User authentication based on 3D facial modeling using partial facial images

Abstract

Systems and techniques for performing user authentication are provided. For example, the process may include acquiring multiple images associated with a user's face and facial expressions, wherein each of the multiple images includes a different portion of the face. An encoder neural network may be used to generate one or more predicted three-dimensional (3D) facial modeling parameters, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the multiple images. A reference 3D facial model associated with the face and facial expressions may be obtained. The error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model may be determined, and the user may be authenticated based on the error being less than a predetermined authentication threshold.

Description

Technical Field

[0001] This disclosure relates generally to systems and techniques for generating three-dimensional (3D) models. For example, aspects of this disclosure relate to user authentication based on three-dimensional facial modeling using partial facial images. Background Technology

[0002] Many devices and systems allow a scene to be captured by generating frames (also called images) and / or video data (including multiple images or frames). For example, a camera or a computing device that includes a camera (e.g., a mobile device (such as a mobile phone or smartphone) that includes one or more cameras) can capture a sequence of frames of a scene. Frame and / or video data can be captured and processed by such devices and systems (e.g., mobile devices, IP cameras, etc.) and can be output for consumption (e.g., displayed on that device and / or other devices). In some cases, frame and / or video data can be captured by such devices and systems and output for processing and / or consumption by other devices.

[0003] Frames can be processed (e.g., using object detection, recognition, segmentation, etc.) to determine the objects present in the frame, which is useful for many applications. For example, a model can be determined to represent the objects in the frame, and this model can be used to facilitate the efficient operation of various systems. Examples of such applications and systems include computer graphics, extended reality (e.g., augmented reality, virtual reality, etc.), image recognition (e.g., face recognition, object recognition, scene recognition, etc.), and feature estimation, etc. Summary of the Invention

[0004] Systems, apparatuses, methods, and computer-readable media for generating one or more models (e.g., one or more 3D facial models) are disclosed. According to at least one example, a method for generating a 3D facial model is provided. The method includes: obtaining a plurality of images associated with a user's face and facial expressions, wherein each corresponding image of the plurality of images includes a different portion of the face; generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; obtaining a reference 3D facial model associated with the face and facial expressions; determining an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticating the user based on the error being less than a predetermined authentication threshold.

[0005] In another example, an apparatus is provided comprising at least one memory (e.g., configured to store data) and at least one processor (e.g., implemented in a circuit) coupled to the at least one memory. The at least one processor is configured to: acquire a plurality of images associated with a user's face and facial expressions, wherein each corresponding image of the plurality of images includes a different portion of the face; generate one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; acquire a reference 3D facial model associated with the face and facial expressions; determine an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticate the user based on the error being less than a predetermined authentication threshold.

[0006] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain a plurality of images associated with a user's face and facial expressions, wherein each of the plurality of images includes a different portion of the face; generate one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; obtain a reference 3D facial model associated with the face and facial expressions; determine an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticate the user based on the error being less than a predetermined authentication threshold.

[0007] In another example, an apparatus is provided comprising: components for acquiring a plurality of images associated with a user’s face and facial expressions, wherein each of the plurality of images includes a different portion of the face; components for generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; components for acquiring a reference 3D facial model associated with the face and facial expressions; components for determining an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and components for authenticating the user based on the error being less than a predetermined authentication threshold.

[0008] In some aspects, one or more of the devices described above are one or more of the following: a camera, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a vehicle or its computing system or device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, a device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors that can be used to determine the position and / or orientation of the device, the state of the device, and / or for other purposes.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all of the drawings, and each claim.

[0010] The foregoing and other features and aspects will become more apparent from the following description, claims and accompanying drawings. Attached Figure Description

[0011] The exemplary aspects of this application are described in detail below with reference to the following figures:

[0012] Figure 1 Examples of a three-dimensional (3D) facial model and a corresponding two-dimensional (2D) facial image overlaid with landmarks projected from the 3D facial model are shown.

[0013] Figure 2 An example head-mounted extended reality (XR) system with a user-facing camera for generating 3D facial models is illustrated according to some examples;

[0014] Figure 3A Examples of partial facial images are shown based on some examples;

[0015] Figure 3B Examples of detailed 3D facial models are provided based on some examples;

[0016] Figure 4 These are illustrations of examples of 3D modeling systems based on some examples;

[0017] Figure 5AThis is a block diagram illustrating an example of training a shape-expression encoder based on some examples to generate predicted shape and expression coefficients for a 3D facial model;

[0018] Figure 5B This is a block diagram illustrating examples of inferences made using a trained shape-expression encoder based on some examples;

[0019] Figure 6 This is a block diagram illustrating an example of training a pose encoder based on some examples to generate predicted pose information;

[0020] Figure 7 This is a block diagram illustrating examples of twin networks that can be trained to determine user identity matching or mismatch based on mouth images, based on some examples.

[0021] Figure 8 This is a flowchart illustrating an example of a process for authenticating a user, based on some examples;

[0022] Figure 9 This is a block diagram illustrating examples of deep learning networks based on some examples;

[0023] Figure 10 This is a block diagram illustrating examples of convolutional neural networks based on some examples; and

[0024] Figure 11 This is a block diagram illustrating an example of a computing system used to implement some of the aspects described herein. Detailed Implementation

[0025] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently, and some may be applied in combination, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation to provide a thorough understanding of the various aspects of this application. However, it will be apparent, however, that the aspects may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0026] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the exemplary aspects will provide those skilled in the art with a description that can be used to implement the exemplary aspects. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0027] The generation of three-dimensional (3D) models of physical objects can be used in many systems and applications, such as extended reality (XR) (e.g., including augmented reality (AR), virtual reality (VR), mixed reality (MR), etc.), robotics, automotive, aviation, 3D scene understanding, object grasping, object tracking, and many other systems and applications. In an AR environment, for example, a user can view an image (also called a frame) that integrates artificial or virtual graphics with the user's natural environment. AR applications allow the manipulation of real-world images to add virtual objects to the image or display virtual objects on a perspective display (making the virtual objects appear to overlay the real-world environment). AR applications can align or register virtual objects with real-world objects (e.g., as observed in the image) in multiple dimensions. For example, real-world objects that exist in reality can be represented using models that are similar to or precisely match the real-world objects. In one example, a model of a virtual vehicle representing a vehicle on a road can be presented through the display of an AR device (e.g., AR glasses, an AR head-mounted display (HMD), or other devices) while the user continues to view their natural environment through the display. The viewer may be able to manipulate the model while viewing a real-world scene. In another example, models with different colors or physical properties in the AR environment can be used to identify and render actual objects located on a table. In some cases, computer-generated copies of artificial virtual objects that do not exist in reality, or actual objects or structures in the user's natural environment, can also be added to the AR environment.

[0028] The increasing use of facial data in applications (e.g., for XR systems, 3D graphics, security, etc.) has led to a significant demand for systems capable of generating detailed 3D facial models (and other object models) efficiently and with high quality. Generating detailed 3D models of objects (e.g., 3D facial models) typically requires expensive equipment and the use of multiple cameras in controlled lighting environments, which hinders large-scale data collection.

[0029] For example, facial data and / or 3D facial models can be used to identify and / or authenticate users of VR, XR, and / or AR systems. In some cases, facial data associated with a user can be captured and used to identify and / or authenticate the user (e.g., using one or more cameras associated with an XR system, one or more HMDs associated with an XR system, etc.).

[0030] Object identification and object authentication (also known as object verification) are two related problems with subtle differences. In some cases, object identification can be defined as a one-to-many problem. For example, facial identification (as an example of object identification) can be used to locate a person from a group of people. Facial identification has many applications, such as for conducting criminal searches. Object authentication can be defined as a one-to-one problem. For example, facial authentication (as an example of object authentication) can be used to check if a person is the person they claim to be (e.g., checking if the claimed person is an individual in an authorized user registration database). Facial authentication has many applications, such as for implementing access control to devices, systems, locations, or other accessible items.

[0031] Using facial identifiers as an illustrative example of object identification, a registry database containing features of registered faces can be used to compare against features of one or more given query face images (e.g., from an input image or frame). Registered faces may include faces registered with the system and stored in a registry database containing known faces. The registered face most similar to the query face image can be determined as a match. Each registered face may be associated with a personal identifier identifying the individual to whom the face belongs. The personal identifier of the matching registered face (the most similar face) is identified as the individual to be identified.

[0032] As described above, object authentication or verification systems can be used to authenticate or verify objects. For example, using facial authentication as an example, an input query image of a face can be compared to a stored or registered representation of an individual's face. Typically, facial authentication requires higher recognition accuracy because it is often associated with access control of a device or system. In this case, no false positives are expected. Facial authentication should be able to identify the individual to whom the face belongs with high accuracy but a low rejection rate. The rejection rate is the percentage of faces that are not recognized due to a match score or classification result below a recognition threshold.

[0033] Biometrics is the science of analyzing specific physical or behavioral characteristics of each individual to authenticate their identity. Biometric-based authentication methods can be used to authenticate people to grant access to devices, systems, locations, or other accessible items. In some cases, biometric authentication allows authentication of an individual based on a set of unique templates (verifiable data). Examples of biometric authentication include facial authentication, fingerprint authentication, voice authentication, and more. For example, facial authentication compares the face of a device user in an input image with known features of the person that user claims to be (e.g., stored in one or more templates) to authenticate that the device user is indeed that person. Similar processes can be performed for fingerprint authentication, voice authentication, and other biometric-based authentication methods.

[0034] Biometric-based user authentication systems typically have at least two steps: a registration step and an authentication step (or testing step). The registration step captures biometric data and stores a representation of the biometric data as a template. This template can then be used in the authentication step. For example, the authentication step can determine the similarity between the template and a representation of the input biometric data, and can use this similarity to determine whether to authenticate the user.

[0035] As mentioned above, facial data and / or 3D facial models can be used to identify and / or authenticate users of VR, XR, and / or AR systems. For example, user identification and / or authentication can be performed based on facial data or images captured using one or more cameras or HMDs associated with an XR system to which the user is being identified or authenticated.

[0036] Performing 3D object reconstruction from one or more images (e.g., to generate a 3D model of an object, such as a facial model) can be challenging. Using a face as an illustrative example of a 3D object, 3D face reconstruction can be difficult due to the need to reconstruct facial geometry (e.g., shape) and facial expression. In some cases, when representing a 3D facial model, it can be viewed from different angles. For example, a 3D facial model in an XR environment can be viewed from different angles by a user moving within that environment. In another example, the 3D facial model can be moved relative to a fixed viewing position (e.g., changing pose). In some cases, maintaining an accurate projection of the 3D facial model onto a 2D image within the field of view (also referred to herein as the viewpoint) can be challenging.

[0037] Furthermore, it can be difficult to accurately reconstruct facial expressions in parts of the face that may undergo significant changes in appearance. In one illustrative example, the eyes of the face may move to extreme gaze directions (e.g., looking to one side, squinting, etc.). In another illustrative example, the upper and lower lips of the mouth are controlled by muscles that allow for a wide variety of difficulties in reconstructing mouth shapes (e.g., smiling, frowning, grimacing, twitching lips, etc.).

[0038] like Figure 1 As illustrated, the white dots overlaid on the 2D facial image 102 can represent projections of the 3D vertices of the 3D facial model 104 back onto the original 2D facial image 102 used to generate the 3D facial model 104. For example, in Figure 1In the illustration, points (which may be called landmarks or 2D landmarks) corresponding to the 3D vertices of the main features of the 3D facial model are depicted as white dots. As shown, landmarks 110, 112, 118, 120, 122, 124, 126, and 128 are included for the contours of the lips, nose, mouth, eyes, eyebrows, etc. Although the 3D facial model 104 may contain a greater number of vertices, for illustrative purposes, only a small number of 3D vertices corresponding to the facial features listed above are shown. Figure 1 In the illustrated example, the marker corresponding to the inner contour 108 of the lower lip of the 3D facial model 104 projected onto the 2D image may include marker 112. Similarly, the marker corresponding to the outer contour 106 of the lower lip of the 3D facial model 104 may include marker 110.

[0039] Figure 1 The outer contour 114 and inner contour 116 of the upper lip of the 3D facial model 104 are also illustrated. In some examples, the markers corresponding to the outer contour 114 of the upper lip may include markers 118 and 124, and the marker corresponding to the inner contour 116 of the upper lip may include marker 120. Additional markers projected from the 3D facial model 104 may include marker 122 corresponding to the left eye, marker 124 corresponding to the right eyebrow, marker 126 corresponding to the entire facial contour, and marker 128 corresponding to the nose. As described above, each of markers 114, 116, 120, 122, 124, 126, and 128 may be generated by projection of the 3D facial model 104 onto the 2D facial image 102.

[0040] Figure 1 A two-dimensional (2D) facial image 102 and a corresponding 3D facial model 104 generated from the 2D facial image 102 using a 3D deformation model (3DMM) are illustrated. In some aspects, the 3D facial model 104 may include a representation of facial expressions in the 2D facial image 102. In one exemplary example, the facial expression representation may be formed by blend shapes. Blend shapes may semantically represent the movement of muscles or parts of facial features (e.g., opening / closing of the jaw, raising / lowering of eyebrows, opening / closing of eyes, etc.). In some cases, each blend shape may be represented by blend shape coefficients paired with a corresponding blend shape vector.

[0041] In some examples, the 3D facial model 104 may include a representation of the facial shape in the 2D facial image 102. In some cases, the facial shape may be represented by facial shape coefficients paired with corresponding facial shape vectors. In some implementations, a 3D model engine (e.g., a machine learning model) may be trained (e.g., during training) to implement a consistent facial shape (e.g., consistent facial shape coefficients) for the 3D facial model, regardless of the pose (e.g., pitch, yaw, and roll) associated with the 3D facial model. For example, when the 3D facial model is rendered as a 2D image for display, projection techniques may be used to project the 3D facial model onto the 2D image. While the 3D model engine implements a consistent facial shape independent of pose, the projected 2D image may have varying degrees of accuracy based on the pose of the 3D facial model captured in the projected 2D image.

[0042] In some cases, it may be difficult to accurately reconstruct a 3D facial model (e.g., including facial model shape and / or facial expression) based on one or more occlusions between the user's face and the camera or imaging device used to capture images for reconstructing the user's 3D facial model. For example, when reconstructing a 3D facial model from images captured using an HMD (e.g., worn by a user in an XR system), an unobstructed frontal view image of the user may not be available. In some aspects, the HMD may include one or more inward-facing cameras that can capture images of the user's face (e.g., or portions thereof) when the HMD is worn by the user. Figure 2 The HMD 202 illustrated may include one or more inward-facing cameras that can be used to capture one or more images of a user's face (e.g., also referred to as a "face image" or "facial image"). In some cases, some (or all) of the facial images captured by the HMD worn by the user may be occluded or otherwise include only a portion of the user's face in the captured facial images. For example, Figure 2 The HMD 202 illustrated herein can use three corresponding inward-facing cameras to capture partial facial images 204A, 204B, and 204C. These partial facial images may also be referred to as “occluded facial images” or “masked facial images”.

[0043] In some respects, an unobstructed frontal view image of the user's face may be unavailable based on the size and / or proximity of the HMD relative to the user's face. For example, the size and / or proximity of the HMD 202 may make it possible for an inward-facing camera included in the HMD 202 to capture partial facial images 204A to 204C, wherein a portion of the user's face is occluded or masked in each of the partial facial images 204A to 204C. In some cases, when reconstructing a 3D facial model of a user using partial facial images (e.g., such as partial facial images captured by an inward-facing camera of the HMD), it may be difficult to accurately capture or determine distinctive facial features that can be used to authenticate the user.

[0044] like Figure 2 As shown, the 3D model generator can generate a 3D facial model 210 using input frames (such as tilted frames 204A, 204B, 204C, and / or 208). Figure 2 As shown, the 3D model fitting engine 206 can also generate textures and / or apply textures to the underlying 3D model (e.g., Figure 1 A 3D facial model 104 is provided to offer a digital representation of the user wearing the head-mounted XR system 202. In an exemplary example, a 3D deformable model (3DMM) may be used to represent the geometry of the user's head.

[0045] Figure 3A An illustrative example is provided of partial facial images 302, 304, and 306, which may correspond to input images provided to a 3D model generator for generating a reconstructed 3D model (e.g., a 3DMM) of a user's face. In some aspects, partial facial images 302, 304, and 306 may be obtained from a head-mounted camera. For example, partial facial images 302, 304, and 306 may be obtained from a camera included in an HMD worn by a user (e.g., worn by a user of an XR system, etc.). In some examples, the head-mounted camera may capture side views of an individual's left eye (e.g., facial image 302) and right eye (e.g., facial image 304), as well as oblique views of the mouth, nose, and jaw (e.g., facial image 306). In some cases, the head-mounted camera may be included in a head-mounted device. In one illustrative example, the head-mounted device may include an XR system. In some cases, it may be desirable to create a 3D facial model (e.g., a 3DMM) of an individual captured in partial facial images 302, 304, and 306. In some cases, the 3D facial model of an individual can be used to generate a 2D image of the individual that provides an accurate representation of the user as the individual's head pose changes.

[0046] In some cases, a 3D facial model (e.g., such as partial facial images 204A, 204B, 204C and / or partial facial images 302, 304, 306) is generated or reconstructed based on partial facial images of the user (e.g., partial facial images 204A, 204B, 204C and / or partial facial images 302, 304, 306). Figure 2 The 3D facial model 210 illustrated may produce unrealistic results in the eye and mouth regions. In some cases, a 3DMM may lack the ability to accurately reproduce a user's mouth and eyes based on the degree or amount of occlusion or masking in the partial facial images used to generate or reconstruct the 3DMM. Unrealistic or inaccurate results in the eye and mouth regions of a 3D facial model (e.g., a 3DMM) reconstructed using a user's facial image may make it difficult to perform user authentication accurately and / or reliably based on the reconstructed 3D facial model.

[0047] Figure 3B An example of a 3D facial model 350 is shown, which can be generated with a high level of accuracy and realistically reflects the shape of an individual's face. Expensive camera equipment that captures an individual's face from multiple angles (e.g., capturing one or more unobstructed views or angles of the individual's face) can be used to generate such models. Figure 3B The image shows a highly detailed 3D facial model. In some cases, the 3D facial model 350 can also be manually edited by a skilled artist to produce an accurate depiction of the individual. The process of generating such a highly detailed 3D facial model can only produce a single model for a specific individual and does not provide a flexible framework for generating 3D models for any individual without a detailed 3D facial model prepared in advance.

[0048] Systems and technologies are needed to generate accurate 3D facial models for a wide variety of facial shapes and expressions, such that these accurate 3D facial models can be used to perform user authentication without requiring highly detailed 3D facial models specifically tailored to each individual to generate accurate representations of facial features. Systems and technologies are also needed to generate accurate 3D facial models for a wide variety of facial shapes and expressions using images of one or more portions of a user's face (e.g., images of portions and / or occlusions captured by a user-facing camera included in a user-worn HMD) as input.

[0049] This document describes systems, apparatus, processes (or methods), and computer-readable media (collectively, the “Systems and Technologies”) for generating accurate 3D facial models based on one or more input images using one or more neural network encoders, wherein the one or more input images include only a portion of a user’s face. For example, one or more user-facing cameras included in an HMD worn by the user may capture one or more input images. HMD images may include, but are not limited to, partial images of the user’s eyes, partial images of the user’s mouth, or partial images of the chin. In some cases, one or more input images may include input images that are partial images (e.g., including a portion but not the entire user’s face) and are tilted images (e.g., frontal views captured at an angle relative to the user’s face or otherwise not depicting the captured portion of the user’s face).

[0050] In some examples, one or more neural networks (e.g., encoder neural networks) may be trained to determine, estimate, or predict shape and expression information based on receiving one or more partial images of a user's face as input. For example, an encoder neural network may be trained to generate multiple three-dimensional (3D) facial modeling parameters based on receiving partial images of the user's eyes (e.g., images of the left and right eyes) and / or partial images of the user's mouth as input. In some aspects, the encoder neural network is trained to generate 3D deformation model (3DMM) parameters or coefficients based on the input images, which can be used to reconstruct or generate a predicted 3D facial model of the user. For example, the encoder neural network may generate multiple 3DMM shape coefficients encoding information indicating the user's identity (e.g., the shape of the user's face), and / or may generate multiple 3DMM expression coefficients encoding information indicating the expressions performed by the user's face.

[0051] In some aspects, one or more neural networks (e.g., encoder neural networks) can be trained to determine, estimate, or predict camera pose information associated with a camera used to capture a partial image of a user's face as input. For example, camera pose information associated with an HMD including a user-facing camera can be predicted by a trained encoder neural network (e.g., a pose prediction network). In some examples, the camera pose information may include 6-DOF pose information associated with one or more user-facing cameras included in the HMD. The predicted pose information may be associated with the user's facial shape and one or more expressions performed by the user (e.g., expressions performed by the user when a partial facial image used as input is captured or obtained).

[0052] While this paper uses the face as an example of a modelable object, the systems and techniques described herein can be applied to generate models for any type of object. Furthermore, for illustrative purposes, the eyes and mouth are used herein as illustrative examples of local feature regions. However, those skilled in the art will understand that the systems and techniques described herein can be performed on other local parts of the face. In one illustrative example, similar systems and techniques can be applied to eyebrows, nose, ears, or any other part of the face. Similar techniques utilizing local features can also be used with objects other than the face at other times.

[0053] The various aspects of the technology described herein will be discussed below with reference to the accompanying drawings. Figure 4 This example illustrates the use of at least one image frame 402 (e.g., from...). Figure 2 An illustration of an example of a 3D modeling system 400 that generates 3D models (e.g., 3D deformable models (3DMMs)) from frames of a user-facing camera in a head-mounted XR system 202. In some examples, one or more image frames 402 may include partial facial images (e.g., images obtained using one or more in-face cameras included in the HMD) Figure 2 Examples of partial facial images 204A, 204B, 204C and / or Figure 3A Examples of facial images 302, 304, and 306 are shown below. Figure 4 As shown, the 3D modeling system 400 includes an image frame engine 404, a 3D model engine 406, and a face reconstruction engine 410. Although the 3D modeling system 400 is shown as including certain components, those skilled in the art will understand that the 3D modeling system 400 may include more than [other components]. Figure 4 The components shown are further components. Components of the 3D modeling system 400 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the 3D modeling system 400 may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or components of the 3D modeling system may include computer software, firmware, or combinations thereof for performing the various operations described herein and / or may be implemented using computer software, firmware, or combinations thereof for performing the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the 3D modeling system 400.

[0054] Image frame engine 404 may acquire or receive image frames 402 captured by an image sensor from a storage device, from memory, from an external source (e.g., a server, external memory accessed via a network, or other external source). In some cases, image frames may be included in a frame sequence (e.g., video, a sequence of standalone or still images, etc.). In one exemplary example, each frame in a frame sequence may include a grayscale component per pixel. Other examples of frames include frames having red (R), green (G), and blue (B) components per pixel (referred to as RGB video including RGB frames), frames having luminance, chroma-blue, chroma-red (YUV, YCbCr, or Y'CbCr) components per pixel, and / or any other suitable type of image. The frame sequence may be captured by one or more cameras, acquired from a storage device, received from another device (e.g., a camera or a device including a camera), or acquired from another source. In some specific implementations, image frame engine 404 may convert or transform the image frame 402 to a grayscale representation or grayscale color space. In some cases, the image frame engine 404 may crop a portion of the image frame 402 corresponding to a face. In some examples, the image frame engine 404 may perform a face detection process and / or a face recognition process to detect and / or recognize faces within the image frame 402. The image frame engine 404 may generate or apply bounding boxes around the face (e.g., Figure 1 The bounding box 130 shown can be used to crop the image data within the bounding box to generate an input image for the 3D model fitting engine 406.

[0055] The 3D model fitting engine 406 may receive an input image (e.g., image frame 402, a cropped bounding box around a face in image frame 402, etc.) from the image frame engine 404. Using the input image, the 3D model fitting engine 406 may perform 3D model fitting techniques to generate a 3D model (e.g., a 3DMM model) of the face (which may include the head of an individual in image frame 402). The 3D model fitting techniques may include solving for the shape coefficient α. i and expression coefficient b j These coefficients can be used to generate a 3DMM representing the face in the input image 402.

[0056] In some examples, the 3D model fitting performed by the 3D model engine 406 may include or be based on pose information related to the head's pose. For example, the pose information may indicate the angular rotation of the head relative to its neutral position. This rotation may be along a first axis (e.g., the yaw axis), a second axis (e.g., the pitch axis), and / or a third axis (e.g., the roll axis). The 3D model generated by the 3D model engine 406 may have different shapes and / or feature coefficients based on specific views or pose information provided as input.

[0057] In some cases, 3D model fitting may also include a focal length for projecting the 3D model onto a 2D image using any suitable projection technique. In some examples, a weakly perspective model may use a focal length generated by the 3D model engine 406 to project the 3D vertices of the 3D model (e.g., a 3DMM) onto a 2D image. In some examples, a fully perspective model may use a focal length generated by the 3D model engine 406 to project the 3D vertices of the 3D model (e.g., a 3DMM) onto a 2D image.

[0058] In another illustrative example, a 3D model can be projected onto an image plane using a fully perspective model. For example, this can be achieved by using the average depth Z in Equation 3. ave Replace with a posed 3D model S P The actual depth Z of each 3D vertex is used to provide full perspective projection. Weak perspective and full perspective modeling techniques for projecting 3D models onto an image plane, as well as any suitable projection techniques, may be used without departing from the scope of this disclosure.

[0059] The facial reconstruction engine 410 can receive coefficients (and any other output parameters such as pose, focal length, etc.) generated by the 3D model engine 406 to generate a 3D model (e.g., a 3DMM). The 3D model can be generated or constructed as a linear combination of an average face (sometimes called a neutral face), facial shape basis vectors, and facial expression basis vectors. The average face can represent an average face that can be transformed (e.g., by shape basis vectors and expression basis vectors) to achieve the desired final 3D facial shape of the 3D model. The facial shape basis vectors can be used to scale the average face. In some cases, the facial shape basis vectors can be used to represent a fat or thin face, a small or large nose, and any adjustments to the basic facial shape. In some implementations, the facial shape basis vectors are determined based on principal component analysis (PCA). In some cases, the facial expression basis vectors can represent facial expressions such as smiling, raising eyebrows, squinting, blinking, frowning, etc.

[0060] A 3D model S (e.g., a 3DMM generated using 3D model fitting techniques) generated by 3D model engine 406 can be a statistical model representing the 3D geometry of an object (e.g., a face). For example, a 3D model (e.g., a 3DMM) can be composed of an average face S0 and facial shape basis terms (also called basis vectors) U. i and facial expression base item V j and facial shape coefficient a i and facial expression coefficient b j It can be represented by a linear combination, for example, as follows:

[0061]

[0062] In the example of formula (1), there are M facial shape coefficients a. i and M facial shape basis vectors U i , where M is an integer greater than or equal to 1. In some specific implementations, the average face S0 and the facial shape basis vector U are... i and facial expression vector V j Each of these can include positional information (e.g., x, y, and z coordinates) of 3D vertices that can be combined to form a 3D model S. In some specific implementations, the facial shape basis vector U i and facial expression vector V j This can be expressed as a positional offset relative to the average face S0, where the coefficient a of the facial shape i and the coefficient b of facial expression j Provide a scaling factor for the corresponding offset vector.

[0063] An illustrative example of facial expression basis vectors is the blend shape. As used herein, a blend shape can correspond to an approximate semantic parameterization of a whole or a part of a facial expression. For example, a blend shape can correspond to a complete facial expression or a “partial” (e.g., “Δ”) facial expression. Examples of partial expressions include raising an eyebrow, closing an eye, moving one side of the face, etc. In one example, a single blend shape can approximate the linearized effect of the movement of a single facial muscle. In some cases, the semantic representation can be modeled as corresponding to the movement of one or more facial muscles.

[0064] In an exemplary example, the 3D model S comprises three thousand 3D vertices. In another exemplary example, M equals 319, which corresponds to 319 facial shape basis vectors U. i and facial shape coefficient a i In some specific implementations, the shape basis vector U i This can include principal component analysis eigenvectors. In some cases, there are N facial expression coefficients b. j There are N facial expression basis vectors, where N is an integer greater than or equal to 1. In some cases, the facial expression vector V... j This can include blended shape vectors. In an exemplary example, N equals 39, which corresponds to 39 facial expression basis vectors V. j and 39 facial expression coefficients b j (For example, 39 blend shapes and 39 blend shape coefficients). In some cases, the result of the linear combination shown in Equation (1) can be a 3D model of a face in a neutral pose (e.g., a 3DMM). In some examples, the 3D model can be rotated using pose information such as yaw, pitch, and roll values ​​to match the pose of the face in image frame 402.

[0065] The facial reconstruction engine 410 can receive pose information from the 3D model engine 406 to generate a 3D model (e.g., a 3DMM) and a projection of the 3D model into a 2D image space. A 3D model S (facial model) can be generated according to formula (1) above. As described above, the 3D model S produces a facial model with a neutral pose. In some cases, the 3D model S can be rotated according to formula (2) below to create a posed 3D model S. P :

[0066] S P =R(α,β,γ)S (2)

[0067] Where R(α,β,γ) is a 3×3 rotation matrix with rotation angles α,β,γ.

[0068] In some cases, any suitable projection technique can be used to project a 3D model onto an image plane. In an exemplary example, weak perspective projection can be used to project a 3D model onto an image plane. The following formula (3) provides how a posed 3D model S can be projected onto an image plane using weak perspective projection. P X and Y components of each 3D vertex coordinate x and y coordinates projected onto a 2D image Example above:

[0069]

[0070] Where f is the predicted focal length output by the 3D model engine 406, and Z ave This is the average depth (or distance) of the face from the weak perspective camera. It can be applied to pose-oriented 3D models. P Each 3D vertex performs the projection of formula (3).

[0071] In some cases, the 3D model engine 306 may include a machine learning model (e.g., a deep neural network, such as a convolutional neural network (CNN), a general adversarial network, an inception network, etc.). The 3D model engine 406 may receive an image frame 402 (and in some cases, an input view) as input. In some cases, the image frame 402 and the view / pose information may be one of multiple images and corresponding views in a training dataset. The training dataset may include a large number of images, including a variety of facial shapes, expressions, and views (e.g., including one or more views of each face depicted in the images), to train the 3D model engine 406 to generate 3D facial models (e.g., 3DMMs) of many kinds of faces and views. In an exemplary example, the training dataset may include one million images and views, two million images and views, or any other suitable large number of images and views. In some cases, the 3D model engine 406 may be trained using the same training dataset through multiple iterations (also called epochs). In some cases, the training dataset used to train the 3D model engine 406 may include images, views, and labels indicating known features in the frames (e.g., during supervised training). In some cases, the training dataset may include annotated ground truth 2D landmark information corresponding to the location of facial features for each image frame 402. In some cases, a landmark algorithm may be used to compute the 2D landmark information from each image frame 402.

[0072] During training, the 3D model engine 406 may utilize one or more loss functions to analyze the error in the 3D model parameters (e.g., the coefficients of Equation (1)). The loss function L can be used to determine the amount of error in the 3D model generated by the 3D model engine 406 relative to the baseline ground truth data provided as input to the 3D model engine 406. An example loss function includes mean squared error (MSE). MSE is defined as... It calculates the sum of half the squares of the actual answer (e.g., the baseline ground truth landmark position) minus the predicted (output) answer. Another example loss function includes normalized average error, where the error between the baseline ground truth landmark and the landmark predicted by the 3D model engine 406 can be normalized by the predicted eye distance of the face.

[0073] Example 2D vertex loss function is shown in the following formula (6):

[0074]

[0075] Here, M is the difference between the 2D landmark positions projected from the baseline ground truth 3D model and the 2D landmark positions projected from the corresponding vertices generated by the 3D model engine 406. M (e.g., an integer) is the number of 2D landmarks. As follows (e.g., regarding...) Figures 5A to 7 To describe this in more detail, v can represent one of several partial facial images provided as input. For example, v can represent one of three HMD views of a user's face (e.g., eyes or mouth). v It can be the set of vertices visible from view v.

[0076] The example 3D vertex loss function is shown in the following formula (7):

[0077]

[0078] Here, It is the difference between the 3D vertex position from the baseline true 3D model and the 3D vertex position of the corresponding vertex generated by the 3D model engine 406, and N (integer) is the number of 3D vertices.

[0079] Boundary marker loss L Landmarks It can be a weighted sum of different 2D and / or 3D landmark loss components, as shown in the following formula (8):

[0080] L Landmarks =λ 3D L 3D +λ 2D L 2D (8)

[0081] Where λ 3D and λ 2D These are the weighting coefficients used for 3D landmark loss and 2D landmark loss, respectively.

[0082] An example regularization loss function is shown in Equation (9), and an example clamping loss function is shown in Equation (10):

[0083]

[0084] Here, α j It is the shape factor. (b) j This can represent facial expression coefficients. In some examples, clamping loss can be used to constrain shape coefficients within fixed boundaries (e.g., to preserve semantic meaning):

[0085]

[0086] Here, β UB,j and β LB,j These are facial expression coefficients b generated by 3D model engine 406. j The upper and lower bound values ​​of each facial expression coefficient in the dataset.

[0087] During inference (e.g., after training is complete), the 3D model engine 406 can output a 3D facial model (e.g., a 3DMM) based on one or more input image frames 402. This is because the shape coefficient α... i Constraints (e.g., the shape consistency loss L in formula (4) ShapeConsistency The shape coefficients are omitted from the loss function shown in Equation (5), and the resulting output 3D facial model for an individual may have different shape coefficients depending on the view (e.g., pose) of the generated 3D facial model. However, eliminating the constraint on shape consistency yields 3D facial models that can generate 2D projected images from different poses.

[0088] Figure 5A A block diagram illustrating an example training process for training a shape-expression encoder to generate shape and expression coefficients that can be used to recreate or generate a 3D facial model (e.g., a 3DMM) of a user's face based on a partial input image of the user's face. In an exemplary example, a shape-expression encoder 522 may be trained to generate shape and expression coefficients 524 based on receiving multiple HMD images 512 as input. In some aspects, the shape-expression encoder 522 may be a neural network encoder, although various other neural networks, machine learning networks, and / or machine learning algorithms and architectures may also be utilized.

[0089] In some aspects, the shape and expression coefficients 524 generated by the trained shape-expression encoder 522 may include multiple shape coefficients of the 3D facial model (e.g., 3DMM) and multiple expression coefficients of the 3D facial model (e.g., 3DMM). For example, the trained shape-expression encoder 522 may be used to generate multiple shape coefficients α. s and multiple facial expression coefficients α e In some examples, the trained shape-expression encoder 522 can generate shape and expression coefficients 524, which include up to 210 shape coefficients α. j And up to 30 facial expression coefficients b j .

[0090] Shape and expression coefficients 524 may be associated with or used to generate a single 3D facial model 552, which is associated with multiple HMD images 512. In some respects, the 3D facial model 552 generated or reconstructed based on shape and expression coefficients 524 may also be referred to as a predicted 3D mesh, a predicted 3D facial model, and / or a predicted 3D facial model.

[0091] For example, the shape and expression coefficient 524 can be used to generate a predicted 3D facial model, which is a 3D deformable mesh (3DMM). The 3D facial model P predicted by the 3DMM can be given by the following equation:

[0092]

[0093] Here, This indicates an average face (e.g., sometimes referred to as a neutral face); α s This represents up to 219 facial shape coefficients (e.g., which can be determined or predicted by a trained shape-expression encoder 522 and included in the output coefficients 524); α e This represents up to 39 facial expression coefficients (e.g., which can be determined or predicted by a trained shape-expression encoder 522 and included in the output coefficients 524); A s Denotes the basis vectors of the facial shape; and A e This represents the basis vector of facial expressions.

[0094] Facial shape coefficient α s The number can be equal to the facial shape basis vector A s The number and facial expression coefficient α e The number can be equal to the facial expression basis vector A e The number. For example, 219 shape coefficients α. s It can be applied to 219 facial shape basis vectors A s And it was used for individual-based facial shape coefficient α s The associated unique shape makes the average face of formula (11) Deformation. Similarly, 39 facial expression coefficients α e It can be applied to 39 facial expression basis vectors A e And it was used based on the facial expression coefficient α e The expression represented or associated with the facial expression coefficient makes the average facial expression in formula (11) equal to the expression in formula (11). Deformation.

[0095] In an exemplary example, the system and techniques can train a shape-expression encoder 522 to estimate (e.g., predict) both shape and expression coefficients using multiple partial HMD images as input. For example, the partial HMD images as input can be compared with... Figure 5AThe RGB partial HMD images 512 depicted are the same or similar. As illustrated, the RGB HMD image 512 may include multiple partial HMD images, each including a different part of the user's face. The first HMD image 512a may be an image of the user's right eye, the second HMD image 512b may be an image of the user's left eye, and the third HMD image 512c may be an image of the user's mouth or chin (e.g., the mouth or chin region or portion of the user's face).

[0096] In some aspects, HMD training data images 512 (e.g., 512a, 512b, 512c) can be generated based on a ground truth frontal image 502. For example, the ground truth frontal image 502 can be a single 2D image of an individual taken from a frontal or frontal viewpoint. The frontal image 502 can be provided to the synthetic HMD image rendering engine 510 along with HMD pose information 504. Using the HMD pose information 504, the synthetic HMD image rendering engine 510 can generate multiple partial HMD images (e.g., such as partial HMD images 512). The HMD pose information 504 can include one or more HMD pose parameters that will be simulated by the synthetic HMD image rendering engine 510 when generating multiple partial HMD training data images 512. In some aspects, the HMD pose information 504 can include 6-DOF HMD pose information (e.g., up / down, left / right, forward / backward, pitch, roll, yaw).

[0097] In some respects, one or more (or all) of the HMD training data images 512 generated by the synthetic HMD image rendering engine 510 may be tilted relative to the ground truth frontal image 502. For example, Figure 5A Examples include a frontal HMD image 512c including the user's mouth, a tilted HMD image 512a including the user's right eye, and a tilted HMD image 512b including the user's left eye. By generating HMD training data images 512 tilted relative to a baseline frontal image 502, the synthetic HMD image rendering engine 510 can simulate or recreate a perspective view of a portion of the face captured by a user captured by a user-facing camera included in the HMD and capturing an image of the user's face while wearing the HMD.

[0098] A ground truth frontal image 502 may be provided to a ground truth stream engine 532 that generates a ground truth 3D mesh 562. The ground truth 3D mesh 562 may be used as a ground truth 3D facial model (e.g., a 3DMM) for training a shape-expression encoder 522 (e.g., based on minimizing the 3D vertex distance loss 545 between the predicted 3D mesh 552 and the ground truth 3D mesh 562, which is generated based on shape and expression coefficients 524 predicted by the shape-expression encoder 522, as will be described in more detail below). In some examples, the ground truth stream engine 532 may be a pre-trained neural network or other machine learning network, encoder, etc., which is pre-trained to generate 3D meshes (e.g., such as the ground truth 3D mesh 562) from a frontal facial image (e.g., such as the ground truth frontal image 502).

[0099] In some cases, the ground truth frontal image data 502 may include a single image, as described above. In some examples, multiple cameras may be used to capture or obtain the ground truth frontal image data 502. For example, the ground truth frontal image data 502 may include depth information determined using multiple cameras. In examples where the ground truth frontal image data 502 is multi-camera data, the ground truth streaming engine 532 may be provided as a pre-trained neural network, machine learning network, encoder, etc., which is pre-trained to generate 3D meshes (e.g., such as ground truth 3D mesh 562) from the multi-camera frontal image data. In some aspects, various other mesh generators may be used to provide the ground truth streaming engine.

[0100] In an exemplary example, a reference truth 3D mesh 562 generated by the reference truth stream engine 532 can be used to retopology the mesh associated with the compositing HMD image rendering engine 510. For example, the compositing HMD image rendering engine 510 can receive a reference truth frontal image 502, a reference truth 3D mesh 562 (e.g., generated from the reference truth frontal image 502), and HMD pose information 504 as input. Using the HMD pose information 504 and the reference truth 3D mesh 562, the compositing HMD image rendering engine 510 can project the 2D frontal image 502 onto the reference truth 3D mesh 562. The HMD pose information 504 can then be used to back-project the 2D frontal image 502 from the reference truth 3D mesh 562 onto the desired 2D HMD image portions 512a to 512c.

[0101] As previously described, the shape-expression encoder 522 can generate predicted shape and expression coefficients 524 based on receiving an RGBHMD image 512 generated from a reference ground truth frontal image 502 as input. In some aspects, the shape-expression encoder 522 can be trained at least in part based on regularization and clamping loss 525. For example, the loss 525 may include a regularization loss L given by the following formula: reg :

[0102]

[0103] The regularization loss of Equation (12) can be determined as the sum of the shape and expression coefficients 524 predicted by the shape-expression encoder 522 (e.g., Equation (12) includes a sum from 1 to 258, representing 219 facial shape coefficients α). s +39 facial expression coefficients α e =258 total coefficients α j ).

[0104] Additionally, the loss 525 may include the clamping loss L given by the following formula. clamp :

[0105]

[0106] The clamping loss of formula (13) can be used to apply the 219 facial shape coefficients α s (For example, based on the clamping loss being determined as the sum obtained over j = 1, ..., 219) constraints are placed between fixed boundaries to preserve semantic meaning. For example, the facial shape coefficient α s (For example, α in formula (13) j It can be constrained at the upper boundary β UB,j With lower boundary β LB,j The upper and lower boundaries preserve the semantic meaning of the facial shape coefficient, which can be predicted by the shape-expression encoder 522, because a facial shape coefficient with values ​​that are too high or too low may represent an average facial shape. The deformation is unrealistic and should not be allowed as output. In some cases, the upper and lower bounds of the clamping loss in Equation (13) can prevent overfitting and preserve the semantic meaning captured by the facial shape coefficients (e.g., the average facial shape coefficient). The unique deformation, which can be applied to recreate the user's unique 3D facial model.

[0107] Return to Figure 5AThe illustrated example training process for training the shape-expression encoder 522, as shown in the figure, involves shape and expression coefficients 524 being used to generate a predicted 3D mesh 552. In an illustrative example, the predicted 3D mesh 552 could be a 3DMM generated using formula (11). For example, this could be achieved by using α included in the shape and expression coefficient predictions 524. s and α e The coefficient values ​​are used as input to Equation (11) to generate the predicted 3D mesh 552, which is the output of the shape-expression encoder 522 during a given training iteration. This system and technique may additionally use 3DMM average face and basis vectors 540 (e.g., combined with shape and expression coefficient predictions 524) to generate the predicted 3D mesh 552. For example, the 3DMM average face and basis vector information 540 may include average face... (For example, where the same average face is used to generate both the baseline ground truth 3D mesh 562 and the predicted 3D mesh 552), multiple face shape basis vectors A s and multiple facial expression basis vectors A e In some respects, the same average face, facial shape basis vectors, and facial expression vectors can be used to generate the predicted 3D mesh 552 and the baseline ground truth 3D mesh 562. In some cases, the same average face, facial shape basis vectors, and facial expression vectors can be used across all training iterations performed to train the shape-expression encoder 522. In some examples, the average face is included in the 3DMM average face and basis vector information 540. These can be general (e.g., individual-independent) average faces and basis vectors.

[0108] The predicted 3D mesh 552 is a 3D facial model (e.g., a 3DMM), which is generated using shape and expression coefficients 524, predicted by a shape-expression encoder 522 using only a portion of the HMD image 512 as input. In some respects, the shape-expression encoder 522 can encode individual-specific details into the facial shape coefficients α. s and / or facial expression coefficient and α e (For example, it is included in the predicted coefficient output 524 generated by the shape-expression encoder 522). For example, individual-specific details may be encoded in the shape and expression coefficients 524, at least in part, based on the fact that the 3DMM average facial and basis vector information 540 is general rather than personalized.

[0109] The 3D vertex distance loss 545 can be determined as the error for a given training iteration, since the 3D vertex distance loss 545 can represent the difference between the predicted 3D mesh 552 and the ground truth 3D mesh 562. In an exemplary example, the 3D vertex distance loss 545 can be the same as or similar to the 3D vertex distance loss in Equation (7). In some respects, as the shape-expression encoder 522 is trained to more accurately predict the shape and expression coefficients 524 of a given set of input HMD training images 512, the resulting predicted 3D mesh 552 will become increasingly similar to the ground truth 3D mesh 562, and the 3D vertex distance loss 545 will decrease.

[0110] In an exemplary example, the shape-expression encoder 522 may be trained, at least in part, based on minimizing the regularization and clamping loss 525 and the 3D vertex distance loss 545 through multiple training iterations. Each training iteration may utilize different sets of benchmark ground truth frontal images 502 and / or HMD training data images 512 (e.g., if the same benchmark ground truth frontal image 502 is used for multiple training iterations, the synthetic HMD image rendering engine 510 may be configured to generate different HMD images from the same benchmark ground truth frontal image 502 for each training iteration).

[0111] In some aspects, the shape-expression encoder 522 may additionally or alternatively be trained at least in part based on the 2D vertex distance loss 575. In one exemplary example, the 2D vertex distance loss 575 may be the same as or similar to the 2D vertex distance loss of Equation (6). For example, the 2D vertex distance loss 575 may be computed based on projecting the predicted 3D mesh 552 onto the HMD image (e.g., indicated at 554) and projecting the ground truth 3D mesh 562 onto the same HMD image (e.g., indicated at 564). In some examples, the projection operations 554 and 564 may be performed using HMD pose information 504 (e.g., the same HMD pose information 504 used to generate the HMD image 512 at the synthetic HMD image rendering engine 510). In one exemplary example, the shape-expression encoder 522 may be trained based on minimizing the sum of the regularization and clamping loss 525, the 3D vertex distance loss 545, and the 2D vertex distance loss 575 in each training iteration.

[0112] In an exemplary example, the shape-expression encoder 522 may be trained using multiple training iterations, wherein some (or all) of the training iterations utilize different sets of ground truth frontal images 502 and / or partial HMD images 512. For example, multiple training iterations may be performed using ground truth frontal images 502 (e.g., and therefore partial HMD images 512) that depict, represent, many different individuals, different lighting conditions, different facial appearances, different facial expressions, etc., or are associated with these.

[0113] Figure 5B This is an illustration of an example of an inference process 500b that can be performed using a trained shape-expression encoder 522 (e.g., in completing the above discussion on...). Figure 5A The training process of the shape-expression encoder 522 is then executed. Figure 5B The inference process. In some respects, it can be performed by XR devices or systems, VR devices or systems, AR devices or systems, and / or other computing devices associated with an HMD worn by the user. Figure 5B The inference process 500b is illustrated below. As will be described in more detail below, an XR device or system can perform user authentication based on the inference process 500b.

[0114] In some examples, the user's facial image captured by the HMD may be an infrared (IR) or near-infrared (NIR) image, rather than the synthetic RGB HMD image 512 used to train the shape-expression encoder 522. For example, because the internal volume of the HMD (e.g., the volume surrounded by the HMD and adjacent to the user's eyes) is a low-light environment, a user-facing camera included inside the HMD may not be able to capture an RGB facial image. The user-facing camera included in the HMD may therefore capture an IR image, a NIR image, or a separate portion of the HMD image rendered in a grayscale color space or other non-RGB color space.

[0115] like Figure 5B As illustrated, user authentication can be performed based on (e.g., from an HMD worn by the user) multiple partial HMD images 506 of the user's face. Multiple partial HMD images 506 can be captured as IR or NIR images. In some cases, the partial HMD images 506 captured inside the HMD can be IR or NIR images (e.g., such as left eye image 506a and right eye image 506b, respectively), while the partial HMD images 506 captured outside the HMD can be RGB images (e.g., such as mouth image 506c). In some cases, all partial HMD images 506 can be IR or NIR images, such as... Figure 5B As depicted in the text.

[0116] After acquiring multiple partial HMD images 506, the NIR-to-RGB transformation engine 505 can be used to convert the HMD images 506 (e.g., as needed) from the NIR color space (where the HMD image was captured) to the RGB color space (where the shape-expression encoder 522 was trained). In some respects, when one or more partial HMD images of the partial HMD images 506 are captured in a color space that is not the NIR color space but is also different from the color space in which the shape-expression encoder 522 was trained, Figure 5B The component described as NIR to RGB transformation engine 505 may be replaced by a suitable transformation engine for converting the color domain of a portion of the HMD image 506 in which it was originally captured (e.g., captured by a user-facing camera included in the HMD) to the color domain (e.g., RGB or others) in which the shape-expression encoder 522 was trained.

[0117] The output of the NIR to RGB transformation engine 505 is a set of RGB HMD images 514, which correspond to the RGB color space (or other color space) in which the shape-expression encoder 522 is trained. For example, the NIR HMD left eye image 506a is transformed into the RGB HMD left eye image 514a, the NIR HMD right eye image 506b is transformed into the RGB HMD right eye image 514b, and the NIR mouth image 506c is transformed into the RGB HMD mouth image 514c.

[0118] Inference (and subsequent user authentication) can be performed by feeding the RGB HMD image 514 as input to a trained shape-expression encoder 522. The trained shape-expression encoder 522 can generate multiple shape and expression coefficients 524 as output, which correspond to the input RGB HMD image 514, as described above. Figure 5A As stated above.

[0119] Using 3DMM average face and basis vectors 540 (e.g., generic or individual-independent 2DMM average face and basis vectors), predicted shape and expression coefficients 524 can be used to generate a predicted 3D mesh 550 corresponding to the user's facial data represented in an RGB HMD image 514, which is fed as input to a trained shape-expression encoder 522 (e.g., also as described above regarding...). Figure 5A The above).

[0120] In an exemplary example, the predicted 3D mesh 550 can be used to perform user authentication. For example, the user authentication may include an initial registration phase where the user is prompted to register or enroll a set of different facial expressions while wearing the HMD. One or more sets of HMD images (e.g., NIR HMD image 506) are captured for each prompt's facial expression for the user to register / enroll, and a registered 3D mesh (not shown) is generated and saved for a unique combination of the user's face and the prompt's facial expression. The above description can be used... Figure 5B The same process is used to generate the registered 3D mesh.

[0121] In some aspects, the trained shape-expression encoder 522 can generate shape and expression coefficients 524 that are unique for each user face shape-user facial expression combination provided during the registration process. In some examples, the registration phase may include storing at least the shape and expression coefficients 524 for the user's registered facial expression combinations. In some aspects, the registration phase may include storing the shape and expression coefficients 524 and the resulting 3D mesh generated using the shape and expression coefficients 524, also for each of the user's registered facial expression combinations. In some aspects, only the resulting 3D mesh may be stored and used for subsequent authentication. In some examples, the registered expressions associated with the user may include multiple different expressions, such as neutral, smiling, angry, etc. In some cases, a single reference may be stored for some (or all) of the registered expressions. In some examples, multiple references may be stored for some (or all) of the registered expressions.

[0122] During the authentication phase, the user may be prompted (e.g., while wearing the HMD) to perform one or more previously registered expressions, as described above. In some examples, the user may be prompted to perform a single expression, multiple expressions from a registered set of expressions, or all expressions from a registered set of expressions. Authentication may be performed based on captured HMD image data of the user performing the prompted expressions (e.g., NIR HMD image 506) and using the captured HMD image data to generate shape and expression coefficients 524 (and / or corresponding 3D mesh 550) of the captured HMD image data.

[0123] In an exemplary example, the shape and expression coefficients 524 predicted during inference of captured HMD image data associated with a user performing a requested expression can be compared with the registered shape and expression coefficients generated during registration (e.g., also by a trained shape-expression encoder 522) when the user is prompted to perform the same expression. In some aspects, the shape and expression coefficients 524 determined during authentication can be validated against a registration objective, which includes the registered shape and expression coefficients generated by the trained shape-expression encoder 522 during registration for the same expression. In some examples, authentication can be based on determining that one or more error values ​​determined between the shape and expression coefficients 524 and the registered shape and expression coefficients are below one or more corresponding predetermined thresholds.

[0124] For example, the weighted absolute error m1 of the shape and expression coefficients can be determined as:

[0125]

[0126] The weighted average vertex error m2 in 3D can be determined as:

[0127]

[0128] The weighted average vertex error m3 in 2D can be determined as:

[0129]

[0130] In some cases, each of the error values ​​in formulas (14) to (16) may be associated with a different predetermined authentication threshold, wherein the user is authenticated if the error value is less than or equal to its corresponding predetermined authentication threshold. In some examples, the error values ​​in formulas (14) to (16) may be associated with the same predetermined authentication threshold, wherein the user is authenticated if at least one of the error values ​​is less than or equal to a single predetermined authentication threshold. In some cases, the user will be authenticated only if multiple or all three error values ​​in formulas (14) to (16) are less than or equal to a predetermined authentication threshold.

[0131] In another illustrative example, a neural network (e.g., a neural network encoder) can be trained to process one or more input HMD images (e.g., such as those mentioned above). Figure 5A and Figure 5B The input HMD images 512, 506, and / or 514 are used to estimate the HMD pose. For example, a neural network encoder can be trained to estimate (e.g., predict) the HMD pose information associated with the main camera of the HMD. Figure 6This is an illustration of an example training process 600 for training a pose network 620 (e.g., also called a pose encoder, neural network pose encoder, etc.) to generate a predicted HMD pose 640 based on an input of an RGB HMD image 612.

[0132] In some respects, Figure 6 The example training process 600 illustrated can be compared with... Figure 5A The example training process illustrated in 500a is the same as or similar to that in which Figure 5A Shape-Face Encoder 522 by Figure 6 The attitude encoder 620 in the image can be replaced. For example, in some aspects, the reference true frontal image 602 can be used with... Figure 5A The baseline true value of the frontal image 502 is the same as or similar to that of the composite HMD image rendering engine 610. Figure 5A The composite HMD image rendering engine 510 is the same as or similar to the baseline truth stream engine 632. Figure 5A The benchmark truth stream engine 532 is the same as or similar to the benchmark truth 3D mesh 662. Figure 5A The reference true 3D mesh 562 is the same as or similar to the reference true 3D mesh 562; and / or the RGB portion of the HMD image 612 may be the same as the reference true 3D mesh 562. Figure 5A The RGB portions of the HMD image are the same as or similar to those in image 512.

[0133] Based on receiving multiple RGB partial HMD images 612 as input, a pose network 620 can be trained to generate HMD pose information 640, wherein the HMD pose information 640 is a predicted HMD pose corresponding to the HMD used to capture one or more HMD images in the HMD images 612 and / or a predicted HMD pose corresponding to at least one user-facing camera included in the HMD and used to capture the HMD images 612. For example, the predicted HMD pose information 640 may be a predicted 6-DOF HMD pose associated with the main camera used to capture the HMD images 612.

[0134] The pose encoder 620 can be trained at least in part based on the L1 loss 645, which can be calculated or otherwise determined between the predicted HMD pose 640 and the corresponding ground truth HMD pose 604. The ground truth HMD pose 604 can be compared with... Figure 5A The HMD pose 604 is the same as or similar to that of the composite HMD image rendering engine 610, as illustrated in the figure. Figure 5A exemplified).

[0135] The baseline ground truth HMD pose 604 can be further provided as input to an HMD projection operation 664, which can be performed to project the baseline ground truth 3D mesh 662 onto the HMD image (e.g., to correspond with the above regarding...). Figure 5A The projection operation 564 illustrated is in the same or similar manner.

[0136] The L1 loss 645 indicates the difference or error between the HMD pose 640 predicted by the pose encoder 620 and the baseline ground truth HMD pose 604, which will be predicted by the pose encoder 620 when it is error-free or trained to 100% accuracy. Based on minimizing the L1 loss 645 during multiple training iterations performed on the pose network 620, the pose encoder 620 can be trained to accurately (or more accurately) generate the predicted HMD pose information 640 for a given input of the RGB partial HMD image 612.

[0137] In some cases, an additional 2D vertex loss of 655 can be used to train the pose encoder 620. For example, it can be used with... Figure 5A The 2D vertex loss 575 is determined in the same or similar manner as the 2D vertex loss 655. In the context of training the pose encoder 620, the predicted HMD pose 640 for a given training iteration can be combined with the ground truth 3D mesh 662 and used to project the ground truth 3D mesh 662 onto the HMD image 612.

[0138] For example, at projection 650, the predicted HMD pose information 640 generated by the pose network 620 can be combined with known HMD camera-specific information (e.g., the relative position of the HMD camera facing the user, focal length, etc.) and used to generate a reference ground truth 3D mesh 662 projected onto the HMD image 612.

[0139] By configuring and generating a projection of the reference ground truth 3D mesh 662 onto the HMD image 612 using the reference ground truth HMD pose information 604 (e.g., together with the same known HMD camera inherent information), a similar projection of the reference ground truth 3D mesh 662 onto the HMD image 612 can be generated at projection 664.

[0140] If the predicted HMD pose information 640 is the same as the ground truth HMD pose information 604, then the 2D vertex projections 650 and 664 will be the same, and the loss or error between the two projections (e.g., the 2D vertex distance loss 655) will be zero. If the predicted HMD pose information 640 is not the same as the ground truth HMD pose information 604, then the corresponding 2D vertex projections 650 and 664 will be different, and the 2D vertex distance loss 655 will have a non-zero value. At least in part based on minimizing the 2D vertex distance loss 655 between the 2D projection of the ground truth 3D mesh 662 using the predicted HMD pose information 640 and the 2D projection of the ground truth 3D mesh 662 using the ground truth HMD pose information 604, the pose network 602 can be trained to generate the predicted HMD pose information 640 accurately (or more accurately) using only multiple RGB partial HMD images 612 as input.

[0141] In an exemplary example, a trained pose encoder 620 can be used, similar to the one described above. Figure 5A and Figure 5B The method described herein includes using a trained shape-expression encoder 522 to perform user authentication. For example, user authentication based on a trained pose encoder 620 may include a user registration phase, which is the same as the user registration phase described above regarding user authentication based on a trained shape-expression encoder 522.

[0142] The authentication phase using the trained pose encoder 620 can be performed by initially prompting the user to perform one or more previously registered expressions (e.g., the same as described above regarding the authentication phase using the trained shape-expression encoder 522). In response to the user performing the prompted expression, the system and technology can capture multiple partial HMD images of the user's face while performing the prompted expression. For example, these multiple partial HMD images can be compared with those described above regarding... Figure 5B The multiple NIR portion HMD images 506 are identical. An NIR-to-RGB conversion engine can be used to match the above description. Figure 5B The NIR-to-RGB conversion engine 505 and the resulting RGB partial HMD image 514 are transformed or converted into multiple RGB partial HMD images in the same or similar manner.

[0143] The trained pose encoder 620 can generate estimated or predicted HMD pose information 640 (e.g., predicted 6DOF HMD pose) based on multiple received RGB partial HMD images as input. Using the predicted HMD pose information 640 and known HMD camera-specific information, a registered mesh (e.g., determined during the registration phase for the current authentication combination of the user's face and expression) can be projected onto the multiple RGB partial HMD images. For example, projection can be performed in the same manner as described above, where the registered mesh for the cues of facial expressions is replaced. Figure 6 The baseline ground truth 3D mesh 662 depicted in the training process.

[0144] In some cases, projecting a registered mesh onto multiple RGB partial HMD images can be performed by projecting one or more registered mesh landmarks onto an RGB partial HMD image and then using a 2D landmark detector to determine one or more detected landmarks in the RGB partial HMD image. For example, landmarks (e.g., among the registered mesh landmarks and detected landmarks) may include facial features and / or expression features, such as the corners of the eyes or mouth.

[0145] In an exemplary example, the detected landmarks determined for the portion of the HMD image being authenticated can be compared with projected landmarks generated by projecting a registered 3D mesh onto the 2D portion of the HMD image using predicted HMD pose information 640. Because the registered 3D mesh is target user specific, the registered mesh will not be aligned with the landmarks of different users (e.g., when different users with different facial shapes perform the same expression, the projected landmarks of different users will be different from the projected landmarks of the user being authenticated).

[0146] In some cases, user authentication can be performed based on the fact that the average landmark error or difference between the landmarks of the 2D projection (e.g., generated by projecting a 3D mesh of the user-expression registration onto a 2D partial HMD image using predicted HMD pose information 640) and the corresponding landmarks included in the partial HMD image captured by the user-facing camera of the HMD is less than or equal to a predetermined threshold. For example, the predetermined threshold can be the average 2D landmark error m4 given by the following formula:

[0147]

[0148] In another exemplary example, a twin network can be used to perform user authentication based on multiple partial HMD images. For example, Figure 7This is a diagram illustrating an example training process for training a Siamese network 700 to determine the similarity between a target HMD mouth image 710a and a source HMD mouth image 710b. In some aspects, the target HMD mouth image 710a may be an HMD mouth image captured during user registration (e.g., in response to a prompting of the user to perform a specified expression, wherein the registered HMD mouth image corresponds to the specified expression and the user and is registered). The source HMD mouth image 710b may be an HMD mouth image captured during the authentication phase (such as when the user is wearing the HMD (e.g., as described above)).

[0149] In some respects, the Siamese network 700 can be trained and implemented (e.g., implemented to perform inference for user authentication) using only HMD mouth images. As previously mentioned, HMD images of the eyes (e.g., such as those about...) Figures 5A to 6 The images described above can only be captured in the IR or NIR domain and are generally not available in the RGB domain. However, based on the fact that HMDs worn by users typically obscure or cover the user's eyes but not their mouth, HMD images of the mouth can be captured in the RGB domain.

[0150] In some respects, the Siamese network 700 can be trained to perform user authentication based on skin texture and color information captured in RGB mouth images (e.g., source RGB mouth image 710a and target RGB mouth image 712b, respectively). A pair of identical mouth encoders 720a, 720b (e.g., capable of processing information similar to that described above) can be used to perform user authentication. Figures 5A to 6 The encoder described is provided in the same or similar manner as a neural network, neural network encoder, etc., to perform training. In an exemplary example, a Siamese network 700 including paired mouth encoders 720a, 720b can be trained over multiple training iterations using multiple paired RGB mouth images. For example, the training data pair of RGB mouth images may include a source RGB mouth image 710a and a target RGB mouth image 712b, which may or may not belong to the same entity. The training data pair may be labeled with an indication of whether the source RGB mouth image and the target RGB mouth image (e.g., 710a, 712b) belong to the same entity. After multiple training iterations, a contrast loss 725 can be determined between the encoded outputs of the first Siamese mouth encoder 720a (e.g., the source encoder) and the second Siamese mouth encoder 720b (e.g., the target encoder), such that the Siamese network 700 is trained to detect identity matching or mismatch between the source RGB mouth image 710a and the target RGB mouth image 712b.

[0151] It can perform authentication based on a trained Siamese network 700a to include the above-mentioned... Figures 5A to 6 The same registration phase applies. A user's set of expressions (e.g., neutral, smiling, angry, etc.) can be registered by prompting the user to perform a specified expression. The captured RGB mouth data can be provided to the trained Siamese network 700 and fixed as a target (e.g., fixed as a target RGB mouth image and / or fixed as a target for match / mismatch detection performed by the trained Siamese network 700). In the authentication phase, the user can be prompted to perform one or more previously registered expressions. The HMD mouth image of the user performing the requested expression for authentication can be captured and fixed as a source (e.g., fixed as a source RGB mouth image and / or fixed as a source for match / mismatch detection performed by the trained Siamese network 700). During authentication, the source-target pair of each facial expression prompted by the user can be passed as input to a trained Siamese network 700a (e.g., an RGB mouth image fixed as the target can be passed to a target mouth encoder 720b, and an RGB mouth image fixed as the source can be passed to a source mouth encoder 720a). This trained Siamese network can generate an indication or determination as output of a match (e.g., in this case, the user is authenticated) or a mismatch (e.g., in this case, the user is not authenticated). In some cases, user authentication can be performed based on determining that the Euclidean distance between the source-target pairs of the RGB HMD mouth images is less than or equal to a predetermined threshold.

[0152] Figure 8 This is a flowchart illustrating an example of a process 800 for authenticating a user. Although the example process 800 depicts a specific sequence of operations, this sequence may be changed without departing from the scope of this disclosure. For example, some of the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality of process 800. In other examples, different components of the example device or system implementing process 800 may perform their functions substantially simultaneously or in a specific order.

[0153] At box 802, process 800 includes acquiring a plurality of images associated with a user's face and facial expressions, wherein each of the plurality of images includes a different portion of the face. For example, these plurality of images may be acquired from a user-facing camera of a head-mounted device (HMD), which in some cases may be connected to... Figure 2 The illustrated HMD 202 is the same as or similar to this one. In some examples, the multiple images may be partial images of the face. For example, the multiple images may include an image of the left eye (e.g., such as...). Figure 2 The illustrated image 204A, Figure 3A The illustrated image 302, Figure 5A The illustrated image 512b Figure 5B The illustrated images 514b and / or 506b Figure 6 The illustrated image 612b, etc.), and the image of the right eye (e.g., such as...) Figure 2 The illustrated image 204B Figure 3A The illustrated image 304 Figure 5A The illustrated image 512a, Figure 5B The illustrated images 514a and / or 506a, Figure 6 The illustrated image 612a, etc.) and the image of the mouth (e.g., such as Figure 2 The illustrated image 204C, Figure 3A The illustrated image 306 Figure 5A The illustrated image 512c Figure 5B The illustrated images are 514c and / or 506c. Figure 6 The illustrated image 612c Figure 7 (Examples 710a and / or 712b, etc.).

[0154] In some examples, obtaining the multiple images associated with the user's face and facial expression may include obtaining a near-infrared (NIR) image of the left eye and a NIR image of the right eye. The left-eye image may be generated based on the left-eye NIR image, wherein the left-eye image is a predicted color image of the left eye. In some cases, the right-eye image may be generated based on the right-eye NIR image, wherein the right-eye image is a predicted color image of the right eye.

[0155] At box 804, process 800 includes generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images. For example, the encoder neural network can be used with... Figure 5A and Figure 5B The illustrated shape-expression encoder 522 is the same as or similar to, and can be used with Figure 6 The illustrated attitude network encoder 620 is the same as or similar to, and / or may be compatible with, the attitude network encoder 620. Figure 7 The illustrated mouth encoder networks 720a and 720b are the same or similar.

[0156] In some cases, the encoder neural network can generate one or more predicted 3D facial modeling parameters, which are 3D deformation model (3DMM) parameters. In some cases, generating the one or more predicted 3D facial modeling parameters includes generating multiple predicted shape coefficients for the 3D facial model using the encoder neural network and generating multiple predicted expression coefficients for the 3D facial model using the encoder neural network. For example, the predicted shape coefficients and / or the predicted expression coefficients may be included in a set of parameters generated by the encoder neural network. Figure 5A and Figure 5B The illustrated shape-expression encoder 522 predicts the shape and expression coefficients 524.

[0157] In some examples, the 3D face modeling parameters used to generate the one or more predictions include camera pose information associated with the predicted images, generated using the encoder neural network. For example, a pose network 620 can be used to generate... Figure 6 The illustrated predicted HMD (e.g., camera) pose information 640. In some cases, the predicted camera pose information may be 6DOF pose information. In some cases, each of the multiple user-facing cameras of a head-mounted device (HMD) may be used to acquire an image included in the multiple images. The predicted camera pose information may be associated with at least a first user-facing camera among the multiple user-facing cameras included in the HMD.

[0158] At box 806, process 800 includes obtaining a reference 3D facial model associated with the face and the facial expression. For example, the reference 3D facial model may be generated based on one or more registered images of the user provided to the encoder neural network, wherein each registered image is associated with the user's face and the user's facial expression. In some examples, the reference 3D facial model may include multiple reference shape coefficients and multiple reference expression coefficients. In some cases, the reference 3D facial model may be generated based on one or more registered images provided to the encoder neural network. Each of the one or more registered images may be associated with the face and the facial expression. The reference 3D model may include one or more registered landmarks associated with the face and the facial expression.

[0159] At box 808, process 800 includes determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model. For example, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model may include determining one or more error values ​​between the predicted shape coefficients and the reference shape coefficients. In some cases, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model may additionally or alternatively include determining one or more error values ​​between the predicted expression coefficients and the reference expression coefficients.

[0160] In some cases, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model involves generating a predicted 3D facial model using the multiple predicted shape coefficients, the multiple predicted expression coefficients, and the average facial components. For example, the average facial components could be the 3DMM average face and basis vectors, such as... Figure 5A and Figure 5B The illustrated 3DMM average face and basis vectors are 540. The predicted 3D face model can be compared with... Figure 5A The illustrated predicted 3D mesh 552 and / or Figure 5B The illustrated predicted 3D mesh 550 is the same as or similar to the one or more predicted 3D facial modeling parameters. In some examples, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model may include determining the average vertex error between multiple vertices included in the predicted 3D facial model and multiple vertices included in the reference 3D facial model. For example, the average vertex error may be compared with... Figure 5A The illustrated 2D vertex distance loss of 575 is the same or similar.

[0161] In some cases, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model can be based on obtaining camera-inherent information associated with the multiple user-facing cameras of the HMD. For example, this camera-inherent information and the predicted camera pose information (e.g., Figure 6 The illustrated HMD pose prediction 640 can be used to project one or more registered landmarks from the reference 3D face model onto the plurality of images. In some examples, the one or more error values ​​can be 2D averaged landmark error values. In some examples, the one or more landmarks included in the plurality of images can be based on providing the plurality of images as input to a two-dimensional (2D) landmark detector (such as...). Figure 6 The detected landmarks are determined by the illustrated 2D vertex distance loss landmark detector 655.

[0162] In some examples, the processes described herein (e.g., process 800 and / or any other processes described herein) may be performed by a computing device, apparatus, or system. In one example, process 800 may be performed by a device having… Figure 11The computing device architecture 1100 is used to perform the computing device or system. The computing device, apparatus, or system may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, an extended reality (XR) headset, a connected watch or smartwatch, or other wearable devices), a server computer, a vehicle or vehicle computing system or apparatus, a robotic device, a laptop computer, a smart TV, a camera, and / or any other computing device with the resource capability to perform the processes described herein (including process 800 and / or other processes described herein). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0163] Components of a computing device may be implemented in a circuit. For example, a component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0164] Process 800 is illustrated as a logic flowchart, the operations of which represent a sequence of operations that can be implemented by hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media, which, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, and data structures that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0165] Additionally, process 800 and / or any other process described herein may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that executes jointly on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0166] Figure 9 This is an exemplary example of a deep learning neural network 900 that can be used by a 3D model training system. Input layer 920 includes input data. In one exemplary example, input layer 920 may include data representing pixels of an input video frame. Neural network 900 includes multiple hidden layers 922a, 922b through 922n. Hidden layers 922a, 922b through 922n include "n" hidden layers, where "n" is an integer greater than or equal to one. The multiple hidden layers can include as many layers as needed for a given application. Neural network 900 also includes an output layer 924 that provides the output produced by the processing performed by hidden layers 922a, 922b through 922n. In one exemplary example, output layer 924 may provide a classification of objects in the input video frame. The classification may include a category identifying the type of object (e.g., person, dog, cat, or other object).

[0167] Neural network 900 is a multi-layered neural network of interconnected nodes. Each node can represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains information while processing it. In some cases, neural network 900 may include a feedforward network, in which case there are no feedback connections where the network's output is fed back into itself. In some cases, neural network 900 may include a recurrent neural network, which may have loops that allow information to be carried across nodes as input is read.

[0168] Information can be exchanged between nodes via node-to-node interconnects between layers. Nodes in input layer 920 can activate a set of nodes in the first hidden layer 922a. For example, as shown, each input node in input layer 920 is connected to each node in the first hidden layer 922a. Nodes in hidden layers 922a, 922b, through 922n can transform information by applying an activation function to the information of each input node. The information derived from this transformation can then be passed to nodes in the next hidden layer 922b, activating those nodes, which can then perform their own specified functions. Example functions include convolution, upsampling, data transformation, and / or any other suitable functions. The output of hidden layer 922b can then activate nodes in the next hidden layer, and so on. Finally, the output of hidden layer 922n can activate one or more nodes in output layer 924, providing the output at those nodes. In some cases, although a node in neural network 900 (e.g., node 926) is shown as having multiple output lines, the node has a single output and is shown as all the lines output from the node representing the same output value.

[0169] In some cases, each node or the interconnection between nodes may have weights, which are a set of parameters derived from the training of the neural network 900. Once the neural network 900 is trained, it can be referred to as a trained neural network, which can be used to classify one or more objects. For example, the interconnection between nodes may represent a piece of information learned about the interconnected nodes. This interconnection may have tunable numerical weights that can be tuned (e.g., based on the training dataset), allowing the neural network 900 to adapt to the input and learn as more and more data is processed.

[0170] Neural network 900 is pre-trained to process features from data in input layer 920 using different hidden layers 922a, 922b to 922n in order to provide output through output layer 924. In an example where neural network 900 is used to identify objects in an image, training data including both images and labels can be used to train neural network 900. For example, training images can be input into the network, where each training image has a label indicating the category of one or more objects in each image (basically, indicating to the network what the objects are and what features they have). In an exemplary example, training images may include images of the number 2, in which case the label of the image could be [0 0 1 0 0 0 0 0 0 0].

[0171] In some cases, the neural network 900 can use a training process called backpropagation to adjust the weights of its nodes. Backpropagation includes forward pass, loss function, backpropagation, and weight update. For each training iteration, forward pass, loss function, backpropagation, and parameter update are performed. This process can be repeated a certain number of iterations for each set of training images until the neural network 900 is trained well enough that the weights of each layer are accurately tuned.

[0172] For an example of identifying objects in an image, the forward pass may include passing a training image through a neural network 900. The weights are initially randomized before training the neural network 900. The image may include, for example, a numerical array representing pixels in the image. Each number in the array may include a value from 0 to 255, describing the pixel intensity at that location in the array. In one example, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or lightness and two chroma components, etc.).

[0173] For the first training iteration of the neural network 900, the output will likely include values ​​due to the weights chosen randomly during initialization, without prioritizing any particular class. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each class may be equal or at least very similar (e.g., 0.1 for each of ten possible classes). Using the initial weights, the neural network 900 cannot determine low-level features and therefore cannot accurately determine what the object's classification might be. A loss function can be used to analyze the error in the output. Any suitable loss function can be defined. An example of a loss function is Mean Squared Error (MSE). MSE is defined as... It calculates the sum of the actual answer minus half the square of the predicted (output) answer. The loss can be set to equal E. total The value of .

[0174] For the first training image, the loss (or error) will be high because the actual value will be very different from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training label. The Neural Network 900 performs backpropagation by determining which inputs (weights) contribute most to the network's loss and can adjust the weights to reduce the loss and eventually minimize it.

[0175] The derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight at a specific layer) can be calculated to determine the weights that contribute the most to the network's loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. A weight update can be expressed as... Where w represents the weight, w i Let represent the initial weights, and η represent the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates, while a lower value indicates smaller weight updates.

[0176] Neural Network 900 can include any suitable deep network. An example includes a Convolutional Neural Network (CNN), which includes an input layer and an output layer, with multiple hidden layers between the input and output layers. (In the following text, relative to...) Figure 10 An example of a CNN is described. The hidden layers of a CNN consist of a series of convolutional layers, non-linear layers, pooling layers (for downsampling), and fully connected layers. A neural network can include any other deep network besides a CNN, such as an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), etc.

[0177] Figure 10 This is an exemplary example of a Convolutional Neural Network 1000 (CNN 1000). The input layer 1020 of the CNN 1000 includes data representing an image. For example, the data may include a numerical array representing the pixels of an image, where each number in the array includes a value from 0 to 255 describing the pixel intensity at the location of that number in the array. Using the previous example above, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or lightness and two chroma components, etc.). The image is passed through a convolutional hidden layer 1022a, an optional non-linear activation layer, a pooling hidden layer 1022b, and a fully connected hidden layer 1022c to obtain an output at the output layer 1024. Although... Figure 10 Only one hidden layer is shown in the diagram, but those skilled in the art will understand that multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers can be included in a CNN 1000. As previously mentioned, the output may indicate a single category of an object, or may include probabilities that best describe the category of an object in an image.

[0178] The first layer of CNN 1000 is a convolutional hidden layer 1022a. Convolutional hidden layer 1022a analyzes the image data input to layer 1020. Each node in convolutional hidden layer 1022a is connected to a region of the input image called a receptive field (pixel). Convolutional hidden layer 1022a can be thought of as one or more filters (each filter corresponding to a different activation or feature map), where each convolutional iteration of the filter is a node or neuron in convolutional hidden layer 1022a. For example, the region of the input image covered by the filter at each convolutional iteration will be the filter's receptive field. In an exemplary example, if the input image comprises a 28×28 array, and each filter (and its corresponding receptive field) is a 5×5 array, then there will be 24×24 nodes in convolutional hidden layer 1022a. Each connection between a node and its receptive field learns weights, and in some cases, learns an overall bias, such that each node learns to analyze its specific local receptive field in the input image. Each node in hidden layer 1022a will have the same weights and biases (called shared weights and shared biases). For example, the filter has a weight (digital) array and the same depth as the input. For the video frame example, the filter would have a depth of 3 (based on the three color components of the input image). An exemplary example of the filter array size is 5×5×3, corresponding to the size of the receptive field of a node.

[0179] The convolutional property of the convolutional hidden layer 1022a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filters of the convolutional hidden layer 1022a may begin at the top left corner of the input image array and may convolve around the input image. As noted above, each convolutional iteration of the filters can be considered as a node or neuron of the convolutional hidden layer 1022a. In each convolutional iteration, the value of the filter is multiplied by the corresponding number of original pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the top left corner of the input image array). The multiplications from each convolutional iteration can be summed to obtain the sum of that iteration or node. Next, the process continues at the next position in the input image based on the receptive field of the next node in the convolutional hidden layer 1022a.

[0180] For example, the filter can move a step size to the next receptive field. The step size can be set to 1 or other suitable amounts. For example, if the step size is set to 1, the filter will move 1 pixel to the right on each convolution iteration. Processing the filter at each unique location in the input volume produces a number representing the filter result at that location, thus determining a sum value for each node of the convolutional hidden layer 1022a.

[0181] The mapping from the input layer to the convolutional hidden layer 1022a is called an activation map (or feature map). An activation map includes values ​​for each node representing the filter results at each location in the input volume. Activation maps may include arrays containing various sums of values ​​produced by the filter for each iteration of the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map would consist of a 24×24 array. The convolutional hidden layer 1022a may include several activation maps to identify multiple features in the image. Figure 10 The example shown includes three activation maps. Using these three activation maps, the convolutional hidden layer 1022a can detect three different types of features, each of which is detectable across the entire image.

[0182] In some examples, nonlinear hidden layers can be applied after the convolutional hidden layer 1022a. Nonlinear layers can be used to introduce nonlinearity into a system that has already computed linear operations. An exemplary example of a nonlinear layer is the Corrected Linear Unit (ReLU) layer. A ReLU layer applies the function f(x) = max(0,x) to all values ​​in the input volume, which changes all negative activations to 0. Therefore, ReLU can increase the nonlinearity of the CNN 1000 without affecting the receptive field of the convolutional hidden layer 1022a.

[0183] A pooling hidden layer 1022b can be applied after the convolutional hidden layer 1022a (and, in use, after the non-linear hidden layer). The pooling hidden layer 1022b is used to simplify the information in the output of the convolutional hidden layer 1022a. For example, the pooling hidden layer 1022b can take each activation map from the output of the convolutional hidden layer 1022a and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by the pooling hidden layer. The pooling hidden layer 1022a uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. Pooling functions (e.g., max pooling filters, L2 norm filters, or other suitable pooling filters) are applied to each activation map included in the convolutional hidden layer 1022a. Figure 10 In the example shown, three pooling filters are used for the three activation maps in the convolutional hidden layer 1022a.

[0184] In some examples, max pooling can be used by applying a max pooling filter (e.g., of 2×2 size) with a step size (e.g., equal to the filter's dimension, such as a step size of 2) to the activation map output from convolutional hidden layer 1022a. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer summarizes a region of 2×2 nodes from the previous layer (each node being a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by the 2×2 max pooling filter at each iteration of the filter, with the maximum of the four values ​​being output as the "maximum" value. If such a max pooling filter is applied to an activation filter of 24×24 nodes from convolutional hidden layer 1022a, the output from pooling hidden layer 1022b will be an array of 12×12 nodes.

[0185] In some examples, L2 norm pooling filters may also be used. L2 norm pooling filters involve calculating the square root of the sum of squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (instead of calculating the maximum value as done in max pooling), and using the calculated value as the output.

[0186] Intuitively, pooling functions (e.g., max pooling, L2-norm pooling, or other pooling functions) determine whether a given feature is found anywhere in a region of an image, discarding the exact location information. This can be done without affecting the results of feature detection, because once a feature has been found, its exact location is less important than its approximate location relative to other features. Max pooling (and other pooling methods) offers the benefit of pooling far fewer features, thus reducing the number of parameters required in subsequent layers of a CNN 1000.

[0187] The final connection in the network is a fully connected layer that connects each node from the pooling hidden layer 1022b to each output node in the output layer 1024. Using the example above, the input layer comprises 28×28 nodes encoding the pixel intensity of the input image, the convolutional hidden layer 1022a comprises 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for filtering) to three activation maps, and the pooling layer 1022b comprises a layer based on applying a max-pooling filter to a 2×2 region on each of the three feature maps. Extending this example, the output layer 1024 may comprise ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 1022b is connected to each node of the output layer 1024.

[0188] The fully connected layer 1022c takes the output of the previous pooling layer 1022b (which should represent an activation map of high-level features) and determines the features most relevant to a particular class. For example, the fully connected layer 1022c can determine the high-level features most relevant to a particular class and may include weights (nodes) for those high-level features. The product between the weights of the fully connected layer 1022c and the pooling hidden layer 1022b can be computed to obtain the probabilities for different classes. For example, if CNN 1000 is used to predict whether an object in a video frame is a person, there will be high values ​​in the activation map representing the high-level features of a person (e.g., two legs, a face at the top of the object, two eyes at the top left and top right of the face, a nose in the middle of the face, a mouth at the bottom of the face, and / or other features common to the person).

[0189] In some examples, the output from output layer 1024 may include an M-dimensional vector (M = 10 in the previous example), where M may include the number of categories from which the program must choose when classifying objects in an image. Other example outputs may also be provided. Each number in the N-dimensional vector may represent the probability that an object belongs to a certain category. In an exemplary example, if the 10-dimensional output vector represents objects of ten different categories as [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates that the probability that the image is an object of the third category (e.g., a dog) is 5%, the probability that the image is an object of the fourth category (e.g., a person) is 80%, and the probability that the image is an object of the sixth category (e.g., a kangaroo) is 15%. The probability for a category can be considered as the confidence level that an object is part of that category.

[0190] Figure 11 This is a diagram illustrating an example of a system used to implement certain aspects of the technology disclosed herein. Specifically, Figure 11 An example of computing system 1100 is illustrated. This computing system can be any computing device, such as an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1105. Connection 1105 can be a physical connection using a bus, or a direct connection to processor 1110, such as in a chipset architecture. Connection 1105 can also be a virtual connection, a networking connection, or a logical connection.

[0191] In some aspects, computing system 1100 is a distributed system in which the functions described herein can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the system components represent a plurality of such components, each of which performs some or all of the functions of the components described herein. In some aspects, the components can be physical or virtual devices.

[0192] Example system 1100 includes at least one processing unit (CPU or processor) 1110 and a connection 1105 that couples various system components, including system memories 1115 such as read-only memory (ROM) 1120 and random access memory (RAM) 1125, to processor 1110. Computing system 1100 may include a cache 1112 of high-speed memory that is directly connected to, close to, or integrated into processor 1110.

[0193] Processor 1110 may include any general-purpose processor and hardware or software services (such as services 1132, 1134, and 1136 stored in storage device 1130 and configured to control processor 1110), as well as dedicated processors in which software instructions are incorporated into the actual processor design. Processor 1110 may be a substantially complete standalone computing system comprising multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0194] To enable user interaction, the computing system 1100 includes an input device 1145 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. The computing system 1100 may also include an output device 1135 that can be one or more of a plurality of output mechanisms. In some cases, a multi-mode system allows the user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 may include a communication interface 1140, which typically controls and manages user input and system output. The communication interface can perform or facilitate the receipt and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, etc. Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs Wireless signal transmission Low-energy (BLE) wireless signal transmission The communication interface 1140 may include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing system 1100 based on one or more signals received from one or more satellites associated with one or more GNSS systems. This includes wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC) wireless signal transmission, microwave access global interoperability (WiMAX) wireless signal transmission, infrared (IR) wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. GNSS systems include, but are not limited to, the U.S. Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no limitations on operation on any particular hardware configuration, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware configurations as they are developed.

[0195] Storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape cassettes, flash memory cards, solid-state storage devices, digital multifunction disks, magnetic tape cassettes, floppy disks, flexible disks, hard disks, magnetic tape, magnetic stripes / strips, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM, rewritable CD, DVD, Blu-ray Disc, holographic disc, another optical medium, secure digital (SD) cards, micro-secure digital (microSD) cards, etc. Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette and / or combinations thereof.

[0196] Storage device 1130 may include software services, servers, etc., which enable the system to perform functions when the code defining such software is executed by processor 1110. In some aspects, hardware services that perform specific functions may include software components for performing functions stored in computer-readable media connected to necessary hardware components such as processor 1110, connection 1105, output device 1135, etc.

[0197] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media that can store data and does not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted using any suitable means, including memory sharing, messaging, token passing, or network transmission.

[0198] In some respects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0199] Specific details are provided in the foregoing description to provide a thorough understanding of the aspects and examples presented herein. However, those skilled in the art will understand that these aspects can be practiced without these specific details. For clarity, in some cases, the technology may be presented as comprising individual functional blocks, including functional blocks containing devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these aspects in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary detail to avoid obscuring the aspects.

[0200] The various aspects described above can be presented as processes or methods, which may be depicted as flowcharts, diagrams, data flow graphs, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many operations within an operation may be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagrams. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0201] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical discs, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0202] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop devices, mobile phones (e.g., smartphones or other types of mobile phones), tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or intercalation cards. By further example, such functionality may also be implemented on different chips or circuit boards between different processes running on a single device.

[0203] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means of providing the functionality described in this disclosure.

[0204] In the foregoing description, aspects of this application have been described with reference to their specific aspects, but those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative aspects of this application have been described in detail herein, it is to be understood that the various inventive concepts can be implemented and employed in a variety of other ways, and the appended claims are intended to be construed as including these variations, unless limited by prior art. Various features and aspects of the above applications may be used individually or in combination. Furthermore, aspects may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that, in alternative aspects, the methods may be performed in a different order than described.

[0205] Those skilled in the art will appreciate that the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced by the less than or equal to (“≤”) and greater than or equal to (“≥”) symbols, respectively, without departing from the scope of this specification.

[0206] When a component is described as being “configured” to perform certain operations, this configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0207] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0208] The claim language or other language that expresses "at least one of" and / or "one or more of" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, the claim language that expresses "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the claim language that expresses "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language that expresses "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the claim language that expresses "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0209] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0210] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication handsets, or integrated circuit devices with multiple uses, including applications in wireless communication handsets and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0211] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0212] The exemplary aspects of this disclosure include:

[0213] Aspect 1: A method for authenticating a user, the method comprising: obtaining a plurality of images associated with the user's face and facial expressions, wherein each corresponding image of the plurality of images includes a different portion of the face; generating one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; obtaining a reference 3D facial model associated with the face and facial expressions; determining an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticating the user based on the error being less than a predetermined authentication threshold.

[0214] Aspect 2: According to the method of aspect 1, the plurality of images are obtained from a user-facing camera of a head-mounted device (HMD).

[0215] Aspect 3: The method according to any one of Aspects 1 to 2, wherein the plurality of images are partial images of the face.

[0216] Aspect 4: The method according to any one of Aspects 1 to 3, wherein the plurality of images includes an image of the left eye, an image of the right eye, and an image of the mouth.

[0217] Aspect 5: The method according to aspect 4, wherein obtaining the plurality of images includes: obtaining a near-infrared (NIR) image of the left eye and a NIR image of the right eye; generating the image of the left eye based on the NIR image of the left eye, wherein the image of the left eye is a predicted color image of the left eye; and generating the image of the right eye based on the NIR image of the right eye, wherein the image of the right eye is a predicted color image of the right eye.

[0218] Aspect 6: The method according to any one of Aspects 1 to 5, wherein the one or more predicted 3D face modeling parameters are 3D deformation model (3DMM) parameters.

[0219] Aspect 7: The method according to any one of Aspects 1 to 6, wherein generating the one or more predicted 3D facial modeling parameters comprises: generating a plurality of predicted shape coefficients for a 3D facial model using the encoder neural network; and generating a plurality of predicted expression coefficients for the 3D facial model using the encoder neural network.

[0220] Aspect 8: The method according to aspect 7, wherein: the reference 3D facial model is generated based on providing one or more registration images to the encoder neural network; and each of the one or more registration images is associated with the face and the facial expression.

[0221] Aspect 9: According to the method of aspect 8, the reference 3D facial model includes a plurality of reference shape coefficients and a plurality of reference expression coefficients.

[0222] Aspect 10: According to the method of aspect 9, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model includes: determining one or more error values ​​between the plurality of predicted shape coefficients and the plurality of reference shape coefficients; and determining one or more error values ​​between the plurality of predicted expression coefficients and the plurality of reference expression coefficients.

[0223] Aspect 11: The method according to any one of Aspects 8 to 10, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: generating a predicted 3D facial model using the plurality of predicted shape coefficients, the plurality of predicted expression coefficients, and an average facial component; and determining an average vertex error between a plurality of vertices included in the predicted 3D facial model and a plurality of vertices included in the reference 3D facial model.

[0224] Aspect 12: The method according to any one of aspects 1 to 11, wherein generating the one or more predicted 3D face modeling parameters comprises: generating predicted camera pose information associated with the plurality of images using the encoder neural network.

[0225] Aspect 13: The method according to aspect 12, wherein: each image included in the plurality of images is obtained using a corresponding user-facing camera among a plurality of user-facing cameras of a head-mounted device (HMD); and the predicted camera pose information is associated with at least a first user-facing camera among the plurality of user-facing cameras included in the HMD.

[0226] Aspect 14: The method according to aspect 13, wherein: the reference 3D facial model is generated based on providing one or more registration images to the encoder neural network, each of the one or more registration images being associated with the face and the facial expression; and the reference 3D facial model includes one or more registrations of landmarks associated with the face and the facial expression.

[0227] Aspect 15: According to the method of aspect 14, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model includes: obtaining camera-inherent information associated with the plurality of user-facing cameras of the HMD; projecting the one or more registered landmarks from the reference 3D facial model onto the plurality of images based on the predicted camera pose information and the camera-inherent information; and determining one or more error values ​​between the projected registered landmarks and one or more landmarks included in the one or more images.

[0228] Aspect 16: According to the method of aspect 15, wherein the one or more error values ​​are two-dimensional (2D) average landmark error values.

[0229] Aspect 17: The method according to aspect 16, wherein the one or more landmarks in the plurality of images are detected landmarks determined based on providing the plurality of images as input to a two-dimensional (2D) landmark detector.

[0230] Aspect 18: An apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: acquire a plurality of images associated with a user's face and facial expressions, wherein each corresponding image of the plurality of images includes a different portion of the face; generate one or more predicted three-dimensional (3D) facial modeling parameters using an encoder neural network, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images; acquire a reference 3D facial model associated with the face and facial expressions; determine an error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model; and authenticate the user based on the error being less than a predetermined authentication threshold.

[0231] Aspect 19: The apparatus according to aspect 18, wherein the plurality of images are obtained from a user-facing camera of a head-mounted device (HMD) and include an image of the left eye, an image of the right eye, and an image of the mouth.

[0232] Aspect 20: The apparatus according to aspect 19, wherein obtaining the plurality of images includes: obtaining a near-infrared (NIR) image of the left eye and a NIR image of the right eye; generating the image of the left eye based on the NIR image of the left eye, wherein the image of the left eye is a predicted color image of the left eye; and generating the image of the right eye based on the NIR image of the right eye, wherein the image of the right eye is a predicted color image of the right eye.

[0233] Aspect 21: The apparatus according to any one of aspects 18 to 20, wherein the one or more predicted 3D facial modeling parameters are 3D deformation model (3DMM) parameters.

[0234] Aspect 22: The apparatus according to any one of aspects 18 to 21, wherein generating the one or more predicted 3D facial modeling parameters comprises: generating a plurality of predicted shape coefficients for a 3D facial model using the encoder neural network; and generating a plurality of predicted expression coefficients for the 3D facial model using the encoder neural network.

[0235] Aspect 23: The apparatus according to aspect 22, wherein: the reference 3D facial model is generated based on providing one or more registered images to the encoder neural network; and each of the one or more registered images is associated with the face and the facial expression.

[0236] Aspect 24: The apparatus according to aspect 23, wherein the reference 3D facial model includes a plurality of reference shape coefficients and a plurality of reference expression coefficients.

[0237] Aspect 25: According to the apparatus of aspect 24, determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model includes: determining one or more error values ​​between the plurality of predicted shape coefficients and the plurality of reference shape coefficients; and determining one or more error values ​​between the plurality of predicted expression coefficients and the plurality of reference expression coefficients.

[0238] Aspect 26: The apparatus according to any one of aspects 23 to 25, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: generating a predicted 3D facial model using the plurality of predicted shape coefficients, the plurality of predicted expression coefficients, and an average facial component; and determining an average vertex error between a plurality of vertices included in the predicted 3D facial model and a plurality of vertices included in the reference 3D facial model.

[0239] Aspect 27: The apparatus according to any one of aspects 18 to 26, wherein generating the one or more predicted 3D face modeling parameters comprises: generating predicted camera pose information associated with the plurality of images using the encoder neural network.

[0240] Aspect 28: The apparatus according to aspect 27, wherein: each image included in the plurality of images is obtained using a corresponding user-facing camera among a plurality of user-facing cameras of a head-mounted device (HMD); and the predicted camera pose information is associated with at least a first user-facing camera among the plurality of user-facing cameras included in the HMD.

[0241] Aspect 29: The apparatus according to aspect 28, wherein: the reference 3D facial model is generated based on providing one or more registered images to the encoder neural network, each of the one or more registered images being associated with the face and the facial expression; and the reference 3D facial model includes one or more registered landmarks associated with the face and the facial expression.

[0242] Aspect 30: The apparatus according to aspect 29, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: obtaining camera-inherent information associated with the plurality of user-facing cameras of the HMD; projecting the one or more registered landmarks from the reference 3D facial model onto the plurality of images based on the predicted camera pose information and the camera-inherent information; and determining one or more error values ​​between the projected registered landmarks and one or more landmarks included in the one or more images.

[0243] Aspect 31: The apparatus according to aspect 30, wherein the one or more error values ​​are two-dimensional (2D) average landmark error values.

[0244] Aspect 32: The apparatus according to aspect 31 includes one or more landmarks in the plurality of images that are detected landmarks determined based on providing the plurality of images as input to a two-dimensional (2D) landmark detector.

[0245] Aspect 33: An apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to perform operations according to any one of aspects 1 to 32.

[0246] Aspect 34: An apparatus comprising components for performing operations according to any one of aspects 1 to 32.

[0247] Aspect 35: A non-transitory computer-readable medium having instructions stored thereon, the instructions causing the one or more processors, when executed, to perform any one of aspects 1 to 32.

Claims

1. A method for authenticating a user, the method comprising: Obtain multiple images associated with the user's face and facial expressions, wherein each of the multiple images includes a different part of the face; An encoder neural network is used to generate one or more predicted 3D facial modeling parameters, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images. Obtain a reference 3D facial model associated with the user's face and registered facial expressions, wherein the reference 3D facial model comprises a 3D mesh generated from multiple registered images of the user's face having the registered facial expressions; Determine the error between predicted expression coefficients included in the one or more predicted 3D facial modeling parameters and reference expression coefficients determined for the reference 3D facial model, wherein the predicted expression coefficients correspond to the facial expression and the reference expression coefficients correspond to the registered facial expression; as well as The user is authenticated based on the error being less than a predetermined authentication threshold.

2. The method of claim 1, wherein the plurality of images are obtained from a user-facing camera of a head-mounted device (HMD).

3. The method of claim 1, wherein the plurality of images are partial images of the face.

4. The method of claim 1, wherein the plurality of images includes an image of the left eye, an image of the right eye, and an image of the mouth.

5. The method of claim 4, wherein obtaining the plurality of images comprises: Obtain the near-infrared NIR image of the left eye and the near-infrared NIR image of the right eye; The image of the left eye is generated based on the near-infrared (NIR) image of the left eye, wherein the image of the left eye is a predicted color image of the left eye; as well as The image of the right eye is generated based on the near-infrared (NIR) image of the right eye, wherein the image of the right eye is a predicted color image of the right eye.

6. The method of claim 1, wherein the one or more predicted 3D facial modeling parameters are 3D deformation model (3DMM) parameters.

7. The method of claim 1, wherein generating the one or more predicted 3D facial modeling parameters comprises: The encoder neural network is used to generate multiple predicted shape coefficients for a 3D facial model; as well as The encoder neural network is used to generate multiple predicted expression coefficients for the three-dimensional facial model.

8. The method according to claim 7, wherein: The reference 3D facial model is generated by providing one or more of the plurality of registered images to the encoder neural network; and Each of the one or more registered images is associated with the user's face and the user's registered facial expression.

9. The method of claim 8, wherein the reference three-dimensional 3D facial model comprises a plurality of reference shape coefficients and a plurality of reference expression coefficients.

10. The method of claim 9, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: Determine one or more error values ​​between the plurality of predicted shape coefficients and the plurality of reference shape coefficients, wherein the plurality of predicted shape coefficients correspond to the facial expression and the plurality of reference shape coefficients correspond to the registered facial expression; as well as Determine one or more error values ​​between the plurality of predicted expression coefficients and the plurality of reference expression coefficients.

11. The method of claim 8, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: A 3D facial model; as well as Determine the average vertex error between multiple vertices included in the predicted 3D facial model and multiple vertices included in the reference 3D facial model.

12. The method of claim 1, wherein generating the one or more predicted 3D facial modeling parameters comprises: The encoder neural network is used to generate predicted camera pose information associated with the plurality of images.

13. The method according to claim 12, wherein: Each of the plurality of images was acquired using a corresponding user-facing camera among a plurality of user-facing cameras of a head-mounted device (HMD); and The predicted camera pose information is associated with at least a first user-facing camera among the plurality of user-facing cameras included in the head-mounted device (HMD).

14. The method of claim 13, wherein: The reference 3D facial model is generated based on one or more registered images provided to the encoder neural network, each of the one or more registered images being associated with the user's face and the user's registered facial expressions; and The reference 3D facial model includes one or more registered landmarks associated with the face and the registered facial expressions.

15. The method of claim 14, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: Obtain camera-specific information associated with the plurality of user-facing cameras of the head-mounted device (HMD); Based on the predicted camera pose information and the camera's inherent information, one or more registered landmarks from the reference 3D facial model are projected onto the plurality of images; as well as Determine one or more error values ​​between the projected registered landmark and one or more landmarks included in the plurality of images.

16. The method of claim 15, wherein the one or more error values ​​are two-dimensional 2D average landmark error values.

17. The method of claim 16, wherein the one or more landmarks in the plurality of images are detected landmarks determined based on providing the plurality of images as input to a two-dimensional 2D landmark detector.

18. An apparatus comprising: At least one memory; and At least one processor, the at least one processor being coupled to the at least one memory and being configured to: Obtain multiple images associated with a user's face and facial expressions, wherein each of the multiple images includes a different part of the face; An encoder neural network is used to generate one or more predicted 3D facial modeling parameters, wherein the encoder neural network generates the one or more predicted 3D facial modeling parameters based on the plurality of images. Obtain a reference 3D facial model associated with the user's face and registered facial expressions, wherein the reference 3D facial model comprises a 3D mesh generated from multiple registered images of the user's face having the registered facial expressions; Determine the error between predicted expression coefficients included in the one or more predicted 3D facial modeling parameters and reference expression coefficients determined for the reference 3D facial model, wherein the predicted expression coefficients correspond to the facial expression and the reference expression coefficients correspond to the registered facial expression; as well as The user is authenticated based on the error being less than a predetermined authentication threshold.

19. The apparatus of claim 18, wherein the plurality of images are obtained from a user-facing camera of a head-mounted device (HMD) and include an image of the left eye, an image of the right eye, and an image of the mouth.

20. The apparatus of claim 19, wherein obtaining the plurality of images comprises: Obtain the near-infrared NIR image of the left eye and the near-infrared NIR image of the right eye; The image is a predicted color image of the left eye; as well as The image of the right eye is generated based on the near-infrared (NIR) image of the right eye, wherein the image of the right eye is a predicted color image of the right eye.

21. The apparatus of claim 18, wherein the one or more predicted 3D facial modeling parameters are 3D deformation model (3DMM) parameters.

22. The apparatus of claim 18, wherein generating the one or more predicted 3D facial modeling parameters comprises: The encoder neural network is used to generate multiple predicted shape coefficients for a 3D facial model; as well as The encoder neural network is used to generate multiple predicted expression coefficients for the three-dimensional facial model.

23. The apparatus according to claim 22, wherein: The reference 3D facial model is generated by providing one or more of the plurality of registered images to the encoder neural network; and Each of the one or more registered images is associated with the user's face and the user's registered facial expression.

24. The apparatus of claim 23, wherein the reference three-dimensional facial model comprises a plurality of reference shape coefficients and a plurality of reference expression coefficients.

25. The apparatus of claim 24, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: Determine one or more error values ​​between the plurality of predicted shape coefficients and the plurality of reference shape coefficients, wherein the plurality of predicted shape coefficients correspond to the facial expression and the plurality of reference shape coefficients correspond to the registered facial expression; as well as Determine one or more error values ​​between the plurality of predicted expression coefficients and the plurality of reference expression coefficients.

26. The apparatus of claim 23, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: The predicted 3D facial model is generated using the multiple predicted shape coefficients, the multiple predicted expression coefficients, and the average facial component. as well as Determine the average vertex error between multiple vertices included in the predicted 3D facial model and multiple vertices included in the reference 3D facial model.

27. The apparatus of claim 18, wherein generating the one or more predicted 3D facial modeling parameters comprises: The encoder neural network is used to generate predicted camera pose information associated with the plurality of images.

28. The apparatus according to claim 27, wherein: Each of the plurality of images was acquired using a corresponding user-facing camera among a plurality of user-facing cameras of a head-mounted device (HMD); and The predicted camera pose information is associated with at least a first user-facing camera among the plurality of user-facing cameras included in the head-mounted device (HMD).

29. The apparatus according to claim 28, wherein: The reference 3D facial model is generated based on one or more registered images provided to the encoder neural network, each of the one or more registered images being associated with the user's face and the user's registered facial expressions; and The reference 3D facial model includes one or more registered landmarks associated with the face and the registered facial expressions.

30. The apparatus of claim 29, wherein determining the error between the one or more predicted 3D facial modeling parameters and the reference 3D facial model comprises: Obtain camera-specific information associated with the plurality of user-facing cameras of the head-mounted device (HMD); Based on the predicted camera pose information and the camera's inherent information, one or more registered landmarks from the reference 3D facial model are projected onto the plurality of images; as well as Determine one or more error values ​​between the projected registered landmark and one or more landmarks included in the plurality of images.

Images