Facial texture synthesis for three-dimensional deformation models

Abstract

Systems and techniques are provided for generating textures of a three-dimensional (3D) face model. For example, a process may include obtaining a first frame including a first portion of a face. In some aspects, the process may include generating a 3D face model based on the first frame, and generating a first face feature corresponding to the first portion of the face. In some examples, the process includes obtaining a second frame, the second frame including a second portion of the face. In some cases, the second portion of the face at least partially overlaps the first portion of the face. In some examples, the process includes combining the first facial feature with the second facial feature to generate an enhanced facial feature, where the combining is performed to enhance an appearance of a selected region of the enhanced facial feature.

Description

Technical Field

[0001] The present disclosure generally relates to systems and techniques for generating three-dimensional (3D) models. For example, aspects of the present disclosure relate to performing facial texture synthesis for a 3D model (eg, a 3D morphable model), such as for a face or other object.

[0002] background

[0003] Many devices and systems allow for capturing a scene by generating frames (also referred to as images) and / or video data (including multiple images or frames) of the scene. For example, a camera or a computing device including a camera (e.g., a mobile device (such as a mobile phone or smart phone) including one or more cameras) can capture a sequence of frames of a scene. Frames and / or video data can be captured and processed by such devices and systems (e.g., mobile devices, IP cameras, etc.) and can be output for consumption (e.g., displayed on the device and / or other devices). In some cases, frames and / or video data can be captured by such devices and systems and output for processing and / or consumption by other devices.

[0004] Frames may be processed (e.g., using object detection, recognition, segmentation, etc.) to determine objects present in the frame, which is useful for many applications. For example, a model may be determined to represent the objects in the frame, and the model may be used to facilitate efficient operation of various systems. Examples of such applications and systems include augmented reality (AR), robotics, automotive and aviation, 3D scene understanding, object grasping, object tracking, and many others. Summary of the invention

[0005] Disclosed are systems, devices, methods, and computer-readable media for generating a texture of a three-dimensional (3D) facial model. According to at least one example, a method for generating a texture of a 3D facial model is provided. The method includes: obtaining a first frame, the first frame including a first portion of a face; generating a 3D facial model based on the first frame; generating a first facial feature corresponding to the first portion of the face; obtaining a second frame, the second frame including a second portion of the face, wherein the second portion of the face at least partially overlaps with the first portion of the face; generating a second facial feature corresponding to the second portion of the face; and combining the first facial feature with the second facial feature to generate an enhanced facial feature, wherein the combining is performed to enhance the appearance of a selected area of ​​the enhanced facial feature.

[0006] In another example, an apparatus for generating a texture of a 3D facial model is provided, the apparatus comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain a first frame, the first frame comprising a first portion of a face; generate a 3D facial model based on the first frame; generate a first facial feature corresponding to the first portion of the face; obtain a second frame, the second frame comprising a second portion of the face, wherein the second portion of the face at least partially overlaps the first portion of the face; generate a second facial feature corresponding to the second portion of the face; and combine the first facial feature with the second facial feature to generate an enhanced facial feature, wherein the combination is performed to enhance the appearance of a selected area of ​​the enhanced facial feature.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: obtain a first frame, the first frame including a first portion of a face; generate a 3D facial model based on the first frame; generate a first facial feature corresponding to the first portion of the face; obtain a second frame, the second frame including a second portion of the face, wherein the second portion of the face at least partially overlaps the first portion of the face; generate a second facial feature corresponding to the second portion of the face; and combine the first facial feature with the second facial feature to generate an enhanced facial feature, wherein the combining is performed to enhance the appearance of a selected area of ​​the enhanced facial feature.

[0008] In another example, an apparatus for generating a texture of a 3D facial model is provided. The apparatus includes: means for obtaining a first frame, the first frame including a first portion of a face; means for generating a 3D facial model based on the first frame; means for generating a first facial feature corresponding to the first portion of the face; means for obtaining a second frame, the second frame including a second portion of the face, wherein the second portion of the face at least partially overlaps the first portion of the face; means for generating a second facial feature corresponding to the second portion of the face; and means for combining the first facial feature with the second facial feature to generate an enhanced facial feature, wherein the combining is performed to enhance the appearance of a selected area of ​​the enhanced facial feature.

[0009] In some aspects, one or more of the above devices are or are part of a vehicle (e.g., a computing device of a vehicle), a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, a device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors that can be used to determine the position and / or posture of the device, the state of the device, and / or for other purposes.

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0013] Figure 1 illustrates an example three-dimensional (3D) facial model and a corresponding two-dimensional (2D) facial image projected from the 3D facial model overlaid with landmarks according to some examples;

[0014] Figure 2 illustrates an example head-mounted XR system with a user-facing camera for generating a 3D facial model according to some examples;

[0015] Figure 3 is a diagram illustrating an example of a 3D modeling system according to some examples;

[0016] Figure 3 is a diagram illustrating another example of a 3D modeling system according to some examples;

[0017] Figure 4A is a block diagram illustrating a non-skewed branch of a training local feature engine according to some examples;

[0018] Figure 4B illustrates example techniques for generating training data for training a local feature engine according to some examples;

[0019] Figure 4C is a block diagram illustrating a tilt branch of a training local feature engine according to some examples;

[0020] Figure 5A is a block diagram illustrating an example facial reconstruction engine according to some examples;

[0021] Figure 5B is a block diagram illustrating an example fusion block according to some examples;

[0022] Figure 6 is a flow chart illustrating an example of a process for training a 3D model according to some examples;

[0023] Figure 7 is a block diagram illustrating an example of a deep learning network according to some examples;

[0024] Figure 8 is a block diagram illustrating an example of a convolutional neural network according to some examples; and

[0025] Fig. 9 is a diagram illustrating an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0026] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects can be implemented without these specific details. Each drawing and description are not intended to be restrictive.

[0027] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes can be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0028] The generation of three-dimensional (3D) models of physical objects can be used in many systems and applications, such as for extended reality (XR) (e.g., including augmented reality (AR), virtual reality (VR), mixed reality (MR), etc.), robotics, automotive, aviation, 3D scene understanding, object grasping, object tracking, and many other systems and applications. In an AR environment, for example, a user can view an image (also referred to as a frame) that includes an integration of artificial or virtual graphics with the user's natural environment. AR applications allow real images to be processed to add virtual objects to the image or display virtual objects on a perspective display (so that the virtual objects appear to be overlaid on the real world environment). AR applications can align or register virtual objects with real-world objects (e.g., as observed in the image) in multiple dimensions. For example, real-world objects that exist in reality can be represented using models that are similar to or exactly match the real-world objects. In one example, a model of a virtual aircraft representing a real aircraft located on a runway can be presented through a display of an AR device (e.g., AR glasses, an AR head-mounted display (HMD), or other device) while the user continues to view his natural environment through the display. The viewer may be able to manipulate the model while viewing the real-world scene. In another example, a real object located on a table may be identified and rendered using a model with a different color or different physical properties in the AR environment. In some cases, artificial virtual objects that do not exist in reality or computer-generated copies of real objects or structures in the user's natural environment may also be added to the AR environment.

[0029] The increasing number of applications using facial data (e.g., for XR systems, for 3D graphics, for security, etc.) has led to a large demand for systems that can generate detailed 3D facial models (as well as 3D models of other objects) in an efficient and high-quality manner. There is also a large demand for generating 3D models of other types of objects, such as 3D models of vehicles (e.g., for autonomous driving systems), 3D models of room layouts (e.g., for XR applications, for navigation by devices, robots, etc.), etc. Generating detailed 3D models of objects (e.g., 3D facial models) typically requires expensive equipment and the use of multiple cameras in an environment with controlled lighting, which hinders large-scale data collection.

[0030] Performing 3D object reconstruction (e.g., to generate a 3D model of an object, such as a face model) from one or more images can be challenging. Using the face as an illustrative example of a 3D object, 3D facial reconstruction can be difficult based on the need to reconstruct facial geometry (e.g., shape) and facial expressions. Additionally, it can be difficult to accurately reconstruct facial expressions for portions of the face that may experience high variations in appearance. In one illustrative example, the eyes of a face may move to extreme gaze directions (e.g., looking to one side, squinting, etc.). In another illustrative example, the upper and lower lips of the mouth of a face are controlled by muscles that allow for a variety of difficulties in reconstructing mouth shapes (e.g., smiling, frowning, pricking teeth, twisting lips, etc.).

[0031] like Figure 1 As illustrated, the white dots overlaid on the 2D facial image 102 may represent the projection of the 3D vertices of the 3D facial model 104 back onto the original 2D facial image 102 used to generate the 3D facial model 104. Figure 1 In the illustration of , points corresponding to 3D vertices of the main features of the 3D facial model (which may be referred to as landmarks or 2D landmarks) are depicted as white dots. As shown, landmarks 110, 112, 118, 120, 122, 124, 126, 128 for the outlines of lips, nose, mouth, eyes, eyebrows, nose, etc. are included. Although the 3D facial model 104 may include a greater number of vertices, for the purpose of illustration, only a small number of projected 3D vertices corresponding to the facial features listed above are shown. Figure 1 In the illustrated example, the landmarks corresponding to the inner contour 108 of the lower lip of the 3D facial model 104 projected onto the 2D image may include the landmark 112. Similarly, the landmarks corresponding to the outer contour 106 of the lower lip of the 3D facial model 104 may include the landmark 110.

[0032] Figure 1 Also illustrated are an outer contour 114 and an inner contour 116 of the upper lip of the 3D facial model 104. In some examples, landmarks corresponding to the outer contour 114 of the upper lip may include landmarks 118 and 124, and landmarks corresponding to the inner contour 116 of the upper lip may include landmark 120. Additional landmarks projected from the 3D facial model 104 may include a landmark 122 corresponding to the left eye, a landmark 124 corresponding to the right eyebrow, a landmark 126 corresponding to the entire facial contour, and a landmark 128 corresponding to the nose. As described above, each of the landmarks 114, 116, 120, 122, 124, 126, and 128 may be generated by the projection of the 3D facial model 104 onto the 2D facial image 102.

[0033] Figure 1A two-dimensional (2D) facial image 102 and a corresponding 3D facial model 104 generated from the 2D facial image 102 using a 3D morphable model (3DMM) are illustrated. In some aspects, the 3D facial model 104 may include a representation of a facial expression in the 2D facial image 102. In one illustrative example, the facial expression representation may be formed by blend shapes. The blend shapes may semantically represent the movement of a muscle or part of a facial feature (e.g., opening / closing of a jaw, raising / lowering of an eyebrow, opening / closing of an eye, etc.). In some cases, each blend shape may be represented by a blend shape coefficient paired with a corresponding blend shape vector.

[0034] In some examples, 3D facial model 104 may include a representation of the facial shape in 2D facial image 102. In some cases, the facial shape may be represented by facial shape coefficients paired with corresponding facial shape vectors. In some implementations, a 3D model engine (e.g., a machine learning model) may be trained (e.g., during a training process) to implement a consistent facial shape (e.g., consistent facial shape coefficients) for the 3D facial model regardless of the pose (e.g., pitch, yaw, and roll) associated with the 3D facial model. For example, when rendering the 3D facial model into a 2D image for display, the 3D facial model may be projected onto the 2D image using projection techniques. Although the 3D model engine implements a consistent facial shape independent of pose, the projected 2D image may have varying degrees of accuracy based on the pose of the 3D facial model captured in the projected 2D image.

[0035] like Figure 2 As shown, the 3D model generator may utilize input frames such as tilted frames 204A, 204B, 204C, and / or 208 to generate a 3D facial model 210. Figure 2 As shown in , the 3D model fitting engine 206 can also generate and / or apply textures to the underlying 3D model (e.g., Figure 1 202) to provide a digital representation of a user wearing the head-mounted XR system 202. In one illustrative example, a 3D morphable model (3DMM) may be used to represent the geometry of the user's head. In some cases, the 3DMM may lack the ability to accurately replicate the user's internal mouth and eyeballs. In some cases, the resulting 3D facial model 210 may produce unrealistic results in the eye and mouth areas.

[0036] In some implementations, a highly detailed 3D facial model (not shown) may be generated using expensive camera equipment that captures the individual's face from multiple angles. In some cases, the 3D facial model may also be manually edited by a skilled artist to produce an accurate depiction of the individual. The process of generating such a highly detailed 3D facial model can only produce a single model for a particular individual and does not provide a flexible framework for generating a 3D model for any individual without preparing a detailed 3D facial model in advance.

[0037] What is needed are systems and techniques for generating accurate 3D facial models for a wide variety of facial expressions that do not require a highly detailed 3D facial model specifically tailored to an individual to generate an accurate representation of facial features.

[0038] Described herein are systems, devices, processes (or methods), and computer-readable media (collectively, "systems and techniques") for generating a 3D model of a specific portion of an object from a video comprising a sequence of frames (e.g., a video having a red (R), green (G), blue (B) format, referred to as an RGB video) or from one or more images (e.g., a grayscale image or an RGB image). In some examples, as described in more detail below, these systems and techniques may generate a 3D model of an object (e.g., a 3D model of a person's face) based on one or more frames depicting the object. The systems and techniques may determine local feature information for producing an accurate depiction of a specific portion of the face (e.g., a mouth, an eye). The systems and techniques may combine the local feature information with features of a full facial model (e.g., using image fusion, inpainting, etc.) to produce a more accurate depiction of the user. In some examples, the systems and techniques may train a machine learning model with training data to generate a local UV texture based on an input frame. In some cases, the systems and techniques may train a machine learning model to combine features from a full facial representation (also referred to herein as global features) with one or more local features (e.g., features for eyes and / or mouth) to generate a synthetic 3D facial model. In some examples, combining is performed to enhance the appearance of selected regions of the face (eg, regions corresponding to local features).

[0039] Although this article uses faces as examples of objects that can be modeled, the systems and techniques described herein can be applied to generate models for any type of object. In addition, for illustrative purposes, eyes and mouths are used herein as illustrative examples of local feature regions. However, a person of ordinary skill will understand that the systems and techniques described herein can be performed for other local parts of the face. In an illustrative example, similar systems and techniques can be applied to eyebrows, noses, ears, or any other part of the face. Similar techniques using local feature generation can also be used with objects other than faces.

[0040] Various aspects of the technology described herein will be discussed below with respect to the various figures. Figure 3 3D modeling system 300 may generate a 3D model (eg, a 3D deformable model (3DMM)) using at least one image frame 302. 3D modeling system 300 may also obtain a local frame (eg, from Figure 2 Frames of a user-facing camera of the head-mounted XR system 202). Figure 3 As shown, the 3D modeling system 300 includes an image frame engine 304, a 3D model fitting engine 306, and a face reconstruction engine 310. Although the 3D modeling system 300 is shown as including certain components, a person of ordinary skill will appreciate that the 3D modeling system 300 may include more than one component. Figure 3 The components shown are more components. The components of the 3D modeling system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some specific implementations, the components of the 3D modeling system 300 may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or the components of the 3D modeling system may include and / or be implemented using computer software, firmware, or a combination thereof for performing the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the 3D modeling system 300.

[0041] The image frame engine 304 may obtain or receive the image frame 302 and / or the partial frame 303 captured by the image sensor from a storage device, from a memory, from an external source (e.g., a server, an external memory accessed via a network, or other external source), etc. In some cases, the image frame may be included in a frame sequence (e.g., a video, an independent or still image sequence, etc.). In an illustrative example, each frame in the frame sequence may include a grayscale component per pixel. Other examples of frames include frames with red (R), green (G), and blue (B) components per pixel (referred to as RGB video including RGB frames), frames with brightness, chroma-blue, chroma-red (YUV, YCbCr, or Y′CbCr) components per pixel, and / or any other suitable type of image. The frame sequence may be captured by one or more cameras, obtained from a storage device, received from another device (e.g., a camera or a device including a camera), or obtained from another source. In some specific implementations, the image frame engine 304 may convert the image frame 302 to grayscale. In some cases, image frame engine 304 may crop a portion of image frame 302 that corresponds to a face. In some examples, image frame engine 304 may perform a facial detection process and / or a facial recognition process to detect and / or recognize a face within image frame 302. Image frame engine 304 may generate or apply a bounding box (e.g., Figure 1 The bounding box 130 shown in FIG. 1 is used to define the image data within the bounding box and to crop the image data within the bounding box to generate an input image for the 3D model fitting engine 306 .

[0042] 3D model fitting engine 306 may receive an input image (e.g., image frame 302, a cropped bounding box around a face in image frame 302, etc.) from image frame engine 304. Using the input image, 3D model fitting engine 306 may perform a 3D model fitting technique to generate a 3D model (e.g., a 3DMM model) of a face (which may include a head of a person in image frame 302). The 3D model fitting technique may include solving for shape coefficients and expression coefficient . In some examples, 3D model fitting may include solving for position information related to the object. In an example where the object is a person's head, the position information may include posture information related to the posture of the head. For example, the posture information may indicate an angular rotation of the head relative to a neutral position of the head. The rotation may be along a first axis (e.g., a yaw axis), a second axis (e.g., a pitch axis), and / or a third axis (e.g., a roll axis). In some cases, 3D model fitting may also include a focal length for projecting the 3D model onto a 2D image using any suitable projection technique. In some examples, a weak perspective model may use a focal length generated by the 3D model fitting engine 306 to project the 3D vertices of a 3D model (e.g., 3DMM) onto a 2D image. In some examples, a full perspective model may use a focal length generated by the 3D model fitting engine 306 to project the 3D vertices of a 3D model (e.g., 3DMM) onto a 2D image.

[0043] The local feature engine 308 may receive one or more input frames (e.g., local frame 303) from the image frame engine 304. In some implementations, the local feature engine 308 may be implemented as a machine learning model (e.g., a deep learning neural network). In some examples, the machine learning model may be trained to generate local textures of parts of the face (such as the eyes and mouth), which may be combined with the full facial texture for use in the 3DMM generated by the 3D model fitting engine 306. In one example, the local frame 303 may include a local texture generated by, for example, Figure 2 Oblique frames 204A, 204B, 204C captured by a user-facing camera on the illustrated head-mounted XR system.

[0044] Reference FIG. 4A to FIG. 4C , used to train local feature engines (e.g., Figure 3 An example training process of the local feature engine 308). Figure 4AIn the illustrated example, the local feature engine 400 includes two separate branches. In an illustrative example, the local feature engine 400 may utilize a variational autoencoder architecture (VAE). In some cases, the variational autoencoder may be trained to generate a latent vector (also referred to herein as a latent code) that provides a statistical representation of a training data set. In a VAE architecture, an encoder (e.g., encoder 412, 452) may encode input data (e.g., input frames), and a decoder (e.g., decoder 420) may generate output data (e.g., output frames). In some cases, the VAE architecture may be trained to generate output frames that are similar to the input frames. During training, the VAE may be configured to generate a latent code (e.g., latent codes 414, 454) that represents the training data set. In some specific implementations, the VAE may be configured to generate latent codes 414, 454 that include assumptions about the underlying distribution of the input data. In an illustrative example, the latent codes 414, 454 may be generated assuming that the input data has a continuous, Gaussian distribution. In Figure 4A In the illustrated example, a non-slanted branch 410 and a slanted branch 450 are illustrated in the local feature engine 400. Both the non-slanted branch 410 and the slanted branch 450 include encoders 412, 452 and potential codes 414, 454. In the illustrated example, the decoder 420 is shared by the two branches 410, 450, and the switch 416 can select which of the two branches 410, 450 provides data to the decoder 420. In some specific implementations, the desired output local UV texture 473A, 473B, 473C of the local feature engine 400 can be a localized area in the UV texture space that can be used to provide additional information for the appearance of eyes and mouths.

[0045] Reference Figure 4B , and illustrates an illustrative example technique for generating training data for training the two branches 410, 450. Figure 4B In the example of , the frontal image 462 can be used to generate a 3DMM 464 using conventional 3DMM generation techniques (e.g., by the 3D model fitting engine 306). In some cases, the frontal image 462 can be aligned with the 3DMM 464 and applied as a texture to provide a 3D representation of the face depicted in the frontal image. As illustrated, the textured 3DMM 466 can be rendered and can be rotated to be viewed at any angle through a virtual camera (not shown). As illustrated, the virtual camera can be configured to capture regions 465A, 465B, 465C of the textured 3DMM 464, such as the left eye, right eye, and mouth. Corresponding tilted frames 467A, 467B, 467C of the left eye, right eye, and mouth, respectively, can be used as Figure 4A The training data of the tilted branch 450.

[0046] In some implementations, during training, a UV face position map 468 (also referred to as a UV texture or UV position map) may be generated for a 3DMM 464 including a face. In some cases, the UV face position map 468 may provide and / or represent a 2D map of a face in an input image. For example, the UV face position map may be a 2D image that records and / or maps the 3D positions of points (e.g., pixels) in a UV space (e.g., a 2D texture coordinate system). U in UV space and V in UV space may represent axes of the UV face position map (e.g., axes of a 2D texture of the face). In an illustrative example, U in UV space may represent a first axis (e.g., a horizontal X-axis) of the UV face position map, and V in UV space may represent a second axis (e.g., a vertical Y-axis) of the UV face position map. In some examples, the UV position map may record, model, identify, represent, and / or calculate the 3D shape, structure, contour, depth, and / or other details of a face (and / or a facial region of a head). In some examples, the 3DMM 464 with the texture provided from the UV face position map 468 may be used to render a 3D digital representation of the frontal image 462. In some implementations, a machine learning model (eg, a neural network) may be used to generate the UV face position map.

[0047] As shown in the illustrated example, the positions of the eyes and mouth in the UV face position map 468 may be fixed at positions 469A, 469B, and 469C corresponding to the left eye, right eye, and mouth, respectively. In some cases, the UV face position map 468 (e.g., UV texture) may be cropped to generate non-slanted input frames 471A, 471B, 471C corresponding to the left eye, right eye, and mouth, respectively. The slanted input frames 467A, 467B, 467C and non-slanted input frames 471A, 471B, 47C may be generated for a large number of faces and facial expressions to create a training data set for training the non-slanted branch 410 and the slanted branch 450 of the local feature engine 400. In an illustrative example, the slanted frames 467A, 467B, 467C may be generated to simulate the facial expressions that may be generated by, for example, Figure 2 Frames provided by a user-facing camera of a head-mounted device of the head-mounted XR system 202.

[0048] Return to Figure 4A In some implementations, the non-slanted branch 410 may be trained before the slanted branch 450. For example, Figure 4A, the desired output of the local feature engine 400 is one or more cropped UV textures (also referred to herein as local UV textures) for partial regions of the full face. The non-tilted branch 410 can be trained to generate an output image that is a local UV texture 473A, 473B, 473C that is the same as the non-tilted input frame 471A, 471B, 471C. In some cases, training the non-tilted branch 410 including the encoder 412, the latent code 414, and the decoder 420 can benefit from the desired output image being the same as the input training image of the training dataset. For example, training the non-tilted branch 410 can first accelerate the convergence of machine learning model parameters (e.g., weights and / or biases) by limiting the variability between the input frames (e.g., 471A, 471B, 47C) and the output local UV textures 473A, 473B, 473C. In contrast, the tilted input frames 467A, 467B, 467C are rotated at different orientations relative to the desired local UV textures 473A, 473B, 473C, resulting in additional variations that must be considered during the training process of the local feature engine 400. After the non-tilted branch 410 is trained, the latent code 414 can provide a compact representation of the encoded non-tilted input frames 471A, 471B, 471C. For example, each of the input frames can have a size of 256 pixels x 256 pixels x 3 colors (e.g., red, green, and blue). In some cases, the latent code 414 can be trained to represent facial frames using a latent code vector that includes 256 features.

[0049] In some cases, a training data set of non-tilted frames 471A, 471B, 471C may be provided as input to the non-tilted branch of 410 for multiple iterations (also referred to as epochs). In some examples, a back-propagation training process may be performed to adjust the weights (and in some cases, other parameters such as biases) of the nodes (e.g., non-tilted branch 410) of the neural network. Figure 8 Described in more detail, back propagation may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, the loss function, the backward pass, and the parameter update may be performed for each training iteration (e.g., for each batch of data from the training set or for the entire training data set). The back propagation process may be repeated for a specific number of iterations for each training data set until the weights of the parameters of the non-tilted branch 410 are accurately tuned.

[0050] During training, a loss function may be used to penalize errors in the output local UV textures 473A, 473B, 473C generated by the VAE (e.g., encoder 412, latent code 414, and decoder 420) of the non-slanted branch 410. In one illustrative example, the reconstruction loss function To train the tilt branch, as shown in the following formula (1):

[0051] (1)

[0052] in is the synthesized output image , is a ground truth image (e.g., input frames 471A, 471B, 471C), and and According to the weighting coefficient and The values ​​of are combined in a weighted sum in a loss function. In an illustrative example, the weight coefficient values ​​can be and .

[0053] The loss is illustrated in the following formula (2).

[0054] (2)

[0055] Very Deep Convolutional Network (VGG) Loss Function A neural network is utilized that can provide a measure of content and / or style loss between an input frame (e.g., 467A, 467B, 467C, 471A, 471B, 471B) and an output image (e.g., 473A, 473B, 473C) produced by the two branches 410, 450 of the local feature engine 400. In one illustrative example, a pre-trained VGG neural network (e.g., a VGG19 convolutional network) comprising 19 convolutional layers can be used. Further details of the VGG19 network are described in "Very Deep Convolutional Networks for Large Scale Image Reconstruction" by Karen Simonyan et al. (2014).

[0056] In some implementations, the Kullback-Leibler (KL) loss The latent code 414 may be used to directly penalize leaning branches during training, as illustrated in the following formula (3):

[0057] (3)

[0058] In some specific implementations, The loss can be used to force the statistical distribution within the latent codes to approach a normal distribution.

[0059] For the tilt branch 450, the total loss function can be the reconstruction loss and KL loss The weighted sum of is shown in the following formula (4):

[0060] (4)

[0061] in and is the weighting factor. In one illustrative example, the weighting factor value may be and .

[0062] Reference Figure 4C , after training the non-slanted branch 410, the local feature engine 400 may select the slanted branch 450 via the switch 416. In some implementations, during the phase of training the slanted branch 450, the parameters of the decoder 420 may be fixed. The loss function for training the non-slanted network may include the reconstruction loss shown in formula (1): and the KL loss as shown in formula (3) In addition, the code loss It can be used to constrain the latent code 454 of the slanted branch to be close to the latent code 414 of the non-slanted branch 410 previously trained as described above. An example code loss is shown in the following formula (5):

[0063] (5)

[0064] in is the vector represented in the latent code 454 of the tilted branch 450, and is the vector represented in the latent code 414 of the non-slanted branch.

[0065] The loss is illustrated in the following formula (6).

[0066] (6)

[0067] The total loss function of the non-tilted branch 410 is shown in the following formula (7):

[0068] (7)

[0069] in , and is the weighting factor. In one illustrative example, the weighting factor value may be , and .

[0070] In some cases, during training of non-slanted branch 450 , encoder 452 may learn to encode slanted frames 467A, 467B, and 467C into non-slanted views of a UV texture as shown in output local UV textures 473A, 473B, 473C.

[0071] In some cases, a training data set of tilted frames 467A, 467B, 467C may be provided as input to the non-tilted branch of 410 for multiple iterations (also referred to as epochs). In some examples, a back-propagation training process may be performed to adjust the weights (and in some cases, other parameters such as biases) of the nodes (e.g., non-tilted branch 410) of the neural network. Figure 8 Described in more detail, back propagation may include a forward pass, a loss function, a backward pass, and a weight update. A forward pass, a loss function, a backward pass, and a parameter update may be performed for each training iteration (e.g., for each batch of data from a training set or for the entire training data set). The back propagation process may be repeated for a specific number of iterations for each training data set until the weights of the parameters of the non-tilted branch 410 are accurately tuned. After training, the latent code 454 may provide a compact representation of the encoded tilted input frames 467A, 467B, 467C. For example, each of the input frames may have a size of 256 pixels × 256 pixels × 3 colors (e.g., red, green, and blue). In some cases, the latent code 454 may be trained to represent a facial image using a latent code vector comprising 256 features.

[0072] During inference (e.g., after training is complete), the tilted branch can receive tilted frames (e.g., from a user-facing camera of a head-mounted XR system) and generate non-tilted cropped UV textures of local facial features such as the mouth and eyes. In some implementations, the non-tilted branch 410 of the local feature engine 400 can be disabled and / or discarded during inference. In some cases, the output of the local feature engine 400 is used to generate a full-face UV texture.

[0073] Reference Figure 3, the facial reconstruction engine 310 may receive the coefficients generated by the 3D model fitting engine 306 to generate a 3D model (e.g., 3DMM). The 3D model may be generated or constructed as a linear combination of an average face (sometimes referred to as a neutral face), a facial shape basis vector, and a facial expression basis vector. The average face may represent an average face that may be transformed (e.g., by a shape basis vector and an expression basis vector) to achieve the desired final 3D facial shape of the 3D model. The facial shape basis vector may be used to scale the proportions of the average face. In some cases, the facial shape basis vector may be used to represent a fat face or a thin face, a small nose or a large nose, and any adjustments to the basic facial shape. In some implementations, the facial shape basis vector is determined based on principal component analysis (PCA). In some cases, the facial expression basis vector may represent facial expressions, such as smiling, raising eyebrows, winking, blinking, frowning, and the like.

[0074] An illustrative example of a facial expression basis vector is a blend shape. As used herein, a blend shape may correspond to an approximate semantic parameterization of all or a portion of a facial expression. For example, a blend shape may correspond to a complete facial expression, or to a "partial" (e.g., "Δ") facial expression. Examples of partial expressions include raising an eyebrow, closing an eye, moving one side of the face, etc. In one example, a single blend shape may approximate the linearized effect of the movement of a single facial muscle. In some cases, a semantic representation may be modeled to correspond to the movement of one or more facial muscles.

[0075] 3D model generated using 3D model fitting technology The 3DMM (eg, generated using a 3DMM fitting technique) may be a statistical model representing the 3D geometry of an object (eg, a face). For example, a 3D model (eg, a 3DMM) may be composed of an average face and the facial shape basis (also called basis vectors) and facial expression bases and facial shape factor and facial expression coefficient It can be expressed as a linear combination of, for example, as follows:

[0076] (8)

[0077] In the example of formula (1), there are M facial shape coefficients and M facial shape basis vectors , where M is an integer greater than or equal to 1. In some implementations, the average face , facial shape basis vector and facial expressions vector Each of the may include may be combined to form a 3D model The position information (e.g., x, y, and z coordinates) of the 3D vertices of the face. In some implementations, the facial shape basis vectors and facial expressions vector It can be expressed as relative to the average face The position shift of and the coefficient of facial expression Provides a scaling factor for the corresponding offset vector. In one illustrative example, the 3D model S includes three thousand 3D vertices. In one illustrative example, M is equal to 219, which corresponds to 219 facial shape basis vectors and facial shape factor In some implementations, the shape basis vectors The principal component analysis feature vector may be included. In some cases, there are N facial expression coefficients and N facial expression basis vectors, where N is an integer greater than or equal to 1. In some cases, the facial expression vector May include blend shape vectors. In one illustrative example, N is equal to 39, which corresponds to 39 facial expression basis vectors and 39 facial expression coefficients (e.g., 39 blend shapes and 39 blend shape coefficients). In some cases, the result of the linear combination shown in equation (1) can be a 3D model (e.g., 3DMM) of the face in a neutral pose. In some examples, the 3D model can be rotated to match the facial pose in image frame 302 using pose information such as yaw, pitch, and roll values.

[0078] In some implementations, the facial reconstruction engine 310 may generate a UV facial position map (also referred to as a UV position map or UV texture) for an input image frame 302 that includes a face. In some examples, the local UV texture output by the local feature engine 400 may be combined with the full facial features represented in the texture map generated by the facial reconstruction engine 310 to improve the quality of the appearance of the parts of the face represented in the local UV texture (e.g., the mouth and eyes). In some cases, the UV facial position map may be applied as a texture to the 3DMM generated by the 3D model fitting engine 306. In some examples, the 3DMM with the texture provided from the UV facial position map may be used to render a 3D digital representation of the input image. In some implementations, a machine learning model (e.g., a neural network) may be used to generate the UV facial position map.

[0079] Figure 5AAn example configuration of a facial reconstruction engine 500 is illustrated that can combine local features from the local feature engine 400 with a full-face UV texture. In some cases, combining local features from the local feature engine 400 with a full-face UV texture can be used to enhance the appearance of portions of the full face that correspond to portions of the face that correspond to local features (e.g., left eye, right eye, and / or mouth). In some cases, Figure 5A The facial reconstruction engine 500 may correspond to Figure 3 As described above, the facial reconstruction engine 500 may receive an input frame 302 of a face and generate a 3DMM and a corresponding full facial UV feature tensor 510. Figure 5A In the illustrated example, the machine learning model including the encoder 512 and the decoder 516 may receive the full facial UV feature tensor 510 as input and generate a combined UV texture 560 as output. In addition, the facial reconstruction engine 500 may include a fusion engine 550 that may combine the local UV textures 473A, 473B, 473C generated by the local feature engine 400 with the full facial features determined based on the full facial UV feature tensor 510 to generate a combined UV texture 560 including additional details provided by the local feature engine 400.

[0080] In one illustrative example (not shown), the local UV textures 473A, 473B, 473C output by the local feature engine 400 may be used to directly replace corresponding pixel locations of the full facial UV texture generated by the face reconstruction engine 500. However, directly replacing the local texture may result in artifacts at the boundaries between the replaced pixels and the original pixels of the full facial UV texture. Figure 5A In the illustrated example, the feature fusion engine 550 may be used to combine the local intermediate features 552 (e.g., features at different resolutions between layers 427A, 427B, 427C of the decoder 420) from the local feature engine 400 with the global intermediate features 517 (e.g., features at different resolutions between layers 514A, 514B, 514C) from the decoder 516 to produce a combined UV texture 560 as output. In some cases, combining the local texture with the full facial UV texture may enhance portions of selected areas (e.g., left eye, right eye, and / or mouth) of the combined UV texture.

[0081] Figure 5BAn example system diagram of a fusion engine 580 is illustrated. In the illustrated example, the intermediate features 552 from the local feature engine 400 are received by the convolution layer 554, and the output of the convolution layer 554 is received by the adaptive instance normalization (AdaIN) block 556. The AdaIN block can adjust the distribution of the local intermediate features 552 to match the distribution of the global intermediate features 517. In an illustrative example, the AdaIN block 556 can subtract the mean of the local intermediate features 552 and divide the local intermediate features 552 from which the mean is subtracted by their variance to normalize the local intermediate features 552 (e.g., to have a Gaussian distribution). In some cases, the AdaIN block 556 can multiply the normalized local intermediate features 552 by the variance of the global intermediate features 517 and add the mean of the global intermediate features 517. In some cases, the AdaIN block 556 can match the pattern of the local intermediate features 552 to the pattern of the global intermediate features 517.

[0082] The output of the AdaIN block 556 may be subtracted from the global features 517 at a difference block 558, and the difference between the global intermediate features 517 and the output of the AdaIN block 556 may be received by a convolutional layer 562. The output of the convolutional layer 562 may be used to generate an attention map 564. In some cases, the attention map may be used to determine weights for combining the local intermediate features 552 with the global intermediate features 517. Based on the determined weights from the generated attention map, the global intermediate features 517 and the local intermediate features 552 may be scaled according to the determined weights at a multiplication block 566 and added together at an addition block 568. The resulting output of the addition block 568 may be a combined feature 569. In some cases, the attention map may determine the weights that provide the best overall appearance of the combined features based on the input global intermediate features 517 and the local intermediate features 552. It should be understood that more or fewer components may be included in the fusion engine 550 without departing from the scope of the present disclosure. In some cases, other techniques besides weighted sums may be used to generate combined features 569. In one illustrative example, global intermediate features 517 may be combined with local intermediate features 552 using an alpha blending technique.

[0083] Return to Figure 5A , the facial reconstruction engine 500 may include a fusion engine 550 between multiple layers of the decoder 516 to combine local intermediate features 552 and global intermediate features 517 at different resolutions, thereby generating combined intermediate features at multiple stages of the decoder 516.

[0084] In some cases, a training data set of oblique frames 467A, 467B, 467C and / or input frames 502 may be provided as input to the local feature engine 400 and the facial reconstruction engine 500, respectively, for multiple iterations (also referred to as epochs). In one illustrative example, the training data set may include oblique frames captured by a multi-view camera system. In some examples, a back-propagation training process may be performed to adjust the weights (and in some cases, other parameters such as biases) of the nodes of the neural network (e.g., the oblique branch 450 of the local feature engine 400, the facial reconstruction engine 500). For example, relative to Figure 8 Described in more detail, back propagation may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, the loss function, the backward pass, and the parameter update may be performed for each training iteration (e.g., for each batch of data from a training set or for the entire training data set). The back propagation process may be repeated for a specific number of iterations for each training data set until the weights of the parameters of the local feature engine 400 and the facial reconstruction engine 500 are accurately tuned.

[0085] During training, a loss function may be used to penalize errors in the combined UV texture 560 generated by the face reconstruction engine 500. In some implementations, the loss function may include a reconstruction loss for the entire face, and a reconstruction loss for a local region (e.g., corresponding to the local UV textures 473A, 473B, 473C) may be used. An example reconstruction loss for the entire face is shown in the following formula (9):

[0086] (9)

[0087] in is the synthesized output image of the facial reconstruction engine 500 (e.g., the combined UV texture 560), is a ground truth image (e.g., input frame 502), , and is the weighting factor. In one illustrative example, the weighting factor value may be and In addition to the above with respect to formulas (1) and (7) and In addition, the multi-scale structural similarity (MS-SSIM) loss function can be used to detect image degradation.

[0088] In addition to the full face loss function, a similar loss function may be applied to each of the cropped local images generated by the tilt branch 450 of the local feature engine 400, as shown in formula (10):

[0089] (10)

[0090] In some implementations, the face reconstruction engine 500 may also be trained using a deep generative neural network model (e.g., a generative adversarial network (GAN)). A GAN is a generative neural network that can learn patterns in input data so that the neural network model can generate new synthetic outputs that are reasonably likely to come from the original data set. The GAN may include two neural networks operating together. One of the neural networks (referred to as a generative neural network or generator, denoted as G(z)) generates the synthetic output, while the other neural network (referred to as a discriminative neural network or discriminator, denoted as D(X)) evaluates the synthetic output for authenticity (whether the synthetic output comes from the original data set, such as the training data set, or is generated by the generator). The generator G(z) may correspond to the face reconstruction engine 310 and / or the face reconstruction engine 500. The generator is trained to try and deceive the discriminator into determining that the synthetic video frame (or group of video frames) generated by the generator is a real video frame (or group of video frames) from a training data set (e.g., a first set of training video data). The training process continues, and the generator becomes better at generating synthetic video frames that look like real video frames. The discriminator continues to find defects in the synthetic video frames, and the generator figures out what the discriminator is looking at to determine the defects in the image. Once the network is trained, the generator is able to produce realistic video frames that the discriminator cannot distinguish from real video frames.

[0091] In an illustrative example, a GAN may generate facial textures, and the goal of the GAN is to generate facial textures with local texture fusion. There is a dual aspect between the generator G and the discriminator D with respect to parameter maximization of the discriminator D. The discriminator D will try to distinguish between real facial textures (from a set of training data) and fake facial textures (generated by the generator G based on a second set of training video data) as well as possible, and the generator G should minimize the ability of the discriminator D to identify fake images. The parameters of the generator G (e.g., the weights of the nodes of the neural network and other parameters in some cases, such as bias) can be adjusted during the training process so that the generator G will output video frames that are indistinguishable from real video frames associated with the second domain. A loss function can be used to analyze the errors in the generator G and the discriminator D. In an illustrative example, a binary cross entropy loss function can be used. In some cases, other loss functions can be used.

[0092] In some specific implementations, the generator G of the GAN can be trained using the loss function shown in the following formula (11):

[0093] (11)

[0094] in is the UV feature tensor (e.g., Figure 5AUV feature tensor 510), Compute synthetic images from the training dataset (e.g., synthetic textures), and is the distribution of synthetic images in the training dataset.

[0095] In some specific implementations, the generator G of the GAN can be trained using the loss function shown in the following formula (12):

[0096] (12)

[0097] in Calculate the expected value of real images y (e.g., ground truth images) in the training dataset, is the probability distribution of real images, represents a real image, and Represents a composite image.

[0098] (13)

[0099] in Represents the layers of the generator network (e.g., Figure 5A 514A, 514B, 514C). As above with respect to Figure 5A As described above, local features (e.g., from Figure 3 The local feature engine 308 and / or Figure 4A The fusion of local features engine 400) with global features (e.g., from the global branch of the face reconstruction engine 500).

[0100] (14)

[0101] Figure 3 The illustrated 3D modeling system 300 illustrates a functional block diagram that can be implemented in hardware, software, or any combination of hardware and software. As an example, the functional blocks shown can use the relationships shown to identify functional blocks that can be converted into a Universal Modeling Language (UML) diagram to identify an exemplary implementation of the 3D modeling system 300 at least in part as an object-oriented arrangement in software. However, the 3D modeling system 300 can be implemented without abstractions and, for example, as a static functional implementation. In some examples, the 3D modeling system 300 can be implemented by a user having Fig. 9 The architecture of computing system 900 is shown as being implemented in whole or in part by a computing device or system.

[0102] Figure 6A flow chart of a process 600 for generating a texture for a 3D facial model is illustrated. At block 602, process 600 includes obtaining a first frame, the first frame including a first portion of a face. In some cases, the first frame is a frontal image of the face in a neutral pose. At block 604, process 600 includes generating a 3D facial model based on the first frame. In one illustrative example, Figure 3 The 3DMM fitting engine 306 can generate a 3D facial model.

[0103] At block 606, process 600 includes generating a first facial feature corresponding to a first portion of a face. In one illustrative example, Figure 3 The facial reconstruction engine 310 may generate a first facial feature.

[0104] At box 608, process 600 includes obtaining a second frame, the second frame including a second portion of the face. In some cases, the second portion of the face at least partially overlaps the first portion of the face. In some aspects, the first portion of the face includes a frontal depiction of the entire face, and the second portion of the face includes a local area of ​​the face that is smaller than the frontal depiction of the entire face. In some examples, the second frame is obtained from a user-facing camera of the head-mounted device. For example, the second frame may include an image of the left eye, the right eye, and / or the mouth.

[0105] At block 610 , process 600 includes generating a second facial feature corresponding to a second portion of the face.

[0106] At block 612, process 600 includes combining a first facial feature with a second facial feature to generate an enhanced facial feature. In some examples, the combining is performed to enhance the appearance of a selected area of ​​the enhanced facial feature. Combining the first facial feature with the second facial feature includes generating a weighted average of the first facial feature and the second facial feature. In some examples, generating the weighted average of the first facial feature and the second facial feature includes: generating an attention map; and based on the attention map, generating weighting coefficients for combining the first facial feature with the second facial feature. In some embodiments, combining the first facial feature with the second facial feature includes combining a first intermediate feature associated with the first facial feature (e.g., from Figure 4A ) and a second intermediate feature associated with the second facial feature (e.g., from Figure 5AIn some examples, the method includes combining a third intermediate feature associated with the first facial feature (e.g., a feature from a different layer 514A, 514B, and / or 514C of decoder 516) and a fourth intermediate feature associated with the second facial feature (e.g., a feature from a different layer 514A, 514B, and / or 514C of decoder 516). Figure 5A In some cases, the first intermediate feature and the second intermediate feature are associated with a first feature resolution, and the third intermediate feature and the fourth intermediate feature are associated with a second feature resolution different from the first feature resolution.

[0107] In some examples, process 600 includes generating a full facial texture associated with the 3D facial model. In some cases, the full facial texture includes enhanced facial features.

[0108] In some cases, the first facial feature is generated by a first machine learning model and the second facial feature is generated by a second machine learning model different from the first machine learning model. In some cases, the second portion of the face includes an oblique view of the face, and the second facial feature includes a portion of a full facial UV texture that is smaller than the full facial UV texture. In some examples, the second machine learning model includes a latent vector corresponding to the oblique view of the face. In some aspects, the first machine learning model and the second machine learning model are trained separately.

[0109] In some aspects, process 600 includes: obtaining a third frame, the third frame including a third portion of the face that is different from the second portion of the face; generating a third facial feature corresponding to the third portion of the face; obtaining a fourth frame, the fourth frame including a fourth portion of the panel that is different from the second portion of the face and the third portion of the face; generating a fourth facial feature corresponding to the fourth portion of the face, and combining the first facial feature with the second facial feature, the third facial feature, and the fourth facial feature. In some examples, the third portion of the face at least partially overlaps the first portion of the face. In some examples, the fourth portion of the face at least partially overlaps the first portion of the face. In some cases, the combining is performed to enhance the appearance of at least one or more of the second portion of the face, the third portion of the face, or the fourth portion of the face. In some examples, the selected area of ​​the enhanced facial feature corresponds to a left eye, a right eye, and a mouth.

[0110] In other examples, the device may include an application or functionality for performing some of the processes described herein (e.g., process 600 and / or any other process described herein). In some examples, the processes described herein (e.g., process 600 and / or any other process described herein) may be performed by a computing device or apparatus. In some examples, process 600 may be performed by the 3D modeling system 300 and / or the facial reconstruction engine 500 described above. In another example, process 600 may be performed by a computer having Fig. 9 The architecture of computing system 900 is shown as being implemented by a computing device or system.

[0111] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, an extended reality (XR) device or system (e.g., a VR headset, an AR headset, AR glasses, or other XR devices or systems), a wearable device (e.g., a connected watch or smartwatch, or other wearable device), a server computer or system, a vehicle or a computing device of a vehicle (e.g., an autonomous vehicle), a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein (including process 600). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0112] The components of the computing device may be implemented in circuits. For example, the components may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0113] Process 600 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0114] Additionally, process 600 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, implemented by hardware, or implemented by a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0115] Figure 7 is an illustrative example of a deep learning neural network 700 that can be used by a 3D model training system. An input layer 720 includes input data. In an illustrative example, the input layer 720 may include data representing pixels of an input video frame. The neural network 700 includes multiple hidden layers 722a, 722b to 722n. The hidden layers 722a, 722b to 722n include "n" hidden layers, where "n" is an integer greater than or equal to one. The multiple hidden layers can include as many layers as required for a given application. The neural network 700 also includes an output layer 724, which provides an output generated by the processing performed by the hidden layers 722a, 722b to 722n. In an illustrative example, the output layer 724 can provide a classification of objects in the input video frame. The classification can include a category that identifies the type of object (e.g., a person, a dog, a cat, or other object).

[0116] Neural network 700 is a multi-layer neural network composed of interconnected nodes. Each node can represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains the information as it processes it. In some cases, neural network 700 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back to itself. In some cases, neural network 700 may include a recursive neural network, which may have loops that allow information to be carried across nodes when reading in input.

[0117] Information can be exchanged between nodes by interconnecting nodes to nodes between each layer. The nodes of input layer 720 can activate a group of nodes in the first hidden layer 722a. For example, as shown in the figure, each input node in the input nodes of input layer 720 is connected to each node in the node of the first hidden layer 722a. The nodes of hidden layers 722a, 722b to 722n can transform the information by applying activation functions to the information of each input node. The information derived from the transformation can then be transferred to the nodes of the next hidden layer 722b and can activate the nodes of the next hidden layer, and the nodes of the next hidden layer can perform their own specified functions. Example functions include convolution, upsampling, data conversion and / or any other suitable function. Then, the output of hidden layer 722b can activate the nodes of the next hidden layer, and so on. The output of last hidden layer 722n can activate one or more nodes of output layer 724, and output is provided at the one or more nodes. In some cases, although a node in neural network 700 (eg, node 726 ) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0118] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 700. Once neural network 700 is trained, it may be referred to as a trained neural network, which may be used to classify one or more objects. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have a tunable digital weight that may be tuned (e.g., based on a training data set), thereby allowing neural network 700 to adapt to the input and be able to learn as more and more data is processed.

[0119] The neural network 700 is pre-trained to process features from the data in the input layer 720 using different hidden layers 722a, 722b to 722n to provide an output through the output layer 724. In an example where the neural network 700 is used to identify objects in an image, the neural network 700 may be trained using training data that includes both images and labels. For example, training images may be input into the network, where each training image has a label indicating the class of one or more objects in each image (essentially, indicating to the network what the objects are and what features they have). In one illustrative example, the training images may include an image of the number 2, in which case the label for the image may be [0 0 1 0 0 0 0 0 0 0].

[0120] In some cases, the neural network 700 may use a training process known as back propagation to adjust the weights of the nodes. Back propagation may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, the loss function, the backward pass, and the parameter update are performed for one training iteration. For each set of training images, this process may be repeated a certain number of iterations until the neural network 700 is trained well enough so that the weights of each layer are accurately tuned.

[0121] For the example of identifying an object in an image, a forward pass may include passing a training image through the neural network 700. Prior to training the neural network 700, the weights are initially randomized. The image may include, for example, an array of numbers representing pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 digital array having 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or a brightness and two chrominance components, etc.).

[0122] For the first training iteration of the neural network 700, the output will likely include values ​​that do not favor any particular class due to the weights randomly selected at initialization. For example, if the output is a vector with probabilities that the object includes different classes, the probability values ​​for each of the different classes may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 700 cannot determine low-level features and therefore cannot accurately determine what the classification of the object is likely to be. A loss function may be used to analyze the error in the output. Any suitable loss function definition may be used. An example of a loss function includes the mean squared error (MSE). The MSE is defined as , which computes the sum of the actual answer minus one-half the square of the predicted (output) answer. The loss can be set equal to The value of .

[0123] For the first training images, the loss (or error) will be high because the actual values ​​will be very different from the predicted outputs. The goal of training is to minimize the amount of loss so that the predicted outputs are the same as the training labels. The neural network 700 can perform a backward pass by determining which inputs (weights) contribute most to the network's loss, and the weights can be adjusted so that the loss decreases and is ultimately minimized.

[0124] The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes most to the loss of the network. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as , where w represents the weight, w i Represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes a larger weight update, while a lower value indicates a smaller weight update.

[0125] Neural network 700 may include any suitable deep network. One example includes a convolutional neural network (CNN) that includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. Figure 8 An example of a CNN is described. The hidden layer of the CNN includes a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. The neural network 700 may include any other deep network other than a CNN, such as an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), etc.

[0126] Figure 8 is an illustrative example of a convolutional neural network 800 (CNN 800). An input layer 820 of CNN 800 includes data representing an image. For example, the data may include an array of numbers representing pixels of the image, where each number in the array includes a value from 0 to 255 describing the intensity of the pixel at the location of the number in the array. Using the previous example from above, the array may include a 28×28×3 digital array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luminance and two chrominance components, etc.). The image may be passed through a convolutional hidden layer 822a, an optional non-linear activation layer, a pooling hidden layer 822b, and a fully connected hidden layer 822c to obtain an output at an output layer 824. Although Figure 8 Only one hidden layer among the hidden layers is shown in , but one of ordinary skill will appreciate that multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 800. As previously described, the output may indicate a single category of an object, or may include a probability of a category that best describes an object in the image.

[0127] The first layer of CNN 800 is a convolutional hidden layer 822a. The convolutional hidden layer 822a analyzes the image data of the input layer 820. Each node of the convolutional hidden layer 822a is connected to an area of ​​nodes (pixels) of the input image called a receptive field. The convolutional hidden layer 822a can be considered as one or more filters (each filter corresponds to a different activation or feature map), where each convolution iteration of the filter is a node or neuron of the convolutional hidden layer 822a. For example, the area of ​​the input image covered by the filter at each convolution iteration will be the receptive field of the filter. In an illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in the convolutional hidden layer 822a. Each connection between a node and the receptive field of that node learns a weight, and in some cases learns an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node of the hidden layer 822a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has an array of weights (numbers) and the same depth as the input. For the video frame example, the filter would have a depth of 3 (according to the three color components of the input image). An illustrative example size of the filter array is 5×5×3, corresponding to the size of the node's receptive field.

[0128] The convolutional nature of the convolutional hidden layer 822a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 822a may start at the upper left corner of the input image array and may be convolved around the input image. As noted above, each convolution iteration of the filter may be considered as a node or neuron of the convolutional hidden layer 822a. In each convolution iteration, the value of the filter is multiplied by the original number of pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the upper left corner of the input image array). The multiplications from each convolution iteration may be added together to obtain the sum for that iteration or node. Next, the process is continued at the next position in the input image according to the receptive field of the next node in the convolutional hidden layer 822a.

[0129] For example, the filter may move a stride amount to the next receptive field. The stride amount may be set to 1 or another suitable amount. For example, if the stride amount is set to 1, the filter will move 1 pixel to the right at each convolution iteration. Processing the filter at each unique position in the input volume produces a number representing the filter result at that position, thereby determining a sum value for each node of the convolution hidden layer 822a.

[0130] The mapping from the input layer to the convolutional hidden layer 822a is called an activation map (or feature map). The activation map includes a value for each node that represents the result of the filter at each location of the input volume. The activation map may include an array that includes various summed values ​​produced by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will include a 24×24 array. The convolutional hidden layer 822a may include several activation maps in order to identify multiple features in the image. Figure 8 The example shown includes three activation maps. Using these three activation maps, the convolutional hidden layer 822a can detect three different types of features, each of which is detectable across the entire image.

[0131] In some examples, a nonlinear hidden layer may be applied after the convolutional hidden layer 822a. A nonlinear layer may be used to introduce nonlinearity into a system that already computes linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer may apply a function f(x)=max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. Thus, the ReLU may increase the nonlinear properties of the CNN 800 without affecting the receptive field of the convolutional hidden layer 822a.

[0132] A pooling hidden layer 822b may be applied after the convolutional hidden layer 822a (and after the nonlinear hidden layer when used). The pooling hidden layer 822b is used to simplify the information in the output from the convolutional hidden layer 822a. For example, the pooling hidden layer 822b may take each activation map output from the convolutional hidden layer 822a and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by a pooling hidden layer. The pooling hidden layer 822a uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. A pooling function (e.g., a maximum pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 822a. In Figure 8 In the example shown in , three pooling filters are used for the three activation maps in the convolutional hidden layer 822a.

[0133] In some examples, max pooling may be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as a stride of 2) to the activation map output from the convolutional hidden layer 822a. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes in the previous layer (each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by a 2×2 max pooling filter at each iteration of the filter, where the maximum of the four values ​​is output as the "maximum" value. If such a max pooling filter is applied to an activation filter with a dimension of 24×24 nodes from the convolutional hidden layer 822a, the output from the pooling hidden layer 822b will be an array of 12×12 nodes.

[0134] In some examples, an L2-norm pooling filter may also be used. The L2-norm pooling filter involves computing the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than computing the maximum value as done in max pooling), and using the computed value as output.

[0135] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found anywhere in a region of an image, and discards the exact location information. This can be done without affecting the results of feature detection, because once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (and other pooling methods) provides the following benefits: far fewer features are pooled, thereby reducing the number of parameters required in subsequent layers of CNN 800.

[0136] The last layer of connections in the network is a fully connected layer that connects each node from the pooling hidden layer 822b to each output node in the output layer 824. Using the above example, the input layer includes 28×28 nodes that encode the pixel intensities of the input image, the convolutional hidden layer 822a includes 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for the filter) to the three activation maps, and the pooling layer 822b includes a layer of 3×12×12 hidden feature nodes based on applying a max pooling filter to a 2×2 region on each of the three feature maps. Extending this example, the output layer 824 may include ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 822b is connected to each node of the output layer 824.

[0137] The fully connected layer 822c can obtain the output of the previous pooling layer 822b (which should represent an activation map of high-level features) and determine the features that are most relevant to a particular category. For example, the fully connected layer 822c layer can determine the high-level features that are most strongly associated with a particular category and can include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 822c and the pooling hidden layer 822b can be calculated to obtain probabilities for different categories. For example, if the CNN 800 is used to predict that the object in the video frame is a person, there will be high values ​​in the activation map representing the high-level features of the person (e.g., there are two legs, there is a face at the top of the object, there are two eyes at the upper left and upper right of the face, there is a nose in the middle of the face, there is a mouth at the bottom of the face, and / or other features common to people).

[0138] In some examples, the output from the output layer 824 may include an M-dimensional vector (in the previous example, M=10), where M may include the number of categories that the program must choose from when classifying an object in an image. Other example outputs may also be provided. Each number in the N-dimensional vector may represent the probability that an object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector represents objects of ten different categories is [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates that the image is a 5% probability of an object of the third category (e.g., a dog), the image is an 80% probability of an object of the fourth category (e.g., a person), and the image is a 15% probability of an object of the sixth category (e.g., a kangaroo). The probability for a category can be considered as the confidence level that the object is part of that category.

[0139] Fig. 9 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig. 9 An example of a computing system 900 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 905. The connection 905 may be a physical connection using a bus, or a direct connection into the processor 910, such as in a chipset architecture. The connection 905 may also be a virtual connection, a networked connection, or a logical connection.

[0140] In some aspects, computing system 900 is a distributed system, where the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each of which performs some or all of the functionality for which the component is described. In some aspects, each component may be a physical or virtual device.

[0141] The example system 900 includes at least one processing unit (CPU or processor) 910 and connections 905 that couple various system components including system memory 915, such as read only memory (ROM) 920 and random access memory (RAM) 925, to the processor 910. The computing system 900 may include a cache 912 of high-speed memory directly connected to the processor 910, in close proximity to the processor, or integrated as part of the processor.

[0142] Processor 910 may include any general purpose processor and hardware or software services, such as services 932, 934, and 936 stored in storage device 930, that are configured to control processor 910 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 910 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0143] To enable user interaction, the computing system 900 includes an input device 945 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 900 may also include an output device 935 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 900. The computing system 900 may include a communication interface 940, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple ® Lightning ® Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Bluetooth ® Wireless signal transfer, Bluetooth ® Low Energy (BLE) wireless signal transfer, IBEACON ®Wireless signal transfer, radio frequency identification (RFID) wireless signal transfer, near field communication (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 802.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transfer, public switched telephone network (PSTN) signal transfer, integrated services digital network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communication interface 940 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 900 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0144] The storage device 930 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick ®card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0145] Storage device 930 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 910, causes the system to perform functions. In some aspects, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to the necessary hardware components (such as processor 910, connection 905, output device 935, etc.).

[0146] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or over a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may have stored thereon code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0147] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0148] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0149] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0150] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0151] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptops, mobile phones (e.g., smart phones or other types of mobile phones), tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0152] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0153] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be appreciated that, in alternative aspects, each method can be performed in a different order than described.

[0154] It should be understood by those skilled in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to (" ”) and greater than or equal to (“ ”) symbol instead.

[0155] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0156] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (eg, connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0157] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0158] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0159] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that, when executed, perform one or more of the above methods. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0160] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0161] Illustrative aspects of the present disclosure include:

[0162] Aspect 1. A method for generating a texture of a 3D facial model, comprising: obtaining a first frame, the first frame including a first part of a face; generating a 3D facial model based on the first frame; generating a first facial feature corresponding to the first part of the face; obtaining a second frame, the second frame including a second part of the face, wherein the second part of the face at least partially overlaps with the first part of the face; generating a second facial feature corresponding to the second part of the face; and combining the first facial feature with the second facial feature to generate an enhanced facial feature, wherein the combining is performed to enhance the appearance of a selected area of ​​the enhanced facial feature.

[0163] Aspect 2. The method according to aspect 1, wherein the first frame is a frontal image of the face in a neutral pose.

[0164] Aspect 3. A method according to any one of Aspects 1 to 2, wherein the second frame is obtained from a user-facing camera of a head-mounted device.

[0165] Aspect 4. The method according to any one of aspects 1 to 3, further comprising generating a full facial texture associated with the 3D facial model, wherein the full facial texture includes the enhanced facial features.

[0166] Aspect 5. A method according to any one of Aspects 1 to 4, wherein the first portion of the face comprises a frontal depiction of the entire face, and the second portion of the face comprises a local area of ​​the face that is smaller than the frontal depiction of the entire face.

[0167] Aspect 6. A method according to any one of aspects 1 to 5, wherein the second facial feature comprises a local feature associated with at least one or more of an eye or a mouth.

[0168] Aspect 7. The method according to any one of aspects 1 to 6, wherein combining the first facial feature with the second facial feature comprises generating a weighted average of the first facial feature and the second facial feature.

[0169] Aspect 8. A method according to any one of Aspects 1 to 7, wherein generating the weighted average of the first facial feature and the second facial feature includes: generating an attention map; and based on the attention map, generating a weighting coefficient for combining the first facial feature with the second facial feature.

[0170] Aspect 9. The method according to any one of Aspects 1 to 8, wherein combining the first facial feature with the second facial feature comprises combining a first intermediate feature associated with the first facial feature and a second intermediate feature associated with the second facial feature.

[0171] Aspect 10. The method according to any one of Aspects 1 to 9 further includes combining a third intermediate feature associated with the first facial feature and a fourth intermediate feature associated with the second facial feature, wherein: the first intermediate feature and the second intermediate feature are associated with a first feature resolution; and the third intermediate feature and the fourth intermediate feature are associated with a second feature resolution different from the first feature resolution.

[0172] Aspect 11. A method according to any one of Aspects 1 to 10, wherein the first facial feature is generated by a first machine learning model and the second facial feature is generated by a second machine learning model different from the first machine learning model.

[0173] Aspect 12. A method according to any one of Aspects 1 to 11, wherein the second portion of the face comprises an oblique view of the face, and the second facial feature comprises a portion of a full facial UV texture, the portion being smaller than the full facial UV texture.

[0174] Aspect 13. A method according to any one of Aspects 1 to 12, wherein the second machine learning model includes a latent vector corresponding to the oblique view of the face.

[0175] Aspect 14. A method according to any one of Aspects 1 to 13, wherein the first machine learning model and the second machine learning model are trained separately.

[0176] Aspect 15. The method according to any one of Aspects 1 to 14, further comprising: obtaining a third frame, the third frame comprising a third part of the face different from the second part of the face, wherein the third part of the face at least partially overlaps with the first part of the face; generating a third facial feature corresponding to the third part of the face; obtaining a fourth frame, the fourth frame comprising a fourth part of the face different from the second part of the face and the third part of the face, wherein the fourth part of the face at least partially overlaps with the first part of the face; generating a fourth facial feature corresponding to the fourth part of the face, and combining the first facial feature with the second facial feature, the third facial feature and the fourth facial feature, wherein the combination is performed to enhance the appearance of at least one or more of the second part of the face, the third part of the face or the fourth part of the face.

[0177] Aspect 16. The method according to any one of aspects 1 to 15, wherein the selected areas of the enhanced facial features correspond to a left eye, a right eye, and a mouth.

[0178] Aspect 17: An apparatus for generating a texture of a 3D facial model. The apparatus comprises at least one memory (e.g., implemented in a circuit) and a processor (or multiple processors) coupled to the at least one memory. The processor (or multiple processors) is configured to: obtain a first frame, the first frame comprising a first portion of a face; generate a 3D facial model based on the first frame; generate a first facial feature corresponding to the first portion of the face; obtain a second frame, the second frame comprising a second portion of the face, wherein the second portion of the face at least partially overlaps with the first portion of the face; generate a second facial feature corresponding to the second portion of the face; and combine the first facial feature with the second facial feature to generate an enhanced facial feature, wherein the combination is performed to enhance the appearance of a selected area of ​​the enhanced facial feature.

[0179] Clause 18: The apparatus of Clause 17, wherein the first frame is a frontal image of a face in a neutral pose.

[0180] Aspect 19: An apparatus according to any one of Aspects 17 to 18, wherein the second frame is obtained from a user-facing camera of a head mounted device.

[0181] Aspect 20: The apparatus according to any one of aspects 17 to 19, wherein the processor is configured to: generate a full facial texture associated with the 3D facial model, wherein the full facial texture includes the enhanced facial features.

[0182] Aspect 21: An apparatus according to any one of Aspects 17 to 20, wherein the first portion of the face comprises a frontal depiction of the entire face, and the second portion of the face comprises a local area of ​​the face that is smaller than the frontal depiction of the entire face.

[0183] Aspect 22: The apparatus of any one of aspects 17 to 21, wherein the second facial feature comprises a local feature associated with at least one or more of an eye or a mouth.

[0184] Aspect 23: The apparatus of any one of aspects 17 to 22, wherein combining the first facial feature with the second facial feature comprises generating a weighted average of the first facial feature and the second facial feature.

[0185] Aspect 24: An apparatus according to any one of Aspects 17 to 23, wherein the processor is configured to: generate an attention map; and based on the attention map, generate a weighting coefficient for combining the first facial feature with the second facial feature.

[0186] Aspect 25: The apparatus according to any one of Aspects 17 to 24, wherein combining the first facial feature with the second facial feature comprises combining a first intermediate feature associated with the first facial feature and a second intermediate feature associated with the second facial feature.

[0187] Aspect 26: An apparatus according to any one of Aspects 17 to 25, wherein the processor is configured to: combine a third intermediate feature associated with the first facial feature and a fourth intermediate feature associated with the second facial feature, wherein: the first intermediate feature and the second intermediate feature are associated with a first feature resolution; and the third intermediate feature and the fourth intermediate feature are associated with a second feature resolution different from the first feature resolution.

[0188] Aspect 27: An apparatus according to any one of Aspects 17 to 26, wherein the first facial feature is generated by a first machine learning model and the second facial feature is generated by a second machine learning model different from the first machine learning model.

[0189] Aspect 28: An apparatus according to any one of Aspects 17 to 27, wherein the second portion of the face comprises an oblique view of the face, and the second facial feature comprises a portion of a full facial UV texture, the portion being smaller than the full facial UV texture.

[0190] Aspect 29: An apparatus according to any one of Aspects 17 to 28, wherein the second machine learning model includes a latent vector corresponding to the oblique view of the face.

[0191] Aspect 30: An apparatus according to any one of Aspects 17 to 29, wherein the first machine learning model and the second machine learning model are trained separately.

[0192] Aspect 31: An apparatus according to any one of Aspects 17 to 30, wherein the processor is configured to: obtain a third frame, the third frame including a third part of the face different from the second part of the face, wherein the third part of the face at least partially overlaps with the first part of the face; generate a third facial feature corresponding to the third part of the face; obtain a fourth frame, the fourth frame including a fourth part of the face different from the second part of the face and the third part of the face, wherein the fourth part of the face at least partially overlaps with the first part of the face; generate a fourth facial feature corresponding to the fourth part of the face, and combine the first facial feature with the second facial feature, the third facial feature and the fourth facial feature, wherein the combination is performed to enhance the appearance of at least one or more of the second part of the face, the third part of the face or the fourth part of the face.

[0193] Aspect 32: The apparatus according to any one of aspects 17 to 31, wherein the selected area of ​​the enhanced facial features corresponds to a left eye, a right eye, and a mouth.

[0194] Aspect 33: A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform any of the operations described in aspects 1 to 32.

[0195] Aspect 34: An apparatus comprising one or more means for performing any of the operations described in aspects 1 to 32.

Claims

1. A method for synthesizing facial features of a three-dimensional (3D) facial model, the method comprising: Obtaining a first frame, the first frame including a first portion of a face; generating a 3D facial model based on the first frame; generating a first facial feature corresponding to the first portion of the face; obtaining a second frame, the second frame comprising a second portion of the face, wherein the second portion of the face at least partially overlaps the first portion of the face; generating a second facial feature corresponding to the second portion of the face; as well as The first facial feature is combined with the second facial feature to generate an enhanced facial feature, wherein the combining is performed to enhance the appearance of a selected area of ​​the enhanced facial feature.

2. The method of claim 1, wherein the first frame is a frontal image of a face in a neutral pose.

3. The method of claim 1, wherein the second frame is obtained from a user-facing camera of a head mounted device. 4 . The method of claim 1 , further comprising generating a full facial texture associated with the 3D facial model, wherein the full facial texture includes the enhanced facial features.

5. The method of claim 1, wherein the first portion of the face comprises a frontal depiction of an entire face, and the second portion of the face comprises a localized area of ​​the face that is smaller than the frontal depiction of the entire face.

6. The method of claim 1, wherein combining the first facial feature with the second facial feature comprises generating a weighted average of the first facial feature and the second facial feature.

7. The method of claim 6, wherein generating the weighted average of the first facial feature and the second facial feature comprises: Generate attention map; as well as Based on the attention map, a weighting coefficient for combining the first facial feature with the second facial feature is generated.

8. The method of claim 1, wherein combining the first facial feature with the second facial feature comprises combining a first intermediate feature associated with the first facial feature and a second intermediate feature associated with the second facial feature.

9. The method of claim 8, further comprising combining a third intermediate feature associated with the first facial feature and a fourth intermediate feature associated with the second facial feature, wherein: The first intermediate feature and the second intermediate feature are associated with a first feature resolution; and The third intermediate features and the fourth intermediate features are associated with a second feature resolution different than the first feature resolution.

10. The method of claim 1, wherein the first facial feature is generated by a first machine learning model and the second facial feature is generated by a second machine learning model different from the first machine learning model.

11. The method of claim 10, wherein the second portion of the face comprises an oblique view of the face, and the second facial feature comprises a portion of a full facial UV texture, the portion being smaller than the full facial UV texture.

12. The method of claim 11, wherein the second machine learning model comprises a latent vector corresponding to the oblique view of the face.

13. The method of claim 10, wherein the first machine learning model and the second machine learning model are trained separately.

14. The method according to claim 1, further comprising: obtaining a third frame, the third frame comprising a third portion of the face different from the second portion of the face, wherein the third portion of the face at least partially overlaps the first portion of the face; generating a third facial feature corresponding to the third portion of the face; obtaining a fourth frame, the fourth frame comprising a fourth portion of the face that is different from the second portion of the face and the third portion of the face, wherein the fourth portion of the face at least partially overlaps with the first portion of the face; generating a fourth facial feature corresponding to the fourth portion of the face, and The first facial feature is combined with the second facial feature, the third facial feature, and the fourth facial feature, wherein the combining is performed to enhance the appearance of at least one or more of the second portion of the face, the third portion of the face, or the fourth portion of the face.

15. The method of claim 14, wherein the selected areas of the enhanced facial features correspond to a left eye, a right eye, and a mouth.

16. An apparatus for synthesizing facial features of a 3D facial model, comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: Obtaining a first frame, the first frame including a first portion of a face; generating a 3D facial model based on the first frame; generating a first facial feature corresponding to the first portion of the face; obtaining a second frame, the second frame comprising a second portion of the face, wherein the second portion of the face at least partially overlaps the first portion of the face; generating a second facial feature corresponding to the second portion of the face; as well as The first facial feature is combined with the second facial feature to generate an enhanced facial feature, wherein the combining is performed to enhance the appearance of a selected area of ​​the enhanced facial feature.

17. The apparatus of claim 16, wherein the first frame is a frontal image of a face in a neutral pose.

18. The apparatus of claim 16, wherein the second frame is obtained from a user-facing camera of a head mounted device.

19. The apparatus of claim 16, wherein the at least one processor is configured to generate a full facial texture associated with the 3D facial model, wherein the full facial texture includes the enhanced facial features.

20. The apparatus of claim 16, wherein the first portion of the face comprises a frontal depiction of an entire face, and the second portion of the face comprises a localized area of ​​the face that is smaller than the frontal depiction of the entire face.

21. The apparatus of claim 16, wherein to combine the first facial feature with the second facial feature, the at least one processor is configured to generate a weighted average of the first facial feature and the second facial feature.

22. The apparatus of claim 21, wherein the at least one processor is configured to: generating an attention map; and Based on the attention map, a weighting coefficient for combining the first facial feature with the second facial feature is generated.

23. The apparatus of claim 16, wherein to combine the first facial feature with the second facial feature, the at least one processor is further configured to combine a first intermediate feature associated with the first facial feature and a second intermediate feature associated with the second facial feature.

24. The apparatus of claim 23, wherein the at least one processor is configured to: combining a third intermediate feature associated with the first facial feature and a fourth intermediate feature associated with the second facial feature, wherein: The first intermediate feature and the second intermediate feature are associated with a first feature resolution; and The third intermediate features and the fourth intermediate features are associated with a second feature resolution different than the first feature resolution.

25. The apparatus of claim 16, wherein the at least one processor is configured to generate the first facial features using a first machine learning model and to generate the second facial features using a second machine learning model different from the first machine learning model.

26. The apparatus of claim 25, wherein the second portion of the face comprises an oblique view of the face, and the second facial feature comprises a portion of a full facial UV texture, the portion being smaller than the full facial UV texture.

27. The apparatus of claim 26, wherein the second machine learning model comprises a latent vector corresponding to the oblique view of the face.

28. An apparatus according to claim 25, wherein the first machine learning model and the second machine learning model are trained separately.

29. The apparatus of claim 16, wherein the at least one processor is configured to: obtaining a third frame, the third frame comprising a third portion of the face different from the second portion of the face, wherein the third portion of the face at least partially overlaps the first portion of the face; generating a third facial feature corresponding to the third portion of the face; obtaining a fourth frame, the fourth frame comprising a fourth portion of the face that is different from the second portion of the face and the third portion of the face, wherein the fourth portion of the face at least partially overlaps with the first portion of the face; generating a fourth facial feature corresponding to the fourth portion of the face, and The first facial feature is combined with the second facial feature, the third facial feature, and the fourth facial feature, wherein the combining is performed to enhance the appearance of at least one or more of the second portion of the face, the third portion of the face, or the fourth portion of the face.

30. The apparatus of claim 29, wherein the selected areas of the enhanced facial features correspond to a left eye, a right eye, and a mouth.

Images