View-related three-dimensional deformation model

Abstract

Systems and techniques for generating a three-dimensional (3D) face model are provided. For example, a process may include obtaining at least one input image associated with a face. In some aspects, the process may include obtaining a pose of a 3D face model associated with the face. In some examples, the process may include generating, by a machine learning model, the 3D face model associated with the face. In some cases, one or more parameters associated with a shape component of the 3D face model are conditioned by the pose. In some implementations, the 3D face model is configured to change in shape based on the pose of the 3D face model associated with the face.

Description

Technical Field

[0001] The present disclosure generally relates to systems and techniques for generating three-dimensional (3D) models of faces. Background Art

[0002] Many devices and systems allow for capturing a scene by generating frames (also referred to as images) and / or video data (including multiple images or frames) of the scene. For example, a camera or a computing device including a camera (e.g., a mobile device (such as a mobile phone or smart phone) including one or more cameras) can capture a sequence of frames of a scene. Frames and / or video data can be captured and processed by such devices and systems (e.g., mobile devices, IP cameras, etc.) and can be output for consumption (e.g., displayed on the device and / or other devices). In some cases, frames and / or video data can be captured by such devices and systems and output for processing and / or consumption by other devices.

[0003] Frames may be processed (e.g., using object detection, recognition, segmentation, etc.) to determine objects present in the frames, which may be useful for many applications. For example, a model may be determined for representing objects in the frames, and the model may be used to facilitate efficient operation of various systems. Examples of such applications and systems include computer graphics, extended reality (e.g., augmented reality, virtual reality, etc.), image recognition (e.g., facial recognition, object recognition, scene recognition, etc.), and feature estimation, among others. Summary of the invention

[0004] In some examples, systems and techniques for generating one or more models (e.g., one or more 3D facial models) are described herein. According to at least one example, a method for generating a 3D facial model is provided. The method includes: obtaining at least one input image associated with a face; obtaining a pose of a 3D facial model associated with the face; and generating, by a machine learning model, a 3D facial model associated with the face. One or more parameters associated with a shape component of the 3D facial model are conditioned on the pose.

[0005] In another example, an apparatus for generating a 3D facial model is provided, the apparatus comprising at least one memory and at least one processor (e.g., configured in a circuit) coupled to the at least one memory. The at least one processor is configured to: obtain at least one input image associated with a face; obtain a pose of a 3D facial model associated with the face; and generate the 3D facial model associated with the face using a machine learning model. One or more parameters associated with a shape component of the 3D facial model are conditioned on the pose.

[0006] In another example, a non-transitory computer-readable medium having stored thereon instructions is provided that, when executed by at least one processor, causes at least one process to: obtain at least one input image associated with a face; obtain a pose of a 3D facial model associated with the face; and generate a 3D facial model associated with the face using a machine learning model. One or more parameters associated with a shape component of the 3D facial model are conditioned on the pose.

[0007] In another example, an apparatus for generating a 3D facial model is provided. The apparatus includes: means for obtaining at least one input image associated with a face; means for obtaining a pose of the 3D facial model associated with the face; and means for generating the 3D facial model associated with the face using a machine learning model. One or more parameters associated with a shape component of the 3D facial model are conditioned on the pose.

[0008] In some aspects, one or more of the above devices are or are part of a vehicle (e.g., a computing device of a vehicle), a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, an apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors that can be used to determine the position and / or posture of the apparatus, the state of the apparatus, and / or for other purposes.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 illustrates an example three-dimensional (3D) facial model and a corresponding two-dimensional (2D) facial image projected from the 3D facial model overlaid with landmarks according to some examples;

[0013] Figure 2A and Figure 2B illustrates an example two-dimensional (2D) image projected from a 3D facial model generated by a non-view-dependent 3D model engine according to some examples;

[0014] Figure 2C illustrates a highly detailed 3D dimensional face model according to some examples;

[0015] Figure 3 is a diagram illustrating an example of a 3D modeling system according to some examples;

[0016] Figure 4 is a diagram illustrating an example view-dependent 3D model engine according to some examples;

[0017] Figure 5 Illustrate an example partial image that may be input to a view-dependent 3D modeling engine according to some examples

[0018] Fig. 6A and Figure 6B is an example of Figure 4 A graph of the cumulative distribution function (CDF) of 3D vertex errors at different locations where view information is injected into an example view-dependent 3D model engine;

[0019] Figure 7 and Figure 8 illustrates an example face model generated by a view-dependent 3D model engine according to some examples;

[0020] Fig. 9A and Fig. 9B illustrates example performance metrics comparing 3D face models generated by a view-dependent 3D model engine and a non-view-dependent 3D model engine according to some examples;

[0021] Fig.10 is a flow chart illustrating an example of a process for generating a 3D model according to some examples;

[0022] Fig.11 is a block diagram illustrating an example of a deep learning network according to some examples;

[0023] Fig.12 is a block diagram illustrating an example of a convolutional neural network according to some examples; and

[0024] Fig.13 is a diagram illustrating an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0025] Some aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes in order to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0026] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes can be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0027] The generation of three-dimensional (3D) models of physical objects can be used in many systems and applications, such as for extended reality (XR) (e.g., including augmented reality (AR), virtual reality (VR), mixed reality (MR), etc.), robotics, automotive, aviation, 3D scene understanding, object grasping, object tracking, and many other systems and applications. In an AR environment, for example, a user can view an image (also referred to as a frame) that includes an integration of artificial or virtual graphics with the user's natural environment. AR applications allow real images to be processed to add virtual objects to the image or display virtual objects on a perspective display (so that the virtual objects appear to be overlaid on the real world environment). AR applications can align or register virtual objects with real-world objects (e.g., as observed in the image) in multiple dimensions. For example, real-world objects that exist in reality can be represented using models that are similar to or accurately match real-world objects. In one example, a model of a virtual vehicle representing a real vehicle on a street can be presented through a display of an AR device (e.g., AR glasses, an AR head-mounted display (HMD), or other device) while the user continues to view his natural environment through the display. The viewer may be able to manipulate the model while viewing the real-world scene. In another example, a real object located on a table may be identified and rendered using a model with a different color or different physical properties in the AR environment. In some cases, artificial virtual objects that do not exist in reality or computer-generated copies of real objects or structures in the user's natural environment may also be added to the AR environment.

[0028] As more and more applications use facial data (e.g., for XR systems, for 3D graphics, for security, etc.), there is a large demand for systems that can generate detailed 3D facial models in an efficient and high-quality manner. There is also a large demand for generating 3D models of other types of objects, such as 3D models of vehicles (e.g., for autonomous driving systems), 3D models of room layouts (e.g., for XR applications, for navigation by devices, robots, etc.), etc. Generating detailed 3D models of objects (e.g., 3D facial models) typically requires expensive equipment and the use of multiple cameras in an environment with controlled lighting, which hinders large-scale data collection.

[0029] Performing 3D object reconstruction (e.g., to generate a 3D model of an object, such as a face model) from one or more images can be challenging. Using the face as an illustrative example of a 3D object, 3D face reconstruction can be difficult based on the need to reconstruct facial geometry (e.g., shape) and facial expressions. In some cases, when representing a 3D face model, the 3D face model can be viewed from different angles. For example, the 3D face model in an XR environment can be viewed from different angles by a user moving within the environment. In another example, the 3D face model can be moved (e.g., change pose) relative to a fixed viewing position. In some cases, it can be challenging to maintain an accurate projection of the 3D face model onto a 2D image within a range of viewing angles (also referred to herein as views).

[0030] like Figure 1 As illustrated, the white dots overlaid on the 2D facial image 102 may represent the projection of the 3D vertices of the 3D facial model 104 back onto the original 2D facial image 102 used to generate the 3D facial model 104. Figure 1 In the illustration of , points corresponding to 3D vertices of the main features of the 3D facial model (which may be referred to as landmarks or 2D landmarks) are depicted as white dots. As shown, landmarks 110, 112, 118, 120, 122, 124, 126, 128 for the outlines of lips, nose, mouth, eyes, eyebrows, nose, etc. are included. Although the 3D facial model 104 may include a greater number of vertices, for the purpose of illustration, only a small number of projected 3D vertices corresponding to the facial features listed above are shown. Figure 1 In the illustrated example, the landmarks corresponding to the inner contour 108 of the lower lip of the 3D facial model 104 projected onto the 2D image may include the landmark 112. Similarly, the landmarks corresponding to the outer contour 106 of the lower lip of the 3D facial model 104 may include the landmark 110.

[0031] Figure 1Also illustrated are an outer contour 114 and an inner contour 116 of the upper lip of the 3D facial model 104. In some examples, landmarks corresponding to the outer contour 114 of the upper lip may include landmarks 118 and 124, and landmarks corresponding to the inner contour 116 of the upper lip may include landmark 120. Additional landmarks projected from the 3D facial model 104 may include a landmark 122 corresponding to the left eye, a landmark 124 corresponding to the right eyebrow, a landmark 126 corresponding to the entire facial contour, and a landmark 128 corresponding to the nose. As described above, each of the landmarks 114, 116, 120, 122, 124, 126, and 128 may be generated by the projection of the 3D facial model 104 onto the 2D facial image 102A.

[0032] Figure 1 A two-dimensional (2D) facial image 102 and a corresponding 3D facial model 104 generated from the 2D facial image 102 using a 3D morphable model (3DMM) are illustrated. In some aspects, the 3D facial model 104 may include a representation of a facial expression in the 2D facial image 102. In one illustrative example, the facial expression representation may be formed by blend shapes. The blend shapes may semantically represent the movement of a muscle or part of a facial feature (e.g., opening / closing of a jaw, raising / lowering of eyebrows, opening / closing of eyes, etc.). In some cases, each blend shape may be represented by a blend shape coefficient paired with a corresponding blend shape vector.

[0033] In some examples, 3D facial model 104 may include a representation of the facial shape in 2D facial image 102. In some cases, the facial shape may be represented by a facial shape coefficient that is paired with a corresponding facial shape vector. In some implementations, a 3D model engine (e.g., a machine learning model) may be trained (e.g., during a training process) to implement a consistent facial shape (e.g., consistent facial shape coefficients) for the 3D facial model regardless of the pose (e.g., pitch, yaw, and roll) associated with the 3D facial model. For example, when rendering the 3D facial model into a 2D image for display, the 3D facial model may be projected onto the 2D image using projection techniques. Although the 3D model engine implements a consistent facial shape independent of pose, the projected 2D image may have varying degrees of accuracy based on the pose of the 3D facial model captured in the projected 2D image.

[0034] As an example, Figure 2A A projected 2D image 205 and a corresponding ground truth image 215 are illustrated. The ground truth image 215 may be an image used to generate a 3D facial model, which in turn is projected in a neutral pose (e.g., a camera position directly facing the projected model) to generate the projected 2D image 205. Figure 2AAs shown, the 2D image 205 projected in the neutral pose provides an accurate depiction of the face in the ground truth image 215 .

[0035] Figure 2B Illustrated in Figure 2A 205, 215 of the same individual depicted in FIG. 2. As illustrated in projected 2D image 225, a 3D facial model generated by a 3D model engine trained to enforce consistent shape regardless of pose may produce distorted projected 2D images (e.g., projected 2D image 225) for rotated poses, despite producing accurate 2D images (e.g., projected 2D image 205) for neutral poses. As can be seen by comparing 2D image 225 to ground truth image 235, although when the 3D model engine generates a consistent facial shape (e.g., facial shape coefficients) regardless of the pose of the 3D facial model, the consistent facial shape of the individual may not produce accurate results regardless of the pose of the 3D facial model.

[0036] Figure 2C A method for generating accurately projected 2D images of different poses of a 3D face model is illustrated. Figure 2C As shown, the 3D facial model 245 can be generated with a high level of accuracy, while realistically reflecting the shape of an individual's face. Figure 2C The highly detailed 3D facial model shown may be generated using expensive camera equipment that captures the individual's face from multiple angles. In some cases, the 3D facial model 245 may also be manually edited by a skilled artist to produce an accurate depiction of the individual. The process of generating such a highly detailed 3D facial model can only produce a single model for a particular individual and does not provide a flexible framework for generating a 3D model for any individual without preparing a detailed 3D facial model in advance.

[0037] Therefore, there is a need for systems and techniques for generating accurate 2D projected images based on a 3D facial model for a wide variety of poses of the 3D facial model, without requiring a highly detailed 3D facial model specifically tailored to the individual.

[0038] Described herein are systems, apparatuses, processes (or methods), and computer-readable media (collectively, "systems and techniques") for generating a 3D model of a particular portion of an object from one or more images (e.g., images having a red (R), green (G), blue (B) format, referred to as RGB images) comprising an image sequence or from frames of one or more videos (e.g., grayscale images or RGB videos). In some examples, as described in more detail below, these systems and techniques may generate a 3D model of an object (e.g., a 3D model of a person's face) based on one or more images depicting the object. These systems and techniques may adjust 3D facial model parameters (e.g., shape coefficients) based on views (e.g., poses) associated with one or more images to ensure that a wide variety of poses (e.g., pitch, yaw, and roll values) can be represented by the 3D facial model, and an accurate 2D image is generated from the 3D facial model. In some examples, these systems and techniques may utilize a trained machine learning model that is trained to generate a view-dependent 3D facial model conditioned on a desired pose or view (e.g., pitch, yaw, and roll) of the 3D facial model.

[0039] Various aspects of the technology described herein will be discussed below with respect to the various figures. Figure 3 is a diagram illustrating an example of a view-dependent 3D modeling system 300 that can generate a 3D model (eg, a 3D morphable model (3DMM)) using at least one image frame 302. Figure 3 As shown, the view-dependent 3D modeling system 300 includes an image frame engine 304, a 3D model engine 306, and a face reconstruction engine 308. Although the view-dependent 3D modeling system 300 is shown as including certain components, a person of ordinary skill will appreciate that the view-dependent 3D modeling system 300 may include more than one component. Figure 3 More or fewer components are shown. The components of the view-dependent 3D modeling system 300 may include software, hardware, or one or more combinations of software and hardware. For example, in some specific implementations, the components of the view-dependent 3D modeling system 300 may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or the components of the view-dependent 3D modeling system may include and / or be implemented using computer software, firmware, or a combination thereof for performing the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the view-dependent 3D modeling system 300.

[0040] The image frame engine 304 may obtain or receive an image frame 302 captured by an image sensor from a storage device, from a memory, from an external source (e.g., a server, an external memory accessed via a network, or other external source), etc. In some cases, the image frame may be included in a sequence of frames (e.g., a video, a sequence of independent or still images, etc.). In an illustrative example, each frame in the sequence of frames may include a grayscale component per pixel. Other examples of frames include frames with red (R), green (G), and blue (B) components per pixel (referred to as RGB video including RGB frames), frames with brightness, chroma-blue, chroma-red (YUV, YCbCr, or Y′CbCr) components per pixel, and / or any other suitable type of image. The sequence of frames may be captured by one or more cameras, obtained from a storage device, received from another device (e.g., a camera or a device including a camera), or obtained from another source. In some specific implementations, the image frame engine 304 may convert the image frame 302 to grayscale. In some cases, the image frame engine 304 may crop a portion of the image frame 302 corresponding to a face. In some examples, image frame engine 304 may perform a facial detection process and / or a facial recognition process to detect and / or identify a face within image frame 302. Image frame engine 304 may generate or apply a bounding box (e.g., Figure 1 The bounding box 130 shown in FIG. 1 ) may be defined, and the image data within the bounding box may be cropped to generate an input image for the 3D model engine 306 .

[0041] 3D model engine 306 may receive an input image (e.g., image frame 302, a cropped bounding box around a face in image frame 302, etc.) from image frame engine 304. 3D model engine 306 may also receive view 303 provided as input to view-dependent 3D modeling system 300. Using image frame 302 and view 303, 3D model engine 306 may perform a 3D model fitting technique to generate a 3D model (e.g., a 3DMM model) of a face (e.g., a face of a person in image frame 302). 3D model fitting techniques may include solving for shape coefficients a i and the expression coefficient b j , these coefficients can be used to generate a 3DMM representing the face in the input image.

[0042] In some examples, the 3D model fitting performed by the 3D model engine 306 may include a view 303, which may include position information related to the object. In an example where the object is a person's head, the view 303 may include posture information related to the posture of the head. For example, the posture information may indicate an angular rotation of the head relative to a neutral position of the head. The rotation may be along a first axis (e.g., a yaw axis), a second axis (e.g., a pitch axis), and / or a third axis (e.g., a roll axis). The 3D model generated by the 3D model engine 306 may have different shapes and / or feature coefficients based on the specific view 303 input into the view-dependent 3D modeling system.

[0043] In some cases, the 3D model fitting may also include a focal length for projecting the 3D model onto the 2D image using any suitable projection technique. In some examples, a weak perspective model may project the 3D vertices of a 3D model (e.g., 3DMM) onto the 2D image using the focal length generated by the 3D model engine 306. In some examples, a full perspective model may project the 3D vertices of a 3D model (e.g., 3DMM) onto the 2D image using the focal length generated by the 3D model engine 306.

[0044] In another illustrative example, the 3D model can be projected onto the image plane using a full perspective model. For example, the average depth Z in Formula 3 can be ave Replaced with posed 3D model S P The actual depth Z of each 3D vertex is used to provide a full perspective projection. Weak perspective model and full perspective model techniques and any suitable projection techniques for projecting a 3D model onto an image plane may be used without departing from the scope of the present disclosure.

[0045] The facial reconstruction engine 308 may receive the coefficients generated by the 3D model engine 306 (as well as any other output parameters, such as pose, focal length, etc.) to generate a 3D model (e.g., 3DMM). The 3D model may be generated or constructed as a linear combination of an average face (sometimes referred to as a neutral face), a facial shape basis vector, and a facial expression basis vector. The average face may represent an average face that may be transformed (e.g., by a shape basis vector and an expression basis vector) to achieve the desired final 3D facial shape of the 3D model. The facial shape basis vectors may be used to scale the proportions of the average face. In some cases, the facial shape basis vectors may be used to represent a fat face or a thin face, a small nose or a large nose, and any adjustments to the basic facial shape. In some implementations, the facial shape basis vectors are determined based on principal component analysis (PCA). In some cases, the facial expression basis vectors may represent facial expressions, such as smiling, raising eyebrows, winking, blinking, frowning, etc.

[0046] The 3D model S (e.g., a 3DMM generated using a 3DMM fitting technique) generated by the 3D model engine 306 using a 3D model fitting technique may be a statistical model representing the 3D geometric shape of an object (e.g., a face). For example, the 3D model (e.g., a 3DMM) may be a product of an average face S0 and a facial shape basis (also referred to as a basis vector) U i and facial expression base V j and the facial shape factor a i and facial expression coefficient b j It can be expressed as a linear combination of, for example, as follows:

[0047]

[0048] In the example of formula (1), there are M facial shape coefficients a i and M facial shape basis vectors U i , where M is an integer greater than or equal to 1. In some implementations, the average face S0, the face shape basis vector U i and facial expression vector V j Each of the face shape basis vectors U may include position information (eg, x, y, and z coordinates) of 3D vertices that can be combined to form the 3D model S. In some implementations, the face shape basis vectors U i and facial expression vector V j It can be expressed as a position shift relative to the average face S0, where the coefficient a of the face shape is i and the coefficient b of facial expression j Provides a scaling factor for the corresponding offset vector.

[0049] An illustrative example of a facial expression basis vector is a blend shape. As used herein, a blend shape may correspond to an approximate semantic parameterization of all or a portion of a facial expression. For example, a blend shape may correspond to a complete facial expression, or to a "partial" (e.g., "Δ") facial expression. Examples of partial expressions include raising an eyebrow, closing an eye, moving one side of the face, etc. In one example, a single blend shape may approximate the linearized effect of the movement of a single facial muscle. In some cases, a semantic representation may be modeled as corresponding to the movement of one or more facial muscles.

[0050] In one illustrative example, the 3D model S includes three thousand 3D vertices. In one illustrative example, M is equal to 319, which corresponds to 319 facial shape basis vectors U i and facial shape coefficient a i In some implementations, the shape basis vectors U i The principal component analysis feature vector may be included. In some cases, there are N facial expression coefficients b jand N facial expression basis vectors, where N is an integer greater than or equal to 1. In some cases, the facial expression vector V j May include blend shape vectors. In one illustrative example, N is equal to 39, which corresponds to 39 facial expression basis vectors V j and 39 facial expression coefficients b j (e.g., 39 blend shapes and 39 blend shape coefficients). In some cases, the result of the linear combination shown in equation (1) can be a 3D model (e.g., 3DMM) of the face in a neutral pose. In some examples, the 3D model can be rotated using pose information such as yaw, pitch, and roll values ​​to match the facial pose in image frame 302.

[0051] The facial reconstruction engine 308 may receive pose information from the 3D model engine 306 to generate a 3D model (e.g., 3DMM) and a projection of the 3D model into the 2D image space. The 3D model S (facial model) may be generated according to the above formula (1). As described above, the 3D model S produces a facial model with a neutral pose. In some cases, the 3D model S may be rotated to create a pose-modified 3D model S according to the following formula (2): P :

[0052] S P =R(α,β,γ)S (2)

[0053] where R(α,β,γ) is a 3×3 rotation matrix with rotation angles α,β,γ.

[0054] In some cases, the 3D model can be projected onto the image plane using any suitable projection technique. In one illustrative example, the 3D model can be projected onto the image plane using weak perspective projection. The following formula (3) provides how the posed 3D model S can be projected onto the image plane using weak perspective projection techniques. P The X and Y components of each 3D vertex coordinate The x and y coordinates of the projected 2D image Example on:

[0055]

[0056] Where f is the predicted focal length output by the 3D model engine 306, Z ave is the average depth (or distance) of the face from the weak perspective camera. P The projection of formula (3) is performed on each 3D vertex of .

[0057] In some cases, the 3D model engine 306 may include a machine learning model (e.g., a deep neural network such as a convolutional neural network (CNN), a general adversarial network, an inception network, etc.). The 3D model engine 306 may receive an image frame 302 and an input view 303 as input. In some cases, the image frame 302 and the view 303 may be one of a plurality of images and corresponding views in a training data set. The training data set may include a large number of images including a variety of facial shapes, expressions, and views (e.g., including one or more views of each face depicted in the image) in order to train the 3D model engine 306 to generate a 3D facial model (e.g., 3DMM) for many kinds of faces and views. In an illustrative example, the training data set may include one million images and views, two million images and views, or any other appropriately large number of images and views. In some cases, the 3D model engine 306 may be trained through multiple iterations (also referred to as epochs) using the same training data set. In some cases, the training data set used to train the 3D model engine 306 may include images, views, and labels indicating known features in the frame (e.g., in a supervised training process). In some cases, the training dataset may include annotated ground truth 2D landmark information corresponding to the locations of facial features for each image frame 302. In some cases, the 2D landmark information may be calculated from each image frame 302 using a landmark algorithm.

[0058] During training, the 3D model engine 306 may utilize one or more loss functions to analyze the errors in the 3D model parameters (e.g., the coefficients of formula (1)). The loss function L may be used to determine the amount of error in the 3D model generated by the 3D model engine 306 relative to the ground truth data provided as input to the 3D model engine 306. An example loss function includes the mean square error (MSE). The MSE is defined as It computes the sum of the actual answer (e.g., ground truth landmark location) minus one-half the square of the predicted (output) answer. Another example loss function includes a normalized mean error, where the error between the ground truth landmarks and the landmarks predicted by the 3D model engine 306 can be normalized by the predicted eye distance of the face.

[0059] Non-view-dependent (non-VD) 3D model engines (e.g., for generating Figure 2A and Figure 2B The loss function L of the 2D projection images 205, 225 in FIG. 1 (not shown here) non-VD An example of is shown in the following formula (4). As shown in formula (4), the loss function for training the 3D model engine 306 may include multiple components:

[0060] L non-VD =L Landmarks +LShapeConsistency +L other (4)

[0061] Where L Landmarks It can be used to represent the 2D and / or 3D landmark errors of 3D models generated by non-VD 3D model engines. ShapeConsistency Can be used to maintain the shape factor a i consistency, and other losses L other Any other loss function that may be applied during the training process may be included. In some cases, L other A clamping loss may be included that limits the facial expression coefficients b during the training process. j and / or for adaptively adjusting the upper and / or lower limits allowed by the facial expression coefficient. other A regularization loss may be included that can be used to prevent the facial shape coefficient a i and / or facial expression coefficient b j Gets too big during training.

[0062] Different from the non-VD 3D model loss function shown in formula (4), the VD 3D model loss function L of the 3D model engine 306 is VD The shape consistency loss L can be omitted ShapeConsistency , as shown in the following formula (5):

[0063] L VD =L Landmarks +L other (5)

[0064] As shown in formula (5), the landmark loss L Landmarks and other losses other It can be used by the VD 3D model loss function of the 3D model engine 306. Landmark loss L Landmarks 2D and / or 3D landmark loss functions may be included. An example 2D loss function is shown in the following formula (6):

[0065]

[0066] in is the difference between the projected 2D landmark position from the ground truth 3D model and the projected 2D landmark position of the corresponding vertex generated by the 3D model engine 306, and M (an integer) is the number of 2D landmarks.

[0067] An example 3D loss function is shown in the following formula (7):

[0068]

[0069] in is the difference between the 3D vertex position from the ground truth 3D model and the 3D vertex position of the corresponding vertex generated by the 3D model engine 306, and N (an integer) is the number of 3D vertices.

[0070] L Landmarks It can be a weighted sum of different 2D and / or 3D landmark loss components, as shown in the following formula (8):

[0071] L Landmarks =λ 3D L 3D +λ 2D L 2D (8)

[0072] where λ 3D and λ 2D are the weighting coefficients for 3D landmark loss and 2D landmark loss, respectively.

[0073] An example regularized loss function is shown in equation (9), and an example clamped loss function is shown in equation (10) below:

[0074]

[0075] where α i is the shape factor and b j is the facial expression coefficient

[0076]

[0077] in and are each facial expression coefficient b generated by the 3D model engine 306 j upper and lower limits.

[0078] During inference (e.g., after training is complete), the 3D model engine 306 may output a 3D face model (e.g., 3DMM) based on the input image frame 302 and the view 304. Since the shape coefficient α is omitted from the loss function shown in formula (5), i The constraints (e.g., the shape consistency loss L in formula (4) ShapeConsistency ), so the resulting individual output 3D facial models may have different shape factors depending on the view input (e.g., pose) of the generated 3D facial model. However, removing the constraint on shape consistency can result in a 3D facial model that can generate 2D projection images from different poses.

[0079] Figure 4 A block diagram of an example view-dependent 3D model engine 400 is illustrated. In some cases, the view-dependent 3D model engine may correspond to Figure 3The 3D model engine 306 . Figure 4 The view-dependent 3D model engine 400 includes a starting module 404, a fully connected layer 406, a summing block 408, a second-level starting module 410, a third-level starting module 412, and a fully connected layer 414. Figure 4 In the illustrated example, the view dependent 3D model engine 400 receives as input three input images 402 and a view 403 .

[0080] Figure 5 An illustrative example of partial facial images 502, 504, and 506 that may correspond to three input images 402 is provided. In some implementations, the partial facial images 502, 504, 506 may be obtained from a head-mounted camera. In some examples, the head-mounted camera may capture side views of the individual's left eye (e.g., facial image 502) and right eye (e.g., facial image 504) and oblique views of the mouth, nose, and jaw. In some cases, the head-mounted camera may be included in a head-mounted device. In an illustrative example, the head-mounted device may include an XR system. In some cases, it may be desirable to create a 3D facial model of the individual captured in the partial facial images 502, 504, 506. In some cases, the 3D facial model of the individual may be used to generate a 2D image of the individual that provides an accurate representation of the user when the pose of the individual's head changes.

[0081] Return to Figure 4 , each of the input images 402 may be input to a corresponding initiation module 404. The initiation module 404 may be used to generate features based on the input image 402. In some implementations, the initiation module 404 may include multiple convolutional neural networks with different kernel sizes (e.g., 1×1, 3×3, and 5×5 pixel kernels), which may be applied to the input images in parallel and combined into a single feature vector output from the initiation module 404 for each corresponding input image 402. In some aspects, the initiation module 404 may detect features of different scales using different convolutional kernels running in parallel, and combine the convolutional network outputs in a feature vector. In an illustrative example, the outputs from the separate convolutional networks within the initiation module 404 may be concatenated into a feature vector and passed to the next stage of the view-dependent 3D model engine 400. The output of each initiation module that receives the input image 402 as input is referred to herein as an input image feature vector.

[0082] like Figure 4 As shown, the view 403 may be input into the fully connected layer 406. In some implementations, the fully connected layer 406 may output a feature vector (referred to herein as a view feature vector) based on the view 403. Figure 4In the illustrated example configuration, at summation block 408, the view feature vector may be individually combined with each feature vector output from initiation module 404. In one illustrative example, summation block 408 may perform an element-by-element addition of the view feature vector and each individual input image feature vector from initiation module 404. The output of each summation block 408 may be passed to a second level initiation module 410. The output of each second level initiation module 410 may be passed to a third level initiation module 412. Thus, the input to the third level initiation module may represent features generated based on all three input images 402 and view 403. In the illustrated example, a fully connected layer 414 receives the output of the third level initiation module 412 and outputs 3D facial model coefficients 416 (e.g., 3DMM coefficients of formula (1)).

[0083] During training, Figure 4 The view-dependent 3D model engine can be used with training Figure 3 The training process is similar to the process described for the 3D model engine 306 of the view-dependent 3D model engine. The loss function may include a landmark loss and other losses (e.g., a regularization loss and / or a clamping loss), as described above with respect to equation (5). In an illustrative example, the landmark loss component for training the view-dependent 3D model engine may include a landmark loss L Landmarks,hmc , as shown in the following formula (11)

[0084] L Landmarks,hmc =λ 3D L 3D +λ 2D L 2D +λ 2D,hmc L 2D,hmc (11)

[0085] The first two terms of formula (11) are f2D and L 3D The loss functions of can correspond to the above formulas (5) and (6) respectively. The additional loss term L 2D,hmc may represent the error between the input image 402 and the projection of the corresponding vertices of the 3D model generated by the view-dependent 3D model engine 400 onto the landmarks of the input image. Therefore, the loss function used to train the view-dependent 3D model engine 400 may utilize three different landmark sources to achieve high landmark accuracy despite allowing for changes in the shape coefficients of the 3D face model coefficients 416.

[0086] The following formula (12) provides an example additional loss function L 2D,hmc :

[0087] L 2D,hmc = 0.33L 2D,LEyE +0.33L 2D,REye +0.33L2D,Mouth (12)

[0088] Where L 2D,LEye ,L 2D,REye ,L 2D,Mouth is the one corresponding to the left eye (e.g. Figure 5 ), right eye (e.g., Figure 5 504) and mouth (e.g., Figure 5 506) corresponding to the input image 402. In some cases, the 2D landmark loss function L 2D,LEye ,L 2D,REye ,L 2D,Mouth It may have the form of formula (6). Therefore, the loss function may represent the error between the projected 2D landmarks of the corresponding 3D vertices from the face model and the 2D landmarks of the ground truth input image 402.

[0089] Return to Figure 3 In some implementations, the face reconstruction engine 308 may generate a UV face position map (also referred to as a UV position map) for an input image frame 302 including a face. In some cases, the UV face position map may provide and / or represent a 2D map of a face in the input image. For example, the UV face position map may be a 2D image that records and / or plots the 3D positions of points (e.g., pixels) in a UV space (e.g., a 2D texture coordinate system). U in UV space and V in UV space may represent axes of the UV face position map (e.g., axes of a 2D texture of the face). In an illustrative example, U in UV space may represent a first axis (e.g., a horizontal X-axis) of the UV face position map, and V in UV space may represent a second axis (e.g., a vertical Y-axis) of the UV face position map. In some examples, the UV position map may record, model, identify, represent, and / or calculate the 3D shape, structure, contour, depth, and / or other details of a face (and / or a facial region of a head). In some cases, the UV face position map may be applied as a texture to the 3DMM generated by the 3D model engine 306. In some examples, a 3DMM with a texture provided from a UV face position map can be used to render a 3D digital representation of an input image. In some implementations, a machine learning model (e.g., a neural network) can be used to generate a UV face position map.

[0090] Figure 3The illustrated 3D modeling system 300 illustrates a functional block diagram that can be implemented in hardware, software, or any combination of hardware and software. As an example, the illustrated functional blocks can use the illustrated relationships to identify functional blocks that can be converted into a Universal Modeling Language (UML) diagram to identify the example implementation of the view-dependent 3D modeling system 300 at least in part as an object-oriented arrangement in software. However, the 3D modeling system 300 can be implemented without abstractions and, for example, as a static functional implementation. In some examples, the view-dependent 3D modeling system 300 can be implemented by a user having Fig.13 The architecture of computing system 1300 is shown implemented in whole or in part by a computing device or system.

[0091] Although specific examples of loss functions (e.g., 2D landmark loss, 3D landmark loss, clamping loss, and regularization loss) are described herein in conjunction with the 3D model engine 306, it should be understood that any suitable loss function may be used to analyze errors in the output of the 3D model engine 306 without departing from the scope of the present disclosure.

[0092] refer to Figure 4 , the view information is included in the layer of the 3D modeling system at the summation block 408 between the initiation module 404 and the second-level initiation module 410. As used herein, the point within the layer of the 3D modeling system where the view information is included is referred to as the injection location. Figure 4 One example injection location of the view feature vector is illustrated in , but other injection locations may be used without departing from the scope of the present disclosure.

[0093] Fig. 6A and Figure 6B Graphs 602, 604, 606 of experimental cumulative distribution function (CDF) of average 3D vertex errors and graphs 612, 614, 616 of maximum 3D vertex errors of 3D face models generated using a view-dependent 3D model engine according to the present disclosure are illustrated. Fig. 6A and Figure 6B In each of the graphs in , the horizontal axis represents the 3D vertex error between the generated 3D face model and the reference truth face model, and the vertical axis indicates the percentage of images having an error less than the horizontal axis value. Figure 4 The injection locations shown in the example view-dependent 3D model engine 400 correspond to the graphs 602 , 612 .

[0094] Another example injection location for view information may be to inject the view information into Figure 4 Each input image 402 is combined as Figure 44. Experimental 3D vertex position errors for a 3D modeling engine similar to the 3D modeling engine 400 but modified with injection positions prior to starting the model are indicated by graphs 604, 614.

[0095] Another example injection location of the view information may be between the third level initiation module 412 and the fully connected layer 414, such as Figure 4 The experimental 3D vertex position errors of a 3D model engine similar to the 3D model engine 400 but modified with injection positions before the fully connected layer 414 are indicated by graphs 606 , 616 .

[0096] Figure 7 and Figure 8 The invention illustrates the use of a view-dependent 3D model engine (e.g., Figure 4 An example 3D facial model generated by a view-dependent 3D model engine 400) based on images of an individual captured with different views (e.g., poses) of the individual's head.

[0097] like Figure 7 7, a ground truth 2D image 702 of an individual depicts a pose facing the upper left corner of 2D image 702. The resulting 3D facial model (e.g., 3DMM) generated by the view-dependent 3D model engine is shown in image 704 in the same pose as ground truth 2D image 702. The same 3D facial model from image 704 is shown in image 706 with a neutral pose (e.g., facing directly toward the camera).

[0098] like Figure 8 As shown, ground truth 2D image 812 of the individual depicts a pose facing the lower right corner of 2D image 812. The resulting 3D facial model (e.g., 3DMM) generated by the view-dependent 3D model engine is shown in image 814 in the same pose as ground truth 2D image 702. The same 3D facial model from image 814 is shown in image 816 with a neutral pose (e.g., facing directly toward the camera).

[0099] Comparing the neutral pose images 706, 816 of the same individual illustrates the results of the relaxation of the shape consistency constraints from the non-view-dependent 3D model engine, as shown in equation (4). Although the underlying 3D meshes of the two 3D face models generated for each pose have different shapes, the projected 2D images produced by the differently shaped 3D face models at different poses can show improved accuracy (e.g., better than the original 3D face models). Figure 2B compared to the 2D image 225 shown).

[0100] Fig. 9A and Fig. 9B A view-dependent 3D modeling system (eg, Figure 4 Comparison of example CDFs of performance metrics of a 3D model engine 400) and a non-view-dependent 3D modeling system (e.g., trained using the loss function in formula (4)). Fig. 9A The graphs 902, 904, 906 of correspond to a 3D model generated based on a set of test images of a first object (e.g., images of a first person captured in different views / poses). Similarly, Fig. 9B The graphs 912, 914, 916 correspond to a 3D model generated based on a set of test images of a second object (e.g., images of a second person captured at different views / poses). In each of the graphs 902, 904, 906, 912, 914, 916, the dashed line is a 3D model generated using a view-dependent 3D model engine (e.g., Figure 4 4. In each of the graphs 902, 904, 906, 912, 914, 916, the solid line is a graph of a 3D model generated using a non-view-dependent 3D modeling system. Fig. 9A and Fig. 9B The graphs include peak signal-to-noise ratio (PSNR) in graphs 902, 912, structural similarity (SSIM) in graphs 904, 914, and perceptual error (PERR) in graphs 906, 916. In some cases, the PERR can be calculated by determining the MSE based on a 2D image of a 3D model projection generated from a test image set. In an illustrative example, the 2D image of a 3D model projection generated from a test image set can be processed by a pre-trained visual geometry group neutral network (e.g., a VGG16 network) including 16 convolutional layers to generate the MSE.

[0101] In the graphs 902, 912, the horizontal axis represents the PSNR score, and the vertical axis represents the percentage of images in the corresponding test image set that produced a PSNR score greater than the PSNR score value on the horizontal axis. In the illustrated graphs 902, 912, lower PSNR scores indicate better performance of the underlying 3D modeling system in generating 3D models in the test configuration.

[0102] In the graphs 904, 914, the horizontal axis represents the SSIM score, and the vertical axis represents the percentage of images in the corresponding test image set that produce an SSIM score greater than the horizontal axis SSIM score value. In the illustrated graphs 904, 912, a lower SSIM score indicates better performance of the underlying 3D modeling system in generating 3D models in the test configuration.

[0103] In the graphs 906, 916, the horizontal axis represents the 3D vertex perception error (PERR), and the vertical axis represents the percentage of images in the corresponding test image set that produce a PERR less than the horizontal axis PERR value. In the illustrated graphs 904, 912, a lower PERR indicates better performance of the underlying 3D modeling system in generating 3D models in the test configuration.

[0104] In each of the graphs 902, 904, 906, 912, 914, 916, the performance of the view-dependent 3D modeling system is better than that of the non-view-dependent 3D modeling system. Fig. 9A and Fig. 9B Provided is a method for using a view-dependent 3D modeling system (e.g., Figure 3 3D modeling system 300, Figure 4 An illustrative example of the benefits of generating a 3D model using a view-dependent 3D model engine 400 .

[0105] Fig.10 A flow chart of a process 1000 for training one or more machine learning models is illustrated. At block 1002, the process 1000 includes obtaining at least one input image associated with a face (e.g., Figure 3 Frame 302, Figure 4 In some examples, at least one input image associated with a face includes two or more partial images of the face. In some cases, at least one input image associated with a face includes an image of a left eye of the face (e.g., Figure 5 ), an image of the right eye of the face (e.g., Figure 5 ) and an image of the mouth of the face (e.g., Figure 5 partial facial image 506).

[0106] At block 1004, process 1000 includes obtaining a pose of a 3D facial model associated with a face (e.g., Figure 3 View 303, Figure 4 In some cases, the pose of the 3D facial model associated with the face includes pitch, yaw, and roll.

[0107] At block 1006, process 1000 includes generating a prediction result by a machine learning model (e.g., Figure 3 3D model engine 306, Figure 4In some cases, one or more parameters associated with a shape component of the 3D facial model are conditioned on a desired pose. In some cases, the 3D facial model is configured to vary in shape based on a pose of the 3D facial model associated with a face. In some examples, one or more parameters associated with the shape component of the 3D facial model include coefficients of the shape component of the 3D facial model (e.g., shape coefficients a i ). In some cases, the one or more parameters associated with the shape component of the 3D model include the facial shape basis vectors of the 3D facial model (the facial shape basis vectors U of formula (1) i ). In some implementations, the 3D face model further includes an average face component and a facial expression component. In some examples, the 3D face model includes a combination of an average face component, a shape component, and a facial expression component.

[0108] In some cases, generating a 3D facial model includes: generating one or more facial image feature vectors based on at least one input image associated with the face; generating a pose feature vector based on a pose of the 3D facial model; and generating a plurality of 3D facial model coefficients associated with the 3D facial model based on the one or more facial image feature vectors and the pose feature vector. In some cases, generating one or more facial image feature vectors includes: generating a first facial image feature vector based on an image of a left eye of the face; generating a second facial image feature vector based on an image of a right eye of the face; and generating a third facial image feature vector based on an image of a mouth of the face. In some cases, process 1000 also includes: combining the first facial image feature vector with the pose feature vector; combining the second facial image feature vector with the pose feature vector; and combining the third facial image feature vector with the pose feature vector. In some implementations, combining the first facial image feature vector, the second facial image feature vector, and the third facial image feature vector with the pose feature vector includes: performing an element-wise sum of the first facial image feature vector and the pose feature vector; performing an element-wise sum of the second facial image feature vector and the pose feature vector; and performing an element-wise sum of the third facial image feature vector and the pose feature vector (see, e.g., Figure 4 The summation block 408).

[0109] In some examples, process 1000 also includes obtaining an additional pose of an additional 3D facial model associated with the face, wherein the additional pose of the additional 3D facial model associated with the face is different from the pose of the 3D facial model associated with the face. In some examples, process 1000 includes generating an additional 3D facial model associated with the face by a machine learning model. In some cases, at least one of the one or more additional parameters associated with the shape component of the additional 3D facial model is different from at least one of the one or more parameters associated with the shape component of the 3D facial model. In some examples, the additional 3D facial model is generated based on at least one input image associated with the face. In some aspects, the 3D facial model associated with the face and the additional 3D facial model associated with the face include a common facial expression. In some cases, process 1000 also includes obtaining at least one additional input image associated with the face. In some specific implementations, the additional 3D facial model is generated based on at least one additional input image associated with the face. In some cases, at least one input image associated with the face is associated with a first facial expression, and at least one additional input image associated with the face is associated with a second facial expression. In some examples, the first facial expression and the second facial expression are different.

[0110] In some examples, a device may include an application or functionality for performing some of the processes described herein (e.g., process 1000 and / or any other process described herein). In some examples, the processes described herein (e.g., process 1000 and / or any other process described herein) may be performed by a computing device or apparatus. In some examples, process 1000 may be performed by the above-described Figure 3 The 3D modeling system 300 and / or Figure 4 In another example, the process 1000 may be performed by a view-dependent 3D model engine 400 having Fig.13 The architecture of computing system 1300 is shown as being implemented by a computing device or system.

[0111] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, an extended reality (XR) device or system (e.g., a VR headset, an AR headset, AR glasses, or other XR devices or systems), a wearable device (e.g., a connected watch or smartwatch, or other wearable device), a server computer or system, a vehicle or a computing device of a vehicle (e.g., an autonomous vehicle), a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein (including processes 1000, 1000). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0112] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0113] Process 1000 is illustrated as a logic flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0114] In addition, process 1000 and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or implemented by a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program that includes multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0115] Fig.11 is an illustrative example of a deep learning neural network 1100 that can be used by a 3D model training system. An input layer 1120 includes input data. In an illustrative example, the input layer 1120 may include data representing pixels of an input video frame. The neural network 1100 includes a plurality of hidden layers 1122a, 1122b to 1122n. The hidden layers 1122a, 1122b to 1122n include "n" hidden layers, where "n" is an integer greater than or equal to one. The plurality of hidden layers may include as many layers as required for a given application. The neural network 1100 also includes an output layer 1124 that provides an output resulting from the processing performed by the hidden layers 1122a, 1122b to 1122n. In an illustrative example, the output layer 1124 may provide a classification of an object in an input video frame. The classification may include a category that identifies a type of object (e.g., a person, dog, cat, or other object).

[0116] Neural network 1100 is a multi-layer neural network composed of interconnected nodes. Each node can represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains the information as it processes it. In some cases, neural network 1100 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back to itself. In some cases, neural network 1100 may include a recursive neural network, which may have loops that allow information to be carried across nodes when reading in input.

[0117] Information can be exchanged between nodes by node-to-node interconnection between each layer. The nodes of input layer 1120 can activate a group of nodes in the first hidden layer 1122a. For example, as shown in the figure, each input node in the input node of input layer 1120 is connected to each node in the node of the first hidden layer 1122a. The nodes of hidden layers 1122a, 1122b to 1122n can transform the information by applying activation function to the information of each input node. The information derived from the transformation can then be transferred to the node of the next hidden layer 1122b and can activate the node of the next hidden layer, and the node of the next hidden layer can perform their own specified functions. Example functions include convolution, upsampling, data conversion and / or any other suitable function. Then, the output of hidden layer 1122b can activate the node of the next hidden layer, and so on. The output of last hidden layer 1122n can activate one or more nodes of output layer 1124, and output is provided at the one or more nodes. In some cases, although a node in neural network 1100 (e.g., node 1126) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0118] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of the neural network 1100. Once the neural network 1100 is trained, it may be referred to as a trained neural network, which may be used to classify one or more objects. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have a tunable digital weight that may be tuned (e.g., based on a training data set), thereby allowing the neural network 1100 to adapt to the input and be able to learn as more and more data is processed.

[0119] The neural network 1100 is pre-trained to process features from the data in the input layer 1120 using different hidden layers 1122a, 1122b to 1122n to provide an output through the output layer 1124. In an example where the neural network 1100 is used to identify objects in an image, the neural network 1100 may be trained using training data that includes both images and labels. For example, training images may be input into the network, where each training image has a label indicating the class of one or more objects in each image (essentially, indicating to the network what the objects are and what features they have). In one illustrative example, the training images may include an image of the number 2, in which case the label for the image may be [00 1 0 0 0 0 0 00].

[0120] In some cases, the neural network 1100 may use a training process known as back propagation to adjust the weights of the nodes. Back propagation may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. For each set of training images, this process may be repeated a certain number of iterations until the neural network 1100 is trained well enough so that the weights of each layer are accurately tuned.

[0121] For the example of identifying an object in an image, a forward pass may include passing a training image through the neural network 1100. Prior to training the neural network 1100, the weights are initially randomized. The image may include, for example, an array of numbers representing pixels of the image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 digital array having 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or a brightness and two chrominance components, etc.).

[0122] For the first training iteration of the neural network 1100, the output will likely include values ​​that do not prioritize any particular class due to the weights randomly selected at initialization. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each of the different classes may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 1100 cannot determine low-level features and therefore cannot accurately determine what the classification of the object may be. A loss function may be used to analyze the errors in the output. Any suitable loss function definition may be used. An example of a loss function includes the mean squared error (MSE). The MSE is defined as It calculates the sum of the actual answer minus one-half the square of the predicted (output) answer. The loss can be set equal to E total The value of .

[0123] For the first training images, the loss (or error) will be high because the actual values ​​will be very different from the predicted outputs. The goal of training is to minimize the amount of loss so that the predicted outputs are the same as the training labels. The neural network 1100 can perform a backward pass by determining which inputs (weights) contribute most to the network's loss, and the weights can be adjusted so that the loss decreases and is ultimately minimized.

[0124] The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes most to the loss of the network. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as Where w represents the weight, w i Represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes a larger weight update, while a lower value indicates a smaller weight update.

[0125] Neural network 1100 may include any suitable deep network. One example includes a convolutional neural network (CNN) that includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. Fig.11 An example of a CNN is described. The hidden layer of the CNN includes a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. The neural network 1100 may include any other deep network other than a CNN, such as an autoencoder, a deep belief network (DBN), a recursive neural network (RNN), etc.

[0126] Fig.12 is an illustrative example of a convolutional neural network 1200 (CNN 1200). An input layer 1220 of CNN 1200 includes data representing an image. For example, the data may include an array of numbers representing pixels of an image, where each number in the array includes a value from 0 to 255 describing the intensity of the pixel at the location of the number in the array. Using the previous example from above, the array may include a 28×28×3 digital array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luminance and two chrominance components, etc.). The image may be passed through a convolutional hidden layer 1222a, an optional non-linear activation layer, a pooling hidden layer 1222b, and a fully connected hidden layer 1222c to obtain an output at an output layer 1224. Although Fig.12 Only one hidden layer among the hidden layers is shown in , but one of ordinary skill will appreciate that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 1200. As previously described, the output may indicate a single category of an object, or may include a probability of a category that best describes an object in the image.

[0127] The first layer of CNN 1200 is a convolutional hidden layer 1222a. Convolutional hidden layer 1222a analyzes the image data of input layer 1220. Each node of convolutional hidden layer 1222a is connected to an area of ​​nodes (pixels) of the input image called a receptive field. Convolutional hidden layer 1222a can be considered as one or more filters (each filter corresponds to a different activation or feature map), where each convolution iteration of the filter is a node or neuron of convolutional hidden layer 1222a. For example, the area of ​​the input image covered by the filter at each convolution iteration will be the receptive field of the filter. In an illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in convolutional hidden layer 1222a. Each connection between a node and the receptive field of the node learns a weight, and in some cases learns an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node of hidden layer 1222a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has an array of weights (numbers) and the same depth as the input. For the video frame example, the filter will have a depth of 3 (according to the three color components of the input image). An illustrative example size of the filter array is 5×5×3, corresponding to the size of the receptive field of the node.

[0128] The convolutional nature of the convolutional hidden layer 1222a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1222a may start at the upper left corner of the input image array and may be convolved around the input image. As noted above, each convolution iteration of the filter may be considered as a node or neuron of the convolutional hidden layer 1222a. In each convolution iteration, the value of the filter is multiplied by the original number of pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the upper left corner of the input image array). The multiplications from each convolution iteration may be added together to obtain the sum for that iteration or node. Next, the process is continued at the next position in the input image according to the receptive field of the next node in the convolutional hidden layer 1222a.

[0129] For example, the filter may move a stride amount to the next receptive field. The stride amount may be set to 1 or another suitable amount. For example, if the stride amount is set to 1, the filter will move 1 pixel to the right at each convolution iteration. Processing the filter at each unique position in the input volume produces a number representing the filter result at that position, thereby determining a sum value for each node of the convolution hidden layer 1222a.

[0130] The mapping from the input layer to the convolutional hidden layer 1222a is called an activation map (or feature map). The activation map includes a value for each node that represents the filter result at each location of the input volume. The activation map may include an array that includes various summed values ​​produced by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will include a 24×24 array. The convolutional hidden layer 1222a may include several activation maps in order to identify multiple features in the image. Fig.12 The example shown includes three activation maps. Using these three activation maps, the convolutional hidden layer 1222a can detect three different types of features, each of which is detectable across the entire image.

[0131] In some examples, a nonlinear hidden layer may be applied after the convolutional hidden layer 1222a. A nonlinear layer may be used to introduce nonlinearity into a system that already computes linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer may apply the function f(x)=max(0,x) to all values ​​in the input volume, which changes all negative activations to 0. Thus, the ReLU may increase the nonlinear properties of the CNN 1200 without affecting the receptive field of the convolutional hidden layer 1222a.

[0132] The pooling hidden layer 1222b may be applied after the convolutional hidden layer 1222a (and after the nonlinear hidden layer when used). The pooling hidden layer 1222b is used to simplify the information in the output from the convolutional hidden layer 1222a. For example, the pooling hidden layer 1222b may take each activation map output from the convolutional hidden layer 1222a and generate a condensed activation map (or feature map) using a pooling function. Max pooling is an example of a function performed by a pooling hidden layer. The pooling hidden layer 1222a uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. A pooling function (e.g., a maximum pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1222a. In Fig.12 In the example shown in , three pooling filters are used for the three activation maps in the convolutional hidden layer 1222a.

[0133] In some examples, max pooling may be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as a stride of 2) to the activation map output from the convolutional hidden layer 1222a. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes in the previous layer (each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by a 2×2 max pooling filter at each iteration of the filter, where the maximum of the four values ​​is output as the "maximum" value. If such a max pooling filter is applied to an activation filter with a dimension of 24×24 nodes from the convolutional hidden layer 1222a, the output from the pooling hidden layer 1222b will be an array of 12×12 nodes.

[0134] In some examples, an L2-norm pooling filter may also be used. The L2-norm pooling filter involves calculating the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than calculating the maximum value as done in max pooling), and using the calculated value as output.

[0135] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found anywhere in a region of an image, and discards the exact location information. This can be done without affecting the results of feature detection, because once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (and other pooling methods) provides the following benefits: far fewer features are pooled, thereby reducing the number of parameters required in subsequent layers of CNN 1200.

[0136] The last layer of connections in the network is a fully connected layer that connects each node from the pooling hidden layer 1222b to each output node in the output layer 1224. Using the above example, the input layer includes 28×28 nodes that encode the pixel intensities of the input image, the convolutional hidden layer 1222a includes 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for the filter) to the three activation maps, and the pooling layer 1222b includes a layer of 3×12×12 hidden feature nodes based on applying a max pooling filter to a 2×2 region on each of the three feature maps. Extending this example, the output layer 1224 may include ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 1222b is connected to each node of the output layer 1224.

[0137] The fully connected layer 1222c can obtain the output of the previous pooling layer 1222b (which should represent an activation map of high-level features) and determine the features most relevant to a particular category. For example, the fully connected layer 1222c layer can determine the high-level features that are most strongly associated with a particular category and can include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 1222c and the pooling hidden layer 1222b can be calculated to obtain probabilities for different categories. For example, if the CNN 1200 is used to predict that the object in the video frame is a person, there will be high values ​​in the activation map representing the high-level features of the person (e.g., there are two legs, there is a face at the top of the object, there are two eyes at the upper left and upper right of the face, there is a nose in the middle of the face, there is a mouth at the bottom of the face, and / or other features common to people).

[0138] In some examples, the output from the output layer 1224 may include an M-dimensional vector (in the previous example, M=10), where M may include the number of categories that the program must choose from when classifying an object in an image. Other example outputs may also be provided. Each number in the N-dimensional vector may represent the probability that an object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector represents objects of ten different categories is [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates that the image is a 5% probability of an object of the third category (e.g., a dog), the image is an 80% probability of an object of the fourth category (e.g., a person), and the image is a 15% probability of an object of the sixth category (e.g., a kangaroo). The probability for a category can be considered as the confidence level that the object is part of that category.

[0139] Fig.13 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.13 An example of a computing system 1300 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using connection 1305. Connection 1305 may be a physical connection using a bus, or a direct connection into processor 1310, such as in a chipset architecture. Connection 1305 may also be a virtual connection, a networked connection, or a logical connection.

[0140] In some embodiments, computing system 1300 is a distributed system, where the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some embodiments, a component may be a physical device or a virtual device.

[0141] The example system 1300 includes at least one processing unit (CPU or processor) 1310 and connections 1305 that couple various system components including system memory 1315, such as read only memory (ROM) 1320 and random access memory (RAM) 1325, to the processor 1310. The computing system 1300 may include a cache 1312 of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 1310.

[0142] Processor 1310 may include any general purpose processor and hardware or software services, such as services 1332, 1334, and 1336 stored in storage device 1330, that are configured to control processor 1310 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1310 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0143] To enable user interaction, the computing system 1300 includes an input device 1345 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1300 may also include an output device 1335 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1300. The computing system 1300 may include a communication interface 1340, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transfer, Low Energy (BLE) wireless signal transfer, Wireless signal transfer, radio frequency identification (RFID) wireless signal transfer, near field communication (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 802.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transfer, public switched telephone network (PSTN) signal transfer, integrated services digital network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communication interface 1340 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1300 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0144] The storage device 1330 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0145] Storage device 1330 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1310, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1310, connection 1305, output device 1335, etc.

[0146] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0147] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0148] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those skilled in the art that embodiments may be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in a combination of software or hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid obscuring these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without necessary details to avoid confusing each embodiment.

[0149] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0150] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0151] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptops, mobile phones (e.g., smart phones or other types of mobile phones), tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0152] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0153] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used in any number of environments and applications beyond the environments and applications described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0154] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0155] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0156] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0157] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0158] Various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. The technician can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as making it depart from the scope of the application.

[0159] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that execute one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0160] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0161] Illustrative aspects of the present disclosure include:

[0162] Aspect 1: A method for generating a three-dimensional (3D) facial model, the method comprising: obtaining at least one input image associated with a face; obtaining a pose of a 3D facial model associated with the face; and generating the 3D facial model associated with the face by a machine learning model, wherein one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose.

[0163] Aspect 2: The method according to aspect 1, wherein the 3D facial model is configured to vary in shape based on the pose of the 3D facial model associated with the face.

[0164] Aspect 3: The method according to any one of Aspects 1 to 2, wherein the posture of the 3D facial model includes pitch, yaw and roll.

[0165] Aspect 4: The method according to any one of aspects 1 to 3, wherein the one or more parameters associated with the shape component of the 3D facial model include coefficients of the shape component of the 3D facial model.

[0166] Aspect 5: The method according to any one of Aspects 1 to 4, wherein the 3D facial model further comprises an average facial component and a facial expression component, and wherein the 3D facial model comprises a combination of the average facial component, the shape component and the facial expression component.

[0167] Aspect 6: According to the method described in any one of Aspects 1 to 5, the method further includes: obtaining an additional posture of an additional 3D facial model associated with the face, wherein the additional posture of the additional 3D facial model associated with the face is different from the posture of the 3D facial model associated with the face; and generating the additional 3D facial model associated with the face by the machine learning model, wherein at least one parameter of the one or more additional parameters associated with the shape component of the additional 3D facial model is different from at least one parameter of the one or more parameters associated with the shape component of the 3D facial model.

[0168] Aspect 7: The method according to any one of aspects 1 to 6, wherein the additional 3D facial model is generated based on the at least one input image associated with the face.

[0169] Aspect 8: The method according to any one of aspects 1 to 7, wherein the 3D facial model associated with the face and the additional 3D facial model associated with the face include a common facial expression.

[0170] Aspect 9: According to the method described in any one of Aspects 1 to 8, the method further includes obtaining at least one additional input image associated with the face, wherein the additional 3D facial model is generated based on the at least one additional input image associated with the face, wherein: the at least one input image associated with the face is associated with a first facial expression; the at least one additional input image associated with the face is associated with a second facial expression; and the first facial expression and the second facial expression are different.

[0171] Aspect 10: A method according to any one of Aspects 1 to 9, wherein generating the 3D facial model comprises: generating one or more facial image feature vectors based on the at least one input image associated with the face; generating a posture feature vector based on the posture of the 3D facial model; and generating a plurality of 3D facial model coefficients associated with the 3D facial model based on the one or more facial image feature vectors and the posture feature vector.

[0172] Aspect 11: The method according to any one of aspects 1 to 10, wherein the at least one input image associated with the face comprises two or more partial images of the face.

[0173] Aspect 12: A method according to any one of Aspects 1 to 11, wherein the at least one input image associated with the face comprises an image of a left eye of the face, an image of a right eye of the face, and an image of a mouth of the face.

[0174] Aspect 13: A method according to any one of Aspects 1 to 12, wherein generating the one or more facial image feature vectors comprises: generating a first facial image feature vector based on the image of the left eye of the face; generating a second facial image feature vector based on the image of the right eye of the face; and generating a third facial image feature vector based on the image of the mouth of the face.

[0175] Aspect 14: According to the method described in any one of Aspects 1 to 13, the method further includes: combining the first facial image feature vector with the posture feature vector; combining the second facial image feature vector with the posture feature vector; and combining the third facial image feature vector with the posture feature vector.

[0176] Aspect 15: A method according to any one of Aspects 1 to 14, wherein combining the first facial image feature vector, the second facial image feature vector and the third facial image feature vector with the posture feature vector comprises: performing an element-by-element summation of the first facial image feature vector and the posture feature vector; performing an element-by-element summation of the second facial image feature vector and the posture feature vector; and performing an element-by-element summation of the third facial image feature vector and the posture feature vector.

[0177] Aspect 16: An apparatus for generating a 3D facial model, the apparatus comprising: a memory and at least one processor coupled to the memory (e.g., implemented in a circuit). The memory may be configured to store data to the processor 1310, such as a system memory 1315, a read-only memory (ROM) 1320, and / or a random access memory (RAM) 1325. The at least one processor is configured to: obtain at least one input image associated with a face; obtain a pose of a 3D facial model associated with the face; and generate the 3D facial model associated with the face by a machine learning model, wherein one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose.

[0178] Aspect 17: The apparatus according to Aspect 16, wherein the 3D facial model is configured to vary in shape based on the pose of the 3D facial model associated with the face.

[0179] Aspect 18: The apparatus according to any one of Aspects 16 to 17, wherein the pose of the 3D facial model includes pitch, yaw and roll.

[0180] Aspect 19: The apparatus according to any one of aspects 16 to 18, wherein the one or more parameters associated with the shape component of the 3D facial model include coefficients of the shape component of the 3D facial model.

[0181] Aspect 20: An apparatus according to any one of Aspects 16 to 19, wherein the 3D facial model further comprises an average facial component and a facial expression component, and wherein the 3D facial model comprises a combination of the average facial component, the shape component and the facial expression component.

[0182] Aspect 21: An apparatus according to any one of Aspects 16 to 20, wherein the at least one product is configured to: obtain an additional posture of an additional 3D facial model associated with the face, wherein the additional posture of the additional 3D facial model associated with the face is different from the posture of the 3D facial model associated with the face; and generate the additional 3D facial model associated with the face by the machine learning model, wherein at least one parameter of the one or more additional parameters associated with a shape component of the additional 3D facial model is different from at least one parameter of the one or more parameters associated with the shape component of the 3D facial model.

[0183] Aspect 22: An apparatus according to any one of aspects 16 to 21, wherein the additional 3D facial model is generated based on the at least one input image associated with the face.

[0184] Aspect 23: The apparatus according to any one of Aspects 16 to 22, wherein the 3D facial model associated with the face and the additional 3D facial model associated with the face include a common facial expression.

[0185] Aspect 24: An apparatus according to any one of Aspects 16 to 23, wherein the at least one processor is configured to: obtain at least one additional input image associated with the face, wherein the additional 3D facial model is generated based on the at least one additional input image associated with the face, wherein: the at least one input image associated with the face is associated with a first facial expression; the at least one additional input image associated with the face is associated with a second facial expression; and the first facial expression and the second facial expression are different.

[0186] Aspect 25: An apparatus according to any one of Aspects 16 to 24, wherein the at least one processor is configured to: generate one or more facial image feature vectors based on the at least one input image associated with the face; generate a posture feature vector based on the posture of the 3D facial model; and generate a plurality of 3D facial model coefficients associated with the 3D facial model based on the one or more facial image feature vectors and the posture feature vector.

[0187] Aspect 26: An apparatus according to any one of Aspects 16 to 25, wherein the at least one input image associated with the face comprises two or more partial images of the face.

[0188] Aspect 27: An apparatus according to any one of Aspects 16 to 26, wherein the at least one input image associated with the face comprises an image of a left eye of the face, an image of a right eye of the face, and an image of a mouth of the face.

[0189] Aspect 28: An apparatus according to any one of Aspects 16 to 27, wherein the at least one processor is configured to: generate a first facial image feature vector based on the image of the left eye of the face; generate a second facial image feature vector based on the image of the right eye of the face; and generate a third facial image feature vector based on the image of the mouth of the face.

[0190] Aspect 29: An apparatus according to any one of Aspects 16 to 28, wherein the at least one processor is configured to: combine the first facial image feature vector with the posture feature vector; combine the second facial image feature vector with the posture feature vector; and combine the third facial image feature vector with the posture feature vector.

[0191] Aspect 30: An apparatus according to any one of Aspects 16 to 29, wherein combining the first facial image feature vector, the second facial image feature vector, and the third facial image feature vector with the posture feature vector comprises: performing an element-by-element summation of the first facial image feature vector and the posture feature vector; performing an element-by-element summation of the second facial image feature vector and the posture feature vector; and performing an element-by-element summation of the third facial image feature vector and the posture feature vector.

[0192] Aspect 31: A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform any of the operations described in aspects 1 to 30.

[0193] Aspect 32: An apparatus comprising one or more means for performing any of the operations described in aspects 1 to 30.

Claims

1. A method for generating a three-dimensional (3D) facial model, the method comprising: obtaining at least one input image associated with a face; obtaining a pose of a 3D facial model associated with the face; as well as The 3D facial model associated with the face is generated by a machine learning model, wherein one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose. 2 . The method of claim 1 , wherein the 3D facial model is configured to vary in shape based on the pose of the 3D facial model associated with the face. 3 . The method of claim 1 , wherein the pose of the 3D facial model comprises pitch, yaw, and roll. 4 . The method of claim 1 , wherein the one or more parameters associated with the shape component of the 3D facial model comprise coefficients of the shape component of the 3D facial model.

5. The method of claim 4, wherein the 3D facial model further comprises an average facial component and a facial expression component, and wherein the 3D facial model comprises a combination of the average facial component, the shape component, and the facial expression component.

6. The method according to claim 1, further comprising: obtaining an additional pose of an additional 3D facial model associated with the face, wherein the additional pose of the additional 3D facial model associated with the face is different from the pose of the 3D facial model associated with the face; and The additional 3D facial model associated with the face is generated by the machine learning model, wherein at least one of the one or more additional parameters associated with a shape component of the additional 3D facial model is different from at least one of the one or more parameters associated with the shape component of the 3D facial model.

7. The method of claim 6, wherein the additional 3D facial model is generated based on the at least one input image associated with the face.

8. The method of claim 7, wherein the 3D facial model associated with the face and the additional 3D facial model associated with the face include common facial expressions.

9. The method of claim 6, further comprising obtaining at least one additional input image associated with the face, wherein the additional 3D facial model is generated based on the at least one additional input image associated with the face, wherein: The at least one input image associated with the face is associated with a first facial expression; The at least one additional input image associated with the face is associated with a second facial expression; as well as The first facial expression and the second facial expression are different.

10. The method of claim 1, wherein generating the 3D facial model comprises: generating one or more facial image feature vectors based on the at least one input image associated with the face; generating a posture feature vector based on the posture of the 3D facial model; and A plurality of 3D facial model coefficients associated with the 3D facial model are generated based on the one or more facial image feature vectors and the pose feature vector.

11. The method of claim 10, wherein the at least one input image associated with the face comprises two or more partial images of the face.

12. The method of claim 11, wherein the at least one input image associated with the face comprises an image of a left eye of the face, an image of a right eye of the face, and an image of a mouth of the face.

13. The method of claim 12, wherein generating the one or more facial image feature vectors comprises: generating a first facial image feature vector based on the image of the left eye of the face; generating a second facial image feature vector based on the image of the right eye of the face; and A third facial image feature vector is generated based on the image of the mouth of the face.

14. The method according to claim 13, further comprising: combining the first facial image feature vector with the posture feature vector; combining the second facial image feature vector with the posture feature vector; as well as The third facial image feature vector is combined with the posture feature vector.

15. The method of claim 14, wherein combining the first facial image feature vector, the second facial image feature vector, and the third facial image feature vector with the posture feature vector comprises: performing an element-by-element summation of the first facial image feature vector and the posture feature vector; performing an element-by-element summation of the second facial image feature vector and the posture feature vector; and An element-wise summation of the third facial image feature vector and the pose feature vector is performed.

16. An apparatus for generating a 3D facial model, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtaining at least one input image associated with a face; obtaining a pose of a 3D facial model associated with the face; as well as The 3D facial model associated with the face is generated by a machine learning model, wherein one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose.

17. The apparatus of claim 16, wherein the 3D facial model is configured to vary in shape based on the pose of the 3D facial model associated with the face.

18. The apparatus of claim 16, wherein the pose of the 3D facial model comprises pitch, yaw, and roll.

19. The apparatus of claim 16, wherein the one or more parameters associated with the shape component of the 3D facial model comprise coefficients of the shape component of the 3D facial model.

20. The apparatus of claim 19, wherein the 3D facial model further comprises an average facial component and a facial expression component, and wherein the 3D facial model comprises a combination of the average facial component, the shape component, and the facial expression component.

21. The apparatus of claim 16, wherein the at least one processor is configured to: obtaining an additional pose of an additional 3D facial model associated with the face, wherein the additional pose of the additional 3D facial model associated with the face is different from the pose of the 3D facial model associated with the face; and The additional 3D facial model associated with the face is generated by the machine learning model, wherein at least one of the one or more additional parameters associated with a shape component of the additional 3D facial model is different from at least one of the one or more parameters associated with the shape component of the 3D facial model.

22. The apparatus of claim 21, wherein the additional 3D facial model is generated based on the at least one input image associated with the face.

23. The apparatus of claim 22, wherein the 3D facial model associated with the face and the additional 3D facial model associated with the face include a common facial expression.

24. The apparatus of claim 21, wherein the at least one processor is configured to: obtaining at least one additional input image associated with the face, wherein the additional 3D facial model is generated based on the at least one additional input image associated with the face, wherein: The at least one input image associated with the face is associated with a first facial expression; The at least one additional input image associated with the face is associated with a second facial expression; as well as The first facial expression and the second facial expression are different.

25. The apparatus of claim 16, wherein the at least one processor is configured to: generating one or more facial image feature vectors based on the at least one input image associated with the face; generating a posture feature vector based on the posture of the 3D facial model; and A plurality of 3D facial model coefficients associated with the 3D facial model are generated based on the one or more facial image feature vectors and the pose feature vector.

26. The apparatus of claim 25, wherein the at least one input image associated with the face comprises two or more partial images of the face.

27. The apparatus of claim 26, wherein the at least one input image associated with the face comprises an image of a left eye of the face, an image of a right eye of the face, and an image of a mouth of the face.

28. The apparatus of claim 27, wherein the at least one processor is configured to: generating a first facial image feature vector based on the image of the left eye of the face; generating a second facial image feature vector based on the image of the right eye of the face; and A third facial image feature vector is generated based on the image of the mouth of the face.

29. The apparatus of claim 28, wherein at least one processor is configured to: combining the first facial image feature vector with the posture feature vector; combining the second facial image feature vector with the posture feature vector; and The third facial image feature vector is combined with the posture feature vector.

30. The apparatus of claim 29, wherein combining the first facial image feature vector, the second facial image feature vector, and the third facial image feature vector with the pose feature vector comprises: performing an element-by-element summation of the first facial image feature vector and the posture feature vector; performing an element-by-element summation of the second facial image feature vector and the posture feature vector; and An element-wise summation of the third facial image feature vector and the pose feature vector is performed.

Images