3D facial capture and modification using image and time tracking neural networks

Abstract

The invention relates to 3D face capture and modification using image and time tracking neural networks. Techniques related to capturing a 3D face using an image and time tracking neural network and modifying an output video using the captured 3D face are discussed. Such techniques include applying a first neural network to an input vector corresponding to a first video image having a face representation to generate a deformation model parameter vector, applying a second neural network to an input vector corresponding to the first video image and a second video image temporally following the first video image to generate a deformation model parameter delta vector, generating a 3D face model of the human face using the deformation model parameter vector and the deformation model parameter delta vector, and generating an output video using the 3D face model.

Description

[0001] Divisional Application Instructions

[0002] This application is a divisional application of the invention patent application with application date of June 14, 2018, application number 201880091642.3, and name “3D facial capture and modification using image and time tracking neural networks”. Technical Field

[0003] The present disclosure relates to 3D facial capture and modification using image and time tracking neural networks. Background Art

[0004] Capturing three-dimensional (3D) faces is an important aspect of visual effects production in film and game animation, as well as other applications. Current techniques include the use of multiple high-resolution cameras, carefully arranged shooting environments, and labeling subjects for video capture. This technique is expensive, time-consuming, and laborious. Other techniques include fitting a 3D deformable model (3DMM) by extracting facial features from an input video image and performing a 3DMM fit to minimize the difference between the input image and an image (e.g., an RGB-D image) rendered using the 3DMM and a 3D camera. However, the former technique is generally unable to capture 3D faces in video, while the latter requires implanting a 3D camera, which can be expensive and complex.

[0005] Therefore, current techniques for estimating facial shape and texture from standard input videos are insufficient in certain scenarios. In view of these and other considerations, improvements are now needed. As the implementation of 3D facial models in various scenarios becomes more common, such improvements may become critical. Summary of the invention

[0006] According to one aspect of the present disclosure, a method is provided, comprising: estimating one or more facial parameters from a first image using a trained model, the one or more facial parameters indicating one or more features of a face captured by the first image; generating a three-dimensional model representing the face from the one or more facial parameters; generating a rendered three-dimensional model by rendering the three-dimensional model using one or more additional facial parameters; and generating a second image from the rendered three-dimensional model.

[0007] According to one aspect of the present disclosure, a computer-readable medium is provided, comprising instructions, which, when executed by at least one processor, enable the processor to perform the method as described above.

[0008] According to one aspect of the present disclosure, a system is provided, comprising: a computer processor for executing computer program instructions; and a computer readable memory for storing computer program instructions, wherein the computer program instructions can be used by the computer processor to execute the method described above.

[0009] According to one aspect of the present disclosure, a device is provided, including a device for executing the method as described above.

[0010] According to one aspect of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by at least one processor, enable the processor to perform the method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The materials described herein are illustrated by way of example and not limitation in the accompanying drawings. For simplicity and clarity of illustration, the elements shown in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. In addition, where considered appropriate, reference numerals are repeated between the drawings to indicate corresponding or similar elements. In the drawings:

[0012] Figure 1 An example system for capturing a 3D face and applying effects based on the captured 3D face is shown;

[0013] Figure 2 Processing an example input image to generate an example input vector is shown;

[0014] Figure 3 Processing of an example morphable model parameter vector to generate an example 3D mesh or facial model is shown;

[0015] Figure 4 shows application of example 3D object elements using a 3D facial model;

[0016] Figure 5 shows application of example texture elements using a 3D facial model;

[0017] Figure 6 An example system for pre-training image CNNs is shown;

[0018] Figure 7 An example system for pre-training a tracking CNN is shown;

[0019] Figure 8 is a flow chart illustrating an example process for training a CNN and implementing it for 3D facial capture and rendering;

[0020] Fig. 9is a flow chart illustrating an example process for processing a video;

[0021] Fig.10 is an illustrative diagram of an example system for processing video;

[0022] Fig.11 is an illustrative diagram of an example system; and

[0023] Fig.12 Example small form factor devices are shown, all arranged in accordance with at least some implementations of the present disclosure. DETAILED DESCRIPTION

[0024] One or more embodiments or implementations are now described with reference to the accompanying drawings. Although specific configurations and arrangements are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the relevant art will recognize that other configurations and arrangements may be used without departing from the spirit and scope of the specification. It will be apparent to those skilled in the relevant art that the techniques and / or arrangements described herein may also be used in various other systems and applications other than those described herein.

[0025] Although the following description sets forth various implementations that may be embodied in architectures such as, for example, a system-on-chip (SoC) architecture, implementations of the techniques and / or arrangements described herein are not limited to a particular architecture and / or computing system, and may be implemented by any architecture and / or computing system for similar purposes. For example, the techniques and / or arrangements described herein may be implemented using various architectures such as multiple integrated circuit (IC) chips and / or packages, and / or various computing devices and / or consumer electronics (CE) devices (e.g., multifunction devices, tablet computers, smart phones, etc.). In addition, although the following description may set forth many specific details (e.g., logical implementations, types, and interrelationships of system components, logical partitioning / integration selections, etc.), the claimed subject matter may be practiced without these specific details. In other cases, certain materials (e.g., control structures and complete software instruction sequences) may not be shown in detail to avoid obscuring the material disclosed herein.

[0026] The materials disclosed herein may be implemented in hardware, firmware, software, or any combination thereof. The materials disclosed herein may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any medium and / or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include a read-only memory (ROM); a random access memory (RAM); a disk storage medium; an optical storage medium; a flash memory device; an electrical, optical, acoustic, or other form of propagation signal (e.g., a carrier wave, an infrared signal, a digital signal, etc.), and the like.

[0027] References in the specification to "one implementation," "implementation," "example implementation," or examples, or embodiments, etc., indicate that the implementation being described may include a particular feature, structure, or characteristic, but that each embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same implementation. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in conjunction with other implementations (whether or not explicitly described herein).

[0028] Methods, devices, apparatus, computing platforms, and articles of manufacture related to capturing parameters for a 3D facial model from a video and modifying the video based on the rendered 3D facial model are described herein.

[0029] As described above, it may be desirable to capture or estimate 3D facial shape and texture by determining parameters of a 3D facial model at high speed and accuracy. The resulting 3D morphable model or parameters for such a morphable model may be used in many applications, such as applications including: visual effects applications, mapping to digital avatars, animation, addition of 2D stickers or 3D dynamic props, virtual reality, photo-realistic 3D facial rendering, assistance in surgery through modeling, etc. The techniques discussed herein provide capabilities for real-time, inexpensive, and easy-to-implement 3D facial capture and video modification (e.g., facial animation), etc.

[0030] As discussed herein, embodiments include processing video frames (which may also be characterized as pictures or images) having representations of human faces. Notably, the video frames are in a standard format, including RGB data, YUV data, etc. without depth information. The input video frames may be preprocessed by cropping (e.g., based on a detected face) and downscaling to, for example, 256×256 pixels. For a first input video frame, an input vector based on a cropped and downscaled region is generated. In an embodiment, the input vector includes RGB data corresponding to the cropped and downscaled region, such that the input vector is a 256×256×3 vector including R, G, and B values ​​for each pixel position. The input vector is provided to a pretrained convolutional neural network (CNN) including a node as an output layer for providing an output vector (i.e., a deformation model parameter vector) including parameters for a 3D facial model. As used herein, the term CNN is used to indicate any neural network for analyzing visual images. For example, a CNN typically includes a feed-forward artificial neural network. The 3D facial model may include any suitable parameters for being implemented by the 3D facial model to generate a representation of a human face, which is characterized herein as a facial mesh. In an embodiment, the morphable model parameter vector includes values ​​for providing geometry, texture (e.g., skin albedo), illumination, and pose to a facial mesh or facial model of a human face.

[0031] In addition, for a temporally subsequent video frame, an input vector (e.g., a temporal tracking input vector) based on the cropped and scaled-down region of the two temporally adjacent frames is generated. In an embodiment, the temporal tracking input vector includes RGB data corresponding to the cropped and scaled-down region of the two temporally adjacent frames, such that the temporal tracking input vector is a 256×256×3×2 vector including R, G, and B values ​​for each pixel position of the two cropped and scaled-down regions. The temporal tracking input vector differs from the previously discussed input vector in that it includes data for the two cropped and scaled-down regions. The temporal tracking input vector is provided to a second pre-trained CNN, which includes a node as an output layer for providing an output vector (i.e., a morphable model parameter delta vector) including delta parameters for a 3D facial model. The morphable model parameter delta vector provides a delta or difference value to each parameter of the previously discussed morphable model parameter vector for the temporally first frame of the two temporally adjacent frames. The morphable model parameter vector for the first frame in time (determined by the first CNN in the case of the first frame in the sequence) is then modified by the morphable model parameter difference vector (e.g., they are added or differed) to generate a morphable model parameter vector for the second frame in time.

[0032] Therefore, a morphable model parameter vector is generated for each frame or certain frames in the video sequence using a spatial or image pre-trained CNN for the first frame in time and a tracking or temporal pre-trained CNN (which is separate from the spatial or image pre-trained CNN) for subsequent frames in time. The 3D facial model discussed previously may be implemented or may be modeled by the morphable model parameter vector to generate a 3D facial mesh or model representing a human face. The input video frame may then be modified based on the 3D facial mesh or model to generate an output video frame, where the output video frame is stored and / or presented to a user. The modification may include any appropriate modification, such as applying a visual effect using information from the 3D facial mesh or model (e.g., applying an effect at the tip of the nose as indicated by the 3D facial mesh or model), by modifying the 3D facial mesh or model itself (e.g., applying a mask texture to an area of ​​the 3D facial mesh or model), etc. In some embodiments, the output video frame is generated using the 3D facial mesh or model without using the input video frame. For example, the output video frames may be generated by rendering an avatar using a 3D facial mesh or model, rendering a photo-realistic image using a 3D facial mesh or model, and so on.

[0033] The technology discussed herein provides: automatically capturing and tracking 3D representations of human faces from, for example, consumer RGB cameras, and providing 2D / 3D special effects in real time based on the 3D representations. Such technology can be implemented in a real-time environment to support real-time 3D facial modeling, tracking, and enhancement (modification) based on images from an RGB camera (as discussed). The technology discussed provides high accuracy, temporal stability, and tracking to improve device performance. In addition, the technology discussed is computationally efficient and provides improved device efficiency and power savings during implementation. The technology discussed herein provides a 3D deformable facial model (e.g., 3D Morphable Model, 3DMM) fitting method that automatically (e.g., without human intervention) determines facial model parameters from input video frames.

[0034] Figure 1 An example system 100 for capturing a 3D face and applying effects based on the captured 3D face is shown, arranged in accordance with at least some implementations of the present disclosure. Figure 1As shown, the system 100 includes an image preprocessing module 121, an image CNN 122, a 3D face rendering module 124, an effect and image output module 126, a (time) tracking CNN 123, a 3D face rendering module 125, and an effect and image output module 127. Also as shown, the image CNN 122, the 3D face rendering module 124, and the effect and image output module 126 can be implemented as a first stage 141 (or a spatial stage) and the tracking CNN 123, the 3D face rendering module 125, and the effect and image output module 127 can be implemented as a second stage 142 (or a temporal tracking stage). The image CNN 122 can be characterized as a spatial CNN, a picture CNN, etc. The tracking CNN 123 can be characterized as a temporal CNN, a temporal tracking CNN, etc. As shown, the image CNN 122 and the tracking CNN 123 are separate CNNs, which have different and separate input layers, internal layers, and output layers. The system 100 can be implemented via any suitable device, such as a personal computer, a laptop computer, a tablet computer, a tablet phone, a smart phone, a digital camera, a game console, a wearable device, a display device, an all-in-one device, a two-in-one device, etc. For example, the system 100 can provide at least a portion of an image signal processing pipeline, which can be implemented in hardware, software, or a combination thereof. As discussed, the system 100 can implement a 3D deformable face model 131. As shown, the system 100 receives an input video 101, which includes any number of input images including representations of human faces.

[0035] The input video 101 may include any appropriate video frames, video pictures, video images, sequences of such video frames, pictures or images, groups of pictures, video data, etc., at any appropriate resolution. For example, the video may be a video graphics array (VGA), high definition (HD), full HD (e.g., 1080p), 4K resolution video, 5K resolution video, etc., and the video may include any number of video frames, sequences of video frames, pictures, groups of pictures, etc. For example, the input video 101 may be a monocular video (e.g., with one field of view) from a single RGB camera. The techniques discussed herein may be discussed interchangeably with respect to video frames, images, and pictures. For example, a frame, picture, or image of color video data may include three chrominance planes (pixel values ​​for chrominance) or channels (e.g., R, G, and B planes), or a luminance plane (e.g., pixel values ​​for luminance) or channels and two chrominance planes (e.g., pixel values ​​for chrominance) or channels. The input video 101 includes any number of frames or images 102, 103, 104, so that the images 102, 103, 104 are in time order. That is, image 102 is before image 103 in time (with any number of intermediate images), and image 103 is before image 104 in time. In an embodiment, image 103 is immediately before image 104 in time (i.e., there is no intermediate image). In an embodiment, the input video 101 is captured in real time via a camera (not shown). In an embodiment, the input video 101 is received from a memory (not shown) or a remote device (not shown). The input video 101 can be characterized as a plurality of sequential video images, pictures, or frames.

[0036] As shown, the image preprocessing module 121 receives input images 102, 103, 104. The input images 102, 103, 104 may include any number of representations of one or more human faces. In an embodiment, the input images 102, 103, 104 include a single representation of a single human face that is changing expression, posture, etc. over time. Image preprocessing as performed by the image preprocessing module 121 may include any appropriate image preprocessing to generate input vectors 112, 113, 114 corresponding to the input images 102, 103, 104. For example, the image preprocessing module 121 may perform face detection, facial landmark detection, etc. to detect representations of human faces in one or more of the input images 102, 103, 104. Based on the detected faces, the image preprocessing module 121 may crop each of the input images 102, 103, 104 into a region including the face and scale the cropped region down to a predefined standard size. In an embodiment, the image preprocessing module 121 detects a face, crops the input image 102, 103, 104 into a predetermined region shape (e.g., a square), scales the region down to a predetermined pixel density (e.g., scaled down to a resolution of 256×256 pixels), and generates each of the input vectors 112, 113, 114 as a concatenation of chroma pixel values ​​for each pixel position in the scaled down region. In an embodiment, each of the input vectors 112, 113, 114 includes 256×256×3 values ​​including a value for each of three (e.g., RGB) chroma values ​​for each scaled down pixel value.

[0037] This cropping and scaling down technique can be performed using any appropriate one or more techniques (e.g., downsampling techniques). In addition, although the discussion is about cropping and scaling down the RGB image to 256×256 pixels, any appropriate image format and region size and shape can be used. In an embodiment, the region is a rectangular region with a height greater than the width. In an embodiment, scaling down is not applied. In any case, each of the input vectors 112, 113, 114 is formatted into (one or more) formats for pre-training image CNN 122 and tracking CNN 123, as will be discussed further below.

[0038] Figure 2 1 shows processing an example input image 102 to generate an example input vector 112, arranged in accordance with at least some implementations of the present disclosure. Figure 2 As shown, at least a portion 201 of the input image 102 (e.g., Figure 2The entire input image 102 or a portion 201 of the input image may be shown to have a representation of a human face 202. As discussed, the representation of the human face 202 may be detected and the input image 102 may be cropped using a bounding box to define a sub-region 203 of the input image 102. The input image 102 is cropped using the sub-region 203 (e.g., keeping pixel values ​​within the sub-region 203 and discarding those pixel values ​​outside the sub-region 203) and the area within the sub-region 203 is scaled down to a predetermined pixel resolution (e.g., 256×256) (as shown by the cropping and downscaling operation 211) to provide an input vector 112. As discussed, the input vector 112 may include a 256×256×3 element vector whose values ​​include each R, G, and B value for each pixel of the cropped and downscaled area within the sub-region 203.

[0039] Back to Figure 1 , the input vector 112 is provided to the image CNN 122. It is noteworthy that the input vector 112 may correspond to the input image 102, which is the first image of the input video 101, the first image of the sequence of the input video 101, the first image after the scene change of the input video 101, etc. Such an image is denoted as I0. The image CNN 122 receives the input vector 112, and the image CNN 122 processes the input vector 112 to generate a deformation model parameter vector 116 (i.e., θ0) corresponding thereto. Here, the deformation model parameter vector is labeled with θ and a subscript, where the subscript indicates the input image corresponding thereto. The image CNN 122 may include any CNN having any number of layers (e.g., 6-8 layers) (including an input layer and an output layer). The input layer has the same number of nodes as the number of elements of the input vector 112 and the output layer has the same number of elements as the deformation model parameter vector 116, which are required for modeling a 3D face model, as discussed further below. The layers of the image CNN 122 may include any suitable processing layers known in the art, such as layers followed by rectified linear units (ReLU), max pooling, and / or local response normalization (LRN). The image CNN 122 is constructed using any suitable technique or techniques (e.g., as described below with respect to Figure 6 The techniques discussed above are used to pre-train the dataset.

[0040] The deformable model parameter vector 116 is provided to the 3D face rendering module 124, which implements the 3D face model 131 using the deformable model parameter vector 116 to generate a resulting face mesh or 3D face. The resulting face mesh includes any appropriate data structure representing a human face, such as per-vertex shape, per-vertex texture (e.g., skin albedo), and illumination parameters. It is worth noting that the resulting face mesh or 3D face can be projected onto a 2D image to generate an output image.

[0041] In an embodiment, the 3D face model 131 is modeled using the morphable model parameter vector 116 to generate a reconstructed 3D face with the correct pose and 3D shape, as described below. The 3D face model 131 (e.g., 3D morphable model, 3DMM) parameterizes the novel face, as shown in the following equations (1), (2), and (3):

[0042]

[0044] Wherein, equation (1) defines the geometry of the reconstructed 3D face, equation (2) defines the skin albedo (e.g., texture or reflectivity) of the reconstructed 3D face, and equation (3) defines the appearance illumination of the reconstructed 3D face. In the above equations, Encode the per-vertex shape (S) and albedo (T) of a predetermined average face, where N is the number of vertices (i.e., the number of vertices). Then use α id ) for the shape identity matrix A id Modifications and (using α exp ) for the shape identity matrix A exp The shape of the predetermined average face is modified by modifying alb ) for the albedo identity matrix A alb The albedo of the predetermined average face is modified by the modification of are the shape unit basis matrix and the albedo basis matrix, respectively, where K = 99, for example, and is a delta expression matrix, where for example K e = 80. The modifications to this base matrix discussed above are characterized herein as coefficients for modifying this matrix. The base matrix is ​​modified using matrix multiplication using shape coefficients, expression coefficients, and albedo coefficients and added to the predetermined average face shape and albedo, respectively, to provide a unique face shape and albedo. Furthermore, in equation (3), T iis the skin albedo of the ith vertex, and The normal is determined as n i The b-th spherical harmonic (SH) basis function of , where B is set to 3.

[0045] Furthermore, the shape, albedo, and illumination in question (e.g., providing a 3D face model) are projected onto a 2D image plane using the perspective camera model based on rotation, translation, and focus parameters. Thus, in order to fully convert the 3D face model 131 into a 2D image plane rendering of the detected face, the morphable model parameter vector 116 includes the following parameters: (respectively, coefficients of the shape unit basis matrix for modifying the 3D facial model 131, coefficients of the expression unit basis matrix for modifying the 3D facial model 131, and coefficients of the texture unit basis matrix for modifying the 3D facial model 131), (illumination parameter), R, T (pose transformation parameters for rotation and translation, respectively), and f (camera parameter for pose transformation), so that any deformation model parameter vector discussed in this article can include 3 rotation parameters, 3 translation parameters, 1 camera parameter, 99 coefficients for modifying the shape unit matrix of the 3D facial model 131, 99 coefficients for modifying the expression unit matrix of the 3D facial model 131, 80 coefficients for modifying the texture unit matrix of the 3D facial model 131, and 27 illumination parameters, a total of 312 parameters. That is, the deformation model parameter vector θ can be represented as follows: θ = {R, T, f, α id ,α alb ,α exp ,γ}.

[0046] Figure 3 1 shows processing of an example morphable model parameter vector 116 to generate an example 3D mesh or facial model, arranged in accordance with at least some implementations of the present disclosure. Figure 3 As shown, the deformation model parameter vector 116 is rendered via the rendering operation 311 discussed above to generate a shape corresponding to the human face 202 (see Figure 2 ) corresponds to a 3D facial model 301. Figure 3In the illustrated example of , for clarity of presentation, the 3D facial model 301 is shown in two perspectives—the upper one facing forward and the lower one rotated. The rendering operation 311 may be applied using any suitable one or more techniques discussed herein, such as application of one or more of rotation parameters (for a total of 312 parameters), translation parameters, camera parameters, coefficients for modifying a shape identity matrix of the 3D facial model 131, coefficients for modifying an expression identity matrix of the 3D facial model 131, coefficients for modifying a texture identity matrix of the 3D facial model 131, and illumination parameters. It is worth noting that not all parameters or elements of the morphable model parameter vector 116 need be applied to render the 3D facial model 301. For example, a texture may be applied to all or part of the 3D facial model 301 as a special effect.

[0047] Back to Figure 1 , then the 3D facial rendering module 124 receives the 3D facial model 131 and implements at least a portion of the morphable model parameter vector 116 via the model (e.g., generates a representative facial model using the morphable model parameter vector 116) to generate a facial mesh or a 3D facial model. Optionally, one or more special effects may be applied based on the facial mesh or the 3D facial model via the effects and image output module 126. The special effects may be applied using any appropriate one or more techniques. In an embodiment, the application of the special effects includes: adding an image element (e.g., an animated sprite, a cropped image, etc.) based on a specific position of the 3D facial model (e.g., at the vertex of the nose, covering the eyes, on the cheeks, etc.), and applying the image element to the input image 102 to generate an output image 118 including the image element. It is noteworthy that the 3D facial model enables accurate application of such image elements.

[0048] Figure 4 4 shows an example 3D object element 401 being applied using a 3D face model 301, arranged in accordance with at least some implementations of the present disclosure. Figure 4As shown, a 3D object element 401 (i.e., a rendering result of a 3D flying butterfly) is applied to an input image 102 using a 3D facial model 301. Specifically, the 3D object element 401 is applied to a specific position 402 of the input image 102 based on the 3D facial model 301. In an embodiment, the 3D object element 401 will be applied to a desired position (e.g., on the forehead, on the tip of the nose, etc.) on the human face 202 and the corresponding 3D facial model 301, and the position is determined within the input image 102 using the morphable model parameter vector 116 when the morphable model parameter vector 116 is applied to the 3D facial model 131 to generate the 3D facial model 301. In an embodiment, the 3D object element 401 is rendered using the 3D facial model, and the resulting rendering is projected onto a 2D image plane. In an embodiment, the 3D facial model 301 is projected onto the 2D image plane of the input image 102, and the pixel positions 402 corresponding to specific vertices of the 3D facial model 301 are used for centering or providing the lower left corner of the 3D object element 401, etc. The 3D object element 401 can then be overlaid on the input image 102 to generate an output image 118 including the 3D object element 401, as shown.

[0049] refer to Figure 1 In other embodiments, a special effect texture is applied to a portion or all of the 3D facial model, the portion of the 3D facial model is converted to the 2D image plane of the input image 102, and the converted portion is overlaid on the input image 102 to generate the output image 118. In an embodiment, the effect and image output module 126 receives the 3D facial model generated by the deformation model parameter vector 116, and applies the texture to a region (e.g., above the eyes when implementing a mask, on the cheeks or eyelids when implementing makeup, on the cleft when implementing a scar, etc.) or the entire 3D facial model. The texture effect can be applied using any appropriate one or more techniques (e.g., texture mapping techniques and / or 3D graphics rendering techniques).

[0050] Figure 5 3D face model 301 is used to apply an example texture element 501, arranged in accordance with at least some implementations of the present disclosure. Figure 5As shown, a texture element 501 (i.e., a decorative mask texture) is applied to an input image 102 using a 3D facial model 301. Specifically, the texture element 501 is applied to a specific region 502 of the input image 102 based on the 3D facial model 301. In an embodiment, the texture element 501 will be applied to a desired region on the human face 202 and the corresponding 3D facial model 301 (e.g., on the forehead, on a portion used to mimic a mask, on the cheek, above the eyes, etc.), and the location is determined within the input image 102 using the morphable model parameter vector 116 when the morphable model parameter vector 116 is applied to the 3D facial model 131 to generate the 3D facial model 301. In an embodiment, the desired texture is applied to the 3D facial model 301, and the resulting textured region is projected onto the 2D image plane of the input image 102. The resulting textured projection is then overlaid onto the input image 102 to generate an output image 118 including the texture element 501, as shown.

[0051] Back to Figure 1 , the effects and image output module 126 may generate an output image 118 based on the rendered 3D facial model using any suitable one or more techniques. In an embodiment, the 3D facial model is generated to be photo-realistic (e.g., to match the human face 202 as closely as possible), converted to an image plane and inserted into a predetermined background image (e.g., the human face is placed on an otherwise animated or previously image-captured character or avatar, or placed in a previously image-captured environment or virtual reality (VR) setting, etc.) to generate the output image 118. In an embodiment, the 3D facial model is textured (e.g., colored to match the character), converted to the image plane of the input image 102 or another image plane and inserted into the input image 102 or a predetermined background image to generate the output image 118. In an embodiment, the 3D facial model is modified or adjusted to match the desired surgical output (e.g., for cosmetic or injury repair), and the adjusted 3D facial model is converted to an output image plane to generate the output image 118 for the surgeon to refer to during surgery. For example, the discussed technology can be used in a variety of contexts, such as creating special visual effects in movies, television, and games, mapping captured (and optionally modified) faces to digital avatars, adding 2D stickers or 3D dynamic props to faces in videos (e.g., replacing the main face with some virtual elements (optionally in real time)), applications in VR social networks, and using reconstructed 3D facial models to help doctors perform surgery.

[0052] As discussed, the image pre-processing module 121 generates input vectors 112, 113, 114 corresponding to the input images 102, 103, 104. As shown, the input vectors 113, 114 corresponding to the input images 103, 104, respectively, are combined to form an input vector 115. For example, the input vectors 113, 114 may include values ​​that include values ​​for each of the three (e.g., RGB) chroma values ​​for each scaled pixel value of the cropped and scaled-down regions as discussed herein. The input vector 115 includes such a combination of values. In an embodiment, the input vector includes a concatenation of the input vectors 113, 114. In an embodiment, the input vector includes 256×256×3×2 values ​​corresponding to values ​​for each of the three (e.g., RGB) chroma values ​​for each of the scaled-down pixel values ​​(e.g., at a resolution of 256×256 pixels) of both the cropped region of the input image 103 and the cropped region of the input image 104. Although discussed with respect to RGB values ​​for a square 256×256 pixel cropped region, any suitable pixel values ​​(e.g., YUV values) and shape and size of the cropped region may be used. Notably, input vector 115 is formatted in a format for pre-training tracking CNN 123.

[0053] As discussed, the input vector 115 corresponds to the input images 103, 104 (respectively labeled I t-1 and I t ). As discussed, input image 104 temporally follows input image 103. In an embodiment, input image 104 temporally follows input image 103 such that there are no intermediate frames between input images 103, 104. In an embodiment, input image 103 is image 10 and input image 104 is a temporally subsequent input image (e.g., I1). The processing discussed with respect to input images 103, 104 may be performed for any number of temporally adjacent input images.

[0054] The input vector 115 is provided to the tracking CNN 123, which processes the input vector 115 to generate a deformation model parameter delta vector 117 (i.e., Δθ t). Here, the deformation model parameter difference vector is labeled with Δθ and a subscript, where the subscript indicates the input image to which the deformation model parameter difference vector corresponds. The tracking CNN 123 may include any CNN with any number of layers (e.g., 6-8 layers) (including input layers and output layers). The input layer has the same number of nodes as the number of elements of the input vector 115 and the output layer has the same number of elements as the deformation model parameter vector 116 and the deformation model parameter difference vector 117, as required to model the 3D face model 131. The layers of the tracking CNN 123 may include any appropriate processing layers known in the art, for example, followed by layers with rectified linear units (ReLU), maximum pooling, and / or local response normalization (LRN). The tracking CNN 123 is a CNN that uses any appropriate one or more techniques (e.g., as described below in this document). Figure 7 Those techniques discussed).

[0055] As used herein, the term morphable model parameter difference vector indicates a vector for adjusting or modifying an element or value of a morphable model parameter vector. The morphable model parameter difference vector includes a difference value for each value or element of the morphable model parameter vector as discussed above. In an embodiment, the morphable model parameter difference vector 117 includes: a difference coefficient value for adjusting the coefficients of the shape unit basis matrix for modifying the 3D facial model 131, a difference coefficient value for adjusting the coefficients of the expression unit basis matrix for modifying the 3D facial model 131, a difference coefficient value for adjusting the coefficients of the texture unit basis matrix for modifying the 3D facial model 131, a difference value or parameter for adjusting the illumination parameter, a difference pose transformation parameter for modifying the rotation and translation values ​​respectively, and a difference camera parameter for adjusting the camera parameter for the pose transformation. That is, the morphable model parameter difference vector Δθ can be characterized as follows: Δθ={ΔR, ΔT, Δf, Δα id ,Δα alb ,Δα exp ,Δγ}.

[0056] For example, the morphable model parameter delta vector 117 may include 312 delta parameters corresponding to the 312 parameters discussed with respect to the morphable model parameter vector. Such modified or adjusted values ​​of the morphable model parameter delta vector 117 may be added to or subtracted from a morphable model parameter vector (e.g., morphable model parameter vector 132) to generate a morphable model parameter vector (θ t That is, the deformation model parameter difference vector 117 (which corresponds to the input vector 115, which in turn corresponds to the temporally adjacent input images 103, 104 (I t-1 ,I t) is added to the input image 103 (I t-1 ) corresponding to the deformation model parameter vector 116 (θ t-1 ) or from the deformation model parameter vector 116 (θ t-1 ) to generate the input image 104 (I t )’s deformation model parameter vector (θ t , not shown). Thus, the output of the tracking CNN 123 is used to provide temporal updates to the deformation model parameter vector across frames.

[0057] As shown, the 3D face rendering module 125 receives the deformation model parameter difference vector 117, the deformation model parameter vector 132 (which may be generated by the image CNN 122, or may be generated by a previous combination of the deformation model parameter vector and the deformation model parameter difference vector), and the 3D face model 131. The 3D face rendering module 125 generates a 3D face model for the current input image (e.g., 1 t ). This modification may be by adding, subtracting, or multiplying the morphable model parameter vector 132 and the morphable model parameter delta vector 117. Then, the 3D face rendering module 125 implements at least a portion of the morphable model parameter vector for the current image based on the 3D face model 131 (e.g., the representative 3D face model is a morphable model parameter vector θ). t The facial mesh or 3D facial model may be generated by the facial mesh or 3D facial model. Optionally, one or more special effects may be applied based on the facial mesh or 3D facial model via the effect and image output module 127.

[0058] The 3D facial rendering module 125 can generate a facial mesh or 3D facial model using any of the techniques discussed herein with respect to the 3D facial rendering module 124. In addition, the effects and image output module 127 can use any of the techniques discussed herein with respect to the effects and image output module 126 to implement special effects and generate an output image 119 corresponding to the input image 104. For the sake of simplicity and clear presentation, this description is not repeated. It is noteworthy that the 3D facial rendering module 125 and the 3D facial rendering module 124 and / or the effects and image output module 126 and the effects and image output module 127 can be implemented separately (as shown in the figure), or they can be implemented together. In an embodiment, the separate implementation of the first stage 141 and the second stage 142 can provide the advantages of parallel processing. For example, one or more processes of the first stage 141 and the second stage 142 can be implemented simultaneously.

[0059] Figure 6An example system 600 for pre-training image CNNs, arranged in accordance with at least some implementations of the present disclosure, is shown. Figure 6 As shown, the system 600 includes an image preprocessing module 121, an image CNN training module 611, a 3D face rendering module 612, and a loss metric module 613. The system 600 can be implemented via any suitable device discussed with respect to the system 100. As shown, the system 600 receives a training corpus 601 of any number of input images (only one of which is shown for clarity), and these input images include representations of human faces.

[0060] As discussed herein, the image preprocessing module 121 receives an input image 602 and generates a corresponding input vector 603 corresponding thereto. The training corpus 601 may include any number of input images 602, such as approximately 100,000 images, each of which includes a representation of a human face. In an embodiment, pre-training the image CNN is self-supervised training, so that the input image 602 of the training corpus 601 is an unlabeled image. The input image 602 and the input vector 603 may have any of the characteristics discussed herein with respect to the input images 102, 103, 104 and the input vectors 112, 113, 114. For example, the image preprocessing module 121 may perform face detection, facial landmark detection, etc. to detect a representation of a human face, crop the input image 602 into an area including a face, scale the cropped area down to a predefined standard pixel resolution, and generate each input vector 603 as follows: a concatenation of chroma pixel values ​​for each pixel position of the scaled down area. In an embodiment, each input vector includes 256x256x3 values ​​including a value for each of three (eg, RGB) chroma values ​​for each scaled-down pixel value.

[0061] In a large number of training iterations, the input vector 603 is iteratively provided to the image CNN training module 611. During training, the CNN parameters 604 are trained based on the input vector 603 and using a cost function to compare the face representation in the input image 602 and the reconstructed 3D face model, wherein the reconstructed 3D face model is generated based on the output of the current image CNN provided by the image CNN training module 611 using the current CNN parameters 604.

[0062] For example, in the current iteration and for the current input vector 603, the image CNN training module 611 uses the current CNN parameters 604 to implement the current image CNN to generate a morphable model parameter vector 605 corresponding thereto. The image CNN may have any of the characteristics discussed herein with respect to the image CNN 122. In addition, the morphable model parameter vector 605 may have any of the characteristics discussed with respect to the morphable model parameter vector 116 or any morphable model parameter vector discussed herein. The morphable model parameter vector 605 is provided to the 3D face rendering module 612, which uses the morphable model parameter vector 605 to implement the 3D face model 131 to generate a resulting face mesh or 3D face in a rendered output image 606 (illustrated herein as R). The resulting face mesh or 3D face includes any appropriate data structure discussed herein. It is noteworthy that the resulting face mesh or 3D face may include or implement all of the details of the morphable model parameter vector 605 to try to match the face representation included in the corresponding input image 602 as closely as possible. Furthermore, the resulting facial mesh or 3D face is projected onto the 2D image to generate a rendered output image 606 to provide a realistic composite image including a representation of a human face.

[0063] As shown, the rendered output image 606 and the corresponding input image 602 are provided to a loss metric module 613, which compares the rendered output image 606 and the corresponding input image 602 via a cost or loss function, which is minimized during the training of the image CNN by the image CNN training module 611 (e.g., by adjusting the CNN parameters 604). The cost or loss function can be any suitable function that provides a cost or loss related to the following: the difference between the rendered output image 606 and the corresponding input image 602 and / or other metrics (e.g., metrics for constraining the deformation model parameter vector). In an embodiment, the cost or loss function is the sum of a metric for photometric alignment between the input image 602 and the rendered output image 606 and a metric for constraining the deformation model parameter vector 605. In an embodiment, the metric for photometric alignment is the sum of per-pixel color differences (e.g., sum of squares, sum of absolute value differences, etc.). In an embodiment, the metric used to constrain the deformation model parameter vector 605 is a function that provides a scalar value corresponding to how much the deformation model parameter vector 605 has changed relative to the mean or median value or relative to the mean or median range of values. In an embodiment, the cost or loss function is provided as shown in equation (4):

[0064] E Loss (θ) = λ p Ephoto (θ)+λ r E reg (θ) (4)

[0066] Among them, E Loss (θ) is a cost or loss function and provides a cost that varies with the current morphable model parameter vector 605 based on the rendered output image 606 and the corresponding input image 602 and the constraint metric of the morphable model parameter vector 605, E pho (θ) is a measure of the dense photometric alignment between the rendered output image 606 and the corresponding input image 602 (e.g., the sum of per-pixel color differences) and is represented as a scalar value, E reg (θ) is a constraint function expressed as a scalar value used to measure the variation of the deformation model parameter vector 605 relative to the mean or median and / or to measure the constraint violation of the deformation model parameter vector 605 on the expected parameter value, λ p is the weighting coefficient for the dense photometric alignment metric, and E r is the weighting coefficient for the constraint function measure.

[0067] After the loss metric module 613 performs the evaluation, the CNN parameters 604 are adjusted (e.g., on a per-image, per-image group, or per-training corpus basis), and this process is repeated until the cost or loss function applied by the loss metric module 613 is minimized. As discussed, in some embodiments, this CNN training is self-supervised, such that the input images 602 of the training corpus 601 are unlabeled images. In addition, this CNN training is end-to-end training, such that the input vectors 603 and the output deformation model parameter vectors 605 are directly evaluated during training without the need to segment the image CNN. Moreover, since the rendering layer (e.g., as implemented by the 3D face rendering module 612) and the loss function (e.g., as implemented by the loss metric module 613) are fully analytical and differentiable, back-forward pass training can be implemented using back-propagation techniques known in the art.

[0068] Figure 7 An example system 700 for pre-training a tracking CNN is shown, arranged in accordance with at least some implementations of the present disclosure. Figure 7As shown, system 700 includes image preprocessing module 121, tracking CNN training module 711, 3D face rendering module 712, loss metric module 713, and optical flow module 714. System 700 can be implemented via any suitable device discussed with respect to system 100. As shown, system 700 receives a training corpus 701 of any number of temporally sequential input images or video frames (only two of which are shown for clarity), each input image or video frame including a representation of a human face.

[0069] As discussed herein, the image preprocessing module 121 receives input images 702, 703 and generates corresponding input vectors 704, 705 corresponding thereto. The training corpus 701 may include any number of sequential input images or frames, for example, approximately 1,000 image or frame sequences, each sequence having approximately 200 sequential images or frames. In an embodiment, pre-training the tracking CNN is self-supervised training, so that the input images of the training corpus 601 are unlabeled images. The input images 702, 703 and the input vectors 704, 705 and the input vector 706 (e.g., a combination of the input vectors 704, 705) may have any characteristics discussed herein with respect to the input images 102, 103, 104 and the input vectors 112, 113, 114, 115. For example, the image preprocessing module 121 can perform face detection, facial landmark detection, etc. to detect representations of human faces, crop the input images 702, 703 into regions including faces, scale the cropped regions down to a predefined standard pixel resolution, and generate each input vector 702, 703 as follows: a concatenation of chroma pixel values ​​for each pixel position of the scaled down region. In an embodiment, each input vector includes 256×256×3 values ​​including values ​​for each of three (e.g., RGB) chroma values ​​for each scaled down pixel value. In addition, the image preprocessing module 121 or another module generates an input vector 706 by concatenating the input vectors 704, 705.

[0070] For example, for each pair of temporally sequential input images 702, 703, an input vector 706 is generated such that the input vector 706 includes pixel values ​​for each region of the input images 702, 703 that includes a face representation. In a large number of training iterations, the input vector 706 is iteratively provided to the tracking CNN training module 711. During training, the CNN parameters 707 are trained based on the input vector 706 and by comparing the face representation in the input image 703 and the reconstructed 3D face model using a cost function, wherein the reconstructed 3D face model is generated based on the output of the current tracking CNN provided by the tracking CNN training module 711 using the current CNN parameters 707.

[0071] For example, in the current iteration and for the current input vector 706, the tracking CNN training module 711 uses the current CNN parameters 707 to implement the current tracking CNN to generate a morphable model parameter delta vector 708 corresponding thereto. The tracking CNN may have any of the characteristics discussed herein with respect to the tracking CNN 123. In addition, the morphable model parameter delta vector 708 may have any of the characteristics discussed with respect to the morphable model parameter delta vector 117 or any morphable model parameter delta vector discussed herein. The morphable model parameter delta vector 708 is provided to the 3D face rendering module 712, which performs the morphable model parameter delta vector 117 by comparing it with the morphable model parameter vector 732 (θ corresponding to the input image 702). t-1 ) is added or subtracted to generate the deformation model parameter vector (θ t , not shown). The deformation model parameter vector 732 may be generated using any suitable technique or techniques. In an embodiment, a pre-trained image CNN 122 (e.g., Figure 6 ) generates a deformation model parameter vector 732 for each input image for use by the 3D face rendering module 712. Then, the 3D face rendering module 712 uses the deformation model parameter vector (θ) for the current input image 703 t ) and the input vector 706 to implement the 3D facial model 131 to render an output image 709 (shown as R t ) is generated from the 2D image 704. The resulting facial mesh or 3D face comprises any suitable data structure discussed herein. Notably, the resulting facial mesh or 3D face may include or implement all details of the morphable model parameter vector to attempt to match as closely as possible the representation of the human face included in the corresponding input image 703. In addition, the resulting facial mesh or 3D face is projected onto the 2D image to generate a rendered output image 709 to provide a realistic composite image including the representation of the human face.

[0072] As shown, the rendered output image 709 and the corresponding input image 702 are provided to a loss metric module 713, which compares the rendered output image 709 with the corresponding input image 703 and with the rendered image 710 corresponding to the morphable model parameter vector 732 via a cost or loss function (wherein a pixel match between the rendered output image and the rendered image 710 is established using an optical flow module 714), and the cost or loss function is minimized during the training of the tracking CNN by the tracking CNN training module 711 (e.g., by adjusting the CNN parameters 707). The cost or loss function can be any suitable function that provides the following items: a cost or loss related to the difference between the rendered output image 709 and the corresponding input image 703, and / or a cost or loss related to the difference between the rendered output image 709 and the rendered image 710. As discussed, the rendered image 710 is based on the morphable model parameter vector 732. In an embodiment, the rendered image 710 is generated by implementing the 3D face model 131 using the morphable model parameter vector 732 (as discussed herein). In an embodiment, deformation model parameter vector 732 is obtained by applying image CNN 122 to input image 702 .

[0073] To compare the rendered output image 709 and the rendered image 710, optical flow data from the optical flow module 714 can be used to establish pixel matches. Such optical flow data can be in any suitable format, for example, per-pixel motion provided by per-pixel displacement and direction data. As shown, the optical flow module 714 can evaluate the sub-regions including the face representation to generate such optical flow data between the input images 702 and 703, as known in the art. The resulting optical flow data is used to establish pixel matches between the rendered output image 709 and the rendered image 710 (for example, this is done by finding matching pixels in the rendered output image 709 for pixels of the rendered image 710 based on the per-pixel displacement and direction data).

[0074] As discussed, the cost or loss function evaluated by the loss metric module 713 can be any suitable function that provides a cost or loss associated with the difference between the rendered output image 709 and the corresponding input image 703, and / or a cost or loss associated with the difference between the rendered output image 709 and the rendered image 710. In an embodiment, the cost or loss function is the sum of a measure of photometric alignment between the rendered output image 709 and the corresponding input image 703, and a measure of temporal matching between the rendered output image 709 and the rendered image 710 (as pixels matched using optical flow data). In an embodiment, the measure of photometric alignment is the sum of per-pixel color differences (e.g., sum of squares, sum of absolute value differences, etc.). Similarly, in an embodiment, the measure of temporal matching is the sum of per-pixel color differences (e.g., sum of squares, sum of absolute value differences, etc.), both of which can be provided as scalar values. In an embodiment, the cost or loss function is provided as shown in equation (5):

[0075] E Loss (θ) = λ s E spatial (θ)+λ t E temporal (θ) (5)

[0077] Among them, E Loss (θ) is a cost or loss function and provides a loss function with respect to the current morphable model parameter vector (θ) based on the rendered output image 709, the input image 703, and the rendered image 710. t ) and the deformation model parameter vector (Δθ t ) The cost of change, E spatial (θ) is a measure of the dense photometric alignment between the rendered output image 709 and the corresponding input image 703 (e.g., the sum of per-pixel color differences) and is represented as a scalar value (e.g., similar to E photo (θ), and provides a measure of how well the rendered image matches the observed face image), and E temporal (θ) is a measure of temporal matching between rendered output image 709 and rendered image 710 (e.g., used to limit large changes between the current frame and the previous frame). As discussed, in an embodiment, E temporal (θ) is the per-pixel color difference between the rendered output image 709 and the rendered image 710. In an embodiment, E temporal (θ) is determined as shown in equation (6):

[0078]

[0079] Among them, R tis the rendered output image 709, R t-1 is the rendered image 710, and Ω is the number of pixels in each image or image portion, and i is a counter variable. As discussed, the pixel correspondence between the rendered output image 709 and the rendered image 710 is determined by the optical flow between the input images 702, 703 (e.g., the input image pair). As discussed, E temporal (θ) ensures temporal tracking in the video tracking result by measuring the temporal coherence between the input images 702 , 703 .

[0080] Figure 8 FIG. 8 is a flow chart illustrating an example process 800 for training CNNs and implementing them for 3D facial capture and rendering, arranged in accordance with at least some implementations of the present disclosure. The process 800 may include: Figure 8 One or more operations 801-816 are shown. Process 800, or portions thereof, may be performed by any device or system discussed herein.

[0081] Process 800 begins at operation 801, where an image CNN is trained. The image CNN may be trained using any suitable one or more techniques. In an embodiment, a predefined CNN such as a feed-forward artificial neural network is modified to include: a number of input nodes equal to the number of elements in an input vector representing an image having a representation of a human face (e.g., equal to the number of elements for the value of each color at each pixel position in a cropped and scaled-down region), and a number of output nodes equal to the number of parameters in a deformation model parameter vector θ. The image CNN is trained to determine parameters (e.g., weights, coefficients, etc.) of the image CNN. In an embodiment, training includes iteratively providing the input vector to the image CNN and modifying its parameters based on a loss or cost function associated with the results from the image CNN. For example, a 3D facial mesh or model is generated using the resulting deformation model parameter vector, which is converted to a 2D image plane. The resulting image is compared to the input image to determine a spatial cost or difference. The spatial cost or difference is added to the cost associated with any deviation of the resulting deformation model parameter vector from the constraints applied thereto to determine a total cost. Throughout the training process, the cost or loss function is minimized across training images and the resulting image CNN parameters are stored for implementation at operation 806 .

[0082] Processing continues at operation 802, where a tracking CNN is trained. The tracking CNN may be trained using any suitable technique or techniques. A predefined CNN, such as a feed-forward artificial neural network, is modified to include: a number of input nodes equal to the number of elements in an input vector representing temporally sequential pairs of images each having a representation of a face (e.g., equal to the number of elements for the value of each color at each pixel position in a cropped and scaled-down region of temporally adjacent images), and a number of output nodes equal to the number of parameters in a deformation model parameter difference vector Δθ. The tracking CNN is trained to determine parameters (e.g., weights, coefficients, etc.) of the tracking CNN. In an embodiment, the training includes iteratively providing the input vector to the tracking CNN, and modifying its parameters based on a loss or cost function associated with the results from the tracking CNN. For example, a 3D face mesh or model is generated using the resulting deformation model parameter difference vector (after first modifying a previous image deformation model parameter vector using the deformation model parameter difference vector), which is converted to a 2D image plane. The resulting image is compared to the input image to determine a spatial cost or difference. The spatial cost or difference is added to the temporal cost, where the temporal cost is determined by comparing the resulting image to a previously rendered image (with pixel matching based on optical flow analysis of the input image pair). Throughout the training process, the cost or loss function is minimized across training images, and the resulting tracking CNN parameters are stored for implementation at operation 811.

[0083] As shown, operations 802 and 803 are performed in a training phase 821, and subsequent operations of process 800 are performed in an implementation phase 831. It should be noted that the training phase 821 and the implementation phase 831 may be performed by different devices.

[0084] Processing continues at operation 803, where an input video is received using any suitable technique or techniques. In an embodiment, the input video is captured at a local device, pre-processed, and provided as a sequence of RGB or YUV images, pictures, or frames. In an embodiment, the input video is received from a local memory or a remote device. Processing continues at operation 804, where an image is selected and cropped and scaled down as needed. In an embodiment, a face is detected within the selected image and cropped around the detected face using a predefined shape (e.g., square, rectangle, etc.). In an embodiment, after cropping, the cropped area is scaled down to a predefined pixel resolution (e.g., N×M pixels).

[0085] Processing continues at decision operation 805, where a determination is made whether the image selected at operation 804 is the first frame in the video sequence. This determination may be made using any appropriate technique or techniques (e.g., scene change detection or a determination that the image is the first available image, etc.). If so, processing continues at operation 806, where an input vector is generated for the current image as follows: chroma pixel values ​​or luma and chroma values ​​for the cropped and scaled region (e.g., N×M×3 values ​​for 3 chroma channels or 1 luma channel and 2 chroma channels). The pixel values ​​may be organized within the vector using any appropriate technique or techniques (e.g., raster scan ordering, etc.).

[0086] Processing continues at operation 807, where the image CNN is applied to the input vector to generate a morphable model parameter vector (θ). In an embodiment, the image CNN trained at operation 801 is applied to the input vector to generate a morphable model parameter vector. The image CNN can be any image CNN discussed herein. In addition, the morphable model parameter vector includes coefficients and parameters for rendering a 3D facial mesh or facial model based on a 3D facial model (e.g., a 3DMM of a face).

[0087] Processing continues at operation 808 where a 3D facial model is rendered based on the morphable model parameter vector, and at operation 809 a visual effect is applied or the image is modified based on the facial mesh or model to generate an output image. In an embodiment, a portion of the parameters of the morphable model parameter vector are used to render a 3D facial mesh or model, which is then modified (e.g., by applying color or texture to the face) to generate a 2D image. In an embodiment, a portion or all of the parameters of the morphable model parameter vector are used to render a 3D facial mesh or model, which is converted to a 2D image and used to generate either a modified version of the current input image as an output image, or a completely new output image. For example, a 2D image of a face may be inserted into an input image or applied to another image (e.g., with a particular background or animation) to generate an output image. In an embodiment, the image is rendered using a 3D facial mesh or model such that the 3D facial mesh or model is inserted into the 3D model prior to conversion to a 2D image. In an embodiment, a photo-realistic version of the 3D facial mesh or model is inserted into the image background.

[0088] Returning to decision operation 805, if the current image is not the first image in the sequence, processing continues at operation 810, where an input vector is generated for an input image pair including the current image and the temporally previous image as follows: chroma pixel values ​​or luma and chroma values ​​for cropped and scaled regions of the current image and the temporally previous image (e.g., N×M×3×2 values ​​for 3 chroma channels or 1 luma channel and 2 chroma channels for both images). The pixel values ​​may be organized within the vector using any suitable technique or techniques (e.g., raster scan ordering the region for the temporally previous image, followed by raster scan ordering the region for the current image, etc.).

[0089] Processing continues at operation 811, where a tracking CNN is applied to the input vector to generate a morphable model parameter delta vector (Δθ). In an embodiment, the tracking CNN trained at operation 802 is applied to the input vector to generate a morphable model parameter delta vector. The tracking CNN can be any tracking CNN discussed herein. In addition, the morphable model parameter delta vector includes adjustments to coefficients and parameters used to render a 3D facial mesh or facial model based on a 3D facial model (e.g., a 3DMM of a face) (e.g., deltas to such values).

[0090] Processing continues at operation 812, where the morphable model parameter delta vector generated at operation 811 is added to a previous morphable model parameter vector for a temporally previous image in the image pair (or the morphable model parameter delta vector generated at operation 811 is subtracted from the previous morphable model parameter vector) to determine a morphable model parameter vector for the current frame. Processing continues at operation 813, where a 3D facial model is rendered based on the morphable model parameter vector, and at operation 814, a visual effect is applied or the image is modified based on the facial mesh or model to generate an output image. In an embodiment, a portion of the parameters of the morphable model parameter vector are used to render a 3D facial mesh or model, which is then modified (e.g., by applying color or texture to the face) to generate a 2D image. In an embodiment, a portion or all of the parameters of the morphable model parameter vector are used to render a 3D facial mesh or model, which is converted to a 2D image and used to generate either a modified version of the current input image as an output image, or a completely new output image. For example, a 2D image of a face may be inserted into an input image or applied to another image (e.g., with a particular background or animation) to generate an output image. In an embodiment, a 3D facial mesh or model is used to render the image, such that the 3D facial mesh or model is inserted into the 3D model before conversion to a 2D image. In an embodiment, a photo-realistic version of the 3D facial mesh or model is inserted into the image background.

[0091] Processing continues from operation 809 or operation 814 at decision operation 815, where it is determined whether the current image is the last image in the sequence. If not, processing continues at operation 804, as discussed above. If yes, processing continues at operation 816, where the generated output image is stored as a video sequence and / or the generated output image is presented to the user as a video via a display.

[0092] Fig. 9 is a flow chart illustrating an example process 900 for processing a video, arranged in accordance with at least some implementations of the present disclosure. The process 900 may include: Fig. 9 One or more operations 901-907 are shown. Process 900 may form at least a portion of a video processing technique. By way of non-limiting example, process 900 may form at least a portion of a processing technique performed by one or more of systems 100, 600, 700 discussed herein. Fig.10 Process 900 is described with reference to system 1000 .

[0093] Fig.10 is an illustrative diagram of an example system 1000 for processing video, arranged in accordance with at least some implementations of the present disclosure. Fig.10As shown, the system 1000 may include a central processor 1001, an image processor 1002, a memory storage device 1003, and a camera 1004. For example, the camera 1004 and a display (not shown) may respectively obtain an input video for processing and display a video modified based on a 3D face model generated as discussed herein. Also as shown, the central processor 1001 may include or implement an image preprocessing module 121, an image CNN 122, a tracking CNN 123, one or more 3D face rendering modules 124, 125, 612, 712, one or more effects and image output modules 126, 127, an image CNN training module 611 and a loss metric module 613, and a tracking CNN training module 711, a loss metric module 713, and an optical flow module 714. In an embodiment, the system 1000 implements only the image preprocessing module 121, the image CNN 122, the tracking CNN 123, the 3D face rendering modules 124, 125, and the image output modules 126, 127 (i.e., in an implementation scenario). In an embodiment, the system 1000 implements only the image preprocessing module 121, the image CNN training module 611 and the loss metric module 613 (i.e., in an image CNN training scenario). In an embodiment, the system 1000 implements only the tracking CNN training module 711, the loss metric module 713 and the optical flow module 714 (i.e., in a tracking CNN training scenario). Such components or modules may be implemented to perform the operations discussed herein. The memory storage device 1003 may store images, image data, input vectors, output vectors, 3D model parameters, deformation model parameter vectors, deformation model parameter delta vectors, effect data, or any other data discussed herein.

[0094] As shown in the figure, in some examples, one or more of the following items are implemented via the central processor 1001: image preprocessing module 121, image CNN 122, tracking CNN 123, one or more 3D face rendering modules 124, 125, 612, 712, one or more effects and image output modules 126, 127, image CNN training module 611 and loss metric module 613, and tracking CNN training module 711, loss metric module 713 and optical flow module 714. In other examples, one or more of the following items or a part of the following items are implemented via the image processor 1002, the video processor, the graphics processor, etc.: image preprocessing module 121, image CNN 122, tracking CNN 123, one or more 3D face rendering modules 124, 125, 612, 712, one or more effects and image output modules 126, 127, image CNN training module 611 and loss metric module 613, and tracking CNN training module 711, loss metric module 713 and optical flow module 714. In another example, one or more of the following items or a portion of the following items are implemented via an image or video processing pipeline or unit: an image preprocessing module 121, an image CNN 122, a tracking CNN 123, one or more 3D face rendering modules 124, 125, 612, 712, one or more effects and image output modules 126, 127, an image CNN training module 611 and a loss metric module 613, and a tracking CNN training module 711, a loss metric module 713 and an optical flow module 714.

[0095] The image processor 1002 may include any number and type of graphics, image, or video processing units that can provide operations as discussed herein. In some examples, the image processor 1002 may be an image signal processor. Such operations may be implemented via software or hardware or a combination thereof. For example, the image processor 1002 may include a circuit system dedicated to manipulating frame data, image data, or video data obtained from the memory storage device 1003. The central processor 1001 may include any number and type of processing units or modules that can provide control and other advanced functions for the system 1000 and / or provide any operations as discussed herein. The memory storage device 1003 may be any type of memory, for example, a volatile memory (e.g., a static random access memory (SRAM), a dynamic random access memory (DRAM), etc.) or a non-volatile memory (e.g., a flash memory, etc.), etc. In a non-limiting example, the memory storage device 1003 may be implemented by a buffer memory.

[0096] In an embodiment, one or more of the following or a portion of the following is implemented via an execution unit (EU) of the image processor 1002: image preprocessing module 121, image CNN 122, tracking CNN 123, one or more 3D face rendering modules 124, 125, 612, 712, one or more effects and image output modules 126, 127, image CNN training module 611 and loss metric module 613, and tracking CNN training module 711, loss metric module 713 and optical flow module 714. The EU may include, for example, a programmable logic or circuit system (e.g., one or more logic cores) that can provide a variety of programmable logic functions. In an embodiment, one or more of the following items or a portion of the following items are implemented via dedicated hardware (e.g., fixed function circuit system, etc.): image preprocessing module 121, image CNN 122, tracking CNN 123, one or more 3D face rendering modules 124, 125, 612, 712, one or more effects and image output modules 126, 127, image CNN training module 611 and loss metric module 613, and tracking CNN training module 711, loss metric module 713 and optical flow module 714. Fixed function circuit system may include dedicated logic or circuit system and may provide a set of fixed function entry points, which may be mapped to dedicated logic for fixed purposes or functions. In some embodiments, one or more of the following items or a portion of the following items may be implemented via application specific integrated circuits (ASICs): image preprocessing module 121, camera and pose estimation module 132, contour determination and matching module 133, linear shape estimation module 134, texture estimation module 135, and shape refinement module 136. ASICs may include integrated circuit systems customized to perform the operations discussed herein. Camera 1004 may include any camera having any suitable lens and image sensor and / or related hardware for capturing images or video.

[0097] Return to Fig. 9 As discussed above, process 900 begins at operation 901 where sequential video images, frames, or pictures with representations of human faces are received for processing. The sequential video images may be in any format and may be received locally from camera 1004 or from memory storage 1003 or from a remote device (not shown).

[0098] Processing continues at operation 902, where a first (image) CNN is applied to a first input vector corresponding to a first video image in the sequence of video images to generate a first morphable model parameter vector. The first video image may be any video image in the sequence of video images. In an embodiment, the first video image is the first video image in the temporally sequential video images. In an embodiment, the first video image is a scene change image or an I-frame, etc. The first input vector may be any input vector for input to the image CNN discussed herein. In an embodiment, the first input vector includes chrominance values ​​for pixels of a first scaled-down sub-region of the first video image. For example, the first scaled-down sub-region may be a cropped and scaled-down region including a representation of a human face. In an embodiment, process 900 further includes: performing face detection on the input image, cropping the image into sub-regions including the detected face, and scaling the cropped region down to a predefined pixel resolution. The first morphable model parameter vector and any other morphable model parameter vectors discussed herein may include any suitable parameters for generating a face model from a 3D face model. In an embodiment, the morphable model parameter vector includes the concatenation of the following coefficients: coefficients for modifying the shape unit matrix of the 3D facial model, coefficients for modifying the expression unit matrix of the 3D facial model, and coefficients for modifying the texture unit matrix of the 3D facial model. Optionally, the morphable model parameter vector may include additional parameters, such as rotation parameters, translation parameters, camera parameters, and illumination parameters.

[0099] Processing continues at operation 903 where a second (tracking) and separate CNN is applied to a second input vector corresponding to a first video image and a second video image in the plurality of sequential video images to generate a morphable model parameter delta vector. The second video image may be any video image that is temporally subsequent to the first video image, e.g., a video image that is immediately temporally subsequent thereto.

[0100] The second input vector can be any input vector for input to the tracking CNN discussed herein. In an embodiment, the second input vector includes chrominance values ​​for pixels of a first scaled-down sub-region of the first video image, and chrominance values ​​for pixels of a second scaled-down sub-region of the second video image. For example, the second scaled-down sub-region can be a cropped and scaled-down region including a face representation similar to the first scaled-down sub-region. In an embodiment, the second input vector includes information of the first input vector (e.g., chrominance values ​​for pixels of the first scaled-down sub-region) concatenated with similar data for the second input image (e.g., chrominance values ​​for pixels of the second scaled-down sub-region). In an embodiment, the first and second input images are characterized as an input image pair.

[0101] In an embodiment, process 900 also includes: applying a second (tracking) CNN to a third input vector corresponding to a second video image and a third video image in a plurality of sequential video images to generate a second morphable model parameter delta vector, such that the third input vector includes a second chroma value (e.g., for the second input image) and a third chroma value for a third pixel of a third scaled-down sub-region of the third video image. For example, the processing discussed with respect to operation 903 may be repeated for any number of subsequent image pairs (e.g., of temporally adjacent images). In an embodiment, each of the first, second, and third chroma values ​​respectively includes a value (e.g., RGB value) for a first, second, and third chroma channel of the input image. The morphable model parameter delta vector and any other morphable model parameter delta vector discussed herein may include any suitable parameters for modifying the morphable model parameter vector, which, after modification, may be used to generate a facial model from a 3D facial model. In an embodiment, the morphable model parameter difference vector comprises a concatenation of second coefficients for modifying each of the following coefficients: coefficients for modifying a shape unit basis matrix before being applied to the 3D facial model, coefficients for modifying an expression unit basis matrix, and coefficients for modifying a texture unit basis matrix. Optionally, the morphable model parameter difference vector may comprise additional parameters or coefficients or values ​​for modifying rotation parameters, translation parameters, camera parameters, and illumination parameters.

[0102] Processing continues at operation 904, where a first facial model of a human face is generated by applying the first morphable model parameter vector to a 3D facial model. The 3D facial model may include any of the models discussed herein (e.g., the models discussed with respect to equations (1), (2), and (3)) or a portion thereof. For example, the 3D facial model may be generated using fewer than all available modeling parameters (e.g., using only shape, using only shape and albedo, etc.). As used herein, the term applying the morphable model parameter vector to the 3D facial model indicates that the 3D model is generated or rendered using the 3D facial model to implement the morphable model parameters. That is, the 3D facial model (e.g., 3DMM) is designed to be modified by using coefficients or parameters implemented using the morphable model parameter vector.

[0103] Processing continues at operation 905, where a second deformation model parameter vector is determined by modifying the first deformation model parameter vector generated at operation 902 using the deformation model parameter difference vector. The first deformation model parameter vector may be modified using the deformation model parameter difference vector using any suitable technique or techniques. In an embodiment, the deformation model parameter difference vector is added to the first deformation model parameter vector to generate the second deformation model parameter vector. In an embodiment, the deformation model parameter difference vector is subtracted from the first deformation model parameter vector to generate the second deformation model parameter vector. In an embodiment, the deformation model parameter difference vector and the first deformation model parameter vector are element-wise multiplied to generate the second deformation model parameter vector.

[0104] Processing continues at operation 906 where a second facial model of the human face is generated by applying the second morphable model parameter vector to the 3D facial model. As discussed with respect to operation 905, the 3D facial model may include any model discussed herein.

[0105] Processing continues at operation 907, where an output video image including image features based at least in part on the first and second facial models is stored to a memory, presented via a display, transmitted to another device, etc. The image features may be based at least in part on the first and second facial models in any suitable manner. In an embodiment, the image features are provided at specific locations in the output image using specific locations (e.g., nose, ears, etc.) of the first and second facial models. In an embodiment, the image features include a rendition of the facial models (e.g., a photo-realistic rendering, a rendering with a replaced texture or portion of a texture, a rendering with a modification to the shape of the facial models, etc.). In an embodiment, the image features are applied to modify the input image. In an embodiment, the image features are part of the newly generated output image. In an embodiment, process 900 also includes applying an effect to at least one of the first or second facial models before storing, presenting, or storing the output video image.

[0106] In an embodiment, process 900 also includes pre-training the first (image) CNN by generating a plurality of training input vectors based on training images (wherein each training image includes a representation of a human face, such that each training input vector includes chromaticity values ​​for a region of the training image including a human face), iteratively applying the first CNN to the plurality of training input vectors to generate a corresponding plurality of morphable model parameter vectors, rendering a resulting output image by applying the plurality of morphable model parameter vectors to a 3D facial model, and minimizing a cost function, wherein the cost function includes a measure of photometric alignment between the training images and the resulting output image and a measure for constraining the morphable model parameter vectors. In an embodiment, pre-training includes self-supervised training such that the training video images include unlabeled images.

[0107] In an embodiment, process 900 also includes pre-training a second (tracking) CNN by generating a plurality of training input vectors based on training video images (wherein each training video image includes a representation of a human face such that each training input vector includes a first chromaticity value for a first region of a first particular training video image and a second chromaticity value for a second region of a second particular training video image that is temporally subsequent to the first particular training video image), iteratively applying the second CNN to the plurality of training input vectors to generate a corresponding plurality of morphable model parameter delta vectors, rendering a first resulting output image by applying the morphable model parameter vectors adjusted using the morphable model parameter delta vectors to a 3D facial model, and minimizing a cost function that includes a measure of photometric alignment between the training video image and the first resulting output image, and a measure of temporal coherence between the first resulting output image and an output image obtained at a previous time. In an embodiment, the measure of temporal coherence includes a per-pixel difference color difference between the first resulting output image and an output image obtained at a previous time, the per-pixel difference color difference being based on a per-pixel correspondence between the first resulting output image and the output image obtained at the previous time, wherein the per-pixel correspondence is generated based on optical flow detection of the first and second particular training video images.

[0108] The various components of the systems described herein may be implemented in software, firmware, and / or hardware and / or any combination thereof. For example, the various components of the systems discussed herein may be provided at least in part by hardware of a computing system-on-chip (SoC) (e.g., such as may be found in, for example, a computing system (e.g., a smart phone)). It will be appreciated by those skilled in the art that the systems described herein may include additional components not depicted in the corresponding figures. For example, the systems discussed herein may include additional components such as a communication module that are not depicted for clarity.

[0109] Although implementations of the example processes discussed herein may include performing all of the operations shown in the order shown, the present disclosure is not limited in this respect, and in various examples, implementations of the example processes herein may include only a subset of the operations shown, the operations performed in an order different from that shown, or additional operations.

[0110] In addition, any one or more operations discussed herein may be performed in response to instructions provided by one or more computer program products. Such a program product may include a signal bearing medium providing instructions, which may provide the functions described herein when executed by, for example, a processor. A computer program product may be provided in any form of one or more machine-readable media. Thus, for example, a processor including one or more graphics processing units or (one or more) processor cores may perform one or more blocks of the example process herein in response to program codes and / or instructions or instruction sets delivered to the processor by one or more machine-readable media. Typically, a machine-readable medium may deliver software in the form of program codes and / or instructions or instruction sets, which may enable any device and / or system described herein to implement at least a portion of the system discussed herein or any other module or component discussed herein.

[0111] As used in any implementation described herein, the term "module" or "component" refers to any combination of software logic, firmware logic, hardware logic, and / or circuitry configured to provide the functionality described herein. Software may be embodied as a software package, code, and / or instruction set or instructions, and "hardware" used in any implementation described herein may include, for example, a single or any combination of hard-wired circuitry, programmable circuitry, state machine circuitry, fixed function circuitry, execution unit circuitry, and / or firmware that stores instructions executed by programmable circuitry. Modules may be collectively or individually embodied as circuitry that forms part of a larger system (e.g., an integrated circuit (IC), a system on a chip (SoC), etc.).

[0112] Fig.111 is an illustrative diagram of an example system 1100 arranged according to at least some implementations of the present disclosure. In various implementations, the system 1100 may be a mobile system, but the system 1100 is not limited to this context. The system 1100 may implement and / or execute any module or technology discussed herein. For example, the system 1100 may be incorporated into a personal computer (PC), a server, a laptop computer, an ultra-laptop computer, a tablet computer, a touchpad, a portable computer, a handheld computer, a palmtop computer, a personal digital assistant (PDA), a cellular phone, a combination cellular phone / PDA, a television, a smart device (e.g., a smart phone, a smart tablet computer, or a smart TV), a mobile Internet device (MID), a messaging device, a data communication device, a camera (e.g., a point-and-shoot camera, a super zoom camera, a digital single-lens reflex (DSLR) camera), etc. In some examples, the system 1100 may be implemented via a cloud computing environment.

[0113] In various implementations, system 1100 includes a platform 1102 coupled to a display 1120. Platform 1102 may receive content from a content device (e.g., content services device(s) 1130 or content delivery device(s) 1140) or other similar content sources. A navigation controller 1150 including one or more navigation features may be used to interact with, for example, platform 1102 and / or display 1120. Each of these components is described in more detail below.

[0114] In various implementations, the platform 1102 may include any combination of the following: a chipset 1105, a processor 1110, a memory 1112, an antenna 1113, a storage device 1114, a graphics subsystem 1115, applications 1116, and / or a radio 1118. The chipset 1105 may provide intercommunication between the processor 1110, the memory 1112, the storage device 1114, the graphics subsystem 1115, the applications 1116, and / or the radio 1118. For example, the chipset 1105 may include a storage device adapter (not depicted) capable of providing intercommunication with the storage device 1114.

[0115] Processor 1110 may be implemented as a complex instruction set computer (CISC) or reduced instruction set computer (RISC) processor, a processor compatible with the x86 instruction set, multi-core, or any other microprocessor or central processing unit (CPU). In various implementations, processor 1110 may be a (one or more) dual-core processor, (one or more) dual-core mobile processor, and the like.

[0116] Memory 1112 may be implemented as a volatile memory device such as, but not limited to, a random access memory (RAM), a dynamic random access memory (DRAM), or a static RAM (SRAM).

[0117] The storage device 1114 may be implemented as a non-volatile storage device, such as, but not limited to, a magnetic disk drive, an optical disk drive, a tape drive, an internal storage device, an attached storage device, flash memory, a battery-backed SDRAM (synchronous DRAM), and / or a network accessible storage device. In various implementations, the storage device 1114 may include technology for increasing storage performance-enhancing protection of valuable digital media when, for example, multiple hard drives are included.

[0118] Image signal processor 1117 may be implemented as a dedicated digital signal processor for image or video frame processing, etc. In some examples, image signal processor 1117 may be implemented based on a single instruction multiple data or multiple instruction multiple data architecture, etc. In some examples, image signal processor 1117 may be characterized as a media processor. As discussed herein, image signal processor 1117 may be implemented based on a system-on-chip architecture and / or based on a multi-core architecture.

[0119] The graphics subsystem 1115 may perform processing of images such as still images or videos for display. The graphics subsystem 1115 may be, for example, a graphics processing unit (GPU) or a visual processing unit (VPU). An analog or digital interface may be used to communicatively couple the graphics subsystem 1115 and the display 1120. For example, the interface may be any of the following: a high-definition multimedia interface, DisplayPort, wireless HDMI, and / or wireless HD compliant technology. The graphics subsystem 1115 may be integrated into the processor 1110 or the chipset 1105. In some implementations, the graphics subsystem 1115 may be a standalone device communicatively coupled to the chipset 1105.

[0120] The graphics and / or video processing techniques described herein can be implemented in various hardware architectures. For example, the graphics and / or video functions can be integrated into a chipset. Alternatively, a discrete graphics and / or video processor can be used. As another implementation, the graphics and / or video functions can be provided by a general-purpose processor including a multi-core processor. In other embodiments, the functions can be implemented in a consumer electronic device.

[0121] Radio 1118 may include one or more radios capable of sending and receiving signals using various suitable wireless communication technologies. Such technologies may involve communications across one or more wireless networks. Example wireless networks include, but are not limited to, wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area network (WMANs), cellular networks, and satellite networks. When communicating across such networks, radio 1118 may operate in accordance with one or more applicable standards in any version.

[0122] In various implementations, the display 1120 may include any television type monitor or display. The display 1120 may include, for example, a computer display screen, a touch screen display, a video monitor, a television-like device, and / or a television. The display 1120 may be digital and / or analog. In various implementations, the display 1120 may be a holographic display. Moreover, the display 1120 may be a transparent surface that can receive a visual projection. Such a projection may convey information, images, and / or objects in various forms. For example, such a projection may be a visual overlay for a mobile augmented reality (MAR) application. Under the control of one or more software applications 1116, the platform 1102 may display a user interface 1122 on the display 1120.

[0123] In various implementations, content services device(s) 1130 may be hosted by any national, international, and / or independent service and thus accessible to platform 1102 via, for example, the Internet. Content services device(s) 1130 may be coupled to platform 1102 and / or to display 1120. Platform 1102 and / or content services device(s) 1130 may be coupled to network 1160 to communicate (e.g., send and / or receive) media information to and from network 1160. Content delivery device(s) 1140 may also be coupled to platform 1102 and / or to display 1120.

[0124] In various implementations, content services device(s) 1130 may include a cable box, a personal computer, a network, a telephone, an Internet-enabled device or appliance capable of delivering digital information and / or content, and any other similar device capable of communicating content unidirectionally or bidirectionally between content providers and platform 1102 and / or display 1120 via network 1160 or directly. It will be appreciated that content may be communicated unidirectionally and / or bidirectionally to and from any of the components in system 1100 and content providers via network 1160. Examples of content may include any media information, including, for example, video, music, medical and gaming information, and the like.

[0125] (One or more) content service device 1130 can receive content such as cable television programs, including media information, digital information, and / or other content. Examples of content providers may include any cable or satellite television or radio or Internet content providers. The examples provided are not meant to limit implementations according to the present disclosure in any way.

[0126] In various implementations, platform 1102 may receive control signals from a navigation controller 1150 having one or more navigation features. The navigation features of navigation controller 1150 may be used to interact with, for example, user interface 1122. In various embodiments, navigation controller 1150 may be a pointing device that may be a computer hardware component (specifically, a human interface device) that allows a user to input spatial (e.g., continuous and multi-dimensional) data into a computer. Many systems (e.g., graphical user interfaces (GUIs)) and televisions and monitors allow a user to control and provide data to a computer or television using physical gestures.

[0127] The movements of the navigation features of navigation controller 1150 may be replicated on a display (e.g., display 1120) through the movements of a pointer, cursor, focus ring, or other visual indicators displayed on the display. For example, under the control of software applications 1116, the navigation features located on navigation controller 1150 may be mapped to virtual navigation features displayed, for example, on user interface 1122. In various embodiments, navigation controller 1150 may not be a separate component, but may be integrated into platform 1102 and / or display 1120. The present disclosure, however, is not limited to the elements or in the context shown or described herein.

[0128] In various implementations, drivers (not shown) may include technology for enabling a user to instantly turn platform 1102 (such as a television) on and off with the touch of a button after initial boot-up (e.g., when enabled). Program logic may allow platform 1102 to stream content to a media adapter or other content service device(s) 1130 or content delivery device(s) 1140 even when the platform is "off." Additionally, chipset 1105 may include hardware and / or software support for 5.1 surround sound audio and / or high definition 7.1 surround sound audio, for example. Drivers may include graphics drivers for integrated graphics platforms. In various embodiments, the graphics driver may include a peripheral component interconnect (PCI) Express graphics card.

[0129] In various implementations, any one or more of the components shown in system 1100 may be integrated. For example, platform 1102 and (one or more) content service device(s) 1130 may be integrated, or platform 1102 and (one or more) content delivery device(s) 1140 may be integrated, or, for example, platform 1102, (one or more) content service device(s) 1130, and (one or more) content delivery device(s) 1140 may be integrated. In various embodiments, platform 1102 and display 1120 may be an integrated unit. For example, display 1120 and (one or more) content service device(s) 1130 may be integrated, or display 1120 and (one or more) content delivery device(s) 1140 may be integrated. These examples are not meant to limit the present disclosure.

[0130] In various embodiments, the system 1100 may be implemented as a wireless system, a wired system, or a combination of the two. When implemented as a wireless system, the system 1100 may include components and interfaces suitable for communicating on a wireless shared medium, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc. Examples of wireless shared media may include a portion of a wireless spectrum, such as an RF spectrum, etc. When implemented as a wired system, the system 1100 may include components and interfaces suitable for communicating on a wired communication medium, such as an input / output (I / O) adapter, a physical connector for connecting an I / O adapter to a corresponding wired communication medium, a network interface card (NIC), a disk controller, a video controller, an audio controller, etc. Examples of wired communication media may include wires, cables, metal leads, printed circuit boards (PCBs), backplanes, switch fabrics, semiconductor materials, twisted pairs, coaxial cables, optical fibers, etc.

[0131] Platform 1102 can establish one or more logical or physical channels to transmit information. The information can include media information and control information. Media information can refer to any data representing content for a user. Examples of content can include, for example, data from: voice conversations, video conferences, streaming video, electronic mail ("email") messages, voice mail messages, alphanumeric symbols, graphics, images, videos, text, etc. Data from voice conversations can be, for example, speech information, silent periods, background noise, comfort noise, tones, etc. Control information can refer to any data representing commands, instructions, or control words for an automated system. For example, control information can be used to route media information through a system, or to instruct a node to process media information in a predetermined manner. However, embodiments are not limited to Fig.11 The elements or situations shown or described in FIG.

[0132] As described above, system 1100 may be embodied in varying physical styles or form factors. Fig.12An example small form factor device 1200 is shown, arranged in accordance with at least some implementations of the present disclosure. In some examples, the system 1100 can be implemented via the device 1200. In other examples, other systems discussed herein, or portions thereof, can be implemented via the device 1200. In various embodiments, for example, the device 1200 can be implemented as a mobile computing device with wireless capabilities. For example, a mobile computing device can refer to any device with a processing system and a mobile power source or power supply (e.g., one or more batteries).

[0133] Examples of mobile computing devices may include a personal computer (PC), a laptop computer, an ultralaptop computer, a tablet computer, a touchpad, a portable computer, a handheld computer, a palmtop computer, a personal digital assistant (PDA), a cellular phone, a combination cellular phone / PDA, a smart device (e.g., a smart phone, a smart tablet, or a smart mobile TV), a mobile Internet device (MID), a messaging device, a data communication device, a camera (e.g., a point-and-shoot camera, a super zoom camera, a digital single-lens reflex (DSLR) camera), and the like.

[0134] Examples of mobile computing devices may also include computers that are arranged to be worn by a person, such as a wrist computer, finger computer, ring computer, eyeglass computer, belt-clip computer, arm-band computer, shoe computers, clothing computers, and other wearable computers. In various embodiments, for example, a mobile computing device may be implemented as a smart phone capable of executing computer applications as well as voice communications and / or data communications. Although some embodiments may be described with a mobile computing device implemented as a smart phone by way of example, it will be appreciated that other embodiments may also be implemented using other wireless mobile computing devices. The embodiments are not limited to this context.

[0135] like Fig.12As shown, device 1200 may include a housing having a front 1201 and a back 1202. Device 1200 includes a display 1204, an input / output (I / O) device 1206, a camera 1215, a camera 1205, and an integrated antenna 1208. Device 1200 may also include a navigation feature 1212. I / O device 1206 may include any appropriate I / O device for inputting information into a mobile computing device. Examples of I / O device 1206 may include an alphanumeric keyboard, a numeric keypad, a touch pad, input keys, buttons, switches, microphones, speakers, voice recognition devices and software, etc. Information may also be input into device 1200 via a microphone (not shown), or may be digitized by a voice recognition device. As shown, device 1200 may include a camera 1205 and a flash 1210 integrated into the back 1202 (or elsewhere) of device 1200 and a camera 1215 integrated into the front 1201 of device 1200. In some embodiments, either or both of cameras 1215, 1205 may be movable relative to display 1204. For example, camera 1215 and / or camera 1205 may be components of an imaging module or pipeline for causing color image data to be processed into streaming video that is output to display 1204 and / or transmitted remotely from device 1200 via antenna 1208. For example, camera 1215 may capture an input image, and an eye contact corrected image may be provided to display 1204 and / or transmitted remotely from device 1200 via antenna 1208.

[0136] Various embodiments can be implemented using hardware elements, software elements, or a combination of the two. The example of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc. The example of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, processes, software interfaces, application program interfaces (APIs), instruction sets, computing codes, computer codes, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and / or software elements may vary according to any number of factors, such as desired computing rates, power levels, thermal tolerance, processing cycle budgets, input data rates, output data rates, memory resources, data bus speeds, and other design or performance constraints.

[0137] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium, which represent various logic within a processor and, when read by a machine, cause the machine to manufacture logic for performing the techniques described herein. Such representations (referred to as IP cores) may be stored on a tangible machine-readable medium and provided to various customers or manufacturing facilities to be loaded into a manufacturing machine that actually manufactures the logic or processor.

[0138] Although certain features set forth herein have been described with reference to various implementations, the description is not intended to be interpreted in a limiting sense. Therefore, various modifications and other implementations of the implementations described herein that are obvious to those skilled in the art to which the present disclosure belongs are considered to fall within the spirit and scope of the present disclosure.

[0139] It will be appreciated that the embodiments are not limited to the described embodiments, but can be practiced with modifications and changes without departing from the scope of the appended claims. For example, the above embodiments may include a specific combination of features. However, the above embodiments are not limited in this respect, and in various implementations, the above embodiments may include: only a subset of such features, a different order of such features, a different combination of such features, and / or additional features other than those features explicitly listed. Therefore, the scope of the embodiments should be determined with reference to the appended claims and the full scope of equivalents to which such claims are assigned.

Claims

1. A method comprising: estimating one or more facial parameters from a first image using the trained model, the one or more facial parameters indicating one or more features of a face captured by the first image; generating a three-dimensional model representing the face of the person from the one or more face parameters; generating a rendered three-dimensional model by rendering the three-dimensional model using the one or more additional facial parameters; and A second image is generated from the rendered three-dimensional model.

2. The method of claim 1, wherein: The one or more facial parameters include facial parameters indicating a shape, posture or expression of the face.

3. The method of claim 1, wherein: The one or more additional facial parameters are estimated using the third image.

4. The method of claim 1, wherein: The three-dimensional model is a three-dimensional deformable model.

5. The method of claim 1, wherein: The rendered three-dimensional model represents an expression or posture different from that of the three-dimensional model.

6. The method of claim 1, wherein: The second image includes a human face having a different expression or posture than the human face captured in the first image.

7. The method of claim 3, wherein: The characteristics of the person having the face captured by the first image are maintained in the third image.

8. A computer-readable medium comprising instructions which, when executed by at least one processor, cause the processor to perform the method of any one of claims 1 to 7.

9. A system comprising: A computer processor for executing computer program instructions; as well as A computer readable memory storing computer program instructions, wherein the computer program instructions can be used by the computer processor to execute the method according to any one of claims 1 to 7.

10. An apparatus comprising means for performing the method according to any one of claims 1 to 7.

11. A computer program product comprising instructions which, when executed by at least one processor, cause the processor to perform the method of any one of claims 1 to 7.

Images