Face tracking based on spatio-temporal aggregation and rigidity prior

Abstract

A computer vision system can extract feature maps from various frames in a video (e.g., monocular video). The frames can include a current frame and one or more historical frames. The computer vision system can aggregate a feature map extracted from the current frame (“current feature map”) with a feature map extracted from each historical frame (“historical feature map”). The computer vision system can use a decoder to determine blend shape coefficients from the aggregated feature maps. The blend shape coefficients can represent a predicted facial expression. Additionally, the computer vision system can use another decoder to determine pose coefficients from the aggregated feature maps, e.g., after applying a semantics-based rigidity prior module to the aggregated feature maps. The pose coefficients can represent a predicted head pose. The computer vision system can generate a facial animation from the blend shape coefficients and the pose coefficients.

Description

[0001] Cross-reference to related applications This application claims the benefit of the entire contents of international application No. PCT / CN2024 / 083753, entitled “Face tracking based on spatiotemporal aggregation and semantically aware rigid priors”, filed on March 26, 2024, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0002] This disclosure relates generally to computer vision, and more specifically to face tracking based on spatiotemporal aggregation and rigid prior. Background Technology

[0003] The past decade has witnessed the rapid development of data processing technologies based on artificial intelligence (AI), particularly those based on deep neural networks (DNNs), such as convolutional neural networks (CNNs). DNNs are widely used in computer vision, speech recognition, image and video processing, primarily due to their ability to surpass human accuracy. Computer vision tasks include methods for acquiring, processing, analyzing, or understanding visual images or videos to generate information, such as face tracking (tracking facial expressions, tracking facial movements, etc.) and motion detection. Attached Figure Description

[0004] The various embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. For ease of description, similar reference numerals denote similar structural elements. In the accompanying figures, embodiments are shown by way of example rather than limitation.

[0005] Figure 1 These are example computer vision systems according to various embodiments.

[0006] Figure 2 It is a memory-based spatiotemporal feature aggregation based on various embodiments.

[0007] Figure 3 These are example read controllers based on various embodiments.

[0008] Figure 4 This is face tracking using a hybrid shape decoder and pose decoder according to various embodiments.

[0009] Figure 5 These are example decoder modules according to various embodiments.

[0010] Figure 6 These are example rigid prior modules based on various embodiments.

[0011] Figure 7 These are example convolutions based on various implementations.

[0012] Figure 8 These are example DNNs based on various embodiments.

[0013] Figure 9 It is an AI-based visual recognition environment according to various embodiments.

[0014] Figure 10 This is a flowchart illustrating a face tracking method according to various embodiments.

[0015] Figure 11 This is a block diagram of an example computing device according to various embodiments. Detailed Implementation

[0016] Many existing methods for 3D face tracking from monocular video lack sufficient accuracy and robustness due to the depth ambiguity and self-occlusion of the monocular input. Some face tracking methods use DNNs to predict 3D facial animation parameters directly from monocular video. These methods typically use a 2D face detector to crop facial regions in each video frame, then train a 3D face tracking network to regress parameters frame-by-frame, and finally use a Kalman filter to smooth the parameters across all video frames. These DNN-based methods have limitations. For example, they require a Kalman filter as a post-processing step to achieve time-consistent 3D face tracking results, leading to additional computational costs during inference. Furthermore, they often lack rigid stability, which is crucial in monocular 3D face tracking due to the depth ambiguity of the monocular input.

[0017] Embodiments of this disclosure can improve at least some of the aforementioned challenges and problems by providing a face tracking method (e.g., facial expression tracking, head pose tracking, etc.) based on spatiotemporal aggregation and semantically aware rigid priors. For example, spatiotemporal features extracted from multiple frames can be aggregated to predict facial expressions (e.g., smiling, frowning, rolling eyes, appearing bored, appearing interested, etc.). By fusing features from multiple frames, tracking accuracy can be improved. Moreover, since a Kalman filter is not required, tracking efficiency can be improved by reducing computational costs during inference. The aggregated spatiotemporal features can also be combined with semantically aware rigid priors to predict head pose. Semantically aware rigid priors can achieve rigid stability and reduce or even eliminate depth shifts during face tracking, thus improving the accuracy of head pose prediction.

[0018] In various embodiments of this disclosure, a computer vision system can extract feature maps from individual frames in a video (e.g., monocular video). These frames may include a current frame and one or more historical frames. The current frame may be the frame with the most recent timestamp, while each historical frame has an earlier timestamp and is temporally preceding the current frame in the video. The computer vision system may aggregate the feature map extracted from the current frame (“current feature map”) with the feature maps extracted from each historical frame (“historical feature map”). In one example, the computer vision system may use one or more transformer blocks to generate intermediate feature maps based on the current feature map and (one or more) historical feature maps. The output of (one or more) transformer blocks may be further concatenated with the feature maps to generate an aggregated feature map. The transformer block may include a transformer encoder or a transformer decoder.

[0019] Computer vision systems can use a decoder to determine blendshape coefficients from an aggregated feature map. These blendshape coefficients can represent the predicted facial expression. The blendshape can be a model of the facial expression (e.g., a 3D model). For example, the blendshape can be an approximate semantic parameterization of the facial expression. Furthermore, computer vision systems can generate rigid transformations from the aggregated feature map by applying a semantically based rigid prior module to it. The computer vision system can use these rigid transformations to predict head pose, for example, by using another decoder. This other decoder can output pose coefficients, such as rotation and translation parameters. The blendshape coefficients and pose coefficients can be used to reconstruct faces in videos. For example, 3D facial animations can be generated.

[0020] For illustrative purposes, specific figures, materials, and configurations have been set forth to provide a thorough understanding of the illustrative implementation. However, it will be apparent to those skilled in the art that this disclosure may be practiced without specific details, and / or may be practiced using only some of the aspects described. In other instances, well-known features have been omitted or simplified so as not to obscure the illustrative embodiments.

[0021] Furthermore, reference has been made to the accompanying drawings, which form part of this disclosure, and practical embodiments are illustrated in the drawings by way of illustration. It should be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of this disclosure. Therefore, the following detailed description should not be construed as limiting.

[0022] Various operations can be described sequentially as a plurality of discrete actions or operations in a manner most conducive to understanding the claimed subject matter. However, the order of description should not be construed as implying that these operations must depend on the order. In particular, these operations may not be performed in the order presented. The described operations may be performed in a different order than in the described embodiments. Various additional operations may be performed, or the described operations may be omitted in additional embodiments.

[0023] For the purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" refers to (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" refers to (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). When used to refer to a measurement range, the term "between" includes the endpoints of the measurement range.

[0024] This description uses the phrases "in one embodiment" or "in an embodiment," both of which can refer to one or more of the same or different embodiments. Terms such as "comprising," "including," "having," etc., used with respect to embodiments of this disclosure are synonyms. This disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to interpret various features of the drawings; however, these terms are merely for ease of discussion and do not imply any desired or required direction. The drawings are not necessarily drawn to scale. Unless otherwise stated, the use of ordinal adjectives such as "first," "second," and "third" to describe common objects indicates only different instances of the similar objects referred to and is not intended to imply that the objects described must be arranged in a given order, whether temporally, spatially, in rank, or otherwise.

[0025] In the following detailed description, terms commonly used by those skilled in the art will be used to describe various aspects of the illustrative implementations in order to convey the substance of their work to others skilled in the art.

[0026] The terms “substantially,” “close to,” “approximately,” “near,” and “about” generally refer to values ​​within + / - 20% of the target value as described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between elements, generally refer to values ​​within + / - 5-20% of the target value as described herein or known in the art.

[0027] Furthermore, the terms “comprising,” “including,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion. For example, a method, process, apparatus, or DNN accelerator that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such a method, process, apparatus, or DNN accelerator. Additionally, the term “or” refers to inclusive “or” rather than exclusive “or.”

[0028] The systems, methods, and apparatuses disclosed herein are innovative in several ways, but none of them alone is responsible for all the desired properties disclosed herein. Details of one or more implementations of the subject matter described herein are set forth in the following description and figures.

[0029] Figure 1 An example computer vision system 100 according to various embodiments is illustrated. The computer vision system 100 includes a face tracking module 110, a training module 130, a verification module 140, and a data storage unit 150. In other embodiments, the computer vision system 100 may include alternative configurations, different, or additional components. Furthermore, the functionality implemented by the components of the computer vision system 100 may be performed by other components included in the computer vision system 100 or by other systems.

[0030] A face tracking module 110 tracks faces in a video. The face tracking module 110 includes a DNN 160, a feature aggregation module 170, a shape blending module 180, and a pose module 190. The DNN 160 can extract features from video frames. In some embodiments, the DNN 160 can extract feature maps from frames. The DNN 160 can operate in an online mode. For example, a sequence of frames in a video can be sequentially input into the DNN 160, and the DNN 160 can generate a feature map from each frame as it is received. The order in which the DNN 160 processes frames can correspond to the order in which the frames are arranged in the video. The DNN 160 can include multiple layers, such as an input layer, one or more hidden layers, and an output layer. One or more layers in the DNN 160 can be convolutional layers.

[0031] The feature maps generated by a DNN 160 can be 2D or 3D tensors. A tensor is a data structure with multiple elements in one or more dimensions. Examples of tensors include vectors (one-dimensional (1D) tensors), matrices (2D tensors), 3D tensors, four-dimensional (4D) tensors, and even tensors with higher dimensions. The dimensions of a tensor can correspond to axes, such as axes in a coordinate system. Dimensions can be measured by the number of data points along an axis. The dimensions of a tensor can define the shape of the tensor. A DNN layer can receive one or more input tensors and compute an output tensor based on those one or more input tensors. Taking a convolutional layer as an example, the input tensors include activation tensors (also called "input feature maps (IFM)" or "input activation tensors") and weight tensors, where the activation tensors include one or more activation values ​​(also called "input elements"). Weight tensors can be kernels (2D weight tensors), filters (3D weight tensors), or filter banks (4D weight tensors). In a convolutional layer, a convolution operation can be performed on the input activation tensor and the weight tensor to compute the output activation tensor.

[0032] Feature aggregation module 170 receives features generated by DNN 160 and aggregates the features. In some embodiments, feature aggregation module 170 can generate an aggregated feature map for each of multiple frames. To generate an aggregated feature map for a frame, feature aggregation module 170 aggregates the feature map extracted by DNN 160 from that frame with one or more other feature maps extracted by DNN 160 from one or more other frames. This frame may be referred to as the current frame. The feature map extracted from this frame may be referred to as the current feature map. One or more other frames may precede this frame in time and may be referred to as one or more historical frames or one or more previous frames. One or more other feature maps extracted from one or more other frames may be referred to as one or more historical feature maps or one or more previous feature maps. Feature aggregation module 170 can aggregate the current feature map with one or more previous feature maps by using one or more transformer decoders. The output of one or more transformer decoders may be further accumulated with the current feature map to generate an aggregated feature map. Further details about feature aggregation module 170 will be provided below. Figure 2 supply.

[0033] The blending shape module 180 predicts facial expressions based on the aggregated feature map generated by the feature aggregation module 170. In some embodiments, blending shapes are used to represent facial expressions. Blending shapes can represent facial expressions such as smiling, frowning, etc. In some embodiments, the blending shape can be a 3D representation of the facial expression (e.g., a 3D model). In some embodiments, the blending shape module 180 may include one or more transformer decoders that can generate one or more blending shape parameters based on the aggregated feature map. The one or more blending shape parameters can be encoded or used to generate a predicted blending shape that represents a prediction of the facial expression shown in the current frame. The one or more blending shape parameters can be deformable models.

[0034] The pose module 190 predicts pose based on the aggregated feature map generated by the feature aggregation module 170. In some embodiments, the pose module 190 may include one or more transformer decoders that can generate one or more pose parameters based on the aggregated feature map. The one or more pose parameters may encode a prediction of the head pose shown in the current frame. The one or more pose parameters may include one or more translation parameters (e.g., global translation parameters) or one or more rotation parameters (e.g., global rotation parameters). Translation parameters may encode the predicted translation of the head. Rotation parameters may encode the predicted rotation of the head.

[0035] In some embodiments, the pose module 190 includes a semantically aware rigidity prior, which can improve the accuracy of pose prediction. The semantically aware rigidity prior can be used by a DNN to compute a reweighted mask based on a segmentation probability map of the face. The aggregated feature map can be multiplied with the reweighted mask before being processed by one or more transformer decoders in the pose module 190. By using the semantically aware rigidity prior, rigid regions of the face (e.g., forehead, nose, etc.) can be enhanced, which can reduce or even eliminate depth drift in rigid regions and contribute to more accurate prediction of rigid motion.

[0036] The face tracking module 110 aggregates spatiotemporal features to achieve accurate and robust 3D face tracking, rather than performing frame-by-frame prediction. Since rigidity can be important in real-time monocular face tracking tasks with head pose ambiguity caused by depth ambiguity, a semantically aware rigidity prior introduced before the transformer decoder can achieve rigidity during 3D face tracking and improve head pose tracking accuracy. Compared to currently available face tracking methods, the face tracking module 110 provides more accurate and efficient 3D face tracking.

[0037] Animation generator 120 generates facial animation based on facial tracking parameters (e.g., blend shape parameters and pose parameters) generated by facial tracking module 110. In some embodiments, animation generator 120 can modify one or more blend shape templates of a face using the facial tracking parameters determined by facial tracking module 110. The blend shape template can represent a specific facial expression. This specific facial expression can be a reference facial expression. In one example, the reference facial expression can be a neutral facial expression, such as a facial expression indicating no strong emotion. Animation generator 120 can generate a new blend shape representing a predicted facial expression based on the blend shape parameters and one or more blend shape templates. Animation generator 120 can also modify the pose of the blend shape based on pose parameters. In some embodiments, animation generator 120 generates a sequence of facial animations for a frame sequence processed by facial tracking module 110. Facial animation can be used in various applications, such as digital humans, video conferencing, movies, games, etc.

[0038] Training module 130 trains face tracking module 110, such as the DNN in face tracking module 110. The DNN in face tracking module 110 may include DNN 160, a transformer decoder in feature aggregation module 170, a transformer decoder in shape blending module 180, a transformer decoder in pose module 190, a CNN in pose module 190, etc. Training module 130 may modify the internal parameters of face tracking module 110 based on the ground truth labels of the training samples and the output generated by face tracking module 110 through processing the training samples. Training samples may include video frames. The ground truth labels of the training samples may include empirical or known face tracking parameters encoding facial expressions and poses shown in the video frames. Training samples may include frame sequences. For each training sample, training module 130 may sample a batch of consecutive frames from the training dataset. Each frame. Training module 130 can also extract and store the previous frames. The features of each frame are used to calculate the loss based on the prediction of the last frame.

[0039] In some embodiments, the training module 130 modifies the internal parameters of the face tracking module 110 to minimize the error between the labels of the training samples generated by the face tracking module 110 and the ground truth labels. In some embodiments, the training module 130 uses a cost function or a loss function to minimize this error. For example, the training module 130 may use mean squared error (MSE) loss on a mixture of shape parameters, rotation parameters, and translation parameters. The MSE loss can be expressed as: in This represents the predicted mixed shape parameters. Indicates the reference truth mixed shape parameter, Indicates the predicted rotation parameters, Represents the rotation parameters of the reference truth. This represents the predicted translation parameters. Indicates the reference truth value translation parameter. It indicates a loss.

[0040] In some embodiments, the training module 130 may also form a validation dataset for validating the performance of the face tracking module 110 after training by the validation module 140. The validation dataset may include validation samples and ground truth labels for the validation samples. The validation dataset may include samples different from those used in the training dataset for training the face tracking module 110. In one embodiment, a portion of the training dataset may be used for initial training of the face tracking module 110, while the remainder of the training dataset may be reserved as a validation subset used by the validation module 140 to validate the performance of the face tracking module 110. A portion of the training dataset that does not include the validation subset may be used to train the face tracking module 110.

[0041] Training module 130 also determines hyperparameters for training face tracking module 110. Hyperparameters are variables that specify the training process. Hyperparameters differ from parameters internal to face tracking module 110 (e.g., weights in convolutional layers, weights in the transformer decoder, etc.). In some embodiments, hyperparameters include variables that determine the architecture of at least a portion of face tracking module 110, such as the number of hidden layers in a DNN 160. Hyperparameters also include variables that determine how face tracking module 110 is trained, such as batch size, number of epochs, etc. Batch size defines the number of training samples to be processed before updating the parameters of face tracking module 110. Batch size is the same as or smaller than the number of samples in the training dataset. The training dataset can be divided into one or more batches. Number of epochs defines the number of times the entire training dataset is forward and backward traversed through the entire network. Number of epochs defines the number of times the deep learning algorithm processes the entire training dataset. An epoch means that each training sample in the training dataset has a chance to update the internal parameters of face tracking module 110. An epoch can include one or more batches. The number of rounds can be 15, 150, 500, 1500 or more.

[0042] Training module 130 can define the architecture of face tracking module 110 (or a part of face tracking module 110, such as a DNN in face tracking module 110), for example, based on some hyperparameters. The architecture of the DNN may include an input layer, an output layer, and multiple hidden layers. The input layer may include a tensor (e.g., a multidimensional array) specifying the properties of the IFM, such as the height, width, and depth of the IFM (e.g., the number of channels in the IFM). The output layer includes the labels of the objects in the input layer. Hidden layers are layers between the input and output layers. Hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully connected layers, layer normalization layers, self-attention layers, cross-attention layers, softmax layers, or logistic layers, etc.

[0043] Training module 130 can train face tracking module 110 for a predetermined number of rounds. The number of rounds is a hyperparameter that defines how many times the deep learning algorithm processes the entire training dataset. One round means that each sample in the training dataset has a chance to update the internal parameters of face tracking module 110. After training module 130 has completed the predetermined number of rounds, training module 130 can stop updating the internal parameters of face tracking module 110, and face tracking module 110 is considered trained.

[0044] The validation module 140 validates the accuracy of the face tracking module 110 after it has been trained. In some embodiments, the validation module 140 inputs samples from a validation dataset into the face tracking module 110 and uses the output of the face tracking module 110 to determine model accuracy. In some embodiments, the validation dataset may consist of some or all of the samples from the training dataset. Alternatively, the validation dataset may include additional samples beyond the training set. In some embodiments, the validation module 140 may determine an accuracy score that measures the precision, recall, or a combination of precision and recall of the DNN. The validation module 140 can use the following metrics to determine the accuracy score: Precision = TP / (TP+FP) and Recall = TP / (TP+FN), where accuracy can be the number of correctly predicted instances (TP or true positives) by the face tracking module 110 divided by the total number of predicted instances (TP+FP or false positives), and recall can be the number of correctly predicted instances (TP) by the face tracking module 110 divided by the total number of objects with the attribute in question (TP+FN or false negatives). The F-score (F-score = 2*PR / (P+R)) unifies precision and recall into a single metric.

[0045] The verification module 140 can compare the accuracy score with a threshold score. In an example where the verification module 140 determines that the accuracy score is below the threshold score, the verification module 140 instructs the training module 130 to retrain the face tracking module 110. In one embodiment, the training module 130 can iteratively retrain the face tracking module 110 until a stopping condition is met, such as an accuracy measurement indicating that the face tracking module 110 may be accurate enough, or that it has already been trained multiple times.

[0046] Data repository 150 stores data received, generated, used, or otherwise associated with the computer vision system 100. For example, data repository 150 stores datasets used by training module 130 and validation module 140. Data repository 150 may also store data generated by training module 130 and validation module 140, such as hyperparameters used to train face tracking module 110, intrinsic parameters of face tracking module 110, etc. As another example, data repository 150 may store video frames to be processed by face tracking module 110 to perform face tracking tasks. Data repository 150 may also store the outputs of face tracking module 110 or its components, such as feature maps, blending shape parameters, pose parameters, segmentation probability maps, etc. Figure 1 In one embodiment, the data repository 150 is a component of the computer vision system 100. In other embodiments, the data repository 150 may be located outside the computer vision system 100 and communicate with the computer vision system 100 via a network.

[0047] Figure 2 Memory-based spatiotemporal feature aggregation according to various embodiments is illustrated. Figure 2 The memory-based spatiotemporal feature aggregation is performed by feature aggregation module 210, which receives the output of backbone network 220. Backbone network 220 can be... Figure 1 An example of DNN 160. (e.g.) Figure 2 As shown, backbone network 220 receives video 201 comprising a sequence of frames 202 (referred to individually as "frame 202"). Video 201 captures one or more faces, such as a human face or the face of another type of object. Video 201 may be monocular video. Frames 202 may be arranged in chronological order. Frames 202 with earlier timestamps may be arranged before frames 202 with later timestamps. In some embodiments, backbone network 220 and feature aggregation module 210 may operate in an online mode. For example, backbone network 220 may receive each frame 202 as it is displayed. The most recently received frame 202 by backbone network 220 may be the current frame, while the other frames 202 arranged before the current frame are one or more historical frames.

[0048] Each frame 202 received by the backbone network 220 is an input to the backbone network 220. The backbone network 220 may be a DNN comprising multiple layers, which may include one or more convolutional layers. The backbone network 220 uses these layers to process the frame 202 and generate feature maps. The feature maps may be 2D or 3D tensors, which include data points (e.g., activation values) representing features extracted by the backbone network 220 from the corresponding frame 202. The feature map of each historical frame is referred to as the previous feature map 203H. The feature map of the current frame is referred to as the current feature map 203P. The (one or more) historical feature maps 203H and the current feature map 203P are collectively referred to as "feature map 203".

[0049] The feature aggregation module 210 receives the feature map 203 from the backbone network 220 and generates spatiotemporal aggregated features based on the feature map 203. The feature aggregation module 210 can be... Figure 1 Example of the feature aggregation module 170. (e.g.) Figure 2 As shown, the feature aggregation module 210 includes a write controller 230, a buffer 240, and a read controller 250. In other embodiments, the feature aggregation module 210 may include alternative configurations, different or additional components. Furthermore, the functions implemented by the components of the feature aggregation module 210 may be performed by other components included in the feature aggregation module 210 or by other modules or systems.

[0050] The write controller 230 can write the feature map 203 generated by the backbone network 220 to the buffer 240. In some embodiments, the write controller 230 can be composed of a buffer with a maximum length of The queue implementation ensures that no more than 100 bytes are stored in buffer 240 at a time. Feature maps 203. Storing a certain number of historical frames can help with effective and efficient face tracking because changes in expression and pose can present smooth transitions in 3D face tracking. Frames 202 closer to the current time can contain more useful information, while frames 202 older than a specific time window are unlikely to help with the prediction of the current frame and may even introduce noise into the prediction. By using a queue of finite length, feature maps 203 older than a specific time window can be removed from the buffer 240, while feature maps 203 within the time window are retained and stored. This ensures the accuracy of the prediction and mitigates the accumulation of errors during the prediction process.

[0051] The read controller 250 reads the feature map 203 stored in the buffer 240. The read controller 250 can aggregate the current feature map 203C with one or more previous feature maps 203P. This aggregation can be at least part of a spatiotemporal aggregation. In some embodiments, the read controller 250 includes one or more transformer decoders that perform the aggregation. In one example, the transformer decoder(s) can aggregate the current feature map 203P (which may be represented as...) ) as the query, and take (one or more) previous feature maps 203P (which can be represented as ( ) serves as the key and value. (One or more) transformer decoders can output intermediate feature maps.

[0052] Then, the intermediate feature map is added to the current feature map 203C by the concatenator 260 via a skip connection. This skip connection can be represented as: in, Indicates (one or more) converter decoders, This represents the spatiotemporal aggregation feature map 204.

[0053] Figure 3 Example read controller 300 according to various embodiments is illustrated. Read controller 300 aggregates features extracted from video frames with different timestamps. In one example, read controller 300 may aggregate features extracted from a time-frame sequence of monocular video to predict facial expressions or head poses captured in one of the frames. Read controller 300 may be... Figure 2 Example of reading controller 250.

[0054] like Figure 3 As shown, the read controller 300 includes a converter decoder 310, referred to solely as "converter decoder 310". Each converter decoder 310 includes multiple layers, such as a self-attention layer 320, a layer normalization layer 330, a cross-attention layer 340, another normalization layer 350, a feedforward layer 360, and another normalization layer 370. In other embodiments, the converter decoder 310 may include different, fewer, or more layers. The read controller 300 may include different numbers of converter decoders 310.

[0055] Self-attention layer 320 or cross-attention layer 340 may have an attention function. In some embodiments, the attention function may map one or more queries and one or more key-value pairs to an output, where the query, each key, each value, or the output may be a tensor, such as a vector. The output may be computed as a weighted sum of values. The weight assigned to each value may be computed by a compatibility function of the query and its corresponding key. Layer normalization layer 330 may connect all positions with a constant number of sequentially performed operations. Self-attention, sometimes called internal attention, may be an attention mechanism that associates different positions of a single sequence to compute a representation of that sequence. Self-attention layer 320 may allow each position in the corresponding transformer decoder 310 to attend to all positions in the transformer decoder 310 up to and including that position.

[0056] like Figure 3 As shown, the self-attention layer 320 receives vector 302C. Vector 302C is generated by flattening the current feature map 301C, which can be generated from the current frame in the video. In some embodiments, the current feature map 301C can be a 2D or 3D tensor, while vector 302C can be a 1D tensor. The data points in the current feature map 301C can be rearranged such that all data points are in one dimension to generate vector 302C. In one example, the current feature map 301C can have three dimensions: X, Y, and Z, where the Z dimension can be a channel dimension. The data points of the current feature map 301C can be rearranged so that all data points are in one of the X, Y, and Z dimensions to form vector 302C. Vector 302C is input into the self-attention layer 320 and can be used as a query, key, and value for the self-attention function. In some embodiments, the self-attention function in the self-attention layer 320 can be represented as: in These represent the query, key, and value, respectively. Represents the self-attention function; This represents the Softmax function.

[0057] The output of the self-attention layer 320 is further accumulated by the splicer 325 with vector 302C. The spliced ​​result is input to the layer normalization layer 330. The layer normalization layer 330 applies layer normalization to the output of the splicer 325. The layer normalization layer 330 can normalize its input (i.e., the output of the splicer 325) across the feature dimension (rather than the batch dimension). The feature dimension can be the dimension of vector 302C or vector 302P.

[0058] In some embodiments, the layer normalization operation may include a series of calculations. Layer normalization layer 330 may include a mean calculation, which can be represented as... ,in This represents the data elements in the input tensor. It can be the position index of a data element in one of the spatial dimensions. It can be the position index of a data element in another spatial dimension. It can be the position index of a data element in the channel dimension. The output of the mean calculation can be a 2D matrix. The mean calculation can be a channel-wise reduction operation. Layer normalization layer 330 can be achieved, for example, by... Each data element is copied at each output point to process the data, in order to... Convert to 3D tensor .

[0059] Layer normalization layer 330 may also include element-wise subtraction, which can be represented as follows: Layer 330, which is a normalized layer, can also perform variance calculation, which can be expressed as: Layer normalization layer 330 can further include division calculations, represented as... . It can be a 2D tensor. Layer normalization layer 330 can also be achieved, for example, by... Copy each data element at each output point to... Convert to 3D tensor Furthermore, layer normalization layer 330 can have element-wise multiplication, represented as... Layer 330, which is a normalized layer, can be further calculated. and . It can be the output of a layer normalization operation.

[0060] The output of the layer normalization layer 330 (which encodes the flattened current features) can be input to the cross-attention layer 340. The cross-attention layer 340 also receives a vector 302P as another input. Vector 302P is generated by flattening a previous feature map 301P, which can be generated from previous frames in the video. In some embodiments, the previous feature map 301P can be a 2D or 3D tensor, while vector 302P can be a 1D tensor. Data points in the previous feature map 301P can be rearranged such that all data points are in one dimension to generate vector 302P. In one example, each previous feature map 301P can have three dimensions: X, Y, and Z, where the Z dimension can be a channel dimension. Data points in the previous feature map 301P can be rearranged so that all data points are in one of the X, Y, and Z dimensions to form vector 302P. Data points in the same previous feature map can be arranged together, for example, in a manner that is not interrupted by any data points in another previous feature map. In some embodiments, vector 302P and vector 302C are in the same dimension. Vector 302C and vector 302P are in... Figure 3 For illustrative purposes, each element is represented by a series of boxes. The number of boxes in vector 302C or vector 302P does not represent the number of data points in vector 302C or vector 302P. In some embodiments, the number of data points in vector 302C may be equal to the number of data points in the current feature map 301C, and the number of data points in vector 302P may be equal to the total number of data points in all previous feature maps 301P.

[0061] The cross-attention layer 340 has a cross-attention function applied to vectors 302P and 302C. Vector 302P can be used as a query, and vector 302C can be used as a key and value. The cross-attention function can be expressed as: Where σ is the Softmax function; Represents vector 302C; Represents vector 302P; These are the projection matrices for the query, key, and value, respectively.

[0062] The output of the cross-attention layer 340 can be summed with the output of the layer normalization layer 330 by another splicer 345. The output of splicer 345 is input to the layer normalization layer 350 to perform another layer normalization operation in the layer normalization layer 350. The output of the layer normalization layer 350 is input to the feedforward layer 360. In some embodiments, the feedforward layer 360 may include linear transformations and nonlinear transformations. The linear transformation may be combined with a fully connected layer (e.g., ...). Figure 8The operations are the same or similar to those in the fully connected layer 830. The nonlinear transformation can be an activation function. In some embodiments, the feedforward layer 360 can multiply the input with the key to obtain a weight for each key. The feedforward layer 360 can use these weights to compute a weighted sum of values ​​to produce an output.

[0063] The output of feedforward layer 360 can be summed with the output of layer normalization layer 350 by splicer 365. The output of splicer 365 can be fed into layer normalization layer 370, where another layer normalization operation is performed. The output of layer normalization layer 370 can be fed into next transformer decoder 310 for further processing. The output of last transformer decoder 310 can constitute the output of read controller 300, which can be an intermediate feature map representing the aggregation of the current feature map 301C and the previous feature map 301P.

[0064] Figure 4 Face tracking using a hybrid shape decoder 410 and a pose decoder 430 coupled to a rigid prior module 420, according to various embodiments, is illustrated. The hybrid shape decoder 410 receives a feature map 401. The feature map 401 may be an aggregated feature map generated based on a feature map extracted from the current frame and one or more feature maps extracted from one or more previous frames. In some embodiments, the feature map 401 is... Figure 1 The feature aggregation module 170 is generated in the middle.

[0065] The hybrid shape decoder 410 also receives multiple latent representations 402, individually referred to as latent representations 402. Each latent representation 402 can be an implicit hybrid shape latent representation. The latent representation 402 can represent various hybrid shapes corresponding to various facial expressions. Each latent representation 402 can be a learnable latent tensor. A learnable latent tensor can be represented as... ,in It is the number of mixed shapes. This represents the number of channels in the latent representation. Each latent tensor can represent or encode a blended shape after training. The blended shape decoder 410 uses the feature map 401 and the latent representation 402 to predict the facial expression of the face shown in the current frame. In some embodiments, the blended shape decoder 410 may include one or more transformer decoders that process the feature map 401 and the latent representation 402.

[0066] The blend shape decoder 410 generates blend shape parameters 403 (also called "blending shape coefficients"), which can encode the predicted facial expression. The blend shape parameters 403 can be represented as: in .

[0067] Feature map 401 is also input to rigid prior module 420. Rigid prior module 420 can reduce or even eliminate depth offset in feature map 401 to achieve better rigidity stabilization. Rigid prior module 420 can use face segmentation map 404 to generate a rigid transformation based on feature map 401. This rigid transformation is input to pose decoder 430. Pose decoder 430 also receives multiple latent representations 405, individually referred to as latent representations 405. Each latent representation 405 can be an implicit pose latent representation. Each latent representation 405 can be a learnable latent tensor. Latent representations 405 can represent various poses. A latent representation 405 can be represented as... The potential tensor, in which It is the number of attitude parameters. This is the number of channels in the latent representation 405. In some embodiments, the attitude decoder 430 may include one or more transformer decoders that process the rigid transformation and the latent representation 405 to predict head attitude.

[0068] The pose decoder 430 can output pose parameters 406 (also called "pose coefficients") that encode the predicted head pose. The pose parameters 406 can be represented as... In some embodiments, the attitude parameters can be decomposed into a six-degree-of-freedom vector, where rotation is determined by three Euler angles. Parameterization, translation by vector express: in, These are the predicted rotation parameters and the predicted translation parameters, respectively.

[0069] The blend shape parameter 403 and pose parameter 406 can be combined with one or more blend shape templates 407 to generate a 3D facial animation 408. The 3D facial animation 408 illustrates the predicted facial expression and the predicted head pose. In some embodiments, the blend shape template can be represented as... ,in It refers to the number of vertices. Given a specific set of blend shape parameters. The predicted blending shape can be obtained in the following ways: By combining the predicted blending shape and the predicted pose, 3D facial animation 408 can be obtained in the following way: in, In homogeneous coordinates .

[0070] Figure 5An example decoder module 500 according to various embodiments is shown. The decoder module 500 may be... Figure 4 Examples of hybrid shape decoder 410 or pose decoder 430. (e.g.) Figure 5 As shown, the decoder module 500 includes a transformer decoder 510, specifically referred to as "transformer decoder 510". Each transformer decoder 510 includes multiple layers, such as a self-attention layer 520, a layer normalization layer 530, a cross-attention layer 540, another normalization layer 550, a feedforward layer 560, and another normalization layer 570. In other embodiments, the transformer decoder 510 may include different, fewer, or more layers. The decoder module 500 may include other numbers of transformer decoders 510. The transformer decoder 510 can be used with... Figure 3 The converter decoder 310 in the text is the same as or similar to the one in the text.

[0071] like Figure 5 As shown, the self-attention layer 520 receives vector 502. Vector 502 is generated by flattening feature map 501, which can be an aggregated feature map. An example of feature map 501 could be... Figure 1 Feature map 501 is used in the model. In some embodiments, feature map 501 can be a 2D or 3D tensor, while vector 502 can be a 1D tensor. Data points in feature map 501 can be rearranged so that all data points are in one dimension to generate vector 502. In one example, feature map 501 can have three dimensions: X, Y, and Z, where the Z dimension can be a channel dimension. Data points in feature map 501 can be rearranged so that all data points are in one of the X, Y, and Z dimensions to form vector 502. Vector 502 is input into self-attention layer 520 and can be used as a query, key, and value for the self-attention function. In some embodiments, the self-attention function in self-attention layer 520 can be represented as: in These represent the query, key, and value, respectively. Represents the self-attention function; This represents the Softmax function.

[0072] The output of the self-attention layer 520 is further summed with vector 502 by the splicer 525. The spliced ​​result is fed into the layer normalization layer 530. The layer normalization layer 530 applies layer normalization to the output of the splicer 525. The layer normalization layer 530 can normalize its input (i.e., the output of the splicer 525) across the feature dimension (rather than the batch dimension). The feature dimension can be the dimension of vector 502.

[0073] The output of the layer normalization layer 530 can be input to the cross-attention layer 540. The cross-attention layer 540 also receives a latent representation 504 (referred to separately as "latent representation 504") as another input. The latent representation 504 can be an implicit mixed shape latent representation, for example, in an embodiment where the decoder module 500 is used to predict mixed shapes. In other embodiments (e.g., an embodiment where the decoder module 500 is used to predict pose), the latent representation 504 can be an implicit pose latent representation.

[0074] In some embodiments, the cross-attention function is applied to the outputs of latent representation 504 and layer normalization layer 530. In some embodiments, cross-attention layer 540 can use latent representation 504 as a query and the output of layer normalization layer 530 (which may represent aggregated spatiotemporal features) as keys and values. This cross-attention function can be combined with the above. Figure 3 The described cross-attention functions are the same or similar, but have different queries, keys, and values.

[0075] The output of the cross-attention layer 540 can be summed with the output of the layer normalization layer 530 by another splicer 545. The output of splicer 545 is input to the layer normalization layer 550 to perform another layer normalization operation. The output of the layer normalization layer 550 is input to the feedforward layer 560. The feedforward layer 560 can have the same or similar computation as the feedforward layer 360.

[0076] The output of feedforward layer 560 can be summed with the output of layer normalization layer 550 by splicer 565. The output of splicer 565 can be input to layer normalization layer 570, where another layer normalization operation is performed. The output of layer normalization layer 570 can be input to the next transformer decoder 510 for further processing. The output of the last transformer decoder 510 can constitute the output of decoder module 500, which can be a mixture of shape coefficients or attitude coefficients.

[0077] Figure 6 An example rigid prior module 600 according to various embodiments is shown. The rigid prior module 600 receives a segmentation map 602. The segmentation map 602 is generated based on an image 601 capturing a head. Image 601 may be a frame from a video. The segmentation map 602 includes information indicating head segmentation in image 601. For example, the segmentation map 602 may include multiple regions, each region potentially corresponding to a different part of the head. In one example, the segmentation map 602 includes regions corresponding to hair, regions corresponding to the face, regions corresponding to the nose, regions corresponding to glasses, etc. The segmentation map 602 may be a segmentation probability map of the current frame. In some embodiments, the segmentation map 602 may be generated by a facial segmentation model that segments the face into different regions. The segmentation map 602 may be... Figure 4Example of a mid-segmented graph 404.

[0078] Different regions can have different properties, such as different weights. In some embodiments, one or more regions in segmentation graph 602 can be identified or labeled as rigid regions(s). Rigid regions can correspond to parts of the head that have no or minimal deformation or change when the head pose changes. Examples of rigid regions include the forehead, nose, etc. Segmentation graph 602 can be used by rigid prior module 600 to improve the rigidity stability of pose prediction.

[0079] The DNN 610 in the rigid prior module 600 receives the segmentation map 602 as input and outputs a reweighted mask 603. The DNN 610 can be a CNN, an example of which could be... Figure 8 The DNN 800 in the DNN. The reweighted mask 603 can be represented as Its spatial dimension is the same as that of feature map 604. Feature map 604 can be an aggregation of feature maps extracted based on the current frame and feature maps extracted based on previous frames. In some embodiments, feature map 604 is composed of... Figure 1 The feature aggregation module 170 is generated in the middle.

[0080] The multiplier 620 in the rigid prior module 600 multiplies the reweighted mask 603 with the feature map 604 and computes the rigid transformation 605. In some embodiments, the rigid transformation 605 is a tensor having the same spatial size or shape as the feature map 604. The values ​​of data points in the rigid transformation 605 may differ from the values ​​in the feature map 604. By multiplying the feature map 604 with the reweighted mask 603, the rigid region can be enhanced. The rigid transformation 605 can be used to predict head pose. In some embodiments, the rigid transformation 605 can be processed by a pose decoder (e.g., pose decoder 430) to generate pose parameters.

[0081] Figure 7 Example convolutions according to various embodiments are shown. This convolution can be a DNN (e.g., ...). Figure 1 DNN160 in Figure 2 The backbone network 220 in China Figure 6 This refers to a deep learning operation within a convolutional layer of a DNN (e.g., DNN 610). This convolution can extract features from the input. The convolution can be performed on the input tensor 710 and the filter 720 (referred to separately as "filter 720"). The result of the convolution is the output tensor 730. In some embodiments, the convolution is performed by a DNN accelerator.

[0082] exist Figure 7In this embodiment, the input tensor 710 includes activation values ​​(also referred to as "input activation values," "elements," or "input elements") arranged in a 3D matrix. Input elements are data points within the input tensor 710. The input tensor 710 has a spatial size. ,in It is the height of the 3D matrix (i.e., the length along the Y-axis, representing the number of activation values ​​in the column of the 3D matrix for each input channel). It is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of activation values ​​in each row of the 3D matrix for each input channel). This is the depth of the 3D matrix (i.e., the length along the Z-axis, representing the number of input channels). For simplicity and illustration, the input tensor 710 has a spatial size of 7×7×3, meaning it includes three input channels, each with a 7×7 2D matrix. Each input element in the input tensor 710 can be represented by (X, Y, Z) coordinates. In other embodiments, the height, width, or depth of the input tensor 710 may be different.

[0083] Each filter 720 includes weights arranged in a 3D matrix. The values ​​of the weights can be determined by training a DNN. The filter 720 has a spatial size. ,in It is the height of the filter (i.e., the length along the Y-axis, representing the number of weights in each column of the core). It is the width of the filter (i.e., the length along the X-axis, representing the number of weights in each row of the core). This is the depth of the filter (i.e., the length along the Z-axis, representing the number of channels). In some embodiments, equal For the sake of simplicity and explanation, Figure 2 Each filter 720 has a spatial size of 7×3×3, meaning that filter 720 includes seven convolutional kernels with a spatial size of 3×3. In other embodiments, the height, width, or depth of filter 720 may be different. The spatial size of the convolutional kernel is smaller than the spatial size of the 2D matrix of each input channel in the input tensor 710.

[0084] Activation values ​​or weights can occupy one or more bytes in memory. The number of bytes for activation values ​​or weights can depend on the data format. For example, when activation values ​​or weights are in INT8 format, the activation value occupies one byte. When activation values ​​or weights are in FP16 format, the activation value or weight occupies two bytes. Other data formats can be used for activation values ​​or weights.

[0085] In convolution, each filter 720 slides over the input tensor 710 and generates a 2D matrix for the output channels in the output tensor 730. Figure 7In this embodiment, the 2D matrix has a spatial size of 5×5. The output tensor 730 includes activation values ​​(also referred to as "output activation values," "elements," or "output elements") arranged in a 3D matrix. Output activation values ​​are data points in the output tensor 730. The output tensor 730 has a spatial size... ,in It is the height of the 3D matrix (i.e., the length along the Y-axis, representing the number of output activation values ​​in the columns of the 2D matrix for each output channel). It is the width of the 3D matrix (i.e., the length along the X-axis, representing the number of output activation values ​​in the rows of the 2D matrix for each output channel). It is the depth of the 3D matrix (i.e., the length along the Z-axis, representing the number of output channels). It can be equal to the number of filters 720 in the convolution. and It can depend on the height and width of the input tensor 710 and each filter 720.

[0086] As part of the convolution, the 3×3×3 sub-tensor 715 in the input tensor 710 (in... Figure 7 (Highlighted with a dotted pattern) and each filter 720 performs a product-accumulate (MAC) operation. The result of performing a MAC operation on subtensor 715 and filter 720 is the output activation value. In some embodiments (e.g., embodiments where the convolution is an integer convolution), the output activation value may include 8 bits, such as one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), the output activation value may include more than one byte. For example, the output element may include two bytes.

[0087] After completing the MAC operations on subtensor 715 and all filters 720, vector 735 is produced. Vector 735 is... Figure 7 Highlighted with a diagonal line. Vector 735 comprises a sequence of output activation values ​​arranged along the Z-axis. The output activation values ​​in vector 735 have the same (X, Y) coordinates, but they correspond to different output channels and have different Z-coordinates. The dimension of vector 735 along the Z-axis can be equal to the total number of output channels in output tensor 730. After generating vector 735, further MAC operations are performed to generate additional vectors until output tensor 730 is produced.

[0088] In some embodiments, the output activation values ​​in the output tensor 730 may be further processed based on one or more activation functions before being stored or input to the next layer of the CNN. Processing based on one or more activation functions may be at least part of post-convolution processing. In some embodiments, post-processing may include one or more other computations, such as offset computation, bias computation, etc. The result of post-processing may be stored in the local memory of the computation block and used as input to the next layer. In some embodiments, the input activation values ​​in the input tensor 710 may be the result of post-processing in the previous layer. Although in Figure 7 The input tensor 710, filter 720, and output tensor 730 are 3D tensors, but in other embodiments, the input tensor 710, filter 720, or output tensor 730 may be 2D tensors.

[0089] Figure 8 An example DNN 800 according to various embodiments is shown. Figure 8 The DNN 800 in the text can be Figure 1 DNN160, Figure 2 China's backbone network 220 Figure 6 Examples include DNN 610, etc. DNN 800 is trained to receive grid-structured data and output information indicating the condition of objects. Figure 8 In one embodiment, the DNN 800 includes a series of layers, including multiple convolutional layers 810 (collectively referred to as "convolutional layers 810"), multiple pooling layers 820 (collectively referred to as "pooling layers 820"), and multiple fully connected layers 830 (collectively referred to as "fully connected layers 830"). In other embodiments, the DNN 800 may include fewer, more, or different layers. During inference in the DNN 800, the layers of the DNN 800 perform tensor computations including a variety of tensor operations, such as convolution (e.g., MAC operations, etc.), pooling operations, element-wise operations (e.g., element-wise addition, element-wise multiplication, etc.), other types of tensor operations, or certain combinations of these operations.

[0090] Convolutional layer 810 summarizes the presence of features input into DNN 800. Convolutional layer 810 acts as a feature extractor. The first layer of DNN 800 is convolutional layer 810. In one example, convolutional layer 810 performs a convolution operation on input tensor 840 (also known as IFM 840) and filter 850. Figure 8As shown, the IFM 840 is represented by a 7×7×3 three-dimensional (3D) matrix. The IFM 840 includes three input channels, each represented by a 7×7 two-dimensional (2D) matrix. Each row of the 7×7 2D matrix contains 7 input elements (also called input points), and each column contains 7 input elements. The filter 850 is represented by a 3×3×3 3D matrix. The filter 850 includes three kernels, each corresponding to a different input channel of the IFM 840. The kernel is a 2D matrix of weights, where the weights are arranged by column and row. The kernel can be smaller than the IFM. Figure 8 In this embodiment, each kernel is represented by a 3×3 2D matrix. Each row of the 3×3 kernel contains 3 weights, and each column also contains 3 weights. The weights can be initialized and updated using gradient descent via backpropagation. The magnitude of the weights can represent the importance of filter 850 in extracting features from IFM 840.

[0091] The convolution involves a MAC operation on the input elements in the IFM 840 and the weights in the filter 850. The convolution can be a standard convolution 863 or a depthwise convolution 883. In a standard convolution 863, the entire filter 850 slides over the IFM 840. All input channels are combined to produce an output tensor 860 (also known as OFM 860). OFM 860 is represented by a 5×5 2D matrix. Each row of the 5×5 2D matrix contains 5 output elements (also called output points), and each column also contains 5 output elements. For illustration, in... Figure 8 In one embodiment, the standard convolution includes a filter. In an embodiment with multiple filters, the standard convolution can produce multiple output channels in the OFM 860.

[0092] The multiplication applied between a local patch of IFM 840 kernel size and the kernel can be a dot product. A dot product is an element-wise multiplication of a local patch of IFM 840 kernel size with the corresponding kernel, followed by summation, always producing a single value. Because it produces a single value, this operation is often called a "scalar product." Using a kernel smaller than IFM 840 is intentional because it allows the same kernel (a set of weights) to be multiplied multiple times by IFM 840 at different points on the IFM 840. Specifically, the kernel is systematically applied from left to right and top to bottom to each overlapping portion or local patch of IFM 840 kernel size. Multiplying the kernel by IFM 840 once results in a single value. Since the kernel is applied multiple times to IFM 840, the result of the multiplication is the output element of a 2D matrix. Thus, the 2D output matrix from the standard convolution 863 (i.e., OFM 860) is called OFM.

[0093] In depthwise convolution 883, the input channels are not combined. Instead, a MAC operation is performed on individual input channels and individual kernels to produce the output channel. For example... Figure 8 As shown, depthwise convolution 883 produces a depth output tensor 880. The depth output tensor 880 is represented by a 5×5×3 3D matrix. The depth output tensor 880 includes three output channels, each represented by a 5×5 2D matrix. Each row and column of the 5×5 2D matrix contains 5 output elements. Each output channel is the result of a MAC operation performed on the input channels of the IFM 840 and the kernel of the filter 850. For example, the first output channel (dot pattern) is the result of a MAC operation on the first input channel (dot pattern) and the first kernel (dot pattern); the second output channel (horizontal stripe pattern) is the result of a MAC operation on the second input channel (horizontal stripe pattern) and the second kernel (horizontal stripe pattern); and the third output channel (diagonal stripe pattern) is the result of a MAC operation on the third input channel (diagonal stripe pattern) and the third kernel (diagonal stripe pattern). In such depthwise convolution, the number of input channels equals the number of output channels, and each output channel corresponds to a different input channel. The input channels and output channels are collectively referred to as depth channels. After depthwise convolution, pointwise convolution 893 is performed on the depth output tensor 880 and the 1×1×3 tensor 890 to produce OFM 860.

[0094] OFM 860 is then passed to the next layer in the sequence. In some embodiments, OFM 860 is passed through an activation function. An example activation function is the Rectified Linear Unit (ReLU). ReLU is a computation that directly returns the value provided as input, or returns 0 if the input is 0 or less. Convolutional layer 810 can receive several images as input and compute the convolution of each of them with each kernel. This process can be repeated several times. For example, OFM 860 is passed to a subsequent convolutional layer 810 (i.e., the convolutional layer 810 in the sequence that produces OFM 860). The subsequent convolutional layer 810 performs convolution on OFM 860 with a new kernel and generates a new feature map. The new feature map can also be normalized and resized. The new feature map can be kernelized again by further subsequent convolutional layers 810, and so on.

[0095] In some embodiments, the convolutional layer 810 has four hyperparameters: the number of kernels, the size of the kernel (e.g., the kernel size is F×F×D pixels), the stride S of dragging the window corresponding to the kernel on the image (e.g., a stride of 1 means moving the window one pixel at a time), and zero padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 810). The convolutional layer 810 can perform various types of convolutions, such as 2D convolution, dilated or dilated convolution, spatially separable convolution, depthwise separable convolution, transposed convolution, etc. The DNN 800 includes 86 convolutional layers 810. In other embodiments, the DNN 800 may include a different number of convolutional layers.

[0096] Pooling layer 820 downsamples the feature map generated by the convolutional layer, for example, by downsampling the presence of features in blocks that summarize the feature map. Pooling layer 820 is positioned between two convolutional layers 810: a pre-convolutional layer 810 (the convolutional layer 810 preceding pooling layer 820 in the layer sequence) and a post-convolutional layer 810 (the convolutional layer 810 following pooling layer 820 in the layer sequence). In some embodiments, pooling layer 820 is added after convolutional layer 810, for example, after an activation function (e.g., ReLU, etc.) has been applied to OFM 860.

[0097] Pooling layer 820 receives feature maps generated by the preceding convolutional layer 810 and applies pooling operations to them. Pooling operations reduce the size of the feature maps while preserving their important characteristics. Therefore, pooling operations improve the efficiency of the CNN and avoid overlearning. Pooling layer 820 can perform pooling operations through average pooling (calculating the average value of each local block on the feature map), max pooling (calculating the maximum value of each local block on the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature map. In various embodiments, the pooling operation is applied with a stride of 2×2 pixels, thus reducing the size of the feature map to half its original size; for example, the number of pixels or values ​​in the feature map is reduced to one-quarter of its original size. In one example, pooling layer 820 applied to a 6×6 feature map produces a 3×3 output pooled feature map. The output of pooling layer 820 is fed into the subsequent convolutional layer 810 for further feature extraction. In some embodiments, pooling layer 820 operates on each feature map separately to create a new set of the same number of pooled feature maps.

[0098] Fully connected layer 830 is the last layer of the CNN. Fully connected layer 830 may or may not be convolutional. Fully connected layer 830 may also be referred to as a linear layer. In some embodiments, fully connected layer 830 (e.g., the first fully connected layer in a DNN 800) may receive input operands. The input operands may define the outputs of convolutional layer 810 and pooling layer 820, and include the values ​​of the final feature map generated by the last pooling layer 820 in the sequence. Fully connected layer 830 may apply a linear transformation to the input operands via a weight matrix. The weight matrix may be the kernel of fully connected layer 830. The linear transformation may include tensor multiplication between the input operands and the weight matrix. The result of the linear transformation may be the output operand. In some embodiments, the fully connected layer may further apply a nonlinear transformation (e.g., by using a nonlinear activation function) to the result of the linear transformation to generate the output operand. The output operand may contain as many elements as there are classes: element i represents the probability that an image belongs to class i. Therefore, each element is between 0 and 1, and the sum of all elements is 1. These probabilities are calculated by the final fully connected layer 830 using either a logistic function (for binary classification) or a SoftMax function (for multi-class classification) as the activation function.

[0099] Figure 9 An AI-based face tracking environment 900 according to various embodiments is illustrated. The AI-based face tracking environment 900 includes a computer vision system 910, a client device 920 (referred to solely as client device 920), and a third-party system 930. In other embodiments, the AI-based face tracking environment 900 may include fewer, more, or different components. For example, the AI-based face tracking environment 900 may include an additional number of client devices 920 or more than one third-party system 930.

[0100] Computer vision system 910 performs face tracking tasks, such as generating facial animations based on video (including monocular video). For example, computer vision system 910 can use a DNN to predict facial expressions and poses based on video frames. In some embodiments, computer vision system 910 can receive video capturing one or more faces and perform face tracking using one or more frames from the video. Computer vision system 910 can receive video from one or more client devices 920 or third-party systems 930. Furthermore, computer vision system 910 can transmit information indicating the face tracking results (e.g., face tracking animation, face tracking parameters, etc.) to one or more client devices 920 or third-party systems 930. An example of computer vision system 910 is... Figure 1 The computer vision system 100 in the middle.

[0101] Client device 920 communicates with computer vision system 910. For example, client device 920 can receive 3D facial animation from computer vision system 910 and display the 3D facial animation to one or more users associated with client device 920. In some embodiments, client device 920 can execute one or more applications, allowing one or more users of client device 920 to interact with computer vision system 910. For example, client device 920 executes a browser application to enable interaction between client device 920 and computer vision system 910. In another embodiment, client device 920 interacts with computer vision system 910 through an application programming interface (API) running on client device 920's native operating system (such as iOS® or Android™).

[0102] Client device 920 may be one or more computing devices capable of receiving user input and sending and / or receiving data via network 940. In one embodiment, client device 920 is a conventional computer system, such as a desktop computer or laptop computer. Alternatively, client device 920 may be a device with computer functionality, such as a personal digital assistant (PDA), mobile phone, smartphone, autonomous vehicle, or other suitable device. Client device 920 is configured to communicate via network 940. In one embodiment, client device 920 is an integrated computing device operating as a standalone networked device. For example, client device 920 includes a display, speakers, microphone, camera, and input devices. In another embodiment, client device 920 is a computing device for coupling to an external media device, such as a television or other external display and / or audio output system. In this embodiment, client device 920 may be coupled to the external media device via a wireless or wired interface and may utilize various functions of the external media device, such as its display, speakers, microphone, camera, and input devices. Here, the client device 920 can be configured to be compatible with general external media devices that do not have software, firmware, or hardware specifically designed for interacting with the client device 920.

[0103] The third-party system 930 is an online system capable of communicating with the computer vision system 910 or at least one client device 920. In some embodiments, the third-party system 930 can provide the computer vision system 910 with data for 3D face tracking. This data may include videos, data for training a DNN, data for validating the DNN, etc. The third-party system 930 may be a social media system, an online image library, an online search system, etc. Furthermore or alternatively, the third-party system 930 can use the results of 3D face tracking in various applications. For example, the third-party system 930 can use the face tracking results of the computer vision system 910 for virtual reality, augmented reality, film and game production, telepresence, etc.

[0104] Computer vision system 910, client device 920, and third-party system 930 are connected via network 940. Network 940 may include any combination of local area network and / or wide area network, using both wired and / or wireless communication systems. In one embodiment, network 940 may use standard communication technologies and / or protocols. For example, network 940 may include communication links using technologies such as Ethernet, 9010.11, Global Microwave Access Interoperability (WiMAX), 3G, 4G, Code Division Multiple Access (CDMA), Digital Subscriber Line (DSL), etc. Examples of network protocols used for communication via network 940 may include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). Data exchanged via network 940 may be represented using any suitable format, such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML). In some embodiments, all or some communication links of network 940 may be encrypted using any suitable technology or combination of technologies.

[0105] Figure 10 This is a flowchart illustrating a face tracking method 1000 according to various embodiments. Method 1000 may be a 3D vision recognition method. Method 1000 may be... Figure 1 The face tracking module 110 in the middle is executed. Although method 1000 is a reference Figure 10 The flowchart shown illustrates this approach, but many other methods can be used alternatively for visual recognition. For example, Figure 10 The execution order of the steps can be changed. As another example, some steps can be changed, deleted, or merged.

[0106] The face tracking module 110 generates a first feature map (1010) based on a first frame from a video capturing a face using a first neural network. In some embodiments, the first neural network is... Figure 1The DNN 160 in the example. In some embodiments, the first frame is the current frame. The first feature map is the current feature map.

[0107] The face tracking module 110 generates a second feature map (1020) based on a second frame in the video using a first neural network. The second frame is temporally later than the first frame in the video. In some embodiments, the first frame is a previous frame. The first feature map is a previous feature map.

[0108] The face tracking module 110 generates (1030) an aggregated feature map by using a first feature map and a second feature map. In some embodiments, the face tracking module 110 writes the first feature map to a buffer after generating the first feature map, and reads the first feature map from the buffer after generating the second feature map.

[0109] The face tracking module 110 determines (1040) one or more blend shape coefficients using an aggregated feature map and a trained blend shape representation via a second neural network. These one or more blend shape coefficients represent the predicted facial expression. In some embodiments, the second neural network may include a transformer decoder. The transformer decoder includes a cross-attention module (e.g., a cross-attention layer) that processes the aggregated feature map as a key or value and processes the trained blend shape representation as a query. In some embodiments, the trained blend shape representation includes multiple vectors, each corresponding to a trained blend shape.

[0110] In some embodiments, the face tracking module 110 generates an intermediate feature map based on the first and second feature maps using a third neural network. In some embodiments, the third neural network may include one or more transformer decoders.

[0111] In some embodiments, the face tracking module 110 further generates a face segmentation probability map of a second frame. The segmentation probability map indicates the segmentation of different parts of the face. The face tracking module 110 generates a reweighted mask by applying a third neural network to the face segmentation probability map. The face tracking module 110 generates a reweighted feature map by aggregating a second feature map with the reweighted mask. The face tracking module 110 also uses the reweighted feature map to determine one or more pose coefficients via a fourth neural network. In some embodiments, the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.

[0112] The face tracking module 110 generates (1150) facial animation based on blended shape coefficients. In some embodiments, the face tracking module 110 generates facial animation based on one or more blended shape coefficients and one or more pose coefficients. In some embodiments, the face tracking module 110 further generates facial animation based on one or more blended shape templates.

[0113] Figure 11 This is a block diagram of an example computing device 1100 according to various embodiments. In some embodiments, the computing device 1100 may be used as at least a part of a computer vision system 100. Multiple components are... Figure 11 The components are shown as being included in computing device 1100, but any one or more of these components may be omitted or copied to suit the application. In some embodiments, some or all of the components included in computing device 1100 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured on a single system-on-a-chip (SoC) die. Furthermore, in various embodiments, computing device 1100 may not include... Figure 11 The computing device 1100 may include one or more of the components shown, but may include interface circuitry for coupling to said one or more components. For example, the computing device 1100 may not include display device 1106, but may include display device interface circuitry (e.g., connectors and driver circuitry) to which display device 1106 may be coupled. In another set of examples, the computing device 1100 may not include audio input device 1118 or audio output device 1108, but may include audio input or output device interface circuitry (e.g., connectors and support circuitry) to which audio input device 1118 or audio output device 1108 may be coupled.

[0114] Computing device 1100 may include processing device 1102 (e.g., one or more processing devices). Processing device 1102 processes electronic data from registers and / or memory to convert the electronic data into other electronic data that can be stored in registers and / or memory. Computing device 1100 may include memory 1104, which itself may include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high-bandwidth memory (HBM), flash memory, solid-state memory, and / or hard disk drive. In some embodiments, memory 1104 may include memory sharing a die with processing device 1102. In some embodiments, memory 1104 includes one or more non-transitory computer-readable media storing instructions executable to perform operations for performing 3D face tracking, for example, as described above. Figure 10 Method 1000 described or combined with the above Figure 1 The computer vision system 100 or its components (e.g., face tracking module 110) described herein perform certain operations. Instructions stored in one or more non-transitory computer-readable media may be executed by processing device 1102.

[0115] In some embodiments, computing device 1100 may include communication chip 1112 (e.g., one or more communication chips). For example, communication chip 1112 may be configured to manage wireless communication for transmitting data to and from computing device 1100. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, technologies, communication channels, etc., that can transmit data via a non-solid medium using modulated electromagnetic radiation. This term does not imply that the associated device does not contain any wires; however, in some embodiments they may be wire-free.

[0116] The 1112 communication chip can implement any of many wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, such as Wi-Fi (IEEE 802.10 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), the Long Term Evolution (LTE) project, and any amendments, updates, and / or revisions (e.g., improved LTE projects, Ultra Mobile Broadband (UMB) projects (also known as "3GPP2"), etc.). Broadband Wireless Access (BWA) networks compatible with IEEE 802.16 are commonly referred to as WiMAX networks, an abbreviation for Global Microwave Access Interoperability, which is a certification mark for products that have passed conformance and interoperability testing of the IEEE 802.16 standard. Communication chip 1112 can operate according to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. Communication chip 1112 can also operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1112 can operate according to Code-Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunication (DECT), Evolution-Data Optimized (EV-DO) and its derivatives, as well as any other wireless protocol specified as 3G, 4G, 5G, etc. In other embodiments, the communication chip 1112 can operate according to other wireless protocols.The computing device 1100 may include an antenna 1122 to facilitate wireless communication and / or receive other wireless communications (e.g., AM or FM radio transmissions).

[0117] In some embodiments, the communication chip 1112 can manage wired communications such as electrical, optical, or any other suitable communication protocol (e.g., Ethernet). As described above, the communication chip 1112 may include multiple communication chips. For example, a first communication chip 1112 may be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1112 may be dedicated to long-range wireless communications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first communication chip 1112 may be dedicated to wireless communications, and the second communication chip 1112 may be dedicated to wired communications.

[0118] The computing device 1100 may include a battery / power circuit 1114. The battery / power circuit 1114 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1100 to a power source (e.g., AC line power) that is separate from the computing device 1100.

[0119] The computing device 1100 may include a display device 1106 (or the corresponding interface circuitry described above). For example, the display device 1106 may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.

[0120] The computing device 1100 may include an audio output device 1108 (or the corresponding interface circuitry described above). For example, the audio output device 1108 may include any device that generates audible indicators, such as a speaker, headphones, or earphones.

[0121] The computing device 1100 may include an audio input device 1118 (or the corresponding interface circuitry described above). The audio input device 1118 may include any device that generates a signal representing sound, such as a microphone, microphone array, or digital musical instrument (e.g., a musical instrument with a Musical Instrument Digital Interface (MIDI) output).

[0122] The computing device 1100 may include a GPS device 1116 (or a corresponding interface circuit as described above). As is known in the art, the GPS device 1116 can communicate with a satellite-based system and can receive the location of the computing device 1100.

[0123] The computing device 1100 may include other output devices 1110 (or corresponding interface circuitry as described above). Examples of other output devices 1110 may include audio codecs, video codecs, printers, wired or wireless transmitters for providing information to other devices, or additional storage devices.

[0124] The computing device 1100 may include other input devices 1120 (or corresponding interface circuitry as described above). Examples of other input devices 1120 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0125] The computing device 1100 can have any desired form factor, such as a handheld or mobile computer system (e.g., a mobile phone, smartphone, mobile internet device, music player, tablet computer, laptop computer, netbook computer, ultrabook computer, personal digital assistant (PDA), ultraportable personal computer, etc.), desktop computer system, server or other networked computing component, printer, scanner, monitor, set-top box, entertainment control unit, vehicle control unit, digital camera, digital video recorder, or wearable computer system. In some embodiments, the computing device 1100 can be any other electronic device that processes data.

[0126] The following paragraphs provide various examples of the embodiments disclosed herein.

[0127] Example 1 provides a method comprising: generating a first feature map based on a first frame in a video capturing a face using a first neural network; generating a second feature map based on a second frame in the video using the first neural network, wherein the second frame is temporally later than the first frame in the video; generating an aggregated feature map using the first feature map and the second feature map; determining one or more blended shape coefficients, the one or more blended shape coefficients representing predicted facial expressions, using the aggregated feature map and a trained blended shape representation via a second neural network; and generating an animation of the face based on the one or more blended shape coefficients.

[0128] Example 2 provides the method described in Example 1, wherein generating the aggregated feature map includes: generating an intermediate feature map using a third neural network based on the first feature map and the second feature map; and aggregating the intermediate feature map with the second feature map.

[0129] Example 3 provides the method described in Example 2, wherein the third neural network includes a transformer decoder.

[0130] Example 4 provides a method according to any one of Examples 1-3, wherein generating the aggregated feature map includes: writing the first feature map to a buffer after generating the first feature map; and reading the first feature map from the buffer after generating the second feature map.

[0131] Example 5 provides a method according to any one of Examples 1-4, wherein the second neural network includes a transformer decoder, the transformer decoder includes a cross-attention module, and the cross-attention module processes the aggregated feature map as a key or value and processes the trained hybrid shape representation as a query.

[0132] Example 6 provides a method as described in any one of Examples 1-5, wherein the trained blend shape representation comprises a plurality of vectors, and each vector corresponds to a trained blend shape.

[0133] Example 7 provides a method as described in any one of Examples 1-6, further comprising: generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different parts of the face; generating a reweighted mask by applying a third neural network to the face segmentation probability map; generating a reweighted feature map by aggregating the second feature map with the reweighted mask; and using the reweighted feature map via a fourth neural network to determine one or more pose coefficients.

[0134] Example 8 provides the method described in Example 7, wherein the third neural network is a convolutional neural network and the fourth neural network includes a transformer decoder.

[0135] Example 9 provides the method described in Example 7 or 8, wherein generating the animation of the face includes: generating the animation of the face based on the one or more blend shape coefficients and the one or more pose coefficients.

[0136] Example 10 provides a method according to any one of Examples 1-9, wherein generating an animation of the face includes: further generating an animation of the face based on one or more blended shape templates, the blended shape templates corresponding to reference facial expressions.

[0137] Example 11 provides one or more non-transitory computer-readable media storing executable instructions to perform operations including: generating a first feature map based on a first frame in a video capturing a face using a first neural network; generating a second feature map based on a second frame in the video using the first neural network, wherein the second frame is temporally later than the first frame in the video; generating an aggregated feature map using the first feature map and the second feature map; determining one or more blended shape coefficients using the aggregated feature map and a trained blended shape representation using a second neural network, the one or more blended shape coefficients representing a predicted facial expression; and generating an animation of the face based on the one or more blended shape coefficients.

[0138] Example 12 provides one or more non-transitory computer-readable media as described in Example 11, wherein generating the aggregated feature map includes: generating an intermediate feature map based on the first feature map and the second feature map using a third neural network; and aggregating the intermediate feature map with the second feature map.

[0139] Example 13 provides one or more non-transitory computer-readable media as described in Example 12, wherein the third neural network includes a transformer decoder.

[0140] Example 14 provides one or more non-transitory computer-readable media as described in any one of Examples 11-13, wherein generating the aggregated feature map includes: writing the first feature map to a buffer after generating the first feature map; and reading the first feature map from the buffer after generating the second feature map.

[0141] Example 15 provides one or more non-transitory computer-readable media as described in any one of Examples 11-14, wherein the second neural network includes a transformer decoder, the transformer decoder includes a cross-attention module, and the cross-attention module processes the aggregated feature map as a key or value and processes the trained hybrid shape representation as a query.

[0142] Example 16 provides one or more non-transitory computer-readable media as described in any one of Examples 11-15, wherein the trained blend shape representation comprises a plurality of vectors, and each vector corresponds to a trained blend shape.

[0143] Example 17 provides one or more non-transitory computer-readable media as described in any one of Examples 11-16, wherein the operation further includes: generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different parts of the face; generating a reweighted mask by applying a third neural network to the face segmentation probability map; generating a reweighted feature map by aggregating the second feature map with the reweighted mask; and using the reweighted feature map by a fourth neural network to determine one or more pose coefficients.

[0144] Example 18 provides one or more non-transitory computer-readable media as described in Example 17, wherein the third neural network is a convolutional neural network and the fourth neural network includes a transformer decoder.

[0145] Example 19 provides one or more non-transitory computer-readable media as described in Example 17 or 18, wherein generating an animation of the face includes: generating an animation of the face based on the one or more blended shape coefficients and the one or more pose coefficients.

[0146] Example 20 provides one or more non-transitory computer-readable media as described in any one of Examples 11-19, wherein generating an animation of the face includes: further generating an animation of the face based on one or more blended shape templates, the blended shape templates corresponding to reference facial expressions.

[0147] Example 21 provides an apparatus comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable storage medium storing computer program instructions executable by the computer processor to perform operations including: generating a first feature map based on a first frame in a video capturing a face using a first neural network; generating a second feature map based on a second frame in the video using the first neural network, wherein the second frame is temporally later than the first frame in the video; generating an aggregated feature map using the first feature map and the second feature map; determining one or more blended shape coefficients using the aggregated feature map and a trained blended shape representation via a second neural network, the one or more blended shape coefficients representing a predicted facial expression; and generating an animation of the face based on the one or more blended shape coefficients.

[0148] Example 22 provides the apparatus of Example 21, wherein generating the aggregated feature map includes: generating an intermediate feature map based on the first feature map and the second feature map using a third neural network; and aggregating the intermediate feature map with the second feature map.

[0149] Example 23 provides the apparatus described in Example 21 or 22, wherein generating the aggregated feature map includes: writing the first feature map to a buffer after generating the first feature map; and reading the first feature map from the buffer after generating the second feature map.

[0150] Example 24 provides an apparatus as described in any one of Examples 21-23, wherein the operation further includes: generating a segmentation probability map of the second frame, the segmentation probability map indicating segmentation of different parts of the face; generating a reweighted mask by applying a third neural network to the face segmentation probability map; generating a reweighted feature map by aggregating the second feature map with the reweighted mask; and using the reweighted feature map via a fourth neural network to determine one or more pose coefficients.

[0151] Example 25 provides the apparatus described in Example 24, wherein the third neural network is a convolutional neural network and the fourth neural network includes a transformer decoder.

[0152] The foregoing description of the embodiments illustrated herein, including the content described in the abstract, is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Although specific implementations and examples of the present disclosure have been described herein for illustrative purposes, various equivalent modifications can be made within the scope of this disclosure, as will be recognized by those skilled in the art. These modifications can be made to the present disclosure based on the foregoing detailed description.

Claims

1. A method comprising: A first feature map is generated using a first neural network based on the first frame of the video capturing the face. The first neural network generates a second feature map based on a second frame in the video, wherein the second frame is later in time than the first frame in the video; An aggregated feature map is generated by using the first feature map and the second feature map; One or more blended shape coefficients, representing predicted facial expressions, are determined using the aggregated feature map and a trained blended shape representation via a second neural network. as well as The animation of the face is generated based on one or more of the hybrid shape coefficients.

2. The method according to claim 1, wherein generating the aggregated feature map comprises: An intermediate feature map is generated using a third neural network based on the first and second feature maps. as well as The intermediate feature map is then aggregated with the second feature map.

3. The method of claim 2, wherein the third neural network includes a transformer decoder.

4. The method according to any one of claims 1-3, wherein generating the aggregated feature map comprises: After generating the first feature map, the first feature map is written to the buffer; as well as After the second feature map is generated, the first feature map is read from the buffer.

5. The method according to any one of claims 1-4, wherein the second neural network includes a transformer decoder, the transformer decoder includes a cross-attention module, and the cross-attention module processes the aggregated feature map as a key or value and processes the trained hybrid shape representation as a query.

6. The method according to any one of claims 1-5, wherein the trained blend shape representation comprises a plurality of vectors, and each vector corresponds to a trained blend shape.

7. The method according to any one of claims 1-6, further comprising: Generate a segmentation probability map of the second frame, the segmentation probability map indicating the segmentation of different parts of the face; A reweighted mask is generated by applying a third neural network to the face segmentation probability map; as well as A reweighted feature map is generated by aggregating the second feature map with the reweighted mask; and One or more pose coefficients are determined using the reweighted feature map via a fourth neural network.

8. The method of claim 7, wherein the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.

9. The method of claim 7 or 8, wherein generating the animation of the face comprises: The animation of the face is generated based on the one or more hybrid shape coefficients and the one or more pose coefficients.

10. The method according to any one of claims 1-9, wherein generating the animation of the face comprises: The animation of the face is also generated based on one or more blended shape templates, which correspond to reference facial expressions.

11. One or more non-transitory computer-readable media storing instructions that are executable to perform operations, the operations including: A first feature map is generated using a first neural network based on the first frame of the video capturing the face. The first neural network generates a second feature map based on a second frame in the video, wherein the second frame is later in time than the first frame in the video; An aggregated feature map is generated by using the first feature map and the second feature map; One or more blended shape coefficients, representing predicted facial expressions, are determined using the aggregated feature map and a trained blended shape representation via a second neural network. as well as The animation of the face is generated based on one or more of the hybrid shape coefficients.

12. The one or more non-transitory computer-readable media of claim 11, wherein generating the aggregated feature map comprises: An intermediate feature map is generated using a third neural network based on the first and second feature maps. as well as The intermediate feature map is then aggregated with the second feature map.

13. One or more non-transitory computer-readable media according to claim 12, wherein the third neural network includes a transformer decoder.

14. One or more non-transitory computer-readable media according to any one of claims 11-13, wherein generating the aggregated feature map comprises: After generating the first feature map, the first feature map is written to the buffer; as well as After the second feature map is generated, the first feature map is read from the buffer.

15. One or more non-transitory computer-readable media according to any one of claims 11-14, wherein the second neural network includes a transformer decoder, the transformer decoder includes a cross-attention module, and the cross-attention module processes the aggregated feature map as a key or value and processes the trained hybrid shape representation as a query.

16. One or more non-transitory computer-readable media according to any one of claims 11-15, wherein the trained hybrid shape representation comprises a plurality of vectors, and each vector corresponds to a trained hybrid shape.

17. One or more non-transitory computer-readable media according to any one of claims 11-16, wherein the operation further comprises: Generate a segmentation probability map of the second frame, the segmentation probability map indicating the segmentation of different parts of the face; A reweighted mask is generated by applying a third neural network to the face segmentation probability map; as well as A reweighted feature map is generated by aggregating the second feature map with the reweighted mask; and One or more pose coefficients are determined using the reweighted feature map via a fourth neural network.

18. The one or more non-transitory computer-readable media of claim 17, wherein the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.

19. One or more non-transitory computer-readable media according to claim 17 or 18, wherein generating the animation of the face comprises: The animation of the face is generated based on the one or more hybrid shape coefficients and the one or more pose coefficients.

20. One or more non-transitory computer-readable media according to any one of claims 11-19, wherein generating the animation of the face comprises: The animation of the face is also generated based on one or more blended shape templates, which correspond to reference facial expressions.

21. An apparatus comprising: A computer processor is used to execute computer program instructions; as well as A non-transitory computer-readable storage device stores computer program instructions that can be executed by the computer processor to perform operations, the operations including: A first feature map is generated using a first neural network based on the first frame of the video capturing the face. The first neural network generates a second feature map based on a second frame in the video, wherein the second frame is later in time than the first frame in the video; An aggregated feature map is generated by using the first feature map and the second feature map; One or more blended shape coefficients, representing predicted facial expressions, are determined using the aggregated feature map and a trained blended shape representation via a second neural network; and The animation of the face is generated based on one or more of the hybrid shape coefficients.

22. The apparatus of claim 21, wherein generating the aggregated feature map comprises: An intermediate feature map is generated using a third neural network based on the first and second feature maps. as well as The intermediate feature map is then aggregated with the second feature map.

23. The apparatus of claim 21 or 22, wherein generating the aggregated feature map comprises: After generating the first feature map, the first feature map is written to the buffer; as well as After the second feature map is generated, the first feature map is read from the buffer.

24. The apparatus according to any one of claims 21-23, wherein the operation further comprises: Generate a segmentation probability map of the second frame, the segmentation probability map indicating the segmentation of different parts of the face; A reweighted mask is generated by applying a third neural network to the face segmentation probability map; as well as A reweighted feature map is generated by aggregating the second feature map with the reweighted mask; and One or more pose coefficients are determined using the reweighted feature map via a fourth neural network.

25. The apparatus of claim 24, wherein the third neural network is a convolutional neural network, and the fourth neural network includes a transformer decoder.

Images