A method, system and medium for rendering interpolated faces from a set of video frames

Abstract

A method for face-based frame rate upsampling for video calling includes receiving a set of video frames corresponding to a video, the set of video frames including a first video frame and a second video frame, the first video frame and the second video frame each including a face, wherein the second video frame is subsequent to the first video frame. The method also includes performing face tracking on the first video frame to identify a first face resampling keyframe, and performing face tracking on the second video frame to identify a second face resampling keyframe. The method also includes deriving an interpolation amount. The method also includes determining a first interpolated face frame based on the first face resampling keyframe and the interpolation amount. The method also includes determining a second interpolated face frame based on the second face resampling keyframe and the interpolation amount. The method also includes rendering the interpolated first face and the interpolated second face. The method also includes displaying a final frame.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 884,135, filed on August 7, 2019, entitled “Face-Based Frame Rate Upsampling for Video Calling,” which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present invention relates to the field of video calling, and in particular to face-based frame rate upsampling for video calling. Background Art

[0004] Video streams on mobile devices, such as those used in video calling applications, can use low frame rates, such as 15 frames per second, to save bandwidth or other resources. However, lower frame rates equate to an unsatisfactory viewer experience. For example, in a video call where objects undergo rapid motion, lower frame rates may cause viewers to perceive stuttering in the motion. Although many mobile devices are capable of rendering higher frame rate videos (e.g., 25 FPS, 30 FPS, 60 FPS, 90 FPS, etc.), bandwidth, battery, or local processing capacity limitations may make high frame rates impractical.

[0005] Traditional motion interpolation techniques use optical flow or voxel flow to detect the movement of pixels between frames. The interframe flows are interpolated to synthesize interpolated frames. However, even when performed on a graphics processing unit of a desktop computer, determining and interpolating the flows is computationally expensive because it requires analyzing each pixel to identify movement between frames. As a result, this approach is too computationally demanding to be used to display video on mobile devices or other devices that are limited by computing power. In addition, traditional motion interpolation works well on televisions where video preprocessing occurs. This is not feasible for video streams that display video in real time.

[0006] The relevant description of the background technology is for the purpose of generally rendering the context of the present disclosure. The work of the inventor of the present invention is described in this background technology section, and various aspects of the description that may not have prior art qualifications at the time of filing are neither explicitly nor implicitly admitted as prior art of the present disclosure. Summary of the invention

[0007] Embodiments are generally directed to a method that includes receiving a set of video frames corresponding to a video, the set of video frames including a first video frame and a second video frame, each of the first video frame and the second video frame each including a face, wherein the second video frame is subsequent to the first video frame. The method also includes performing face tracking on the first video frame to identify a first face resampling keyframe, the first face resampling keyframe including a first head transformation matrix and first facial landmark vertices. The method also includes performing face tracking on the second video frame to identify a second face resampling keyframe, the second face resampling keyframe including a second head transformation matrix and second facial landmark vertices. The method also includes deriving an interpolation amount. The method also includes determining a first interpolated facial frame based on the first face resampling keyframe and the interpolation amount. The method also includes determining a second interpolated facial frame based on the second face resampling keyframe and the interpolation amount. The method also includes interpolating a first background of the first face resampling keyframe and a second background of the second face resampling keyframe based on the interpolation amount. The method also includes rendering the interpolated first face and the interpolated second face. The method also includes displaying a final frame based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face.

[0008] In some embodiments, the method further comprises blending the first background with the second background to obtain a blended background, blending the interpolated first face with the interpolated second face to obtain a blended interpolated face, and generating a final frame by placing the smoothed face on top of the blended interpolated face and the blended background. In some embodiments, determining the first interpolated face frame comprises: extracting a translation vector, a rotation quaternion, and a scale vector using the first head transformation matrix, linearly interpolating the translation vector, interpolating the rotation quaternion using linear interpolation to generate an interpolated rotation quaternion, interpolating the scale vector using linear interpolation to generate an interpolated scale vector, composing an interpolated translation-rotation-scale matrix based on the interpolated translation vector, the interpolated rotation quaternion, and the interpolated scale vector. And using the interpolated translation-rotation-scale matrix to calculate an interpolated position of the interpolated first face. In some embodiments, the interpolation amount is derived from (a) the duration between the first face resampling keyframe and the second face resampling keyframe and (b) the current rendering time. In some embodiments, determining the second interpolated face frame comprises calculating a displacement of each vertex in the second facial landmark vertices. In some embodiments, interpolating the first background and the second background is done with alpha blending. In some embodiments, rendering includes at least one of feathering an edge of a face based on an interpolation amount or fading between a first interpolated face frame and a second interpolated face frame. In some embodiments, performing face tracking on the first video frame further includes determining a first texture coordinate and a timestamp of a first face resampling keyframe, and applying the first texture coordinate to a first facial landmark vertex. In some embodiments, the first background and the face are identified by performing red-green-blue (RGB) face tracking on the first video frame.

[0009] A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more computers, causing the one or more computers to perform operations including: receiving a set of video frames corresponding to a video, the set of video frames comprising a first video frame and a second video frame, the first video frame and the second video frame each containing a face, wherein the second video frame follows the first video frame. Performing facial tracking on the first video frame to identify a first facial resampling key frame, the first facial resampling key frame includes a first head transformation matrix and first facial landmark vertices, performing facial tracking on the second video frame to identify a second facial resampling key frame, the second facial resampling key frame includes a second head transformation matrix and second facial landmark vertices, and deriving an interpolation amount; determining a first interpolated facial frame based on the first facial resampling key frame and the interpolation amount; determining a second interpolated facial frame based on the second facial resampling key frame and the interpolation amount; interpolating a first background of the first facial resampling key frame and a second background of the second facial resampling key frame based on the interpolation amount, rendering the interpolated first face and the interpolated second face; and displaying a final frame based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face.

[0010] In some embodiments, the operation further comprises: blending the first background with the second background to obtain a blended background, blending the interpolated first face with the interpolated second face to obtain a blended interpolated face, and generating the final frame by placing a smooth face on top of the blended interpolated face and the blended background. In some embodiments, determining the first interpolated face frame comprises: extracting a translation vector, a rotation quaternion, and a scale vector using a first head transformation matrix, interpolating the translation vector using linear interpolation, interpolating the rotation quaternion using linear interpolation to generate an interpolated rotation quaternion, interpolating the scale vector using linear interpolation to generate an interpolated scale vector, composing an interpolated translation-rotation-scale matrix based on the interpolated translation vector, the interpolated rotation quaternion, and the interpolated scale vector. And calculating an interpolated position of the interpolated first face using the interpolated translation-rotation-scale matrix. In some embodiments, the interpolation amount is derived based on (a) the duration between the first face resampling keyframe and the second face resampling keyframe and (b) the current rendering time. In some embodiments, determining the second interpolated face frame comprises calculating a displacement of each vertex in the second facial landmark vertices.

[0011] The system may include one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving a set of video frames corresponding to a video, the set of video frames comprising a first video frame and a second video frame, the first video frame and the second video frame each including a face, wherein the second video frame is after the first video frame. Performing facial tracking on the first video frame to identify a first facial resampling key frame, the first facial resampling key frame includes a first head transformation matrix and first facial landmark vertices, performing facial tracking on the second video frame to identify a second facial resampling key frame, the second facial resampling key frame includes a second head transformation matrix and second facial landmark vertices, and deriving an interpolation amount; determining a first interpolated facial frame based on the first facial resampling key frame and the interpolation amount; determining a second interpolated facial frame based on the second facial resampling key frame and the interpolation amount; interpolating a first background of the first facial resampling key frame and a second background of the second facial resampling key frame based on the interpolation amount, rendering the interpolated first face and the interpolated second face; and displaying a final frame based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face.

[0012] In some embodiments, the operation further comprises: blending the first background with the second background, blending the interpolated first face with the interpolated second face, and generating the final frame by placing a smooth face on top of the blended interpolated face and the blended background. In some embodiments, determining the first interpolated face frame comprises: extracting a translation vector, a rotation quaternion, and a scale vector using a first head transformation matrix, interpolating the translation vector using linear interpolation; interpolating the rotation quaternion using linear interpolation to generate an interpolated rotation quaternion; interpolating the scale vector using linear interpolation to generate an interpolated scale vector, composing an interpolated translation-rotation-scale matrix based on the interpolated translation vector, the interpolated rotation quaternion, and the interpolated scale vector. And calculating an interpolated position of the interpolated first face using the interpolated translation-rotation-scale matrix. In some embodiments, the interpolation amount is derived based on (a) the duration between the first face resampling keyframe and the second face resampling keyframe and (b) the current rendering time. In some embodiments, determining the second interpolated face frame comprises calculating the displacement of each vertex in the second facial landmark vertices. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present disclosure is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references are used to refer to similar elements.

[0014] Figure 1A block diagram is shown of an example environment for generating interpolated video frames in accordance with some embodiments.

[0015] Figure 2 A block diagram of an example computing device that generates interpolated video frames in accordance with some embodiments is shown.

[0016] Figure 3A An example of a first face resampling keyframe is shown in accordance with some embodiments.

[0017] Figure 3B An example of a first face resampling keyframe is shown in accordance with some embodiments.

[0018] Figure 3C An output image with texture from a first frame warped based on an interpolation grid is shown in accordance with some embodiments.

[0019] Figure 3D An output image with texture from a second frame warped based on an interpolation grid is shown in accordance with some embodiments.

[0020] Figure 4A An example of input video frames of a video transmitted at 7.5 frames per second (FPS) is shown in accordance with some embodiments.

[0021] Figure 4B An interpolated first face and an interpolated second face are shown at 240 FPS in accordance with some embodiments.

[0022] Figure 5A An example of input video frames of a video transmitted at 7.5 FPS is shown in accordance with some embodiments.

[0023] Figure 5B A video frame at 240 FPS is shown with a fade in preparation for an upcoming smooth face being drawn on top, in accordance with some embodiments.

[0024] Fig. 6A An example of input video frames of a video transmitted at 7.5 FPS is shown in accordance with some embodiments.

[0025] Figure 6B An example blended frame without a background is shown, rendered at 240 FPS based on a first interpolated facial frame and a second interpolated facial frame rendered at 240 FPS, in accordance with some embodiments.

[0026] Fig. 7A An example of input video frames of a video transmitted at 7.5 FPS is shown in accordance with some embodiments.

[0027] Figure 7B The final frame is shown displayed on a user device according to some embodiments.

[0028] Figure 8 Two frame sequences are shown, one at a low frame rate without interpolation and the other at a higher frame rate where additional frames are interpolated and added to the frame sequence, according to some embodiments.

[0029] Fig.9A and 9B A flow chart is shown of an example method of generating an interpolated video frame in accordance with some embodiments. DETAILED DESCRIPTION

[0030] A planar transformer extending on seven layers substantially parallel to each other's planes is used as an illustrative example; in a row of consecutive layers, perpendicular to the layers are called the first layer S1, the second layer P1, the third layer S2, the fourth layer P2, the fifth layer S3, the sixth layer P3 and the seventh layer S4.

[0031] In some embodiments, a mobile device receives a set of video frames corresponding to a video. For example, the set of video frames may include a first video frame and a second video frame each including a face. The second video frame may be after the first video frame. The mobile device may perform facial tracking on the first video frame to identify a first facial resampling keyframe, wherein the first facial resampling keyframe includes a first head transformation matrix and a first facial landmark vertex. Facial tracking may be performed on the second video frame to identify a second facial resampling keyframe, wherein the second facial resampling keyframe includes a second head transformation matrix and a second facial landmark vertex. An interpolation amount is derived. A first interpolated facial frame is determined based on the first facial resampling keyframe and the interpolation amount. A second interpolated facial frame is determined based on the second facial resampling keyframe and the interpolation amount. A first background of the first facial resampling keyframe and a second background of the second facial resampling keyframe are interpolated based on the interpolation amount. The interpolated first face and the interpolated second face are rendered. A final frame based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face is displayed.

[0032] The various embodiments described below have several advantages. First, the processing is performed by the user device displaying the video. As a result, the video application uses reduced computing power to display the video with a perceived higher frame rate. Second, even when the video is received using end-to-end encryption between the sender device and the receiver device displaying the video, the embodiments provide a frame rate higher than the received video frame rate. Third, interpolation is computationally efficient because the structure of the video frame is interpolated rather than the texture.

[0033] Example System

[0034] Figure 1A block diagram of an example environment 100 for generating interpolated video frames is shown. The system 100 shown includes a video server 101, user devices 115a, 115n, and a network 105. Users 125a, 125n may be associated with respective user devices 115a, 115n. In some embodiments, the system 100 may include Figure 1 Other servers or devices not shown. Figure 1 In the drawings and the remaining drawings, a letter following a reference number, e.g., "115a", indicates a reference to the component with that specific reference number. A reference number in text without a following letter, e.g., "115", indicates a general reference to an embodiment of the component with that reference number.

[0035] The video server 101 may include a processor, a memory, and a network communication capability. In some embodiments, the video server 101 is a hardware server. The video server 101 is communicatively coupled to the network 105 via a signal line 102. The signal line 102 may be a wired connection, such as Ethernet, a coaxial cable, a fiber optic cable, etc., or a wireless connection, such as Wi-Fi, Bluetooth or other wireless technologies. In some embodiments, the video server 101 sends data to and receives data from one or more user devices 115a, 115n via the network 105. The video server 101 may include a video application 103a and a database 199.

[0036] The video application 103a may include code and routines operable to receive a video stream from one user device 115a and send the video stream to another user device 115n. For example, the video stream may be encrypted end-to-end so that the video server 101 relays the video stream as data without performing video processing. In some embodiments, the video application 103 may be implemented using hardware including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), any other type of processor, or a combination thereof. In some embodiments, the video application 103 may be implemented using a combination of hardware and software.

[0037] The database 199 may store social network data associated with the user 125 , user preferences of the user 125 , and the like.

[0038] In the case where the systems and methods discussed herein can collect or use personal information about a user (e.g., user data, information about a user's social network, a user's location, a user's biometric information, a user's activities and / or demographic information, video application 103 storage and analysis of videos, etc.), the user is provided with an opportunity to control whether personal information is collected, whether personal information is stored, whether personal information is used, whether images or videos are analyzed, and how user personal information is collected, stored, and used. That is, the systems and methods discussed herein can collect, store, and / or use user personal information only when explicit authorization is received from the relevant user. For example, the user is provided with control over whether a program or feature collects user information about the particular user or other users associated with the program or feature. One or more options are presented to each user for whom personal information is to be collected to allow control over the collection of information related to the user, thereby providing permission or authorization as to whether information is collected and as to which parts of the information are to be collected. For example, one or more such control options may be provided to the user via a communication network. In addition, before certain data is stored or used, the data may be processed in one or more ways to remove personally identifiable information. As an example, the user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0039] User device 115 may be a computing device including memory and a hardware processor. For example, user device 115 may include a desktop computer, a mobile device, a tablet computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game console, a portable music player, a reader device, or other electronic device capable of accessing network 105.

[0040] In the illustrated embodiment, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Signal lines 108 and 110 may be wired connections, such as Ethernet, coaxial cable, fiber optic cable, etc., or wireless connections, such as Wi-Fi, Bluetooth or other wireless technologies. User devices 115a, 115n are accessed by users 125a, 125n, respectively. Figure 1 The user devices 115a, 115n in FIG. 1 are used as examples. Figure 1 Two user devices 115 a and 115 n are shown, but the present disclosure is applicable to system architectures having one or more user devices 115 .

[0041] In some embodiments, user device 115 may be a wearable device worn by user 125. For example, user device 115n is included as part of a clip (e.g., a wristband), as part of jewelry, or as part of a pair of glasses. In another example, user device 115n may be a smartwatch. User 125n may view a video generated by video application 103c on a display of user device 115n worn by user 125n. For example, video application 103c may display interpolated video frames on user device 115n.

[0042] In some embodiments, user device 115a may include a video application 103b that generates video for a video call. User device 115a may send a video stream directly to user device 115n during a video call via network 105 or via video server 101. User device 115n may include a video application 103c that displays a video stream (one-way video) or a video application executed on another device (two-way video or as part of a video call application).

[0043] The user device 115n is a receiver device that includes the video application 103c. The user device 115n receives a set of video frames corresponding to a video. The video application 103c identifies faces and backgrounds in the frames, constructs interpolated face frames, interpolates the backgrounds of the frames, renders the interpolated faces, and displays a final frame based on the interpolated faces and the backgrounds in the interpolated frames.

[0044] In the illustrated embodiment, the entities of system 100 are communicatively coupled via a network 105. Network 105 can be of a conventional type, wired or wireless, and can have a variety of different configurations, including a star configuration, a token ring configuration, or other configurations. In addition, network 105 can include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), and / or other interconnected data paths through which multiple devices can communicate. In some embodiments, network 105 can be a peer-to-peer network. Network 105 can also be coupled to a telecommunications network or include portions of a telecommunications network for sending data in a variety of different communication protocols. In some embodiments, network 105 includes a communications network specified by IEEE 902.11, a wireless local area network (WLAN), computer communications, or a cellular communications network for sending and receiving data, including via short message service (SMS), multimedia messaging service (MMS), hypertext transfer protocol (HTTP), direct data connection, email, etc. Although Figure 1 One network 105 is shown coupled to the user device 115 and the video server 101, but one or more networks 105 may actually be coupled to these entities.

[0045] Example computing device

[0046] Figure 2 A block diagram of an example computing device 200 that generates interpolated video frames is shown. The computing device 200 may be a user device 115 or a video server 101. The computing device 200 may include a processor 235, a memory 237, a communication unit 239, a display 241, and a storage device 247. Depending on the type of computing device 200, additional components may be present, or some of the previous components may be omitted. The video application 103 may be stored in the memory 237. In some embodiments, the computing device 200 may include other components not listed here, such as a battery, etc. The components of the computing device 200 may be communicatively coupled via a bus 220.

[0047] Processor 235 includes an arithmetic logic unit, a microprocessor, a general purpose controller, or some other processor array to perform calculations and provide instructions to the display device. Processor 235 processes data and may include a variety of computing architectures, including a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or an architecture that implements a combination of instruction sets. Although Figure 2 A single processor 235 is included, but multiple processors 235 may be included. Other processors, operating systems, sensors, displays, and physical configurations may be part of the computing device 200. The processor 235 is coupled to the bus 220 to communicate with the other components via signal line 222.

[0048] Memory 237 stores instructions and / or data that can be executed by processor 235. The instructions may include code for executing the techniques described herein. Memory 237 may be a dynamic random access memory (DRAM) device, a static RAM, or other memory device. In some embodiments, memory 237 also includes non-volatile memory, such as a static random access memory (SRAM) device or a flash memory, or similar permanent storage devices and media, including a hard drive, a compact disc read-only memory (CD-ROM) device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other large-capacity storage device for storing information on a more permanent basis. Memory 237 includes code and routines that are operable to execute video application 103, which will be described in more detail below. Memory 237 is coupled to bus 220 to communicate with other components via signal line 224.

[0049] The communication unit 239 sends data to and receives data from at least one of the user device 115 and the video server 101. In some embodiments, the communication unit 239 includes a wireless transceiver for exchanging data with the user device 115, the video server 101, or other communication channels using one or more wireless communication methods (including IEEE 802.11, IEEE 802.16, Bluetooth, or other suitable wireless communication methods). The communication unit 239 is coupled to the bus 220 to communicate with other components via the signal line 226.

[0050] In some embodiments, the communication unit 239 includes a cellular communication transceiver for sending and receiving data over a cellular communication network, including via short message service (SMS), multimedia message service (MMS), hypertext transfer protocol (HTTP), direct data connection, email, or other suitable types of electronic communication. In some embodiments, the communication unit 239 includes a wired port and a wireless transceiver. The communication unit 239 also provides other conventional connections to the network 105 for distributing files and / or media objects using standard network protocols, including but not limited to user datagram protocol (UDP), TCP / IP, HTTP, HTTP secure (HTTPS), simple mail transfer protocol (SMTP), SPDY, Quick UDP Internet Connection (QUIC), etc. The communication unit 239 is coupled to the bus 220 to communicate with other components via signal line 226.

[0051] Display 241 may include hardware operable to display graphics data received from video application 103. For example, display 241 may render graphics to display video frames of a video. Display 241 may be any type of display, such as a liquid crystal display (LCD), OLED, etc. In some embodiments, display 241 may be a projection screen. In some embodiments, for example, when device 241 is an augmented reality device, display 241 may be a stereoscopic display. Display 241 is coupled to bus 220 to communicate with other components via signal line 228.

[0052] Storage device 247 may be a non-transitory computer-readable storage medium that stores data that provides the functionality described herein. In an embodiment where computing device 200 is video server 101, storage device 247 may include Figure 1199 in the database. Storage device 247 can be a DRAM device, an SRAM device, a flash memory, or some other storage device. In some embodiments, storage device 247 also includes non-volatile memory or similar permanent storage devices and media, including a hard drive, a CD-ROM device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other mass storage device for permanent storage of information. Storage device 247 is coupled to bus 220 to communicate with other components via signal line 230.

[0053] The video application 103 may include a decoder 202 , a video analyzer 204 , a face resampler 206 , and a user interface module 208 .

[0054] Decoder 202 decodes encoded video frames. In some embodiments, decoder 202 includes a set of executable instructions processed by processor 235 to decode encoded video frames, such as video frames received from a sending device participating in a video call with computing device 200. In some embodiments, decoder 202 is stored in memory 237 of computing device 200 and is accessible and executable by processor 235.

[0055] In some embodiments, the decoder 202 receives a set of encoded video frames via the communication unit 239. The decoder 202 decodes the video frames, for example, by decrypting (e.g., when the video is encrypted) and decompressing the encoded video frames. Once the decoder 202 decodes the video frames, in conventional video processing, the video frames are typically rendered for display. However, according to the techniques described herein, the decoder 202 skips scheduling of video frames for presentation, for example, when the frame rate of the received video is lower than the frame rate of the video application 103. As a result, instead of scheduling video frames for presentation, the decoder 202 provides each of the decoded video frames to the video analyzer 204. For example, the set of video frames includes a first video frame and a second video frame, wherein the second video frame is after the first video frame, for example, associated with a timestamp later than the timestamp associated with the first video frame, so that the second video frame occupies a later position in the frame sequence than the first video frame. In some embodiments, the second video frame is directly after the first video frame, for example, two frames are identified for sequential display without other frames in between.

[0056] Video analyzer 204 performs facial tracking. In some embodiments, video analyzer 204 includes an instruction set that can be performed by processor 235 for facial tracking. In some embodiments, video analyzer 204 is stored in memory 237 of computing device 200 and can be accessed and executed by processor 235.

[0057] The video analyzer 204 runs at a frequency with a marker signal (clock signal) that is independent of the video frame rate of the user device 115. For example, the video analyzer 204 runs at a frequency of 60 Hz, and there is an independent clock running at 60 Hz that is linked to the refresh / render loop for displaying video frames.

[0058] The video analyzer 204 receives a set of decoded video frames and identifies, for each frame, a background and a face in the decoded video frame. For example, the video analyzer 204 identifies a first background and a face in a first video frame and a second background and a face in a second video frame. The video analyzer 204 may distinguish between the face and the background in the frames by applying a facial tracking technique. For example, the video analyzer 204 may apply a red-green-blue (RGB) facial tracking technique that identifies a facial resampling keyframe, texture coordinates (e.g., RGB texture) of the facial resampling keyframe, and a timestamp of the video frame.

[0059] In some embodiments, the face resampling keyframe includes two values: a head transformation matrix and a set of landmark vertices. The head transformation matrix is ​​a 4×4 matrix M that represents the head transformation. The head transformation matrix transforms from an arbitrary initial model space (initial position, orientation, and scale) to the face space (including translation, rotation, and scale). A set of landmark vertices V describes the set of landmark vertices in the face space.

[0060] In some embodiments, the video analyzer 204 uses machine learning to perform real-time face tracking of a decoded set of video frames. In some embodiments, the trained model can be trained, for example, based on training data for which permission to use user data for training has been explicitly obtained from the user. The training data can include any data that is permitted for training, such as video and corresponding metadata, such as synthetic or computer-generated data, data that is permitted for training, etc. The training data can include images and videos in which face resampling keyframes have been identified in the images and video frames. The images and videos can be from videos stored in the database 199 of the video server 101.

[0061] In some embodiments, a trained model can be obtained based on supervised learning, for example, based on training data including images and videos and corresponding metadata. For example, a trained model can include a model form or structure (e.g., describing the number and organization of multiple nodes with associated weights in a neural network layer). In some embodiments, a model can be trained to detect and track faces in a video. For example, a video can be provided, and data indicating facial locations (e.g., facial landmarks) can also be provided as input to an untrained model. The model can be trained (using supervised learning) to predict facial landmarks. The model can be used to detect facial landmarks in a video, and the predicted landmark locations can be compared with ground truth data. The difference between the two can be calculated and provided as training feedback. The weights of one or more neural network nodes or other parameters of the model can be adjusted based on the feedback. Training is completed when the model prediction reaches an accuracy threshold.

[0062] In some embodiments, the user helps create the training data by providing user input. The user may be asked to identify the location of a face in a video frame. As a result of the user input, the training data may have an accurate identification of the location of the face, and use it to determine the face resampling keyframes for the video frame. In this example, and other examples of identifying the location of the face, the face detection does not include determining the identity, e.g., no facial recognition is performed. In addition, the training data may represent an application environment, e.g., where the application environment is an environment for detecting faces in a video call using a module device.

[0063] Based on the training data, the video analyzer 204 may include a trained model capable of generating face resampled keyframes based on images and videos. The trained model may be a temporal action localization model. In various embodiments, the video analyzer 204 may implement the trained model using a Bayesian classifier, a support vector machine, a neural network, or other learning techniques.

[0064] In some embodiments, the trained model may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., a "hidden layer" between an input layer and an output layer, each layer being a linear network), a convolutional neural network (e.g., a network that segments or divides the input data into multiple parts or slices, processes each slice separately using one or more neural network layers, and merges the results from the processing of each slice), a sequence-to-sequence neural network (e.g., a network that takes sequential data such as words in a sentence, frames in a video, etc. as input and generates a sequence of results as output), etc. The model form or structure may specify the connectivity between various nodes and the organization of nodes into layers. For example, the nodes of the first layer (e.g., the input layer) may receive data as input data or application data. Such data may include, for example, one or more pixels per node, for example, when the trained model is used for image analysis. Subsequent intermediate layers may receive the output of the nodes of the previous layer as input according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. The last layer (e.g., the output layer) generates an output of a face resampled keyframe and other metadata. For example, the output may be a face resampled keyframe, texture coordinates of the face resampled keyframe, and a time stamp of the video frame. In some embodiments, the model form or structure may also specify the number and / or type of nodes in each layer.

[0065] In various embodiments, the trained model may include multiple nodes arranged in layers of each model structure or form. In some embodiments, the node may be a computational node without memory, for example, configured to process one input unit to generate one output unit. The calculation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum using a bias or intercept value to generate a node output. In some embodiments, the calculation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, calculations of multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using separate processing units of a general processing unit or a dedicated neural circuit. In some embodiments, the node may include memory, for example, being able to store and use one or more earlier inputs when processing subsequent inputs. For example, a node with memory may include a long short-term memory (LSTM) node. The LSTM node may use memory to maintain a "state" that allows the node to work like a finite state machine (FSM). Models with such nodes can be used to process sequential data, such as frames in a video.

[0066] In some embodiments, the trained model may include embeddings or weights for each node. For example, the trained model may be initialized as a plurality of nodes organized into layers as specified by the model form or structure. Upon initialization, corresponding weights may be applied to connections between each pair of nodes connected in each model form, e.g., nodes in successive layers of a neural network. For example, corresponding weights may be randomly assigned or initialized to default values. The trained model may then be trained, e.g., using data, to generate results.

[0067] The face resampler 206 generates an interpolated face frame. In some embodiments, the resampler 206 includes an instruction set executable by the processor 235 to generate the interpolated face frame. In some embodiments, the recognition module 206 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0068] In some embodiments, the face resampler 206 is associated with a finite size queue or buffer that stores face resampled keyframes, texture coordinates, and timestamps of video frames. In some embodiments, the buffer has a finite storage capacity, and the face resampler 206 removes old face resampled keyframes each time a new face resampled keyframe is stored. For example, two face resampled keyframes may be used to generate an interpolated frame. Other examples are possible, such as three or four resampled keyframes to account for the rate of motion of objects between video frames.

[0069] For each instance that rendering of a video frame will occur, the face resampler 206 generates an interpolation amount t from the duration between face resampling keyframes (i.e., the time each frame is normally displayed and the current rendering time relative to that duration). For example, the face resampler 206 generates an interpolation amount t between 0 and 1 based on the current rendering time and the frame duration, which is defined as the time between a first face resampling keyframe and a second face resampling keyframe. In some embodiments, the interpolation amount is defined as:

[0070]

[0071] In other words, the interpolation amount is a number between 0 and 1 that represents the relative distance in time of the interpolated facial frame between the first and second video frames.

[0072] The face resampler 206 performs interpolation, which includes applying an interpolation algorithm to (1) decompose the face resampled key frame using the head transformation matrix to extract a translation vector, a rotation quaternion, and a scale vector; (2) linearly interpolate the translation vector; (3) interpolate the rotation quaternion using spherical linear interpolation to generate an interpolated rotation quaternion; (4) linearly interpolate the scale vector to generate an interpolated scale vector; (5) compose an interpolated translation, rotation, and scale matrix; (6) calculate a new interpolated position of the interpolated frame; and (7) render the interpolated face frame (i.e., upsampled frame) using the interpolated vertices.

[0073] The face resampler 206 may decompose the first face resampled keyframe and the second face resampled keyframe. For example, the face resampler 206 decomposes the first face resampled keyframe using the head transformation matrix M_first and decomposes the second face resampled keyframe using the head transformation matrix M_second to determine: (1) a translation vector having x, y, and z coordinates for the first face resampled keyframe T_first and the second face resampled keyframe T_second; (2) a rotation quaternion having x, y, z, and w coordinates for the first face resampled keyframe R_first and the second face resampled keyframe R_second; and (3) a scale vector having x, y, and z coordinates for the first face resampled keyframe S_first and the second face resampled keyframe S_second.

[0074] In some embodiments, the face resampler 206 linearly interpolates the translation vectors for the first face resampled keyframe T_first and the second face resampled keyframe T_second to generate an interpolated translation vector using the following equation:

[0075] T 插值 =T first *(1.0-t)+T second *t (2)

[0076] In some embodiments, the face resampler 206 interpolates R_first and R_second using spherical linear interpolation (SLERP) to generate an interpolated rotation quaternion using the following equation:

[0077] R 插值 =SLERP(R first ,R second ,t) (3)

[0078] In some embodiments, the face resampler 206 linearly interpolates the scaled vectors S_first and S_second using the following equation to generate an interpolated scaled vector:

[0079] S 插值=S first *(1.0-t)+S second *t (4)

[0080] In some embodiments, the face resampler 206 composes an interpolated translation rotation scale matrix that transforms from the model space to the face space from the above interpolated values ​​using the following equation:

[0081] M 插值 = translation vector (T 插值 )*Rotation quaternion (R 插值 )*Scaling vector (S 插值 )(5)

[0082] In some embodiments, the face resampler 206 calculates new interpolated positions of the first interpolated frame and the second interpolated frame for each vertex v_first in V_first (first facial landmark vertex) and each vertex v_second in V_second (second facial landmark vertex) by calculating linear interpolation model space vertices using the following equations, wherein the linear interpolation model space vertices include facial in-motion (e.g., smiling, raising eyebrows) that are not included in the overall transformation:

[0083] v 第一模型位置 = inverse(M first )*v first (6)

[0084] v 第二模型位置 = inverse(M second )*v second (7)

[0085] v 插值的模型位置 =v 第一模型位置 *(1.0-t)+v 第二模型位置 *t (8)

[0086] In some embodiments, the face resampler 206 calculates new interpolated face-space vertices shared between the first frame and the second frame using the following equation:

[0087] v 插值 =M 插值 *v 插值的模型位置 (9)

[0088] In some embodiments, the face resampler 206 uses the interpolated face-space vertices to render the upsampled frame.

[0089] The face resampler 206 generates an interpolated face frame for each face resampled key frame, wherein the texture coordinates in each interpolated face frame are equal to the texture coordinates of its corresponding face resampled key frame. The landmark vertices are interpolated according to the interpolation amount to create interpolated mesh vertices. The interpolated mesh vertices are a combination of the interpolated head transformation matrix and the mesh vertices from the face resampled key frame. The landmark vertices are the same between each interpolated face frame, but because the texture coordinates correspond to the corresponding face resampled key frame, different texture coordinates are applied to the landmark vertices. For example, the face resampler 206 creates a first interpolated face frame for a first face resampled key frame and a second interpolated face frame for a second face resampled key frame. The face in the first video frame and the face in the second video frame are effectively stretched so that they are aligned at the same intermediate position.

[0090] Go to Figure 3A , shows an example of a first face resampled keyframe 300. This is the input to the interpolation algorithm at time 0. Figure 3B An example of a second face resampled keyframe 325 is shown. This is the input to the interpolation algorithm at time 1. As shown, Figure 3A and 3B The landmark vertices (intersection points of the mesh) in are different. For example, in Figure 3A In the Figure 3B In the figure, the face is slightly rotated and Figure 3A In comparison, the mouth is less open, which causes the corresponding vertices to be in different positions.

[0091] Figure 3C An output image 350 is shown with texture from a first face resampled keyframe based on an interpolation grid warp without blending. Figure 3D An output image 375 is shown with texture from the second face resampled keyframe based on interpolation grid warping without blending. Figure 3C and 3D The landmark vertices in are the same, but because the texture changes between the two frames, different texture coordinates are applied.

[0092] Go to Figure 4A , shows an example of an input video frame 400 of a video sent at 7.5 FPS. Figure 4B An interpolated first face 425 and an interpolated second face 430 are shown at 240 FPS in accordance with some embodiments.

[0093] In some embodiments, the face resampler 206 interpolates a first background of a first face resampled key frame and a second background of a second face resampled key frame based on the interpolation amount. The face resampler 206 may determine the first background and the second background based on the position of the face in the first frame and the second frame determined by the face tracking algorithm. The face resampler 206 may interpolate the backgrounds of the key frames using alpha blending. In some embodiments, the blending may include a fade between the first background and the second background so that when the frames are displayed sequentially at a high frame rate, the upcoming smooth face rendered on top is less visually jittery to the viewer. In some embodiments, performing the interpolation may include buffering at least two frames used for the interpolation. Due to such buffering, in some embodiments, the playback of the video includes a single frame delay introduced into the video stream.

[0094] Go to Figure 5A , an example of an input video frame 500 at 7.5 FPS is shown according to some embodiments. Figure 5B A video frame 525 at 240 FPS is shown with the background faded in preparation for the upcoming smoothed face rendering on top.

[0095] The face resampler 206 renders the first interpolated face and the second interpolated face. In some embodiments, the face resampler 206 performs at least one of feathering of facial edges (which may make the edges of the interpolated facial frame less noticeable to a viewer) and fading between the first interpolated facial frame and the second interpolated facial frame, depending on the amount of interpolation.

[0096] Go to Fig. 6A , an example of an input video frame 600 transmitted at 7.5 FPS is shown according to some embodiments. Figure 6B An example blended face frame 625 is shown blended from a first interpolated face frame and a second interpolated face frame. The blended face frame (along with other blended frames) is rendered at 240 FPS and is Figure 6B is shown without background for illustration purposes. Figure 6B In the example shown, the face resampler 206 performs at least one of feathering and fading to smooth the face.

[0097] The face resampler 206 instructs the display 241 to display a final frame based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face. For example, the face resampler 206 places the smoothed face on top of a blended face frame having a blended background generated by a fade between the first background and the second background. The face resampler 206 uses the face tracking information to align the smoothed face with the blended face frame while maintaining the feathering of the blended face frame.

[0098] Go to Fig. 7A, shows an example of an input video frame 700 at 7.5 FPS. Figure 7B A final frame 725 displayed by the user device 115 is shown. The final frame 725 is an example of a blended facial frame combined with a blended background. As a result of inserting one or more blended facial frames as intermediate frames between the first frame and the second frame, the perceived frame rate is higher when the video is displayed and results in a positive user experience because the quality of the video is improved (e.g., abrupt motion transitions that may be perceived at lower frames are no longer perceptible), while the sender device does not need to send the video stream at a high frame rate, which requires greater network bandwidth. For example, the generation of interpolated frames as described herein enables playback of a low frame rate (e.g., 7.5 FPS) received video stream at a high frame rate (e.g., 30 FPS, 60 FPS, 240 FPS, etc.) by inserting interpolated frames between consecutive frames of the received video.

[0099] Go to Figure 8 , two frame sequences are shown. Frame sequence 810 is from the original video and includes frames 811, 812, 813, 814, 815, 816, 817, 818, and 819. When the video is transmitted at a low frame rate, only frames 811, 815, and 819 are included for display at the low frame rate without interpolation. Frames 812, 813, 814, 816, 817, and 818 are not included. This is problematic because when the video is displayed, the user may perceive sudden motion, such as the video jumping between frames, such as when a person's mouth suddenly closes between frames 811 and 815 and eyes change dramatically between frames 815 and 819.

[0100] Frame sequence 820 includes upsampled video and includes frames 811, 822, 823, 824, 815, 826, 827, 828, and 819 displayed at a higher frame rate. As described herein, additional frames are obtained by interpolation from a received video stream including frames 811, 815, and 819. The additional frames in frame sequence 820 are 822, 823, 824, 826, 827, and 828, and are obtained and added to the sequence by interpolation according to the techniques described herein. As a result of using interpolated frames, frame sequence 820 can be displayed at a higher frame rate (because intermediate frames 822-824 and 826-828 are available) without jitter, while the bandwidth used to receive the video remains the same as the bandwidth used for frame sequence 810.

[0101] The user interface module 208 generates graphical data for displaying the user interface. In some embodiments, the user interface module 208 includes an instruction set that can be executed by the processor 235 to generate graphical data. In some embodiments, the user interface module 208 is stored in the memory 237 of the computing device 200 and can be accessed and executed by the processor 235.

[0102] In some embodiments, the user interface module 208 generates graphical data for displaying the video at a perceived higher frame rate, for example, by inserting interpolated frames obtained from the face resampler 206 between consecutive frames of the received video. In some embodiments, the user interface module 208 also generates user interface options associated with the video application. For example, the user interface includes options for making a video call, adding other users to the video call, changing settings associated with the video call, etc.

[0103] In some embodiments, the number of intermediate frames generated between the first frame and the second frame is based on the difference between the frame rate of the received video and the frame rate of the video playback. For example, if the video is received at 10FPS and played at 30FPS, 2 intermediate frames are generated for each pair of consecutive frames of the video. In this case, if the original video has 100 frames (corresponding to 10 seconds of video), 200 intermediate frames are generated to obtain 300 frames of video, which can play back 10 seconds of video at 30FPS. The frame rate for video playback can be selected based on various factors, such as the refresh rate of the display 241, the computing power of the device 200, the motion rate in the received video, etc.

[0104] Example Method

[0105] Fig.9A and 9B A flow chart of an example method for generating an interpolated video frame is shown. The method 900 is performed by a video application 103 stored on a computing device 200 (eg, a user device 115, a video server 101, or a portion of a user device 115 and a portion of a video server 101).

[0106] At block 902, a set of video frames corresponding to a video is received. The set of video frames includes a first video frame and a second video frame, each of the first video frame and the second video frame each including a face, wherein the second video frame is subsequent to the first video frame, e.g., a timestamp associated with the second video frame has a higher value than a timestamp associated with the first video frame, such that during playback of the video, the first frame is displayed before the second frame is displayed.

[0107] At block 904, face tracking is performed on the first video frame to identify a first face resampling keyframe, the first face resampling keyframe comprising a first head transformation matrix and first facial landmark vertices. At block 906, face tracking is performed on the second video frame to identify a second face resampling keyframe, the second face resampling keyframe comprising a second head transformation matrix and second facial landmark vertices.

[0108] At block 908, an interpolation amount is derived. At block 910, a first interpolated facial frame is determined based on the first facial resampling keyframe and the interpolation amount. At block 912, a second interpolated facial frame is determined based on the second facial resampling keyframe and the interpolation amount. At block 914, a first background of the first facial resampling keyframe and a second background of the second facial resampling keyframe are interpolated based on the interpolation amount. In some embodiments, block 914 may be omitted. At block 916, the interpolated first face and the interpolated second face are rendered. For example, the interpolated first face and the interpolated second face are rendered as a blended interpolated face. At block 918, a final frame based on the interpolated first background, the interpolated second background, the interpolated first face, and the second interpolated second face is displayed. For example, the blended interpolated face is added to the blended background to form a final frame. The final frame is inserted at a corresponding position in the frame sequence, for example, between the first video frame and the second video frame, so that during playback, it is displayed in the order of the first frame, the final frame, and the second frame. If more than one interpolated frame is generated between the first video frame and the second video frame (e.g., when the interpolation amount is less than t) to allow playback at a higher frame rate (e.g., 30 frames per second from a 10 frames per second video), each interpolated frame is inserted at the corresponding position in the sequence.

[0109] In the above description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the specification. However, it is apparent to those skilled in the art that the present disclosure can be practiced without these specific details. In some instances, structures and devices are shown in block diagram form to avoid obscuring the description. For example, the above embodiments may be described primarily with reference to user interfaces and specific hardware. However, the embodiments may be applicable to any type of computing device that can receive data and commands, and any peripheral device that provides services.

[0110] Mentioning "some embodiments" or "some examples" in the specification means that a particular feature, structure or characteristic described in conjunction with the embodiments or examples may be included in at least one implementation of the specification. The phrase "in some embodiments" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0111] Some portions of the above detailed description are rendered in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, considered to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0112] It should be remembered, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, as will be apparent from the following discussion, it should be understood that throughout the specification, discussions using terms including "processing" or "computing" or "calculating" or "determining" or "displaying" and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0113] Embodiments of the present specification may also relate to a processor for performing one or more steps of the above method. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to any type of disk, including an optical disk, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory including a USB key with non-volatile memory, or any type of medium suitable for storing electronic instructions. Each is coupled to a computer system bus.

[0114] This specification can take the form of some complete hardware embodiments, some complete software embodiments or some embodiments including hardware and software elements. In some embodiments, the specification is implemented with software, which includes but is not limited to firmware, resident software, microcode, etc.

[0115] Furthermore, the description may take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in conjunction with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any means that can contain, store, communicate, propagate or transport the program for use by or in conjunction with an instruction execution system, apparatus or device.

[0116] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to a memory element via a system bus. The memory element may include local memory used during actual execution of the program code, bulk storage, and cache memories that provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.

[0117] In the cases discussed above where the system collects or uses personal information, the system provides the user with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current location), or to control whether and / or how content that may be more relevant to the user is received from the server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be summarized where location information is obtained (e.g., at the city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user may have control over how information is collected about the user and server usage.

Claims

1. A method for rendering interpolated faces from a set of video frames, characterized in that include: Receiving a set of video frames corresponding to a video, the set of video frames comprising a first video frame and a second video frame, the first video frame and the second video frame each comprising a face, wherein the second video frame is subsequent to the first video frame; Performing facial tracking on the first video frame to identify a first facial resampling key frame, the first facial resampling key frame comprising a first head transformation matrix and first facial landmark vertices; Performing facial tracking on the second video frame to identify a second facial resampling key frame, the second facial resampling key frame comprising a second head transformation matrix and second facial landmark vertices, wherein the head transformation matrix represents a head transformation from an initial model space to a facial space, and the facial landmark vertices describe landmark vertices in the facial space; generating an interpolation amount between 0 and 1 based on a current rendering time and a frame duration, the frame duration being a time between the first face resampling key frame and the second face resampling key frame; Determining a first interpolated facial frame based on the first facial resampling key frame and the interpolation amount; Determining a second interpolated facial frame based on the second facial resampling key frame and the interpolation amount; interpolating a first background of the first face resampling key frame and a second background of the second face resampling key frame based on the interpolation amount; rendering an interpolated first face in the first interpolated facial frame and an interpolated second face in the second interpolated facial frame; and A final frame based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face is displayed.

2. The method according to claim 1, characterized in that Also includes: mixing the interpolated first background with the interpolated second background to obtain a mixed background; blending the interpolated first face with the interpolated second face to obtain a blended interpolated face; and The final frame is generated by placing a smoothed face on top of the blended interpolated face and the blended background, the smoothed face being obtained by feathering or fading the blended interpolated face.

3. The method according to claim 1, characterized in that Determining the first interpolated facial frame includes: Using the first head transformation matrix to extract a translation vector, a rotation quaternion, and a scaling vector; interpolating the translation vector using linear interpolation; interpolating the rotation quaternion using linear interpolation to generate an interpolated rotation quaternion; interpolating the scaling vector using linear interpolation to generate an interpolated scaling vector; composing an interpolated translation-rotation-scaling matrix based on the interpolated translation vector, the interpolated rotation quaternion and the interpolated scaling vector; and An interpolated position of the interpolated first face is calculated using the interpolated translation-rotation-scaling matrix.

4. The method according to claim 1, characterized in that Interpolating the first background and the second background is done using alpha blending.

5. The method according to claim 1, characterized in that Rendering includes at least one of feathering an edge of the face or fading between the first interpolated facial frame and the second interpolated facial frame based on the interpolation amount.

6. The method according to claim 1, characterized in that: Performing facial tracking on the first video frame further includes determining first texture coordinates and a timestamp of the first facial resampled keyframe; and The first texture coordinate is applied to the first facial landmark vertex.

7. The method according to claim 1, characterized in that The first background and the face are identified by performing red-green-blue (RGB) face tracking on the first video frame.

8. A non-transitory computer readable medium having instructions stored thereon, characterized in that: When executed by one or more computers, the instructions cause the one or more computers to perform operations including: Receiving a set of video frames corresponding to a video, the set of video frames comprising a first video frame and a second video frame, the first video frame and the second video frame each comprising a face, wherein the second video frame is subsequent to the first video frame; Performing facial tracking on the first video frame to identify a first facial resampling key frame, the first facial resampling key frame comprising a first head transformation matrix and first facial landmark vertices; performing facial tracking on the second video frame to identify a second facial resampling key frame, the second facial resampling key frame comprising a second head transformation matrix and second facial landmark vertices, wherein the head transformation matrix represents a head transformation from an initial model space to a facial space, and the facial landmark vertices describe landmark vertices in the facial space; generating an interpolation amount between 0 and 1 based on a current rendering time and a frame duration, the frame duration being a time between the first face resampling key frame and the second face resampling key frame; Determining a first interpolated facial frame based on the first facial resampling key frame and the interpolation amount; Determining a second interpolated facial frame based on the second facial resampling key frame and the interpolation amount; interpolating a first background of the first face resampling key frame and a second background of the second face resampling key frame based on the interpolation amount; rendering an interpolated first face in the first interpolated facial frame and an interpolated second face in the second interpolated facial frame; and A final frame based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face is displayed.

9. The computer-readable medium of claim 8, wherein: The operations also include: mixing the interpolated first background with the interpolated second background to obtain a mixed background; blending the interpolated first face with the interpolated second face to obtain a blended interpolated face; and The final frame is generated by placing a smoothed face on top of the blended interpolated face and the blended background, the smoothed face being obtained by feathering or fading the blended interpolated face.

10. The computer-readable medium of claim 8, wherein: Determining the first interpolated facial frame includes: Using the first head transformation matrix to extract a translation vector, a rotation quaternion, and a scaling vector; interpolating the translation vector using linear interpolation; interpolating the rotation quaternion using linear interpolation to generate an interpolated rotation quaternion; interpolating the scaling vector using linear interpolation to generate an interpolated scaling vector; composing an interpolated translation-rotation-scaling matrix based on the interpolated translation vector, the interpolated rotation quaternion and the interpolated scaling vector; and An interpolated position of the interpolated first face is calculated using the interpolated translation-rotation-scaling matrix.

11. The computer-readable medium of claim 8, wherein: Interpolating the first background and the second background is done using alpha blending.

12. A system for rendering interpolated faces from a set of video frames, characterized in that include: one or more processors; and A memory storing instructions that, when executed by the one or more processors, cause the one or more processors to: Receiving a set of video frames corresponding to a video, the set of video frames comprising a first video frame and a second video frame, the first video frame and the second video frame each comprising a face, wherein the second video frame is subsequent to the first video frame; Performing facial tracking on the first video frame to identify a first facial resampling key frame, the first facial resampling key frame comprising a first head transformation matrix and first facial landmark vertices; performing facial tracking on the second video frame to identify a second facial resampling key frame, the second facial resampling key frame comprising a second head transformation matrix and second facial landmark vertices, wherein the head transformation matrix represents a head transformation from an initial model space to a facial space, and the facial landmark vertices describe landmark vertices in the facial space; generating an interpolation amount between 0 and 1 based on a current rendering time and a frame duration, the frame duration being a time between the first face resampling key frame and the second face resampling key frame; Determining a first interpolated facial frame based on the first facial resampling key frame and the interpolation amount; Determining a second interpolated facial frame based on the second facial resampling key frame and the interpolation amount; interpolating a first background of the first face resampling key frame and a second background of the second face resampling key frame based on the interpolation amount; rendering an interpolated first face in the first interpolated facial frame and an interpolated second face in the second interpolated facial frame; and A final frame based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face is displayed.

13. The system of claim 12, wherein: The operations also include: mixing the interpolated first background with the interpolated second background to obtain a mixed background; blending the interpolated first face with the interpolated second face to obtain a blended interpolated face; and The final frame is generated by placing a smoothed face on top of the blended interpolated face and the blended background, the smoothed face being obtained by feathering or fading the blended interpolated face.

14. The system of claim 12, wherein: Determining the first interpolated facial frame includes: Using the first head transformation matrix to extract a translation vector, a rotation quaternion, and a scaling vector; interpolating the translation vector using linear interpolation; interpolating the rotation quaternion using linear interpolation to generate an interpolated rotation quaternion; Using linear interpolation to scale the vector to generate an interpolated scale vector; composing an interpolated translation-rotation-scaling matrix based on the interpolated translation vector, the interpolated rotation quaternion and the interpolated scaling vector; and An interpolated position of the interpolated first face is calculated using the interpolated translation-rotation-scaling matrix.

Images