Robust Facial Animation from Video and Audio

Abstract

Implementations described herein relate to a method, system, and computer-readable medium for generating animation for a 3D avatar from input video and audio captured on a client device. A camera may capture facial video, and a trained face detection model and a trained regression model output a set of video FACS weights, head pose, and facial landmarks that will be converted into animation for the 3D avatar. In addition, a microphone may capture audio emitted by a user, and a trained face motion detection model and a trained regression model output a set of audio FACS weights. In addition, a blending term is provided for identifying time progression in the audio. A modular blending component blends the video FACS weights and audio FACS weights based on the blending term to create final FACS weights for animating the user's avatar, a character's clothing, or another animation-enabled construct.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 440,993, entitled "ROBUST FACIAL ANIMATION FROM VIDEO USING NEURAL NETWORKS," filed January 25, 2023, the entire contents of which are incorporated herein by reference.

[0002] FIELD Embodiments relate generally to computer-based virtual experiences, and more particularly to methods, systems, and computer-readable media for robust facial animation from real-time video. [Background technology]

[0003] Some online platforms (e.g., gaming platforms, media exchange platforms, etc.) allow users to connect with each other, interact with each other (e.g., in-game), create games, and share information with each other over the Internet. Users of online platforms may participate in multiplayer gaming or virtual environments (e.g., three-dimensional environments), design custom gaming environments, design characters and avatars, decorate avatars, exchange virtual items / objects with other users, communicate with other users using voice or text messaging, etc. Such environments, such as metaverse or multiverse environments, may also allow participating users to share, sell, or trade objects they create with other users. Summary of the Invention [Problem to be solved by the invention]

[0004] Users who interact with one another may use interactive interfaces that involve the presentation of the users' avatars. Animating the avatars traditionally may involve having the user input the desired gestures, movements, and other similar pre-configured animation details, and presenting the animation based on the user's input. Such traditional approaches have drawbacks.

[0005] The background description provided herein is intended to provide a context for the present disclosure. None of the work of the inventors named herein that falls within the scope of what is described in this background section, nor any aspect of the description that may not otherwise qualify as prior art at the time of filing, is admitted expressly or implicitly as prior art to the present disclosure. [Means for solving the problem]

[0006]

[0003] According to one embodiment, a computer-implemented method includes receiving input video frames, receiving input audio frames and mixing terms, the input audio frames including audio associated with the input video frames, obtaining video Facial Activity Coding System (FACS) weights from a first trained machine learning model based on the input video frames, obtaining audio FACS weights from a second trained machine learning model based on the input audio frames, combining the video FACS weights and the audio FACS weights to obtain final FACS weights, the combination being based at least in part on the mixing terms, and outputting the final FACS weights to drive facial animation of a 3D model.

[0007] Various implementations and variations of the computer-implemented method are disclosed.

[0008] In some implementations, the first trained machine learning model and the second trained machine learning model are trained in a two-stage semi-supervised training process.

[0009] In some implementations, the first trained machine learning model comprises at least one encoder and at least three task-specific decoders.

[0010] In some implementations, a first task-specific decoder of the at least three task-specific decoders is configured to output a predicted head pose, a second task-specific decoder of the at least three task-specific decoders is configured to output a probability that a face is visible in an input video frame of the received input video frames, and a third task-specific decoder of the at least three task-specific decoders is configured to output facial landmarks.

[0011] In some implementations, at least one of the at least three task-specific decoders comprises a causal convolutional layer applied across the time dimension.

[0012] In some implementations, the second trained machine learning model comprises at least one encoder and at least two task-specific decoders.

[0013] In some implementations, a first task-specific decoder of the at least two task-specific decoders is configured to output audio FACS weights, and a second task-specific decoder of the at least two task-specific decoders is configured to output mixing terms.

[0014] According to another aspect, a system is provided, comprising: a memory having instructions stored thereon; and a processing device coupled to the memory, the processing device configured to access the memory and execute the instructions, causing the processing device to perform operations comprising: receiving input video frames, receiving input audio frames and mixing terms, the input audio frames including audio associated with the input video frames, obtaining video Facial Activity Coding System (FACS) weights from a first trained machine learning model based on the input video frames, obtaining audio FACS weights from a second trained machine learning model based on the input audio frames, combining the video FACS weights and the audio FACS weights to obtain final FACS weights, the combining based at least in part on the mixing terms, and outputting the final FACS weights to drive facial animation of a 3D model.

[0015] Various implementations and variations of the system are disclosed.

[0016] In some implementations, the first trained machine learning model and the second trained machine learning model are trained in a two-stage semi-supervised training process.

[0017] In some implementations, the first trained machine learning model comprises at least one encoder and at least three task-specific decoders.

[0018] In some implementations, a first task-specific decoder of the at least three task-specific decoders is configured to output a predicted head pose, a second task-specific decoder of the at least three task-specific decoders is configured to output a probability that a face is visible in an input video frame of the received input video frames, and a third task-specific decoder of the at least three task-specific decoders is configured to output facial landmarks.

[0019] In some implementations, at least one of the at least three task-specific decoders comprises a causal convolutional layer applied across the time dimension.

[0020] In some implementations, the second trained machine learning model comprises at least one encoder and at least two task-specific decoders.

[0021] In some implementations, a first task-specific decoder of the at least two task-specific decoders is configured to output audio FACS weights, and a second task-specific decoder of the at least two task-specific decoders is configured to output mixing terms.

[0022] According to another aspect, a non-transitory computer-readable medium is provided that, when executed by a processing device, causes the processing device to perform operations comprising receiving input video frames, receiving input audio frames and mixing terms, where the input audio frames include audio associated with the input video frames, obtaining video Facial Activity Coding System (FACS) weights from a first trained machine learning model based on the input video frames, obtaining audio FACS weights from a second trained machine learning model based on the input audio frames, combining the video FACS weights and the audio FACS weights to obtain final FACS weights, where the combining is based at least in part on the mixing terms, and outputting the final FACS weights to drive facial animation of a 3D model.

[0023] Various implementations and variations of non-transitory computer-readable media are disclosed.

[0024] In some implementations, the first trained machine learning model and the second trained machine learning model are trained in a two-stage semi-supervised training process.

[0025] In some implementations, the first trained machine learning model comprises at least one encoder and at least three task-specific decoders.

[0026] In some implementations, a first task-specific decoder of the at least three task-specific decoders is configured to output a predicted head pose, a second task-specific decoder of the at least three task-specific decoders is configured to output a probability that a face is visible in an input video frame of the received input video frames, and a third task-specific decoder of the at least three task-specific decoders is configured to output facial landmarks.

[0027] In some implementations, at least one of the at least three task-specific decoders comprises a causal convolutional layer applied across the time dimension.

[0028] In some implementations, the second trained machine learning model comprises at least one encoder and at least two task-specific decoders, a first task-specific decoder of the at least two task-specific decoders configured to output audio FACS weights, and a second task-specific decoder of the at least two task-specific decoders configured to output mixing terms.

[0029] According to yet other aspects, portions, features, and implementations of the systems, methods, and non-transitory computer-readable media may be combined to form additional aspects, including some aspects that omit and / or modify some or portions of individual components or features, and may include additional components or features, and / or other modifications, and all such modifications are within the scope of the present disclosure. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 illustrates an exemplary network environment, according to some implementations. [Figure 2] FIG. 1 is a diagram of a facial animation engine, according to some implementations. [Figure 3] FIG. 1 is a diagram of a video animation component, according to some implementations. [Figure 4] FIG. 1 is a diagram of an audio animation component, according to some implementations. [Figure 5A] 1 is a flowchart of an example method for training portions of a video animation component, according to some implementations. [Figure 5B] 1 is a flowchart of an example method for training portions of a video animation component, according to some implementations. [Figure 6A] 1 is a flowchart of an example method for training portions of an audio animation component, according to some implementations. [Figure 6B] 1 is a flowchart of an example method for training portions of an audio animation component, according to some implementations. [Figure 6C] FIG. 1 illustrates an exemplary training environment for training portions of an audio animation component, according to some implementations. [Figure 7] 1 is a flowchart of an exemplary method for real-time robust facial animation from video and audio, according to some implementations. [Figure 8] FIG. 1 is a block diagram illustrating an example computing device that may be used to implement one or more features described herein, according to some implementations. DETAILED DESCRIPTION OF THE INVENTION

[0031] One or more implementations described herein relate to real-time, robust animation from video and audio. Functionality may include automatically creating animations of three-dimensional (3D) avatars based on input video and audio received from client devices.

[0032] The features described herein provide for automatic face detection in video, automatic detection / identification of facial movements from phonemes in audio, regression of parameters used to animate a 3D avatar from the detected faces and facial movements, and creation of animation of the 3D avatar based on the parameters. The trained model receives audio and video inputs and outputs audio and video Facial Activity Coding System (FACS) weights. A modularity mixing component combines the audio FACS weights and video FACS weights to obtain final FACS weights for facial animation.

[0033] The trained models can be deployed on client devices for use by users who want automatically created animations for their associated avatars. The client devices can also be configured to operatively communicate with online platforms, such as virtual experience (VE) platforms, so that their associated avatars can be richly animated for presentation in communication interfaces (e.g., video chat), richly animated within the virtual experience (e.g., a richly animated face on a surrogate virtual body), richly animated in animated videos sent to other users (e.g., by sending recordings of the animated avatars through chat or other features), and richly animated in other parts of the online platform.

[0034] Online virtual experience platforms (also called "user-generated content platforms" or "user-generated content systems") provide various ways for users to interact with each other. For example, users of an online virtual experience platform may create experiences, games, or other content or resources (e.g., characters, graphics, items, etc. for gameplay within a virtual world) within the platform.

[0035] Users of an online virtual experience platform may collaborate toward a common goal in playing or creating a game, share various virtual items, send electronic messages to one another, etc. Users of an online virtual experience platform may, for example, interact with the environment and play the game, including characters (avatars) or other game objects and mechanisms. The online virtual experience platform may also allow users of the platform to communicate with one another. For example, users of an online virtual experience platform may communicate with one another using voice messages (e.g., via voice chat), text messaging, video messaging, or a combination of the above. Some online virtual experience platforms may provide a virtual three-dimensional environment in which users can represent themselves using avatars or virtual representations of themselves.

[0036] To help enhance the entertainment value of the online virtual experience platform, the platform may provide a facial animation engine to facilitate automatically animating avatars. The facial animation engine may allow users to request or select options for animation, including, for example, animating the avatar's face or body based on a live video feed and / or live audio transmitted from the client device.

[0037] For example, a user may allow an application on a user device associated with an online virtual experience platform to access the camera and microphone. Video produced by the camera may be interpreted to extract gestures or other information that facilitates animation of an avatar based on the extracted gestures. Additionally, audio captured by the microphone may be interpreted to extract facial movements. The user may also enhance facial animation through directed control input to move other body parts or exaggerate facial gestures.

[0038] In situations where some implementations discussed herein may obtain or use user data (e.g., user images, user video, user audio, user attributes, user behavior data on the platform, user search history, purchased and / or viewed items, user friends on the platform, etc.), the user is given the option to control whether and how such information is collected, stored, or used. That is, the implementations discussed herein collect, store, and / or use user information only after receiving explicit user approval and in accordance with applicable regulations.

[0039] The user is given control over whether the program or feature collects user information about that particular user or other users that is important to the program or feature. Each user about whom information is to be collected is presented with options (e.g., via a user interface) that allow the user to exercise control over information collection that is important to that user and provide permission or approval for whether and what portion of the information will be collected. Additionally, some data may be modified in one or more ways before storage or use so that personally identifiable information is removed. As one example, a user's identification information may be modified (e.g., by substitution with a pseudonym, numeric values, etc.) so that personally identifiable information cannot be determined. In another example, a user's geographic location may be generalized to a broader area (e.g., city, zip code, state, country, etc.).

[0040] As described herein, automatic face detection in video, automatic facial movement detection / identification from phonemes in audio, regression of parameters used to animate a 3D avatar from detected faces and facial movements, and creation of 3D avatar animation based on the parameters may be provided by a trained model. Model training may include two or more stages, for example, where a portion of the model (e.g., an encoder) is trained first, followed by training of another portion of the model (e.g., one or more decoders) using the output from the trained encoder. Such two-stage training may overcome drawbacks associated with typical automatic animation, including jitter, delay, exaggerated movements, minimal movements, unrealistic movements, and others. Technical effects and benefits of two-stage training may include a reduction in training cycles, which may improve energy consumption (e.g., by reducing training time, energy usage of computational resources used for training is reduced), reduce computation cycles (e.g., by reducing training time, the total number of computation cycles may be reduced), improve storage requirements (e.g., by using both synthetic and real data, some of the training data may be created during training rather than stored), and others.

[0041] As further described herein, the deployed trained model may operate to provide realistic and stable automatic avatar animation. For example, realistic animation may refer to animation with natural-looking movements where facial movements and audio are matched. For example, stable animation may refer to animation without jarring transitions between successive frames in a sequence of animation frames. The trained model may overcome drawbacks associated with typical automatic animation, including jitter, delay, exaggerated movements, minimal movements, unrealistic movements, and others. Technical effects and benefits of the deployed trained model may include improved energy consumption at client devices (e.g., deploying a trained model with adjustable level of detail may reduce computational cycles and therefore energy usage at client devices), improved energy consumption at server devices (e.g., deploying a trained model with improved efficiency of FACS output generation may reduce computational cycles and therefore energy usage at server devices), and others.

[0042] These and other advantages of the present disclosure will be apparent from the included description and associated drawings. Turning now to Figure 1, an exemplary system architecture in which models may be trained and / or deployed is detailed.

[0043] Figure 1: Exemplary system architecture 1 illustrates an exemplary network environment 100 according to some implementations of the present disclosure. The network environment 100 (also referred to herein as a “system”) includes an online virtual experience platform 102, a first client device 110, and a second client device 116 (collectively referred to herein as “client devices 110 / 116”), all connected via a network 122. The online virtual experience platform 102 may include, among other things, a virtual experience (VE) engine 104, one or more virtual experiences 105, a communications engine 106, a facial animation engine 107, and a data store 108. The client device 110 may include a virtual experience application 112. The client device 116 may include a virtual experience application 118. Users 114 and 120 may use the client devices 110 and 116, respectively, to interact with the online virtual experience platform 102.

[0044] For illustrative purposes, a network environment 100 is provided. In some implementations, the network environment 100 may include the same, fewer, more, or different elements configured in the same or different manner as shown in FIG.

[0045] In some implementations, the network 122 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wired network (e.g., an Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi network, or a wireless LAN (WLAN)), a cellular network (e.g., a Long Term Evolution (LTE) network), a router, a hub, a switch, a server computer, or a combination thereof.

[0046] In some implementations, the data store 108 may be a non-transitory computer-readable memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. The data store 108 may also include multiple storage components (e.g., multiple drives or multiple databases), which may span multiple computing devices (e.g., multiple server computers).

[0047] In some implementations, the online virtual experience platform 102 may include a server having one or more computing devices (e.g., a cloud computing system, a rack-mounted server, a server computer, a cluster of physical servers, a virtual server, etc.) In some implementations, the server may be included in the online virtual experience platform 102, may be a separate system, or may be part of another system or platform.

[0048] In some implementations, the online virtual experience platform 102 may include one or more computing devices (such as a rack-mounted server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memory, databases), networks, software components, and / or hardware components that may be used to perform operations on the online virtual experience platform 102 and provide users with access to the online virtual experience platform 102. The online virtual experience platform 102 may also include a website (e.g., one or more web pages) or application backend software that may be used to provide users with access to content provided by the online virtual experience platform 102. For example, users may access the online virtual experience platform 102 using virtual experience applications 112 / 118 on client devices 110 / 116, respectively.

[0049] In some implementations, the online virtual experience platform 102 may include some type of social network that provides connections between users, or some type of user-generated content system that allows users (e.g., end users or consumers) to communicate with other users via the online virtual experience platform 102, where the communication may include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., synchronous and / or asynchronous text-based communication). In some implementations of the present disclosure, a "user" may be represented as a single individual. However, other implementations of the present disclosure encompass a "user" (e.g., a creating user) being a set of users or an entity controlled by an automated source. For example, a set of individual users that federate as a community or group in a user-generated content system may be considered a "user."

[0050] In some implementations, the online virtual experience platform 102 may be a virtual gaming platform. For example, the gaming platform may offer single-player or multiplayer games to a community of users who may access or interact with games (e.g., user-generated games or other games) using client devices 110 / 116 over the network 122. In some implementations, the games (also referred to herein as “video games,” “online games,” or “virtual games”) may be, for example, two-dimensional (2D) games, three-dimensional (3D) games (e.g., 3D user-generated games), virtual reality (VR) games, or augmented reality (AR) games. In some implementations, users may find games and game items and participate in gameplay with other users in one or more games. In some implementations, games may be played in real time with other users of the game.

[0051] In some implementations, other collaboration platforms may be used with the robust animation functionality described herein instead of or in addition to the online virtual experience platform 102. For example, social networking platforms, video chat platforms, messaging platforms, user content creation platforms, virtual meeting platforms, etc. may be used with the robust animation functionality described herein to facilitate fast, robust, and accurate representation of a user's facial movements to a virtual avatar based on input video and / or audio.

[0052] In some implementations, "gameplay" may refer to the interaction of one or more players using a client device (e.g., 110 and / or 116) within a game or experience (e.g., VE 105), or the presentation of interactions on a display or other output device of client device 110 or 116.

[0053] One or more virtual experiences 105 are provided by an online virtual experience platform. In some implementations, the virtual experiences 105 may include electronic files that may be executed or loaded using software, firmware, or hardware configured to present virtual content (e.g., digital media items) to an entity. In some implementations, a virtual experience application 112 / 118 may execute and render the virtual experience 105 with respect to a virtual experience engine 104. In some implementations, the virtual experiences 105 may have a common set of rules or a common goal, and the environments of the virtual experiences 105 share that common set of rules or common goal. In some implementations, different virtual experiences may have different rules or goals from one another. Similarly, or alternatively, some virtual experiences may have no goal at all; the purpose is for users to socially interact.

[0054] In some implementations, a virtual experience may have one or more environments (also referred to herein as "gaming environments" or "virtual environments"), and the environments may be connected. An example of an environment may be a three-dimensional (3D) environment. One or more environments of a virtual experience 105 may be collectively referred to herein as a "world" or a "virtual world" or a "virtual universe" or a "metaverse." An example of a world may be the 3D world of the virtual experience 105. For example, a user may build a virtual environment that is connected to another virtual environment created by another user. Characters of a virtual experience may cross virtual boundaries into adjacent virtual environments.

[0055] It may be noted that a 3D environment or world uses graphics that use three-dimensional representations of geometric data that represent the virtual content (or at least present the content in a way that appears to be 3D content, regardless of whether a 3D representation of the geometric data is used). A 2D environment or world uses graphics that use two-dimensional representations of geometric data that represent the virtual content.

[0056] In some implementations, the online virtual experience platform 102 can host one or more virtual experiences 105 and enable users to interact with the virtual experiences 105 (e.g., explore games, VE-related content, or other content) using virtual experience applications 112 / 118 on client devices 110 / 116. Users (e.g., 114 and / or 120) of the online virtual experience platform 102 can play, create, interact with, or build the virtual experiences 105, explore the virtual experiences 105, communicate with other users, create and build objects (e.g., also referred to herein as “items” or “game objects” or “virtual game items”) of the virtual experience 105, and / or explore objects. For example, in creating a user-generated virtual item, a user may, among other things, create a character, decorations for a character, one or more virtual environments for an interactive experience, or build a structure to be used in the virtual experience 105.

[0057] In some implementations, users may buy, sell, or trade virtual objects, such as in-platform currency (e.g., virtual currency), with other users of the online virtual experience platform 102. In some implementations, the online virtual experience platform 102 may transmit virtual content to a virtual experience application (e.g., 112, 118). In some implementations, virtual content (also referred to herein as "content") may refer to any data or software instructions (e.g., virtual objects, experiences, user information, videos, images, commands, media items, etc.) related to the online virtual experience platform 102 or a virtual experience application.

[0058] In some implementations, a virtual object (also referred to herein as an "item" or "object" or "virtual game item") may refer to an object used, created, shared, or otherwise depicted in a virtual experience application 105 of the online virtual experience platform 102 or a virtual experience application 112 or 118 of a client device 110 / 116. For example, a virtual object may include parts, models, characters, tools, weapons, clothing, buildings, vehicles, currency, plants, animals, components of the foregoing (e.g., windows of a building), etc.

[0059] It may be noted that the online virtual experience platform 102 hosting the virtual experience 105 is provided for purposes of illustration and not limitation. In some implementations, the online virtual experience platform 102 may host one or more media items, which may include communication messages from one user to one or more other users. Media items may include, but are not limited to, digital videos, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, e-books, e-magazines, e-newspapers, digital audiobooks, e-journals, web blogs, real simple syndication (RSS) feeds, e-comics, software applications, etc. In some implementations, the media items may be electronic files that may be executed or loaded using software, firmware, or hardware configured to present the digital media items to entities.

[0060] In some implementations, a virtual experience 105 may be associated with a particular user or a particular group of users (e.g., a private experience), or may be made generally available to users of the online virtual experience platform 102 (e.g., a public experience). In some implementations where the online virtual experience platform 102 associates one or more virtual experiences 105 with a particular user or group of users, the online virtual experience platform 102 may associate the virtual experience 105 with a particular user using user account information (e.g., a user account identifier such as a username and password). Similarly, in some implementations, the online virtual experience platform 102 may associate a particular developer or group of developers with a virtual experience 105 using developer account information (e.g., a developer account identifier such as a username and password).

[0061] In some implementations, the online virtual experience platform 102 or the client device 110 / 116 may include a virtual experience engine 104 or a virtual experience application 112 / 118. The virtual experience engine 104 may include a virtual experience application similar to the virtual experience application 112 / 118. In some implementations, the virtual experience engine 104 may be used for the development or execution of the virtual experience 105. For example, the virtual experience engine 104 may include, among other things, a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), an audio engine, scripting functions, an animation engine, an artificial intelligence engine, networking functions, streaming functions, memory management functions, threading functions, scene graph functions, or video support for cinematics. Components of the virtual experience engine 104 may generate commands (e.g., rendering commands, collision commands, physics commands, etc.) that help compute and render the virtual experience. In some implementations, the virtual experience applications 112 / 118 of the client devices 110 / 116 may each operate independently, in cooperation with the virtual experience engine 104 of the online virtual experience platform 102, or a combination of both.

[0062] In some implementations, both the online virtual experience platform 102 and the client devices 110 / 116 run virtual experience engines (104, 112, and 118, respectively). The online virtual experience platform 102, using the virtual experience engine 104, may perform some or all virtual experience engine functions (e.g., generate physics commands, rendering commands, etc.) or offload some or all virtual experience engine functions to the virtual experience engine 104 of the client device 110. In some implementations, each virtual experience 105 may have a different ratio between virtual experience engine functions running on the online virtual experience platform 102 and virtual experience engine functions running on the client devices 110 and 116.

[0063] For example, the virtual experience engine 104 of the online virtual experience platform 102 may be used to generate physics commands when there is a collision between at least two game objects, while additional virtual experience engine functions (e.g., rendering generation commands) may be offloaded to the client device 110. In some implementations, the ratio of virtual experience engine functions performed by the online virtual experience platform 102 to those performed by the client device 110 may be changed (e.g., dynamically) based on interactivity conditions. For example, if the number of users participating in the virtual experience 105 exceeds a threshold number, the online virtual experience platform 102 may execute one or more virtual experience engine functions that were previously performed by the client device 110 or 116.

[0064] For example, a user may be interacting with the virtual experience 105 on client devices 110 and 116 and may send control instructions (e.g., user input such as right, left, up, down, user selection, or character position and velocity information) to the online virtual experience platform 102. After receiving the control instructions from client devices 110 and 116, the online virtual experience platform 102 may send interaction instructions (e.g., position and velocity information for characters participating in the virtual experience, or commands such as rendering commands, collision commands, etc.) to client devices 110 and 116 based on the control instructions. For example, the online virtual experience platform 102 may perform one or more logical operations on the control instructions (e.g., using the virtual experience engine 104) to generate interaction instructions for client devices 110 and 116. In other cases, the online virtual experience platform 102 may pass one or more of the control instructions from one client device 110 to other client devices (e.g., 116) participating in the virtual experience 105. Client devices 110 and 116 may use the instructions to render the experience for presentation on the displays of client devices 110 and 116 .

[0065] In some implementations, control instructions may refer to instructions indicating in-experience behavior of a user's character or avatar. For example, control instructions may include user inputs for controlling in-experience behavior, such as right, left, up, down, user selection, gyroscope position and orientation data, force sensor data, etc. Control instructions may include character position and velocity information. In some implementations, control instructions are sent directly to the online virtual experience platform 102. In other implementations, control instructions may be sent from the client device 110 to another client device (e.g., 116), which generates play instructions using the local virtual experience engine 104. Control instructions may include instructions for playing voice communication messages or other sounds from another user on an audio device (e.g., speaker, headphones, etc.), instructions for moving a character or avatar, and other instructions.

[0066] In some implementations, interaction instructions or playback instructions may refer to instructions that enable client device 110 (or 116) to render the movement of elements of a virtual experience, such as a multiplayer game. The instructions may include one or more of user input (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.). As described more fully herein, other instructions may include facial animation instructions extracted through analysis of input video of a user's face to direct in real time the virtual facial animation of a virtual avatar representing the user. Thus, while interaction instructions may include user input to directly control some body movement of a character, interaction instructions may also include gestures extracted from video of the user.

[0067] In some implementations, characters (or virtual objects in general) are constructed from components that automatically combine to aid the user in editing, and one or more of these components may be selected by the user. One or more characters (also referred to herein as “avatars” or “models”) may be associated with a user, who may control the characters to facilitate the user's interaction with the virtual experience 105. In some implementations, characters may include components such as body parts (e.g., hair, arms, legs, etc.) and accessories (e.g., T-shirts, glasses, decorative images, tools, etc.). In some implementations, customizable character body parts include, among others, head type, body part type (arms, legs, torso, and hands), face type, hair type, and skin type. In some implementations, customizable accessories include clothing (e.g., shirts, pants, hats, shoes, glasses, etc.), weapons, or other tools.

[0068] In some implementations, the user may also adjust the dimensions of the character (e.g., height, width, or depth) or the dimensions of components of the character. In some implementations, the user may adjust the proportions of the character (e.g., stocky, anatomical, etc.). It may be noted that in some implementations, the character may not include a character object (e.g., a body part, etc.), although the user may adjust the character (without the character object) to facilitate the user's interaction with the game (e.g., a puzzle game in which there is no rendered character game object, but the user still controls the character to control in-game actions).

[0069] In some implementations, components such as body parts may be basic geometric shapes such as blocks, cylinders, spheres, or some other basic shape such as a wedge, torus, tube, groove, etc. In some implementations, a creator module may publish a user's character for viewing or use by other users of the online virtual experience platform 102. In some implementations, creating, modifying, or customizing a character, other virtual object, virtual experience 105, or virtual environment may be performed by a user using a user interface (e.g., a developer interface), with or without scripting (or with or without an application programming interface (API)). It may be noted that, for purposes of illustration and not limitation, characters are described as having a humanoid form. It may further be noted that characters may have any form, such as a vehicle, an animal, an inanimate object, or other creative form.

[0070] In some implementations, the online virtual experience platform 102 may store characters created by a user in the data store 108. In some implementations, the online virtual experience platform 102 maintains a character catalog and an experience catalog that may be presented to a user via the virtual experience engine 104, the virtual experience 105, and / or the client device 110 / 116. In some implementations, the experience catalog includes images of various experiences stored in the online virtual experience platform 102. In addition, a user may select a character (e.g., a character created by the user or another user) from the character catalog to participate in the selected experience. The character catalog includes images of characters stored in the online virtual experience platform 102. In some implementations, one or more of the characters in the character catalog may be created or customized by the user. In some implementations, the selected character may have a character setting that defines one or more of the character's components.

[0071] In some implementations, a user's character can include a configuration of components, and the configuration and appearance of the components, and more generally, the character's appearance, may be defined by a character setting. In some implementations, a user's character's character setting may be selected at least in part by the user. In other implementations, a user may select a character with a default character setting or a character setting selected by another user. For example, a user may select a default character from a character catalog with a predefined character setting, and the user may further customize the default character by changing parts of the character setting (e.g., adding a customized logo to a shirt). A character setting may be associated with a particular character by the online virtual experience platform 102.

[0072] In some implementations, client devices 110 or 116 may each include a computing device such as a personal computer (PC), a mobile device (e.g., a laptop, a mobile phone, a smartphone, a tablet computer, or a netbook computer), a network-connected television, a game console, etc. In some implementations, client devices 110 or 116 may also be referred to as “user devices.” In some implementations, one or more client devices 110 or 116 may connect to the online virtual experience platform 102 at any given moment. It may be noted that the number of client devices 110 or 116 is given by way of example and not limitation. In some implementations, any number of client devices 110 or 116 may be used.

[0073] In some implementations, each client device 110 or 116 may include an instance of a virtual experience application 112 or 118, respectively. In one implementation, the virtual experience application 112 or 118 may enable a user to use and interact with the online virtual experience platform 102, such as searching for particular experiences or other content, control a virtual character in a virtual game hosted by the online virtual experience platform 102, or view or upload content such as the virtual experience 105, images, video items, web pages, documents, etc. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, present, or manipulate content (e.g., virtual characters in a virtual environment, etc.) served by a web server. In another example, the virtual experience application may be a native application (e.g., a mobile application, app, or program) that is installed and runs locally on the client device 110 or 116 and enables a user to interact with the online virtual experience platform 102. The virtual experience application may render, display, or present content (e.g., a web page, a user interface, a media viewer) to the user. In some implementations, the virtual experience application may also include an embedded media player (e.g., a Flash player) that is embedded in a web page.

[0074] According to aspects of the present disclosure, the virtual experience application 112 / 118 may be an online virtual experience platform application through which a user builds, creates, edits, and uploads content to the online virtual experience platform 102, as well as interacts with the online virtual experience platform 102 (e.g., plays and interacts with a virtual experience 105 hosted by the online virtual experience platform 102). Thus, the virtual experience application 112 / 118 may be provided to the client device 110 or 116 by the online virtual experience platform 102. In another example, the virtual experience application 112 / 118 may be an application downloaded from a server.

[0075] In some implementations, a user may log in to the online virtual experience platform 102 through a virtual experience application. A user may access a user account by providing user account information (e.g., a username and password), and the user account is associated with one or more characters available to participate in one or more virtual experiences 105 of the online virtual experience platform 102.

[0076] In general, functions described as being performed by the online virtual experience platform 102 may also be performed by the client device 110 or 116 or a server, as appropriate, in other implementations. In addition, functionality attributed to a particular component may be performed by different or multiple components working together. The online virtual experience platform 102 is not limited to use on websites, as it may also be accessed as a service offered to other systems or devices through an appropriate application programming interface (API).

[0077] In some implementations, the online virtual experience platform 102 may include a communications engine 106. In some implementations, the communications engine 106 may be a system, application, or module that enables the online virtual experience platform 102 to provide video communications capabilities to users, allowing users to experience virtual chats or virtual video conferences using the online virtual experience platform 102 and their associated virtual representatives. For example, users may design and build virtual avatars and use the virtual avatars through chat functionality provided by the communications engine 106.

[0078] In various implementations, the platform 102 may provide chat and / or avatar animation functionality through the communications engine 106 and other components, such as an animation engine or facial animation engine. In these and other implementations, the communications engine 106 may utilize automatic generation of FACS weights to animate avatars representing users utilizing the chat functionality, such that the users are presented as animated avatars. For example, rather than transmitting video over the network 122, the communications engine 106 may transmit audio data and FACS weights. In response to receiving the audio data and FACS weights, the receiving VE application 112 may animate the avatar and present the animated avatar in a graphical user interface (GUI) or another interface. Thus, users may "chat through avatars" rather than being presented with live video.

[0079] In some implementations, the online virtual experience platform 102 may include a facial animation engine 107. In some implementations, the facial animation engine 107 may be a system, application, or module that implements face detection (e.g., from video analysis), facial movement detection (e.g., from audio analysis), and regression models to create robust real-time animation of a user's avatar or character's face based on both audio and video signals. The animation may be based on the user's actual face and utterances, etc., and may include smiles, blinks, winks, frowns, head poses, and other gestures extracted from input video of the user's face and audio of the user's mouth utterance phase. While shown as running directly on the online virtual experience platform 102, it should be understood that the face detection, audio processing, and regression models may be implemented on each client device 110, 116, for example, or on other devices.

[0080] Facial animation engine 107, in conjunction with communications engine 106, may provide some or all of the avatar chat functionality described above. For example, facial animation engine 107 may receive input video and input audio and output associated FACS weights to communications engine 106 (or to VE engine 104). Communications engine 106 may subsequently (or substantially simultaneously) transmit the FACS weights and user audio data for presentation through the chat reception interface and associated components.

[0081] In some implementations, facial animation engine 107 may also be used to present realistic avatar animation within virtual experience 105, within 3D environments during gameplay or virtual activities, and elsewhere. Thus, while described in some implementations as relating to chat functionality, the functionality provided by facial animation engine 107 may also be used to animate avatars within games, within scene rendering (e.g., for video game cutscenes or animated videos), within virtual experiences (e.g., to increase user enjoyment and engagement and improve the immersive experience provided by the 3D environment), and elsewhere.

[0082] The various functions and components associated with facial animation engine 107 are described in detail below with respect to FIGS.

[0083] Figure 2: Facial animation engine 2 is a diagram of a facial animation engine 107, according to some implementations. As shown, the facial animation engine 107 includes a video animation component 204. The video animation component 204 generates FACS weights s determined from input video frames 202. V The facial animation output may be a software component deployed on a computing device that includes one or more models that are trained to provide robust facial animation output as a

[0084] The input video frames 202 may be received from a camera, an image capture device, or another device / technique (e.g., stored video, re-created video, synthesized video, etc.). The input video frames may be sent to a video animation component 204, which then calculates the FACS weights s VFor example, the video animation component 204 may output FACS weights based on the video, which may include facial movements such as movements of the eyebrows, eyes, eyelids, nose, ears, cheeks, chin, neck, lips, mouth, and others.

[0085] The facial animation engine 107 further includes an audio animation component 210. The audio animation component 210 uses FACS weights s determined from the input audio 208. a The facial animation output may be a software component deployed on a computing device that includes one or more models that are trained to provide robust facial animation output as a

[0086] The input audio 208 may be received from a microphone, an audio capture device, or another device / technique (e.g., stored audio, recreated audio, synthesized audio, etc.). The input audio 208 may be sent to an audio animation component 210, which then calculates the FACS weights s a and audio signal α. For example, the audio animation component 210 may output FACS weights based on audio and / or phonemes, which may include facial movements such as movements of the lips, mouth, tongue, jaw, and others. a may also include values ​​representing opening the mouth in surprise, pursing the lips, stretching the lips, and others that can be determined from the audio information. Additionally, the audio animation component 210 may determine if the speaker is speaking and adjust and / or output α to represent the detected speech and / or utterance.

[0087] The facial animation engine 107 may further include a modularity mixing component 206. The modularity mixing component 206 is deployed in a computing device and is configured to calculate FACS weights s based on a weighted linear function and / or a voice activity signal α. V and s a For example, in some implementations, the modularity mixing component 206 may be a software component configured to mix or blend the FACS weights s based on Equation 1 provided below: V and s a may be mixed or fused together. Formula 1: FACS i =FACS v,i (1-∝)+∝(w a,i FACS a,i +w v,i FACS v,i )

[0088] In Equation 1, the mixed or fused results are i The video animation component output is represented by FACS v,i and audio animation component output FACS a,i Note that i represents an index value for an individual frame or grouping of frames of the input video. Furthermore, α represents the speech activity signal, and w a,i and w v,i are the blending weights for the audio and video results, respectively. Note that if there is no speech detected in the input audio 208, the audio animation component 210 outputs α as 0, and the predictions from the video animation component 204 drive the final FACS weights 212. If voice activity is detected, the value of α transitions from 0 to 1, and the final FACS weights 212 are a weighted combination of the results from both the video animation component 204 and the audio animation component 210.

[0089] The operations and components associated with the video animation component 204 are described in detail below with respect to FIG.

[0090] Figure 3: Video Animation Component 3 is a diagram of the video animation component 204, according to some implementations. As shown, the video animation component 204 is configured to receive input video frames 302. The input video frames may be provided to an initial face proposal network 304. In some implementations, the input video frames may also be directed to bypass the face proposal network 304 upon face detection (e.g., elements 310 and 320).

[0091] The face proposal network 304 may include both a proposal network 306 (labeled P-Net) and a refinement network 308 (labeled R-Net). The P-Net 306 may be configured to determine face proposals. For example, the P-Net 306 may be configured to determine whether a human face is located within a video frame of the input video frame 302.

[0092] The R-Net 308 may be configured to filter the determined face proposals received from the P-Net 306. The R-Net 308 may further be configured to refine the determined face proposals within the appropriate bounding box. Upon detecting the user's face and refining the detected face within the bounding box, the face proposal network 304 may be bypassed in the OR function 310. Bypassing the face proposal network 304 may reduce the overall computation cycles by reducing the video FACS weights s V This can improve the speed at which the

[0093] One or both of the P-Net 306 and the R-Net 308 may be part of a Multi-task Cascaded Convolutional Network (MTCNN) of the video animation component 204. In that respect, the P-Net 306 and the R-Net 308 may be early stages of the MTCNN.

[0094] In refining the bounding box and / or bypassing the face proposal network 304, the two level of detail decision blocks 312 and 320 may operate in a coordinated manner. For example, at a first level of detail, the decision block 312 may direct the refined input video frame and bounding box to the B-Net 314, which represents a base level of detail. For example, at a second level of detail, the decision block 320 may direct the refined input video frame and bounding box to the H-Net 314, which represents a higher level of detail than the first level of detail. Furthermore, in some implementations, both the B-Net 314 and the H-Net 322 may be used, resulting in a highest level of detail that is higher than both the first and second levels of detail.

[0095] The B-Net 314 (also called Basic-Net) includes a feature encoder 316 and up to four decoders 318. The four decoders 318 are a landmark decoder D1, a FACS decoder D2, and a feature vector decoder D3. v1 , head pose decoder D z , and the face probability decoder D p In some implementations, the feature encoder 316 is a convolutional neural network. In some implementations, the landmark decoder D1, the FACS decoder D v1 , head pose decoder D z , and the face probability decoder D p is a relatively small, time-aware neural network that is configured to input the high-level features output by the feature encoder 316.

[0096] Note that, unlike convolutions applied across the spatial dimension, the causal convolutional layers of decoder 318 are applied across the temporal dimension. In this regard, decoder 318 (and 326, described below) enable implicit learning of filtering functions that reduce jitter while maintaining responsiveness in the FACS weight output. Furthermore, the availability of temporal information allows decoder 318 to compensate for low-quality features output by feature encoder 316, so the relatively small size of the decoder allows for a reduction in the size of feature encoder 316 without compromising output quality.

[0097] Landmark decoder D1, FACS decoder D v1 , head pose decoder D z , and the face probability decoder D p Note that because FACS decoder D1 is trained on a continuous sequence of video data rather than a single image, it can be trained to associate temporal relationships between input video frames 302. The landmark decoder D1 may be configured to decode facial landmarks for use as input to the H-Net 322. If the H-Net 322 is bypassed (e.g., at the OR operation 328 and the switching / decision block 320), the FACS decoder D1 may be configured to decode facial landmarks for use as input to the H-Net 322. v1 may be configured to generate the FACS weights as output. z may be configured to output a head pose signal zv to aid in realistic head poses in the animation of avatars. In addition, a face probability decoder D p may be configured to operate the decision block 312 to bypass the face suggestion network 304.

[0098] To aid in bypassing the face suggestion network 304 and / or the B-Net 314 and / or the H-Net 322, other forms of diversion blocks may be implemented in place of the specific diversion components shown.

[0099] The H-Net 322 (also called HiFi-Net) includes a feature encoder 324 and a decoder 326. The decoder 326 is a FACS decoder D vh In some implementations, the feature encoder 324 is a convolutional neural network. In some implementations, the FACS decoder D vh is a relatively small, time-aware neural network that is configured to input the high-level features output by the feature encoder 324.

[0100] The B-Net314 and H-Net322 and their respective encoders 316, 324 differ in the input resolution and overall size of the associated convolutional neural networks. For example, the B-Net314 and associated encoder 316 are of smaller input resolution compared to the H-Net322 and associated encoder 324. Thus, in operation, a higher-resolution output may be obtained from the H-Net322 compared to the B-Net314. Thus, different client devices with different computational capabilities may be used to implement face tracking functionality as described herein; with appropriate routing, a low-end device may use the B-Net314 to provide automatic and robust animation, while a high-end device may implement either the B-Net314 or the H-Net322.

[0101] Additionally, the B-Net 314 and H-Net 322 and their respective encoders 316, 324 may be operable to run frame-by-frame (e.g., 30 frames per second [fps]) or subsample the input and run at a lower frame rate (e.g., 15 fps). Producing output at 15 fps may degrade visual quality in some circumstances. However, in some implementations, for "skipped" input frames when running the encoder at 15 fps, the system takes as input the associated encoder feature vector from the previous frame, which allows for extrapolation of the FACS output and may produce results that are quite similar or nearly identical to the output when running the encoder at 30 fps. Note that this interpolation may be achieved through the causal convolutional architecture described above, which results in a form of temporal filtering.

[0102] As shown, the video animation component 204 generates video animation FACS weights s V The video animation FACS weights are then output. V may be used for robust facial animation with or without audio-animation FACS weights, as described above with respect to modularity mixing component 206 and Equation 1. In implementations where audio is available and / or selected by the user, audio animation component 210 uses audio-animation FACS weights s a as an output.

[0103] Figure 4: Audio Animation Component 4 is a diagram of an audio animation component 210, according to some implementations. As shown, the audio animation component 210 can be configured to receive input audio 402. In some implementations, the input audio 402 can be captured through a microphone or synthesized for non-verbal users.

[0104] The input audio 402 may be processed by the feature extraction component 404 to extract features at a first hop length and a first window length. In some implementations, the feature extraction component is configured to extract 26 features at a hop length of 16 ms and a window length of 16 ms. Other variations are applicable depending on the specific implementation.

[0105] In some implementations, five consecutive frames of extracted features are packed together to form an input package for the audio network 406. In some implementations, the audio network 406 is configured to include a feature encoder 408 and up to two decoders 410, 412. The feature encoder 408 may include a stack of causal convolutional layers followed by batch normalization. The causal convolutional layers are configured to examine sequence values ​​from the past (e.g., in a temporal sense) and thus may widen the receptive field without introducing delay. Additional layers may be configured to operate along the time dimension to preserve temporal embeddings. The additional layers may be recurrent neural networks, such as long short-term memory layers (LSTM layers).

[0106] The decoder 410 receives the audio FACS weights s a , while decoder 412 may be configured to provide audio signal α as described above. Thus, decoders 410 and 412 can operate as dedicated to a specific task, resulting in improved efficiency and reduced computational cost compared to larger decoders for multiple tasks.

[0107] Both the video animation component 204 and the audio animation component 210 are trained in a multi-phase / multi-stage training process, whereby the associated encoder is trained first, followed by training of the decoder while the encoder weights are frozen / fixed based on previous training data. Below, training of the video animation component 204 is described with reference to Figure 5, and training of the audio animation component 210 is described with reference to Figure 6.

[0108] Figure 5: Training the video animation component 5A and 5B show a flowchart 500 of an exemplary method for training a portion of a video animation component 204 according to some implementations. In some implementations, the method 500 may be implemented, for example, on a server system, such as the online virtual experience platform 102 as shown in FIG. 1. In some implementations, some or all of the method 500 may be implemented on a system, such as one or more client devices 110 and 116 as shown in FIG. 1, and / or on both the server system and one or more client systems. In the described example, the implementation system includes one or more processors or processing circuits and one or more storage devices, such as a database or other accessible storage. In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of the method 500. The method 500 begins at block 502.

[0109] In block 502, facial landmarks are obtained from both real and synthetic video training data. For example, training a FACS model directly from synthetic data in a supervised manner is not straightforward because the domain gap between synthetic data and actual real faces is large. Therefore, the feature representations learned by a network trained on the synthetic dataset are different from those needed for real data. Therefore, a model trained only on the synthetic dataset is not able to generalize. Therefore, both the real dataset and the synthetic dataset are used for training, and facial landmarks are obtained for the first stage of encoder training. Although facial landmarks are not used in the regression step described above with reference to FIG. 3, the landmarks help the model to learn representations that are valid for both real and synthetic data. Block 502 is followed by block 504.

[0110] In block 504, training data is provided to the encoder model for training. For example, real video frames and synthetic video frames are provided as input to the associated encoder of the video animation component 204. Block 504 is followed by block 506.

[0111] In block 506, the acquired facial landmarks are provided to a landmark decoder D1. In this way, the landmark decoder D1 can be trained simultaneously with the encoder. Block 506 is followed by block 508.

[0112] At block 508, the encoder and / or landmark decoder are tuned based on two or more loss terms. For example, the two or more terms may be linearly combined such that the encoder and / or landmark decoder may be trained together.

[0113] The first loss term is L LMK, which is the location loss for the landmarks. The root mean square error of the regressed location can be used for the location loss. The second loss term is L CON , which is a consistency loss for landmarks. The consistency loss for landmarks may encourage landmark predictions to be equivariant under different transformations, which may enable using authentic image pairs without landmark annotations. Based on their respective outputs and the calculated loss terms, internal weights for each of the encoder and landmark decoder may be adjusted. Block 510 follows block 508.

[0114] In block 510, it is determined whether all training data has been input or whether training is otherwise complete. For example, the achievement of a threshold number of training epochs or the input of a threshold amount of images may indicate that the training process is complete. Other thresholds or conditions may also be applicable during training. If training is complete, block 512 may follow block 510. Otherwise, block 510 may be followed by block 504, where additional training data is input to the encoder.

[0115] If training is complete in block 512, the internal weights for the encoder are fixed and a second stage of training may begin in block 514. Block 514 is followed by block 516.

[0116] In block 516, an output from the trained encoder is obtained. For example, additional training sets or other training data may be provided as input to the trained encoder. The encoder output may be obtained for training an individual decoder. Block 516 is followed by block 518.

[0117] At block 518, the encoder output is fed to an untrained decoder (e.g., FACS decoder D v1 , head pose decoder D z , and the face probability decoder Dp ) are provided to each decoder. For example, high-level encoded features output by a trained encoder are input to each decoder. Block 520 follows block 518.

[0118] At block 520, the decoder is adjusted based on three or more loss terms. For example, three or more loss terms may be linearly combined. The first loss term is L POS , which is the position loss for the FACS weights. The mean squared error is L POS The second loss term is L VEL This is the velocity loss. VEL The use of can help reduce jitter by promoting smoothness in dynamic representations. The third loss term is L ACC , which is the acceleration regularization loss. To reduce FACS weight jitter, a regularization term for acceleration is added (e.g., its weight is kept low to preserve responsiveness). Block 522 follows block 520.

[0119] In block 522, it is determined whether all training data has been input or whether training is otherwise complete. For example, the achievement of a threshold number of training epochs or the input of a threshold amount of encoded features may indicate that the training process is complete. Other thresholds or conditions for training may also be applicable. If training is complete, block 522 may be followed by block 524. Otherwise, block 522 may be followed by block 516, where additional training data (e.g., encoded features from a trained encoder) is input to the decoder.

[0120] At block 524, if training is complete, the internal weights for the decoder are fixed and the trained encoder and decoder may be output and / or deployed.

[0121] As described above, a two-stage training approach is used to train the video animation component 204 and associated subcomponents. As described below, a two-stage approach is also used to train the audio animation component 210.

[0122] Figure 6: Training the Audio Animation Component 6A and 6B show a flowchart 600 of an exemplary method for training portions of an audio animation component according to some implementations. In some implementations, the method 600 may be implemented, for example, on a server system, such as on the online virtual experience platform 102 as shown in FIG. 1. In some implementations, some or all of the method 600 may be implemented on a system such as one or more client devices 110 and 116 shown in FIG. 1 and / or on both the server system and one or more client systems. In the described example, the implementation system includes one or more processors or processing circuits and one or more storage devices, such as a database or other accessible storage. In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of the method 600. The method 600 starts at block 602.

[0123] Audio samples with time-aligned labels are obtained in block 602. Block 602 is followed by block 604.

[0124] In block 604, the training data is augmented. For example, audio samples are augmented with various randomly selected noises and pitch transformations. To improve the robustness of the model to various types of noise, the input audio samples are augmented by adding randomly generated white noise or by selecting pre-recorded ambient noises such as street, restaurant, wind, or rain sounds.

[0125] The impulse response convolution may add reverberation to the input audio to simulate people speaking in various virtual rooms. In addition, pitch variations, gain variations, and rate variations may also be used to further enhance the audio samples. Block 606 follows block 604.

[0126] The enhanced audio samples are provided to an encoder and an additional phoneme layer or phoneme decoder in block 606. For example, with reference to Figure 6C, in an initial training stage labeled 680, enhanced audio samples 670 may be provided to the encoder 408 and the additional phoneme decoder 675.

[0127] Although the phoneme decoder 675 is not a permanent component or otherwise used in operation, phoneme prediction tasks may result in clearer or more consistent output. Intuitively, vocal speech is strongly correlated with visual speech units (e.g., jaw, lip, and mouth movements). Therefore, training for phoneme recognition tasks favors an internal representation that can be converted into a continuous analog of a viseme sequence represented as a FACS curve. In some implementations, the phoneme decoder 675 is implemented as a fully connected layer after embedding to estimate phoneme labels. Returning to FIG. 6A, block 608 follows block 606.

[0128] In block 608, the encoder is adjusted based on the loss term. For example, only the connectionist temporal classification (CTC) loss may be calculated and used to adjust the encoder. Block 610 follows block 608.

[0129] In block 610, it is determined whether all training data has been input or whether training is otherwise complete. For example, the achievement of a threshold number of training epochs or the input of a threshold amount of images may indicate that the training process is complete. Other thresholds or conditions for training may also be applicable. If training is complete, block 612 may follow block 610. Otherwise, block 610 is followed by block 604, where additional training data is input to the encoder.

[0130] If training is complete in block 612, the internal weights for the encoder are fixed and a second stage of training may begin in block 614. Block 614 is followed by block 616.

[0131] In block 616, output from the trained encoder is obtained. For example, additional training sets or other training data may be provided as input to the trained encoder. The encoder output may be obtained for training the decoder. Block 616 is followed by block 618.

[0132] In block 618, the encoder output (e.g., a phoneme-aware representation of the input data) is provided to an untrained decoder. For example, referring to FIG. 6C, the second stage of training 685 may include training both decoder 410 and decoder 412 using phoneme-aware encoded features provided by trained encoder 408, which are based on input audio samples 680. Note that while training data 670 comprises audio samples from multiple speakers, training data 680 is based on a single speaker. In this way, the encoder is not biased toward a single speaker's voice. Returning to FIG. 6B, block 620 follows block 618.

[0133] At block 620, the decoder is adjusted based on three or more loss terms. For example, three or more loss terms may be linearly combined. The first loss term is LPOS , which is the position loss for the FACS weights. The mean squared error is L POS The second loss term is L VEL This is the velocity loss. VEL The use of can help reduce jitter by promoting smoothness in dynamic representations. The third loss term is L VAD , which is the cross-entropy loss. POS and L VEL Note that both may be implemented as smoothed L1 losses, while the same loss term is the L2 loss in video network training. Block 622 follows block 620.

[0134] In block 622, it is determined whether all training data has been input or whether training is otherwise complete. For example, the achievement of a threshold number of training epochs or the input of a threshold amount of encoded features may indicate that the training process is complete. Other training thresholds or conditions may also be applicable. If training is complete, block 624 may follow block 622. Otherwise, block 622 may be followed by block 616, where additional training data (e.g., encoded features from a trained encoder) is input to the decoder.

[0135] At block 624, if training is complete, the internal weights for the decoder are fixed and the trained encoder and decoder may be output and / or deployed.

[0136] As described above, both the video animation component and the audio animation component may be trained using a two-stage approach, which results in more robust FACS weight outputs for the animation. Below, the functionality and operation associated with the deployed and trained models are described with respect to FIG.

[0137] Figure 7: Animating an avatar using the trained model 7 is a flowchart of an example method 700 for real-time robust facial animation from video and audio according to some implementations. In some implementations, method 700 may be implemented, for example, in a server system, such as in the online virtual experience platform 102 as shown in FIG. 1. In some implementations, some or all of method 700 may be implemented in a system such as one or more client devices 110 and 116 as shown in FIG. 1 and / or in both the server system and one or more client systems. In the described example, implementing a system includes one or more processors or processing circuits and one or more storage devices, such as a database or other accessible storage. In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 700.

[0138] To provide avatar animation, faces may be detected from the input video, and facial landmarks, head pose, tongue state, etc. may be determined and utilized to animate the corresponding avatar face. Additionally, audio may be received from a microphone, and facial movements may be determined from phonemes. Before performing face detection or analysis, the user is given an indication that such techniques will be utilized for avatar animation. If the user declines permission, video- and / or audio-based facial animation is turned off (e.g., default animation may be used, or animation may be based on user-provided audio and / or other user-authorized input, such as text input). User-provided video and / or audio are utilized specifically for avatar animation and are not stored. The user can opt out of video analysis, audio analysis, and animation generation at any time. Furthermore, face detection is performed to detect the location of faces within the video; i.e., face recognition is not performed. If the user authorizes the use of video and audio analysis for avatar animation, method 700 begins at block 702 and block 706.

[0139] In block 702, an input video frame is received from a user device. For example, the user's face may be captured with a camera or image capture device. In some implementations, the user may select an option in an interface to allow image capture. In these and other examples, the user may opt out of automatic animation from the video and / or audio. Block 702 is followed by block 704.

[0140] At block 704, video FACS weights are obtained from the trained machine learning model. For example, the video FACS weights may be obtained from the video animation component 204. The video FACS weights may be based on a first level of detail (e.g., utilizing only B-Net314), a second level of detail (e.g., utilizing only H-Net322), or a third level of detail (e.g., utilizing both B-Net314 and H-Net322).

[0141] In block 706, an input audio frame and a mixing term (e.g., an active audio input signal α, an active speech signal, an unmuted audio signal, and / or others) are received from a user device. The mixing term may ramp from a value of 0 (no audio or speech present) to a value of 1. The mixing term is described in detail above with respect to Equation 1. Block 706 is followed by block 708.

[0142] In block 708, audio FACS weights are obtained from a trained machine learning model. For example, the audio FACS weights may be obtained from the audio animation component 210. Block 710 may follow blocks 704 and 708.

[0143] In block 710, the audio FACS weights and the video FACS weights are combined using a modularity mixing component. This combination may include linearly combining the audio FACS weights and the video FACS weights. In addition, this combination may be based on a mixing term. For example, if the user is not actively speaking, the mixing term may be close to (or is) 0, and therefore the audio FACS weights are not combined. In addition, for example, if the user is actively speaking, a mixing term may be used to combine the audio FACS weights with the video FACS weights, as described in detail above with respect to Equation 1. Block 712 follows block 710.

[0144] In block 712, the combined FACS weights are output as final FACS weights for the animation of the user's avatar, character clothing, 3D model, or any other animation-enabled construct. For example, the final FACS weights may be a linearly combined mixture or fusion of both video and audio FACS weights. In this manner, the avatar may be animated even when the user's face is partially obscured while the user is actively speaking. For example, audio may be used to generate FACS weights that provide lip, chin, and / or mouth movement based solely on the audio. Furthermore, even when the user is not actively speaking, video FACS weights may be used to animate the avatar's eyes, mouth, head, and other features based on actual movement even without speech. These and other features provide a robust facial animation framework that overcomes many shortcomings and offers technical advantages and benefits, including reduced computational cost, improved energy efficiency, improved storage usage, improved bandwidth usage, and others.

[0145] Blocks 702-712 may be performed (or repeated) in a different order than described above, and / or one or more blocks may be omitted, modified, combined with other blocks, supplemented with blocks, etc. Method 700 may be performed on a server (e.g., 102) and / or on a client device (e.g., 110 or 116). Furthermore, portions of method 700 may be combined and performed sequentially or in parallel, according to any desired implementation.

[0146] As described above, techniques for robust facial animation include the implementation on a client device of a trained face detection model, an audio model, and a trained regression model, which may output video and audio FACS weights that are linearly combined in a modularity mixing component based on mixing terms to create final FACS weights for a user's avatar's facial animation, a character's clothing, a 3D model, or another animation-enabled construct.

[0147] A more detailed description of various computing devices that may be used to implement the various devices and components shown in FIGS. 1-4 is provided below with respect to FIG.

[0148] 8 is a block diagram of an exemplary computing device 800 that may be used to implement one or more features described herein, according to some implementations. In one example, the device 800 may be used to implement a computer device (e.g., 102, 110, and / or 116 of FIG. 1 ) and to perform implementations of suitable methods described herein. The computing device 800 may be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 800 may be a mainframe computer, a desktop computer, a workstation, a portable computer, or an electronic device (such as a portable device, a mobile device, a mobile phone, a smartphone, a tablet computer, a television, a TV set-top box, a personal digital assistant (PDA), a media player, a gaming device, a wearable device, etc.). In some implementations, the device 800 includes a processor 802, a memory 804, an input / output (I / O) interface 806, and audio / video input / output devices 814 (e.g., a display screen, a touch screen, display goggles or glasses, audio speakers, a microphone, etc.).

[0149] Processor 802 may be one or more processors and / or processing circuits for executing program code and controlling the basic operations of device 800. A "processor" includes any suitable hardware and / or software system, mechanism, or component that processes data, signals, or other information. A processor may include a general-purpose central processing unit (CPU), multiple processing units, a system with dedicated circuitry to accomplish functions, or other systems. Processing need not be limited to a particular geographic location or have time constraints. For example, a processor may perform its functions in "real time," "offline," "batch mode," etc. Portions of processing may be performed at different times and in different locations by different (or the same) processing systems. A computer may be any processor in communication with memory.

[0150] Memory 804 is typically provided in device 800 for access by processor 802, is suitable for storing instructions for execution by the processor, and may be any suitable processor-readable storage medium, e.g., random access memory (RAM), read-only memory (ROM), electronically erasable read-only memory (EEPROM), flash memory, etc., located remotely from and / or integrated with processor 802. Memory 804 may store software operated by processor 802 on server device 800, including operating system 808, applications 810, and associated data 812. In some implementations, applications 810 may include instructions that enable processor 802 to perform some or all of the functions described herein, e.g., the methods of FIGS. 5-7. In some implementations, applications 810 may also include one or more trained models for generating robust real-time animation based on input video, as described herein.

[0151] For example, memory 804 may include software instructions for an application 810 that can provide an animated avatar within an online virtual experience platform (e.g., 102) based on a user's facial movements captured by a camera (or another device) and audio captured via a microphone (or another device). Alternatively, any of the software in memory 804 may be stored in any other suitable storage location or computer-readable medium. Additionally, memory 804 (and / or other connected storage devices) may store instructions and data used in the features described herein. Memory 804 and any other type of storage (such as magnetic disks, optical disks, magnetic tape, or other tangible media) may be considered to be “storage” or “storage devices.”

[0152] The I / O interface 806 may provide functionality to allow the server device 800 to interface with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data store 108), and input / output devices may communicate via the interface 806. In some implementations, the I / O interface may be connected to interface devices including input devices (keyboards, pointing devices, touchscreens, microphones, cameras, scanners, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc.).

[0153] For ease of illustration, FIG. 8 shows one block for each of the processor 802, memory 804, I / O interface 806, software blocks 808 and 810, and database 812. These blocks may represent one or more processors or processing circuits, operating systems, memories, I / O interfaces, applications, and / or software modules. In other implementations, the device 800 may not have all of the components shown and / or may have other elements, including other types of elements, instead of or in addition to those shown herein. Although the online virtual experience platform 102 is described as performing the operations as described in some implementations herein, any suitable component or combination of components of the online virtual experience platform 102 or a similar system, or any suitable processor or processors associated with such a system, may perform the described operations.

[0154] User devices may also implement and / or be used in conjunction with features described herein. An exemplary user device may be a computing device including several components similar to device 800, e.g., processor 802, memory 804, and I / O interface 806. An operating system, software, and applications suitable for the client device may be provided in the memory and used by the processor. The I / O interface for the client device may be connected to a network communication device and to input and output devices, e.g., a microphone for capturing sound, a camera for capturing images or video, an audio speaker device for outputting sound, a display device for outputting images or video, or other output devices. A display device within audio / video input / output device(s) 814 may be connected to (or included in) device 800, e.g., to display pre-processing and post-processing images as described herein; such a display device may include any suitable display device, e.g., an LCD, LED, or plasma display screen, a CRT, a television, a monitor, a touchscreen, a 3D display screen, a projector, or other visual display device. Some implementations can provide an audio output device, for example, text-to-speech output or synthesis.

[0155] Methods, blocks, and / or operations described herein may, where appropriate, be performed in a different order than shown or described and / or performed concurrently (partially or fully) with other blocks or operations. Some blocks or operations may be performed for one portion of data and then performed again later, for example, for another portion of data. In various implementations, not all of the blocks and operations described need be performed. In some implementations, blocks and operations may be performed multiple times, in different orders, and / or at different times, such as in a method.

[0156] In some implementations, some or all of the methods may be implemented in a system such as one or more client devices. In some implementations, one or more methods described herein may be implemented, for example, in a server system and / or in both a server system and a client system. In some implementations, different components of one or more servers and / or clients may perform different blocks, operations, or other portions of a method.

[0157] One or more methods described herein (e.g., methods 500, 600, 680, 685, and / or 700) may be implemented by computer program instructions or code that may be executed on a computer. For example, the code may be implemented by one or more digital processors (e.g., microprocessors or other processing circuits) and may be stored in a computer program product including a non-transitory computer-readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), flash memory, rigid magnetic disks, optical disks, solid-state memory drives, etc. The program instructions may also be contained in or provided as electronic signals, for example, in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods may be implemented in hardware (such as logic gates) or a combination of hardware and software. Exemplary hardware may be a programmable processor (e.g., a field programmable gate array (FPGA), complex programmable logic device), general-purpose processor, graphics processor, application-specific integrated circuit (ASIC), etc. One or more methods may be implemented as part of or a component of an application running on the system, or as an application or software running in conjunction with other applications and the operating system.

[0158] One or more methods described herein may be implemented in a standalone program that may run on any type of computing device, a program running on a web browser, a mobile application (“app”) running on a mobile computing device (e.g., a mobile phone, a smartphone, a tablet computer, a wearable device (wristwatch, armband, jewelry, headwear, goggles, glasses, etc.), a laptop computer, etc.). In one example, a client / server architecture may be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives final output data from the server for output (e.g., for display). In another example, all computations may be performed within a mobile app (and / or other apps) on the mobile computing device. In another example, computations may be split between the mobile computing device and one or more server devices.

[0159] Although described with reference to specific implementations, these specific implementations are exemplary only and not limiting, and the concepts illustrated in the examples may be applied to other examples and implementations.

[0160] In situations where some implementations discussed herein may obtain or use user data (e.g., user images, user video, user audio, user attributes, user behavior data on the platform, user search history, purchased and / or viewed items, user friends on the platform, etc.), the user is given the option to control whether and how such information is collected, stored, or used. That is, the implementations discussed herein collect, store, and / or use user information only with explicit user authorization and in accordance with applicable regulations.

[0161] The user is given control over whether the program or feature collects user information about that particular user or other users associated with that program or feature. Each user about whom information will be collected is presented with options (e.g., via a user interface) to enable the user to exercise control over the collection of information associated with that user and provide permission or approval for whether and what portions of the information will be collected. Additionally, some data may be modified in one or more ways before storage or use so that personally identifiable information is removed. As one example, a user's identifying information may be modified (e.g., by substitution with a pseudonym, numeric values, etc.) so that personally identifiable information cannot be determined. In another example, a user's geographic location may be generalized to a broader area (e.g., city, zip code, state, country, etc.).

[0162] It should be noted that the functional blocks, operations, features, methods, devices, and systems described in this disclosure may be combined or divided into different combinations of systems, devices, and functional blocks that would be known to one of ordinary skill in the art. Any suitable programming language and programming techniques may be used to implement the routines of a particular implementation. Various programming techniques, such as procedural or object-oriented programming techniques, may be utilized. The routines may be executed on a single processing device or on multiple processors. While steps, operations, or computations may be presented in a particular order, the order may be changed in different particular implementations. In some implementations, multiple steps or operations shown herein as sequential may be performed simultaneously.

[0163] Below, additional details and a description of the figures are provided where appropriate, followed by claims directed to one or more aspects of the disclosure. [Explanation of symbols]

[0164] 100 Network Environment 102 Online Virtual Experience Platform 104 Virtual Experience (VE) Engine 105 Virtual Experience, VE 106 Communication Engine 107 Facial Animation Engine 108 Datastores 110 client devices 112 Virtual Experience Applications, VE Applications 114 users 116 client devices 118 Virtual Experience Applications, VE Applications 120 users 122 Network 202 input video frames 204 Video Animation Component 206 Mixed Modularity Components 208 Input Audio 210 Audio Animation Component 212 FACS weights 302 input video frames 304 Face Suggestion Network 306 P-NET 308 R-NET 310 elements 312 Specificity determination block 314 B-NET 316 Feature Encoder 318 Decoder 320 elements 322 H-NET 324 Feature Encoder 326 decoder 328 Logical OR operation 402 Input Audio 404 Feature Extraction Component 406 Audio Network 408 Feature Encoder 410 decoder 412 decoder 670 training data 675 Additive Music Decoder 680 training data 800 computing devices 802 processor 804 memory 806 I / O interface 808 Operating System 810 Application 812 Data, Database 814 Audio / Video Input / Output Devices, I / O Devices

Claims

1. 1. A computer-implemented method comprising: receiving an input video frame; receiving an input audio frame and a mixing term, the input audio frame comprising audio associated with the input video frame; obtaining video facial activity coding system (FACS) weights from a first trained machine learning model based on the input video frames; obtaining audio FACS weights from a second trained machine learning model based on the input audio frames; combining the video FACS weights and the audio FACS weights to obtain final FACS weights, the combining being based at least in part on the mixing terms; and outputting the final FACS weights to drive facial animation of a 3D model.

2. 10. The computer-implemented method of claim 1, wherein the first trained machine learning model and the second trained machine learning model are trained in a two-stage semi-supervised training process.

3. 10. The computer-implemented method of claim 1, wherein the first trained machine learning model comprises at least one encoder and at least three task-specific decoders.

4. a first task-specific decoder of the at least three task-specific decoders configured to output a predicted head pose; a second task-specific decoder of the at least three task-specific decoders configured to output a probability that a face is visible in an input video frame of the received input video frames; 4. The computer-implemented method of claim 3, wherein a third task-specific decoder of the at least three task-specific decoders is configured to output facial landmarks.

5. 5. The computer-implemented method of claim 4, wherein at least one of the at least three task-specific decoders comprises a causal convolutional layer applied across a time dimension.

6. 10. The computer-implemented method of claim 1, wherein the second trained machine learning model comprises at least one encoder and at least two task-specific decoders.

7. 7. The computer-implemented method of claim 6, wherein a first task-specific decoder of the at least two task-specific decoders is configured to output the audio FACS weights, and a second task-specific decoder of the at least two task-specific decoders is configured to output the mixing terms.

8. 1. A system comprising: a memory in which instructions are stored; a processing device coupled to the memory, the processing device configured to access the memory and execute the instructions, the instructions causing the processing device to: receiving an input video frame; receiving an input audio frame and a mixing term, the input audio frame including audio associated with the input video frame; Obtaining video facial activity coding system (FACS) weights from a first trained machine learning model based on the input video frames; and obtaining audio FACS weights from a second trained machine learning model based on the input audio frames; and combining the video FACS weights and the audio FACS weights to obtain final FACS weights, the combining being based at least in part on the mixing terms; outputting the final FACS weights to drive facial animation of a 3D model; and A system that performs an operation comprising:

9. 10. The system of claim 8, wherein the first trained machine learning model and the second trained machine learning model are trained in a two-stage semi-supervised training process.

10. 10. The system of claim 8, wherein the first trained machine learning model comprises at least one encoder and at least three task-specific decoders.

11. a first task-specific decoder of the at least three task-specific decoders configured to output a predicted head pose; a second task-specific decoder of the at least three task-specific decoders configured to output a probability that a face is visible in an input video frame of the received input video frames; 11. The system of claim 10, wherein a third task-specific decoder of the at least three task-specific decoders is configured to output facial landmarks.

12. 12. The system of claim 11, wherein at least one of the at least three task-specific decoders comprises a causal convolutional layer applied across a time dimension.

13. 10. The system of claim 8, wherein the second trained machine learning model comprises at least one encoder and at least two task-specific decoders.

14. 14. The system of claim 13, wherein a first task-specific decoder of the at least two task-specific decoders is configured to output the audio FACS weights, and a second task-specific decoder of the at least two task-specific decoders is configured to output the mixing terms.

15. A non-transitory computer-readable medium storing instructions, the instructions causing the processing device to: receiving an input video frame; receiving an input audio frame and a mixing term, the input audio frame including audio associated with the input video frame; Obtaining video facial activity coding system (FACS) weights from a first trained machine learning model based on the input video frames; and obtaining audio FACS weights from a second trained machine learning model based on the input audio frames; and combining the video FACS weights and the audio FACS weights to obtain final FACS weights, the combining being based at least in part on the mixing terms; outputting the final FACS weights to drive facial animation of a 3D model; and 10. A non-transitory computer-readable medium for causing a computer to perform operations comprising:

16. 16. The non-transitory computer-readable medium of claim 15, wherein the first trained machine learning model and the second trained machine learning model are trained in a two-stage semi-supervised training process.

17. 16. The non-transitory computer-readable medium of claim 15, wherein the first trained machine learning model comprises at least one encoder and at least three task-specific decoders.

18. a first task-specific decoder of the at least three task-specific decoders configured to output a predicted head pose; a second task-specific decoder of the at least three task-specific decoders configured to output a probability that a face is visible in an input video frame of the received input video frames; 20. The non-transitory computer-readable medium of claim 17, wherein a third task-specific decoder of the at least three task-specific decoders is configured to output facial landmarks.

19. 20. The non-transitory computer-readable medium of claim 18, wherein at least one of the at least three task-specific decoders comprises a causal convolutional layer applied across a time dimension.

20. 16. The non-transitory computer-readable medium of claim 15, wherein the second trained machine learning model comprises at least one encoder and at least two task-specific decoders, a first task-specific decoder of the at least two task-specific decoders configured to output the audio FACS weights, and a second task-specific decoder of the at least two task-specific decoders configured to output the mixing terms.

Images