Adaptive bone joint smoothing

Abstract

Various aspects of the disclosure relate to methods and systems for performing operations, including a computer-readable storage medium storing a program, the operations comprising: receiving a video, the video including a depiction of a body of a user; detecting a plurality of skeletal joints of the depicted body in the video; tracking motion of the plurality of skeletal joints in a set of frames of the video; and smoothing motion of a first set of the plurality of skeletal joints between frames in the first set of frames independently of smoothing motion of a second set of the plurality of skeletal joints in the first set of frames.

Description

[0001] CLAIM OF PRIORITY

[0002] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 198,553, filed October 27, 2020, and U.S. Patent Application Serial No. 16 / 949,607, filed November 6, 2020, the entire contents of each of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates generally to tracking a user’s motion in a video. BACKGROUND

[0004] Image processing systems can be used to react to different actions performed by a user that are captured by a camera. Such systems can modify a presentation of a virtual object or perform other application-specific functions. Such systems can encounter presentation problems due to environmental conditions, user actions, unexpected visual interruptions between the camera and the object being presented, and the like. Presentation problems can also arise from failing to accurately detect user actions. This can cause the virtual object to disappear or otherwise behave abnormally, or cause incorrect functions to be performed, which breaks the illusion that the virtual object exists in the real world. BRIEF DESCRIPTION OF DRAWINGS

[0005] In the drawings, which are not necessarily drawn to scale, like numerals can describe similar components in different views. To easily identify the discussion of any particular element or action, one or more of the highest three digits of a reference number can refer to that element within the figure in which it is first introduced. Some embodiments are illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which:

[0006] Figure 1 is a block diagram illustrating an example messaging system for exchanging data (e.g., messages and associated content) over a network, in accordance with example embodiments.

[0007] Figure 2 is a block diagram illustrating further details of the messaging system of Figure 1 , in accordance with example embodiments.

[0008] Figure 3 is a schematic diagram illustrating data that can be stored in a database of the messaging server system, in accordance with example embodiments.

[0009] Figure 4 is a schematic diagram illustrating the structure of a message generated by a messaging client for communication, in accordance with example embodiments.

[0010] Figure 5Ais a block diagram illustrating an example body tracking system, in accordance with an example embodiment.

[0011] Figure 5B is a block diagram illustrating an example smoothing module, in accordance with an example embodiment.

[0012] Figure 6 is a diagram illustrating body joint positions for tracking a body in a video, in accordance with an example embodiment.

[0013] Figure 7 is a flowchart illustrating example operations of a body tracking system, in accordance with an example embodiment.

[0014] Figure 8 illustrates illustrative inputs and outputs of a body tracking system, in accordance with an example embodiment.

[0015] Figure 9 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions can be executed to cause the machine to perform any one or more of the methodologies discussed herein, according to some examples.

[0016] Figure 10 is a block diagram illustrating a software architecture, wherein an example can be implemented. DETAILED DESCRIPTION

[0017] The description that follows includes systems, methods, techniques, instruction sequences, and computing machine program products embodying illustrative embodiments of the disclosure. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments. It will be apparent, however, to one skilled in the art that embodiments can be practiced without some or all of these specific details. In general, well-known instruction instances, protocols, structures, and techniques have not been shown in detail.

[0018] Generally, virtual reality (VR) and augmented reality (AR) systems display an avatar representing a given user by capturing an image of the user and additionally using a depth sensor to obtain a depth map of the real-world human depicted in the image. By processing the depth map together with the image, VR systems and AR systems can detect and mimic or react to actions performed by the user. While such systems work well in presenting an avatar of the user and reacting to actions performed by the user (e.g., modifying a virtual object or performing an application function), the need for a depth sensor limits the range of applications thereof. This is because adding a depth sensor to a user device in order to display an avatar or perform an action in response to a user-detected gesture increases the overall cost and complexity of the device, making it less attractive.

[0019] Certain systems attempt to detect user actions such as gestures received in a video based on the joint positions of a user's body depicted in the image. Such systems rely on predetermined angles for various joint positions to be specified. These systems calculate the angles of the detected joint positions in the image in real-time for comparison with the predetermined angles for detecting a given gesture. Inputting such angles requires a certain level of skill and consumes a significant amount of time, particularly when tracking a large subset of gestures. This reduces the scalability of the system and increases the overall amount of resources consumed by such systems. Even so, tracking the motion of joints in a set of video frames can be noisy. That is, the motion of a pair of joints can not be accurately determined and detected in a set of adjacent frames. This can also cause the corresponding motion to appear jerky. Some systems apply a single smoothing filter collectively to the joints. However, sometimes no smoothing can be needed at all, and applying a smoothing filter requires a significant amount of processing and memory resources, which slows down the overall detection system. This makes such systems difficult to implement and apply to real-time images or videos being received.

[0020] The disclosed implementations improve the efficiency of use of electronic devices by independently applying smoothing filters to the skeletal joints of a user's body in a video on individual key pairs or groups of joints. In this way, if one group of joints needs more smoothing than another group of joints, the smoothing filters can be adapted to apply different amounts of smoothing to the different groups of joints. This improves the overall responsiveness of the filters applied to smooth the joints of a tracked person in a video, as unnecessary smoothing can be reduced or avoided altogether. In particular, the disclosed implementations receive a video that includes a depiction of a user's body and detect a plurality of skeletal joints of the body depicted in the video. The disclosed implementations track the motion of the plurality of skeletal joints in a set of frames of the video and independently of (or separately from) smoothing the motion of a second group of the plurality of skeletal joints in the first set of frames, smooth the motion of a first group of the plurality of skeletal joints between frames in the first set of frames.

[0021] In particular, noise representing motion of one or more joints of a person in a video can be measured, and multiple smoothing filters (e.g., having different smoothing characteristics) can be applied in parallel based on the measured noise to smooth motion of a given joint or set of joints. That is, if the noise exceeds a threshold, more weight can be applied to the output of a smoothing filter having a first characteristic (e.g., an aggressive motion filter) to smooth motion of a given joint compared to the weight applied to the output of a smoothing filter having a second characteristic (e.g., a gentle motion filter). The weighted outputs of the two smoothing filters can be aggregated to smooth the overall motion of the given joint. In this way, the responsiveness (e.g., increased or decreased) of the motion filter applied to smooth the motion of the joint can be controlled without applying too much smoothing (e.g., over-smoothing) or not enough smoothing (e.g., under-smoothing) to the motion. For example, if there is a noise measurement that exceeds a threshold, the aggressive filter can be weighted more heavily compared to the gentle filter, which decreases responsiveness but increases smoothing. If there is a noise measurement that is less than a threshold, the aggressive filter can be weighted less heavily compared to the gentle filter, which increases responsiveness but decreases smoothing.

[0022] Virtual objects (e.g., three-dimensional objects such as 3D annotations, emojis, characters, avatars, animations, looping animations of personalized avatars or characters, looping or non-looping animated graphics such as dancing hot dogs, stylized words with animation, etc.) can be generated directly from single red-green-blue (RGB) videos depicting a real-world user’s body in response to tracking skeletal joints and smoothing individual sets of joints separately. The disclosed implementations generate virtual objects without also requiring a depth map of the real-world user’s body. This enables user devices with simple RGB cameras (without depth sensors) to accurately and quickly surface animated virtual objects in VR or AR applications based on a real-world user’s body, allowing users to interact with VR or AR content in a more realistic environment.

[0023] Networked computing environment

[0024] Figure 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each of which hosts several applications including messaging clients 104 and AR / VR applications 105. Each messaging client 104 and AR / VR application 105 is communicatively coupled to a messaging server system 108, as well as other instances of messaging clients 104 and AR / VR applications 105, via a network 106 (e.g., the Internet).

[0025] The messaging client 104 and the AR / VR application 105 are able to communicate and exchange data with one another and with the messaging server system 108 via the network 106. The data exchanged among the messaging clients 104, and between the messaging clients 104 and the messaging server system 108 includes functionality (e.g., commands for invoking functions) as well as payload data (e.g., textual data, audio, video or other multimedia data).

[0026] The AR / VR application 105 is an application that includes a set of functions that allow the client device 102 to access a body tracking system 126. In some implementations, the AR / VR application 105 is a component or feature that is part of the messaging client 104. The AR / VR application 105 uses an RGB camera to capture one or more images (e.g., videos) of a user’s real-world body. The AR / VR application 105 applies one or more trained machine learning techniques to the captured body images to detect the skeletal joint positions of the body. The AR / VR application 105 generates an outline of the user’s body, for example, by drawing lines connecting the detected different skeletal joint positions. The AR / VR application 105 tracks the motion of the skeletal joints in a set of video frames. For example, the AR / VR application 105 selects pairs of adjacent joints (e.g., pairs of shoulder and elbow joints and pairs of hip and knee joints).

[0027] The AR / VR application 105 tracks the motion of the pairs of joints in a set of video frames and measures the noise of each pair of joints detected in the set of frames. Based on the measured amount of noise, the AR / VR application 105 modifies the smoothing parameters of one or more smoothing filters applied to different pairs of joints. In one example, a first smoothing parameter is applied to a first set of filters that are applied to a first pair of joints, and a second smoothing parameter is applied to a second set of filters that are applied to a second pair of joints. The first smoothing parameter can cause the first set of filters to apply a greater amount of smoothing to the first pair of skeletal joints in a subsequent set of frames or the current set of frames of the video (e.g., due to detecting an amount of noise that is greater than a threshold). The second smoothing parameter can cause the second set of filters to apply a lesser amount of smoothing to the second pair of skeletal joints in the current or subsequent set of frames of the video (e.g., due to detecting an amount of noise that is less than a threshold). In this way, different amounts of smoothing are applied on a per-joint pair or per-joint group basis as video depicting a person is received and captured in real-time. In another example, multiple smoothing filters can be applied in parallel to the same group of joints, but the outputs of the smoothing filters can be weighted and aggregated according to the amount of noise in the motion of the group of joints in a window of frames. That is, the weights of the smoothing filters applied to the same group of joints can be controlled and changed according to the amount of noise in the motion of the joints measured in a set of frames.

[0028] In some implementations, the AR / VR application 105 detects the joint positions of the body depicted in the captured RGB image directly from the captured RGB image by obtaining a first trained machine learning technique from local or remote storage. The first trained machine learning technique processes the captured RGB image to extract one or more features from the RGB image that correspond to the body depicted in the captured RGB image. These features are analyzed to identify one or more skeletal joints and their respective arrangements relative to one another. In particular, these features are analyzed to determine the joint positions of a specified set of skeletal joints. The AR / VR application 105 also obtains a second trained machine learning technique to process one or more previously captured frames (e.g., video frames 1-2 seconds prior to the RGB image) to estimate or predict the skeletal joint positions of subsequent frames. A threshold number of video frames (which can be user-defined, previously specified, and / or dynamically determined) can be continuously or periodically stored in a buffer such that the threshold number of video frames prior to the current RGB image can be accessed by the second trained machine learning technique. The output or prediction of skeletal joint positions by the second trained machine learning technique is used to filter or refine the skeletal joint positions identified by the first trained machine learning technique. In some cases, the second trained machine learning technique processes the skeletal joint positions identified by the first trained machine learning technique along with the previously captured frames to filter or refine the estimated skeletal joint positions. Techniques for detecting skeletal joint positions are described in greater detail in commonly owned U.S. Patent Application No. 16 / 710,980 to Assouline et al., filed December 11, 2019, the entirety of which is incorporated by reference herein.

[0029] The messaging server system 108 provides server-side functionality via the network 106 to particular messaging clients 104. While certain functions of the messaging system 100 are described herein as being performed by messaging clients 104 or by the messaging server system 108, it can be possible for certain functionality to be performed by both a messaging client 104 and by the messaging server system 108, depending on the design choices made by the developers of the messaging system 100. For example, it can be technically preferable to initially deploy certain technology and functions within the messaging server system 108, but to later migrate this technology and functionality to the messaging client 104 as the client devices 102 become sufficiently powerful to support the additional processing requirements.

[0030] The messaging server system 108 supports a variety of services and operations that are provided to the messaging client 104. Such operations include transmitting data to, receiving data from, and processing data generated by the messaging client 104. This data can include message content, client device information, geolocation information, media

[0031] Turning now specifically to the messaging server system 108, an Application Program Interface (API) server 110 is coupled to, and provides a programmatic interface to, the application server 112, which is operable to manage a plurality of messaging applications. The application server 112 is communicably coupled to a database server 118, which facilitates access to a database 120 that stores client information, account information, message content, media, user-generated content, and other related information. Similarly, a web server 124 is coupled to the application server 112, and provides web-based interfaces to the application server 112. To this end, the web server 124 processes incoming network requests using the Hypertext Transfer Protocol (HTTP) and several related protocols.

[0032] The Application Program Interface (API) server 110 receives and transmits message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the Application Program Interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that the messaging client 104 can use to call functions used by the application server 112. The Application Program Interface (API) server 110 exposes various functions supported by the application server 112, including: account registration; login functionality; sending messages from a particular messaging client 104 to another messaging client 104 via the application server 112; sending media files (e.g., images or videos) from the messaging client 104 to the messaging server 114 and for possible access by another messaging client 104; setting a collection of media data (e.g., a story); retrieving a friend list of a user of the client device 102; retrieving such a collection; retrieving messages and content; adding and deleting entities (e.g., friends) to / from an entity graph (e.g., a social graph); locating friends in a social graph, and opening (e.g., related to the messaging client 104) application events.

[0033] The application servers 112 host a number of server applications and subsystems, including for example a messaging server 114, an image processing server 116, a body tracking system 126, and a social network server 122. The messaging server 114 implements a number of message processing technologies and functions, particularly related to the aggregation and other processing of content (e.g., textual and multimedia content) included in messages received from multiple instances of the messaging client 104. As will be described in further detail, textual and media content from multiple sources can be aggregated into content collections (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 104. Such processing can also be performed on the server side by the messaging server 114, taking into account hardware requirements for other processor and memory intensive processing of data.

[0034] The application servers 112 also include an image processing server 116 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payloads of messages sent from or received at the messaging server 114.

[0035] The social network server 122 supports various social networking functions and services and makes these functions and services available to the messaging server 114. To this end, the social network server 122 maintains and accesses an entity graph 306 (as shown in Figure 3 Examples of functions and services supported by the social network server 122 include identifying other users in the messaging system 100 that have a relationship or are "following" a particular user, as well as identifying interests and other entities of a particular user.

[0036] System Architecture

[0037] Figure 2 is a block diagram illustrating further details regarding the messaging system 100, according to some examples. In particular, the messaging system 100 is shown to include the messaging client 104 and the application servers 112. The messaging system 100 includes a number of subsystems that are supported on the client side by the messaging client 104 and on the server side by the application servers 112. These subsystems include, for example, a ephemeral timer system 202, a collection management system 204, an augmentation system 206, a map system 208, a game system 210. In some implementations, the augmentation system 206 implements some or all of the functionality of the body tracking system 126.

[0038] The ephemeral timer system 202 is responsible for implementing temporary or time-limited access to content by the messaging client 104 and the messaging server 114. The ephemeral timer system 202 includes a number of timers that selectively enable access (e.g., for presentation and display) of messages and associated content via the messaging client 104 based on a duration and display parameters associated with a message or collection of messages (e.g., a story). Additional details regarding the operation of the ephemeral timer system 202 are provided below.

[0039] The collection management system 204 is responsible for managing groups and collections of media (e.g., collections of text, image, video, and audio data). Collections of content (e.g., messages including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be made available for a specified period of time, e.g., for the duration of an event to which the content relates. For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing icons to the user interface of the messaging client 104 that provide notification of the existence of particular collections.

[0040] In addition, the collection management system 204 also includes a curation interface 212 that allows collection managers to be able to manage and curate particular collections of content. For example, the curation interface 212 enables event organizers to curate collections of content related to a particular event (e.g., to delete inappropriate content or redundant messages). In addition, the collection management system 204 employs machine vision (or image recognition techniques) and content rules to automatically curate collections of content. In certain examples, users can be paid compensation for including user-generated content into a collection. In such cases, the curation management system 204 operates to automatically pay such users for use of their content.

[0041] The augmentation system 206 provides various functionality that enables users to augment (e.g., annotate or otherwise modify or edit) media content associated with messages. For example, the augmentation system 206 provides functionality related to generating and publishing media overlays for messages handled by the messaging system 100. The augmentation system 206 is operable to supply media overlays or augmentations (e.g., image filters) to the messaging client 104 based on a geographic location of the client device 102. In another example, the augmentation system 206 is operable to supply media overlays to the messaging client 104 based on other information such as social network information of a user of the client device 102. The media overlays can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects can be applied to a media content item (e.g., a photo) at the client device 102. For example, the media overlays can include text or images that can be overlaid on a photo taken by the client device 102. In another example, the media overlays include a location identification overlay (e.g., Venice Beach), a live event name or business name overlay (e.g., Beachside Café). In another example, the augmentation system 206 uses a geographic location of the client device 102 to identify a media overlay that includes a name of a business at the geographic location of the client device 102. The media overlays can include other indicia associated with the business. The media overlays can be stored in the database 120 and accessed by the database server 118.

[0042] In some cases, the body tracking system 126 or portions of the body tracking system 126 can be implemented by or included in the augmentation system 206.

[0043] In some examples, the augmentation system 206 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify an environment in which a particular media overlay should be provided to other users. The augmentation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0044] In other examples, the augmentation system 206 provides a business-based publishing platform that enables businesses to select a particular media overlay associated with a geographic location via a bidding process. For example, the augmentation system 206 associates a media overlay of a highest bidding business with a corresponding geographic location for a predefined amount of time.

[0045] The map system 208 provides various geographic location functionality and supports the presentation by the messaging client 104 of map-based media content and messages. For example, the map system 208 enables the display of user icons or avatars (e.g., stored in the profile data 308) on a map to indicate the current or past locations of a user's "friends" and media content (e.g., collections of messages including photos and videos) generated by such friends in the context of a map. For example, on a map interface of the messaging client 104, a message posted by a user to the messaging system 100 from a particular geographic location can be displayed to a particular user's "friends" in the context of a map at that particular location. A user can also share his or her location and status information with other users of the messaging system 100 via the messaging client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users in the context of a map interface of the messaging client 104.

[0046] The game system 210 provides various game functionality in the context of the messaging client 104. The messaging client 104 provides a game interface that provides a list of available games that can be launched by a user in the context of the messaging client 104 and played with other users of the messaging system 100. The messaging system 100 also enables a particular user to invite other users to participate in playing a particular game by issuing invitations from the messaging client 104 to the other users. The messaging client 104 also supports both voice messaging and text messaging (e.g., chat) in the context of playing games, provides a leaderboard for games, and also supports the provision of in-game rewards (e.g., game coins and items).

[0047] Data Architecture

[0048] Figure 3 FIG. 3 is a schematic diagram illustrating a data structure 300 that can be stored in the databases 120 of the messaging server system 108, in accordance with certain examples. While the contents of the databases 120 are illustrated as including several tables, it will be appreciated that data can be stored in other types of data structures (e.g., as an object-oriented database).

[0049] The databases 120 include message data stored within a messages table 302. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. The messages table 302 is described in more detail below with reference to FIG. 4. Figure 4 Additional details are described with respect to information that can be included in a message and included in the message data stored in the messages table 302.

[0050] The entity table 304 stores entity data and is linked to (e.g., references) the entity graph 306 and profile data 308. Entities for which records are maintained within the entity table 304 can include individuals, corporate entities, organizations, objects, places, events, and the like. Regardless of entity type, any entity for which the messaging server system 108 stores data can be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0051] The entity graph 306 stores information about relationships and associations between entities. Such relationships can be social, professional (e.g., working at a common company or organization), interest-based, or activity-based, merely as examples.

[0052] The profile data 308 stores various types of profile data about particular entities. Based on privacy settings specified by a particular entity, the profile data 308 can be selectively used and presented to other users of the messaging system 100. In the case of an entity that is a person, the profile data 308 includes, for example, a user name, phone number, address, settings (e.g., notification and privacy settings), and a user-selected avatar representation (or collection of such avatar representations). The particular user can then selectively include one or more of these avatar representations in the content of messages communicated via the messaging system 100 as well as on map interfaces displayed by the messaging client 104 to other users. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can select to convey at a particular time.

[0053] In the case of an entity that is a group, the profile data 308 for the group can similarly include one or more avatar representations associated with the group in addition to the group name, members, and various settings (e.g., notifications) for the related group.

[0054] The database 120 also stores augmentation data, e.g., overlays or filters, in an augmentation table 310. The augmentation data is associated with and applied to videos (whose video data is stored in a video table 314) and images (whose image data is stored in an image table 316).

[0055] In one example, the filter is an overlay that is displayed as an overlay on the image or video during presentation to the receiving user. The filter can be of various types, including a user-selected filter from a set of filters presented by the messaging client 104 to the sending user when the sending user is composing a message. Other types of filters include a geolocation filter (also referred to as a geo filter) that can be presented to the sending user based on a geolocation. For example, a geolocation filter specific to a nearby or special location can be presented by the messaging client 104 within the user interface based on geolocation information determined by a global positioning system (GPS) unit of the client device 102.

[0056] Another type of filter is a data filter that can be selectively presented to the sending user by the messaging client 104 based on other inputs or information collected by the client device 102 during the message creation process. Examples of data filters include a current temperature at a particular location, a current speed at which the sending user is traveling, a battery life of the client device 102, or a current time.

[0057] Other augmentation data that can be stored within the image table 316 includes augmented reality content items (e.g., corresponding to an application lens or augmented reality experience). The augmented reality content items can be real-time special effects and sounds that can be added to the image or video.

[0058] The smoothing filters 306 store a plurality of smoothing filters used by the body tracking system 126. The smoothing filters can apply different amounts or types of smoothing to the joint or group or pair of joints to which they are applied. The smoothing filters can each include a parameter that adjusts the filter coefficients to increase or decrease the amount of smoothing applied by the given smoothing filter.

[0059] As described above, augmentation data includes augmented reality content items, overlays, image transformations, AR images, and similar items that refer to modifications that can be applied to image data (e.g., video or images). This includes real-time modifications that modify images as they are captured using device sensors (e.g., one or more cameras) of the client device 102 and then displayed on the screen of the client device 102 with the modifications. This also includes modifications to stored content (e.g., video segments in a library that can be modified). For example, in a client device 102 that has access to multiple augmented reality content items, a user can use multiple augmented reality content items for a single video segment to see how different augmented reality content items would modify the stored segment. For example, multiple augmented reality content items that apply different pseudo-random motion models can be applied to the same content by selecting different augmented reality content items for the same content. Similarly, real-time video capture can be used with shown modifications to show how video images currently being captured by the sensors of the client device 102 would modify the captured data. Such data can simply be shown on screen without being stored in memory, or the content captured by the device sensors can be recorded and stored in memory with or without the modifications (or both). In some systems, a preview feature can show how different augmented reality content items would look at the same time in different windows in the display. For example, this can enable multiple windows with different pseudo-random animations to be viewed on the display at the same time.

[0060] Accordingly, data using augmented reality content items and various systems or other such transformation systems that use the data to modify content can involve: detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.); tracking of such objects as they leave, enter, and move around the field of view in video frames; and modification or transformation of such objects as they are tracked. In various implementations, different methods for implementing such transformations can be used. Some examples can involve generating three-dimensional mesh models of one or more objects, and using transformations of the models and animated textures within the video to implement the transformations. In other examples, tracking of points on objects can be used to place images or textures (which can be two-dimensional or three-dimensional) at the tracked locations. In still further examples, neural network analysis of video frames can be used to place images, models, or textures in content (e.g., images or frames of a video). Accordingly, augmented reality content items refer both to images, models, and textures used to create transformations in content, and to additional modeling and analysis information needed to implement such transformations with object detection, tracking, and placement.

[0061] Real-time video processing can be performed with any kind of video data (e.g., video streams, video files, etc.) saved in the memory of any kind of computerized system. For example, a user can load a video file and save it in the memory of a device, or can generate a video stream using the sensors of the device. Moreover, any object can be processed using computer animation models, such as parts of human faces and bodies, animals, or non-biological objects (e.g., chairs, cars, or other objects).

[0062] In some examples, when a particular modification is selected together with the content to be transformed, the elements to be transformed are identified by the computing device, and then the elements to be transformed are detected and tracked if they exist in the frames of the video stream. The elements of the object are modified according to the modification request, thereby transforming the frames of the video stream. For different kinds of transformations, the transformation of the frames of the video stream can be performed by different methods. For example, for frame transformations that mainly involve changing the form of the elements of the object, feature points of each element of the object are computed (e.g., using an Active Shape Model (ASM) or other known methods). Then, a mesh based on the feature points is generated for each of the at least one element of the object. The mesh is used in the following stages of tracking the elements of the object in the video stream. During the tracking process, the mentioned mesh of each element is aligned with the position of each element. Then, additional points are generated on the mesh. A first set of points is generated for each element based on the modification request, and a second set of points is generated for each element based on the first set of points and the modification request. Then, the frames of the video stream can be transformed by modifying the elements of the object based on the first set of points and the second set of points and the mesh. In such a method, the background of the modified object can also be changed or distorted by tracking and modifying the background.

[0063] In some examples, the transformation of changing some regions of the object using the elements of the object can be performed by computing feature points of each element of the object and generating a mesh based on the computed feature points. Points are generated on the mesh, and then various regions based on the points are generated. Then, the elements of the object are tracked by aligning the regions of each element with the position of each of the at least one element, and the properties of the regions can be modified based on the modification request, thereby transforming the frames of the video stream. Depending on the specific modification request, the properties of the mentioned regions can be transformed in different ways. Such modifications can involve: changing the color of the region; removing at least part of the region from the frames of the video stream; including one or more new objects in the region based on the modification request; and modifying or distorting the region or the elements of the object. In various implementations, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some of the feature points can be selected as control points for determining the entire state space of options for the model animation.

[0064] In some examples of computer animation models that use face detection to transform image data, a particular face detection algorithm (e.g., Viola-Jones) is used to detect faces on an image. An Active Shape Model (ASM) algorithm is then applied to the face region of the image to detect facial feature landmarks.

[0065] In other examples, other methods and algorithms suitable for face detection can be used. For example, in some implementations, landmarks are used to locate features, which are distinguishable points that are present in most images under consideration. For example, for face landmarks, the position of the left eye pupil can be used. If the initial landmark is not identifiable (e.g., if a person has an eye patch), a secondary landmark can be used. Such a landmark identification process can be used for any such object. In some examples, a set of landmarks forms a shape. A shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another using a similarity transformation (allowing for translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0066] In some examples, the search for landmarks begins from an average shape that is aligned with the position and size of the face determined by the global face detector. Such a search then repeats the following steps: a tentative shape is suggested by adjusting the position of the shape points through template matching of the image texture around each point, and then conforming the tentative shape to the global shape model until convergence occurs. In some systems, the individual template matches are unreliable, and the shape model pools the results of the weak template matches to form a stronger overall classifier. The entire search is repeated at each level of an image pyramid, from coarse resolution to fine resolution.

[0067] The transformation system can capture images or video streams on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102, while maintaining a suitable user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, emotion transfers (e.g., changing a face from a frown to a smile), state transfers (e.g., aging a subject, reducing apparent age, changing gender), style transfers, graphic element applications, and any other suitable image or video manipulations implemented by convolutional neural networks that have been configured to execute efficiently on the client device 102.

[0068] In some examples, a computer animation model for transforming image data can be used by a system in which a user can capture an image or video stream of the user (e.g., a selfie) using a client device 102 having a neural network operating as part of a messaging client 104 operating on the client device 102. The transform system operating within the messaging client 104 determines that a face is present within the image or video stream and provides a modification icon associated with a computer animation model for transforming the image data, or the computer animation model can be presented in association with an interface described herein. The modification icon includes a change that can be the basis for modifying the face of the user within the image or video stream as part of a modification operation. Once the modification icon is selected, the transform system initiates a process to convert the image of the user to reflect the selected modification icon (e.g., generate a smiling face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on the client device 102. The transform system can implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, the user can capture an image or video stream and the results of the modification are presented in real-time or near real-time once the modification icon is selected. Further, the modification can be persistent while a video stream is being captured and the selected modification icon remains toggled. Machine-taught neural networks can be used to implement such modifications.

[0069] The graphical user interface presenting the modifications performed by the transform system can supply additional interaction options for the user. Such options can be based on the interface used to initiate the selection of a particular computer animation model and content capture (e.g., initiated from a content creator user interface). In various implementations, the modification can be persistent after an initial selection of a modification icon. The user can turn on or off the modification by tapping or otherwise selecting the face modified by the transform system and store it for later viewing or browsing to other areas of the imaging application. In cases where multiple faces are modified by the transform system, the user can turn on or off the modification globally by tapping or selecting the individual face modified and displayed within the graphical user interface. In some implementations, individual faces within a group of multiple faces can be modified individually or such modifications can be toggled individually by tapping or selecting individual faces or a series of individual faces displayed within the graphical user interface.

[0070] Story table 312 stores data regarding collections of messages and associated image, video, or audio data that are assembled into collections (e.g., stories or libraries). The creation of a particular collection can be initiated by a particular user (e.g., each user for which a record is maintained in entity table 304). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of messaging client 104 can include a user-selectable icon to enable a sending user to add particular content to his or her personal story.

[0071] A collection can also constitute a "live story" that is a collection of content from multiple users, created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Users whose client devices are enabled for location services and are at a public location event at a particular time can be presented with an option to contribute content to a particular live story, e.g., via the user interface of messaging client 104. A user can be identified with a live story by messaging client 104 based on his or her location. The end result is a "live story" told from the perspective of the community.

[0072] Another type of collection of content is referred to as a "location story," which enables users whose client devices 102 are located within a particular geographic location (e.g., at a college or university campus) to contribute to a particular collection. In some implementations, contributions to a location story can require secondary authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student of a university campus).

[0073] As mentioned above, video table 314 stores video data that, in one example, is associated with messages for which records are maintained within message table 302. Similarly, image table 316 stores image data that is associated with messages for which message data is stored in entity table 304. Entity table 304 can associate various augmentations from augmentation table 310 with various images and videos stored in image table 316 and video table 314.

[0074] Data Communication Architecture

[0075] Figure 4is a schematic diagram showing the structure of a message 400 generated by a messaging client 104 for transmission to another messaging client 104 or messaging server 114, according to some examples. The contents of the particular message 400 are used to populate the message table 302 stored within the database 120, which is accessible by the messaging server 114. Similarly, the contents of the message 400 are stored in memory as "in-transit" or "in-flight" data for the client device 102 or application server 112. The message 400 is shown to include the following example components:

[0076] • message identifier 402: a unique identifier that identifies the message 400.

[0077] • message text payload 404: text to be generated by a user via a user interface of the client device 102 and included in the message 400.

[0078] • message image payload 406: image data captured by a camera component of the client device 102 or retrieved from a memory component of the client device 102 and included in the message 400. Image data for sent or received messages 400 can be stored in the image table 316.

[0079] • message video payload 408: video data captured by a camera component or retrieved from a memory component of the client device 102 and included in the message 400. Video data for sent or received messages 400 can be stored in the video table 314.

[0080] • message audio payload 410: audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400.

[0081] • message augmentation data 412: augmentation data (e.g., filters, stickers, or other annotations or augmentations) representing augmentations to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400. Augmentation data for sent or received messages 400 can be stored in the augmentation table 310.

[0082] • message duration parameter 414: a parameter value indicating an amount of time in seconds that the content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) is to be presented to a user via the messaging client 104 or is accessible by a user.

[0083] • message geolocation parameters 416: geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 416 values can be included in the payload, each of which is associated with a content item included in the content (e.g., a particular image in the message image payload 406, or a particular video in the message video payload 408).

[0084] • message story identifiers 418: identifier values that identify one or more content collections (e.g.,“stories” identified in the story table 312) with which particular content items in the message image payload 406 of the message 400 are associated. For example, multiple images within the message image payload 406 can each be associated with multiple content collections using identifier values.

[0085] • message tags 420: each message 400 can be tagged with multiple tags, each of which indicates a subject matter of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value indicating the relevant animal can be included within the message tags 420. Tag values can be manually generated based on user input, or can be automatically generated using, for example, image recognition.

[0086] • message sender identifier 422: an identifier (e.g., a messaging system identifier, email address, or device identifier) that indicates a user of the client device 102 on which the message 400 was generated and from which the message 400 was sent.

[0087] • message recipient identifier 424: an identifier (e.g., a messaging system identifier, email address, or device identifier) that indicates a user of the client device 102 to which the message 400 is addressed.

[0088] The content (e.g., values) of the various components of the message 400 can be pointers to locations in tables within which the content data values are stored. For example, the image values in the message image payload 406 can be pointers to locations (or addresses of locations) within the image table 316. Similarly, the values within the message video payload 408 can point to data stored within the video table 314, the values stored within the message augmentation data 412 can point to data stored in the augmentation table 310, the values stored within the message story identifiers 418 can point to data stored in the story table 312, and the values stored within the message sender identifier 422 and the message recipient identifier 424 can point to user records stored within the entity table 304.

[0089] Figure 5Ais a block diagram illustrating an example body tracking system 126 according to an example embodiment. The body tracking system 126 operates on a set of input data (e.g., a video 501 depicting a real-world body of a user). The body tracking system 126 includes a machine learning technique module 512, a skeletal joint location module 514, a smoothing module 516, and a virtual object display module 520.

[0090] In some embodiments, the body tracking system 126 includes a user detection component (not shown). In response to receiving an input from a user to activate a front-facing or rear-facing camera of the client device 102, the user detection component is activated. Once activated, the user detection component analyzes features of a set of images captured by the camera of the client device. The user detection component determines whether the features match an object corresponding to a human or a person. In this case, the user detection component initiates a process for tracking the motion of the user in frames (e.g., a set of images) of the video. Specifically, the user detection component instructs the machine learning technique module 512, the skeletal joint location module 514, and the smoothing module 516 to track the skeletal joints of the user and to independently smooth the motion of one set of joints or to smooth the motion of another set of joints in the frames of the video. In some cases, the user detection component is part of the machine learning technique module 512.

[0091] The machine learning technique module 512 (e.g., a deep neural network) extracts one or more features from a given input video to estimate skeletal joint locations of skeletal joints depicted in the video. For example, the machine learning technique module 512 obtains a given video depicting a user’s face, arms, torso, hips, and legs. The machine learning technique module 512 extracts features corresponding to the user’s face and arms from the video to identify one or more skeletal joints (e.g., the joints shown in FIG. 5, including left / right wrist joints, left / right elbow joints, left / right shoulder joints, and a nose position). Figure 6 The machine learning technique module 512 (e.g., a deep neural network) extracts one or more features from a given input video to estimate skeletal joint locations of skeletal joints depicted in the video. For example, the machine learning technique module 512 obtains a given video depicting a user’s face, arms, torso, hips, and legs. The machine learning technique module 512 extracts features corresponding to the user’s face and arms from the video to identify one or more skeletal joints (e.g., the joints shown in FIG. 5, including left / right wrist joints, left / right elbow joints, left / right shoulder joints, and a nose position).

[0092] The machine learning technique module 512 generates pairs or groups of joints to be smoothed. For example, the machine learning technique module 512 communicates with the skeletal joint position module 514 to specify different groups of joints to be tracked and smoothed independently of other groups of joints. As an example, the skeletal joint position module 514 can specify the neck joint and the shoulder joint as a first pair of joints to be tracked and smoothed together; can specify the hip joint, the left leg joint, and the left knee joint as another group of joints to be tracked and smoothed together, and so on. Any number of joints less than all of the detected joints (e.g., one joint, two adjacent joints, three adjacent joints) can be included in a given group of joints that are tracked and smoothed together. In this way, the skeletal joint position module 514 tracks the joints that are part of one group separately from the joints that are part of another group in a group of video frames to measure noise, and adjusts the smoothing applied to one group independently of the smoothing applied to the joints of the other group.

[0093] The extracted features of the machine learning technique module 512 are provided to the skeletal joint position module 514. The skeletal joint position module 514 analyzes the skeletal joint features to determine the coordinates of particular skeletal joints. For example, the skeletal joint position module 514 determines the x, y coordinates of particular points of each skeletal joint, such as the x, y coordinates of the left wrist, the x, y coordinates of the left elbow, the x, y coordinates of the left shoulder, the x, y coordinates of the nose (or other facial features such as the mouth, ears, or eyes), the x, y coordinates of the right wrist, the x, y coordinates of the right elbow, and the x, y coordinates of the right shoulder. Using these coordinates, the skeletal joint position module 514 specifies pairs or groups of joints.

[0094] In some cases, a neural network or machine learning technique can be applied to automatically select the joints to be tracked and smoothed as part of each group adaptively. For example, a machine learning technique can be trained based on a set of training video to predict that a first group of joints (e.g., the neck joint, the left shoulder joint, and the left elbow joint) move more or cause a greater amount of noise in a group of frames than another group of joints (e.g., the hip joint and the left leg joint). In this case, when operating on a new group of video frames, the machine learning technique can instruct the skeletal joint position module 514 to form a first group of joints that includes the neck joint, the left shoulder joint, and the left elbow joint, and to exclude the hip joint and the left leg joint from the first group. Alternatively, the skeletal joint position module 514 can only track and smooth groups of joints that are predicted by the matching learning technique to have an amount of noise greater than a threshold value based on a set of training video. In some other cases, the groups of joints that are tracked and smoothed are manually specified by a user.

[0095] The smoothing module 516 retrieves a set of smoothing filters 306 and applies the smoothing filters 306 to a first set of joints provided by the skeletal joint position module 514 independently of applying the smoothing filters 306 to a second set of joints provided by the skeletal joint position module 514. For example, the smoothing module 516 applies a first set of smoothing filters in parallel to a first set of skeletal joints of the plurality of skeletal joints. The smoothing module 516 applies a second set of smoothing filters in parallel to a second set of skeletal joints of the plurality of skeletal joints. The smoothing module 516 adapts a first parameter of the first set of smoothing filters independently of a second parameter of the second set of smoothing filters. For example, the smoothing module 516 adapts a weight of each filter in the first set of smoothing filters to control an amount of smoothing applied to the first set of joints. The smoothing module 516 aggregates weighted outputs of the filters applied to the first set of joints.

[0096] In some implementations, the smoothing module 516 accesses a set of previous frames (e.g., 1-2 seconds of past video). The smoothing module 516 analyzes motion of a skeletal joint or a set of skeletal joints in the set of previous frames. For example, after or before measuring a signal quality parameter representative of an amount of noise in motion of a first set of skeletal joints appearing in the previous frames, the smoothing module 516 applies a plurality of smoothing filters to the skeletal joint. The smoothing module 516 also applies a same or different plurality of smoothing filters to a second set of skeletal joints, the plurality of smoothing filters having a same or different smoothing parameter than the smoothing filters applied to the first set of skeletal joints. In some implementations, the plurality of smoothing filters are applied in parallel to the first set of skeletal joints and the plurality of smoothing filters are also applied in parallel to the second set of skeletal joints. That is, a first smoothing filter and a second smoothing filter of the plurality of smoothing filters are applied simultaneously or in parallel to smooth motion of the first set of skeletal joints in the set of previous frames. As another example, a second smoothing filter and a third smoothing filter of the plurality of smoothing filters are applied simultaneously or in parallel to smooth motion of the second set of skeletal joints in the set of previous frames. Outputs of the smoothing filters applied to the first set of skeletal joints / second set of skeletal joints are weighted and summed based on the signal quality parameter.

[0097] The smoothing module 516 measures signal stability using the signal stability parameter for each of the first set of skeletal joints and the second set of skeletal joints. Specifically, the smoothing module 516 measures the amount of noise resulting from smoothing the first set of skeletal joints in a previous frame, and separately measures the amount of noise resulting from smoothing the second set of skeletal joints in a previous frame. Based on the amount of noise resulting from the smoothing operation, the smoothing module 516 adjusts the smoothing parameters of the smoothing filters applied to the different set of skeletal joints and to the same set of skeletal joints (e.g., by controlling the amount of weight applied to each of the multiple filters applied to the same set of skeletal joints). In some implementations, the smoothing module 516 interpolates the motion and smoothed output of the first skeletal joints to measure the noise, and separately interpolates the motion and smoothed output of the second skeletal joints to measure the noise in the motion of the different set of joints.

[0098] In one example, the smoothing module 516 determines that the interpolated motion of the first set of skeletal joints results in noise that exceeds a threshold. That is, the signal stability parameter representing the noise in the motion of the first set of skeletal joints that is measured and calculated can exceed a threshold. In this case, the amount of noise can indicate that additional smoothing is needed. Accordingly, the smoothing module 516 increases the smoothing parameters of the multiple smoothing filters (or a subset of the smoothing filters) applied to the first set of skeletal joints when receiving and processing a subsequent set of video frames or for the current set of frames for which the noise was measured. That is, the smoothing module 516 can measure the noise in a set of frames, and then adjust the amount of smoothing applied to the joints of the person depicted in that set of frames based on the measured noise. As an example, the smoothing module 516 controls the weights of the smoothing filters applied in parallel to the joints of a given set based on the signal stability parameter. One smoothing filter having one characteristic can have a greater weight than a second smoothing filter having another characteristic that is applied in parallel to the same set of joints.

[0099] In another example, the smoothing module 516 determines that the interpolated motion of the second set of skeletal joints results in noise that does not exceed (or is less than) a threshold. That is, the signal stability parameter representing the noise in the motion of the second set of skeletal joints that is measured and calculated can not exceed a threshold. In this case, the amount of noise can indicate that less smoothing is needed. Accordingly, the smoothing module 516 decreases the smoothing parameters of the multiple smoothing filters (or a subset of the smoothing filters) applied to the second set of skeletal joints when receiving and processing a subsequent set of video frames or for the current set of frames for which the noise was measured. In one implementation, the smoothing applied to the first set of skeletal joints can be different than the smoothing applied to the second set of skeletal joints at the same time.

[0100] In some cases, the threshold for noise caused by the motion of the first set of skeletal joints can be different than the threshold for noise caused by the motion of the second set of skeletal joints. In some cases, the two thresholds can be the same. That is, each set of skeletal joints can be associated with a different threshold, which their respective signal stability parameters are compared against to adjust the smoothing parameter.

[0101] Figure 5B is a block diagram illustrating an example smoothing module 516, according to an example implementation. The smoothing module 516 includes or has access to a frame store 590, a filter update module 592, a smoothing filter 306, an adaptive filter computation module 593, and a filtered 3D skeletal joints module 594.

[0102] The frame store 590 includes a window of previous frames of the received real-time video 501, as well as the keypoints and scores of those frames. The number of previous frames can include any number of frames (e.g., 2 frames, 10 frames, 100 frames, or any other suitable number). The smoothing module 516 initializes a set of K motion filters by accessing the smoothing filter 306. The number K in the set of motion filters can be selected based on the number of joints to be tracked and smoothed or the number of sets of joints to be tracked and smoothed. For example, if there are three sets of joints to be tracked and smoothed, three different motion filters can be selected to be configured with different respective characteristics.

[0103] In some implementations, each motion filter in the set of K motion filters includes X different versions. As an example, the motion filters can include a soft characteristic and an aggressive characteristic, where the responsiveness of the motion filter can be increased when the motion filter is configured to the soft characteristic than when configured to the aggressive characteristic. The amount of smoothing (e.g., the value of the weight applied to the output of the smoothing filter) is controlled such that the responsiveness of the smoothing filter is increased without over-smoothing or under-smoothing. That is, when the motion of the joints in a given frame window has noise that exceeds a threshold, the output of the aggressive filter is weighted more heavily than the output of the soft filter applied to the same set of joints, which decreases the responsiveness but increases the amount of smoothing. When the motion of the joints in a given frame window has noise that is less than a threshold, the output of the aggressive filter is weighted less heavily than the output of the soft filter applied to the same set of joints, which increases the responsiveness but decreases the amount of smoothing. Initially, the filter update module 592 communicates with the smoothing filter 306 to obtain the set of K motion filters and an initial set of characteristics for each obtained filter.

[0104] After a number of frames in the frame store 590 are processed to evaluate the signal quality parameter S, the filter update module 592 adjusts or selects the characteristics of each obtained filter. The smoothing module 516 can compute the signal quality parameter S according to the standard deviation of the differences between adjacent frames in the set of frames. In particular, the smoothing module 516 computes the signal quality parameter S according to the following equation: s t = std t ({d i |t-T...t}), where T is the number of frames in the set of frames in the frame store 590, d i = x i -x i-1 is the difference between two adjacent frames.

[0105] In some implementations, if the signal quality parameter S associated with the first set of joints is greater than a specified threshold, the first filter selected to apply to the first set of joints can be configured to have a soft characteristic. As another example, if the signal quality parameter S associated with the first set of joints is less than a specified threshold, the second filter selected to apply to the first set of joints can be configured to have an aggressive characteristic. In some cases, the output of the filter is specified and controlled by a weight. That is, a higher weight value applied to an aggressive filter than a soft filter can result in a greater amount of smoothing applied to the set of skeletal joints but a decrease in responsiveness. A lower weight value applied to an aggressive filter than a soft filter can result in a lesser amount of smoothing applied to the set of skeletal joints but an increase in responsiveness.

[0106] In some implementations, the filter update module 592 adjusts the weight between a soft filter and an aggressive filter applied in parallel to a given set of joints based on the number of stable frames in which the signal quality parameter S is measured to be below a predetermined threshold. This value is averaged over a window of frames in the frame store 590 based on the weight associated with the respective filter. In particular, the filter update module 592 computes a first weight associated with a soft filter characteristic according to w soft = E[s t < threshold] and a second weight associated with an aggressive filter characteristic according to w aggressive = E[s t > threshold]. In this way, if the signal quality parameter S (e.g., a standard deviation measure of stability over a number of past frames) is below a threshold, a soft weight (w soft ) is selected to be applied to the corresponding set of joints by the motion filter. If the signal quality parameter S (e.g., a standard deviation measure of stability over a number of past frames) is above or equal to the threshold, an aggressive weight (w aggressive) applied to the corresponding set of joints. As an example, if the signal quality parameter S (e.g., a standard deviation measure of stability over a number of past frames) is above or equal to a threshold, the weight of the output of the aggressive filter is increased relative to the weight applied to the output of the more gentle filter to aggregate the smoothing of the motion of the skeletal joints.

[0107] Multiple filters can be applied in parallel to the same set of joints. Each of the multiple filters can include different filter characteristics. For example, a gentle motion filter can be applied in parallel to an aggressive motion filter to a first set of joints. If the signal quality parameter S (e.g., a standard deviation measure of stability over a number of past frames) is above or equal to a threshold, the filter update module 592 can assign a lower weight to the output of the gentle motion filter compared to the weight assigned to the output of the aggressive motion filter. As another example, if the signal quality parameter S (e.g., a standard deviation measure of stability over a number of past frames) is below a threshold, the filter update module 592 can assign a higher weight to the output of the gentle motion filter compared to the weight assigned to the output of the aggressive motion filter. The smoothed motion of the 3D skeletal joints corresponding to the set of joints can then be based on a weighted combination of the gentle motion filter and the aggressive motion filter.

[0108] The adaptive filter computation module 593 aggregates a set of filters with respective weights. In some implementations, the adaptive filter computation module 593 retrieves a set of frames from the frame store 590 and applies the filters with associated weights to the corresponding set of skeletal joints using the filter 3D skeletal joints module 594. In particular, the adaptive filter computation module 593 receives the raw 3D skeletal joints from the skeletal joint position module 514 and filters the motion of the 3D skeletal joints in a set of frames in the frame store 590 (or frames received in a subsequent frame window) using the aggregated set of filters. For example, the adaptive filter computation module 593 computes the final signal as follows: where f i (x) represents, for all filters X, the motion filter i applied to a set of joints x with a given weight w i In this way, each filter applied to the same set of joints can be assigned a different weight and the outputs of all filters applied to the same set of joints are aggregated to smooth the motion of the joints.

[0109] The filtered 3D skeletal joints are output as the weighted filter outputs f iThe average of (x). Specifically, the adaptive filter computation module 593 obtains a first frame from the frame store 590 and identifies a set of joints of a person depicted in the first frame. The adaptive filter computation module 593 smooths the motion of the set of joints from the first frame to a next adjacent frame or in the set of frames according to a motion filter with selected filter characteristics for the set of joints (e.g., an aggressive filter or a soft filter or a weighted combination of an aggressive filter and a soft filter). In some cases, the adaptive filter computation module 593 applies a soft filter to a set of joints in a first set of frames that appear in the set of frames in the frame store 590 and then switches to applying an aggressive filter to the same set of joints in a second set of frames that appear in the set of frames in the frame store 590. This can be a result in a case where a signal quality parameter for the second set of frames is lower than the signal quality parameter for the first set of frames for the set of joints. In some cases, in a case where a signal quality parameter for a set of frames is below a threshold, the adaptive filter computation module 593 applies a weight with a first value to a soft filter to smooth a set of joints in a first set of frames in a set of frames that appear in the frame store 590 and applies a weight with a second value to an aggressive filter to smooth the same set of joints in the first set of frames. This results in a greater amount of smoothing being applied to the set of joints in the first set of frames.

[0110] In this way, the smoothing module 516 operates on a set of previously received video frames to compute a signal quality parameter that represents noise of motion of a set of joints or multiple sets of joints of an object or person depicted in the set of frames. Based on the signal quality, the smoothing module 516 selectively and adaptively modifies an amount of smoothing (e.g., how aggressive or soft the motion filter characteristics are set) applied to individual joints in the set of joints. The respective joints in the previously received frames are then smoothed using respective filters with the same or different motion filter characteristics. After or during smoothing of the motion of the joints in the previously received frames, the avatar or virtual object can be adaptively moved in a similarly smoothed manner in accordance with the smoothed motion of the joints of the real-world object or person.

[0111] The virtual object display module 520 can adjust the skeletal rig of a given avatar based on the smoothed skeletal joints detected and tracked from the user's video. The virtual object display module 520 adjusts, for example, the way the avatar moves in the image by modifying the avatar's pose, visual attributes, and / or position in the image. In some implementations, the virtual object display module 520 incorporates the adjusted avatar into the received video depicting the user's body such that both the adjusted avatar and the user are presented simultaneously in the video. The image is provided by the virtual object display module 520 to the client device 102, which can then be transmitted to another user or stored for later access and display.

[0112] Figure 7 is a flowchart showing example operations of the body tracking system 126 in performing the process 700, according to an example implementation. The process 700 can be implemented in computer-readable instructions for execution by one or more processors such that the operations of the process 700 can be performed, in part or in whole, by functional parts of the messaging server system 108 and / or the AR / VR application 105; thus, the process 700 is described below by way of example with reference thereto. However, in other implementations, at least some of the operations of the process 700 can be deployed on various other hardware configurations. Thus, the process 700 is not intended to be limited to the messaging server system 108 and can be implemented in whole or in part by any other component. Some or all of the operations of the process 700 can be parallel, out of order, or omitted entirely.

[0113] At operation 701, the body tracking system 127 receives a video that includes a depiction of a user’s body. For example, the machine learning techniques module 512 receives the video 501 that depicts the user’s body. The machine learning techniques module 512 extracts one or more features from the image that are indicative of a skeletal joint.

[0114] At operation 702, the body tracking system 126 detects a plurality of skeletal joints of the body depicted in the video. For example, the skeletal joint position module 514 receives the output of the machine learning techniques module 512 and labels or generates a depiction of the skeletal joints (as shown in FIG. 5B) for tracking the joints in a set of frames. Figure 6

[0115] At operation 703, the body tracking system 126 tracks the motion of the plurality of skeletal joints in a set of video frames. For example, the skeletal joint position module 514 collects different sets of skeletal joints or pairs of skeletal joints into different groups for tracking and smoothing. In some cases, the skeletal joint position module 514 groups joints that are determined to cause the largest amount of noise according to the machine learning techniques into the same group, as long as they are adjacent joints.

[0116] ​At operation 704, the body tracking system 126 independently smooths the motion of a first set of the plurality of skeletal joints in the first set of frames from the motion of a second set of the plurality of skeletal joints in the first set of frames. For example, the smoothing module 516 applies a first set of smoothing filters with a set of smoothing parameters to the first set of skeletal joints (e.g., the neck and left shoulder joints) and a second set of smoothing filters with another set of smoothing parameters to the second set of skeletal joints (e.g., the hip and right knee joints). The smoothing module 516 can also apply multiple smoothing filters with different characteristics to the same set of joints or a single joint in parallel. The smoothing module 516 assigns weights to the multiple smoothing filters based on signal quality parameters that represent noise in the motion of the joints to which the filters are applied. The smoothing module 516 then aggregates the weighted sum of the multiple smoothing filters applied to the same joint or set of joints to adaptively smooth the motion of the joint in the set of frames.

[0117] Although the flow diagrams described can show operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process is terminated when its operations are completed. A process can correspond to a method, a procedure, an algorithm, etc. The operations of a method can be performed entirely or partially in parallel with some or all of the operations of another method, and can be performed by any number of different systems (such as the systems described herein) or any portion thereof (such as a processor included in any system).

[0118] Figure 8 Illustrative inputs and outputs of the body tracking system 126 are shown in accordance with example implementations. As shown, in the video, an avatar is presented with a person. The motion of the person's skeletal joints is smoothed by the body tracking system 126. As a result, the motion of the avatar's rig is also smoothed, resulting in the motion of the avatar presented on the display appearing more realistic and lifelike. That is, the avatar is moved to mimic the motion of the person in the video in a smoothed manner. In some cases, the motion of one portion of the avatar or one portion of the avatar's skeletal rig is smoothed independently from the motion of another portion of the avatar. In some cases, the set of smoothed person's skeletal joints matches the set of smoothed avatar's skeletal joints. In this way, different portions or skeletal joints of the avatar can be smoothed separately using different smoothing filters or smoothing filters with different smoothing parameters.

[0119] Machine Architecture

[0120] Figure 9is a diagrammatic representation of the machine 900 in which instructions 908 (e.g., software, programs, applications, applets, apps, or other executable code) for causing the machine 900 to perform any one or more of the methodologies discussed herein can be executed. For example, the instructions 908 can cause the machine 900 to execute any one or more of the methods described herein. The instructions 908 transform the general, non-programmed machine 900 into a particular machine 900 programmed to carry out the described and illustrated functions in the manner described. The machine 900 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 900 can operate in the capacity of a server machine or a client machine in server-client network environments, or as a peer machine in peer-to-peer (or distributed) network environments. The machine 900 can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 908, sequentially or otherwise, that specify actions to be taken by machine 900. Further, while only a single machine 900 is illustrated, the term “machine” shall also be taken to include a collection of machines 900 that individually or jointly execute the instructions 908 to perform any one or more of the methodologies discussed herein. For example, the machine 900 can include any of the client device 102 or any of the number of server devices that form a part of the messaging server system 108. In some examples, the machine 900 can further include both a client system and a server system, where certain operations of a particular method or algorithm are performed on the server-side and where certain operations of the particular method or algorithm are performed on the client-side.

[0121] The machine 900 can include processors 902, memory 904, and input / output (I / O) components 938, which can be configured to communicate with each other via a bus 940. In an example, the processor(s) 902 (e.g., a central processing unit (CPU), a reduced instruction set computer (RISC) processor, a complex instruction set computer (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 906 and a processor 910 that execute the instructions 908. The term “processor” is intended to include a multi-core processor that can include two or more independent processors (sometimes referred to as “cores”) that can execute instructions contemporaneously. Although FIG. 9 shows the processor(s) 902 that include the processor 906 and the processor 910, the processor(s) 902 can include more than two processors.Figure 9 Multiple processors 902 are shown, but a machine 900 can include a single processor with single cores, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core each, multiple processors with multiple cores each, or any combination thereof.

[0122] The storage 904 includes a main memory 912, a static memory 914, and a storage unit 916 each of which can be accessed via the bus 940 by the processor 902. The main memory 912, static memory 914, and the storage unit 916 store the instructions 908 which implement any one or more of the methodologies described herein. The instructions 908 can also reside, completely or

[0123] The I / O components 938 can include various components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so forth. The specific I / O components 938 included in the machine 900 will depend on the type and Figure 9 Many of the I / O components 938 can be considered communication devices, including for example, the user input components 926 and the user output components 924. The I / O components 938 can include many other components not shown in FIG. 9. In various examples, the I / O components 938 can include user output components 924 and user input components 926. The user output components 924 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The user input components 926 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or another pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0124] In another example, the I / O components 938 can include biometric components 928, motion components 930, environmental components 932, or position components 934, among a myriad of other components. For example, biometric components 928 include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like.

[0125] The environmental components 932 include, for example, one or more cameras (with still and / or video capabilities), illumination sensors (e.g., photometers), temperature sensors (e.g., one or more thermometers that detect ambient temperature), humidity sensors, pressure sensors (e.g., barometers), acoustic sensors (e.g., one or more microphones that detect background noise), proximity sensors (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to a physical environment.

[0126] With respect to cameras, the client device 102 can have a camera system including, for example, a front-facing camera on a front surface of the client device 102 and a rear-facing camera on a rear surface of the client device 102. The front-facing camera can be used, for example, to capture still and video images of a user of the client device 102 (e.g., a “selfie”), which can then be enhanced with the above-described enhancement data (e.g., filters). The rear-facing camera can be used, for example, to capture still and video images in a more conventional camera mode, which are similarly enhanced with the enhancement data. In addition to the front-facing and rear-facing cameras, the client device 102 can include a 360° camera for capturing 360° photos and videos.

[0127] Furthermore, the camera system of the client device 102 can include dual rear-facing cameras (e.g., a primary camera and a depth-sensing camera), or even triple, quad, or penta rear-facing camera configurations on the front and rear sides of the client device 102. These multi-camera systems can include, for example, a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0128] The position component 934 includes location sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect atmospheric pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like.

[0129] Communication can be implemented using a wide variety of technologies. The I / O component 938 also includes a communication component 936 that can be operable to couple the machine 900 to a network 920 or devices 922 via respective coupling or connections. For example, the communication component 936 can include a network interface component or other suitable components. In further examples, the communication component 936 can include a wired communication component, a wireless communication component, a cellular communication component, a Near Field Communication (NFC) component, components (e.g., low energy), components, and other communication components to provide communication via other modalities. The devices 922 can be another machine or any of a wide assortment of peripheral devices (e.g., a peripheral device coupled via a USB).

[0130] Moreover, the communication component 936 can detect identifiers or include components operable to detect identifiers. For example, the communication component 936 can include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes), or acoustic detection components (e.g., microphones to identify a special coded signal produced by a microphone). Moreover, various information can be derived from the communication component 936, such as location via internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting an NFC beacon signal that can indicate a particular location, and so forth.

[0131] The various memories (e.g., the main memory 912, the static memory 914, and the memory of the processor 902) and the storage unit 916 can store one or more sets of instructions and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. These instructions (e.g., instructions 908), when executed by the processor 902, cause various operations to implement the disclosed examples.

[0132] The instructions 908 can be transmitted or received using a transmission medium via the network 920 (for example, a communication network including the Internet) and using any one of multiple, well-known transfer protocols (for example, hypertext transfer protocol (HTTP)). Similarly, the instructions 908 can be transmitted or received using a transmission medium via the coupling (for example, a peer-to-peer coupling) to the devices 922.

[0133] Software Architecture

[0134] Figure 10 is a block diagram 1000 illustrating a software architecture 1004, which can be installed on any one or more of the devices described herein. The software architecture 1004 is supported by hardware of a machine 1002, such as the machine 1002 includes processors 1020, memory 1026, and I / O components 1038. In this example, the software architecture 1004 can be conceptualized as a stack of layers, where each layer provides particular functionality. The software architecture 1004 includes layers such as an operating system 1012, libraries 1010, frameworks 1008, and applications 1006. Operationally, the applications 1006 and / or other components within the software architecture 1004 can invoke API calls 1050 through the software stack and receive messages 1052 in response to the API calls 1050.

[0135] The operating system 1012 manages hardware resources and provides common services. The operating system 1012 includes, for example, a kernel 1014, services 1016, and drivers 1022. The kernel 1014 acts as an abstraction layer between the hardware and the other software layers. For example, the kernel 1014 provides memory management, processor management (for example, scheduling), component management, networking, and security settings, among other functionality. The services 1016 can provide other common services that the or Low-power wireless drivers, flash memory drivers, serial communication drivers (for example, USB drivers), storage device drivers (for example, RAID drivers, etc.), and / or the like. Audio drivers, power management drivers, and / or the like.

[0136] The libraries 1010 provide a common low-level infrastructure used by the applications 1006. The libraries 1010 can include system libraries 1018 (e.g., C standard library) providing functionality such as memory allocation functions, string manipulation functions, mathematic functions, and the like. Further, the libraries 1010 can include API libraries 1024, such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., an OpenGL framework used to render two and three dimensional graphics on a display), database libraries (e.g., SQLite to provide various relational database functions), web libraries (e.g., WebKit to provide web browsing functionality), and the like. The libraries 1010 can also include a wide variety of other libraries 1028 to provide many other APIs to the applications 1006.

[0137] The frameworks 1008 provide a common high-level infrastructure used by the applications 1006. For example, the frameworks 1008 provide various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The frameworks 1008 can provide a broad spectrum of other APIs that can be used by the applications 1006, some of which are specific to a particular operating system or platform.

[0138] In an example, the applications 1006 include a home application 1036, a contacts application 1030, a browser application 1032, a book reader application 1034, a location application 1042, a media application 1044, a messaging application 1046, a game application 1048, and a broad assortment of other applications such as a third party application 1040. The applications 1006 are programs that execute functions defined in the programs. Various programming languages can be employed to create one or more of the applications 1006, structured in a variety of manners, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language). In a specific example, the third party application 1040 (e.g., an application developed by an entity other than the vendor of the particular platform) can be an Android TM or IOS TM application developed using the ANDROID TM or IOS TM software development kit (SDK) Mobile software running on the mobile operating system of the Phone or another mobile operating system. In this example, the third-party application 1040 can activate API calls 1050 provided by the operating system 1012 to facilitate the functionality described herein.

[0139] Glossary

[0140] "Carrier signal" means any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions can be transmitted or received via the network interface device using a transmission medium.

[0141] "Client device" means any machine that interfaces to a communications network to access resources from a server or other client device. A client device can be, without limitation, a mobile telephone, desktop computer, laptop computer, portable digital assistants (PDAs), smart phones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access a network.

[0142] "Communication network" means one or more portions of a network that can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or a portion of a network can include a wireless or cellular network, and a coupling can be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling can implement any of a variety of types of data transfer technology, such as Single Carrier Radio Transmission Technology (lxRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standard, others defined by various standards setting organizations, other long range protocols, or other data transfer technology.

[0143] A“component” refers to a device, physical entity or logic having boundaries defined by function or subroutine calls, branches points, APIs, or other technologies that provide for the partitioning or modularization of a particular processing or control function. Components can be combined via their interfaces to execute machine processes, such as those processes described herein. A component can be a packaged functional hardware unit designed for use with other components and can be a part of a program that usually performs a particular function of related functions. Components can constitute either software components (e.g., code embodied on a machine-readable medium) or hardware components.

[0144] A“hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein. A hardware component can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component can be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the configured functions and are no longer general-purpose processors.

[0145] It will be appreciated that the decision to implement a hardware component mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) can be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner and / or to perform certain operations described herein. Considering implementations in which hardware components are temporarily configured (e.g., programmed), each of the hardware components need not be configured or instantiated at any one instance in time. For example, where a hardware component comprises a general-purpose processor configured by software to become a special-purpose processor, the general-purpose processor can be configured as different special-purpose processors (e.g., comprising different hardware components) at different times. Software could thereby configure a general-purpose processor to become a particular hardware component at one instance of time and a different hardware component at a different instance of time. A hardware component can provide information to, and receive information from, other hardware components. Accordingly, the described hardware components can be regarded as being communicatively coupled. Where multiple hardware components exist contemporaneously, communications might be achieved through signal transmission (e.g., over appropriate circuits and buses) between or among two or more of the hardware components. In implementations in which multiple hardware components are configured or instantiated at different times, communications between such hardware components can be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware components have access. For example, one hardware component performs an operation and stores the output of that operation in a memory device to which it is communicatively coupled. A further hardware component can then access the memory device to retrieve and process the data stored therein. Hardware components can also initiate communications with input or output devices, and can operate on resources (e.g., information) stored within memory structures.

[0146] The various operations of example methods described herein can be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that conduct operations to execute one or more operations or functions described herein. As used herein, "processor- implemented component" refers to a hardware component implemented with one or more processors. Similarly, the methods described herein can be at least partially processor-implemented, with a particular one or more processors being an example of hardware. For example, at least some of the operations of a method can be performed by one or more processors 902 or processor-implemented components. Moreover, a one or more processors can operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an API). The performance of certain of the operations can be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processors or processor-implemented components can be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processors or processor-implemented components can be distributed across a number of geographic locations.

[0147] "Computer-readable storage media" refers to both machine storage media and transmission media. Thus, the term includes both storage devices / media and carrier waves / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0148] "Transient message" refers to a message that is accessible for a time-limited duration. The transient message can be text, image, video, etc. The access time for the transient message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the receiver. Regardless of the setting technique, the message is temporary.

[0149] “machine-storage medium” refers to a single or multiple storage devices and media (e.g., a centralized or distributed database, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be understood to include, but not be limited to, solid-state memories, and optical and magnetic media, including memory internal or external to processors. Specific examples of machine-storage media, computer-storage media, and device-storage media include non-volatile memory, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine-storage medium,” “device-storage medium,” “computer-storage medium,” and “device-storage medium” mean the same thing and can be used interchangeably in this disclosure. The terms “machine-storage medium,” “computer-storage medium,” and “device-storage medium” specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term “signal medium.”

[0150] “non-transitory computer-readable storage medium” refers to a tangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine.

[0151] “signal medium” refers to any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and that causes a machine to perform any of the disclosed methods. The term “signal medium” shall be taken to include any form of a modulated data signal, carrier wave, and so on. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms “transmission medium” and “signal medium” mean the same thing and can be used interchangeably in this disclosure.

[0152] Changes and modifications can be made to the disclosed embodiments without departing from the scope of the present disclosure. These and other changes or modifications are intended to be included within the scope of the present disclosure, as expressed in the appended claims.

Claims

1. A method comprising: receiving, by one or more processors, a video, the video comprising a depiction of a body of a user; detecting, by the one or more processors, a plurality of skeletal joints of the body depicted in the video; tracking motion of the plurality of skeletal joints in a set of frames of the video; and smoothing motion of a first set of the plurality of skeletal joints between frames of the first set of frames independently of smoothing motion of a second set of the plurality of skeletal joints in the first set of frames to enable different amounts of smoothing to be applied to the first set of the plurality of skeletal joints and the second set of the plurality of skeletal joints.

2. The method of claim 1, further comprising: controlling an amount of smoothing applied to the first set of the plurality of skeletal joints based on a signal stability parameter.

3. The method of any one of claims 1-2, wherein, the first set of the plurality of skeletal joints comprises a first pair of adjacent joints, and wherein the second set of the plurality of skeletal joints comprises a second pair of adjacent joints.

4. The method of any one of claims 1-3, further comprising: applying a first set of smoothing filters to the first set of the plurality of skeletal joints; applying a second set of smoothing filters to the second set of the plurality of skeletal joints; and adjusting a first parameter of the first set of smoothing filters independently of a second parameter of the second set of smoothing filters.

5. The method of claim 4, wherein, the first set of smoothing filters and the second set of smoothing filters comprise a plurality of smoothing filters.

6. The method of any one of claims 4-5, further comprising: computing a first signal stability parameter representing a first amount of noise resulting from motion of the first set of the plurality of skeletal joints between frames of the first set of frames; and modifying the first parameter based on a value of the first signal stability parameter.

7. The method of any of claims 4-6, further comprising: in response to determining that the first amount of noise represented by the first signal stability parameter exceeds a threshold, modifying the first parameter to increase an amount of smoothing applied to the second set of frames by the first set of smoothing filters.

8. The method of any one of claims 4-7, further comprising: computing a second signal stability parameter representing a second amount of noise resulting from motion of the second set of the plurality of skeletal joints between frames of the first set of frames; and modifying the second parameter based on a value of the second signal stability parameter.

9. The method of any one of claims 4-8, further comprising: in response to determining that the second amount of noise represented by the first signal stability parameter is less than the threshold, modifying the second parameter to decrease an amount of smoothing applied to the second set of frames by the second set of smoothing filters.

10. The method of claim 4, wherein, applying the first set of smoothing filters to the first set of the plurality of skeletal joints comprises applying the plurality of smoothing filters in parallel to the first set of the plurality of skeletal joints.

11. The method of any one of claims 4-10, further comprising: interpolating outputs of the plurality of smoothing filters.

12. The method of claim 11, wherein, performing the interpolating of the outputs based on a signal stability parameter representing an amount of noise between the outputs of the plurality of smoothing filters.

13. The method of any one of claims 1-12, further comprising: generating an avatar based on the smoothed motion of the plurality of skeletal joints.

14. A system comprising: a processor configured to perform operations comprising: receiving a video comprising a depiction of a body of a user; detecting a plurality of skeletal joints of the body depicted in the video; tracking motion of the plurality of skeletal joints in a set of frames of the video; and smoothing motion of a first set of the plurality of skeletal joints between frames in a first set of frames independently of smoothing motion of a second set of the plurality of skeletal joints in the first set of frames to enable different amounts of smoothing to be applied to the first set of the plurality of skeletal joints and the second set of the plurality of skeletal joints.

15. The system of claim 14, wherein, the operations further comprising controlling an amount of smoothing applied to the first set of the plurality of skeletal joints based on a signal stability parameter.

16. The system of any one of claims 14-15, wherein, the first set of the plurality of skeletal joints comprises a first pair of adjacent joints, and wherein the second set of the plurality of skeletal joints comprises a second pair of adjacent joints.

17. The system of any one of claims 14-16, wherein, the operations further comprising: applying a first set of smoothing filters to the first set of the plurality of skeletal joints; applying a second set of smoothing filters to the second set of the plurality of skeletal joints; and adjusting a first parameter of the first set of smoothing filters independently of a second parameter of the second set of smoothing filters.

18. The system of claim 17, wherein, the first set of smoothing filters and the second set of smoothing filters comprise a plurality of smoothing filters.

19. The system of any one of claims 17-18, wherein, the operations further comprising: computing a first signal stability parameter representing a first amount of noise resulting from motion of the first set of the plurality of skeletal joints between frames in the first set of frames; and modifying the first parameter based on a value of the first signal stability parameter.

20. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: receiving a video comprising a depiction of a body of a user; detecting a plurality of skeletal joints of the body depicted in the video; tracking motion of the plurality of skeletal joints in a set of frames of the video; and smoothing motion of a first set of the plurality of skeletal joints between frames in a first set of frames independently of smoothing motion of a second set of the plurality of skeletal joints in the first set of frames to enable different amounts of smoothing to be applied to the first set of the plurality of skeletal joints and the second set of the plurality of skeletal joints. ​

Images