Skeletal tracking using previous frames

Abstract

Aspects of the disclosure relate to a system comprising a computer-readable storage medium storing a program and a method for detecting a user gesture. The program and method comprise the operations of receiving a monocular image comprising a depiction of a body of a user; detecting a plurality of skeletal joints of the body based on the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; filtering the plurality of skeletal joints of the body detected based on the monocular image using the video feed; and determining a gesture represented by the body depicted in the monocular image based on the filtered plurality of skeletal joints of the body.

Description

[0001] Priority Statement

[0002] This application claims priority to U.S. Patent Application No. 16 / 710,980, filed December 11, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to generating virtual objects based on images depicting users. Background Technology

[0004] Virtual rendering systems can be used to create engaging and fun augmented reality experiences where virtual object graphics (such as avatars) appear to exist in the real world. However, such systems can encounter rendering problems due to environmental conditions, user actions, unexpected visual interruptions between the camera and the object being rendered, etc. This can cause virtual objects to disappear or otherwise appear irregular, thus breaking the illusion that the virtual objects exist in the real world. Attached Figure Description

[0005] In drawings that are not necessarily drawn to scale, the same reference numerals may describe the same parts in different views. For ease of identification of any particular element or action being discussed, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced. Some embodiments are shown in the accompanying drawings by way of example rather than limitation, in which:

[0006] Figure 1 This is a block diagram illustrating an example messaging system for exchanging data (e.g., messages and associated content) over a network, according to an example implementation.

[0007] Figure 2 This is a schematic diagram illustrating data that can be stored in a database of a message transceiver server system according to an example implementation.

[0008] Figure 3 This is a schematic diagram illustrating the structure of a message for communication generated by a messaging client application according to an example implementation.

[0009] Figure 4 This is a block diagram illustrating an example body pose estimation system according to an example implementation.

[0010] Figure 5 This is a diagram illustrating the positions of body joints used to determine body posture according to an example embodiment.

[0011] Figure 6 This is a flowchart illustrating an example operation of a body posture estimation system according to an example implementation.

[0012] Figures 7A to 8C These are illustrative inputs and outputs of a body posture estimation system based on an example implementation.

[0013] Figure 9 This is a block diagram illustrating a representative software architecture that can be used in conjunction with various hardware architectures described herein, according to an example implementation.

[0014] Figure 10 This is a block diagram illustrating components of a machine according to an example embodiment, capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and executing any or more of the methods discussed herein. Detailed Implementation

[0015] The following description includes systems, methods, techniques, instruction sequences, and computer program products embodying illustrative embodiments of this disclosure. In the following description, numerous specific details are set forth for illustrative purposes to provide an understanding of various embodiments. However, it will be apparent to those skilled in the art that embodiments can be practiced without these specific details. Generally, well-known examples of instructions, protocols, structures, and techniques are not necessarily shown in detail.

[0016] Typically, virtual reality (VR) and augmented reality (AR) systems display an avatar representing a given user by capturing an image of the user and additionally using a depth sensor to obtain a depth map depicting a real-world human body in the image. By processing the depth map and the image together, VR and AR systems can mimic actions performed by the user. While such systems can represent user avatars well, the requirement for depth sensors limits their applications. This is because adding depth sensors to the user device to display the avatar increases the overall cost and complexity of the device, thus reducing their appeal.

[0017] Furthermore, while some systems use real-time skeletal trackers to identify user actions, such trackers are noisy and require temporal filtering. This reduces their overall efficiency and consumes too many resources to apply them to mobile devices. Filtering based on classical computer vision and signal analysis can slightly improve the results, but it is not sufficient.

[0018] The disclosed implementation improves the efficiency of using electronic devices by applying a first machine learning technique to identify the skeletal joints of a user's body from a single image of the user, and then applying a second machine learning technique to the image features of previously received user video frames and the current frame to predict the skeletal joint positions in the current frame (e.g., a single image) to filter or improve the identified skeletal joints. Specifically, a video depicting a user's body is received. The current frame of the video is processed using the first machine learning technique to identify skeletal joint positions. A set of previous frames (e.g., video 1 to 2 seconds before the current frame) is processed using the second machine learning technique to predict the skeletal joint positions in subsequent frames (e.g., the current frame). The skeletal joint positions identified for the current frame using the first machine learning technique are compared with the skeletal joint positions predicted based on the previous frames using the second machine learning technique. Any deviations or differences between the skeletal joint positions are then corrected using the second machine learning technique to improve and filter the skeletal joint positions identified for the current frame using the first machine learning technique.

[0019] Skeletal joints can then be used to modify the rendering of one or more virtual objects (e.g., simulating poses corresponding to skeletal joints). Virtual objects (e.g., 3D objects such as 3D captions, emojis, characters, avatars, animations, looping animations of personalized avatars or characters, looping or non-looping animated graphics such as dancing hot dogs, animated artistic fonts, etc.) are generated directly from a single red, green, and blue (RGB) image (e.g., a monocular RGB image) or a video depicting an RGB image of a real-world user's body. The disclosed implementation generates virtual objects without acquiring a depth map of the real-world user's body. This enables user devices with simple RGB cameras (without depth sensors) to accurately and quickly render animated virtual objects based on the real-world user's body pose in VR or AR applications, allowing users to interact with VR or AR content in a more realistic environment.

[0020] In some implementations, a virtual object is selected by the user and added to an RGB image, and the pose of the virtual object is modified to match the pose of the user's body, thus providing the illusion that the selected virtual object is part of a real-world scene and is mimicking the user. Specifically, the disclosed implementation uses machine learning techniques to process an image depicting the user's body to identify skeletal joints. The pose represented by the identified skeletal joints is determined. After the pose represented by the skeletal joints is determined, the corresponding skeletal joints of the avatar's skeleton are adjusted to change the avatar's pose to match the pose represented by the identified skeletal joints of the user's body. An avatar with a modified pose matching the pose represented by the identified skeletal joints of the user's body is generated and displayed to the user.

[0021] Figure 1This is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) via network 106. The messaging system 100 includes a plurality of client devices 102, each of which hosts multiple applications including a messaging client application 104 and an AR / VR application 105. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104, the AR / VR application 105, and the messaging server system 108 via network 106 (e.g., the Internet).

[0022] Therefore, each messaging client application 104 and AR / VR application 105 can communicate and exchange data with another messaging client application 104 and AR / VR application 105, as well as with the messaging server system 108, via network 106. The data exchanged between messaging client application 104 and AR / VR application 105, and between messaging client application 104 and messaging server system 108, includes functions (e.g., commands to invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0023] AR / VR application 105 is an application that includes a set of functions that allow client device 102 to access body pose estimation system 124. In some implementations, AR / VR application 105 is a component or feature as part of messaging client application 104. AR / VR application 105 uses an RGB camera to capture monocular images of the user's real-world body. AR / VR application 105 applies various trained machine learning techniques to the captured images of the body, as well as one or more previous frames depicting the body, to generate an avatar. For example, the avatar is generated to have the same pose as the user's body depicted in the images. As another example, the avatar is generated to be displayed simultaneously with the user, and the avatar's position changes between frames based on the user's relative position in the images. For example, the avatar's position changes to maintain a constant distance between the avatar and the user, so the avatar appears to follow the user. In some implementations, AR / VR application 105 continuously captures images of the user's body in real-time or periodically to continuously or periodically update the generated avatar. This allows the user to move around in the real world and view the avatar updates in real time. AR / VR application 105 presents various content (e.g., messages, games, advertisements, etc.) and allows users to modify the pose of their avatar in AR / VR application 105 to interact with other virtual content (e.g., the avatar's pose can be modified to pick a virtual apple from a virtual tree). After the avatar is generated, the user can perform actions or gestures to capture images of the user and the avatar to send to another user.

[0024] To enable AR / VR application 105 to generate avatars directly from captured RGB images, AR / VR application 105 acquires a first trained machine learning technique from body pose estimation system 124 and / or messaging server system 108. The first trained machine learning technique processes the captured RGB images to extract one or more features corresponding to the body depicted in the captured RGB images. Features are analyzed to identify one or more skeletal joints and their corresponding alignments relative to each other. Specifically, features are analyzed to determine the joint positions of a specified set of skeletal joints. AR / VR application 105 also acquires a second trained machine learning technique from body pose estimation system 124 and / or messaging server system 108. The second trained machine learning technique processes one or more previously captured frames (e.g., video frames 1 to 2 seconds prior to the RGB image) to estimate or predict the skeletal joint positions of subsequent frames. A threshold number of seconds (which can be user-defined, previously specified, and / or dynamically determined) of video frames can be stored continuously or periodically in a buffer, allowing access to video frames preceding the current RGB image via the second trained machine learning technique. The output or prediction of skeletal joint positions from a second-trained machine learning technique is used to filter or improve the skeletal joint positions identified by the first-trained machine learning technique. In some cases, the second-trained machine learning technique processes the skeletal joint positions identified by the first-trained machine learning technique together with previously captured frames to filter or improve the estimated skeletal joint positions.

[0025] Joint positions are provided to a database with offsets to identify poses within the offsets of the determined joint positions. After the pose is identified, an avatar is obtained and the avatar's skeleton is adjusted to mimic or replicate the identified pose. In some implementations, the avatar's skeleton is adjusted directly based on the determined joint positions rather than on the identified pose. The avatar is then generated for display based on the skeleton adjusted to mimic the body pose depicted in the captured image.

[0026] In some implementations, the RGB image is the first frame of a video with multiple frames. In this case, after detecting the user's body in the first frame using a first machine learning technique and a second machine learning technique, the body pose estimation system 124 estimates the position and proportion of the user's body in a second frame with multiple frames. The second frame may be adjacent to the first frame. In some implementations, a third machine learning technique is applied to the first frame to predict or estimate the position and proportion of the user's body in the second frame.

[0027] During training, the body pose estimation system 124 acquires a first plurality of input training images including different real-world body poses. These training images also provide real-world information about the body pose depicted in each image. A first machine learning technique (e.g., a deep neural network) is trained based on features from the plurality of training images. Specifically, the first machine learning technique extracts one or more features from a given training image and estimates the body pose by analyzing the joint positions of the body depicted in the given training image. The first machine learning technique acquires real-world information corresponding to the training images and adjusts or updates one or more coefficients to improve subsequent estimations of the body pose depicted in subsequent images.

[0028] During training, the body pose estimation system 124 acquires a first plurality of input training videos (each with a number of frames corresponding to a threshold video duration, for example, 1 to 2 seconds) including different real-world body poses. These training videos also provide real-world information with body skeletal joint positions relative to subsequent frames of each video. That is, the first training video can be associated with real-world information identifying the skeletal joint positions of the body depicted in frames immediately following the last frame of the first training video. A second machine learning technique (e.g., a neural network) is trained based on the features of the plurality of training videos. Specifically, the second machine learning technique extracts one or more features from a given training video and estimates or predicts skeletal joint positions in subsequent frames associated with the last frame of the training video by analyzing the joint positions of the body depicted in the given training video. The second machine learning technique acquires real-world information corresponding to the training videos and adjusts or updates one or more coefficients to improve subsequent estimates of body poses depicted in subsequent videos.

[0029] In some implementations, a third machine learning technique (e.g., a deep neural network) extracts one or more features from a given training image and estimates body poses in subsequent images adjacent to the given training image by analyzing the joint positions of the body depicted in the given training image. The third machine learning technique acquires ground truth information corresponding to the training image, which identifies body poses in subsequent images, and adjusts one or more coefficients to improve subsequent estimates of body poses depicted in subsequent images, as well as estimates of body poses and proportions in subsequent images. In some implementations, during training, the third machine learning technique acquires a reference 3D depth map for each training image and uses the reference 3D depth map to estimate body poses.

[0030] The messaging server system 108 provides server-side functionality to a specific messaging client application 104 via network 106. While some functions of the messaging system 100 are described herein as being performed by either the messaging client application 104 or the messaging server system 108, it should be understood that the location of certain functions within either the messaging client application 104 or the messaging server system 108 is a design choice. For example, it is technically preferred that certain technologies and functions be initially deployed within the messaging server system 108, but later migrated to the messaging client application 104 on the client device 102 with sufficient processing power.

[0031] The messaging server system 108 supports various services and operations provided to the messaging client application 104. Such operations include sending data to and receiving data from the messaging client application 104, and processing data generated by the messaging client application 104. This data may include, for example, message content, client device information, geolocation information, media annotations and overlays, virtual objects, message content persistence conditions, social network information, and life event information. Data exchange within the messaging system 100 is activated and controlled through functions available via the user interface (UI) of the messaging client application 104.

[0032] Now, specifically to message transceiver server system 108, application interface (API) server 110 is coupled to application server 112 and provides a programming interface to application server 112. Application server 112 is communicatively coupled to database server 118, which provides easy access to database 120, in which data associated with messages processed by application server 112 is stored.

[0033] Specifically, API server 110 handles the receiving and sending of message data (e.g., commands and message payloads) between client device 102 and application server 112. Specifically, API server 110 provides a set of interfaces (e.g., routines and protocols) that message-receiving client application 104 can call or query to activate the functionality of application server 112. API server 110 displays various functions supported by application server 112, including: account registration; login functionality; sending messages from one messaging client application 104 to another messaging client application 104 via application server 112; sending media files (e.g., images or videos) from messaging client application 104 to messaging server application 114 and making them accessible to another messaging client application 104; setting up media data collections (e.g., stories); retrieving such collections; retrieving the friend list of the user of client device 102; retrieving messages and content; adding and deleting friends in the social graph; the location of friends in the social graph; accessing user conversation data; accessing avatar information stored on messaging server system 108; and opening application events (e.g., related to messaging client application 104).

[0034] Application server 112 hosts several applications and subsystems, including message transceiver server application 114, image processing system 116, social networking system 122, and body pose estimation system 124. Message transceiver server application 114 implements several message processing techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of message transceiver client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into content collections (e.g., referred to as stories or galleries). These collections are then made available to message transceiver client application 104 by message transceiver server application 114. Given the hardware requirements for such processing, additional processor- and memory-intensive processing of the data can also be performed on the server side by message transceiver server application 114.

[0035] Application server 112 also includes an image processing system 116, which is typically dedicated to performing various image processing operations on images or videos received within the payload of messages at message transceiver server application 114. A portion of the image processing system 116 may also be implemented by body pose estimation system 124.

[0036] Social networking system 122 supports various social networking features and services, and makes these features and services available to messaging server application 114. To this end, social networking system 122 maintains and accesses an entity graph within database 120. Examples of features and services supported by social networking system 122 include identifying other users of messaging system 100 who have relationships with a particular user or who the particular user is "following," as well as identifying other entities and interests of a particular user. Such other users may also be referred to as the user's friends.

[0037] Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120, in which data associated with messages processed by message sending and receiving server application 114 is stored.

[0038] Figure 2 This is a schematic diagram 200 illustrating data that can be stored in a database 120 of a message transceiver server system 108 according to some example implementations. Although the contents of the database 120 are shown to include several tables, it should be understood that the data can be stored in other types of data structures (e.g., as an object-oriented database).

[0039] Database 120 includes message data stored in message table 214. Entity table 202 stores entity data, including entity diagram 204. Entities maintaining records in entity table 202 can include individuals, company entities, organizations, objects, locations, events, etc. Regardless of type, any entity whose data is stored in message transceiver server system 108 can be an identifiable entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0040] Entity Graph 204 also stores information about the relationships and associations between entities. For example, such relationships can be social, professional (e.g., working in the same company or organization), interest-based, or activity-based.

[0041] Message table 214 may store a collection of conversations between a user and one or more friends or entities. Message table 214 may include various attributes for each conversation, such as a list of participants, the size of the conversation (e.g., the number of users and / or the number of messages), the chat color of the conversation, a unique identifier for the conversation, and any other conversation-related characteristics.

[0042] Database 120 also stores annotation data in annotation table 212 as an example of filters. Database 120 also stores the annotation content received in annotation table 212. The filters storing data in annotation table 212 are associated with and applied to videos (whose data is stored in video table 210) and / or images (whose data is stored in image table 208). In one example, a filter is an overlay displayed as an overlay on an image or video during presentation to the receiving user. Filters can be of various types, including filters selected by the user from a filter gallery presented to the sending user by the messaging client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the sending user based on geographic location. For example, a geolocation filter specific to a nearby or particular location can be presented by the messaging client application 104 within the UI based on geographic location information determined by the Global Positioning System (GPS) unit of the client device 102. Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client application 104 based on other inputs or information collected by the client device 102 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed of the sending user, the battery life of the client device 102, or the current time.

[0043] Other annotation data that can be stored in image table 208 can be real-time specific effects and sounds that can be added to images or videos.

[0044] As described above, video table 210 stores video data, which in one embodiment is associated with messages for which records are maintained in message table 214. Similarly, image table 208 stores image data associated with messages whose message data is stored in entity table 202. Entity table 202 can associate various annotations from annotation table 212 with various images and videos stored in image table 208 and video table 210.

[0045] The trained machine learning technique 207 stores parameters that have been trained during the training of the body pose estimation system 124. For example, the trained machine learning technique 207 stores trained parameters of one or more neural network machine learning techniques.

[0046] Body posture training image 209 stores multiple images depicting real-world body postures. The multiple images stored in body posture training image 209 include various depictions of real-world body postures and 3D depth information captured from a 3D depth sensor for each image. The multiple images also include realistic information identifying the body posture and actual joint positions of one or more skeletal joints. Figure 5 An example of skeletal joint positions identified for a given pose is shown. These skeletal joint positions include the positions of the right wrist, right elbow, right shoulder, nose on the face, left shoulder, left elbow, and left wrist. Skeletal joint positions can be determined relative to each other (e.g., how high or low one skeletal joint is relative to another, or how high or low one skeletal joint is relative to the nose) to determine a given pose. These body pose training images 209 are used by a body pose estimation system 124 to train a first machine learning technique for determining body pose and skeletal joint positions from received RGB monocular images of the user's body.

[0047] Body pose training images 209 store multiple videos (1- to 2-second video clips) depicting real-world body poses. The multiple videos stored in body pose training images 209 include various depictions of real-world body poses. The multiple videos also include real-world information identifying the actual skeletal joint positions of the body depicted in subsequent frames relative to the last frame in each of the multiple videos. These body pose training images 209 are used by body pose estimation system 124 to train a second machine learning technique for predicting skeletal joint positions in subsequent frames based on received RGB monocular video of the user's body.

[0048] Return to Figure 2 Story table 206 stores data related to messages and collections of associated image, video, or audio data, compiled into collections (e.g., stories or galleries). The creation of a specific collection can be initiated by a specific user (e.g., each user for whom records are maintained in entity table 202). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. For this purpose, the UI of the messaging client application 104 can include user-selectable icons that allow the sending user to add specific content to his or her personal story.

[0049] Collections can also constitute "life stories," which are collections of content from multiple users, created manually, automatically, or using a combination of manual and automatic technologies. For example, a "life story" can form a curated flow of user-submitted content from various locations and events. Users whose client devices have location services enabled and are at a common location event at a specific time can be presented with options, for example, via the UI of messaging client application 104, to contribute content to a specific life story. The messaging client application 104 can identify life stories to a user based on their location. The end result is a "life story" told from a community perspective.

[0050] Another type of content collection is called a "location story," which allows users whose client devices 102 are located in a specific geographic location (e.g., on a college or university campus) to contribute to a specific collection. In some implementations, contributing to a location story may require secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., a student on a university campus).

[0051] Figure 3 This is a schematic diagram illustrating the structure of a message 300 according to some embodiments. The message 300 is generated by a message transceiver client application 104 for transmission to another message transceiver client application 104 or a message transceiver server application 114. The content of a particular message 300 is used to populate a message table 214 stored in a database 120, which is accessible by the message server application 114. Similarly, the content of the message 300 is stored in memory as “in transit” or “in flight” data for the client device 102 or application server 112. The message 300 is shown to include the following components:

[0052] • Message Identifier 302: A unique identifier that identifies message 300.

[0053] • Message text payload 304: The text to be generated by the user via the UI of the client device 102 and included in message 300.

[0054] • Message image payload 306: Image data captured by the camera device component of the client device 102 or retrieved from the memory of the client device 102 and included in the message 300.

[0055] • Message video payload 308: Video data captured by the camera device component or retrieved from the memory component of the client device 102 and included in the message 300.

[0056] • Message audio payload 310: Audio data captured by the microphone or retrieved from the memory component of the client device 102 and included in the message 300.

[0057] • Message annotation 312: Annotation data (e.g., filters, stickers, or other enhancements) representing annotations to be applied to message image payload 306, message video payload 308, or message audio payload 310 of message 300.

[0058] • Message duration parameter 314: A parameter value that indicates the amount of time, in seconds, during which the content of the message (e.g., message image payload 306, message video payload 308, message audio payload 310) will be presented to the user or made accessible to the user via the message sending and receiving client application 104.

[0059] • Message geolocation parameter 316: Geographic location data (e.g., latitude and longitude coordinates) associated with the content payload of the message. The payload may include multiple message geolocation parameter 316 values, each of which is associated with a content item included in the content (e.g., a specific image within the message image payload 306, or a specific video within the message video payload 308).

[0060] • Message Story Identifier 318: An identifier value that identifies one or more content sets (e.g., "story") to which a specific content item in the message image payload 306 of message 300 is associated. For example, multiple images within the message image payload 306 may each be associated with multiple content sets using identifier values.

[0061] • Message Tag 320: Each message 300 can be labeled with multiple tags, each tag indicating the subject of the content included in the message payload. For example, if a specific image in the message image payload 306 depicts an animal (e.g., a lion), a tag value indicating the relevant animal can be included in the message tag 320. The tag values ​​can be generated manually based on user input, or automatically using, for example, image recognition.

[0062] • Message sender identifier 322: An identifier (e.g., message sending system identifier, email address, or device identifier) ​​indicating the user of the client device 102 on which message 300 is generated and from which message 300 is sent.

[0063] • Message receiver identifier 324: An identifier (e.g., a messaging system identifier, email address, or device identifier) ​​for the user of the client device 102 to which message 300 is addressed. In the case of a conversation between multiple users, the identifier can identify each user involved in the conversation.

[0064] The content (e.g., values) of various components of message 300 can be pointers to locations of stored content data values ​​in tables. For example, image values ​​in message image payload 306 can be pointers to locations (or addresses) within image table 208. Similarly, values ​​in message video payload 308 can point to data stored in video table 210, values ​​stored in message annotation 312 can point to data stored in annotation table 212, values ​​stored in message story identifier 318 can point to data stored in story table 206, and values ​​stored in message sender identifier 322 and message receiver identifier 324 can point to user records stored in entity table 202.

[0065] Figure 4 This is a block diagram illustrating an example body pose estimation system 124 according to an exemplary implementation. The body pose estimation system 124 operates on a set of input data (e.g., a monocular image 401 depicting a user's real body, body pose training image data 402, a monocular video 403 depicting the user's body, and body pose training video data 404). This set of input data is obtained from body pose training images 209 stored in a database 200 during the training phase and from the RGB camera of the client device 102 when the AR / VR application 105 is being used. The body pose estimation system 124 includes a first machine learning technology module 412, a skeletal joint position module 414, a second machine learning technology module 417, a pose determination module 416, a virtual object modification module 418, a virtual object mode selection module 419, and a virtual object display module 420.

[0066] During training, according to the disclosed implementation, the body pose estimation system 124 receives given training images (e.g., monocular images 401 depicting a real-world body, such as images of a user's face, arms, torso, hips, and legs) from body pose training image data 402. The body pose estimation system 124 applies one or more machine learning techniques to the given training images using a first machine learning technique module 412. The first machine learning technique module 412 extracts one or more features from the given training images to estimate the skeletal joint positions depicted in the images. For example, the first machine learning technique module 412 acquires given training images depicting a user's face, arms, torso, hips, and legs. The first machine learning technique module 412 extracts features corresponding to the user's face and arms from the images to identify one or more skeletal joints (e.g., ...). Figure 5 The joints shown include the left / right wrist joint, left / right elbow joint, left / right shoulder joint, and nose position.

[0067] The first machine learning module 412 determines the relative positions of the identified skeletal joints to determine the pose corresponding to the skeletal joint positions. In one embodiment, the first machine learning module 412 uses depth information associated with a given training image to help determine the skeletal joint positions and poses of the body. The first machine learning module 412 compares the determined skeletal joint positions and poses with the real skeletal joint positions and poses provided as part of the body pose training image data 402. Based on a difference threshold for the comparison, the first machine learning module 412 updates one or more coefficients and acquires one or more additional body pose training images. After a specified number of epochs or after the training images have been processed and / or when the difference threshold reaches a specified value, the first machine learning module 412 completes training and the parameters and coefficients of the first machine learning module 412 are stored in the trained machine learning module 207. In some embodiments, the first machine learning module 412 is partially implemented by a skeletal joint position module 414 and a pose determination module 416.

[0068] During training, according to the disclosed implementation, the body pose estimation system 124 receives a given training video (e.g., a monocular video 403 depicting a user's body, which depicts a real-world body, such as the user's face, arms, torso, hips, and legs) from body pose training image data 402. The body pose estimation system 124 applies one or more machine learning techniques to the given training video using a second machine learning technique module 417. The second machine learning technique module 417 extracts one or more features from the given training video to predict the skeletal joint positions in frames following the last frame of the video. For example, the second machine learning technique module 417 acquires a given training video depicting movement of a set of frames spanning 1 to 2 seconds, showing the user's face, arms, torso, hips, and legs. The second machine learning technique module 417 extracts features corresponding to the user's face and arms from the video to predict one or more skeletal joints in frames following the last frame of the video (e.g., ...). Figure 5 The joints shown include the left / right wrist joints, left / right elbow joints, left / right shoulder joints, and nose position. In some cases, frames after the last frame of the video may be unavailable and not received by the second machine learning module 417, but the second machine learning module 417 predicts the skeletal joint positions of subsequent frames based on previous frames.

[0069] The second machine learning module 417 predicts the skeletal joint positions of one or more subsequent frames following a given training video. For example, the second machine learning module 417 may process frames 2 through 25 of a given video and frame 26 of the same video to predict the skeletal joint positions of the body depicted in frame 26 of the same video. The second machine learning module 417 compares the determined / predicted skeletal joint positions with the true skeletal joint positions provided as part of the body pose training image data 402. The true skeletal joint positions can provide the true skeletal joint positions of the subsequent frame 26 regarding the movement of the body depicted in frames 2 through 25. Based on a difference threshold for the comparison, the second machine learning module 417 updates one or more coefficients and acquires one or more additional body pose training videos. After processing a specified number of periods or batches of training videos and / or when the difference threshold reaches a specified value, the second machine learning module 417 completes training and the parameters and coefficients of the second machine learning module 417 are stored in the trained machine learning module 207. In some embodiments, a portion of the second machine learning module 417 is implemented by the skeletal joint position module 414 and the pose determination module 416.

[0070] In the example, the second machine learning module 417 is trained to recognize the movement of skeletal joint positions across a set of consecutive frames. The second machine learning module 417 may receive the current video frame and / or the skeletal joint positions determined based on that current video frame. The second machine learning module 417 may process image features from a set of previous frames depicting body movement and the current frame together, and predict the estimated skeletal joint positions for the next frame. Based on this prediction, the second machine learning module 417 compares the predicted estimated skeletal joint positions with the skeletal joint positions determined by the first machine learning module 412 for the current frame. The second machine learning module 417 may then determine any necessary biases and corrections based on the comparison.

[0071] Specifically, the second machine learning module 417 (e.g., from the first machine learning module 412) receives a set of skeletal joint positions already determined for the current frame. The set of skeletal joint positions may indicate a specific coordinate (4, 6) in a two-dimensional (2D) or three-dimensional (3D) space for the left wrist, and another specific coordinate (10, 8) in a 2D or 3D space for the right wrist. The second machine learning module 417 processes a sequence of video frames immediately preceding the current frame and image features of the current frame. The second machine learning module 417 analyzes the movement of the skeletal joints across the video frame sequence to predict the estimated position of the skeletal joints in the current frame or frames following the current frame. As an example, the second machine learning module 417 predicts the coordinates of the left wrist as (4, 7) and the coordinates of the right wrist as (10, 8). The second machine learning module 417 compares the predicted coordinates with coordinates determined for the current frame (e.g., by the first machine learning module 412). The second machine learning module 417 may filter or correct at least some mismatched coordinates. In this example, the second machine learning technology module 417 determines that the left wrist coordinates (4, 6) do not match the coordinates (4, 7) predicted based on previous video frames, and therefore corrects the skeletal joint position to (4, 7) in the set of skeletal joint positions.

[0072] Following training, according to the disclosed implementation, the body pose estimation system 124 receives an input image 401 (e.g., a monocular image depicting a real-world body, such as an image of a user's face, arms, torso, hips, and legs) as a single RGB image from the client device 102. The body pose estimation system 124 applies a first trained machine learning module 412 to the received input image 401 to extract one or more features representing the skeletal joints of the body depicted in image 401. The body pose estimation system 124 applies a second trained machine learning module 417 to the received monocular video 403 depicting the user's body to extract one or more features representing the skeletal joints of the body depicted in the monocular video 403 and to generate predictions or estimates of the skeletal joints in subsequent frames.

[0073] In some implementations, the rate at which features are extracted from a video stream depicting a user's body is based on how close or far the user is relative to the image capture device. For example, if the user is positioned far from the image capture device and appears very small in the video image, then the user's body features are analyzed less frequently, thus the user's body pose is analyzed less frequently (e.g., once every 5 frames). If the user is close to the image capture device and appears large in the video image, then the user's body features are analyzed more frequently, thus the user's body pose is analyzed more frequently (e.g., once every 2 frames). The extracted features are provided to a skeletal joint location module 414. The skeletal joint location module 414 analyzes the skeletal joint features to determine the coordinates of specific skeletal joints. For example, the skeletal joint location module 414 determines the x, y coordinates of specific points for each skeletal joint, such as the x, y coordinates of the left wrist, left elbow, left shoulder, nose (or other facial features, such as mouth, ear, or eyes), right wrist, right elbow, and right shoulder. The skeletal joint position module 414 provides the x, y coordinates of specified skeletal joints to the second machine learning technology module 417. In some embodiments, the skeletal joint position module 414 provides the filtered x, y coordinates of skeletal joints to the virtual object modification module 418. In one embodiment, the x, y coordinates represent the x, y offset of each skeletal joint relative to the neutral pose of the body.

[0074] The second machine learning module 417 analyzes the movement of skeletal joints on a sequence of video frames received prior to the monocular image 401 depicting the user's body to predict the estimated position of the skeletal joints in the monocular image 401 depicting the user's body. The number of previous video frames analyzed by the second machine learning module 417 or the duration of previously received video segments can be set, predetermined, or dynamically adjusted by the user. In some cases, the number of previous video frames analyzed may be increased (e.g., from 1 second video to 2 seconds video) if the amount of filtering or correction of the skeletal joint positions determined by the first machine learning module 412 exceeds a specified threshold. In some cases, the number of previous video frames analyzed may be increased (e.g., from 1 second video to 2 seconds video) based on the distance between the user and the camera exceeding a specified threshold. The second machine learning module 417 can filter or correct at least some of the coordinates of one or more skeletal joint positions provided by the skeletal joint position module 414 that do not match the skeletal joint positions predicted by the second machine learning module 417 based on previous video frames. In this example, the second machine learning technology module 417 determines that the left wrist coordinates (4, 6) do not match the coordinates (4, 7) predicted based on previous video frames, and therefore corrects the skeletal joint position to (4, 7) in the set of skeletal joint positions.

[0075] The pose determination module 416 determines the relative position of each skeletal joint received from the second machine learning technology module 417. For example, the pose determination module 416 determines that the left wrist is lower than the right wrist in the y-direction by more than a first specified amount but less than a second specified amount. Based on this determination, the pose determination module 416 estimates that the pose depicted in the image corresponds to a first pose. As another example, the pose determination module 416 determines that the left wrist is lower than the right wrist in the y-direction by more than a first specified amount and a second specified amount. Based on this determination, the pose determination module 416 estimates that the pose depicted in the image corresponds to a second pose. In some embodiments, the pose determination module 416 searches a database based on the x, y coordinates received from the skeletal joint position module 414 to identify poses within predetermined thresholds of the x, y coordinates.

[0076] The virtual object mode selection module 419 receives a selection of a virtualization mode from the client device 102. For example, a list of mode options may be presented to the user of the AR / VR application 105. In response to receiving the user's selection of a given mode option from the list, the given mode is provided to the virtual object mode selection module 419 as a selection of the virtualization mode. Mode options may include single avatar emulation options, multiple avatar emulation options, avatar visual control options, avatar follow options, and avatar virtual world interaction options. The virtualization mode selection controls how the position of the user's skeletal joints affects the avatar's skeleton or the avatar's position relative to the user on the display. Mode options may include a list of different forms and types of avatars for the user to choose from. The selected avatar is then used to obtain the corresponding skeleton to adjust the skeleton according to the user's posture.

[0077] The virtual object modification module 418 can adjust the skeleton of a given avatar based on the mode selected by the virtual object mode selection module 419 and the posture or skeletal joint position received from the skeletal joint position module 414 and / or posture determination module 416. The virtual object modification module 418 adjusts the presentation of the avatar in the image, for example, by changing the avatar's posture, visual attributes, and / or position. The adjusted avatar is provided by the virtual object modification module 418 to the virtual object display module 420. The virtual object display module 420 combines the adjusted avatar into a received monocular image depicting the user's body, thereby simultaneously presenting the adjusted avatar and the user in the image. This image is provided by the virtual object display module 420 to the client device 102, and can then be sent to another user or stored for later access and display.

[0078] For example, the selection of a single avatar mimicry option instructs the virtual object modification module 418 to modify the avatar's skeleton based on the relative distances between the skeletal joints of the user's body. In this way, the skeleton is modified to replicate or mimic the pose represented by the user's body. Specifically, if it is determined that the left wrist is offset higher than the neutral position of the left wrist in the captured RGB image by a certain proportion or percentage, then the corresponding avatar skeleton's left wrist is raised to a position also higher than the neutral position of the avatar's left wrist by the same proportion or percentage. Similarly, each skeletal joint is adjusted to replicate the relative positions of the skeletal joints of the body depicted in the image. For example, as... Figure 7A As shown, the user's body 723 is identified, and the position of the user's left wrist joint 723 is determined to be lower than the neutral left wrist joint position by a first amount, while the position of the user's right wrist joint 723 is determined to be higher than the neutral right wrist joint position by a second amount. In one embodiment, Figure 5 This represents the neutral skeletal joint positions of the user's body. The skeletal joint positions of the avatar 734's skeleton are similarly adjusted relative to their corresponding neutral positions to offset by similar first and second amounts. As a result, the avatar 734 appears to mimic or replicate the posture represented by the user's body 723.

[0079] As another example, the selection of multiple avatar imitation options instructs the virtual object modification module 418 to modify the skeleton of multiple copies of the avatar based on the relative distances between the skeletal joints of the user's body. Multiple avatar imitation options enable... Figure 8A The avatar depicted in the first neutral position 1020 shown in image 1011 is animated in a pose corresponding to the user's body as shown in image 1013. Specifically, the multiple avatars shown in image 1011 may appear in different neutral poses (e.g., crawling on the floor on all fours). In response to the selection of multiple avatar imitation options, pose estimation system 124 analyzes the sequence of received monocular RGB images to determine when the user's pose matches a specified pose. In response to determining that the user's pose matches a specified pose, pose estimation system 124 causes all avatars to replicate the user's pose (e.g., standing on two legs and positioning the left and right arms in a similar pose to the user). The avatar's skeleton is adjusted in a similar manner to that for a single avatar when a single avatar imitation option is selected. Another example is shown in images 1019 and 1015. In image 1013, the avatar 1030 transitions from a neutral position (lying flat on the floor) to a pose position 1032 that mimics the user's posture (e.g., standing on two legs and positioning the left and right arms in a posture similar to the user's). When the avatar is a collection of paper pieces, the paper pieces are animated as flying around until they are arranged in a certain order and in a posture that matches the user's body posture in front of or behind the user.

[0080] As another example, the selection of avatar visualization control options instructs the virtual object modification module 418 to modify the avatar's visual attributes based on the user's posture. For example, the avatar is presented on the screen as moving from top to bottom or from left to right at a given rate. Specifically, Figure 8B The cloud shown in image 1110 can be animated to move downwards across the screen in front of the user. The rate at which the cloud moves across the screen can be increased or decreased based on the user's posture or how slowly or quickly the user transitions from one posture to another. As another example, an avatar (e.g., a cloud) can be placed on the user's hand, as shown in image 1112. As the user changes their posture from a vertical and straight hand suspended in the air to a horizontally outstretched hand, the avatar continues to follow and position itself on the user's hand, but the visual properties of the avatar change (e.g., the cloud now presents a rainbow above the user, with the rainbow shifting from the first cloud on one hand to the second cloud on the second hand), as shown in image 1114. Changes in visual properties can also include changing the face depicted on the avatar from a smile to a frown, or the avatar's color can be changed based on changes in the user's skeletal joint position.

[0081] As another example, the selection of avatar virtual world interaction instructs virtual object modification module 418 to allow the avatar to interact with another virtual object in an image. For example, the avatar is displayed in an image that includes virtual objects (e.g., a virtual tree and virtual apples on the tree). The avatar's pose and movement track and mimic the user's skeletal joint positions in a manner that allows the avatar to interact with the virtual objects. Specifically, Figure 8C A virtual apple 810 and an avatar 812 are shown, their arms having left and right wrists positioned horizontally relative to each other in a specific vertical direction. In this position, the avatar 812's right hand is a first distance from the virtual apple 810. When the user's body 820, depicted in the image, raises its right arm, the avatar 812's right arm 816 also raises relative to its left arm, which in turn raises the avatar 812's right wrist. In this position, the avatar 812's right wrist and right elbow are higher than its left wrist and left elbow, and the avatar 812's right hand is a second distance from the virtual apple 810. The distance between the avatar 812's right hand and the virtual apple 810 is determined to be less than a threshold; therefore, the virtual apple 810 separates from the virtual tree 814 and is fixed to the avatar 812's right hand. As shown, the virtual apple 810 is no longer presented on the virtual tree 814, and when the user lowers their arms, the avatar's left and right arms are also lowered to show that the right hand is now holding the virtual apple 810. In this way, users can make Avatar 812 mimic their gestures to interact with one or more virtual objects in an image or video.

[0082] As another example, selecting the avatar follow option instructs the virtual object modification module 418 to maintain a given distance between the avatar and the user. Specifically, this option causes the avatar to follow the user across a series of images. For example, the avatar's position in an image depicting the user's body changes as the user's body position changes within the image. In particular, if the user moves a certain amount to the right, the displayed avatar's position also moves the same amount to the right. Figure 8C An avatar is shown at a first position 830 relative to the user depicted in the image and at a first distance between the avatar and the user in 2D or 3D space. As the user moves to a second position, the avatar is moved to a second position 832 to maintain the same distance from the user depicted in the image.

[0083] Figure 6 This is a flowchart illustrating example operations of the body pose estimation system 124 in execution process 600 according to an exemplary embodiment. Process 600 can be implemented as computer-readable instructions executable by one or more processors, such that the operation of process 600 can be performed partially or entirely by functional components of the messaging server system 108 and / or the AR / VR application 105; therefore, process 600 is described below by way of example with reference to it. However, in other embodiments, at least some operations of process 600 can be deployed on various other hardware configurations. Therefore, process 600 is not intended to be limited to messaging server system 108 and can be implemented wholly or partially by any other component. Some or all operations of process 600 can be performed in parallel, out of order, or completely omitted.

[0084] At operation 601, the body pose estimation system 124 receives a monocular image including a depiction of the user's body. For example, the first machine learning technology module 412 receives the monocular image 401 depicting the user's body. The first machine learning technology module 412 extracts one or more features from the image indicating skeletal joints.

[0085] At operation 602, the body pose estimation system 124 detects multiple skeletal joints of the body depicted in the monocular image 401. For example, the skeletal joint position module 414 processes features extracted by the first machine learning technique module 412 to determine the position of a set of skeletal joints. This position can be determined relative to the neutral position of each skeletal joint.

[0086] At operation 603, the body pose estimation system 124 accesses a video feed including multiple monocular images received prior to the monocular image. For example, the pose determination module 416 accesses 1 to 2 seconds of video received prior to the current image processed by the first machine learning technique module 412.

[0087] At operation 604, the body pose estimation system 124 uses video feed filtering to filter multiple skeletal joints of the body detected based on monocular images. For example, a second machine learning module 417 predicts the skeletal joint positions of the current frame based on skeletal joint positions in one or more previous frames (e.g., based on body movement in the previous 1 to 2 seconds of video). The second machine learning module 417 filters or corrects any mismatches between the skeletal joint positions output by the first machine learning module 412 based on the actually received current video frame and the predicted skeletal joint positions output by the second machine learning module 417 based on previous video frames and / or image features of the current frame.

[0088] At operation 605, the body pose estimation system 124 determines the pose of the body representation depicted in the monocular image based on multiple skeletal joints of the filtered body.

[0089] Figures 7A to 7C Illustrative inputs and outputs of a body posture estimation system 124 according to an example implementation are shown. Figures 7A to 7C The inputs and outputs shown can be implemented by AR / VR application 105. In some embodiments, a first user accesses body pose estimation system 124 to exchange images depicting the user and a corresponding avatar with another user. Screen 711, including an avatar selection area and an option to start exchanging images, is presented to the first user. After the first user selects an avatar and chooses the start option, instructions are presented to the first user on screen 712 to position the first user at a specified distance from the image capture device. For example, the first user is instructed to step back so that their body fits into the image captured by the image capture device.

[0090] In one implementation, as shown on screen 712, the first user is instructed to move far enough away from the image capture device until a predetermined set of skeletal joints becomes visible, but not all skeletal joints. Specifically, the body pose estimation system 124 may only require the wrist, elbow, shoulder, and nose positions to be visible in the image, without requiring the leg positions. In some implementations, the skeletal joints of the first user visible in the image cause the corresponding skeletal joints of the avatar to be adjusted. For example, if only the first user's arm is visible in the image, only the avatar's arm is adjusted to mimic the first user's arm position. If the user's entire body (including the user's legs) is visible, the entire avatar skeleton (including the avatar's legs) is adjusted to mimic the first user's body pose.

[0091] The user's body posture is captured on screen 713, and the body posture estimation system 124 generates an avatar 734 with the same or similar posture as the first user for display on screen 714. The first user can say a predetermined word, or, if the user maintains the same posture for a threshold time period, a screenshot or image is captured, characterized by the user in that posture and the avatar mimicking the user's posture.

[0092] like Figure 7B As shown, screenshot 741 is displayed to the first user on screen 721. Blank space 760 may be included in screen 721 to indicate that a corresponding screenshot from the second user is about to be received. The first user can select or press the blank area 760 to display a list of recipients on screen 722. Specifically, screen 722 displays the first user's list of friends. The first user can select a given friend "Matt," and the body posture estimation system 124 sends screenshot 741 to the selected friend "Matt."

[0093] like Figure 7C As shown, the second user 731 follows a similar sequence to the first user so that the second avatar 732 mimics the posture of the second user 731, as shown on screen 730. Specifically, screen 730 is provided on a different mobile device associated with the second user 731. The second user 731 selects the second avatar 732 and is instructed to back up so that the body posture estimation system 124 can estimate the body posture of the user 731 and adjust the selected second avatar 732 to mimic the body posture of the second user. The body posture estimation system 124 captures an image 751 depicting the second user 731 and the second avatar 732 with a given posture and presents the captured image 751 on screen 740. The captured image 751 is automatically sent to the first user. The captured image 751 depicting the second user 731 and the second avatar 732 with a given posture is presented together with a screenshot 741 depicting the first user and the first avatar with another posture.

[0094] Figure 9 This is a block diagram illustrating example software architecture 906, which can be used in conjunction with various hardware architectures described herein. Figure 9 This is a non-limiting example of software architecture, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. Software Architecture 906 can be applied to, for example... Figure 10 The execution is performed on the hardware of machine 1000, which includes processor 1004, memory 1014, and input / output (I / O) components 1018, etc. A representative hardware layer 952 is shown and can represent, for example... Figure 10The machine 1000. A representative hardware layer 952 includes a processing unit 954 having associated executable instructions 904. The executable instructions 904 represent executable instructions of the software architecture 906, including implementations of the methods, components, etc., described herein. Hardware layer 952 also includes a memory and / or storage module memory / storage device 956, which also has executable instructions 904. Hardware layer 952 may also include other hardware 958.

[0095] exist Figure 9 In the example architecture, software architecture 906 can be conceptualized as a stack of layers, where each layer provides specific functionality. For example, software architecture 906 may include layers such as operating system 902, libraries 920, framework / middleware 918, applications 916, and presentation layer 914. Operationally, applications 916 and / or other components within a layer can activate application programming interface (API) calls 908 through the software stack and receive messages 912 in response to API calls 908. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide a framework / middleware 918, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0096] Operating system 902 manages hardware resources and provides public services. Operating system 902 may include, for example, a kernel 922, services 924, and drivers 926. Kernel 922 can serve as an abstraction layer between hardware and other software layers. For example, kernel 922 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 924 can provide other public services to other software layers. Drivers 926 are responsible for controlling the underlying hardware or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 926 may include display drivers, camera drivers, Bluetooth drivers, etc. Drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0097] Library 920 provides common infrastructure used by application 916 and / or other components and / or layers. Library 920 provides functionality that allows other software components to perform tasks more easily than by directly interfacing with the functions of the underlying operating system 902 (e.g., kernel 922, services 924, and / or drivers 926). Library 920 may include system libraries 944 (e.g., the C standard library), which provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Furthermore, Library 920 may include API libraries 946, such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats such as MPREG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (e.g., OpenGL frameworks for rendering 2D and 3D graphical content on a display), database libraries (e.g., SQLite providing various relational database functionalities), web libraries (e.g., WebKit providing web browsing functionality), etc. Library 920 may also include a wide variety of other libraries 948 to provide many other APIs to application 916 and other software components / modules.

[0098] The framework / middleware 918 (sometimes also called middleware) provides a higher level of common infrastructure that can be used by applications 916 and / or other software components / modules. For example, the framework / middleware 918 can provide various UI (GUI) functionalities, advanced resource management, advanced location services, etc. The framework / middleware 918 can provide a wide range of other APIs that can be utilized by applications 916 and / or other software components / modules, some of which may be specific to a particular operating system 902 or platform.

[0099] Application 916 includes built-in applications 938 and / or third-party applications 940. Examples of representative built-in applications 938 may include, but are not limited to: contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 940 may include applications developed by entities other than platform-specific vendors using the Android™ or iOS™ Software Development Kit (SDK), and may be based on platforms such as iOS™, Android™, and... Mobile software running on the phone's mobile operating system or other mobile operating systems. Third-party application 940 can activate API calls 908 provided by the mobile operating system (e.g., operating system 902) to facilitate the functions described herein.

[0100] Application 916 can use built-in operating system functions (e.g., kernel 922, services 924, and / or drivers 926), libraries 920, and frameworks / middleware 918 to create a UI for interacting with the system's user. Alternatively or additionally, in some systems, interaction with the user can occur through a presentation layer, such as presentation layer 914. In these systems, the application / component "logic" can be separated from the application / component's user-interacting aspects.

[0101] Figure 10 This is a block diagram illustrating components of a machine 1000, according to some example embodiments, capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any or more of the methods discussed herein. Specifically, Figure 10 A schematic diagram of machine 1000 is shown as an example of a computer system, in which instructions 1010 (e.g., software, programs, applications, applets, apps, or other executable code) can be executed to cause machine 1000 to perform any or more of the methods discussed herein. Therefore, instructions 1010 can be used to implement the modules or components described herein. Instructions 1010 transform the general, non-programmable machine 1000 into a specific machine 1000 programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, machine 1000 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1000 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1000 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1010 specifying actions to be taken by machine 1000. Furthermore, although only a single machine 1000 is shown, the term "machine" should also be considered to include a collection of machines that individually or jointly execute instructions 1010 to perform any one or more of the methods discussed herein.

[0102] Machine 1000 may include processor 1004, memory / storage device 1006, and I / O unit 1018, which may be configured to communicate with each other, for example, via bus 1002. In an example embodiment, processor 1004 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 1008 and processor 1012 capable of executing instructions 1010. The term "processor" is intended to include multi-core processor 1004, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 10 Multiprocessor 1004 is shown, but machine 1000 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0103] Memory / storage device 1006 may include memory 1014, such as main memory or other memory storage devices, and storage cells 1016, both of which are accessed by processor 1004, for example, via bus 1002. Storage cells 1016 and memory 1014 store instructions 1010 embodying any one or more of the methods or functions described herein. Instructions 1010 may also reside wholly or partially in memory 1014, in storage cells 1016, in at least one of the processor 1004 (e.g., in the processor's cache memory), or in any suitable combination thereof during execution by machine 1000. Therefore, memory 1014, storage cells 1016, and the memory of processor 1004 are examples of machine-readable media.

[0104] I / O component 1018 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurement results, etc. The specific I / O component 1018 included in a particular machine 1000 will depend on the type of machine. For example, a portable machine such as a mobile phone is likely to include a touch input device or other such input mechanism, while a headless server machine is likely not to include such a touch input device. It will be understood that I / O component 1018 may include... Figure 10Many other components are not shown. The grouping of I / O components 1018 according to function is only for the sake of simplifying the following discussion, and this grouping is by no means limiting. In various example embodiments, I / O components 1018 may include output components 1026 and input components 1028. Output components 1026 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), auditory components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 1028 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide position and / or force for touch or touch gestures), audio input components (e.g., microphones), etc.

[0105] In other example implementations, I / O component 1018 may include biometric component 1039, motion component 1034, environmental component 1036, or positioning component 1038, as well as a wide range of other components. For example, biometric component 1039 may include components for detecting expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 1034 may include: accelerometer component (e.g., accelerometer), gravity sensor component, rotation sensor component (e.g., gyroscope), etc. Environmental component 1036 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an hearing sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects the concentration of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 1038 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0106] A variety of technologies can be used to implement communication. I / O component 1018 may include communication component 1040, which is operable to couple machine 1000 to network 1037 or device 1029 via coupling 1024 and coupling 1222, respectively. For example, communication component 1040 may include a network interface component or other suitable device to interface with network 1037. In another example, communication component 1040 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, Bluetooth component, etc. Components (e.g., Bluetooth) (low power consumption) Components and other communication components that provide communication via other modes. Device 1029 can be any of other machines or various peripheral devices (e.g., a peripheral device coupled via USB).

[0107] Furthermore, the communication component 1040 can detect identifiers or include components operable to detect identifiers. For example, the communication component 1040 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, data matrices, data symbols, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes) or an auditory detection component (e.g., a microphone for identifying audio signals from tags). Additionally, various information can be obtained via the communication component 1040, such as location via Internet Protocol (IP), etc. The location of signal triangulation, the location of NFC beacon signals that can be detected to indicate a specific location, etc.

[0108] Glossary:

[0109] In this context, "carrier signal" refers to any intangible medium capable of storing, encoding, or carrying transient or non-transient instructions executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions can be sent or received over a network using transient or non-transient transmission media via network interface devices and using any of a number of well-known transmission protocols.

[0110] In this context, "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, PDAs, smartphones, tablet computers, ultrabooks, netbooks, multiple laptops, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0111] In this context, "communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a POTS (Plain Old-Style Telephone Service) network, a cellular telephone network, a wireless network, etc. A network, another type of network, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any of a variety of data transmission technologies, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution of GSM (EDGE), the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other standards defined by various standards-setting organizations, other telematics protocols, or other data transmission technologies.

[0112] In this context, "short message" refers to a message that can be accessed for a limited time. Short messages can be text, images, videos, etc. The access time for a short message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0113] In this context, "machine-readable medium" means a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)) and / or any suitable combination thereof. The term "machine-readable medium" should be considered to include a single medium or multiple media capable of storing instructions (e.g., a centralized or distributed database or associated cache and server). The term "machine-readable medium" will also be considered to include any medium or combination of media capable of storing machine-executable instructions (e.g., code) such that, when executed by one or more processors of the machine, the instructions cause the machine to perform any or more methods described herein. Therefore, "machine-readable medium" refers to a single storage device or apparatus, and a "cloud-based" storage system or storage network comprising multiple storage devices or apparatuses. The term "machine-readable medium" does not include signals themselves.

[0114] In this context, a “component” refers to a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularity for a particular processing or control function. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components and can be part of a program that typically performs a specific function within a related function. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various example implementations, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to perform certain operations described herein.

[0115] Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field-programmable gate array (FPGA) or an ASIC. A hardware component may also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific part of a machine) uniquely tailored to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to implement a hardware component mechanically, either in a dedicated and permanently configured circuit or in a temporarily configured (e.g., software-configured) circuit, may be made for cost and time considerations. Accordingly, the phrase “hardware component” (or “hardware-implemented component”) should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate or perform certain operations described herein. Consider implementations where hardware components are temporarily configured (e.g., programmed), eliminating the need to configure or instantiate each hardware component at any given time. For example, where the hardware components include a general-purpose processor configured by software as a dedicated processor, this general-purpose processor can be configured as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to constitute a specific hardware component, for example, at one time and as different hardware components at different times.

[0116] Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled. In the presence of multiple hardware components, communication can be achieved through signal transmission (e.g., via appropriate circuitry and buses) between or among two or more hardware components. In embodiments where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessed by the multiple hardware components and retrieving information from that memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. Other hardware components can then access the memory device at a subsequent time to retrieve and process the stored output.

[0117] Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., information collection). Various operations of the example methods described herein can be performed, at least partially, by one or more processors configured, either temporarily (e.g., by software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented, at least partially, by processors, where a particular one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can also operate to support the execution of relevant operations in a "cloud computing" environment or as "Software as a Service" (SaaS). For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of certain operations can be distributed across processors, rather than residing within a single machine, but deployed across multiple machines. In some example implementations, the processor or processor-implemented components may reside in a single geographic location (e.g., in a home environment, office environment, or server cluster). In other example implementations, the processor or processor-implemented components may be distributed across several geographic locations.

[0118] In this context, "processor" refers to any circuit or virtual circuit (physical circuitry simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals used to operate the machine. For example, a processor can be a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an ASIC, a Radio Frequency Integrated Circuit (RFIC), or any combination thereof. A processor can also be a multi-core processor with two or more independent processors (sometimes called "cores") capable of executing instructions simultaneously.

[0119] In this context, a "timestamp" refers to a sequence of characters or encoded information that identifies when an event occurred, such as giving a date and time of day, sometimes accurate to a fraction of a second.

[0120] Changes and modifications may be made to the disclosed embodiments without departing from the scope of this disclosure. Such and other changes or modifications are intended to be included within the scope of this disclosure and are set forth in the appended claims.

Claims

1. A method for determining a pose represented by a body depicted in a monocular image, the method comprising: A monocular image, comprising a depiction of the user's body, is received by one or more processors; the monocular image is the first frame of a video. The one or more processors detect multiple skeletal joints of the body based on the monocular image; A first machine learning technique is used to identify multiple skeletal joint features in the monocular image, wherein the first machine learning technique is configured to process the monocular image to extract one or more skeletal joint features corresponding to the body depicted in the monocular image, and wherein the positions of the multiple skeletal joints are detected based on the identified multiple skeletal joint features. Access includes a video feed of multiple monocular images received prior to the monocular image, the multiple monocular images being frames in the video preceding the first frame; The video feed is used to filter the plurality of skeletal joints of the body detected based on the monocular image, wherein filtering the plurality of skeletal joints includes applying the video feed to a second machine learning technique, wherein the second machine learning technique is configured to predict the location of the skeletal joints. The second machine learning technique is configured to process the plurality of monocular images in the video feed to predict the skeletal joint positions of the first frame, and Filtering the plurality of skeletal joints includes: comparing the positions of skeletal joints detected by the first machine learning technique with the predicted positions of skeletal joints, and filtering at least some of the skeletal joints whose positions detected by the first machine learning technique do not match the predicted positions of skeletal joints; and The pose represented by the body as depicted in the monocular image is determined based on multiple skeletal joints of the filtered body.

2. The method according to claim 1, wherein, The video feed applied to the second machine learning technique is dynamically adjusted such that if the amount of filtering exceeds a specified threshold, the number of frames in the video feed is increased.

3. The method according to claim 1, wherein, The video feed applied to the second machine learning technique is dynamically adjusted such that the number of frames in the video feed is increased when the distance between the user and the camera device exceeds a specified threshold.

4. The method according to claim 1, further comprising: The pose of the avatar is modified by one or more processors by adjusting a set of skeletal joints of the avatar's skeleton based on multiple skeletal joints of the filtered body, in order to match the pose represented by the body depicted in the monocular image. as well as An avatar with a modified pose that matches the pose represented by the body depicted in the monocular image is generated by one or more processors for display.

5. The method according to claim 1, wherein, The first machine learning technique includes a first deep neural network.

6. The method of claim 5, further comprising training the first deep neural network by performing the following operations: Receive training data, which includes multiple training monocular images and real skeletal joint information for each of the multiple training monocular images, each of the multiple training monocular images depicting a different body posture; The first deep neural network is applied to the first training monocular image among the plurality of training monocular images to estimate the skeletal joints of the body depicted in the first training monocular image; Calculate the deviation between the estimated skeletal joints of the body and the real skeletal joint information associated with the first training monocular image; The parameters of the first deep neural network are updated based on the calculated bias; as well as The steps of applying, calculating, and updating are repeated for each of the plurality of training monocular images.

7. The method according to claim 1, wherein, The second machine learning technique includes a second deep neural network, and wherein the filtering includes comparing predictions of skeletal joint positions of the monocular image provided by the second deep neural network with multiple skeletal joint features of the monocular image identified using a first machine learning technique.

8. The method of claim 7, further comprising training the second deep neural network by performing the following operations: Receive training data, which includes multiple training videos and real skeletal joint information for each of the multiple training videos, each of the multiple training videos depicting a different body posture; The second deep neural network is applied to the first training video among the plurality of training videos to predict the skeletal joints of the body in frames following the first training video; Calculate the deviation between the predicted skeletal joints of the body and the actual skeletal joint information associated with the first training video; The parameters of the second deep neural network are updated based on the calculated bias; as well as The steps of applying, calculating, and updating are repeated for each of the multiple training videos.

9. The method according to claim 1, wherein, The plurality of monocular images are received within a threshold number of seconds prior to receiving the monocular image.

10. The method of claim 4, further comprising selecting, by the one or more processors, an avatar associated with the skeleton from a plurality of avatars.

11. The method according to claim 1, further comprising: Receive a second video comprising multiple monocular images, the multiple monocular images including a depiction of the user's body; Track changes in the multiple skeletal joints across the multiple monocular images; Changes in the body's posture are detected by tracking changes in the multiple skeletal joints; as well as The avatar's pose can be modified continuously or periodically to match changes in the pose of the body representation.

12. The method of claim 1, further comprising enabling the avatar to interact with a virtual object depicted in a given image based on the determined pose.

13. The method according to claim 1, wherein, The detection, filtering, and determination steps are performed without accessing depth information from the depth sensor.

14. The method according to claim 1, wherein, Detecting multiple skeletal joints of the body includes identifying points associated with the right wrist, right elbow, right shoulder, nose on the user's face, left shoulder, left elbow, and left wrist, respectively.

15. The method according to claim 1, wherein, The rate at which the multiple skeletal joints are detected is adjusted based on the user's position relative to the image capture device.

16. A system for determining a pose represented by a body in a monocular image, the system comprising: A processor configured to perform operations including: Receive a monocular image including a depiction of the user's body, the monocular image being the first frame of a video; Detect multiple skeletal joints of the body based on the monocular image; A first machine learning technique is used to identify multiple skeletal joint features in the monocular image, wherein the first machine learning technique is configured to process the monocular image to extract one or more skeletal joint features corresponding to the body depicted in the monocular image, and wherein the positions of the multiple skeletal joints are detected based on the identified multiple skeletal joint features. Access includes a video feed of multiple monocular images received prior to the monocular image, the multiple monocular images being frames in the video preceding the first frame; The video feed is used to filter multiple skeletal joints of the body detected based on the monocular image, wherein filtering the multiple skeletal joints includes applying the video feed to a second machine learning technique, wherein the second machine learning technique is configured to predict the location of the skeletal joints. The second machine learning technique is configured to process the plurality of monocular images in the video feed to predict the skeletal joint positions of the first frame, and Filtering the plurality of skeletal joints includes: comparing the positions of skeletal joints detected by the first machine learning technique with the predicted positions of skeletal joints, and filtering at least some of the skeletal joints whose positions detected by the first machine learning technique do not match the predicted positions of skeletal joints; and The pose of the body depicted in the monocular image is determined based on multiple skeletal joints of the filtered body.

17. The system according to claim 16, wherein, The second machine learning technique includes a deep neural network, and the operation further includes training the second machine learning technique in the following manner: Receive training data, which includes multiple training videos and real skeletal joint information for each of the multiple training videos, each of the multiple training videos depicting a different body posture; The deep neural network is applied to a first training video among the plurality of training videos to estimate the skeletal joints of the body depicted in the first training video; Calculate the deviation between the estimated skeletal joints of the body and the actual skeletal joint information associated with the first training video; The parameters of the deep neural network are updated based on the calculated bias; and The steps of applying, calculating, and updating are repeated for each of the multiple training videos.

18. A non-transitory machine-readable storage medium for determining the pose of a body representation depicted in a monocular image, wherein the non-transitory machine-readable storage medium includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations, the operations including: Receive a monocular image including a depiction of the user's body, the monocular image being the first frame of a video; Detect multiple skeletal joints of the body based on the monocular image; A first machine learning technique is used to identify multiple skeletal joint features in the monocular image, wherein the first machine learning technique is configured to process the monocular image to extract one or more skeletal joint features corresponding to the body depicted in the monocular image, and wherein the positions of the multiple skeletal joints are detected based on the identified multiple skeletal joint features. Access includes a video feed of multiple monocular images received prior to the monocular image, the multiple monocular images being frames in the video preceding the first frame; The video feed is used to filter multiple skeletal joints of the body detected based on the monocular image, wherein filtering the multiple skeletal joints includes applying the video feed to a second machine learning technique, wherein the second machine learning technique is configured to predict the location of the skeletal joints. The second machine learning technique is configured to process the plurality of monocular images in the video feed to predict the skeletal joint positions of the first frame, and Filtering the plurality of skeletal joints includes: comparing the positions of skeletal joints detected by the first machine learning technique with the predicted positions of skeletal joints, and filtering at least some of the skeletal joints whose positions detected by the first machine learning technique do not match the predicted positions of skeletal joints; and The pose of the body depicted in the monocular image is determined based on multiple skeletal joints of the filtered body.

Images