Systems and methods for predicting lower body posture

Abstract

The computing system may receive sensor data from one or more sensors coupled to the user. Based on the sensor data, the computing system may generate an upper body pose corresponding to a first portion of the user's body, which may include the user's head and arms. The computing system may process the user's upper body pose using a machine learning model to generate a lower body pose corresponding to a second portion of the user's body, which may include the user's legs. The computing system may generate a full body pose of the user based on the upper body pose and the lower body pose.

Description

[Technical field]

[0001] The present disclosure relates generally to predicting a user's lower body posture. [Background technology]

[0002] Machine learning can be used to enable machines to automatically detect and process objects. In general, machine learning typically involves processing a training dataset according to a machine learning model and updating the model based on the training algorithm to progressively "learn" features in the dataset that predict a desired output. One example of a machine learning model is a neural network, which is a network of interconnected nodes. Groups of nodes can be arranged in layers. The first layer of the network, which takes in input data, can be called the input layer. The last layer, which outputs data from the network, can be called the output layer. There can be any number of internal hidden layers that map nodes in the input layer to nodes in the output layer. In a feedforward neural network, the output of the nodes in each layer, except the output layer, is configured to feed forward to nodes in the subsequent layer.

[0003] Artificial reality is a form of reality that is conditioned in some way prior to presentation to a user and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or any combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., photos of the real world). Artificial reality content may include video, audio, haptic feedback, or some combination thereof. Any of them may be presented in a single channel or multiple channels (such as stereo video that creates a three-dimensional effect for the viewer). Artificial reality may relate to, for example, applications, products, accessories, services, or some combination thereof that are used to create content in the artificial reality and / or are used in the artificial reality (e.g., perform activities in the artificial reality). A virtual reality system that provides virtual reality content may be implemented on a variety of platforms, including a head mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing virtual reality content to one or more viewers. Summary of the Invention

[0004] The embodiments described herein relate to a method for generating one or more body postures of a user associated with one or more components of a virtual reality system. To enable the computing system to generate accurate postures of one or more joints that make up the body posture, the computing system may receive one or more sensor data or image data from one or more components of the virtual reality system (e.g., sensor data from a motion tracking sensor or image data received from one or more cameras). Using various techniques described herein, the image data and sensor data enable the computing system to accurately generate an upper body posture of the user based on one or more sensor data or image data associated with the virtual reality system. However, while this sensor data and image data is effective for generating a posture of the upper body of the user, it is often of limited use when generating a posture of the lower body of the user.

[0005] To remedy this problem, certain embodiments described herein utilize a machine learning model to generate lower body poses in response to receiving the generated upper body poses of a user. In certain embodiments, the machine learning model may be trained to receive the generated upper body poses and generate corresponding lower body poses. In certain embodiments, the machine learning model may be based on a Generative Adversarial Network (GAN) and may be trained using one or more training poses. Based on these training poses, the machine learning model may learn how to generate lifelike lower body poses. Once trained, the machine learning model may receive the generated upper body poses and output corresponding lower body or full body poses that may be used in various applications. For example, the output full body poses may be used to generate an avatar of the user in a virtual reality or artificial reality space.

[0006] According to a first aspect of the present disclosure, a method is provided that includes: a computing system receiving sensor data captured by one or more sensors coupled to a user; generating an upper body pose corresponding to a first part of the user's body, including the user's head and arms, based on the sensor data; generating a lower body pose corresponding to a second part of the user's body, including the user's legs, by processing the upper body pose using a machine learning model; and generating a full body pose of the user based on the upper body pose and the lower body pose.

[0007] The machine learning model may be trained using a second machine learning model that is trained to determine whether a given whole-body pose is likely to have been generated using the machine learning model.

[0008] The machine learning model may be optimized during training to cause a second machine learning model to erroneously determine that a given whole-body pose generated using the machine learning model is unlikely to have been generated using the machine learning model.

[0009] The machine learning model may be trained by generating a second lower body pose by processing a second upper body pose using the machine learning model, generating a second full body pose based on the second upper body pose and the second lower body pose, using the second machine learning model to determine whether the second full body pose could have been generated using the machine learning model, and updating the machine learning model based on the determination of the second machine learning model.

[0010] The one or more sensors may be associated with a head-mounted device worn by a user. Generating the upper body pose may include determining a plurality of poses corresponding to a plurality of predetermined body parts of the user based on the sensor data, and inferring the upper body pose based on the plurality of poses.

[0011] The multiple poses may include a wrist pose corresponding to a wrist of a user. The wrist pose may be determined based on one or more images captured by a head-mounted device worn by the user. The one or more images may represent (1) the user's wrist, or (2) a device held by the user.

[0012] The method may further include determining contextual information associated with a time at which the sensor data was captured. The lower body pose may be generated by further processing the contextual information using a machine learning model.

[0013] The context information may include the application with which the user is interacting when the upper body pose is determined. The method may further include generating an avatar of the user based on the whole body pose.

[0014] According to a second aspect of the present disclosure, one or more computer-readable non-transitory storage media are provided that include instructions that, when executed by a server, cause the server to perform the method of the first aspect of the present disclosure.

[0015] According to a third aspect of the present disclosure, there is provided a system comprising one or more processors and one or more computer-readable non-transitory storage media of the second aspect of the present disclosure coupled to the one or more processors.

[0016] The embodiments disclosed herein are merely examples, and the scope of the disclosure is not limited thereto. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments according to the invention are disclosed in the appended claims, which are particularly directed to methods, storage media, and systems, and any feature described in one claim category, e.g., a method, may also be claimed in another claim category, e.g., a system. Dependencies or references in the appended claims have been selected for formality reasons only. However, any subject matter resulting from an intentional reference (especially a multiple claim dependency) to any preceding claim may also be claimed. As a result, any combination of claims and features thereof may be disclosed and claimed regardless of the dependencies selected in the appended claims. The subject matter that may be claimed includes not only combinations of features recited in the appended claims, but also any other combination of features in the claims, and each feature recited in a claim may be combined with any other feature or other combination of features in the claim. Furthermore, any of the embodiments and features described or shown in this specification may be claimed in a separate claim and / or in any combination with any of the embodiments or features described or shown in this specification or with any of the features of the appended claims. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 illustrates an exemplary virtual reality system. [Diagram 2] FIG. 1 illustrates sample body postures associated with a user. [Diagram 3] FIG. 1 illustrates an example of using a machine learning model to generate lower body poses utilizing known predicted upper body poses. [Figure 4] FIG. 1 illustrates an arrangement for training a machine learning model for lower body pose prediction. [Diagram 5]FIG. 2 illustrates an exemplary method for training a generator. [Figure 6] FIG. 13 is a diagram showing generation of a whole body posture from a generated upper body posture and a generated lower body posture. [Figure 7] FIG. 2 illustrates an exemplary method for training a classifier. [Figure 8] FIG. 1 illustrates an example method for generating a full body pose of a user based on upper and lower body poses. [Figure 9] FIG. 1 illustrates an exemplary network environment associated with a social networking system. [Figure 10] FIG. 1 illustrates an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] FIG. 1 illustrates an exemplary virtual reality system 100. In a particular example, the virtual reality system 100 may include a headset 104, a controller 106, and a computing system 108. A user 102 may wear the headset 104, which may display visual virtual reality content to the user 102. The headset 104 may include an audio device that may provide audio virtual reality content to the user 102. The headset 104 may include one or more cameras 110 that may capture images and videos of the environment. The headset 104 may include an eye-tracking system for determining a vergence distance of the user 102. The vergence distance may be a distance from the user's eyes to an object (e.g., a real-world object or a virtual object in a virtual space) to which the user's eyes are converged. The headset 104 may be referred to as a head-mounted display (HMD). In a particular example, the computing system 108 may determine a pose of the headset 104 associated with the user 102. The headset pose may be determined by utilizing either sensor data or image data received by the computing system 108.

[0019] One or more controllers 106 may be paired with the virtual reality system 100. In a particular example, the one or more controllers 106 may be equipped with at least one inertial measurement unit (IMU) and infrared (IR) light emitting diodes (LEDs) for the virtual reality system 100 to estimate the pose of the controller and / or track the position of the controller so that the user can perform certain functions via the controller. The one or more controllers 106 may be equipped with one or more trackable markers distributed to be tracked by the computing system 108. The one or more controllers 106 may include a trackpad and one or more buttons. The one or more controllers 106 may receive input from the user 102 and relay the input to the computing system 108. The one or more controllers 106 may also provide haptic feedback to the user 102. The computing system 108 may be connected to the headset 104 and the one or more controllers 106 via a cable or wireless connection. The one or more controllers 106 may include a combination of hardware, software, and / or firmware not explicitly illustrated herein so as to not obscure other aspects of the disclosure.

[0020] In a particular example, the computing system 108 may receive sensor data from one or more sensors of the artificial reality system 100. In a particular example, the one or more sensors may be coupled to the user 102. In a particular example, the one or more sensors may be associated with a headset 104 worn by the user. By way of example and not by way of limitation, the headset 104 may include a gyroscope or inertial measurement unit that tracks the real-time movements of the user and outputs sensor data that represents or describes the movements. The sensor data provided by such motion tracking sensors may be used by a VR application to determine the user's current orientation and provide the orientation to a rendering engine to direct / reorient a virtual camera in 3D space. By way of another example and not by way of limitation, the one or more controllers 106 may include an inertial measurement unit (IMU) and infrared (IR) light emitting diodes (LEDs) configured to collect and transmit IMU sensor data to the computing system 108. In particular examples, the computing system 108 may utilize one or more sensor data with one or more tracking techniques (e.g., without limitation, SLAM tracking or IR-based tracking) to determine the pose of one or more components of the artificial reality system 100.

[0021] In certain examples, the computing system 108 may receive one or more image data from one or more components of the virtual reality system 100. In certain examples, this image data includes image data captured from one or more cameras 110 associated with the virtual reality system 100. For example, FIG. 1 illustrates one or more cameras 110 coupled within the headset 104. These one or more cameras may be positioned to capture one or more images associated with various perspectives (e.g., without limitation, one or more downward-facing cameras associated with the headset 104 (e.g., toward the feet of the user 102 while standing)).

[0022] In a particular example, the computing system 108 may determine a controller pose of one or more controllers 106 associated with the user 102. The controller pose associated with the user 102 may be determined by utilizing sensor or image data received by the computing system 108 and utilizing one or more techniques, such as, but not limited to, computer vision techniques (e.g., image classification). Methods for determining controller pose are further described in U.S. Patent Application No. 16 / 734,172, entitled “Joint Infrared and Visible Light Visual-Inertial Object Tracking,” filed on January 3, 2020, which is incorporated by reference herein in its entirety.

[0023] In particular examples, the computing system 108 may control the headset 104 and one or more controllers 106 to provide artificial reality content to and receive input from the user 102. The computing system 108 may be a standalone host computer system, an on-board computer system integrated with the headset 104, a mobile device, or any other hardware platform capable of providing artificial reality content to and receiving input from the user 102.

[0024] 2 illustrates a sample full body pose associated with a user 102. In a particular example, the computing system 108 may generate a full body pose 200 associated with the user 102, including an upper body pose 205 and a lower body pose 215. The full body pose 200 associated with the user 102 may attempt to reproduce the position and orientation of one or more joints of the user 102 utilizing the virtual reality system 100 at a particular time. In a particular example, the full body pose 200 associated with the user 102 comprises a skeletal frame of inverse kinematics ("skeleton" or "body pose"), which may comprise a list of one or more joints.

[0025] The upper body posture 205 may correspond to a portion of the body of the user 102. In a particular example, the upper body posture 205 may correspond to a portion of the body of the user 102 including at least the head and arms of the user 102. In a particular example, the upper body posture 205 may be generated by determining a plurality of postures corresponding to a plurality of predetermined body parts or joints of the user 102 (e.g., the head or wrists of the user 102). In a particular example, the upper body posture 205 may further include, by way of example and not limitation, a posture of one or more joints associated with the upper body of the user 102, such as, but not limited to, a head posture 210, a wrist posture 220, an elbow posture 220, a shoulder posture 240, a neck posture 250, or an upper spine posture 260.

[0026] The lower body posture 215 may correspond to a portion of the body of the user 102. In certain examples, the lower body posture 215 may correspond to a portion of the body of the user 102 including at least the legs of the user 102. In certain examples, the lower body posture 215 may further include, by way of example and not limitation, joint postures of one or more joints associated with the lower body of the user 102, such as, but not limited to, lower spine posture 270, hip posture 280, knee posture 290, or ankle posture 295.

[0027] In certain examples, one or more joint poses that make up the full body pose 220, upper body pose 205, or lower body pose 215 may be represented through a subset of parameters that represent, for example, but not by way of limitation, the position and / or orientation of each joint in the body pose 200. The nonlinear solver may parameterize each joint pose associated with the user 102 by seven degrees of freedom: three translational values ​​(e.g., x, y, z), three rotational values ​​(e.g., Euler angles in radians), and one uniform scale value. In certain examples, these parameters may be represented using one or more coordinate systems, for example, but not by way of limitation, via an absolute global coordinate system (e.g., x, y, z) or via a local coordinate system relative to a parent joint (e.g., but not by way of limitation, the head joint).

[0028] In certain examples, the full body pose 200, the upper body pose 205, or the lower body pose 215, and one or more joint poses that comprise the full body pose 200, the upper body pose 205, or the lower body pose 215 may be determined using sensor data or image data received from one or more components of the artificial reality system 100. In certain examples, the one or more joint poses that comprise the full body pose 200 may be generated using a combination of one or more techniques, such as, by way of example and not limitation, localization techniques (e.g., SLAM), machine learning techniques (e.g., neural networks), known spatial relationships with one or more components of the artificial reality system (e.g., known spatial relationships between the headset 104 and the head pose 210), visualization techniques (e.g., image segmentation), or optimization techniques (e.g., nonlinear solvers). In certain examples, one or more of these techniques may be utilized separately or with one or more other techniques. This full body pose 200 associated with the user 102 may be useful for various applications as described herein.

[0029] In certain examples, the computing system 108 may generate an upper body pose 205 for the user 102. The upper body pose 205 may be generated based on at least sensor data or image data. In certain examples, the computing system 108 may determine the upper body pose 205 using one or more poses corresponding to a plurality of predetermined body parts of the user 102, such as, but not limited to, joint poses of one or more joints that make up the upper body pose 200 (e.g., head pose 210 or wrist pose 220).

[0030] In a particular example, the one or more poses may include a head pose 210 associated with the user 102. The head pose 210 may include a position and orientation of a head joint of the user 102 while wearing the headset 104. The head pose 210 associated with the user 102 may be determined by the computing system 108 utilizing any of the sensor data and / or image data received by the computing system 108. In a particular example, this head pose 210 associated with the user 102 may be determined based on a pose of the headset 104 and a known spatial relationship between the headset 104 and the head of the user 102. In a particular example, the head pose 210 associated with the user 102 may be determined based on sensor data associated with the headset 104 worn by the user 102.

[0031] In a particular example, the one or more postures may include a wrist posture 220 associated with the user 102. The wrist posture 220 may include a position and orientation of a wrist joint of the user 102 while interacting with the artificial reality system 100. The wrist posture 220 associated with the user 102 may be determined by the computing system 108 utilizing either sensor data and / or image data received by the computing system 108. In a particular example, the image data may include one or more images captured by a headset 104 worn by the user 102. These one or more images may represent the wrist of the user 102 or a device (e.g., the controller 106) held by the user 102. In a particular example, the computing system 108 may determine the wrist posture 220 utilizing the one or more images using one or more computer vision techniques such as, for example, but not limited to, image classification or object detection. In a particular example, the wrist posture 220 may be determined based on the controller posture and a known spatial relationship between the controller 106 and the wrist of the user 102.

[0032] In certain examples, the upper body pose 205 may be inferred based on multiple poses, such as, but not limited to, head pose 210 or wrist pose 220. In certain examples, the upper body pose 205 may be inferred by a nonlinear kinematic optimization solver ("nonlinear solver") to infer one or more joint poses that make up the upper body pose 205. In certain examples, the nonlinear solver may include a C++ library built for inverse kinematics with a large set of common error functions that cover a wide range of applications. In certain examples, the nonlinear solver may provide one or more helper functions for tasks typically associated with global inverse kinematics problems (e.g., error functions for joint and skeletal structures, meshes and linear blend skinning, common constraints), or one or more helper functions for mesh deformations (e.g., Laplacian surface deformations), or one or more IO functions for various file formats.

[0033] In a particular example, the nonlinear solver may infer one or more joint poses that make up the upper body pose 205. The one or more joint poses that make up the upper body pose 205 associated with the user 102 inferred by the nonlinear solver may comprise a kinematic hierarchy at a particular time or state, stored as a list of one or more joint poses. In a particular example, the nonlinear solver may include one or more basic solvers, such as, but not limited to, an L-BFGS or Gauss-Newton solver, that are supported for inferring the poses of one or more joints that make up the upper body pose 205. In a particular example, the nonlinear solver may utilize a skeletal solver function to solve a single frame (single body pose) of inverse kinematics. For example, the skeletal solver function may take as input a current set of one or more parameters (e.g., joint poses) and optimize the parameters of the active subset given a defined error function. The contract of the nonlinear solver is to minimize the error value of the current function (e.g., the skeletal solver function) to infer an accurate upper body pose 205.

[0034] In particular examples, the nonlinear solver may represent one or more joint poses via a subset of parameters representing, for example, but not limited to, the position or orientation of each joint in the upper body pose 205. In particular examples, the nonlinear solver may parameterize each joint pose by seven degrees of freedom: three translation values ​​(e.g., x, y, z), three rotation values ​​(e.g., Euler angles in radians), and one uniform scale value. In particular examples, these parameters may be represented using one or more coordinate systems, for example, but not limited to, via an absolute global coordinate system (e.g., x, y, z) or via a local coordinate system relative to a parent joint (e.g., but not limited to, the head joint).

[0035] In certain examples, the nonlinear solver may assign one or more variable limits (e.g., minimum or maximum values) for each joint posture parameter. In certain examples, the nonlinear solver may assign a predefined static weight for each joint posture parameter. These predefined static weights may be determined based on, for example, but not by way of limitation, the accuracy of the sensor data used to determine the value of each variable. For example, the joint posture parameters representing the head posture 210 and wrist posture 220 associated with the user 102 may be assigned a higher static weight than one or more other joint postures or variables within the joint posture parameters because they are determined using a more accurate method, such as the SLAM techniques described herein. In certain examples, the nonlinear solver may use the joint parameters and predefined static weights to infer an upper body posture 205 that infers the most likely posture of one or more joints of the user 102 at a particular time or state.

[0036] While these techniques utilized alone or in combination typically allow for reliable determination of the upper body pose 205 and associated joints that make up the upper body pose 2015 (e.g., head pose 210, wrist pose 220, elbow pose 230, etc.), these techniques may be unreliable and inaccurate for determining the lower body pose 215 and associated joints that make up the lower body pose 215 (e.g., hip pose 280, knee pose 290, ankle pose 295, etc.). Because the user 102 is wearing only the headset 104 and one or more controllers 106, there may be limitations on the sensor or image data associated with the virtual reality system 100. For example, image data captured from one or more cameras 110 may not include at least a portion of the lower body of the user 102, resulting in little information to determine or generate an accurate lower body pose 215 associated with the user 102.

[0037] To remedy this problem, in a particular example, the computing system 108 may utilize one or more techniques described herein to generate the lower body pose 215 of the user 102. In a particular example, the lower body pose 215 may be generated by processing the upper body pose 205 using a machine learning model. FIG. 3 illustrates an example of utilizing an input upper body pose to generate a lower body pose. In particular, FIG. 3 illustrates an input upper body pose 205 associated with the user 102 of the artificial reality system 100, generated using the method described herein and received by the machine learning model 300. In a particular example, the machine learning model 300 may be based on a generative adversarial network (GAN). Using a particular example described herein, the machine learning model 300 may utilize the upper body pose 205 to generate the lower body pose 215. In a particular example, the computing system may combine the generated upper body pose 205 with the generated lower body pose 215 to generate a full body pose instead of or in addition to the lower body pose 215 generated by the machine learning model.

[0038] FIG. 4 illustrates a configuration for training a generative adversarial network (GAN) 400 for pose prediction. The GAN may include two separate neural networks, a generator 405 (interchangeably referred to herein as “G”) and a discriminator 410 (interchangeably referred to herein as “D”). In a particular example, the generator 405 and the discriminator 410 may be implemented as neural networks, for example and without limitation, although any network architecture suitable for the operations described herein may be utilized. In broad overview, the generator 405 may be configured to receive as input the generated upper body poses 205 and output the generated lower body poses 215. In a particular example, the upper body poses 205 may be combined with the lower body poses 215 to generate the whole body poses 425. In a particular example, the discriminator 410 may be configured to discriminate between “fake” full body poses 425 (including lower body poses output by the generator 405) and “real” training lower body poses 435 from a training pose database 440 that are not generated by the generator 405. In a particular example, the one or more training full body poses 435 may include full body poses from one or more images. The generator 405 and the discriminator 410 may be considered to be in an adversarial relationship because the objective of the generator 405 is to generate fake poses that fool the discriminator 410 (in other words, to increase the error rate of the discriminator 410) and the objective of the discriminator 410 is to correctly distinguish between the “fake” poses from the generator 405 and the “real” poses. In a particular example, a machine learning model (e.g., generator 405) is optimized during training to cause a second machine learning model (e.g., discriminator 410) to erroneously determine that a given whole-body pose generated using the machine learning model is unlikely to have been generated using the machine learning model.

[0039] In a particular example, each training full body pose 435 in the training pose dataset may be a “real” pose in the sense that they are not generated in whole or in part by the model 400. In a particular example, each training full body pose 435 may be obtained automatically by searching for different poses of a person from a pose database, where each of the different poses includes a full body pose 435 of the person. These “real” training full body poses 435 serve as ground truth for training the network of the classifier 410 to identify which full body poses are “real” and which are “fake”. Also, the training full body poses 435 may represent, for example, but not limited to, a person sitting, standing, or kneeling, or in another pose that is the same or different from the generated upper body pose 205 or the generated lower body pose 215 output by the generator 405. During operation, it may be unknown what body poses the machine learning model 400 will be asked to process, so randomness in the poses helps make the trained machine learning model 400 more robust.

[0040] In a particular example, the training of the machine learning model 200 may be performed simultaneously or in stages. For example, the first stage may be for training the generator 405. The second stage may be for training the discriminator 410 based on the output of the generator 405. The third stage may be for retraining / refining the generator 405 to better "fool" the trained discriminator 410. As another example, the training of the generator 405 and the discriminator 410 may be performed simultaneously.

[0041] In a particular example, the generator 405 may be configured to receive the upper body poses 205 and generate a time series of lower body poses 215, e.g., a series of lower body poses of a user standing, sitting, or walking. Rather than generating a single lower body pose 215, the generator 405 may generate a time series of lower body poses that may be combined with corresponding upper body poses to generate a time series of full body poses. In a particular example, the discriminator 410 may be configured and trained to determine whether the input series of poses is temporally consistent to discriminate between a "fake" time series of full body poses (including the time series of lower body poses output by the generator 405) and a "real" time series of training lower body poses from the training pose database 440 that are not generated by the generator 405. In a particular example, the time series of training full body poses may include full body poses from one or more images. In this temporal GAN, the generator 405 and the discriminator 410 may be considered to be in an adversarial relationship because the objective of the generator 405 is to generate fake time series poses for the discriminator 410 (in other words, to increase the error rate of the discriminator 410), and the objective of the discriminator 410 is to correctly distinguish between the “fake” time series whole body poses from the generator 405 and the “real” time series whole body poses. Once trained, the temporal generator can be trained to output a time series of authentic lower body poses given the input upper body poses 205. Although this disclosure primarily describes training and utilizing only one pose at a time using the methods described herein for readability and clarity, a GAN may instead be trained and utilized to output a time series of lower body poses using the same methods described herein.

[0042] FIG. 5 illustrates an exemplary method 500 for training a generator based on loss, according to a particular example. In broad overview, during training of the generator 405, the parameters of the generator 405 may be iteratively updated based on a comparison between the generated “fake” full body poses 425 (including the lower body poses 215 generated by the generator) and the corresponding training full body poses 435. The objective of the training is to maximize the prediction loss 460 for the “fake” poses. In doing so, the goal is to have the generator 405 learn how to generate upper body poses 215 based on the upper body poses 205 that can be used to generate “fake” full body poses 425 that look sufficiently “real”.

[0043] The method may begin at step 510, where a computing system may receive upper body poses 205 corresponding to a first part of a body of a user 102, the first part of the body including the head and arms of the user 102. The upper body poses 205 input to the generator 405 may be upper body poses generated based on sensor data and / or image data using one or more techniques described herein, or the upper body poses 205 input to the generator 405 may be upper body poses extracted from one or more real images of a person.

[0044] In step 520, the machine learning model may generate a lower body pose 215 corresponding to a second part of the body of the user 102 by processing the upper body pose 205. Here, the second part of the body includes the legs of the user 102. In a particular example, the generator 405 may generate a lower body pose 215 based on the received upper body pose 205. In a particular example, the generated lower body pose 215 corresponds to the second part of the body including the legs of the user. The generated lower body pose 215 may be used to generate a "fake" full body pose 425 that includes the generated lower body pose 215.

[0045] In a particular example, the computing system may determine context information associated with the artificial reality system 100. In a particular example, the generator may receive the context information. The generator 405 may utilize this context information to generate a lower body pose. This context information may be associated with a particular time when the sensor data and / or image data was captured. For example, without limitation, the context information may include information regarding whether the user 102 is sitting or standing. In another example, the context information may include information regarding an application (e.g., without limitation, a artificial reality application) with which the user is interacting when the upper body pose is determined (e.g., whether the user is interacting with an application associated with a business meeting or whether the user is interacting with an application associated with a game, such as a dance contest). In a particular example, the generator 405 may receive one or more context information associated with the artificial reality system 100. In a particular example, the lower body pose 215 is generated by further processing the context information using a machine learning model.

[0046] In a particular example, the computing system may determine one or more physical constraints associated with the upper body posture or the lower body posture, for example, using a physics-aware data augmentation method. The generator 405 may receive and utilize one or more of these physical constraints to generate a more realistic lower body posture or a more realistic time series of lower body postures. These physical constraints may be associated with one or more joint postures of the user, for example, but not limited to, a physical constraint on the range of motion for a knee joint posture that simulates one or more physical constraints of the human body. In a particular example, the computing system may determine one or more physical constraints when generating the time series of lower body postures. For example, but not limited to, the computing system may determine a time limit on the rate of acceleration or jitter of one or more joints between one or more successive postures in the time series of postures. In a particular example, the generator 405 may use the constraint to determine a more realistic time series of lower body postures.

[0047] In step 530, the system may generate a full body posture 425 based on the upper body posture 205 and the lower body posture 215. FIG. 6 illustrates generating a full body posture from the generated upper body posture and the generated lower body posture. In a particular example, the upper body posture 215 is generated using a method described herein and corresponds to a first part of the user's body, the first part of the body including the user's head and arms. In a particular example, the lower body posture 215 is generated using a method described herein and corresponds to a second part of the user's body, the second part of the body including the user's legs. In a particular example, the full body posture 425 may be generated by the machine learning model 400. In a particular example, the full body posture 425 may be generated by a computing system in a post-processing step.

[0048] In step 540, the system may use the discriminator 410 to determine whether the full body pose 425 is likely to have been generated using the lower body pose 215 generated by the generator 405 (i.e., whether it is “fake” or “real”). In a particular example, the goal of training the discriminator is to have it learn how to correctly distinguish between “real” and “fake” full body poses. For example, for each full body pose, the discriminator 410 may output a value between 0 and 1 that represents the confidence or probability / likelihood that the pose is generated by the generator (i.e., is “fake”). For example, a value closer to 1 may indicate a higher probability / confidence that the pose is “real” (i.e., not generated by the generator). A value closer to 0 may indicate a lower probability / confidence that the pose is “real” (which implicitly means a higher probability / confidence that the pose is “fake”). These predictions can be compared to known labels of poses that indicate which are "fake" and which are "real".

[0049] In a particular example, the system may calculate a loss based on a determination of D of whether it is likely that a full body pose was generated using the lower body pose 215 generated by the generator 405. For example, the loss may be calculated based on a comparison of the prediction made by the discriminator (e.g., a confidence / probability score) with a known label (which may be an implicit label) that the full body pose is "fake". In a particular example, for a given pose, the prediction of the discriminator 410 (represented as an output of a value between 0 and 1 representing the confidence or probability / likelihood of the pose being generated by the generator 405) may be compared with the known labels of the poses indicating which are "fake" and which are "real". For example, if a full body pose is known to be "fake", a prediction closer to 1 may result in a higher loss and a prediction closer to 0 may result in a lower loss. In other words, the loss may be a measure of the accuracy of the prediction of the discriminator 410.

[0050] In step 550, the system may update the generator based on the loss. The parameters of the generator may be updated with the goal of maximizing the loss output by the discriminator 410. Since the goal of the generator is to "trick" the discriminator into thinking that the whole body poses generated using the lower body poses 215 generated by the generator are "real", the training algorithm may be configured to optimize the loss. In other words, the parameters of the generator are updated with the goal of optimizing the generator to generate lower body poses (which are subsequently used to generate whole body poses) that increase the loss of the discriminator's predictions (i.e., increase the erroneous predictions that a given whole body pose generated using the lower body poses 215 generated by the generator 405 is unlikely to have been generated using the generator 405).

[0051] Next, in step 560, the system may determine whether training of the generator 405 is complete. In certain examples, the system may determine whether training is complete based on one or more termination criteria. For example, the system may determine that training is complete if the loss is below a predetermined threshold and / or if the change in loss over the last several iterations has stabilized (e.g., fluctuated within a predetermined range). Alternatively or additionally, training may be considered complete if a predetermined number of training iterations have been completed or if a predetermined number of training samples have been used. If training is not yet complete, the system may, in certain examples, repeat the process beginning with step 510 to continue training the generator. Alternatively, if training is determined to be complete, the trained generator may be used in operation.

[0052] The training objective is for the generator 405 to learn to generate poses 215 that fool the discriminator 410 into thinking the poses are “real”. In a particular example, the generator 405 may generate lower body poses 215, and the discriminator 410 may predict the likelihood that the resulting full body poses 425 are “real” or “fake” 450. If a high prediction value represents that the discriminator 410 believes that the pose is more likely to be “real”, then the training objective may be expressed in terms of maximizing the prediction value of the discriminator 410 for the “fake” full body poses 215. If the correctness of the prediction of the discriminator 410 for the “fake” full body poses 215 is expressed as a loss 460, then the training objective may be expressed in terms of maximizing that loss 460. Based on the loss function and the training objective, the parameters of the generator 405 may be iteratively updated after each training such that the generator 405 becomes more capable of generating lower body poses 215 that can "fool" the discriminator 410. Thus, once trained, the generator 405 may be used to process a given upper body pose to automatically generate realistic lower body poses.

[0053] Returning to FIG. 4 , according to a particular example, the generator 405 can be used to simultaneously train the classifier 410 while the generator 405 is being trained. In broad overview, the generator 405 can process a given upper body pose 205 to generate a lower body pose 215. A computing system can utilize the upper body pose 205 and the lower body pose 215 to generate a full body pose 425. The generated full body pose 425 can be provided as input to the classifier 410 along with one or more training full body poses 435 from a training pose database 440.

[0054] FIG. 7 illustrates an example method 700 for training the classifier 410. In a particular example, the generator 405 may be used to train the classifier 410. In a particular example, the generator 405 and the classifier 410 may be trained simultaneously. The classifier 410 may be tasked with processing input poses (e.g., the “fake” whole body poses 425 and the “real” training whole body poses 435) and predicting which are “real” and which are “fake” 450. Conceptually, the goal of training the classifier 410 may be to maximize the prediction value of the classifier 410 for the generated whole body poses 425 and the training whole body poses 435, and minimize the prediction value of the classifier 410 for the generated whole body poses 425. In other words, if the correctness of the predictions of the classifier 410 for the "fake" poses (e.g., generated whole body poses 425) and the "real" poses (i.e., training whole body poses 435) are represented as losses 460 and 465, respectively, the goal of training the classifier 410 is to minimize the loss (i.e., minimize inaccurate predictions). Based on the loss function and the training objective, the parameters of the classifier 410 may be iteratively updated after each prediction such that the classifier 410 becomes more capable of distinguishing between the "real" and "fake" whole body poses.

[0055] The method may begin at step 710, where a computing system may receive upper body poses 205 corresponding to a first part of a body of a user 102, where the first part of the body includes the head and arms of the user 102. The upper body poses 205 input to the generator 405 may be upper body poses generated based on sensor data and / or image data using one or more techniques described herein, or the upper body poses 205 input to the generator 405 may be upper body poses extracted from one or more real images of a person. In certain examples, step 710 may proceed similarly to step 510 described herein.

[0056] In step 720, the machine learning model may generate a lower body pose 215 corresponding to a second part of the body of the user 102 by processing the upper body pose 205. Here, the second part of the body includes the legs of the user 102. In a particular example, the generator 405 may generate a lower body pose 215 based on the received upper body pose 205. In a particular example, the generated lower body pose 215 corresponds to the second part of the body including the legs of the user. In a particular example, the generator 405 may generate the lower body pose 215 based on its current parameters, which may be iteratively updated during training so that the generator gains a higher ability to generate authentic lower body poses. The generated lower body pose 215 may be used to generate a "fake" full body pose 425 that includes the generated lower body pose 215. In a particular example, step 720 may proceed similarly to step 520 described herein.

[0057] In a particular example, the computing system may determine context information associated with the artificial reality system 100. In a particular example, the generator may receive the context information. The generator 405 may utilize this context information to generate a lower body pose. This context information may be associated with a particular time when the sensor data and / or image data was captured. For example, without limitation, the context information may include information regarding whether the user 102 is sitting or standing. In another example, the context information may include information regarding an application (e.g., without limitation, a artificial reality application) with which the user is interacting when the upper body pose is determined (e.g., whether the user is interacting with an application associated with a business meeting or whether the user is interacting with an application associated with a game, such as a dance contest). In a particular example, the generator 405 may receive one or more context information associated with the artificial reality system 100. In a particular example, the lower body pose 215 is generated by further processing the context information using a machine learning model.

[0058] In step 730, the system may generate full body pose 425 based on upper body pose 205 and lower body pose 215. In certain examples, full body pose 425 may be generated by machine learning model 400. In certain examples, full body pose 425 may be generated by a computing system in a post-processing step. In certain examples, step 730 may proceed similarly to step 530 described herein.

[0059] In step 740, the system may use the discriminator 410 to determine whether the full body pose 425 is likely to have been generated using a lower body pose 215 generated by using the generator 405 (i.e., whether it is “fake” or “real”). In a particular example, the goal of training the discriminator is to have it learn how to correctly distinguish between “real” and “fake” full body poses. For example, for each full body pose, the discriminator 410 may output a value between 0 and 1 that represents the confidence or probability / likelihood that the pose is generated by the generator (i.e., is “fake”). For example, a value closer to 1 may indicate a higher probability / confidence that the pose is “real” (i.e., not generated by the generator). A value closer to 0 may indicate a lower probability / confidence that the pose is “real” (which implicitly means a higher probability / confidence that the pose is “fake”). These predictions may be compared to known labels of poses that indicate which are "fake" and which are "real." In particular examples, step 740 may proceed similarly to step 540 described herein.

[0060] In a particular example, the system may calculate a loss based on the discriminator's determination of whether a full body pose is likely to have been generated using the lower body pose 215 generated by the generator 405. For example, the loss may be calculated based on a comparison of the prediction made by the discriminator (e.g., a confidence / probability score) with a known label (which may be an implicit label) that the full body pose is "fake". In a particular example, for a given pose, the discriminator's 410 prediction (represented as an output of a value between 0 and 1 representing the confidence or probability / likelihood of the pose being generated by the generator 405) may be compared with the known labels of the poses indicating which are "fake" and which are "real". For example, if a full body pose is known to be "fake", a prediction closer to 1 may result in a higher loss and a prediction closer to 0 may result in a lower loss. In other words, the loss may be a measure of the accuracy of the discriminator's 410 prediction.

[0061] In step 750, the system may update the classifier based on the losses. The parameters of the classifier 410 may be updated with the goal of minimizing the losses that the classifier outputs. In a particular example, these losses may be backpropagated and used by a training algorithm to update the parameters of the classifier 410 so that over the course of training, the classifier becomes increasingly better at distinguishing between "fake" and "real" poses. The goal of the training algorithm may be to minimize the losses (i.e., minimize inaccurate predictions).

[0062] Then, in step 760, the system may determine whether training of the classifier 405 is complete. In certain examples, the system may determine whether training is complete based on one or more termination criteria. For example, the system may determine that training is complete if the loss is below a predetermined threshold and / or if the change in loss over the last several iterations has stabilized (e.g., fluctuated within a predetermined range). Alternatively or additionally, training may be considered complete if a predetermined number of training iterations have been completed or if a predetermined number of training samples have been used. If training is not yet complete, the system may repeat the process beginning with step 710 to continue training the classifier, in certain examples.

[0063] Although this disclosure describes and illustrates certain steps of the method of Figure 7 as occurring in a particular order, this disclosure contemplates that any suitable steps of the method of Figure 7 be occurring in any suitable order. Additionally, while this disclosure describes and illustrates an exemplary method for training a classifier that includes certain steps of the method of Figure 7, this disclosure contemplates any suitable method for training a classifier that includes any suitable steps, which may include all, some, or none of the steps of the method of Figure 7, where appropriate. Additionally, while this disclosure describes and illustrates certain components, devices, or systems that perform certain steps of the method of Figure 7, this disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of Figure 7.

[0064] Returning to FIG. 5, the generator 405, once trained, may be configured to receive input upper body poses 205 and generate lower body poses 215. After training, the poses are realistic, and when combined with the upper body poses 205, the computing system generates realistic full body poses 425. Furthermore, once the generator is trained, it may be used to generate lower body poses based on any input upper body pose (in other words, not limited to generating lower body poses based on poses that appear in the training pose dataset). Also, the trained generator may be distributed to different platforms different from the training system, including, for example, a user's mobile device or other personal computing device.

[0065] In step 570, the trained generator may access upper body poses 205. The upper body poses 205 may correspond to a first part of a body of the user 102, where the first part of the body includes the head and arms of the user 102. The upper body poses 205 input to the trained generator may be upper body poses generated based on sensor data using one or more techniques described herein, or the upper body poses 205 input to the trained generator 405 may be upper body poses from one or more real images of a person.

[0066] Then, in step 580, the trained generator may generate a lower body pose corresponding to a second part of the user's body. The lower body pose may correspond to a second part of the user's body, where the second part of the body includes the legs of the user 102. In a particular example, the trained generator may generate a lower body pose 215 based on the received upper body pose 205. In a particular example, the generated lower body pose 215 corresponds to the second part of the user's body, where the legs include the legs.

[0067] In certain examples, the trained generator may receive and process contextual information that specifies further information for generating an accurate lower body pose 215. In certain examples, the contextual information may be associated with the time that the sensor data used to generate the upper body pose 205 was captured. For example, the generator may receive contextual information that includes the application that the user 102 was interacting with when the sensor data was captured, such as a virtual reality application used for a virtual business meeting. Based on this information, the trained generator 405 may be more likely to generate a lower body pose 215 that is typical of a user sitting or standing in a work environment.

[0068] Returning to FIG. 6 , in certain examples, the computing system may generate a full body pose 425 based on the upper body pose 205 and the lower body pose 215. In certain examples, the full body pose 425 may be generated by the machine learning model 400. In certain examples, the full body pose 425 may be generated by the computing system in a post-processing step. In certain examples, this full body pose 425 may be utilized for various applications. For example, the computing system 108 may utilize the full body pose 425 to generate an avatar of the user 102 in a virtual reality or artificial reality space. In certain examples, the computing system 108 may utilize only a portion of the full body pose 425 for various applications, such as an inferred pose of only the upper body pose (e.g., from the user's head to the user's hips) or one or more joints (e.g., elbow joint 230 or knee joint 290), by way of example and not limitation.

[0069] Although this disclosure describes and illustrates certain steps of the method of FIG. 5 as occurring in a particular order, this disclosure contemplates that any suitable steps of the method of FIG. 5 be occurring in any suitable order. Additionally, although this disclosure describes and illustrates an exemplary method for training a generator based on an encoded representation of features that includes certain steps of the method of FIG. 5, this disclosure contemplates any suitable method for training a generator that includes any suitable steps, which may include all, some, or none of the steps of the method of FIG. 5, as appropriate. Additionally, although this disclosure describes and illustrates certain components, devices, or systems that perform certain steps of the method of FIG. 5, this disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of FIG. 5.

[0070] Although this disclosure describes and illustrates certain steps of the method of FIG. 4 as occurring in a particular order, this disclosure contemplates that any suitable steps of the method of FIG. 4 be occurring in any suitable order. Additionally, while this disclosure describes and illustrates an exemplary method for training a generator based on a recursive loss that includes certain steps of the method of FIG. 4, this disclosure contemplates any suitable method for training a generator based on a recursive loss that includes any suitable steps that may include all, some, or none of the steps of the method of FIG. 4, as appropriate. Additionally, while this disclosure describes and illustrates certain components, devices, or systems that perform certain steps of the method of FIG. 4, this disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of FIG. 4.

[0071] FIG. 8 illustrates an exemplary method 800 for generating a full body posture of a user based on an upper body posture and a lower body posture. The method may begin at step 810, where a computing system may receive sensor data captured by one or more sensors coupled to the user. In a particular example, the one or more sensors may be coupled to the user 102. In a particular example, the one or more sensors may be associated with a headset 104 worn by the user. By way of example and not by way of limitation, the headset 104 may include a gyroscope or inertial measurement unit that tracks real-time movements of the user and outputs sensor data that represents or describes the movements. By way of another example and not by way of limitation, the one or more controllers 106 may include an inertial measurement unit (IMU) and infrared (IR) light emitting diodes (LEDs) configured to collect and transmit IMU sensor data to the computing system 108. By way of another example, the sensor data may include one or more image data from one or more components of the virtual reality system 100.

[0072] In step 820, the computing system may generate an upper body pose corresponding to a first part of the user's body, where the first part of the body includes the user's head and arms. In a particular example, the computing system may utilize sensor data to generate the upper body pose. In a particular example, the computing system 108 may determine a plurality of poses corresponding to a plurality of predetermined body parts of the user 102, such as, but not limited to, joint poses (e.g., head pose 210 or wrist pose 220). In a particular example, the one or more joint poses constituting the upper body pose 200 may be determined or inferred using a combination of one or more techniques, such as, but not limited to, localization techniques (e.g., SLAM), machine learning techniques (e.g., neural networks), known spatial relationships with one or more artificial reality system components (e.g., known spatial relationships between the headset 104 and the head pose 210), visualization techniques (e.g., image segmentation), or optimization techniques (e.g., nonlinear solvers).

[0073] In step 830, the computing system may generate a lower body pose corresponding to a second part of the user's body, where the second part of the body includes the user's legs. In a particular example, the computing system may utilize a machine learning model to generate the lower body pose, such as, but not limited to, a generative adversarial network (GAN), including a generator network and a discriminator network. In a particular example, the generator may receive contextual information associated with the upper body pose. The generator may utilize this contextual information to generate the lower body pose. This contextual information may be associated with a particular time when the sensor data and / or image data was captured.

[0074] In step 840, the computing system may generate a full body pose for the user based on the upper body pose and the lower body pose. In certain examples, the full body pose may be generated by a machine learning model. In certain examples, the full body pose may be generated by the computing system in a post-processing step. In certain examples, the full body pose may be utilized in various applications. For example, the computing system may utilize the full body pose to generate an avatar of the user in a virtual reality or artificial reality space.

[0075] Particular examples may repeat one or more steps of the method of FIG. 8, where appropriate. Although the present disclosure describes and illustrates certain steps of the method of FIG. 8 as occurring in a particular order, the present disclosure contemplates any suitable steps of the method of FIG. 8 occurring in any suitable order. Additionally, while the present disclosure describes and illustrates an exemplary method for generating a full body posture of a user based on upper and lower body postures that includes certain steps of the method of FIG. 8, the present disclosure contemplates any suitable method for generating a full body posture of a user based on upper and lower body postures that includes any suitable steps, which may include all, some, or none of the steps of the method of FIG. 8, where appropriate. Additionally, while the present disclosure describes and illustrates certain components, devices, or systems that perform certain steps of the method of FIG. 8, the present disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of FIG. 8.

[0076] FIG. 9 illustrates an exemplary network environment 900 associated with a social networking system. The network environment 900 includes a client system 930, a social networking system 960, and a third party system 970 connected to one another by a network 910. Although FIG. 9 illustrates a particular configuration of the client system 930, the social networking system 960, the third party system 970, and the network 910, this disclosure contemplates any suitable configuration of the client system 930, the social networking system 960, the third party system 970, and the network 910. By way of example and not limitation, two or more of the client system 930, the social networking system 960, and the third party system 970 may be directly connected to one another, bypassing the network 910. As another example, two or more of the client system 930, the social networking system 960, and the third party system 970 may be physically or logically located in whole or in part at the same location as one another. 9 illustrates a particular number of client systems 930, social networking systems 960, third party systems 970, and networks 910, this disclosure contemplates any suitable number of client systems 930, social networking systems 960, third party systems 970, and networks 910. By way of example, and not limitation, network environment 900 may include multiple client systems 930, social networking systems 960, third party systems 970, and networks 910.

[0077] This disclosure contemplates any suitable network 910. By way of example, and not limitation, one or more portions of network 910 may include an ad-hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular network, or a combination of two or more of these. Network 910 may include one or more networks 910.

[0078] Links 950 may connect client systems 930, social networking systems 960, and third party systems 970 to the communications network 910 or to each other. This disclosure contemplates any suitable links 950. In particular examples, one or more links 950 include one or more wireline links (such as, for example, Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)), wireless links (such as, for example, Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical links (such as, for example, Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)). In particular examples, one or more links 950 each include an ad-hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a cellular technology based network, a satellite communication technology based network, another link 950, or a combination of two or more such links 950. The links 950 need not necessarily be the same throughout the network environment 900. One or more first links 950 may differ from one or more second links 950 in one or more respects.

[0079] In certain examples, client system 930 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more such components, and is capable of performing suitable functions implemented or supported by client system 930. By way of example and not limitation, client system 930 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a mobile phone, a smart phone, an augmented / virtual reality device, other suitable electronic device, or any suitable combination thereof. This disclosure contemplates any suitable client system 930. A client system 930 may enable a network user at client system 930 to access network 910. A client system 930 may enable its user to communicate with other users of other client systems 930.

[0080] In a particular example, the client system 930 may include a web browser 932 and may have one or more add-ons, plug-ins, or other extensions. A user of the client system 930 may enter a uniform resource locator (URL) or other address that directs the web browser 932 to a particular server (such as the server 962, or a server associated with the third-party system 970), and the web browser 932 may generate a hypertext transfer protocol (HTTP) request and communicate the HTTP request to the server. The server may accept the HTTP request and communicate one or more hypertext markup language (HTML) files to the client system 930 in response to the HTTP request. The client system 930 may render a web page based on the HTML files from the server for presentation to the user. This disclosure contemplates any suitable web page file. By way of example, and not limitation, a web page may be rendered from an HTML file, an Extensible Hypertext Markup Language (XHTML) file, or an Extensible Markup Language (XML) file, depending on particular needs. Such a page may execute scripts, a combination of markup language and scripts, and the like. As used herein, a reference to a web page encompasses, where appropriate, one or more corresponding web page files (that a browser may use to render the web page), and vice versa.

[0081] In certain examples, social networking system 960 may be a network-addressable computing system that may host an online social network. Social networking system 960 may generate, store, receive, and transmit social networking data, such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to an online social network. Social networking system 960 may be accessed directly or via network 910 by other components of network environment 900. By way of example and not limitation, client system 930 may access social networking system 960 directly or via network 910 using a web browser 932 or a native application associated with social networking system 960 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In certain examples, social networking system 960 may include one or more servers 962. Each server 962 may be a single server or a distributed server across multiple computers or multiple data centers. The servers 962 may be of various types, such as, for example, but not limited to, a web server, a news server, a mail server, a message server, an advertisement server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In certain examples, each server 962 may include hardware, software, or embedded logic components, or a combination of two or more such components, for performing suitable functions implemented or supported by the server 962. In certain examples, the social networking system 960 may include one or more data stores 964. The data stores 964 may be used to store various types of information.In particular examples, information stored in data stores 964 may be organized according to a particular data structure. In particular examples, each data store 964 may be a relational database, a columnar database, a correlation database, or other suitable database. Although this disclosure describes or illustrates particular types of databases, this disclosure contemplates any suitable type of database. Particular examples may provide an interface that allows a client system 930, a social networking system 960, or a third party system 970 to manage, retrieve, modify, add, or delete information stored in data stores 964.

[0082] In certain examples, the social networking system 960 may store one or more social graphs in one or more data stores 964. In certain examples, the social graph may include multiple nodes, which may include multiple user nodes (each corresponding to a particular user) or multiple concept nodes (each corresponding to a particular concept), and multiple edges connecting the nodes. The social networking system 960 may provide users of an online social network with the ability to communicate and interact with other users. In certain examples, a user may join an online social network via the social networking system 960 and then add connections (e.g., relationships) to some other users of the social networking system 960 with whom the user wishes to connect. As used herein, the term "friend" may refer to any other users of the social networking system 960 with whom the user has connected, associated, or formed a relationship via the social networking system 960.

[0083] In particular examples, social networking system 960 may provide users with the ability to take actions on various types of items or objects supported by social networking system 960. By way of example and not limitation, items and objects may include groups or social networks to which a user of social networking system 960 may belong, events or calendar entries in which a user may be interested, computer-based applications that a user may use, transactions that allow a user to buy or sell items via the service, interactions with advertisements that a user may perform, or other suitable items or objects. A user may interact with anything that can be represented within social networking system 960 or by an external system of third-party system 970 that is separate from social networking system 960 and coupled to social networking system 960 via network 910.

[0084] In particular examples, social networking system 960 may be capable of linking various entities. By way of example and not limitation, social networking system 960 may enable users to interact with each other as well as receive content from third party system 970 or other entities, or may enable users to interact with these entities through application programming interfaces (APIs) or other communication channels.

[0085] In certain examples, third-party system 970 may include one or more types of servers, one or more data stores, one or more interfaces, including, but not limited to, for example, an API, one or more web services, one or more content sources, one or more networks, or any other suitable components with which the servers may communicate. Third-party system 970 may be operated by an entity different from the entity that operates social networking system 960. However, in certain examples, social networking system 960 and third-party system 970 may work in conjunction with one another to provide social networking services to users of social networking system 960 or third-party system 970. In this sense, social networking system 960 may provide a platform or backbone that other systems, such as third-party system 970, may use to provide social networking services and functionality to users over the Internet.

[0086] In particular examples, third party systems 970 may include third party content object providers. A third party content object provider may include one or more sources of content objects that may be communicated to client systems 930. By way of example and not by way of limitation, the content objects may include information about matters or activities of interest to a user, such as, for example, movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not by way of limitation, the content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.

[0087] In certain examples, social networking system 960 also includes user-generated content objects that may increase user interaction with social networking system 960. User-generated content may include anything that a user may add, upload, transmit, or "post" to social networking system 960. By way of example and not limitation, a user communicates posts from client system 930 to social networking system 960. Posts may include data such as status updates or other text data, location information, photos, videos, links, music, or other similar data or media. Content may also be added to social networking system 960 by third parties through "communication channels," such as news feeds or streams.

[0088] In certain examples, the social networking system 960 may include various servers, subsystems, programs, modules, logs, and data stores. In certain examples, the social networking system 960 may include one or more of a web server, an action logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, an authorization / privacy server, a search module, an ad targeting module, a user interface module, a user profile store, a connectivity store, a third-party content store, or a location store. The social networking system 960 may also include suitable components such as network interfaces, security mechanisms, load balancers, failover servers, administration and network operations consoles, other suitable components, or any suitable combination thereof. In certain examples, the social networking system 960 may include one or more user profile stores for storing user profiles. The user profiles may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information such as work experience, educational background, hobbies or preferences, interests, affinities, or location. The interest information may include interests related to one or more categories. The categories may be general or specific. By way of example and not limitation, if a user "likes" articles about shoe brands, the category may be the brand or the general category of "shoes" or "clothing". The connection store may be used to store connection information about users. The connection information may indicate users who have similar or common work experiences, group memberships, hobbies, educational backgrounds, or who are somehow related to or share a common attribute. The connection information may also include user-defined connections between different users and content (both internal and external).A web server may be used to link the social networking system 960 to one or more client systems 930 or one or more third party systems 970 via the network 910. The web server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 960 and the one or more client systems 930. An API request server may enable the third party system 970 to access information from the social networking system 960 by calling one or more APIs. An action logger may be used to receive communication information from the web server regarding a user's actions on or off the social networking system 960. Along with the action log, a third party content object log may be maintained of user exposure to third party content objects. A notification controller may provide information about the content objects to the client system 930. The information may be pushed to the client system 930 as a notification or the information may be pulled from the client system 930 in response to a request received from the client system 930. An authorization server may be used to enforce one or more privacy settings of a user of the social networking system 960. A user's privacy settings determine how certain information associated with the user may be shared. The authorization server may allow a user to opt-in or opt-out of having their actions recorded by the social networking system 960 or shared with other systems (e.g., third party system 970), such as by setting appropriate privacy settings. A third party content object store may be used to store content objects received from third parties, such as the third party system 970. A location store may be used to store location information received from the client system 930 associated with the user.The advertising billing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to users in the form of notifications.

[0089] FIG. 10 illustrates an exemplary computer system 1000. In certain examples, one or more computer systems 1000 perform one or more steps of one or more methods described or illustrated herein. In certain examples, one or more computer systems 1000 provide functionality described or illustrated herein. In certain examples, software operating on one or more computer systems 1000 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Certain examples include one or more portions of one or more computer systems 1000. As used herein, references to a computer system may encompass a computing device, and vice versa, where appropriate. Additionally, references to a computer system may encompass one or more computer systems, where appropriate.

[0090] This disclosure contemplates any suitable number of computer systems 1000. This disclosure contemplates computer system 1000 taking any suitable physical form. By way of example and not limitation, computer system 1000 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (e.g., a computer on module (COM) or system on module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these. Where appropriate, computer system 1000 may include one or more computer systems 1000, may be unitary or distributed, or may reside in a cloud that may span multiple locations, multiple machines, multiple data centers, or include one or more cloud components in one or more networks. Where appropriate, the one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or time limitations. By way of example and not limitation, the one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. The one or more computer systems 1000 may perform one or more steps of one or more methods described or illustrated herein at different times or in different locations, where appropriate.

[0091] In a particular example, computer system 1000 includes a processor 1002, memory 1004, storage 1006, an input / output (I / O) interface 1008, a communication interface 1010, and a bus 1012. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0092] In a particular example, the processor 1002 includes hardware for executing instructions, such as those that make up a computer program. By way of example and not limitation, to execute instructions, the processor 1002 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 1004, or storage 1006, decode and execute those instructions, and then write one or more results to an internal register, an internal cache, memory 1004, or storage 1006. In a particular example, the processor 1002 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates the processor 1002 including any suitable number of any suitable internal caches, where appropriate. By way of example and not limitation, the processor 1002 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in an instruction cache may be copies of instructions in memory 1004 or storage 1006, and the instruction cache may speed up retrieval of those instructions by the processor 1002. The data in the data cache may be a copy of data in memory 1004 or storage 1006 for instructions executing in the processor 1002 to operate on, a result of a previous instruction executed by the processor 1002 for access by subsequent instructions executing in the processor 1002 or for writing to the memory 1004 or storage 1006, or other suitable data. The data cache may speed up read or write operations by the processor 1002. The TLB may speed up virtual-address translation for the processor 1002. In a particular example, the processor 1002 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates the processor 1002 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, the processor 1002 may include one or more arithmetic logic units (ALUs), be a multi-core processor, or include one or more processors 1002.Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0093] In a particular example, the memory 1004 includes a main memory for storing instructions for the processor 1002 to execute or data for the processor 1002 to operate on. By way of example and not limitation, the computer system 1000 may load instructions into the memory 1004 from the storage 1006 or another source (such as, for example, another computer system 1000). The processor 1002 may then load the instructions from the memory 1004 into an internal register or cache. To execute instructions, the processor 1002 may retrieve instructions from the internal register or cache and decode those instructions. During or after the execution of instructions, the processor 1002 may write one or more results (which may be intermediate or final results) to an internal register or cache. The processor 1002 may then write one or more of those results to the memory 1004. In particular examples, the processor 1002 executes only instructions in one or more internal registers or internal caches, or in memory 1004 (as opposed to storage 1006 or elsewhere), and operates only on data in one or more internal registers or internal caches, or in memory 1004 (as opposed to storage 1006 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 1002 to the memory 1004. The bus 1012 may include one or more memory buses, as described below. In particular examples, one or more memory management units (MMUs) reside between the processor 1002 and the memory 1004 and enable access to the memory 1004 requested by the processor 1002. In particular examples, the memory 1004 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. This RAM may be dynamic RAM (DRAM) or static RAM (SRAM), where appropriate. Furthermore, this RAM may be single-ported or multi-ported RAM, where appropriate. This disclosure contemplates any suitable RAM. Memory 1004 may include one or more memories 1004, where appropriate.Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0094] In a particular example, storage 1006 includes mass storage for data or instructions. By way of example and not limitation, storage 1006 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Storage 1006 may include removable or non-removable (or fixed) media, as appropriate. Storage 1006 may be internal or external to computer system 1000, as appropriate. In a particular example, storage 1006 is a non-volatile solid-state memory. In a particular example, storage 1006 includes read-only memory (ROM). As appropriate, this ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. This disclosure contemplates mass storage 1006 taking any suitable physical form. Storage 1006 may include one or more storage control units that facilitate communication between processor 1002 and storage 1006, where appropriate. Storage 1006 may include one or more storages 1006, where appropriate. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0095] In particular examples, I / O interface 1008 includes hardware, software, or both that provide one or more interfaces for communication between computer system 1000 and one or more I / O devices. Computer system 1000 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices may enable communication between a person and computer system 1000. By way of example and not limitation, an I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet, a touch screen, a trackball, a video camera, another suitable I / O device, or a combination of two or more of these. An I / O device may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interfaces 1008 for those I / O devices. Where appropriate, I / O interface 1008 may include one or more device or software drivers that enable processor 1002 to drive one or more of these I / O devices. I / O interface 1008 may include one or more I / O interfaces 1008, where appropriate. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0096] In a particular example, the communication interface 1010 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication, etc.) between the computer system 1000 and one or more other computer systems 1000 or one or more networks. By way of example and not limitation, the communication interface 1010 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. The present disclosure contemplates any suitable network and any suitable communication interface 1010 for that network. By way of example and not limitation, the computer system 1000 may communicate with an ad-hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 1000 may communicate with a wireless PAN (WPAN) (e.g., a BLUETOOTH® WPAN), a WI-FI network, a WI-MAX network, a cellular network (e.g., a Global System for Mobile Communications (GSM®) network), or other suitable wireless network, or a combination of two or more of these. Computer system 1000 may include any suitable communications interface 1010 for any of these networks, as appropriate. Communications interface 1010 may include one or more communications interfaces 1010, as appropriate. Although this disclosure describes and illustrates a particular communications interface, this disclosure contemplates any suitable communications interface.

[0097] In particular examples, bus 1012 includes hardware, software, or both that couple components of computer system 1000 together. By way of example, and not limitation, bus 1012 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard (EISA) bus, a Front Side Bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus, or a combination of two or more of these. Bus 1012 may include one or more buses 1012, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0098] As used herein, the one or more computer-readable non-transitory storage media may include, where appropriate, one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. The computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

[0099] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or the context indicates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or the context indicates otherwise. Furthermore, "and" is both jointly and severally, unless expressly indicated otherwise or the context indicates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or the context indicates otherwise.

[0100] The scope of the present disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the examples and embodiments described or illustrated herein that would be understood by a person skilled in the art. The scope of the present disclosure is not limited to the embodiments described or illustrated herein. Furthermore, although the present disclosure describes and illustrates each example herein as including specific components, elements, features, functions, operations, or steps, any of these examples may include any combination or substitution of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by a person skilled in the art. Furthermore, references in the appended claims to an apparatus or system, or an apparatus or system component, that is adapted, arranged, capable, configured, enabled, operable, or operative to perform a particular function encompass that apparatus, system, component, or whether the apparatus, system, component, or particular function thereof is activated, turned on, or unlocked, so long as the apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. In addition, although the present disclosure has described or illustrated particular examples as providing certain advantages, a particular example may provide none of these advantages, or some or all of these advantages.

Claims

1. 1. A method, comprising: receiving sensor data captured by one or more sensors coupled to a user; generating an upper body pose corresponding to a first portion of the user's body, including the user's head and arms, based on the sensor data; generating a lower body pose corresponding to a second portion of the user's body, the lower body pose including the user's legs, by processing the upper body pose using a machine learning model; generating a full body pose of the user based on the upper body pose and the lower body pose.

2. 2. The method of claim 1 , wherein the machine learning model is trained using a second machine learning model trained to determine whether a given whole body pose is likely to have been generated using the machine learning model.

3. 3. The method of claim 1 or 2, wherein the machine learning model is optimized during training to cause a second machine learning model to erroneously determine that a given whole-body pose generated using the machine learning model is unlikely to have been generated using the machine learning model.

4. The machine learning model is generating a second lower body pose by processing a second upper body pose with the machine learning model; generating a second whole body pose based on the second upper body pose and the second lower body pose; using a second machine learning model to determine whether the second whole body pose was likely generated using the machine learning model; The method of claim 1 , further comprising: training the machine learning model by updating the machine learning model based on the determination of the second machine learning model.

5. The method of claim 1 , wherein the one or more sensors are associated with a head-mounted device worn by a user.

6. The step of generating an upper body posture includes: determining a plurality of poses corresponding to a plurality of predetermined body parts of a user based on the sensor data; The method of claim 1 , further comprising: inferring the upper body pose based on the plurality of poses.

7. 7. The method of claim 6, wherein the multiple poses include a wrist pose corresponding to a user's wrist, the wrist pose being determined based on one or more images captured by a head-mounted device worn by the user, the one or more images representing (1) the user's wrist, or (2) a device held by the user.

8. determining context information associated with a time at which the sensor data was captured; The method of claim 1 , wherein the lower body poses are generated by further processing the contextual information using the machine learning model.

9. The method of claim 8 , wherein the contextual information includes an application with which a user is interacting when the upper body pose is determined.

10. The method of claim 1 , further comprising generating an avatar of the user based on the whole body pose.

11. One or more computer-readable, non-transitory storage media containing instructions that, when executed by a server, cause the server to perform the method of any one of claims 1 to 10.

12. 1. A system comprising: one or more processors; 12. A system comprising: the one or more computer-readable non-transitory storage media of claim 11 coupled to the one or more processors.

Images