Concurrent human body pose estimation for virtual representations

Abstract

A method for virtually representing a human body pose includes receiving positioning data that details parameters of one or more body parts of a human user based at least in part on input from one or more sensors. One or more mapping constraints are maintained that associate the model articulation representation to the target articulation representation. A model pose of the model articulation representation and a target pose of the target articulation representation are simultaneously estimated based at least in part on the positioning data and the one or more mapping constraints. A previously trained attitude optimization machine is trained with training positioning data having a base fact tag for the model articulation representation. The target articulation representation is output for display with the target pose as a virtual representation of the human user.

Description

Background Art

[0001] Information about a human user's "pose" (e.g., the position of one or more of the user's body parts) can be mapped onto a virtual articulated representation that is rendered for display. For example, when a human user participates in a virtual environment as part of a virtual reality experience, their representation within the virtual environment can be rendered with a pose that appears similar to the user's real-world pose. The user's real-world pose can be converted to a pose of the virtual articulated representation by a previously trained model (such as a neural network), which can be trained to output a pose for the same virtual articulated representation that is ultimately rendered for display.

[0002] However, in some cases, it may be desirable to display representations of human users that differ from the representations for which the model has training data annotated with ground truth labels. For example, different users may choose to be represented in a virtual environment using different virtual representations that have different proportions and / or a different number of joints than the training representations (e.g., users may be represented in a video game by different cartoon characters with different body proportions, skeletons, and / or other aspects).

[0003] This situation can be handled by using the model to first output a pose for the training representation and then transferring the pose to the display representation. However, this can result in a less accurate reproduction of the user's real-world pose and / or increase the computational burden required to output the pose due to the need to perform additional optimization steps on the display representation. Summary of the invention

[0004] This summary is provided to introduce some concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.

[0005] A method for virtually representing a human body posture comprises: receiving positioning data, the positioning data detailing parameters for one or more body parts of a human user based at least in part on input from one or more sensors. Maintaining one or more mapping constraints that associate a model articulated representation with a target articulated representation. Simultaneously estimating a model posture of the model articulated representation and a target posture of the target articulated representation based at least in part on the positioning data and the one or more mapping constraints. A previously trained posture optimization machine is trained using training positioning data having ground truth labels for the model articulated representation. The target articulated representation is output for display as a virtual representation of the human user along with the target posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A human user is schematically illustrated being virtually represented as an articulated representation in a virtual environment.

[0007] Figure 2 An example method for virtually representing a human body pose is illustrated.

[0008] Figure 3 Receiving positioning data for one or more body parts of a human user is schematically illustrated.

[0009] Figure 4 One or more mapping constraints associated with maintaining a model articulated representation to a target articulated representation are schematically illustrated.

[0010] Figure 5A and Figure 5B Schematic illustration of a pose optimization machine that simultaneously estimates a model pose and a target pose.

[0011] Figure 6 An example computing system is schematically illustrated. DETAILED DESCRIPTION

[0012] Sometimes it is desirable to display a virtual articulated representation in a pose that appears similar to the real-world pose of a human user. Figure 1 A schematic diagram is provided showing a human user 100 in a real world environment 102. As shown, gestures of the human user are applied to an articulated representation 104 that virtually represents the human user in a virtual environment 106. In other words, as the human user moves parts of his body in the real world environment, the user's movements are translated into corresponding movements of the articulated representation 104 in the virtual environment 106. For example, the articulated representation may be an avatar of the human user 100 in a virtual reality video game, teleconferencing, and / or other virtual reality experiences. By mapping the user's real world movements to their virtual articulated representation, the immersion of the human user and any other participants in the virtual experience may be improved.

[0013] exist Figure 1 , the articulated representation 104 and the virtual environment 106 are presented on an electronic display device 108. The electronic display device can have any suitable form factor and utilize any suitable underlying display technology. For example, the electronic display device can be a near-eye display device (e.g., integrated into a head-mounted display), a television, a computer monitor, a smartphone, a tablet computer, a laptop computer display, a projector system, etc. It is worth noting that, as will be described in detail below, the articulated representation does not necessarily have to be presented by the same device that estimates the pose of the representation based on the real-world pose of a human user.

[0014] In some cases, the pose of the virtual articulated representation may be output by a previously trained model, which, as a non-limiting example, may include a neural network. As discussed above, in some cases, the model may be trained using training data having ground truth labels for the same virtual articulated representation displayed in a virtual environment. For example, the training data may include positioning data for a body part of a human training subject (e.g., rotation parameters corresponding to the subject's joints), labeled with ground truth labels, the positioning data being associated with the pose of the corresponding joint of the virtual articulated representation, the pose of the corresponding joint of the virtual articulated representation matching the subject's real-world pose. Based on such training data, the model may output a pose for the virtual articulated representation for any suitable set of input positioning data.

[0015] However, in some cases, the virtual articulated representation displayed may differ from the representation used to train the model. To address this issue, the pose estimated for the trained articulated representation may be transferred to the displayed articulated representation, although this may result in a less accurate representation of the real-world pose of a human user unless computational resources are used to supplement the optimization step. In another approach, the pose for the displayed representation may be output directly without the benefit of labeled training data for the trained representation, which limits the possible poses, which may result in a less accurate reproduction of the real-world pose of a human user.

[0016] Thus, the present disclosure relates to techniques for simultaneously estimating pose for both a model articulated representation (e.g., a representation that a model is trained to output a pose estimate) and a target articulated representation (e.g., an articulated representation that will be displayed as a virtual representation of a human user). More specifically, according to the techniques described herein, a computing system receives positioning data that details parameters for one or more body parts of a human user based at least in part on input from one or more sensors. For example, this can include output from an inertial measurement unit (IMU) of a head mounted display device (HMD) worn on the user's head, and / or output from a suitable camera (e.g., an HMD camera configured to image the user's hands). The system also maintains one or more mapping constraints that associate the model articulated representation to the target articulated representation—e.g., for a joint of the target articulated representation, a joint mapping constraint that specifies mapping of one or more joints of the model articulated representation to a target joint. Based at least in part on the positioning data and the mapping constraints, a pose optimization machine simultaneously estimates a model pose for the model articulated representation and a target pose for the target articulated representation. Once estimated, the target articulated representation may be displayed along with the target pose as a virtual representation of the human user.

[0017] The posture optimization machine can be trained using training positioning data with ground truth labels for model articulated representations. However, in some cases, the training positioning data may lack ground truth labels for target articulated representations. In this way, advantageously, the technology in this article can allow accurate reproduction of the real-world posture of human users without the need for computationally expensive training for each different possible target articulated representation. For example, when participating in a virtual environment, users may be able to choose from a variety of different avatars used to represent them in the virtual environment, and in some cases, users may have the ability to dynamically change their appearance during a session - for example, from one target articulated representation to another target articulated representation. New target articulated representations can be added to the menu of representations available to the user without the computational cost of retraining the model for that particular representation.

[0018] The techniques described herein may provide technical advantages by reducing the consumption of computational resources while accurately reproducing the real-world pose of a human user, and allowing the accurate pose to be applied to any of a plurality of different target articulated representations. This is achieved by simultaneously estimating the target pose and the model pose, whereas other schemes may perform additional pose transfer and optimization steps. Furthermore, the techniques described herein are associated with specific mathematical operations that provide the technical purpose of simultaneously estimating two different poses corresponding to two different articulated representations based on positioning data and mapping constraints.

[0019] Figure 2 An example method 200 for virtually representing a human body posture is illustrated. Method 200 may be implemented by any suitable computing system of one or more computing devices. Any computing device implementing method 200 may have any suitable functionality, form factor, and hardware configuration. Where two or more different devices implement the steps of method 200, they may in some cases communicate via a suitable computer network—for example, using a client / server model. In some cases, method 200 may be implemented by any suitable computing system of one or more computing devices. Any computing device implementing method 200 may have any suitable functionality, form factor, and hardware configuration. Where two or more different devices implement the steps of method 200, they may in some cases communicate via a suitable computer network—for example, using a client / server model. Figure 6 The described computing system 600 is implemented.

[0020] At 202, method 200 includes receiving positioning data that specifies parameters for one or more body parts of a human user based at least in part on input from one or more sensors. Figure 3 This is schematically illustrated, again showing a human user 100. In some cases, one or more body parts of the human user may include a head of the human user, such as head 300 of human user 100. In some examples, one or more body parts of the human user may further include one or both hands of the human user, such as hands 302A and 302B of human user 100. In general, the system may derive positioning data for any or all of the different body parts (e.g., head, arms, hands, fingers, legs, feet) of the human user.

[0021] As used herein, "positioning data" includes any suitable computer data that specifies or can be used to estimate the position, orientation, and / or movement of a body part of a human user in a real-world environment. For example, the positioning data may include or be derived from the output of one or more position sensors (such as an IMU), which may be integrated into one or more devices and held or worn by the corresponding body parts of the human user. As another example, the positioning data may include or be derived from image or video data output by a suitable camera, which is configured to image visible portions of one or more body parts of a human user in a real-world environment.

[0022] As used herein, positioning data details "parameters" for one or more body parts of a human user. This means that the positioning data includes information that specifies or can be used to derive the position, orientation and / or movement of the human body part. For example, where a human hand is visible in an image captured by a camera, in some cases the image data may be processed to estimate the position of the human hand relative to the real world environment and / or relative to the camera or another reference frame. Thus, in this case, positioning data is derived from the output of a sensor (e.g., a camera), and the "parameters" detailed by the positioning data are estimates of the position of the human hand.

[0023] However, as another example, "positioning data" may refer to the raw output of one or more sensors - for example, in some cases, a pose optimization machine may be trained to estimate the pose of a virtual articulated representation based on the raw sensor output. For example, a pose optimization machine may receive image data from a camera as input, and may have been previously trained to output a pose based at least in part on the image pixel data. Thus, the "parameters" detailed by the positioning data include pixels depicting a person's hand, which pixels may be used to estimate the position of the hand.

[0024] Figure 3 Schematically illustrates different types of sensors, wherein the output from the sensors may include or be used to derive the positioning data. Figure 3 In the method 200, a human user 100 wears a head mounted display device (HMD) 304 on their head 300, wherein the HMD 304 may be configured to perform one or more steps of the method 200. For example, the HMD may include an IMU configured to detect changes in the position and / or orientation of the HMD (e.g., via a suitable combination of an accelerometer, a gyroscope, and / or a magnetometer). In this way, changes in position / orientation detected by an IMU integrated into the HMD may be interpreted as changes in the position / orientation of the user's head - e.g., positioning data for the head can be derived from the output of the HMD's integrated IMU.

[0025] In addition, Figure 3 In the embodiment of the present invention, human user 100 wears position sensors 306A and 306B, which can be configured to detect the movement of the user's hands and report it to HMD 304, and / or another computing system configured to receive the positioning data. For example, the wearable position sensors can include respective IMUs and / or other suitable position sensors for detecting the movement of the user's hands.

[0026] In some cases, the positioning data may include, or be derived from, the output of a handheld position sensor in addition to or in lieu of wearable position sensors 306A and 306B. For example, a user may hold one or more handheld game controllers, each of which includes an internal position sensor.

[0027] As another example, the positioning data may include or be derived from the output of a camera configured to image one or more body parts of a human user. Figure 3 In FIG. 3 , HMD 304 includes an onboard camera 308, which is represented by a white circle. The arrow extending away from the white circle represents the field of view (FOV) of the camera within the real world environment. Figure 3 In the depicted scene, a human hand 302B will be visible in an image of the real world environment captured by camera 308. The computing system can be configured to identify pixels of the hand in the captured image (e.g., via an estimator trained with suitable machine learning configured to identify human hands in image data), and estimate the position of the hand based on the size and position of the hand within the captured image. Alternatively, as discussed above, the output from the camera can be provided directly to a pose optimization machine configured to generate a pose for the virtual articulated representation.

[0028] It will be appreciated that at any given time, either, both, or none of the user's hands may be visible in the FOV of camera 308. Thus, in some cases, the number of different body parts for which positioning data is received and used to output a pose for a virtual articulated representation may vary over time—e.g., at some points, positioning data is received for both hands, while at other points, positioning data is not received for either hand.

[0029] Additionally, in some cases, the output from camera 308 can be used to detect changes in the position and / or orientation of the camera, and thereby detect changes in the head of a human user (in the case where the camera is integrated into an HMD worn on the user's head). For example, the computing system can identify one or more landmarks in an image of the real-world environment captured by camera 308. As the position and / or orientation of the camera changes, the image space size and / or position of the identified landmarks will change. By detecting and analyzing such changes, movement of the camera (and thereby the user's head) can be estimated.

[0030] In some cases, positioning data may be received from and / or derived from the output of one or more cameras in addition to or in lieu of cameras integrated into the HMD. Figure 3, an external camera 310 is configured to image a portion of a real-world environment in which a human user 100 is visible. The FOV of the camera 310 is indicated by the dashed arrow extending away from the camera. The position of any or all body parts of the human user (e.g., head 300, hands 306A / 306B) can be detected in the images captured by the camera 310. Thus, the output of the camera 310 can include positioning data for any or all body parts of the human user, and / or the output from the camera can be used to derive positioning data.

[0031] In some cases, one or both of cameras 308 and 310 may be implemented as depth cameras. For example, the camera(s) may be configured to emit structured light patterns, or to calculate the time of flight for photons to leave a light emitter, reflect from an object in the environment, and hit the camera's image sensor. This may enable the camera(s) to more accurately detect the position of a human user's body part relative to the camera and / or the surrounding environment. The cameras may be sensitive to any suitable wavelength of light - e.g., visible light and / or infrared light.

[0032] It will be understood that the specific sensors described above are not limited and do not all have to be used together. For example, an implementation may receive output from an IMU and camera integrated into the HMD, or derive positioning data from it, without using an external camera or any wearable / handheld position sensor. In addition, it will be understood that positioning data may be received in or derived from the output of one or more sensors not explicitly described herein. More specifically, the present disclosure generally assumes that appropriate positioning data is received, and does not consider the specific sources and processing that may be applied to such positioning data.

[0033] Briefly return to Figure 2 , at 204, method 200 includes maintaining one or more mapping constraints that associate a model articulated representation to a target articulated representation. As discussed above, a "model" articulated representation refers to a representation that a pose optimization machine has previously trained using training data with ground truth labels. In this way, when provided with a suitable set of positioning data at runtime, the pose optimization machine can output a pose for the model articulated representation that is similar to a real-world pose of a human user referred to by the positioning data. The model articulated representation is associated to a "target" articulated representation via one or more mapping constraints, and the "target" articulated representation is intended to be displayed as part of a virtual experience.

[0034] against Figure 4This is illustrated schematically, which shows an example model articulated representation 400. It will be appreciated that the model articulated representation 400 is highly simplified and is provided for illustration purposes only. Generally speaking, the model articulated representation will take the form of a digital representation of a human body, wherein different parts of the representation are coupled via multiple joints. However, the scale of the representation and the specific number of limbs, joints and other movable body parts may vary depending on the implementation. As a non-limiting example, the model articulated representation can be implemented as an SMPL (skinned multi-person linear model) representation.

[0035] exist Figure 4 In the example of Figure 1 As discussed above, the target articulated representation is rendered for display in the virtual environment and can be displayed together with the target pose output by the pose optimization machine. However, it will be understood that the target articulated representation 104 is a non-limiting example and the techniques described herein can be applied to any of a variety of different suitable representations of a human user.

[0036] For example, the target articulated representation can have any suitable appearance and proportions, and can have any suitable number of limbs, joints, and / or other movable body parts. Although the target articulated representation may generally be humanoid in appearance (e.g., because it represents a real-world human user), this need not always be the case—for example, the target articulated representation may resemble a non-human animal, a fictional character, or nearly any other suitable avatar with multiple movable body parts. In addition, the target articulated representation may differ from the model articulated representation in any number of different ways. For example, the model articulated representation may have a different number of joints, a different number of limbs (and / or other movable body parts), different proportions, a different appearance, etc. than the target articulated representation.

[0037] The model articulated representation and the target articulated representation are associated by one or more mapping constraints 402. As will be described in detail below, the pose optimization machine estimates poses for both the model and target articulated representations simultaneously by finding poses that are consistent with the positioning data and also satisfy the mapping constraints. Thus, the form of a "mapping constraint" may be any suitable function or relationship that affects the range of possible poses of the target articulated representation based on some property of the model articulated representation.

[0038] As a non-limiting example, one or more mapping constraints may include a joint mapping constraint 404. The joint mapping constraint specifies a set of one or more joints in the model articulated representation that are mapped to the joints of the target articulated representation for the joints of the target articulated representation. For example, the model articulated representation 400 includes multiple joints, two of which are labeled 403A and 403B, which correspond to the shoulder joint and the elbow joint. The target articulated representation 104 includes similar joints 405A and 405B. Therefore, multiple mapping constraints may include different joint mapping constraints for the joints 405A and 405B of the target representation, indicating that such joints are mapped to the joints 403A and 403B of the model representation.

[0039] In the above examples, there is a one-to-one correspondence between the joints in the target articulated representation and the joints in the model articulated representation. However, it will be appreciated that this is not always the case. More specifically, in some examples, two or more different joints in the model articulated representation may be mapped to a single joint in the target articulated representation. For example, the target articulated representation may include fewer spinal joints than the model articulated representation. Therefore, a joint mapping constraint may specify that two or more different spinal joints in the model representation affect one spinal joint in the target representation.

[0040] In some cases, the joint mapping constraint may further specify a weight for each of the joints in the model articulated representation that is mapped to a joint of the target articulated representation. For example, when only one joint in the model articulated representation is mapped to a particular joint of the target articulated representation, the weight of the model joint may be 100%. When two model joints are mapped to the target joint, the weights of the two model joints may be 50% and 50%, 30% and 70%, 10% and 90%, etc. It will be understood that the joints in the target articulated representation may be mapped to any number of different joints in the model articulated representation, and each of such joints may have any suitable weight.

[0041] In order to describe the joint mapping constraints mathematically, let the model articulation representation include a set of joints And the target articulated representation includes a set of joints The i-th joint of the articulated representation of the model is And all The kth joint of the target articulated representation is And all Joint mapping constraints can be described as moving from the destination rig Joints in To source rack An ordered list of joints in Each joint has a scalar weight α i :

[0042] in

[0043] As another example, one or more mapping constraints may include a root relative correspondence constraint 406. A root relative correspondence constraint may specify that a rotation of a model joint applied to the model articulated representation relative to a root of the model articulated representation in the model pose has at least a threshold similarity to a rotation of a corresponding target joint of the target articulated representation relative to a root of the target articulated representation in the target pose.

[0044] In some cases, the pose of an articulated representation can be expressed as rotations of multiple joints of the representation expressed in a joint-local format, referring to the rotations of the joints relative to the local coordinate system of the joints. In contrast, "rig-space" rotations can be defined by the rotations and translations of individual joints relative to the root of the articulated representation. The root of the articulated representation stores the global position of the articulated representation. In SMPL representation In the specific example of is the root. In many cases, the target articulated representation will also use the pelvis joint as the root, although this is not necessarily always the case. When the root of the target articulated representation is defined at a different joint than the model articulated representation, a re-rooting process can be applied to traverse the kinematic tree of the target articulated representation. This can be implemented to derive an alternative version of the target articulated representation that uses the same root as the model articulated representation.

[0045] In any case, the root-relative correspondence constraints can cause the model and target poses output for the model and target articulated representations to appear substantially similar. For example, when the model articulated representation has a pose with a hand in front of the representation's head (e.g., covering the representation's face), the pose can be expressed as rotations of multiple joints of the model representation relative to the root (e.g., pelvis) of the model representation. A substantially similar pose can be achieved when similar rotations are applied to corresponding joints of the target articulated representation relative to the root of the target representation. When rotations are expressed using an axis-angle (e.g., yaw, pitch, roll) convention, the root-relative rotation for a given joint can be referred to as a local rotation, or "local theta." Thus, local rotations for non-root joints can be expressed as in, It's the root joint.

[0046] Thus, a root-relative correspondence constraint may specify that the local rotation applied to any given joint in the target articulated representation should have at least a threshold similarity to the local rotation applied to one or more joints in the model articulated representation that are mapped to the target joint. In the case where two or more model joints are mapped to a target joint, a weighted average of the local rotations may be used. The root-relative correspondence constraint may specify any suitable threshold - for example, a threshold of 10% may be used. It is noteworthy that in some cases, it may not be desirable to apply exactly the same local rotations to corresponding joints in the model and target representations - for example, situations such as where different scales or limb lengths between the two representations would result in unrealistic or unnatural poses (such as where two hands intersect each other).

[0047] As another example, the one or more mapping constraints may include a default pose mapping constraint that specifies that one or more rotations are to be applied to one or more joints of the target articulated representation to match the default pose of the model articulated representation. This approach may be used where the model and target articulated representations have different default poses (also referred to as "bind" poses or "rest" poses). Figure 4 In this example of the present invention, the model articulated representation 400 is shown in a default pose 407A, sometimes referred to as an "A-pose," in which the arms of the representation are tilted downward and the hands are at a similar height to the pelvis. In contrast, the target articulated representation 104 is shown in a default pose 407B, referred to as a "T-pose," in which the arms are extended perpendicular to the torso. It will be understood that these default poses are non-limiting examples, and each of the model and target articulated representations may use any suitable default pose.

[0048] The default pose mapping constraint can be expressed mathematically as a function that maps the default pose of each joint to the default pose in the joint local space. Apply Rotation R i In this way, the new pose of the target articulated representation may be substantially similar to the default pose of the model articulated representation. This may be referred to as the adjusted binding pose of the target articulated representation.

[0049]

[0050] As another example, one or more mapping constraints may include a pose continuity constraint that specifies frame-to-frame variation limits for one or more joints of the model articulated representation and one or more joints of the target articulated representation. It is noteworthy that, as will be described in detail below, in some cases the model pose and target pose for the model and target articulated representations may be estimated at least in part based on one or more previous model poses and previous target poses estimated at one or more previous time frames. Thus, the pose continuity constraint may limit the extent to which the state of a joint (e.g., a local rotation of a joint) can change from one time frame to another.

[0051] The specific frame-to-frame variation limit used may vary from one implementation to another, and will generally depend on the frame rate at which the new pose is estimated. Any suitable frame rate may be used—as a non-limiting example, poses may be estimated at a rate of 30 frames per second (FPS) or 60 FPS. Similarly, as an example, a frame-to-frame variation limit may specify that the local rotation of a joint should not vary by more than 10° from one frame to another.

[0052] It will be appreciated that the specific mapping constraints described above are non-limiting examples. In general, the set of mapping constraints used to estimate the model and target poses need not include each of the constraints described above, and in some cases, may include one or more mapping constraints in addition to those described above. More specifically, the form of a "mapping constraint" may be any suitable function or relationship that affects the range of possible poses of the target articulated representation based on some characteristic of the model articulated representation.

[0053] In view of the above mapping constraints, a non-limiting example algorithm is provided that can be used to apply a given pose to a target articulated representation starting from its default pose. Specifically, a mapping from joint-local space to frame-space can be determined for the target articulated representation in an adjusted bind pose: For simplicity, It is represented by j dst ,and It is represented by j src .

[0054] for J in dst :

[0055]

[0056] GetAnimPose(j src ) provides the model representation of joint j in the form of angle-axis srcThe local rotation function of the joint, where the axis of rotation is represented by the normalized 3D vector Specify , and the amount of rotation is expressed as an angle θ∈[-π,π].

[0057] The local rotation of the joint in the frame space is expressed in the model representation (e.g., SMPL representation) as the local transformation R of the frame src The weighting (through the weight α i ) contributions (e.g., rotations expressed relative to the root of a joint representation).

[0058] Mapping ExpMapToMatrix(R′) converts the rotation from exponential map form Function that maps to a rotation matrix. The exponential form of a rotation is the unit length rotation axis times the rotation angle. In this form, it is easier to interpolate between rotations.

[0059] For servings: The algorithm first uses Q jntLocal→rig (j dst ) maps from the joint local space to the rack space, applying the transformation R in the rack local space src , using Q rig→jntLocal (j dst ) from the frame space back to the joint local space, and then, using Map the destination rack to the adjusted bind pose.

[0060] SetPose(j dst ,R dst ) Set the destination joint j dst The joint-local pose is set to rotate R dst .

[0061] The above algorithm may be implemented using any suitable alternative representation of rotations other than matrices (e.g., quaternions) - it depends on whether the chosen representation is more suitable for the implementation framework.

[0062] It is worth noting that the above algorithms only focus on applying pose to the target articulated representation. Therefore, we briefly return to Figure 2 At 206, method 200 includes simultaneously estimating a model pose of the model articulated representation and a target pose of the target articulated representation via a previously trained pose optimization machine. Based at least in part on the positioning data (e.g., as described above with respect to Figure 3 ) and mapping constraints (e.g., as described above with respect to Figure 4 described) to estimate the model pose and target pose.

[0063] against Figure 5A and Figure 5B This is schematically illustrated. Specifically, Figure 5A An example posture optimization machine 500 is schematically shown, which can be implemented as any suitable combination of computer logic components. As a non-limiting example, the posture optimization machine 500 can be as follows for Figure 6 The described logic subsystem 602 is implemented.

[0064] As in Figure 5A As shown in , the posture optimization machine estimates the model posture 502A for the model articulated representation and the target posture 502B for the target articulated representation simultaneously. This is done at least in part based on positioning data 504 and one or more mapping constraints 506. In some cases, the posture estimation can be done at least in part based on one or more previous model postures and the previous target posture estimated on one or more previous time frames. Therefore, the posture optimization machine 500 also stores a plurality of previous postures 506, which can be expressed as a plurality of local rotations of the joints of each model. As discussed above, in some cases, one or more mapping constraints can include a posture continuity constraint, which imposes a frame-to-frame restriction that constrains the degree to which the local rotation of a given joint can change from one frame to another.

[0065] Figure 5B Schematic diagram illustrating the application of the estimated model and target poses to the model and target articulated representations. Specifically, Figure 5B The model articulated representation 400 and the target articulated representation 104 are again depicted in terms of their corresponding default poses 407A and 407B. Then, the poses of the articulated redirection are changed so that the model articulated representation 400 adopts the model pose 502A and the target articulated representation 104 adopts the target pose 502B. It is worth noting that each of these poses is similar to Figure 3 The real-world posture of human users 100 is similar.

[0066] It will be appreciated that the model pose and the target pose are estimated simultaneously. In other words, compared to the alternative, the pose optimization machine does not first output the model pose for the model articulated representation and then transfer the pose to the target articulated representation. More precisely, pose estimation can be described as finding two different poses that satisfy a set of constraints for two different representations at the same time. For example, the pose of the model articulated representation may be constrained by the previous training of the pose optimization machine so as to output possible human poses given a set of positioning data inputs, while the pose of the target articulated representation may be constrained by one or more mapping constraints that associate the target articulated representation to the model articulated representation.

[0067] In addition, in some cases, pose estimation can be performed by a previously trained machine learning (ML) model 508 implemented by a pose optimization machine. As discussed above, the previously trained pose optimization machine is trained using positioning data with ground truth labels for the model's articulated representation. For example, the pose optimization machine can implement a previously trained neural network or other suitable model by providing training positioning data that specifies rotational parameters for one or more joints of a training object, while the ground truth labels can specify a pose of the model's articulated representation that will be consistent with the real-world pose of the training object. Further details on example machine learning (ML) and / or artificial intelligence (AI) techniques that can be used to implement the techniques described herein are provided below with respect to Figure 6 Given.

[0068] The posture optimization machine may be configured in some cases to output a posture based on sparse input positioning data. In other words, the posture optimization machine may be trained to output a posture estimate based on more input parameters than the posture optimization machine typically receives at runtime. For example, the posture optimization machine may be trained to receive inputs corresponding to the user's head, arms, hands, torso, legs, feet, and / or other joints (or movable body parts) of the training subject. As training proceeds, one or more of these inputs may be selectively masked to improve the robustness of the model when fewer inputs are provided. At runtime, the posture optimization machine may in some cases only receive input data corresponding to the user's head, and optionally input data for one or two hands (e.g., when the hands happen to be visible in the camera FOV).

[0069] In other words, the positioning data provided to the pose optimization machine may include rotation parameters for n joints of a human user, while the pose optimization machine is previously trained to receive rotation parameters for n+m joints as input, where m≥ 1. Estimating the model pose may then include estimating rotation parameters for n+m model joints of the model articulated representation based at least on the rotation parameters for the n joints, and in the absence of rotation parameters for the m joints.

[0070] In addition, in some cases, the training positioning data for the posture optimization machine may not include the ground truth label for the target articulated representation. More specifically, as discussed above, the target articulated representation is related to the model articulated representation via one or more mapping constraints, which usually constrains the target posture to be substantially similar to the model posture. Since the training positioning data has the ground truth label for the model articulated representation, the model posture output at runtime can be limited to a relatively "possible" posture for a human user, and the posture can be related to the target posture via one or more suitable constraints. This provides flexibility for the type of target articulated representation that can be used without compromising the fidelity of the posture reproduction. For example, advantageously, the technology described in this article can enable users to select their desired target representation from a variety of options when joining a virtual experience, or even change their target representation during a virtual session.

[0071] An example implementation is now described to illustrate the various operations performed by the pose optimization machine to estimate the pose of the model and the target simultaneously. First, for optimizing rotations: For a model articulated representation, given R in axis-angle format src The derivative of src(AA) , R in axis-angle format can be calculated from quaternions by applying the chain rule dst The derivative of dst(AA) , to determine R dst(AA) How to respond to R src(AA) changes with the changes of.

[0072]

[0073] Translations can be transferred intact, as they can be optimized later using inverse kinematics constraints. Root rotations and positions can also be transferred intact from the model articulated representation to the target articulated representation.

[0074] Let θ dst corresponds to R dst The local rotation of the joint, and θ src corresponds to R srclocal rotations of joints. For each frame, given a set of input localization data, the goal is to predict a new model and target pose that is consistent with the localization data and (if available) the previous pose and localization data. To this end, the pose optimization machine can implement the following process: (1) Compute the local to world transformation of each bone / joint in the kinematic tree of the model representation, given the root position and rotation of the model representation and the local rotations of the joints of the model representation. (2) Compute the derivatives of the target pose parameters with respect to the source pose parameters (also known as "retargeting"). (3) Compute the local to world transformation of the target articulated representation. (4) Optimize the model and target pose parameters to minimize the error in the position of one or more body parts (e.g., head and hands) of the target representation. Step (4) can be accomplished at least in part based on knowledge of one or previous model / target poses and the transfer of derivatives between the model and target articulated representations, while satisfying one or more continuity constraints (e.g., frame-to-frame pose continuity, gravity continuity, trajectory continuity).

[0075] In some cases, the optimization may be performed by simultaneously minimizing the energy for the model and target articulated representations. Thus, the model and target poses estimated by the pose optimization machine may be those that minimize the associated energies. For example, as a non-limiting example, the optimization may be performed via a least squares optimization method.

[0076] For a particular implementation where the positioning data includes the output of a position sensor corresponding to the user's head, let L = Loss, λ = the weight associated with the loss term, T Head = as the prediction transformation of the SE3 matrix for the head sensor, The SE3 transform of the internal sensor expresses the input sensor measurement, and D SE3 [T1, T2] = L2 distance between two SE3 transformations T1 and T2.

[0077]

[0078] Here, the term group L i Indicates that in θ dst The goal is to minimize the distance between the observations from the input sensors and the model predictions for the target articulated representation. src The term L defined above prior is included as an example of a regularizer that encodes prior knowledge about the distribution of plausible human poses.

[0079] The optimization problem can be rewritten as:

[0080]

[0081] In this formula, the function θ dst =ρ(θ src ) represents the redirected mapping from the model articulated representation to the target articulated representation.

[0082] In this example, the three data items correspond to three position sensors for the user's head and hands, and a regularizer is used to encode prior knowledge of plausible human poses:

[0083]

[0084] In general, any appropriate number of additional data items (e.g., joint rotations) and / or additional regularizers (e.g., corresponding to gravity constraints, trajectory constraints, range of motion constraints), etc., may be added depending on the implementation.

[0085] The following is a non-limiting example implementation of redirecting pose derivatives via the chain rule:

[0086]

[0087] This involves a triple matrix multiplication of dimensions (3D×4D)×(4D×4S)×(4S×3S) of the target articulated representation consisting of D joints and the model articulated representation consisting of S joints. In some cases, the matrices involved may be relatively sparse (e.g., with a sparsity factor of ~2%), although the exact sparsity may depend on the joint mapping between the two articulated representations.

[0088] However, in some cases, considering sparsity alone may not be sufficient for real-time implementation because the processing may not be fast enough. Therefore, the techniques described in this article can be effectively implemented by: (1) exploiting the problem structure (e.g., the lower / upper triangular structure of the matrix), (2) optimizing the computational scheduling by observing that the matrix has a 3×4 or 4×3 or 4×4 block sparse structure, enabling the use of specialized compressed storage for block sparse matrices, and (3) optimizing data access by performing block-local matrix multiplication through specialized mathematical functions. This further exploits the known problem structure and optimizes intermediate storage by anticipating future access patterns. For example, the result of (4D×4S)×(4S×3S) can be written row by row, but read column by column.

[0089] Utilizing the above techniques, advantageously, the speed of processing can be increased by two orders of magnitude. This enables real-time concurrent estimation of model and target poses without the need to use low-level native code or dedicated hardware acceleration. However, it will be appreciated that the specific algorithms and calculations described above are non-limiting examples. In other examples, the pose optimization machine can estimate the model and target poses simultaneously by other suitable means.

[0090] Briefly return to Figure 2 At 208, method 200 includes outputting the target articulated representation and the target pose as a virtual representation of the human user for display. Figure 1 In FIG. 1 , the target articulated representation 104 is presented by the electronic display device 108. Figure 1 As discussed, the display device presenting the target articulated representation may take any suitable form and may use any suitable underlying display technology. In some examples, the target articulated representation may be displayed by a near-eye display of an HMD—for example, for use by a different user who is also participating in the virtual experience.

[0091] In some cases, "outputting" the target articulated representation for display may include rendering the target articulated representation and the target pose as an image frame. The image frame may then be sent to a suitable display device - for example, via a computer network such as the Internet. In other examples, the computing system need not render the target articulated representation as an image. Instead, after estimating the target pose, parameters for the target pose (e.g., local rotations of the joints of the target pose) may be transmitted to a separate computing device, which may be configured to apply the parameters to its own local instance of the target articulated representation and then render the target articulated representation for display.

[0092] The methods and processes described herein may be connected to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as an executable computer application, a network accessible computing service, an application program interface (API), a library, or a combination of the above content and / or other computing resources.

[0093] Figure 6 Schematically illustrated is a simplified representation of a computing system 600 that is configured to provide any or all of the computing functionality described herein. Computing system 600 may take the form of one or more personal computers, network accessible server computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), virtual / augmented / mixed reality computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, and / or other computing devices.

[0094] The computing system 600 includes a logic subsystem 602 and a storage subsystem 604. The computing system 600 may optionally include a display subsystem 606, an input subsystem 608, a communication subsystem 610, and / or Figure 6 Other subsystems not shown.

[0095] The logic subsystem 602 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions as part of one or more applications, services, or other logical structures. The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processor of the logic subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The individual components of the logic subsystem may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Aspects of the logic subsystem may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0096] The storage subsystem 604 includes one or more physical devices configured to temporarily and / or permanently store computer information, such as data and instructions executable by the logic subsystem. When the storage subsystem includes two or more devices, these devices can be spliced ​​and / or remotely located. The storage subsystem 604 can include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable and / or content addressable devices. The storage subsystem 604 can include removable and / or built-in devices. When the logic subsystem executes instructions, the state of the storage subsystem 604 may be transformed-for example, to store different data.

[0097] Aspects of logic subsystem 602 and storage subsystem 604 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOC), and complex programmable logic devices (CPLD).

[0098] The logic subsystem and the storage subsystem can collaborate to instantiate one or more logic machines. As used herein, the term "machine" is used to collectively refer to a combination of hardware, firmware, software, instructions, and / or any other components that collaborate to provide computer functions. In other words, a "machine" is never an abstract concept, but always has a tangible form. A machine can be instantiated by a single computing device, or a machine can include two or more subcomponents instantiated by two or more different computing devices. In some implementations, a machine includes a local component (e.g., a software application executed by a computer processor) that collaborates with a remote component (e.g., a cloud computing service provided by a server computer network). The software and / or other instructions that give a particular machine its functionality can optionally be stored on one or more suitable storage devices as one or more unexecuted modules.

[0099] The machine may be implemented using any appropriate combination of state-of-the-art and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) techniques. Non-limiting examples of techniques that may be incorporated into implementations of one or more machines include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and aggregate features across one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI smart knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, component and / or dependency parsing, and / or intent recognition, segmentation models, and / or super-segmentation models (e.g., hidden dynamic models)).

[0100] In some examples, the methods and processes described herein may be implemented using one or more differentiable functions, where the gradient of the differentiable function may be calculated and / or estimated based on the inputs and / or outputs of the differentiable function (e.g., based on training data and / or based on an objective function). Such methods and processes may be determined, at least in part, by a set of trainable parameters. Thus, the trainable parameters for a particular method or process may be adjusted by any suitable training procedure to continually improve the functionality of the method or process.

[0101] Non-limiting examples of training procedures for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning), and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual bandit methods, maximum likelihood methods, and / or expectation maximization. In some examples, multiple methods, processes, and / or components of the systems described herein can be trained simultaneously for an objective function that measures the collective functional performance of multiple components (e.g., for reinforced feedback and / or for labeled training data). Training multiple methods, processes, and / or components simultaneously can improve such collective functions. In some examples, one or more methods, processes, and / or components can be trained independently of other components (e.g., offline training on historical data).

[0102] When display subsystem 606 is included, display subsystem 606 can be used to present a visual representation of the data stored by storage subsystem 604. This visual representation can take the form of a graphical user interface (GUI). Display subsystem 606 can include one or more display devices using almost any type of technology. In some implementations, the display subsystem can include one or more virtual reality, augmented reality, or mixed reality displays.

[0103] When an input subsystem 608 is included, the input subsystem 608 may include or interface with one or more input devices. An input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or may be peripheral, and the conversion and / or processing of input actions may be performed on or off the board. Example NUI components may include microphones for voice and / or sound recognition; infrared, color, stereo and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers and / or gyroscopes for motion detection and / or intent recognition.

[0104] When a communication subsystem 610 is included, the communication subsystem 610 can be configured to communicatively connect the computing system 600 to one or more other computing devices. The communication subsystem 610 can include wired and / or wireless communication devices compatible with one or more different communication protocols. The communication subsystem can be configured to communicate via a personal, local area network, and / or wide area network.

[0105] The methods and processes disclosed herein can be configured to enable users and / or any other person to control any private and / or potentially sensitive data. Whenever data is stored, accessed and / or processed, it must be processed in accordance with privacy and / or security standards. When collecting user data, users or other stakeholders can specify how the data is used and / or stored. Regardless of the purpose for which user data is collected, user data must be collected with the greatest respect for user privacy (for example, user data can only be collected with the explicit consent of the user who owns the data, and / or the user who owns the data is notified whenever the user data is collected). If the data is to be published for access by anyone other than the user or for any decision-making process, the user's consent can be obtained before the data is used and / or published. Users can choose to join and / or exit data collection at any time. After the data has been collected, the user can issue a command to delete the data and / or restrict access to the data. To further protect user privacy, all potentially sensitive data can be optionally encrypted and / or anonymized when feasible. Users can specify parts of the data, metadata, or statistics / results of processing data to be published to other parties, for example, for further processing. Private and / or confidential data may be kept completely confidential, e.g., only temporarily decrypted for processing, or only decrypted when processed on a user device, and otherwise stored in encrypted form. The user may hold and control the encryption keys for the encrypted data. Alternatively or in addition, the user may designate a trusted third party to hold and control the encryption keys for the encrypted data, e.g., to provide the user with access to the data in accordance with an appropriate authentication protocol.

[0106] When the methods and processes described herein include ML and / or AI components, the ML and / or AI components can make decisions based at least in part on the training of the components with respect to the training data. Thus, the ML and / or AI components can be trained on different, representative data sets that include sufficient relevant data for different users and / or user groups. Specifically, the training data sets are inclusive with respect to different human individuals and groups, so that as the ML and / or AI components are trained, their performance with respect to the user experience of the users and / or user groups is improved.

[0107] Additionally, ML and / or AI components can be trained to make decisions so as to minimize potential bias against human individuals and / or groups. For example, when AI systems are used to evaluate any qualitative and / or quantitative information about human individuals or groups, they can be trained to be invariant to differences between individuals or groups that are not intended to be measured by the qualitative and / or quantitative assessments, for example, so that any decision is not affected in unexpected ways by differences between individuals and groups.

[0108] ML and / or AI components may be designed to provide context for how they operate so that implementers of the ML and / or AI systems can consider decisions / evaluations made by the systems. For example, ML and / or AI systems may be configured for replicable behavior, e.g., when they make pseudo-random decisions, a random seed may be used and recorded so that the decision can be replicated later. As another example, data used to train and / or test ML and / or AI systems may be collated and maintained to facilitate future investigations of the behavior of the ML and / or AI systems with respect to the data. Additionally, ML and / or AI systems may be continuously monitored to identify potential biases, errors, and / or unexpected results.

[0109] The present disclosure is described by way of example and with reference to the accompanying drawings. Components, process steps and other elements that may be substantially the same in one or more figures are identified in an equivalent manner and described with minimal repetition. However, it should be noted that elements identified in an equivalent manner may also differ to a certain extent. It should be further noted that some figures may be schematic and not drawn to scale. The various drawing scales, aspect ratios and number of components shown in the figures may be deliberately distorted to make certain features or relationships easier to see.

[0110] In one example, a method for virtually representing a human body posture includes: receiving positioning data, the positioning data describing in detail parameters of one or more body parts for a human user based at least in part on input from one or more sensors; maintaining one or more mapping constraints, the one or more mapping constraints associating a model articulated representation to a target articulated representation; at least in part based on the positioning data and the one or more mapping constraints, estimating a model pose of the model articulated representation and a target pose of the target articulated representation via a previously trained posture optimization machine, wherein the previously trained posture optimization machine is trained using training positioning data with ground truth labels for the model articulated representation; and outputting the target articulated representation with the target pose as a virtual representation of the human user for display. In this example or any other example, the training positioning data does not have a ground truth label for the target articulated representation. In this example or any other example, the one or more mapping constraints include joint mapping constraints, which specify a set of one or more joints in the model articulated representation for a joint of the target articulated representation. In this example or any other example, the joint mapping constraint further specifies a weight for each of the one or more joints in the model articulated representation that are mapped to joints of the target articulated representation. In this example or any other example, the one or more mapping constraints include a root-relative correspondence constraint that specifies that the rotation of a model joint of the model articulated representation relative to the root of the model articulated representation in the model pose has at least a threshold similarity with the rotation of one or more corresponding target joints of the target articulated representation relative to the root of the target articulated representation in the target pose. In this example or any other example, the one or more mapping constraints include a default pose transformation constraint that specifies one or more rotations to be applied to one or more joints of the target articulated representation to match the default pose of the model articulated representation. In this example or any other example, the model pose and the target pose are further estimated based at least in part on one or more previous model poses and previous target poses estimated at one or more previous time frames. In this example or any other example, one or more mapping constraints include a pose continuity constraint that specifies frame-to-frame variation limits for one or more joints of the model articulated representation and one or more joints of the target articulated representation. In this or any other example, the model articulated representation includes a different number of joints than the target articulated representation.In this example or any other example, the positioning data is sparse positioning data, which includes rotation parameters for n joints of the human user; the posture optimization machine is previously trained to receive rotation parameters for n+m joints as input, where m≥1, and estimating the model posture includes: estimating the rotation parameters for n+m model joints of the model articulated representation based at least on the rotation parameters for n joints instead of the rotation parameters for m joints. In this example or any other example, one or more body parts of the human user include the head of the human user. In this example or any other example, one or more body parts of the human user further include one or two hands of the human user. In this example or any other example, the one or more sensors include a camera configured to image at least one body part of the human user. In this example or any other example, the one or more sensors include one or more position sensors configured to be held or worn by at least one body part of the human user.

[0111] In one example, a computing system includes: a logic subsystem; and a storage subsystem that stores instructions executable by the logic subsystem to: receive positioning data based at least in part on input from one or more sensors, the positioning data detailing parameters for one or more body parts of a human user; maintain one or more mapping constraints that associate a model articulated representation to a target articulated representation; estimate a model pose of the model articulated representation and a target pose of the target articulated representation simultaneously via a previously trained pose optimization machine based at least in part on the positioning data and the one or more mapping constraints, wherein the previously trained pose optimization machine is trained using training positioning data having ground truth labels for the model articulated representation; and output the target articulated representation having the target pose as a virtual representation of the human user for display. In this example or any other example, the training positioning data does not have ground truth labels for the target articulated representation. In this example or any other example, the one or more mapping constraints include joint mapping constraints, which specify a set of one or more joints in the model articulated representation for a joint of the target articulated representation. In this example or any other example, the model articulated representation includes a different number of joints than the target articulated representation. In this example or any other example, the positioning data is sparse positioning data, which includes rotation parameters for n joints of a human user; the posture optimization machine is previously trained to receive rotation parameters for n+m joints as input, where m≥1, and estimating the model pose includes: estimating rotation parameters for n+m model joints of the model articulated representation based at least on the rotation parameters for n joints instead of the rotation parameters for m joints.

[0112] In one example, a method for virtually representing a human body posture comprises: receiving sparse positioning data, the sparse positioning data being based at least in part on input from one or more sensors and detailing parameters for a head and one or two hands of a human user; maintaining one or more mapping constraints, the one or more mapping constraints associating a model articulated representation to a target articulated representation, the model articulated representation having a different number of joints than the target articulated representation; simultaneously estimating a model pose of the model articulated representation and a target pose of the target articulated representation via a previously trained posture optimization machine based at least in part on the sparse positioning data and the one or more mapping constraints, wherein the previously trained posture optimization machine is trained using training positioning data having ground truth labels for the model articulated representation; and outputting the target articulated representation having the target pose as a virtual representation of the human user for display.

[0113] It is understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various behaviors described and / or described may be in the order described and / or described, in other orders, in parallel, or may be omitted. Likewise, the order of the above process may be changed.

[0114] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A method for virtually representing a human body posture, the method comprising: receiving positioning data based at least in part on input from one or more sensors, the positioning data detailing parameters for one or more body parts of a human user; maintaining one or more mapping constraints that relate the model articulated representation to the target articulated representation; simultaneously estimating a model pose of the model articulated representation and a target pose of the target articulated representation via a previously trained pose optimization machine based at least in part on the positioning data and the one or more mapping constraints, wherein the previously trained pose optimization machine is trained using training positioning data having ground truth labels for the model articulated representation; as well as The target articulated representation having the target pose is output as a virtual representation of the human user for display.

2. The method according to claim 1, wherein: The training localization data does not have ground truth labels for the target articulated representation.

3. The method according to claim 1, wherein: The one or more mapping constraints include a joint mapping constraint that specifies, for a joint of the target articulated representation, a set of one or more joints in the model articulated representation.

4. The method according to claim 3, wherein: The joint mapping constraint further specifies, for each of the one or more joints in the model articulated representation, a weight that is mapped to a joint of the target articulated representation.

5. The method according to claim 1, wherein: The one or more mapping constraints include a root-relative correspondence constraint, which specifies that the rotation of a model joint of the model articulated representation relative to the root of the model articulated representation in the model pose has at least a threshold similarity with the rotation of one or more corresponding target joints of the target articulated representation relative to the root of the target articulated representation in the target pose.

6. The method according to claim 1, wherein: The one or more mapping constraints include a default pose transformation constraint that specifies one or more rotations to be applied to one or more joints of the target articulated representation to match a default pose of the model articulated representation.

7. The method according to claim 1, wherein: The model pose and the target pose are further estimated based at least in part on one or more previous model poses and previous target poses estimated over one or more previous time frames.

8. The method according to claim 7, wherein: The one or more mapping constraints include a pose continuity constraint that specifies frame-to-frame variation limits for one or more joints of the model articulated representation and one or more joints of the target articulated representation.

9. The method according to claim 1, wherein: The model articulated representation includes a different number of joints than the target articulated representation.

10. The method according to claim 1, wherein: The positioning data is sparse positioning data, which includes rotation parameters for n joints of the human user; the posture optimization machine is previously trained to receive rotation parameters for n+m joints as input, where m≥1, and estimating the model posture includes: estimating the rotation parameters for the n+m model joints of the model articulated representation based at least on the rotation parameters for the n joints rather than the rotation parameters for the m joints.

11. The method according to claim 1, wherein: The one or more body parts of the human user include a head of the human user.

12. The method according to claim 11, wherein: The one or more body parts of the human user include one or both hands of the human user.

13. The method according to claim 1, wherein: The one or more sensors include a camera configured to image at least one body part of the human user.

14. The method according to claim 1, wherein: The one or more sensors include one or more position sensors configured to be held or worn by at least one body part of the human user.

15. A computing system comprising: Logical subsystem; as well as A storage subsystem that stores instructions executable by the logic subsystem for: receiving positioning data based at least in part on input from one or more sensors, the positioning data detailing parameters for one or more body parts of a human user; maintaining one or more mapping constraints that relate the model articulated representation to the target articulated representation; simultaneously estimating a model pose of the model articulated representation and a target pose of the target articulated representation via a previously trained pose optimization machine based at least in part on the positioning data and the one or more mapping constraints, wherein the previously trained pose optimization machine is trained using training positioning data having ground truth labels for the model articulated representation; as well as The target articulated representation having the target pose is output as a virtual representation of the human user for display.

Images