Gesture prediction for articulated objects
Through machine learning models, predicting the pose of articulated objects, using less than all joints to solve the problem of pose inaccurateness in virtual reality, improving the accuracy of immersion and posture recognition, and reducing computing resource consumption.
Patent Information
- Application Number
- CN202280093011.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-06-13
- Filing Date
- 2022-11-29
- Publication Date
- 2024-11-08
AI Technical Summary
When the prior art uses virtual reality to accurately reconstruct the real-world pose of human users, it is difficult to accurately predict the orientation of other joints due to the spatial information of the head and hands, resulting in inaccurate virtual poses.
Through machine learning models, predict the pose of the articulated object, receive spatial information of less than all joints, and use previously trained models to predict the pose. The model is trained to receive more joint information and gradually mask the input of some joints to achieve the pose Accurate prediction.
It improves the immersion of virtual reality experience and the accuracy of posture recognition, reduces the amount of data input and computing resource consumption, and realizes accurate posture prediction in sparse input situations.
Smart Images

Figure CN118922800A_ABST
Abstract
Description
Background Art
[0001] Information about the "pose" of an articulated object (e.g., the position and orientation of one or more body parts of a human user) can be mapped onto a rendered virtual articulated representation for display. For example, when a human user is participating in a virtual environment as part of a virtual reality experience, their representation within the virtual environment can be rendered with a pose that appears similar to the user's real-world pose.
[0002] However, accurately representing the real-world pose of a human user (or other suitable articulated object) using a virtual representation often requires relatively detailed information about the pose / orientation of the user's body parts, and such information is not always available. For example, when using a head-mounted display device to provide a virtual reality experience, the display device may only receive spatial information about the user's head and optionally the hands. In many cases, this is insufficient to accurately recreate the real-world pose of a human user. Summary of the invention
[0003] This summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. In addition, the claimed subject matter is not limited to implementations that overcome any or all of the deficiencies noted in any part of this disclosure.
[0004] A method for predicting a pose of an articulated object, comprising: receiving spatial information for n joints of the articulated object, passing the spatial information for the n joints to a machine learning model previously trained to receive spatial information for n+m joints as input, where m>=1, and receiving a pose prediction for the articulated object from the machine learning model as output, the pose prediction being based on at least the spatial information for the n joints without requiring spatial information for the m joints. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] Figure 1 An articulated object in the form of a human user represented as a virtual representation in a virtual environment is schematically illustrated.
[0006] Figure 2 An exemplary method for predicting the pose of an articulated object is illustrated.
[0007] Figure 3 A sensor for outputting spatial information on joints of a human body is schematically illustrated.
[0008] Figure 4 A machine learning model that outputs a predicted pose for an exemplary articulated object based on spatial information is schematically illustrated.
[0009] Figures 5A-5D Schematic illustration of training a machine learning model over multiple training iterations.
[0010] Figure 6 The application of multiple intermediate transformations to output a pose prediction for an articulated object is schematically illustrated.
[0011] Figure 7 A machine learning model that outputs a predicted pose for another exemplary articulated object based on spatial information is schematically illustrated.
[0012] Figure 8 An exemplary computing system is schematically illustrated. DETAILED DESCRIPTION
[0013] Sometimes it is desirable to display a virtual articulated representation that has a pose that appears similar to the real-world pose of the articulated object. For example, the present disclosure is primarily concerned with situations where the articulated object is a human user. The human user may be represented in a virtual environment—for example, as part of a virtual reality experience (such as a video game or a teleconference session)—by a virtual representation that appears to have a pose that is the same as the user's real-world pose. Accurately recreating the user's real-world pose in a virtual environment can improve the immersion of the virtual reality experience for the human user and any other users who are interacting with the user.
[0014] Relative to Figure 1 A schematic illustration of this is provided, which shows a human user 100 in a real-world environment 102. As shown, the human user's gestures are applied to an articulated representation 104, which virtually represents the human user in a virtual environment 106. In other words, as the human user moves parts of his or her body in the real-world environment, the user's movements are translated into corresponding movements of the articulated representation 104 in the virtual environment 106. For example, the articulated representation may be an avatar of the human user 100 in a virtual reality video game, a teleconference, and / or other virtual reality experience.
[0015] exist Figure 1 In the embodiment of the present invention, the articulated representation 104 and the virtual environment 106 are presented on an electronic display device 108. The electronic display device can have any suitable form factor and utilize any suitable underlying display technology. For example, the electronic display device can be a near-eye display device (e.g., integrated into a head-mounted display), a television, a computer monitor, a smart phone, a tablet computer, a laptop display, a projector system, etc.
[0016] It should be understood that the present disclosure is not limited to situations where the articulated object is the entire body of a human user. On the contrary, in another example to be described below, the articulated object can take the form of a human user's hand or any other alternative part of the human body. Additionally or alternatively, an "articulated object" can include a non-human object such as an animal or an inanimate object such as a machine. In general, an "articulated object" can take the form of any structure including a plurality of different parts, which are coupled to each other via a plurality of joints so that the position and / or orientation of one or more movable parts of the articulated object can be changed relative to the rest of the articulated object.
[0017] Furthermore, it should be appreciated that the predicted pose output of an articulated object need not be graphically rendered for display—e.g., as part of a virtual reality experience or the like. Rather, in some cases, the predicted pose of the articulated object may be used for other purposes that do not involve displaying any representation of the real-world articulated object. As a non-limiting example, the predicted pose of the articulated object may be used for gesture recognition—e.g., a human user may control a computing system by performing various predefined gestures (e.g., waving their hands, swinging their arms), and the performance of these gestures may be detected based on the predicted pose output for the human user via the techniques described herein.
[0018] A computing device attempting to map the pose of an articulated object onto a virtual representation may have only limited information about the position / orientation of parts of the articulated object in the real world. For example, when a human user is using a head-mounted display device, the device may only have spatial information about the user's head (e.g., via a position sensor integrated into the display device) and one, both, or neither of the user's hands (e.g., via a position sensor integrated into a handheld controller and / or via images captured by one or more appropriate cameras). In other words, the computing system only has information about a relatively small number of joints of the articulated object (e.g., a person's head and wrist joints), and this can make it difficult to correctly predict the orientation of other joints of the articulated object (e.g., a person's shoulder, torso, and lower limbs). This can compromise the accuracy of the pose predicted by the computing system—e.g., resulting in the human user being represented in a virtual environment using a virtual pose that is inconsistent with his or her real-world pose.
[0019] Accordingly, the present disclosure relates to techniques for predicting the pose of an articulated object. Specifically, a machine learning model receives spatial information for n different joints of the articulated object, wherein the n joints are less than all joints of the articulated object. For example, in the case of a human user, the n joints may include a head joint and / or one or two wrist joints of the human user, which are associated with spatial information detailing parameters of the user's head and / or hands.
[0020] The machine learning model is previously trained to receive as input spatial information for n+m joints of the articulated object, where m is greater than or equal to 1. For example, during initial training, the machine learning model may receive input data corresponding to substantially all joints of the articulated object—for example, the n+m joints may include every joint of the articulated object. However, in other examples, the n+m joints may include fewer than all of the joints of the articulated object. During training, the input data provided to the machine learning model may be gradually masked—for example, input data corresponding to some of the m joints may be replaced with predefined values representing masked joints or simply omitted. In other words, the machine learning model is trained to accurately predict the pose of the articulated object based on gradually decreasing information about the position / orientation of the various movable parts of the articulated object.
[0021] In this way, the machine learning model can accurately predict the pose of the articulated object at runtime based on sparse inputs - for example, based on at least spatial information for the n joints but not for the m joints. In other words, the machine learning model outputs a prediction of the current pose of the articulated object (e.g., a human user in the real world), where the pose prediction can take any suitable form - for example, a list of joint rotation values, predicted positions / orientations of various parts of the articulated object, and / or another form depending on how the pose prediction will be used.
[0022] The technology in this article can advantageously allow the real-world posture of an articulated object (e.g., a human user) to be accurately recreated without extensive information about the orientation of each joint of the articulated object. In other words, the technology described in this article can provide technical benefits of improving human-computer interaction by more accurately recreating the real-world posture of a human user. For example, this can increase the immersion of a virtual reality experience in which a user participates and / or improve the accuracy of a gesture recognition system. The technology described in this article can reduce the technical benefits of reducing the consumption of computing resources while accurately recreating the real-world posture of a human user by reducing the amount of data that must be collected as input to the posture prediction process. In addition, the technology in this article is associated with a specific mathematical operation that provides a technical purpose of predicting the posture of an articulated object based on sparse input data.
[0023] Specifically, the present disclosure describes a scheme for posture prediction based on a conditional normalization process for sparse input. Specifically, it is possible to estimate the conditional distribution of the whole body posture of a human subject via a flow-based model given head and optional hand data, which enables a reversible mapping between a three-dimensional posture distribution and a base distribution. Then, the reversibility of the model allows learning of probabilistic mappings from the conditional distribution to high-likelihood regions in the same base distribution. This can be referred to as a flow-based avatar generation model (FLAG). The strengths of this design may include: first, the use of a flow-based generation model enables exact posture probability calculations, as opposed to the approximate likelihoods seen in other schemes. Second, the reversibility of the generation model allows the calculation of oracle potential codes. Then, during training, the oracle potential code can play the role of the ground truth for the mapping function. This enables learning a representative mapping from the head and optional hands of the observed human subject to the latent space, thereby allowing the technology described in this article to be used as a strong prediction model. Finally, when optimizing in pose space or latent space, using the predicted model as pose prior can provide a good initialization in the latent space, thereby improving optimization efficiency.
[0024] Figure 2 An exemplary method 200 for predicting the pose of an articulated object is illustrated. The method 200 may be implemented by any suitable computing system of one or more computing devices. Any computing devices implementing the method 200 may each have any suitable capabilities, form factors, and hardware configurations. Where two or more different devices implement the steps of the method 200, they may in some cases communicate via a suitable computer network—for example, using a client / server model. In some cases, the following may be used with respect to Figure 8The computing system 800 is described to implement the method 200 .
[0025] At 202, method 200 includes receiving spatial information for n joints of an articulated object. As discussed above, an "articulated object" can take the form of any structure including a plurality of different parts that are coupled to each other via a plurality of joints, thereby enabling the position and / or orientation of one or more movable parts of the articulated object to be changed. Specifically, the articulated object can be a body of a human user.
[0026] The system receives spatial information for n joints of the articulated object, the n joints including joints less than all joints of the articulated object. As used herein, "spatial information" may include any appropriate computer data that specifies or can be used to estimate the position and / or orientation of a portion of an articulated object (e.g., a portion of a human body) in a real-world environment. This can be expressed relative to the joints connected to the articulated object - for example, the spatial information for the joints is expressed as one or more rotations applied to the joints relative to one or more rotation axes - which can be used to infer the state of the body part connected to the joints. Additionally or alternatively, the spatial information can be expressed relative to the body part connected to the joints - for example, the spatial information for the joints is expressed as the position and orientation of the six degrees of freedom (6-DOF) of the connected body part - which can be used to infer the state of the joints. As an example, the n joints may include the head joints of the human body, and the spatial information for the head joints may detail parameters for the head of the human body. Additionally or alternatively, the n joints may include one or more wrist joints of the human body, and the spatial information for the one or more wrist joints may detail parameters for one or more hands of the human body. Furthermore, it should be understood that "joints" need not refer to actual anatomical joints of the articulated object. Instead, in some cases, "joints" may refer to virtual joints having spatial information reflecting the position / orientation / rotation of a body part of a base object, which may be affected by one or more anatomical joints having different positions relative to the articulated object.
[0027] Again, the human user 100 is shown Figure 3 A schematic illustration is given for this. The human user has a head 300 and two hands 302A and 302B. The computing system may receive spatial information for one or more joints of the human user 100 (e.g., spatial information for n joints, where the human body includes more than n joints), the n joints may include head and / or wrist joints that enable the head 300 and / or hands 302A / B to move.
[0028] The spatial information for the n joints of the articulated object may be derived from positioning data output by one or more sensors. These sensors may be integrated into one or more devices that are held or worn by corresponding body parts of a human user. For example, the sensors may include one or more inertial measurement units (IMUs) integrated into a head-mounted display device and / or a handheld controller. As another example, the sensors may include one or more cameras configured to image a portion of the real-world environment in which a portion of the articulated object (e.g., a user's hand) is visible.
[0029] As used herein, spatial information details "parameters" for one or more parts of the articulated object (e.g., a human body part). This means that the spatial information includes information that specifies or can be used to derive the position and / or orientation of the part of the articulated object, the position and / or orientation being determined by the orientation (e.g., expressed as one or more rotations) of one or more joints of the articulated object. For example, in the case where a human hand is visible in an image captured by a camera, the image data may in some cases be processed to estimate the position and orientation of the human hand relative to the real world environment and / or relative to another camera or another reference frame, the position and orientation being affected by the wrist joint to which the human hand is connected. Thus, in this case, the spatial information is derived from the output of a sensor (e.g., a camera), and the "parameters" detailed by the spatial information are estimates of the position / orientation of the human hand and / or the rotations of the wrist joint.
[0030] However, as another example, "spatial information" may refer to the raw output of one or more sensors - for example, the system may predict the pose of the articulated object based at least in part on the raw output of the sensors. For example, the system may receive image data from a camera as input, and may have been previously trained to output a predicted pose based at least in part on the image pixel data. Thus, "parameters" detailed by the spatial information include pixels depicting a human hand, which can be used to estimate the position and orientation of the hand and / or the rotation of the wrist joint.
[0031] Figure 3 Schematically illustrates different types of sensors, wherein the output from the sensors may include or be used to derive the spatial information. Figure 3In the method 200, a human user 100 wears a head mounted display device (HMD) 304 on his head 300, wherein the HMD 304 may be configured to perform one or more steps of the method 200. For example, the HMD may include an IMU configured to detect changes in the position and / or orientation of the HMD (e.g., via a suitable combination of an accelerometer, a gyroscope, and / or a magnetometer). In this way, changes in position / orientation detected by the IMU integrated into the HMD may be interpreted as changes in the position / orientation of the user's head, and thus as changes in the rotation of the head joints - for example, spatial information for the head joints can be derived from the output of the HMD's integrated IMU.
[0032] In addition, Figure 3 In the embodiment, the human user 100 is holding position sensors 306A and 306B, which can be configured to detect the movement of the user's hands and report it to the HMD 304 and / or another computing system configured to receive the spatial information. The position sensors can include, for example, corresponding IMUs and / or other suitable position sensors for detecting the movement of the user's hands and / or the rotation of the user's wrists.
[0033] In some cases, the spatial information may include or be derived from the output of a wearable position sensor in addition to or as an alternative to handheld position sensors 306A and 306B. For example, a user may wear one or more hand-worn or wrist-worn devices (e.g., a ring or watch) that are configured to output appropriate spatial information related to movement of the user's hand and / or rotation of the user's wrist.
[0034] As another example, the spatial information may include or be derived from the output of a camera configured to image one or more portions of the articulated object. Figure 3 In FIG. 3 , HMD 304 includes an onboard camera 308, which is represented by a white circle. The arrow extending from the white circle represents the field of view (FOV) of the camera within the real world environment. Figure 3 In the depicted scenario, a human hand 302B will be visible in an image of the real-world environment captured by camera 308. The computing system may be configured to identify pixels of the hand in the captured image (e.g., via an appropriate machine learning trained estimator configured to identify a human hand in image data), and estimate the position of the hand based on the size and position of the hand within the captured image. Alternatively, as discussed above, the output from the camera may be provided directly to a machine learning model configured to output a pose prediction based at least in part on input in the form of image data.
[0035] It should be appreciated that at any given time, either, both, or neither of the user's hands may be visible in the FOV of camera 308. Thus, the number of different joints for which spatial information may be received may vary over time in some cases—e.g., at some points in time, spatial information may be received for both wrist joints, while at other points in time, spatial information may not be received for either wrist joint.
[0036] In addition, in some cases, the output from the camera 308 can be used to detect changes in the position and / or orientation of the camera, and thereby the head of a human user (in the case where the camera is integrated into an HMD worn on the user's head). For example, the computing system can identify one or more landmarks in the image of the real-world environment captured by the camera 308. As the position and / or orientation of the camera changes, the image space size and / or position of the identified landmarks will also change. By detecting and analyzing such changes, the movement of the camera (and thereby the user's head) can be estimated, which can be used to infer the rotation of the head joints.
[0037] In some cases, camera 308 may be implemented as a depth camera. For example, the camera may be configured to emit a pattern of structured light or calculate the flight time of photons that leave a light emitter, are reflected off objects in the environment, and impinge on the camera's image sensor. This may enable the camera to more accurately detect the position of a human user's body part relative to the camera and / or the surrounding environment. The camera may be sensitive to any suitable wavelength of light - for example, visible light and / or infrared light.
[0038] It should be understood that the specific sensors described above are non-limiting and need not all be used together. For example, one embodiment may receive output from an IMU and a camera integrated into the HMD or derive spatial information based on the output without using any wearable / handheld position sensors. In other words, the one or more sensors may include a camera configured to image a body part of a human body and one or both of position sensors configured to be held or worn by at least one body part of a human body (and / or a part of another suitable articulated object). In addition, it should be understood that spatial information may be received from the output of one or more sensors not explicitly described herein or may be derived based on the output. Instead, the present disclosure generally assumes that appropriate spatial information is received and that it is unclear as to its source and the processing that may be applied to such spatial information.
[0039] Simply return to Figure 2At 204, method 200 includes passing spatial information for n joints to a machine learning model that was previously trained to receive spatial information for n+m joints as input, where m>=1. In other words, the machine learning model receives spatial information for fewer joints (e.g., n joints) than it was previously trained to receive (e.g., n+m joints). At 206, method 200 includes receiving a pose prediction for the articulated object as an output from the machine learning model, the pose prediction being based at least on the spatial information for the n joints without requiring spatial information for the m joints. In other words, the machine learning model predicts the full pose of the articulated object even if it is not provided with spatial information for the m joints.
[0040] With respect to the exemplary machine learning model 400 shown Figure 4 The process is schematically illustrated. The training and operation of the machine learning model will be described in more detail below. In general, the machine learning model can be implemented by any suitable machine learning (ML) and / or artificial intelligence (AI) technology. Figure 8 to describe examples of appropriate ML and AI techniques.
[0041] exist Figure 4 , the machine learning model receives spatial information 402 corresponding to three different joints J1, J2, and J3. As discussed above, the spatial information for a joint can take the form of any suitable computer data that specifies the position and / or orientation of a body part connected to the joint or can be used to derive the position and / or orientation. As an example, the spatial information can directly specify the position and orientation of the body part (e.g., a 6-DOF pose of the body part), and / or the spatial information can specify one or more rotations of the joint relative to one or more rotational axes. Figure 4 , joints J1 , J2 and J3 correspond to the head joint 404A and two wrist joints 404B / 404C of the human user 100, as indicated by the shaded circles superimposed on the body of the human user.
[0042] In this example, the n joints include three joints, which correspond to the head joints and wrist joints of the human body. However, it should be understood that this is non-limiting, and the n joints may include any appropriate number of joints of the articulated object, which is less than all the joints of the articulated object. In some cases, the n joints may include only one joint. In addition, in some cases, the number of joints included in the n joints may change over time. For example, as discussed above, in some cases the user's hands can be tracked via a camera integrated into a head-mounted display device. At different times, one, both, or none of the user's hands are visible in the camera's field of view. Therefore, as an example, the number of joints included in the n joints may depend on which hand of the user is visible in the camera's field of view in some cases. In general, the machine learning model can receive spatial information corresponding to any number of joints (when such information is available), and the number is in the range from one joint of the articulated object (e.g., only the head of a human user) to each of its joints. In some cases, the n+m joints may include every joint of the articulated object, or may include fewer than all of the joints of the articulated object.
[0043] Based on the input spatial information 402, the machine learning model outputs a predicted pose 406 for the articulated object. For illustrative purposes, the predicted pose is shown as if it were applied to a simplified representation 408 of a human body. The simplified representation includes a number of joints represented by white circles, some of which are labeled as joints 410A, 410B, and 410C. For example, these may directly correspond to the head joints and wrist joints 404A-C of the human user 100.
[0044] However, it will be appreciated that the predicted pose output by the machine learning model may be expressed in any suitable manner. Generally speaking, the predicted pose takes the form of an appropriate computer data structure that can be used to recreate, mimic, or otherwise conceptualize the real-world pose of the articulated object, as predicted based on input data provided to the machine learning model. As an example, the predicted pose may include a list of joint rotations for a set of joints. The set of joints may correspond directly to the joints of the articulated object. In one example, the machine learning model may output predicted spatial information for a set of output joints corresponding to all n+m joints of the articulated object. For example, in Figure 4 The machine learning model outputs a set of joints J1-J NPredicted spatial information 412. These may correspond to the n joints for which the input spatial information is provided to the machine learning model and the m joints for which the machine learning model was previously trained.
[0045] Additionally or alternatively, the machine learning model may output predicted spatial information corresponding to joints of a virtual articulated representation to be rendered for explicit use, which may have a different number and / or distribution of joints from the articulated object—for example, a human user may be represented by an avatar having cartoon or non-human proportions. As an example, the predicted spatial information may correspond to the joints of a skinned multi-person linear model (SMPL) representation. In other words, the joints of the virtual representation of the articulated representation (e.g., simplified representation 408) may not necessarily have a 1:1 correspondence with the joints of the articulated object. Thus, the joints for which the predicted spatial information is output by the machine learning model may not directly correspond to the n+m joints of the articulated object. For example, the virtual representation may have fewer spinal joints than the articulated object.
[0046] Furthermore, it should be appreciated that the predicted pose may be expressed in other ways than as a set of joint rotations. For example, the predicted pose may be specified relative to a body part—e.g., the position and orientation of one or more parts of the articulated object, or one or more parts of a virtual avatar that will be displayed as a representation of the virtual object.
[0047] The machine learning model may be trained in any suitable manner. In some examples, the machine learning model may have been previously trained using training input data for the articulated object with ground truth labels. In other words, the machine learning model may be provided with training spatial information for the joints of the articulated object, labeled with ground truth labels that specify the actual pose of the articulated object to which the spatial information corresponds. Figure 8 A non-limiting example of a suitable training process is described below.
[0048] As discussed above, the machine learning model can be trained to receive spatial information for n+m joints as input. For the first training iteration, this can include providing training input data for all n+m joints to the machine learning model. In a series of subsequent training iterations, the training input data for the m joints can be gradually masked. For example, on the second training iteration, the first joint of the m joints can be masked, where, as an example, the spatial information for the joint in the training data set is replaced with a predefined value representing the masked joint, or the spatial information is simply omitted. On the third training iteration, the second joint of the m joints can be masked, and so on, until each of the m joints is masked, and only the spatial information for the n joints is provided to the machine learning model. In general, on each successive training iteration, one or more of the m joints can be masked. For example, on some training iterations, more than one new joint of the m joints can be masked. Similarly, at some training iterations, new joints among the m joints may not be masked, and / or one or more previously masked joints among the m joints may no longer be masked.
[0049] Relative to Figures 5A-5D The process is schematically illustrated. Specifically, Figure 5A In the example, the training input data set is provided to the machine learning model 400. In this example, the training input data includes spatial information corresponding to a plurality of different postures of the articulated object (including a first posture 502A and a second posture 502B). It should be understood that in Figures 5A-5D The specific poses shown in are non-limiting, and the training input data set may include training data for any suitable number of different poses of the articulated object (including only one pose).
[0050] For illustrative purposes, the different poses represented in the training input data are shown as if they were applied to a simplified representation of a human body. However, it should be understood that the training input data may be expressed in any suitable form and is not typically rendered as if it were applied to a graphical representation of the articulated object.
[0051] exist Figure 5A In , spatial information for each of the n+m joints of the articulated object is provided to the machine learning model. This is indicated by the white fill pattern used for each of the circles representing joints in the simplified representation of the human body. Figure 5B , one of the joints 504A is masked, as indicated by the black fill pattern used for the circle representing the joint 504A. Figure 5AA first training iteration of a training process may be represented, wherein spatial information for the n+m joints is provided to the machine learning model. Figure 5B 504A of the m joints is masked. It should be understood that the "second" training iteration is not necessarily continuous with respect to the first training iteration - instead, spatial information for the n+m joints can be provided to the model for multiple consecutive training iterations before masking any joints.
[0052] exist Figure 5C In , the second joint 504B of the m joints of the articulated representation is masked. Similarly, in Figure 5D , masking the third joint of the m joints. As discussed above, this can continue for multiple training iterations until the spatial information for each of the m joints is masked, and only the spatial information for the n joints is provided to the machine learning model.
[0053] Any appropriate criteria can be used to select which joints of the m joints should be masked for a given training iteration. In one example, gradually masking the training input data for each joint of the m joints includes masking the next joint of the m joints along the kinematic tree of the articulated object toward the root of the kinematic tree on each training iteration. For example, the root of the kinematic tree can be a pelvis. Thus, during each training iteration, it is possible to start at the limbs (e.g., the foot) and move toward the pelvis (e.g., the lower body joints, then the arm joints, then the spinal joints) to mask the spatial information for the m joints. In another embodiment, masking can start at the root of the kinematic tree and then move toward the limbs.
[0054] In another exemplary embodiment, gradually masking the training input data for each of the m joints includes randomly selecting one of the m joints for masking at each training iteration. In general, the spatial information for the m joints can be masked in any suitable order over multiple training iterations, and the order can be determined in any suitable manner depending on the implementation.
[0055] In addition, Figures 5A-5D In an example of , a plurality of different poses of the articulated object are represented in the training input data. In such a case, gradually masking the training input data for each of the m joints comprises masking the same joint of the m joints for each of the plurality of different poses at each training iteration. Figures 5A-5D, where the same joints are masked for each of poses 502A and 502B. However, in other examples, this may not always be the case. Instead, different joints in the m joints may be masked on the same training iteration, and / or different numbers of joints may be masked between different poses represented by the training input data.
[0056] An exemplary scheme for training and operating a suitable machine learning model will now be described, wherein the machine learning model implements a normalized stream and wherein the articulated object is the body of a human subject. The normalized stream as a likelihood-based generative model can provide a way to express probability distributions of data. It should be noted that the normalized stream may only require the definition of a simple base distribution (also known as a prior) and a series of bijective transformations. These bijective transformations can allow the model to map data to a latent space and vice versa.
[0057] Given the input data In the case of , the goal is to learn the joint distribution of the data. The normalization flow models x as a real vector sampled from the chosen underlying distribution pz(z) The transformation T of , which can be as simple as a multivariate normal distribution. With T (and thus T-1) being reversible and differentiable, and using the change of variables formula, the density of x can be obtained as:
[0058] p x (x) = p z (z)|det J T (z)| -1
[0059] Among them, J T is the Jacobian of T. Since z = T -1 (x), and can also be related to x and T -1 The Jacobian of p x (x) is written as:
[0060]
[0061] Instead of a single transformation, multiple simple transformations can be composed to form a complex transformation. Among them, T i z i-1 Transform to z i , z0 is a latent variable in the underlying distribution, and x=z K This can be constructed using a neural network that maximizes the log-likelihood of the data. logp(x) can be written as:
[0062]
[0063] SMPL can be described as a parametric generative model for human body meshes. SMPL can receive as input a 3D human pose in axis angle representation θ and body shape parameters β, and generate a body mesh represented as a 3×6890 matrix M=SMPL(θ,β). With this, it is possible to define To calculate the position and / or orientation for the head and hands.
[0064] The goal of the pose prediction process can be defined as given the sparse observations x H and shape parameter β to generate the complete body pose x θ .x θ ∈R 3×J represents the joint rotations as axis-angle vectors for J body joints, and x H ∈R 9×K represents the global 6D joint rotation and the 3D joint position for each of the K=3 observations (eg, head and hands). This information can be obtained from a parametric model of the human body (eg, SMPL).
[0065] A kind of according to x H Generate x θ A valuable approach is to use a conditional flow-based model f θ Given the observed x H and β. Although this scheme can effectively provide the probability of a given posture, the generation process is still incomplete; in order to H In order to generate a novel pose with β and β, it may be necessary to sample the latent variables. However, the sampling process may not depend on the observations in particular. While other schemes rely on the latent space with the mean z = 0 (a vector of all zeros) as the latent code to generate the full pose, the present disclosure contemplates a representation x that is superior to z = 0. θ In fact, although z = 0 may be the most likely latent code in the underlying distribution, it does not necessarily translate into the most likely pose in the pose space, because by f θ There may be a change in the volume of the distribution for the transformation of H and β to estimate the sub-region N(μ) in the normalized flow basis distribution H ,Σ H ), from which latent variables can be sampled to generate the full body pose.
[0066] At test time, in order to H and β, the complete body posture can be generated by z H ~N(μ H ,ΣH ) samples the potential code and uses it as z θ is an approximation of , which is the potential code for generating the complete body posture. θ =f θ (z H ,[x H ,β]) uses this latent estimate to generate the full body pose.
[0067] Can use the normalized flow model to θ Model the distribution of θ Is for x H and β set the conditional RealNVP. This can be done by writing x via a simple reversible transformation θ This is achieved by mapping from a pose distribution to a base distribution (and vice versa), where each transformation can stretch or shrink its input distribution. In other words, the machine learning model may include a normalization flow that applies multiple reversible transformations to the spatial information for the n joints to output the pose prediction.
[0068] This is relative to Figure 6 Specifically, Figure 6 Illustrated is a set of joint space information 600 to which a series of intermediate transformations 602A-602D are applied, ultimately resulting in an output pose prediction 604. For purposes of illustration, the resulting output pose 606A-606D for each intermediate transformation is shown as if it were applied to a simplified representation of a human body.
[0069] However, it should be understood that for illustrative purposes, Figure 6 The process illustrated in FIG. 1 is highly simplified. It will be appreciated that any suitable number of intermediate transformations may be applied to the joint space information and various steps of the pose prediction process may be omitted. Furthermore, it will be appreciated that the output of each transformation operation need not necessarily be applied to the joint space information as in FIG. Figure 6 , instead of the graphical representation of the articulated objects shown in , this is done only to illustrate the cumulative, incremental effect of each intermediate transformation.
[0070] In the case where multiple reversible transformations are applied to the spatial information for the n joints, in some cases, intermediate supervision can be used during training to influence the contribution of one or more of the reversible transformations. This can be used to address a situation in which the largest contributor to outputting an accurate pose prediction for an articulated object is the last reversible transformation in the sequence. In other words, previously training the machine learning model can include applying intermediate supervision by providing the ground truth pose of the articulated object to one or more intermediate reversible transformations of the multiple reversible transformations. Thus, in addition to having the ground truth pose as input to the last transformation block, the ground truth pose can be input to the intermediate transformation blocks as if they were the last blocks of the sub-network. This is possible because f θ The transformations in do not modify the data scale. Therefore, the intermediate transformation blocks can be encouraged to produce reasonable human poses and fully exercise the capacity of these blocks. Doing so can have the effect of increasing the contribution of the intermediate reversible transformations to the final output of accurate pose predictions for the articulated object.
[0071] In order to H To generate novel poses in the case of β and β, we can sample the latent variable z from the underlying distribution and use it to generate poses x^ θ =f θ (z,[x H ,β]). Since the oracle latent code is known during training, the model can be trained so that it learns to transform the condition (x H , and β) to regions in the underlying distribution where z has high probability. Utilizing z during training allows accounting for changes in the volume of the underlying distribution, and thus changes in the probability mass around the latent code (when adjusted via f θ When transforming from the base distribution to the pose space), the region of interest can be modeled using Gaussian to learn its parameters μ H and Σ H =diag(σ H ) 2 .
[0072] Such a mapping can have two desirable properties: (i) The mapping can be expressive, such that it can generate a representation of the whole body given sparse observations. This can be used to estimate a subregion of the underlying distribution that represents the whole body. (ii) The mapping can account for uncertainty in the representation of the human body given sparse observations. When only the head and hands are observed, there are multiple plausible whole body poses. For each plausible pose, a corresponding subregion in the underlying distribution can be determined.
[0073] To this end, the mapping function can be modeled using a transformer-based model (with the help of a transformer encoder), thereby utilizing a self-attention mechanism that learns the relationship between different joints in the body during training. In short, the transformer encoder can receive x H and β as input, and estimate N(μ H ,Σ H ), where μ H Trained as oracle latent code z * A good approximation of .
[0074] In order to make such a distribution representative of the entire body, x can be generated based on the output of the transformer encoder θ , which was originally designed to reconstruct x based on the entire body joints θ And gradually reduce joint visibility (via masking) in the encoding until only the head and hands are provided, as discussed above. To further help the Transformer learn a representation of the body, the masked joints can be predicted given the observed joints. Such gradual masking and prediction can enable the model to infer a representation of the entire body by focusing on the available joints in the input (layers). To obtain a compact representation coming out of the Transformer encoder, pooling can be applied over the range of each output joint. H ) so that only unmasked joints (e.g., head representation and hand representation) are taken.
[0075] The output of the transformer encoder can be deterministic in some cases—for example, the encoder does not give an uncertainty estimate for the predicted pose. Therefore, a categorical latent space [13, 30, 33] can be modeled over a range of human poses based on the output of the transformer encoder. Discrete latent variables from this distribution can be sampled to generate x for the defined auxiliary task. θ , or use the entire latent representation to estimate N(μ H ,Σ H ), which includes information about plausible poses and associated uncertainties. In order to efficiently model the complex distribution of human motion, a relatively large latent space can be used, resulting in a relatively large number of latent classes. To remedy this, a 2D classification latent space can be used. Specifically, G-dimensional latent variables can be modeled, each responsible for M modes. This provides the advantage of using M G The ability of a unique hot latent code.
[0076] The training dataset may include a variety of 3D human models, where each sample is a triplet (x θ ,x H,β), and where β is the SMPL shape parameter. The loss function is given by
[0077] L=λ nll L nll +λ mjp L mjp +λ rec L rec +λ lra L lra
[0078] where λ is the weight associated with each term.
[0079] L nll :This term promotes the model to make model f θ The following x θ The negative log-likelihood of is minimized. In addition, we can consider the intermediate supervision result of f θ The log-likelihood values produced by the sub-network.
[0080]
[0081] Where S is f θ The set of subnetworks of (e.g., for a predefined set of is from block T i to T1), p s θ (x θ ) is in subnetwork sx θ The log-likelihood value of s are the weights associated to the sub-networks, which are proportional to the number of transform blocks in each sub-network.
[0082] L mjp : To train the auxiliary task of masked joint prediction, the following function can be used:
[0083]
[0084] Among them, J masked is a list of masked joints, x j P YesR 9 The representation of the jth joint in (6D rotation and 3D position), and x^ j P are the corresponding predictions from the network.
[0085] L rec : This term acts on the output of the auxiliary task of decoding the whole body pose from the classification latent space of the transformer with discrete latent variable sampling, which aims to guide the construction of a meaningful discrete latent space:
[0086]
[0087] L lra :This term promotes learning Gaussian distribution N(μ H ,Σ H ), under which the oracle latent variable z has high probability:
[0088]
[0089] in, is the estimated sub-region of the underlying distribution.
[0090] Although the entire model can be trained in an end-to-end fashion, first training f θ Subsequent training of the latent region approximator can be efficient. The second training phase can be relatively fast—e.g., about 4 GPU hours. θ This two-stage training may also be useful as a base model and only train the mapping function for other data modalities (e.g., body landmarks or environment scans).
[0091] At a given x H and β can be calculated by first calculating μ given the observations H , then use μ H As z θ The approximation of the pose x^ θ =f θ (μ H ,[x H ,β]) to generate the whole body pose. To further enhance the quality of the generated pose, a flow-based model can be used as a pose prior in the optimization, thereby minimizing a cost function with respect to the prior and the data. The optimization can be done in pose space or latent space.
[0092] The optimizer seeks plausible human poses θ under which the observations x are matched. H . θ can be optimized by minimizing the cost:
[0093]
[0094] The optimizer seeks to match the observations x H Under the model of plausible posture, the latent variable z is obtained. When using the generative function of posture prior (f θ )When generating a posture, it is possible to optimize for z by minimizing the cost.
[0095]
[0096] where logp(z) is the log-likelihood of z optimized under the underlying distribution:
[0097] N(0,I),θ^=f θ (z,[x H ,β])
[0098] And r = || z - μ H || is a regularization term, which is used to implicitly prevent the latent code from being too far away from the initial guess.
[0099] The present disclosure focuses primarily on situations where the articulated object is the entire body of a human subject. However, as discussed above, this is not a limitation. Instead, in other examples, the articulated object may take the form of a replacement part of a human body, a non-human animal, or an inanimate object such as a machine.
[0100] to this end, Figure 7 The schematic diagram shows an exemplary case where the articulated object is a human hand instead of the entire human body. Figure 7 An exemplary machine learning model 700 implemented as described above is shown. The machine learning model 700 receives spatial information for joints J1, J2, and J3 corresponding to three joints 704A-C of an articulated object, which in this case takes the form of a human hand 706. Specifically, in this case, the n joints include one or more finger joints of a human hand. The spatial information for the one or more finger joints details parameters of one or more fingers or finger segments of the human hand. For example, the spatial information can specify the position / orientation of the fingers of the human hand and / or the rotation applied to the joints of the hand.
[0101] Spatial information for the joint may be collected in any suitable manner, for example, via position sensor 708. As one example, the position sensor may take the form of a camera configured to image a human hand. As another example, the position sensor may include a suitable radio frequency antenna configured to expose the surface of a human hand to an electric field and assess how the electric field impedance at the antenna is affected by movement and proximity of conductive human skin.
[0102] Based on the input spatial information 702, the machine learning model outputs a set of predicted spatial information 710, which can constitute the predicted pose of the articulated object. As discussed above, such spatial information can be expressed as the position / orientation of a body part (e.g., a finger or finger segment) of the articulated object, one or more rotations applied to the joints of the articulated object, and / or expressed in another suitable manner.
[0103] The methods and processes described herein may be bundled with a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as an executable computer application, a network accessible computing service, an application programming interface (API), a library, or a combination of the above options and / or other computing resources.
[0104] Figure 8 Schematically illustrated is a simplified representation of a computing system 800 configured to provide all of the computing functionality described herein. The computing system 800 may take the form of one or more personal computers, network accessible server computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones) virtual / augmented reality / mixed reality computing devices, wearable computing devices, the Internet of Things (IoT), embedded computing devices, and / or other computing devices.
[0105] The computing system 800 includes a logic subsystem 802 and a storage subsystem 804. The computing system 800 may optionally include a display subsystem 806, an input subsystem 808, a communication subsystem 810, and / or Figure 8 Other subsystems not shown.
[0106] The logic subsystem 802 includes one or more physical devices configured to execute instructions. For example, the logic subsystem can be configured to execute instructions as part of one or more applications, services, or other logical structures. The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processor of the logic subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic subsystem may be distributed among two or more separate devices, which may be in a remote location and / or configured to implement collaborative processing. Various aspects of the logic subsystem may be virtualized and may be executed by a remotely accessible networked computing device configured in accordance with a cloud computing configuration.
[0107] The storage subsystem 804 includes one or more physical devices configured to temporarily and / or permanently store computer information (e.g., data and instructions executable by the logic subsystem). When the storage subsystem includes two or more devices, these devices may be located at a common location and / or may be located at a remote location. The storage subsystem 804 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices. The storage subsystem 804 may include removable and / or built-in devices. When the logic subsystem executes instructions, the state of the storage subsystem 804 may be transformed, for example, to store different data.
[0108] Various aspects of the logic subsystem 802 and the storage subsystem 804 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include program specific and application specific integrated circuits (PASIC / ASIC), program specific and application specific standard products (PSSP / ASSP), systems on chips (SOC), and complex programmable logic devices (CPLD).
[0109] The logic subsystem and the storage subsystem can collaborate to instantiate one or more logic machines. As used herein, the term "machine" is used to collectively refer to a combination of hardware, firmware, software, instructions, and / or any other components that cooperate to provide computer functions. In other words, a "machine" is by no means an abstract concept and always has a tangible form. A machine can be instantiated by a single computing device, or a machine can include two or more subcomponents instantiated by two or more different computing devices. In some embodiments, the machine includes a local component (e.g., a software application executed by a computer processor) that collaborates with a remote component (e.g., a cloud computing service provided by a network of server computers). Software and / or other instructions that give functionality to a particular machine can be optionally stored as one or more unexecuted modules on one or more appropriate storage devices.
[0110] The machine may be implemented using any suitable combination of existing and / or future machine learning (ML) techniques, artificial intelligence (AI) techniques, and / or natural language processing (NLP) techniques. Non-limiting examples of techniques that may be incorporated into one or more machine embodiments include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and pool features across one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or natural language processing techniques (e.g., word segmentation, stemming, component and / or dependency analysis, and / or intent recognition, segmentation models, and / or super-segmentation models (e.g., hidden dynamic models)).
[0111] In some examples, one or more differentiable functions can be used to implement the methods and processes described herein, wherein the gradient of the differentiable function can be calculated and / or estimated relative to the input and / or output of the differentiable function (e.g., relative to training data and / or relative to an objective function). Such methods and processes can be determined at least in part by a set of trainable parameters. Accordingly, the trainable parameters of a particular method or process can be adjusted by any appropriate training process to continuously improve the operation of the method or process.
[0112] Non-limiting examples of training processes for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization algorithm), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on categories derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning), and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual bandit methods, maximum likelihood, and / or expectation maximization. In some examples, multiple methods, processes, and / or components of a system can be trained simultaneously relative to an objective function that measures the performance of the collection of components (e.g., relative to reinforced feedback and / or relative to labeled training data). Training multiple methods, processes, and / or components simultaneously can improve such collection operation. In some examples, one or more methods, processes, and / or components can be trained independently of other components (e.g., offline training based on historical data).
[0113] When display subsystem 806 is included, it can be used to present a visual representation of the data stored by storage subsystem 804. This visual representation can take the form of a graphical user interface (GUI). Display subsystem 806 can include one or more display devices using almost any type of technology. In some embodiments, the display subsystem can include one or more visual displays, augmented reality displays, or mixed reality displays.
[0114] When an input subsystem 808 is included, it may include an interface with one or more input devices. An input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input subsystem may include or interface with a selected natural user input (NUI) component system. Such a component system may be integrated or peripheral, and the transduction and / or processing of input actions may be on-board or off-board processing. Exemplary NUI component systems may include: a microphone for conversation and / or speech recognition; an infrared, color, stereoscopic vision, and / or depth camera for machine vision and / or gesture recognition; a head tracker, an eye tracker, an accelerometer, and / or a gyroscope for motion detection and / or intent recognition.
[0115] When a communication subsystem 810 is included, it can be configured to communicatively couple the computing system 800 with one or more other computing devices. The communication subsystem 810 can include wired and / or wireless communication devices compatible with one or more different communication protocols. The communication subsystem can be configured to communicate over a personal network, a local network, and / or a wide area network.
[0116] The methods and processes described in this article can be configured to provide control of proprietary and / or possibly sensitive data to users and / or any other person. Whenever storing, accessing and / or processing data, these data can be manipulated according to privacy and / or security standards. When collecting user data, users or other stakeholders can specify how to use and / or store the data. As long as user data is collected for any purpose, the user data can be collected only under the extremely respectful user privacy (for example, user data can be collected only when the user with the data provides confirmatory consent and / or the user with the data can be notified whenever the user data is collected). If the data will be released for access by anyone other than the user or used for any decision-making process, the user's consent can be collected before using and / or releasing the data. The user can choose to join and / or opt out of data collection at any time. After the data has been collected, the user can issue an order to delete the data and / or limit access to the data. All potentially sensitive data can be optionally encrypted and / or anonymous (when feasible), thereby further protecting user privacy. The user can specify the statistics / results of the part, metadata or data processing of the data for release to all parties, for example, for further processing. Proprietary and / or confidential data may be kept completely private, e.g., only temporarily decrypted for processing, or only decrypted for processing on a user device, but otherwise stored in encrypted form. The user may hold and control the encryption keys for the encrypted data. Alternatively or in addition, the user may designate a trusted third party to hold and control the encryption keys for the encrypted data, e.g., to provide the user with access to the data in accordance with an appropriate authentication protocol.
[0117] When the methods and processes described herein incorporate ML and / or AI components, the ML and / or AI components may make decisions based at least in part on the training of these components with respect to the associated training data. Accordingly, the ML and / or AI components may be trained on a diverse set of representative data, which may include sufficient relevant data for a diverse set of users and / or user groups. Specifically, training data sets may be included that are relevant to different individuals and different groups of people, thereby improving the performance of these components in terms of the user experience of the users and / or user groups as the ML and / or AI components are trained.
[0118] ML and / or AI components may additionally be trained to make decisions that minimize possible biases against individuals and / or populations. For example, when AI systems are employed to assess any qualitative and / or quantitative information about an individual or population, these systems may be trained to be invariant to differences between individuals or populations that are not intended to be measured by qualitative and / or quantitative assessments, such that (for example) any decision is not influenced in an unintended manner by differences between individuals and populations.
[0119] ML and / or AI systems may be designed to provide context about how they work, so that implementers of ML and / or AI systems can consider decisions / evaluations made by these systems. For example, ML and / or AI systems may be configured to implement replicable behavior, e.g., when they make pseudo-random decisions, a random seed may be used and recorded, so that the decision can be replicated later. As another example, data used to train and / or test ML and / or AI systems may be curated and maintained to facilitate future investigations of the behavior of the ML and / or AI systems with respect to said data. Furthermore, ML and / or AI systems may be monitored on an ongoing basis to identify possible biases, errors, and / or unintended results.
[0120] The present disclosure is described by way of example with reference to the associated drawings. Substantially identical components, process steps, and other elements are co-identified in one or more of the drawings and described with minimal repetition. However, it should be noted that the co-identified elements may also differ to some extent. It will be further noted that some of the drawings may be schematic and not drawn to scale. The various drawing scales, aspect ratios, and numbers of components shown in the drawings may be purposefully distorted to make certain features or relationships easier to see.
[0121] In one example, a method for predicting the posture of an articulated object includes: receiving spatial information for n joints of the articulated object; passing the spatial information for the n joints to a machine learning model, the machine learning model previously trained to receive spatial information for n+m joints as input, where m>=1; and receiving a posture prediction for the articulated object as an output from the machine learning model, the posture prediction being based at least on the spatial information for the n joints without spatial information for the m joints. In this example or any other example, the articulated object is a human body. In this example or any other example, the n joints include a head joint of a human body, and the spatial information for the head joints details parameters for the head of the human body. In this example or any other example, the n joints include one or more wrist joints of a human body, and the spatial information for the one or more wrist joints details parameters for one or more corresponding hands of the human body. In this example or any other example, the spatial information for the n joints of the articulated object is derived from positioning data output by one or more sensors. In this example or any other example, the one or more sensors include one or both of a camera configured to image one or more body parts of a human body and a position sensor configured to be held or worn by at least one body part of a human body. In this example or any other example, the articulated object is a human hand. In this example or any other example, the n joints include one or more finger joints of a human hand, and the parameters of one or more fingers or finger segments of the human hand are detailed for the spatial information of the one or more finger joints. In this example or any other example, the machine learning model is previously trained using training input data with ground truth labels for the articulated object. In this example or any other example, previously training the machine learning model includes providing the machine learning model with training input data for all n+m joints for the first training iteration, and gradually masking the training input data for one or more of the m joints in a series of subsequent training iterations. In this example or any other example, gradually masking the training input data for each of the m joints includes masking one or more of the next joints of the m joints along the kinematic tree of the articulated object toward the root of the kinematic tree at each training iteration. In this example or any other example, progressively masking the training input data for each of the m joints includes randomly selecting one or more of the m joints to mask at each training iteration.In this example or any other example, the training input data includes spatial information corresponding to a plurality of different poses of the articulated object, and wherein the training input data that progressively masks one or more of the m joints includes masking the same joint of the m joints for each of the plurality of different poses at each training iteration. In this example or any other example, the machine learning model includes a normalization stream that applies a plurality of reversible transformations to the spatial information of the n joints to output a pose prediction. In this example or any other example, previously training the machine learning model includes applying intermediate supervision by providing a ground truth pose of the articulated object to one or more intermediate reversible transformations of the plurality of reversible transformations. In this example or any other example, the pose prediction includes predicted spatial information for all n+m joints of the articulated object.
[0122] In one example, a computing system includes: a logic subsystem; and a storage subsystem storing instructions, the instructions being executable by the logic subsystem for: receiving spatial information for n joints of the articulated object; passing the spatial information for the n joints to a machine learning model, the machine learning model previously trained to receive spatial information for n+m joints as input, where m>=1; and receiving a posture prediction for the articulated object as an output from the machine learning model, the posture prediction being based at least on the spatial information for the n joints without requiring spatial information for the m joints. In this example or any other example, the articulated object is a human body, and wherein the n joints include a head joint of the human body and one or more wrist joints of the human body. In this example or any other example, previously training the machine learning model includes providing the machine learning model with training input data for all n+m joints for a first training iteration, and in a series of subsequent training iterations, gradually masking the training input data for each of the m joints.
[0123] In one example, a method for predicting the posture of a human body includes: receiving spatial information of n joints of the human body, the n joints including the head joints of the human body and one or more wrist joints of the human body; passing the spatial information of the n joints to a machine learning model, the machine learning model being previously trained to receive spatial information for n+m joints as input, wherein m>=1, the machine learning model being previously trained by providing training input data for all n+m joints to the machine learning model for a first training iteration, and gradually masking the training input data for each of the m joints in a series of subsequent training iterations; and receiving a posture prediction for the human body as an output from the machine learning model, the posture prediction being based at least on the spatial information for the n joints without requiring spatial information for the m joints.
[0124] It should be understood that the configuration and / or method described in this article are exemplary in nature, and these specific embodiments or examples should not be considered in a restrictive sense, because there are many variations that are possible. The specific routine or method described in this article can represent one or more of any number of processing strategies. Like this, the various actions illustrated and / or described can be in the order illustrated and / or described, in other orders or performed in parallel, or can be omitted. Similarly, the order of the process described above can be changed.
[0125] The subject matter of the present disclosure includes all novel and nonobvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, and any and all equivalents thereof.
Claims
1. A method for predicting the pose of an articulated object, comprising: Receiving spatial information of n joints of the articulated object; Passing the spatial information for the n joints to a machine learning model previously trained to receive as input the spatial information for n+m joints, where m>=1; as well as A pose prediction for the articulated object is received as an output from the machine learning model, the pose prediction being based at least on spatial information for the n joints without requiring spatial information for the m joints.
2. The method according to claim 1, wherein: The articulated object is a human body.
3. The method according to claim 2, wherein: The n joints include a head joint of the human body, and spatial information of the head joint details parameters for the head of the human body.
4. The method according to claim 3, wherein: The n joints include one or more wrist joints of the human body, and spatial information of the one or more wrist joints details parameters of one or more corresponding hands of the human body.
5. The method according to claim 2, wherein: Spatial information for the n joints of the articulated object is derived from positioning data output by one or more sensors.
6. The method according to claim 5, wherein: The one or more sensors include one or both of: a camera configured to image one or more body parts of the human body and a position sensor configured to be held or worn by at least one body part of the human body.
7. The method according to claim 1, wherein: The articulated object is a human hand.
8. The method according to claim 7, wherein: The n joints include one or more finger joints of the human hand, and spatial information of the one or more finger joints details parameters for one or more fingers or finger segments of the human hand.
9. The method according to claim 1, wherein: The machine learning model was previously trained using training input data with ground truth labels for the articulated objects.
10. The method according to claim 1, wherein: Previously training the machine learning model includes: providing the machine learning model with training input data for all n+m joints for a first training iteration, and in a series of subsequent training iterations, gradually masking the training input data for one or more of the m joints.
11. The method according to claim 10, wherein: Progressively masking the training input data for each of the m joints includes masking one or more next joints of the m joints along the kinematic tree of the articulated object toward a root of the kinematic tree at each training iteration.
12. The method according to claim 10, wherein: Gradually masking the training input data for each of the m joints includes randomly selecting one or more of the m joints for masking at each training iteration.
13. The method according to claim 10, wherein: The training input data includes spatial information corresponding to multiple different postures of the articulated object, and wherein gradually masking the training input data for one or more of the m joints includes: masking the same joint of the m joints for each of the multiple different postures at each training iteration.
14. The method according to claim 1, wherein: The machine learning model includes a normalization flow that applies a plurality of reversible transformations to the spatial information for the n joints to output the pose prediction.
15. The method according to claim 14, wherein: Prior training of the machine learning model included applying intermediate supervision by providing a ground truth pose of the articulated object to one or more intermediate reversible transforms of the plurality of reversible transforms.
