Training machine learning models to predict joint locations using intermediate keypoints

Abstract

Introduced here are computer-implemented platforms (also referred to as "pose estimation platforms") that are able to estimate poses of individuals performing physical activities. By predicting locations of joints and intermediate points between joints, the pose estimation platforms are able to more accurately estimate poses and thereby provide more relevant feedback to individuals.

Description

TRAINING MACHINE LEARNING MODELS TO PREDICT JOINT LOCATIONS USING INTERMEDIATE KEYPOINTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to US Provisional Application No. 63 / 663,018, titled “Training Machine Learning Models to Predict Joint Locations Using Intermediate Keypoints” and filed on June 21 , 2024, which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] Various embodiments concern computer programs and associated computer-implemented techniques for estimating poses of a living body and providing appropriate feedback to promote completion of physical activities.BACKGROUND

[0003] Pose estimation (also called “pose detection”) is an active area of study in the field of computer vision. Over the last several years, tens - if not hundreds - of different approaches have been proposed in an effort to solve the problem of pose detection. Many of these approaches rely on machine learning due to its programmatic approach to learning what constitutes a pose.

[0004] As a field of artificial intelligence, computer vision enables machines to perform image processing tasks with the aim of imitating human vision. Pose estimation is an example of a computer vision task that generally includes detecting, associating, and tracking the movements of a person. This is commonly done by identifying “key points” that are semantically important to understanding pose. Examples of key points include “head,” “left shoulder,” “right shoulder,” “left knee,” and “right knee.” Insights into posture and movement can be drawn from analysis of these key points.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 includes several examples of traditional pose estimation.

[0006] Figure 2 illustrates a network environment that includes a pose estimation platform that is executed by a computing device.

[0007] Figure 3 illustrates an example of a computing device that is able to execute a pose estimation platform.

[0008] Figure 4 illustrates a training flow for estimating poses using intermediate keypoints.

[0009] Figure 5 illustrates traditional loss and distance-to-bone loss during training.

[0010] Figure 6 illustrates an inferencing flow for estimating poses using intermediate keypoints.

[0011] Figure 7 illustrates a digital image and an estimated pose.

[0012] Figure 8 illustrates a digital image and an estimated pose.

[0013] Figure 9 includes a block diagram illustrating an example of a processing system in which at least some operations described herein can be implemented.

[0014] Various features of the technology described herein will become more apparent to those skilled in the art from a study of the Detailed Description in conjunction with the drawings. Various embodiments are depicted in the drawings for the purpose of illustration. However, those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technology. Accordingly, although specific embodiments are shown in the drawings, the technology is amenable to various modifications.DETAILED DESCRIPTION

[0015] Over the last several years, significant advances have been made in the field of computer vision. This has resulted in the development of sophisticated pose estimation programs (also called “pose estimators” or “pose predictors”) that are designed to perform pose estimation in either two dimensions or three dimensions. Two-dimensional (“2D”) pose estimators predict the 2D spatial locations of key points, generally through the analysis of the pixels of a single digital image. Three-dimensional (“3D”) pose estimators predict the 3D spatial arrangement of key points, generally through the analysis of the pixels of multiple digital images, for example, consecutive frames in a video or a single digital image in combination with another type of data generated by, for example, an inertial measurement unit (“IMU”) or Light Detection and Ranging (“LiDAR”) unit.

[0016] Pose estimators - both 2D and 3D - continue to be applied to different contexts, and as such, continue to be used to help solve different problems. One problem for which pose estimators have proven to be particularly useful is monitoring the performance of physical activities. Consider, for example, a scenario where an individual is instructed or prompted to perform a physical activity by a computer program. By applying a pose estimator to digital images of the individual, the computer program can glean insight into the performance of the physical activity. Historically, the individual may have instead been asked to summarize her performance of the physical activity (e.g., in terms of difficulty); however, this type of manual feedback tends to be inaccurate and inconsistent. Due to their consistent, programmatic nature, pose estimators allow for more accurate monitoring of performances of physical activities.

[0017] Accuracy is especially important when the pose estimator is responsible for monitoring physical activities that have meaningful real-world impact, such as on the health and wellness of the individual responsible for performing the physical activities. Exercise therapy is an intervention technique that utilizes physical activities as the principal treatment for addressing the symptoms of musculoskeletal (“MSK”) conditions, such asacute physical ailments and chronic physical ailments. Exercise therapy programs (or simply “programs”) generally involve a plan for performing physical activities during exercise therapy sessions (or simply “sessions”) that occur on a periodic basis. Normally, the purpose of a program is to either restore normal MS K functionality or reduce the pain caused by a physical ailment, which may have been caused by injury or disease.

[0018] Typically, a pose estimator defines a set of key points of interest for a model to identify on images of individuals performing a physical activity.Such key points tend to correspond to human joints (e.g., shoulder, elbow, wrist, etc.). The model learns the relationships between these key points, especially between pairs of key points corresponding to adjacent body joints (e.g., elbow and wrist). Such key points are rigidly connected and tend to move jointly with a specific distance between them. As such, predicting the locations of key points independently from each other fails to leverage all possible information contained in the relationships between joints. As a result, estimated poses may fail to adequately represent the poses of individuals. Figure 1 shows examples of a pose estimator producing an inaccurate estimated pose.

[0019] Introduced here is an approach for training machine learning models (also called “models” or simply “models”) to rely on intermediate key points between anatomical locations when predicting the locations of those anatomical locations. Note that, for the purpose of illustration, embodiments are generally described in the context of establishing the locations of joints. However, the features of those embodiments may be similarly applicable if the model is trained to predict the location of another type of anatomical location (e.g., referenced by a bone, such as the skull; a region, such as the thoracic region or lumbar region; or soft tissue, such as the eye).

[0020] As further discussed below, a model can be trained to receive, as input during a training operation, intermediate key points between joints in addition to the key points indicating joints. Inputting the intermediate key points forces the model to learn more complex reasoning about the relationships between the joints. When applied as part of an inferencingoperation, the trained model need not output the intermediate key points. Rather, the trained model can rely on the complex reasoning learned during the training operation to more accurately predict the locations of joints. The approach thus improves accuracy of pose estimators without being computationally costly.

[0021] For example, given a training dataset that includes a set of images of one or more individuals performing a physical activity, a pose estimation platform may:• Receive a training file that specifies intermediate key points between joints that are representative of key points;• Provide the training dataset and training file to a model, as input, to produce a trained model that is able to estimate poses based on key points and intermediate key points; and• Cause the trained model to update its predictions based on the accuracy of the predicted key points and intermediate key points.

[0022] This training approach may improve the accuracy of real-time pose estimation for an individual as she performs a physical activity. These estimated poses - or indicia that are visually representative thereof - may be presented for display on an interface that is accessible via a computing device. The computing device may be associated with the individual and may be responsible for generating the digital images from which the poses are estimated. The computing device may additionally provide feedback with respect to the individual’s performance of a physical activity based on the estimated poses.

[0023] For example, once the model is trained using the intermediate key points, the pose estimation platform may use the model to more accurately estimate poses for individuals. For example, the pose estimation platform may:Receive an image of an individual performing a physical activity;• Apply the trained model to the image to generate, as output, predicted locations of joints that are collectively representative of an estimated pose of the individual;• Determine the individual’s performance of the physical activity based on the estimated pose; and• Provide feedback to the individual based on the individual’s performance.

[0024] Another benefit is that this approach may be generic to a large variety of physical activities, though specific parameters and templates can be defined per physical activities. Accordingly, a set of models corresponding to different physical activities may be trained and then released, but additional models corresponding to new physical activities may be added to the set or existing models corresponding to existing physical activities could be removed from the set.

[0025] For the purpose of illustration, embodiments may be described with reference to exercises that are performed during sessions as part of a program. However, a pose estimation platform could be designed to accurately estimate poses of other physical activities, such as sporting activities, cooking activities, art activities, and the like. Accordingly, the approach described herein could be used to accurately predict poses and provide feedback regarding performance of nearly any physical activity.

[0026] Moreover, embodiments may be described in the context of computer-executable instructions for the purpose of illustration. However, aspects of the approach could be implemented via hardware or firmware instead of, or in addition to, software. As an example, the pose estimation platform may be embodied as a computer program that offers support for completing exercises during sessions as part of a program, determines which physical activities are appropriate for a user given performance during past sessions, and enables feedback for the user based on the estimated poses of the user.Terminology

[0027] References in the present disclosure to “an embodiment” or “some embodiments” mean that the feature, function, structure, or characteristic being described is included in at least one embodiment. Occurrences of such phrases do not necessarily refer to the same embodiment, nor are they necessarily referring to alternative embodiments that are mutually exclusive of one another.

[0028] Unless the context clearly requires otherwise, the terms “comprise,” “comprising,” and “comprised of” are to be construed in an inclusive sense rather than an exclusive or exhaustive sense. That is, in the sense of “including but not limited to.” The term “based on” is also to be construed in an inclusive sense. Thus, the term “based on” is intended to mean “based at least in part on.”

[0029] The terms “connected,” “coupled,” and variants thereof are intended to include any connection or coupling between two or more elements, either direct or indirect. The connection or coupling can be physical, logical, or a combination thereof. For example, elements may be electrically or communicatively coupled to one another despite not sharing a physical connection.

[0030] The term “module” may refer broadly to software, firmware, hardware, or combinations thereof. Modules are typically functional components that generate one or more outputs based on one or more inputs. A computer program may include or utilize one or more modules. For example, a computer program may utilize multiple modules that are responsible for completing different tasks, or a computer program may utilize a single module that is responsible for completing all tasks.

[0031] When used in reference to a list of multiple items, the word “or” is intended to cover all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of items in the list.Overview of Pose Estimation Platform

[0032] A pose estimation platform may be responsible for monitoring the motion of an individual (also called a “user,” “patient,” or “participant”) throughanalysis of digital images that contain her and are captured as she completes a physical activity. As an example, the pose estimation platform may guide the user through exercise therapy sessions (or simply “sessions”) that are performed as part of an exercise therapy program (or simply “program”) by monitoring pose in an ongoing manner. As part of the program, the user may be requested to engage with the pose estimation platform on a periodic basis. The frequency with which the user is requested to engage with the pose estimation platform may be based on factors such as the anatomical region for which therapy is needed, the MSK condition for which therapy is needed, the difficulty of the program, the age of the user, the amount of progress that has been achieved, and the like. Note that because the motion of the user is generally monitored through the continual analysis of pose, the pose estimation platform could also be called a “pose monitoring platform.”

[0033] As the user performs exercises, she may be recorded by a camera of a computing device. Normally, the camera is part of the computing device on which the motion monitoring is executed or accessed. For example, in order to initiate a session, the user may initiate a mobile application that is stored on, and executable by, her mobile phone or tablet computer, and the mobile application may instruct the user to position her mobile phone or tablet computer in such a manner that one of its cameras can record her as exercises are performed. Note that, in some embodiments, the camera is part of another computing device. For example, the camera may be included in a peripheral computing device, such as a web camera (also called a “webcam”), that is connected to the computing device. By examining the digital images that are output by the camera, the pose estimation platform can monitor performance of the exercises by estimating the pose of the user over time.

[0034] As mentioned above, the pose estimation platform could alternatively estimate poses in contexts that are unrelated to healthcare, for example, to improve technique. As an example, the pose estimation platform may estimate the pose of an individual while she completes a sporting activity (e.g., performs a dance move, performs a yoga move, shoots a basketball, throws a baseball, swings a golf club), a cooking activity, an art activity, etc.Accordingly, while embodiments may be described in the context of a user who completes an exercise during a session as part of a program, the features of those embodiments may be similarly applicable to individuals performing other types of physical activities. Individuals whose performances of physical activities are analyzed may be referred to as “users” of the pose estimation platform, even if these individuals have little to no opportunity to interact with the pose estimation platform.

[0035] Figure 2 illustrates a network environment 200 that includes a pose estimation platform 202 that is executed by a computing device 204. Users can interact with the pose estimation platform 202 via interfaces 206. For example, users may be able to access interfaces that are designed to guide them through physical activities, indicate progress, present feedback, etc. As another example, users may be able to access interfaces through which information regarding completed physical activities can be reviewed, feedback can be provided, etc. Thus, interfaces 206 may serve as informative spaces, or the interfaces 206 may serve as collaborative spaces through which users and coaches can communicate with one another.

[0036] As shown in Figure 2, the pose estimation platform 202 may reside in a network environment 200. Thus, the computing device on which the pose estimation platform 202 is executing may be connected to one or more networks 208A-B. Depending on its nature, the computing device 204 could be connected to a personal area network (“PAN”), local area network (“LAN”), wide area network (“WAN”), metropolitan area network (“MAN”), or cellular network. For example, if the computing device 204 is a mobile phone, then the computing device 204 may be connected to a computer server of a server system 210 via the Internet. As another example, if the computing device 204 is a computer server, then the computing device 204 may be accessible to users via respective computing devices that are connected to the Internet via LANs.

[0037] The interfaces 206 may be accessible via a web browser, desktop application, mobile application, or another form of computer program. For example, to interact with the pose estimation platform 202, a user may initiatea web browser on the computing device 204 and then navigate to a web address associated with the pose estimation platform 202. As another example, a user may access, via a desktop application or mobile application, interfaces that are generated by the pose estimation platform 202 through which she can select physical activities to complete, review analyses of her performance of the physical activities, and the like. Accordingly, interfaces generated by the pose estimation platform 202 may be accessible via various computing devices, including mobile phones, tablet computers, desktop computers, wearable electronic devices (e.g., watches or fitness accessories), virtual reality systems, augmented reality systems, and the like.

[0038] Generally, the pose estimation platform 202 is hosted, at least partially, on the computing device 204 that is responsible for generating the digital images to be analyzed, as further discussed below. For example, the pose estimation platform 202 may be embodied as a mobile application executing on a mobile phone or tablet computer. In such embodiments, the instructions that, when executed, implement the pose estimation platform 202 may reside largely or entirely on the mobile phone or tablet computer. Note, however, that the mobile application may be able to access a server system 210 on which other aspects of the pose estimation platform 202 are hosted.

[0039] In some embodiments, aspects of the pose estimation platform 202 are executed by a cloud computing service operated by, for example, Amazon Web Services®, Google Cloud Platform™, or Microsoft Azure®. Accordingly, the computing device 204 may be representative of a computer server that is part of a server system 210. Often, the server system 210 comprises multiple computer servers. These computer servers can include information regarding different physical activities; computer-implemented models (or simply “models”) that indicate how anatomical regions should move when a given physical activity is performed; computer-implemented templates (or simply “templates”) that indicate how anatomical regions should be positioned when partially or fully engaged in a given physical activity; algorithms for processing image data from which spatial position of anatomical regions can be computed, inferred, or otherwise determined; user data such as name, age,weight, ailment, enrolled program, duration of enrollment, and number of physical activities completed; and other assets.

[0040] Figure 3 illustrates an example of a computing device 300 that is able to execute a pose estimation platform 312. As mentioned above, the pose estimation platform 312 can facilitate the performance of physical activities by a user, for example, by providing instruction or encouragement. As shown in Figure 3, the computing device 300 can include a processor 302, memory 304, display mechanism 306, communication module 308, image sensor 310A, audio output mechanism 322, and audio input mechanism 324. Each of these components is discussed in greater detail below.

[0041] Those skilled in the art will recognize that different combinations of these components may be present depending on the nature of the computing device 300. For example, if the computing device 300 is a computer server that is part of a server system (e.g., server system 210 of Figure 2), then the computing device 300 may not include the display mechanism 306, image sensor 310A, audio output mechanism 322, or audio input mechanism 324, though the computing device 300 may be communicatively connectable to another computing device that does include a display mechanism, an image sensor, an audio output mechanism, or an audio input mechanism.

[0042] The processor 302 can have generic characteristics similar to general-purpose processors, or the processor 302 may be an applicationspecific integrated circuit (“ASIC”) that provides control functions to the computing device 300. As shown in Figure 3, the processor 302 can be coupled to all components of the computing device 300, either directly or indirectly, for communication purposes.

[0043] The memory 304 may be comprised of any suitable type of storage medium, such as static random-access memory (“SRAM”), dynamic randomaccess memory (“DRAM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, or registers. In addition to storing instructions that can be executed by the processor 302, the memory 304 can also store data generated by the processor 302 (e.g., when executing themodules of the pose estimation platform 312) and produced, retrieved, or obtained by the other components of the computing device 300. For example, data received by the communication module 308 from a source external to the computing device 300 (e.g., image sensor 310B) may be stored in the memory 304, or data produced by the image sensor 310A may be stored in the memory 304. Note that the memory 304 is merely an abstract representation of a storage environment. The memory 304 could be comprised of actual integrated circuits (also referred to as “chips”).

[0044] The display mechanism 306 can be any mechanism that is operable to visually convey information to a user. For example, the display mechanism 306 may be a panel that includes light-emitting diodes (“LEDs”), organic LEDs, liquid crystal elements, or electrophoretic elements. In some embodiments, the display mechanism 306 is touch sensitive. Thus, a user may be able to provide input to the pose estimation platform 312 by interacting with the display mechanism 306. Alternatively, the user may be able to provide input to the pose estimation platform 312 through some other control mechanism.

[0045] The communication module 308 may be responsible for managing communications external to the computing device 300. For example, the communication module 308 may be responsible for managing communications with other computing devices (e.g., server system 210 of Figure 2, or a camera peripheral such as video camera or webcam). The communication module 308 may be wireless communication circuitry that is designed to establish communication channels with other computing devices. Examples of wireless communication circuitry include 2.4 gigahertz (“GHz”) and 5 GHz chipsets compatible with Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 - also referred to as “Wi-Fi chipsets.” Alternatively, the communication module 308 may be representative of a chipset configured for Bluetooth®, Near Field Communication (“NFC”), and the like. Some computing devices - like mobile phones and tablet computers - are able to wirelessly communicate via separate channels. Accordingly, the communication module 308 may be one of multiple communication modulesimplemented in the computing device 300. As an example, the communication module 308 may initiate and then maintain one communication channel with a camera peripheral (e.g., via Bluetooth), and the communication module 308 may initiate and then maintain another communication channel with a server system (e.g., via the Internet).

[0046] The nature, number, and type of communication channels established by the computing device 300 - and more specifically, the communication module 308 - may depend on the sources from which data is received by the pose estimation platform 312 and the destinations to which data is transmitted by the pose estimation platform 312. Assume, for example, that the computing device 300 is representative of a mobile phone or tablet computer that is associated with (e.g., owned by) a user. In some embodiments the communication module 308 may only externally communicate with a computer server, while in other embodiments the communication module 308 may also externally communicate with a source from which to receive image data. The source could be another computing device (e.g., a mobile phone or camera peripheral that includes an image sensor 310B) to which the mobile device is communicatively connected. Image data could be received from the source even if the mobile phone generates its own image data. Thus, image data could be acquired from multiple sources, and these image data may correspond to different perspectives of the user performing a physical activity. Regardless of the number of sources, image data - or analyses of the image data - may be transmitted to the computer server for storage in a digital profile that is associated with the user. The same may be true if the pose estimation platform 312 only acquires image data generated by the image sensor 310A. The image data may initially be analyzed by the pose estimation platform 312, and then the image data - or analyses of the image data - may be transmitted to the computer server for storage in the digital profile.

[0047] The image sensor 310A may be any electronic sensor that is able to detect and convey information in order to generate images, generally in the form of image data (also called “pixel data”). Examples of image sensorsinclude charge-coupled device (“CCD”) sensors and complementary metal- oxide semiconductor (“CMOS”) sensors. The image sensor 31 OA may be part of a camera module (or simply “camera”) that is implemented in the computing device 300. In some embodiments, the image sensor 31 OA is one of multiple image sensors implemented in the computing device 300. For example, the image sensor 31 OA could be included in a front- or rear-facing camera on a mobile phone. Alternatively, the image sensor 31 OA may be externally connected to the computing device 300 such that the image sensor 310A captures image data of an environment and sends the image data to the pose estimation platform 312.

[0048] For convenience, the pose estimation platform 312 may be referred to as a computer program that resides in the memory 304. However, the pose estimation platform 312 could be comprised of hardware or firmware in addition to, or instead of, software. In accordance with embodiments described herein, the pose estimation platform 312 may include a processing module 314, pose estimating module 316, analysis module 318, and graphical user interface (“GUI”) module 320. These modules can be an integral part of the pose estimation platform 312. Alternatively, these modules can be logically separate from the pose estimation platform 312 but operate “alongside” it. Together, these modules may enable the pose estimation platform 312 to programmatically monitor motion of users during the performance of physical activities, such as exercises, through analysis of digital images generated by the image sensor.

[0049] The processing module 314 can process image data obtained from the image sensor 310A over the course of a session. The image data may be used to infer a spatial position or orientation of one or more anatomical regions as further discussed below. The image data may be representative of a series of digital images. These digital images may be discretely captured by the image sensor 310A over time, such that each digital image captures the user at different stages of performing a physical activity. In some embodiments, these digital images may be representative of frames of avideo that is captured by the image sensor 310. In such embodiments, the image data could also be called “video data.”

[0050] The image data may be used to infer a spatial position of one or more anatomical regions as further discussed below. For example, the processing module 314 may perform operations (e.g., filtering noise, changing contrast, reducing size) to ensure that the data can be handled by the other modules of the pose estimation platform 312. As another example, the processing module 314 may temporally align the data with data obtained from another source (e.g., another image sensor) if multiple data are to be used to establish the spatial position of the anatomical regions of interest.

[0051] Moreover, the processing module 314 may be responsible for processing information input by users through interfaces generated by the GUI module 320. For example, the GUI module 320 may be configured to generate a series of interfaces that are presented in succession to a user as she completes physical activities as part of a session. On some or all of these interfaces, the user may be prompted to provide input. For example, the user may be requested to indicate (e.g., via a verbal command or tactile command provided via, for example, the display mechanism 306) that she is ready to proceed with the next physical activity, that she completed the last physical activity, that she would like to temporarily pause the session, etc. These inputs can be examined by the processing module 314 before information indicative of these inputs is forwarded to another module.

[0052] The pose estimating module 316 (or simply “estimating module”) may be responsible for estimating the pose of the user through analysis of image data, in accordance with the approach further discussed below. Specifically, the pose estimating module 316 can create, based on a digital image (e.g., generated by the image sensor 310A or image sensor 310B), a skeletal frame that specifies a spatial position of each of multiple anatomical regions. For example, the pose estimating module 316 can apply a computer- implemented model (or simply “model”) called a pose estimator to the digital image, so as to produce the skeletal frame. In some embodiments the pose estimator is designed and trained to identify a predetermined number and / ortype of anatomical regions (e.g., left and right wrist, left and right elbow, left and right shoulder, left and right hip, left and right knee, left and right ankle, or any combination thereof), while in other embodiments the pose estimator is designed and trained to identify all anatomical regions of a certain type (e.g., all joints) that are visible in the digital image provided as input. As discussed in greater detail below, the pose estimator may estimate poses based on joint positions using intermediate keypoints to improve accuracy. The pose estimator could be a neural network that, when applied to the digital image, analyzes the pixels to independently identify digital features that are representative of each anatomical region of interest.

[0053] The analysis module 318 may be responsible for establishing the locations of anatomical regions of interest based on the outputs produced by the pose estimating module 316. Referring again to the aforementioned examples, the analysis module 318 could establish the locations of joints based on an analysis of the skeletal frame. Moreover, the analysis module 318 may be responsible for determining appropriate feedback for the user based on the outputs produced by the pose estimating module 316, in accordance with the approach further discussed below. Specifically, the analysis module 318 may determine an appropriate personalized recommendation for the user based on her current position and a determination as to how her current position compares to a template that is associated with the physical activity that she has been instructed to perform.

[0054] Other modules could also be included in some embodiments. For example, the pose estimation platform 312 may include a training module (not shown) that is responsible for training the pose estimator that is employed by the pose estimating module 316. As another example, the pose estimation platform 312 may include a template generating module (not shown) that is responsible for generating templates that are used by the analysis module 318 to determine which recommendations, if any, are appropriate for a user given her current position.

[0055] Similarly, other components could be implemented in, or accessible to, the computing device 300 in some embodiments. For example, someembodiments of the computing device 300 include an audio output mechanism 322 and / or an audio input mechanism 324. The audio output mechanism 322 may be any apparatus that is able to convert electrical impulses into sound. One example of an audio output mechanism is a loudspeaker (or simply “speaker”). Meanwhile, the audio input mechanism 324 may be any apparatus that is able to convert sound into electrical impulses. One example of an audio input mechanism is a microphone. Together, the audio output and input mechanisms 322 and 324 may enable feedback, such as personalized recommendation as further discussed below, to be audibly provided to the user. Assume, for example, that the user has been instructed to perform a physical activity while being recorded by the image sensor 310A. In such a scenario, the user may be audibly encouraged - in a personalized manner - via the audio output mechanism 322.Estimating Poses Using Intermediate Keypoints

[0056] Earlier attempts to improve the robustness of pose estimation platforms include the explicit computation of 2D bone maps, which encode the direction of the connections between keypoints in a separate map for each coordinate. In addition to a computational increase, such approaches also require expensive off-network decoding stages that enable fully leveraging the benefits of the maps. Consequently, such approaches are inefficient. Introduced here is an approach for training machine learning models to rely on intermediate key points between anatomical locations when predicting the locations of those anatomical locations.

[0057] First, the pose estimation platform may obtain a training dataset that includes digital images. The digital images may each include an individual who is performing a physical activity. In some embodiments, the digital images may relate to a context for which the pose estimation platform is training a model. For example, the digital images may relate to a particular physical activity (e.g., yoga) or other activity (e.g., cooking). In some embodiments, the training dataset may include labels. For example, in order for the learning algorithm to create an accurate model, the training dataset must contain the answer to be predicted - commonly called the “target.” Inthis scenario, the labels can be used to indicate positions of joints of each individual. Through the learning process, the learning algorithm can discover patterns in the training dataset, and the learning algorithm can output the model that captures these patterns.

[0058] In some embodiments, the pose estimation platform may obtain a training file that specifies locations of a plurality of intermediate points, each of which is located between a different pair of joints. Such keypoints are defined as “intermediate” because they are located between two already existing keypoints, and they are computed dynamically during training. For example, the intermediate keypoints between two joints may approximate the bone connecting the two joints. The intermediate keypoints move rigidly in the context of the other keypoints. The model may use these intermediate keypoints to aggregate further visual evidence towards the prediction of a particular keypoint in a given location, as it has learned to also be aware of the areas in between keypoints. For example, the intermediate key points allow the model to “walk” between pairs of joints during the training operation. In some embodiments, the training file may specify multiple intermediate keypoints between each pair of joints. In the extreme case, if the training file were to specify a very large number of intermediate keypoints between two joints, the model would be predicting a discrete sampling of the actual bone connecting the two joints.

[0059] The pose estimation platform may provide the training dataset and the training file to a machine learning algorithm as input, so as to produce a model that is able to estimate poses based on an analysis of locations of joints and intermediate points. The intermediate keypoints are not exposed by the model, which retains its original set of keypoints as output. Instead, they are predicted internally and disregarded during inferencing. During training, however, the intermediate keypoints are trained as one would train any keypoint. By tasking the model with predicting such intermediate keypoints, the pose estimation platform forces the model to be more conscious of the location of the original keypoints, as there is a strong correlation between the original keypoints and the corresponding intermediate keypoints. In someembodiments, the machine learning model may be trained to estimate poses based on locations of joints and intermediate points. For example, the estimated poses may be digital representations of the individuals pictured in the training dataset.

[0060] Figure 4 illustrates a training flow for estimating poses using intermediate keypoints. For example, digital image 401 may represent a digital image included in the training dataset. The digital image 401 may include labels 402 indicating the locations of joints of the pictured individual. The training file may specify a number of intermediate keypoints between each pair of joints. For example, the pose estimation platform may add intermediate keypoints 404, as shown in digital image 403. The pose estimation platform may then input digital image 403 into a machine learning algorithm to train a model to predict the keypoints. For example, digital image 405 may represent the predicted keypoints. The model may generate predicted original keypoints 406 (e.g., predicting the original labels) as well as predicted intermediate keypoints 408 (e.g., predicting the intermediate keypoints). In some embodiments, the intermediate keypoints may be disregarded when outputting predictions during the inferencing stage.

[0061] Evaluating whether the model performs as expected is an important part of the training process. The digital images to which the model is applied during the inferencing stage has an unknown target, and therefore it is important to check the accuracy of the model on data for which the target is known. That is where the validating dataset can be used, namely, as a means of assessing accuracy as a proxy for predictive accuracy on digital images.

[0062] To properly evaluate the model, the pose estimation platform can hold onto a portion of the original dataset - namely, the validating dataset - that is used only for validation. Evaluating the predictive accuracy of the model with the same training dataset supplied to the learning algorithm is not useful, as the model would be rewarded for remembering the training dataset rather than learning to generalize from it. Specifically, digital images in the validating dataset can be supplied to the model that produces, as output, estimated poses. Predicted joint locations can then be compared to thelabeled joint locations created for those digital images in order to establish a metric that indicates how well the model performs. The metric may be representative of an indication of how well the predictions output by the model match the labels.

[0063] In response to a determination that the metric exceeds a threshold - indicating that performance of the model is sufficient - the pose estimation platform can store the now-validated model in memory (e.g., memory 906 of FIG. 9). In response to a determination that the metric does not exceed the threshold - indicating that performance of the model is not sufficient - the pose estimation platform may perform some other action. In some embodiments, the pose estimation platform may cause the model to update the model based on the accuracy. For example, the pose estimation platform may determine a first accuracy of first locations (e.g., the original keypoints) and a second accuracy of second locations (e.g., the intermediate keypoints). Based on determining that the first accuracy, the second accuracy, or both do not exceed a threshold, the pose estimation platform may cause the model to update. For example, the pose estimation platform may initiate a retraining operation in which aspects (e.g., weights) of the model are adjusted to see whether performance can be improved.

[0064] In some embodiments, the pose estimation platform may use a loss function specifically tailored to train the intermediate keypoints. Traditionally, keypoints may be supervised by the distance between the predicted and the ground-truth keypoints. This loss is capable of making the model learn accurate locations for the intermediate keypoints, which already leads to improvements in localizing the original keypoints. However, in this context, the most important property of the intermediate keypoints is being located on the “bone” (e.g., straight line) uniting two adjacent keypoints, rather than the exact distance to either of the two original joints. For example, it is not imperative that the predicted intermediate keypoints be equidistant from both corresponding original keypoints. Rather, the predicted intermediate keypoints should be located on the straight line connecting two adjacent keypoints. Thus, the pose estimation platform may use a complementary loss, e.g.,“distance-to-bone,” that penalizes how far the predicted intermediate keypoint is from the line uniting the joints. Therefore, this loss focuses on training the model to predict locations of the intermediate keypoints that fall on the line connecting two original keypoints, rather than being equidistant from both corresponding original keypoints (e.g., falling at the midpoint between both corresponding original keypoints). In some embodiments, the pose estimation platform may use the distance-to-bone loss (e.g., with its contribution weighted by 0.1) alongside a traditional loss used to train all keypoints.

[0065] Figure 5 illustrates traditional loss and distance-to-bone loss during training. For example, Figure 5 includes the original keypoints 502 and the bone 508 (e.g., straight line) connecting the original keypoints 502. The target intermediate keypoint 504 may be the location that a model is attempting to predict. The predicted intermediate keypoint 506 may be the location that the model actually predicts. The traditional loss 510 may be a distance between the predicted intermediate keypoint 506 and the target intermediate keypoint 504. The distance-to-bone loss 512 may be a distance from the predicted intermediate keypoint 506 to the closest point on the bone 508. In some embodiments, the pose estimation platform may use a combination of the distance-to-bone loss 512 (e.g., with its contribution weighted by 0.1) alongside the traditional loss 510 to train the model to more accurately predict the locations of intermediate keypoints.

[0066] After training, the pose estimation platform may be tasked with implementing the model. The pose estimation platform may obtain a digital image of a person who is performing a physical activity. For example, the digital image may show the person performing the same physical activity as the individuals pictured in the training dataset or a different physical activity. The pose estimation platform may provide the digital image to the machine learning model as input. For example, the machine learning model is trained to estimate poses based on locations of joints and intermediate points predicted for the digital image input. In some embodiments, inputting the digital image may cause the machine learning model to generate an estimated pose of the person based on an analysis of locations of the jointsand intermediate points. In some embodiments, the machine learning model may only predict positions for the joints and may disregard the intermediate keypoints. At a high level, the joint locations output by the model are representative of labels that are predicted by the model through an analysis of the digital image.

[0067] Figure 6 illustrates an inferencing flow for estimating poses using intermediate keypoints. For example, Figure 6 includes a digital image 601 illustrating a person performing a physical activity. In some embodiments, the digital image 601 may be an image that was not included in the training dataset. In some embodiments, the physical activity performed by the person in the digital image 601 may be the same as the physical activity performed by individuals pictured in the training dataset. The pose estimation platform may apply the trained model to the digital image 601. The model may predict the original keypoints 606 and the intermediate keypoints 608. The model may output the original keypoints 606 and may disregard the intermediate keypoints 608, as shown in digital image 603.

[0068] In some embodiments, the model may generate an estimated pose (e.g., a digital representation of the individual’s pose) based on the predicted keypoints. For example, Figure 7 illustrates a digital image 701 and an estimated pose 703. The pose estimation platform may input the digital image 701 into the model to cause the model to predict the joint locations (e.g., as shown in the digital image 701) and generate the estimated pose 703. Figure 8 illustrates a digital image 801 and an estimated pose 803. The pose estimation platform may input the digital image 801 into the model to cause the model to predict the joint locations (e.g., as shown in the digital image 801) and to generate the estimated pose 803.

[0069] The pose estimation platform may determine, based on the estimated pose, a current state of performance of the physical activity. For example, for each of the estimated poses, the pose estimation platform can compare that estimated pose to some or all of the states defined in a template and then identify a given state that is most similar to that estimated pose. By doing this in an ongoing manner, the motion monitoring platform canestablish, in real time, a current state of the individual in performing the physical activity. Based on the current state, the pose estimation platform can identify appropriate feedback to convey to the individual. Because this feedback is tailored to the current state, it is more likely to be effective in achieving its goal (e.g., improving performance of the physical activity or improving adherence to a program requiring completion of sessions over time). In some embodiments, the pose estimation platform may cause digital presentation of the appropriate feedback via a device associated with the person.

[0070] In some embodiments, the pose estimation platform may adjust a visual representation of the person to reflect the current state of performance of the physical activity. For example, the visual representation may include an estimated pose (e.g., the estimated pose 703, as in Figure 7, or the estimated pose 803, as in Figure 8). In some embodiments, the visual representation may include an avatar or animation representing the person. In some embodiments, the visual representation may be updated to reflect the movement of the person in real time. In some embodiments, the visual representation may change colors to reflect the current state of the person. For example, the entire visual representation or certain portions of the visual representation may turn a certain color (e.g., red) to reflect a current state that is incorrect, dangerous, harmful, or otherwise negative. In some embodiments, the entire visual representation or certain portions of the visual representation may turn a certain color (e.g., green) to reflect a current state that is correct, beneficial, or otherwise positive.Processing System

[0071] Figure 9 includes a block diagram illustrating an example of a processing system 900 in which at least some operations described herein can be implemented. For example, components of the processing system 900 may be hosted on a computing device that includes a pose estimation platform (e.g., pose estimation platform 202 of Figure 2 or pose estimation platform 312 of Figure 3).

[0072] The processing system 900 can include a processor 902, main memory 906, non-volatile memory 910, network adapter 912, video display mechanism 918, input / output devices 920, control device 922 (e.g., a keyboard or pointing device such as a computer mouse or trackpad), drive unit 924 including a storage medium 926, and signal generation device 930 that are communicatively connected to a bus 916. The bus 916 is illustrated as an abstraction that represents one or more physical buses or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus 916, therefore, can include a system bus, a Peripheral Component Interconnect (“PCI”) bus or PCI-Express bus, a HyperTransport (“HT”) bus, an Industry Standard Architecture (“ISA”) bus, a Small Computer System Interface (“SCSI”) bus, a Universal Serial Bus (“USB”) data interface, an Inter-Integrated Circuit (“l2C”) bus, or a high-performance serial bus developed in accordance with Institute of Electrical and Electronics Engineers (“IEEE”) 1394.

[0073] While the main memory 906, non-volatile memory 910, and storage medium 926 are shown to be a single medium, the terms “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized / distributed database and / or associated caches and servers) that store one or more sets of instructions 928. The terms “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the processing system 900.

[0074] In general, the routines executed to implement the embodiments of the disclosure can be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically comprise one or more instructions (e.g., instructions 904, 908, 928) set at various times in various memory and storage devices in a computing device. When read and executed by the processor 902, the instruction(s) cause the processing system 900 to perform operations to execute elements involving the various aspects of the present disclosure.

[0075] Further examples of machine- and computer-readable media include recordable-type media, such as volatile memory and non-volatile memory 910, removable disks, hard disk drives, and optical disks (e.g., Compact Disk Read-Only Memory (“CD-ROMs”) and Digital Versatile Disks (“DVDs”)), and transmission-type media, such as digital and analog communication links.

[0076] The network adapter 912 enables the processing system 900 to mediate data in a network 914 with an entity that is external to the processing system 900 through any communication protocol supported by the processing system 900 and the external entity. The network adapter 912 can include a network adaptor card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, bridge router, a hub, a digital media receiver, a repeater, or any combination thereof.Remarks

[0077] The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated.

[0078] Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments can vary considerably in their implementation details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted toany specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the embodiments.

[0079] The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.

Claims

CLAIMSWhat is claimed is:1 . A method comprising: obtaining a training dataset that includes a plurality of digital images, each of which includes an individual who is performing a physical activity and a first plurality of labels indicating positions of a plurality of joints; obtaining a training file that specifies locations of a plurality of intermediate points, each of which is located between a different pair of the plurality of joints; providing the training dataset and the training file to a machine learning algorithm as input, so as to produce a model that is able to estimate poses based on an analysis of locations of joints and intermediate points; obtaining a digital image of a person who is performing the physical activity; providing the digital image to the model as input, so as to cause the model to generate an estimated pose of the person based on an analysis of locations of the plurality of joints and the plurality of intermediate points while only predicting positions for the plurality of joints; determining, based on the estimated pose, a current state of performance of the physical activity; and adjusting a visual representation of the person to reflect the current state of performance of the physical activity.

2. The method of claim 1 , further comprising causing the model to update based on a first accuracy of predicted positions for the plurality of joints and a second accuracy of predicted locations of the plurality of intermediate points, wherein the second accuracy is based on a first distance between each predicted location of a respective intermediate point and a straight line connecting a respective pair of joints.

3. The method of claim 2, wherein the second accuracy is further based on a second distance between each predicted location of the respective intermediate point and a midpoint between the respective pair of joints.

4. The method of claim 3, further comprising determining the second accuracy based on a weighting of the first distance and the second distance to determine the second accuracy.

5. The method of claim 2, wherein the model is trained to rely on the predicted locations of the plurality of intermediate points to determine the predicted positions for the plurality of joints.

6. The method of claim 1 , further comprising: identifying appropriate feedback for the person based on the estimated pose; and causing digital presentation of the appropriate feedback via a device associated with the person.

7. The method of claim 1 , wherein the current state is determined from among a set of states, and wherein each state in the set of states corresponds to a different temporal position or a different spatial position.

8. The method of claim 1 , wherein the estimated pose is representative of a collection of predicted positions for the plurality of joints.

9. The method of claim 1 , wherein adjusting the visual representation of the person to reflect the current state of performance of the physical activity involves adjusting one or more colors of the visual representation to reflect the current state of performance of the physical activity.

10. The method of claim 1 , wherein the digital image is generated by a camera directed at the person.

11. A system for training machine learning models to predict joint locations, the system comprising: one or more processors communicatively coupled to a storage device, wherein the one or more processors execute instructions that are stored in the storage device to cause the system to: obtain a training dataset that includes a plurality of images, each of which includes an individual who is performing a physical activity and a plurality of labels indicating positions of a plurality of joints; obtain a training file that specifies locations of a plurality of intermediate points, each of which is located between a different pair of the plurality of joints; provide the training dataset and the training file to a machine learning algorithm as input, so as to produce a model that is able to predict first locations of the plurality of joints and second locations of the plurality of intermediate points; and cause the model to update based on a first accuracy of the first locations and a second accuracy of the second locations.

12. The system of claim 11 , wherein the second accuracy is based on a first distance between each predicted location of a respective intermediate point and a straight line connecting a respective pair of joints.

13. The system of claim 12, wherein the second accuracy is further based on a second distance between each predicted location of the respective intermediate point and a midpoint between the respective pair of joints.

14. The system of claim 13, wherein the instructions further cause the system to determine the second accuracy based on a weighting of the first distance and the second distance to determine the second accuracy.

15. The system of claim 11 , wherein the model is trained to rely on predicted locations of the plurality of intermediate points to determine predicted positions for the plurality of joints.

16. One or more non-transitory media with instructions stored thereon that, when executed by a processor of a computing device, cause the computing device to perform operations comprising: obtaining a digital image of a person who is performing a physical activity; providing the digital image to a machine learning model so as to cause the machine learning model to generate an estimated pose of the person based on an analysis of locations of a plurality of joints and a plurality of intermediate points while only predicting positions for the plurality of joints, wherein the machine learning model is trained to estimate poses based on locations of joints and intermediate points; determining, based on the estimated pose, a current state of performance of the physical activity; and adjusting a visual representation of the person to reflect the current state of performance of the physical activity.

17. The one or more non-transitory media of claim 16, further comprising: identifying appropriate feedback for the person based on the estimated pose; and causing digital presentation of the appropriate feedback via a device associated with the person.

18. The one or more non-transitory media of claim 16, wherein the current state is determined from among a set of states, and wherein each state in the set of states corresponds to a different temporal position or a different spatial position.

19. The one or more non-transitory media of claim 16, wherein the estimated pose is representative of a collection of predicted positions for the plurality of joints.

20. The one or more non-transitory media of claim 16, wherein adjusting the visual representation of the person to reflect the current state of performance of the physical activity involves adjusting one or more colors of the visual representation to reflect the current state of performance of the physical activity.

Images