Perspective-corrected, three-dimensional annotation tool for labeling anatomical positions of living bodies in digital images

Abstract

Introduced here is an approach employed by a computer program to facilitate more accurate labeling of the three-dimensional (3D) spatial locations of anatomical landmarks of a living body in a digital image. With the approach disclosed herein, annotators can more easily and accurately label anatomical landmarks, leading to better training datasets that, in turn, lead to the development of more accurate pose estimators. Historically, significant amounts of training data were necessary to train pose estimators. With these conventional computer programs, it was not unusual for anatomical landmarks to be unlabeled or mislabeled. In order to be "safe," training datasets were expanded to ensure that appropriate patterns could be learned. With the approach disclosed herein, annotators can be more confident in their labels and, as such, the volume of training data—and therefore, the time and computational resources necessary for the training operation—can be lessened.

Description

PATENTAttorney Docket No. 125847.8032. WO01PERSPECTIVE-CORRECTED, THREE-DIMENSIONAL ANNOTATION TOOL FOR LABELING ANATOMICAL POSITIONS OF LIVING BODIES IN DIGITAL IMAGESTECHNICAL FIELD

[0001] Various embodiments concern computer programs and associated computer-implemented techniques for facilitating the labeling of anatomical positions of living bodies in digital images.BACKGROUND

[0002] Pose estimation (also called “pose detection” or “pose prediction”) is an active area of study in the field of computer vision. Over the last several years, tens — if not hundreds — of different approaches have been proposed in an effort to solve the problem of pose estimation. Many of these approaches rely on machine learning due to its programmatic approach to learning what constitutes a pose.

[0003] As a field of artificial intelligence, computer vision enables machines to perform image processing tasks with the aim of imitating human vision. Pose estimation is an example of a computer vision task that generally involves using a machine learning model (or simply “model”) to detect, associate, and track the movements of a person. This is commonly done by training the model to identify “key points” that are semantically important to understanding pose. Examples of key points include “head,” “left shoulder,” “right shoulder,” “left knee,” and “right knee.” Insights into posture and movement can be drawn from analysis of these key points.11805473883PATENTAttorney Docket No. 125847.8032. WO01BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 illustrates a network environment that includes a motion monitoring platform that is executed by a computing device.

[0005] Figure 2 illustrates an example of a computing device that is able to execute a motion monitoring platform.

[0006] Figure 3 includes a schematic diagram of an approach to facilitating labeling of anatomical positions of living bodies in digital images with a computer program executing on a computing device.

[0007] Figures 4A-C illustrate an exemplary computer program interface for facilitating labeling of anatomical positions of living bodies in digital images.

[0008] Figure 5 includes a high-level diagrammatic illustration of a process for labeling of anatomical positions of living bodies in digital images.

[0009] Figures 6A-B include illustrations of different field of view projections used for placing a virtual camera onto a two-dimensional digital image of a living body.

[0010] Figure 7 includes an illustration of a projection of a two-dimensional digital image of a living body onto a three-dimensional representation of the living body.

[0011] Figure 8 is a flow diagram of a process for facilitating labeling of anatomical positions of living bodies in digital images.

[0012] Figure 9 is a flow diagram of a process for facilitating labeling of anatomical positions of living bodies in digital images.

[0013] Figure 10 is a flow diagram of a process for facilitating labeling of anatomical positions of living bodies in digital images.

[0014] Figure 11 includes a block diagram illustrating an example of a processing system in which at least some operations described herein can be implemented.21805473883PATENTAttorney Docket No. 125847.8032. WO01

[0015] Various features of the technology described herein will become more apparent to those skilled in the art from a study of the Detailed Description in conjunction with the drawings. Various embodiments are depicted in the drawings for the purpose of illustration. However, those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technology. Accordingly, although specific embodiments are shown in the drawings, the technology is amenable to various modifications.31805473883PATENTAttorney Docket No. 125847.8032. WO01DETAILED DESCRIPTION

[0016] Over the last several years, significant advances have been made in the field of computer vision. This has resulted in the development of sophisticated pose estimation models (also called “pose estimators,” “pose detectors,” or “pose predictors”) that are designed to perform pose estimation in either two dimensions or three dimensions. Two-dimensional (“2D”) pose estimators predict the 2D spatial locations of key points, generally through the analysis of the pixels of a single digital image. Three-dimensional (“3D”) pose estimators predict the 3D spatial arrangement of key points, generally through the analysis of the pixels of multiple digital images, for example, consecutive frames in a video, or a single digital image in combination with another type of data generated by, for example, an inertial measurement unit (“IMU”) or Light Detection and Ranging (“LiDAR”) unit.

[0017] Pose estimators — both 2D and 3D — continue to be applied to different contexts, and as such, continue to be used to help solve different problems. One problem for which pose estimators have proven to be particularly useful is monitoring the performance of physical activities. Consider, for example, a scenario where an individual is instructed or prompted to perform a physical activity by a computer program. By applying a pose estimator to digital images of the individual, the computer program can glean insight into performance of the physical activity. Historically, the individual may have instead been asked to summarize her performance of the physical activity (e.g., in terms of difficulty); however, this type of manual feedback tends to be inaccurate and inconsistent. Due to their consistent, programmatic nature, pose estimators allow for more accurate monitoring of performances of physical activities.

[0018] In order to achieve good performance, a pose estimator must be properly designed and trained to identify key points of interest. The process of training a pose estimator generally involves providing a machine learning algorithm (or simply “learning algorithm”) with a labeled dataset from which to41805473883PATENTAttorney Docket No. 125847.8032. WO01 learn as part of a training operation. Generally, this dataset — commonly called the “training dataset” — includes the target that the pose estimator is being trained to predict. As part of the training operation, the learning algorithm discovers relationships between the target and various attributes in the training dataset. Said another way, the learning algorithm finds patterns in the training dataset that relate the attributes to the target and then outputs the pose estimator that captures these patterns.

[0019] Assume, for example, that an individual is interested in training a pose estimator to predict whether a living body has proper form while performing an exercise. In such a scenario, the individual would provide, to a learning algorithm, a training dataset that includes a series of digital images for which the target (i.e., whether form is proper or improper) is known. Each digital image may be associated with one or more informative labels (or simply “labels”) that indicate the position of anatomical landmarks. The term “anatomical landmark” may be used to refer to a biologically meaningful loci of a living body that can be unambiguously defined and repeatedly located with high accuracy. Appending labels to the digital images provides context so that a learning algorithm can learn from the training dataset and produce the pose estimator.

[0020] Many, if not most, practical pose estimators are developed through supervised learning, where a learning algorithm is provided a training dataset — with labels — from which relationships between inputs (e.g., locations of key points) and outputs (e.g., a determination of whether form is proper) can be learned in order to make correct predictions. Traditionally, these labels have been acquired in one of four ways. First, a computer program could create synthetic, rendered data in the form of a 3D avatar that a person is asked to annotate. Second, a person could be prompted to perform the exercise while wearing trackable markers in a sophisticated motion capture studio, and the labels could be fully or semi-autonomously generated based on the analysis of digital images of the performance of the exercise. Third, semi-supervised or unsupervised learning approaches, where a pose estimator learns from partially51805473883PATENTAttorney Docket No. 125847.8032. WO01 labeled or entirely unlabeled image data, can be employed to obtain 3D labels of persons from images or frames. Fourth, humans could be asked to make judgments about unlabeled digital images. For example, these humans — commonly called “labelers” or “annotators” — may be asked to label anatomical landmarks in all of the digital images in a training dataset where “pose is proper” is true. Conventional computer programs are generally not well suited for this, however. Not only do conventional computer programs lack sophistication, but they do not account for the perspective of the digital images being labeled or the proportions of the living bodies included in the digital images.

[0021] Introduced here is an approach employed by a computer program to facilitate more accurate labeling of the 3D spatial locations of anatomical landmarks of a living body in a digital image. With the approach disclosed herein, annotators can more easily and accurately label anatomical landmarks, leading to better training datasets that, in turn, lead to the development of more accurate pose estimators. Note that with the approach, training of pose estimators may also be a more resource-efficient process. Historically, significant amounts of training data were necessary to train pose estimators due to the limitations of conventional computer programs. With these conventional computer programs, it was not unusual for anatomical landmarks to be unlabeled or mislabeled. In order to be “safe,” training datasets were expanded to ensure that appropriate patterns could be learned. With the approach disclosed herein, annotators can be more confident in their labels and, as such, the volume of training data — and therefore, the time and computational resources necessary for the training operation — can be lessened.

[0022] Initially, an annotator may be prompted to label the 2D spatial locations of different anatomical landmarks on a digital image that is presented on an interface. After these anatomical landmarks have been labeled, the annotator can progress to annotating (i) an estimate of the field of the view of the camera used to capture the digital image, (ii) an estimate of the proportions of the living body being labeled, and (iii) the depth of the anatomical landmarks in 3D space.61805473883PATENTAttorney Docket No. 125847.8032. WO01As further discussed below, the input of these parameters can manipulate a rigged, anatomically correct avatar that is projected onto the digital image.

[0023] Several aspects of the approach allow highly accurate labels to be produced in a more efficient manner. First, anatomical landmarks — as identified in 2D — can be labeled on the digital image before labeling the relative depth of these anatomical landmarks, allowing the labeling process to be completed more quickly and accurately in a planar space. Second, the perspective of the camera used to capture the digital image can be modeled by estimating the field of view (“FoV”) and then reprojecting a translucent, anatomically correct avatar of the living body back onto the digital image to ensure correctness. Third, the key points labeled by the annotator can be divorced from the anatomical landmarks that are output by the computer program by treating the manually labeled key points as positional targets of important anatomical landmarks that are used to deform the rigged, anatomically correct avatar of the living body on which the anatomical landmarks are accurately and consistently placed. Fourth, the entire approach can be optimized based on continuously given inputs (e.g., target positions of anatomical landmarks, parameters of body proportions, parameters of camera, etc.) to minimize error between the 2D spatial locations established for the anatomical landmarks in the digital image and the reprojected 3D spatial locations estimated for the anatomical landmarks in connection with the avatar.

[0024] For the purpose of illustration, embodiments may be described in the context of computer-executable instructions. However, aspects of the approach could be implemented via hardware or firmware instead of, or in addition to, software. As an example, a motion monitoring platform may be embodied as a computer program that offers support for monitoring motion of living bodies in real time, for example, by applying a pose estimator to digital images of those living bodies completing exercises. Through the motion monitoring platform, annotators may also be able to label anatomical landmarks in accordance with the approach disclosed herein, and these labeled anatomical landmarks may be used, directly or indirectly, to train the pose estimator.71805473883PATENTAttorney Docket No. 125847.8032. WO01Terminology

[0025] References in the present disclosure to “an embodiment” or “some embodiments” mean that the feature, function, structure, or characteristic being described is included in at least one embodiment. Occurrences of such phrases do not necessarily refer to the same embodiment nor are they necessarily referring to alternative embodiments that are mutually exclusive of one another.

[0026] Unless the context clearly requires otherwise, the terms “comprise,” “comprising,” and “comprised of” are to be construed in an inclusive sense rather than an exclusive or exhaustive sense. That is, in the sense of “including but not limited to.” The term “based on” is also to be construed in an inclusive sense.Thus, the term “based on” is intended to mean “based at least in part on.”

[0027] The terms “connected,” “coupled,” and variants thereof are intended to include any connection or coupling between two or more elements, either direct or indirect. The connection or coupling can be physical, logical, or a combination thereof. For example, elements may be electrically or communicatively coupled to one another despite not sharing a physical connection.

[0028] The term “module” may refer broadly to software, firmware, hardware, or combinations thereof. Modules are typically functional components that generate one or more outputs based on one or more inputs. A computer program may include or utilize one or more modules. For example, a computer program may utilize multiple modules that are responsible for completing different tasks, or a computer program may utilize a single module that is responsible for completing all tasks.

[0029] When used in reference to a list of multiple items, the word “or” is intended to cover all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of items in the list.Overview of Motion Monitoring Platform81805473883PATENTAttorney Docket No. 125847.8032. WO01

[0030] A motion monitoring platform may be responsible for monitoring the motion of an individual (also called a “participant”) through analysis of digital images that contain her and are captured as she completes a physical activity. As an example, the motion monitoring platform may guide the participant through exercise therapy sessions (or simply “sessions”) that are performed as part of an exercise therapy program (or simply “program”). As part of the program, the participant may be requested to engage with the motion monitoring platform on a periodic basis. The frequency with which the participant is requested to engage with the motion monitoring platform may be based on factors such as the anatomical region for which therapy is needed, the musculoskeletal (“MSK”) condition for which therapy is needed, the difficulty of the program, the age of the participant, the amount of progress that has been achieved, and the like.

[0031] As the participant performs exercises, she may be recorded by a camera of a computing device. Normally, the camera is part of the computing device on which the motion monitoring platform is executed or accessed. For example, in order to initiate a session, the participant may initiate a mobile application that is stored on, and executable by, her mobile phone or tablet computer, and the mobile application may instruct the participant to position her mobile phone or tablet computer in such a manner that one of its cameras can record her as exercises are performed. Note that, in some embodiments, the camera is part of another computing device. For example, the camera may be included in a peripheral computing device, such as a web camera (also called a “webcam”), that is connected to the computing device. By applying a pose estimator to the digital images that are output by the camera, the motion monitoring platform can monitor performance of the exercises by estimating the pose of the participant over time.

[0032] The motion monitoring platform could alternatively estimate pose in contexts that are unrelated to healthcare, for example, to improve technique. As an example, the motion monitoring platform may apply a pose estimator to digital images that correspond to a performance of a sporting activity (e.g., a dance91805473883PATENTAttorney Docket No. 125847.8032. WO01 move, a yoga move, a basketball shot or pass, a baseball throw, a swing of a baseball bat or a golf club), a cooking activity, an art activity, etc. Accordingly, while embodiments may be described in the context of a participant who completes an exercise during a session as part of a program, the features of those embodiments may be similarly applicable to participants performing other types of physical activities. Participants whose performances of physical activities are analyzed may be referred to as “users” of the motion monitoring platform, even if these participants have little to no opportunity to interact with the motion monitoring platform.

[0033] As mentioned above, in order to achieve good performance, the pose estimator must be properly designed and trained to identify key points of interest. The process of training a pose estimator generally involves providing a learning algorithm with a labeled training dataset from which to learn. Referring again to the aforementioned example involving pose estimation in the context of digital therapy, the labeled training dataset may include digital images with corresponding labels. With the approach disclosed herein, annotators can more easily and accurately identify appropriate labels for the digital images. Annotators that are responsible for labeling digital images to be used for training purposes may also be referred to as “users” of the motion monitoring platform.

[0034] Figure 1 illustrates a network environment 100 that includes a motion monitoring platform 102 that is executed by a computing device 104. Users can interact with the motion monitoring platform 102 via interfaces 106. For example, participants may be able to access interfaces that are designed to guide them through physical activities, indicate progress, present feedback, etc. As another example, annotators may be able to access interfaces through which digital images can be viewed and labeled, as further discussed below.

[0035] As shown in Figure 1 , the motion monitoring platform 102 may reside in a network environment 100. Thus, the computing device 104 on which the motion monitoring platform 102 is executing may be connected to one or more101805473883PATENTAttorney Docket No. 125847.8032. WO01 networks 108A-B. Depending on its nature, the computing device 104 could be connected to a personal area network (“PAN”), local area network (“LAN”), wide area network (“WAN”), metropolitan area network (“MAN”), or cellular network. For example, if the computing device 104 is a mobile phone, then the computing device 104 may be connected to a computer server of a server system 110 via the Internet. As another example, if the computing device 104 is a computer server, then the computing device 104 may be accessible to users via respective computing devices that are connected to the Internet via LANs.

[0036] The interfaces 106 may be accessible via a web browser, desktop application, mobile application, or another form of computer program. For example, to interact with the motion monitoring platform 102, a user may initiate a web browser on the computing device 104 and then navigate to a web address associated with the motion monitoring platform 102. As another example, a user may access, via a desktop application or mobile application, interfaces that are generated by the motion monitoring platform 102 through which she can perform activities. The nature of the activities may depend on the type of user. For example, participants may be able to select physical activities to complete, review analyses of her performance of the physical activities, and the like, while annotators may be able to select digital images to label, identify appropriate labels for digital images, and review labels identified — either manually or automatically — for digital images. Accordingly, interfaces generated by the motion monitoring platform 102 may be accessible via different types of computer programs and different types of computing devices, including mobile phones, tablet computers, desktop computers, wearable electronic devices (e.g., watches or fitness accessories), virtual reality systems, augmented reality systems, and the like.

[0037] In some embodiments, the motion monitoring platform 102 is hosted, at least partially, on the computing device 104 that is responsible for generating, processing, or storing the digital images to be analyzed. For example, the motion monitoring platform 102 may be embodied as a mobile application executing on a111805473883PATENTAttorney Docket No. 125847.8032. WO01 mobile phone or tablet computer. In such embodiments, the instructions that, when executed, implement the motion monitoring platform 102 may reside largely or entirely on the mobile phone or tablet computer. Note, however, that the mobile application may be able to access a server system 110 on which other aspects of the motion monitoring platform 102 are hosted.

[0038] Aspects of the motion monitoring platform 102 could alternatively or additionally be executed by a cloud computing service operated by, for example, Amazon Web Services®, Google Cloud Platform™, or Microsoft Azure®. Accordingly, the computing device 104 may be representative of a computer server that is part of a server system 110. Often, the server system 110 is comprised of multiple computer servers. These computer servers can include information regarding different physical activities; models that indicate how anatomical regions should move when a given physical activity is performed; computer-implemented templates (or simply "templates”) that indicate how anatomical regions should be positioned when partially or fully engaged in a given physical activity; algorithms for processing image data from which spatial position of anatomical regions can be computed, inferred, or otherwise determined; user data such as name, age, weight, ailment, enrolled program, duration of enrollment, and number of physical activities completed; training datasets that include digital images and corresponding labels; and other assets.

[0039] Figure 2 illustrates an example of a computing device 200 that is able to execute a motion monitoring platform 212. As mentioned above, the motion monitoring platform 212 can facilitate the performance of physical activities by a participant, for example, by providing instruction as she completes physical activities. The motion monitoring platform 212 can also facilitate the labeling of digital images by an annotator by displaying the digital image for labeling and prompting the annotator to label key points on the image (e.g., anatomical landmarks) and / or fill in perspective parameters related to the camera used to capture the image (e.g., FoV, proportion of the living body in the image, depth of an anatomical landmark labeled in the image, etc.). The annotated digital image121805473883PATENTAttorney Docket No. 125847.8032. WO01 can be used to generate a perspective-corrected, 3D representation of the participant performing the physical activity depicted in the image, as described herein. As shown in Figure 2, the computing device 200 can include a processor 202, a memory 204, a display mechanism 206, a communication module 208, and an image sensor 210A. Each of these components is discussed in greater detail below.

[0040] Those skilled in the art will recognize that different combinations of these components may be present depending on the nature of the computing device 200. For example, if the computing device 200 is a computer server that is part of a server system (e.g., server system 110 of Figure 1 ), then the computing device 200 may not include the display mechanism 206 or image sensor 210A, though the computing device 200 may be communicatively connectable to another computing device that does include a display mechanism or an image sensor.

[0041] The processor 202 can have generic characteristics similar to general- purpose processors, or the processor 202 may be an application-specific integrated circuit (“ASIC”) that provides control functions to the computing device 200. As shown in Figure 2, the processor 202 can be coupled to all components of the computing device 200, either directly or indirectly, for communication purposes.

[0042] The memory 204 may be comprised of any suitable type of storage medium, such as static random-access memory (“SRAM”), dynamic randomaccess memory (“DRAM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, or registers. In addition to storing instructions that can be executed by the processor 202, the memory 204 can also store data generated by the processor 202 (e.g., when executing the modules of the motion monitoring platform 212) and produced, retrieved, or obtained by the other components of the computing device 200. For example, data received by the communication module 208 from a source external to the computing device 200131805473883PATENTAttorney Docket No. 125847.8032. WO01(e.g., image sensor 21 OB) may be stored in the memory 204, or data produced by the image sensor 210A may be stored in the memory 204. Note that the memory 204 is merely an abstract representation of a storage environment. The memory 204 could be comprised of actual integrated circuits (also referred to as “chips”).

[0043] The display mechanism 206 can be any mechanism that is operable to visually convey information to a user. For example, the display mechanism 206 may be a panel that includes light-emitting diodes (“LEDs”), organic LEDs, liquid crystal elements, or electrophoretic elements. In some embodiments, the display mechanism 206 is touch-sensitive. Thus, a user may be able to provide input to the motion monitoring platform 212 by interacting with the display mechanism 206. Alternatively, the user may be able to provide input to the motion monitoring platform 212 through some other control mechanism.

[0044] The communication module 208 may be responsible for managing communications external to the computing device 200. For example, the communication module 208 may be responsible for managing communications with other computing devices (e.g., server system 110 of Figure 1 , or a camera peripheral such as a video camera or webcam). The communication module 208 may be wireless communication circuitry that is designed to establish communication channels with other computing devices. Examples of wireless communication circuitry include 2.4 gigahertz (“GHz”) and 5 GHz chipsets compatible with Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 (also referred to as “Wi-Fi chipsets”). Alternatively, the communication module 208 may be representative of a chipset configured for Bluetooth®, Near Field Communication (“NFC”), and the like. Some computing devices — like mobile phones and tablet computers — are able to wirelessly communicate via separate channels. Accordingly, the communication module 208 may be one of multiple communication modules implemented in the computing device 200. As an example, the communication module 208 may initiate and then maintain one communication channel with a camera peripheral (e.g., via Bluetooth), and the141805473883PATENTAttorney Docket No. 125847.8032. WO01 communication module 208 may initiate and then maintain another communication channel with a server system (e.g., via the Internet).

[0045] The nature, number, and type of communication channels established by the computing device 200 — and more specifically, the communication module 208 — may depend on the sources from which data is received by the motion monitoring platform 212 and the destinations to which data is transmitted by the motion monitoring platform 212. Assume, for example, that the computing device 200 is representative of a laptop computer that is associated with (e.g., owned by) an annotator. In some embodiments, the communication module 208 may only externally communicate with a computer server, while in other embodiments the communication module 208 may also externally communicate with a source from which to receive image data. The source could be another computing device (e.g., a tablet computer, mobile phone, or camera peripheral that includes an image sensor 210B) to which the laptop computer is communicatively connected. Consider, for example, a scenario where an annotator captures a digital image of herself performing an exercise and then reviews the digital image to label anatomical landmarks in accordance with the approach disclosed herein. Regardless of the number of sources, analyses of image data may be transmitted to the computer server for storage in a digital profile. The digital profile could be associated with the exercise being performed in the digital image, the person captured in the digital image, or the annotator responsible for labeling the digital image. The same may be true if the motion monitoring platform 212 only acquires image data generated by the image sensor 210A. The image data may initially be analyzed by the motion monitoring platform 212 and then analyses of the image data may be transmitted to the computer server for storage in the digital profile.

[0046] The image sensor 210A may be any electronic sensor that is able to detect and convey information in order to generate images, generally in the form of image data (also called “pixel data”). Examples of image sensors include charge-coupled device (“CCD”) sensors and complementary metal-oxide151805473883PATENTAttorney Docket No. 125847.8032. WO01 semiconductor (“CMOS”) sensors. The image sensor 210A may be part of a camera module (or simply “camera”) that is implemented in the computing device 200. In some embodiments, the image sensor 210A is one of multiple image sensors implemented in the computing device 200. For example, the image sensor 210A could be included in a front- or rear-facing camera on a mobile phone. Additionally or alternatively, the motion monitoring platform 212 may obtain image data that is generated by an image sensor 210B that is external to the computing device 200.

[0047] For convenience, the motion monitoring platform 212 may be referred to as a computer program that resides in the memory 204. However, the motion monitoring platform 212 could be comprised of hardware or firmware in addition to, or instead of, software. In accordance with embodiments described herein, the motion monitoring platform 212 may include a processing module 214, modeling module 216, rigging module 218, training module 220, and graphical user interface (“GUI”) module 222. These modules can be an integral part of the motion monitoring platform 212. Alternatively, these modules can be logically separate from the motion monitoring platform 212 but operate “alongside” it. Together, these modules may enable the motion monitoring platform 212 to programmatically facilitate more accurate labeling of the 3D spatial locations of anatomical landmarks of living bodies in digital images.

[0048] The processing module 214 can process image data (e.g., obtained from image sensor 210A or image sensor 210B, or retrieved from memory 204 or a database external to the computing device 200). In some embodiments, the image data includes digital images that were captured independently of one another, for example, in different settings, with different living bodies contained therein, etc. In other embodiments, the image data includes a related series of digital images that were discretely captured in rapid succession such that each digital image captured a living body at different stages of performing a physical activity. In such embodiments, these digital images may be representative of frames of a video, and therefore the image data could also be called “video data.”161805473883PATENTAttorney Docket No. 125847.8032. WO01

[0049] As further discussed below, the image data — or analyses of the image data — may be used to train a pose estimator. To ensure that the image data can be readily handled by the other modules of the motion monitoring platform 212, the processing module 214 may perform operations (e.g., filtering noise, changing contrast, reducing size) on the image data. Moreover, the processing module 214 may be responsible for processing information input by users through interfaces generated by the GUI module 222. For example, the GUI module 222 may be configured to generate interfaces through which an annotator is able to identify the locations of anatomical landmarks of a living body in a digital image, estimate parameters (e.g., FoV of the camera that captured the digital image, proportions of the living body, depths of the anatomical landmarks in 3D space), and review a translucent, anatomically correct avatar projected onto the digital image. On some or all of these interfaces, the user may be prompted to provide input. For example, the user may be requested to indicate (e.g., via a tactile command provided, for example, through the display mechanism 206 or with a pointing device, such as a computer mouse) the locations of the anatomical landmarks by interacting with the digital image. As another example, the user may be requested to specify (e.g., via entry of alphanumeric characters) the parameters. These inputs can be examined by the processing module 214 before information indicative of these inputs is forwarded to another module.

[0050] As shown in Figures 4A-C, the GUI module 222 may generate an interface on which a digital image is presented for review by an annotator. Through the interface, the annotator may be prompted to label the 2D spatial locations of different anatomical landmarks on the digital image. For example, the GUI module 222 may specify, on the interface, one or more anatomical landmarks to be labeled by the annotator. Identifying these anatomical landmarks in 2D space — on the digital image — may allow the annotator to complete the labeling process more quickly and accurately because she is able to work in a planar space.171805473883PATENTAttorney Docket No. 125847.8032. WO01

[0051] In some embodiments, these anatomical landmarks are predetermined. For example, these anatomical landmarks may be representative of a predefined set (e.g., specifying head, right shoulder, left shoulder, right elbow, left elbow, etc.), only some of which may actually be visible in the digital image. In other embodiments, these anatomical landmarks are dynamically determined by the motion monitoring platform 212. For example, the motion monitoring platform 212 may apply, to the digital image, a model that produces, as output, a prediction regarding which anatomical landmarks are visible within the bounds of the digital image. In other embodiments, these anatomical landmarks are specified by the annotator. In such embodiments, the motion monitoring platform 212 may rely on the annotator to specify which landmarks are visible within the bounds of the digital image, for example, by inferring that the anatomical landmarks for which the annotator labels 2D spatial locations are the only anatomical landmarks that are visible within the bounds of the digital image.

[0052] After these anatomical landmarks have been labeled, the annotator can progress to annotating (i) an estimate of the field of the view of the camera used to capture the digital image, (ii) an estimate of the proportions of the living body being labeled, and (iii) the depth of the anatomical landmarks in 3D space. These parameters may be input through the same interface on which the digital image is presented or a different interface. Additionally or alternatively, these parameters may be estimated, inferred, or otherwise determined by the motion monitoring platform 212. For example, the modeling module 216 may model the perspective of the camera used to capture the digital image by establishing the FoV (e.g., based on the estimate from the annotator), while the rigging module 218 may programmatically associate the anatomical landmarks with corresponding segments of an anatomically correct avatar. Moreover, the rigging module 218 may cause the avatar to be projected onto the digital image such that the annotator is able to ensure that the anatomical landmarks were properly associated with the corresponding segments of the avatar. Accordingly, the181805473883PATENTAttorney Docket No. 125847.8032. WO01 rigging module 218 may be responsible for “rigging” an avatar based on the anatomical landmarks labeled by the annotator as well as other inputs provided by the annotator.

[0053] Note that the rigging module 218 may divorce the anatomical landmarks — and more specifically, the actual key points labeled by the annotator — from the avatar to some degree by treating those actual key points as positional targets for corresponding segments of the avatar. With these positional targets, the rigging module 218 can deform the avatar to “rig” it to the living body in the digital image. Such an approach to deforming the avatar in a personalized manner ensures that the segments are accurately and consistently placed on appropriate anatomical landmarks.

[0054] The training module 220, meanwhile, may be responsible for using the rigged avatar to train a pose estimator. Specifically, the training module 220 may assign the rigged avatar to be part of a labeled dataset to be used to train the pose estimator. The rigged avatar may be one of multiple rigged avatars in the labeled dataset. These rigged avatars may be associated with different digital images (e.g., different frames in the same video), different living bodies, different targets (e.g., proper form versus improper form), etc. The training module 220 could also optimize the aforementioned process of generating training data based on continuously given inputs (e.g., target positions of anatomical landmarks, parameters of body proportions, parameters of camera, etc.) to minimize error between the 2D spatial locations established for the anatomical landmarks in the digital image and the reprojected 3D spatial locations estimated for the anatomical landmarks in connection with the avatar.Approaches to Facilitating Production of Perspective-Corrected, 3D Representations

[0055] Various attempts have been made to improve the performance of pose estimators. Consider, for example, an exercise therapy program that requires exercises to be performed by an individual to achieve a therapeutic goal. To191805473883PATENTAttorney Docket No. 125847.8032. WO01 achieve good performance in such programs, it is crucial that a pose estimator associated with the therapy program is properly designed and trained to identify key points of interest to accurately assess the individual’s movements. The process of training a pose estimator generally involves providing a learning algorithm with a labeled training dataset from which to learn. In the context of digital therapy, this labeled training dataset may include digital images with corresponding labels. However, obtaining accurate and comprehensive labeled datasets can be challenging.

[0056] Introduced here is an approach to facilitating the production of perspective-corrected, 3D representations based on annotations of 2D images. This approach offers several advantages over conventional methods that do not obtain 3D labels of persons from digital images and / or frames via an annotation system or tool, including but not limited to reduced distortion of 3D representations through perspective correction and improved fidelity by using a human anatomical skeleton to guide annotation. The approach described herein enables annotators to more easily and accurately identify appropriate labels for the digital images, which are essential for training the pose estimator.

[0057] Figure 3 includes a schematic diagram of an approach to facilitating labeling of anatomical positions of living bodies in digital images with a computer program executing on a computing device.

[0058] As mentioned above, the motion monitoring platform can include a data processing component — namely, a processing module (e.g., processing module 214, as shown in Figure 2). Initially, a motion monitoring platform (e.g., motion monitoring platform 212) can retrieve image data from one or more sources (e.g., at an operation 301 , through the processing module 214 of Figure 2). For example, the motion monitoring platform can retrieve image data from one or more image sensors associated with the motion monitoring platform and used to capture images of participants performing physical activities. Additionally, the motion monitoring platform can retrieve image data from a server system201805473883PATENTAttorney Docket No. 125847.8032. WO01(e.g., the server system 110 of Figure 1 ), including images recorded and saved by the annotator through a computer program. The retrieved image data can also include perspective parameters related to the camera used to capture the image (e.g., FoV, proportion of the living body in the image, depth of an anatomical landmark labeled in the image, etc.). By retrieving the image data, the motion monitoring platform can process and format the digital image into an annotatable 2D image accessible to an annotator via an interface for annotation as described below.

[0059] The data processing (e.g., operation 302) can include processing sensor data by the processing module 214 to produce processed image data, which can be utilized by a GUI module (e.g., the GUI module 222 of Figure 2), as described later herein. For example, the processing module 214 may filter noise, change contrast, and reduce the size of the image data. To illustrate, the processing module 214 can handle digital images captured independently or in rapid succession, ensuring that they are suitable for further analysis. The motion monitoring platform can filter, align, and / or transform such sensor readings and / or stored image data to enhance the quality and usability of the data. By processing the image data, the motion monitoring platform can enable evaluation and annotation of the 2D image by an annotator to properly label and store annotated versions of the image with anatomical landmarks. For example, the processing module 214 may adjust the brightness and contrast of the images to highlight specific anatomical features, making it easier for annotators to identify and label these landmarks accurately.

[0060] Although the processes described herein may be categorized as being performed by other modules (e.g., the modeling module 216, the rigging module 218, the training module 220, the GUI module 222 of Figure 2), the processing module 214 or any of the modules described herein can be responsible for retrieving, processing, and examining the sensor data, stored image data, or annotated data received to prepare inputs for another module (e.g., the modeling211805473883PATENTAttorney Docket No. 125847.8032. WO01 module 216, the rigging module 218, the training module 220 of Figure 2), as described herein.

[0061] The GUI module 222 can be responsible for generating a computer program interface for an annotator to interact with (e.g., at operation 303). As shown in Figure 4A, the GUI module 222 can generate an exemplary computer program interface for facilitating the labeling of anatomical positions of living bodies in the digital images by the annotator. The computer program interface can include one or more interface panels that allow the annotator to load an image dataset (e.g., a plurality of digital images in a series of a living body performing a physical activity), save an annotation, reset a camera of the motion monitoring platform (e.g., to prompt an individual to perform another physical activity or reperform the physical activity and capture additional digital images), invalidate a digital image or set of digital images for various reasons (e.g., poor quality, incorrect positioning, etc.), scroll between a series of digital images to facilitate the annotation of anatomical landmarks, exit the computer program interface, and / or the like. It is worth noting that the computer program interface of Figures 4A-C is an exemplary computer program interface and can include various additional GUIs, pop-ups, panels, etc., that readily facilitate proper labeling and review of the digital image by the annotator. Therefore, the computer program interface can be generated a plurality of times to, for example, include different views (e.g., 2D view, perspective views, free views, etc.) depending on the annotation to be provided and / or reviewed, as described further herein.

[0062] In some embodiments, the computer program interface facilitates the annotator in inputting 2D labels for anatomical positions of the living body in the digital image (e.g., at operation 304). For example, as further illustrated in Figure 4A, the computer program interface enables the annotator to populate one or more text box callouts by interfacing with a pop-up generated by the GUI module 222 for labeling the anatomical landmarks. The annotator may interact with the computer program by, for example, typing on an external or digital keyboard,221805473883PATENTAttorney Docket No. 125847.8032. WO01 pressing one or more tactile buttons on the digital interface, and / or speaking into a microphone to provide audio feedback to an interface communicably coupled to the computer program for labeling the 2D image. The computer program interface can include text box callouts automatically generated based on the processed image data (e.g., by the processing module 214), highlighting certain features for the annotator to label, and / or text box callouts can be manually input by the annotator for labeling the anatomical landmarks.

[0063] Once the anatomical landmarks have been labeled by the annotator, the computer program interface can further facilitate the annotator in inputting parameter inputs related to the digital image (e.g., at operation 305). For example, the parameter inputs can include body proportions, camera parameters, and other capture parameters related to the camera used to capture the image, such as FoV, proportion of the living body in the image, depth of an anatomical landmark labeled in the image, and / or the like. As shown in Figure 4B, the computer program interface can enable the annotator to input perspective parameters associated with the digital camera used to capture the digital image by interfacing with a pop-up generated by the GUI module 222 for inputting the additional parameters. In other words, the computer program interface can facilitate requesting the annotator to specify parameter inputs (e.g., via entry of alphanumeric characters). The parameter inputs can be examined by the processing module 214 before information indicative of the inputs is forwarded to additional modules (e.g., the modeling module 216 and rigging module 218 of Figure 2). The parameters can be input through the same interface on which the digital image is presented or through a different interface. Additionally or alternatively, the parameters may be estimated, inferred, or otherwise determined by the motion monitoring platform.

[0064] As described above, the motion monitoring platform (e.g., the modeling module 216 of Figure 2) can estimate the FoV of the camera used to capture the digital image (e.g., at operation 306). In some embodiments, the modeling module 216 estimates the FoV using the parameters input by the annotator231805473883PATENTAttorney Docket No. 125847.8032. WO01 and / or stored perspective parameters associated with generally similar digital images (e.g., in the server system 110 of Figure 1 ). The perspective parameters can include focal length (e.g., the distance between the lens and the image sensor), sensor size (e.g., the dimensions of the camera’s image sensor), and distance to the subject (e.g., how far the subject is from the camera). In some embodiments, the focal length determines the magnification and angle of view, sensor size affects the FoV and depth of field, and distance to the subject influences the perspective and scale of the image. Additionally or alternatively, the modeling module 216 can model the perspective of the camera used to capture the digital image using the estimated perspective parameters. For example, the perspective of the camera can be determined by the camera’s position and orientation relative to the subject such that the 2D image can be accurately mapped onto a 3D model or avatar. The perspective of the camera can be used to rig an anatomically correct avatar onto the labeled 2D image to produce the rigged avatar, as described further herein.

[0065] The motion monitoring platform (e.g., the rigging module 218 of Figure 2) can associate the anatomical landmarks input by the annotator with corresponding segments of an anatomically correct avatar (e.g., at operation 307). For example, the rigging module 218 can map the input anatomical landmarks to corresponding joints, bones, or limbs of the avatar. The anatomically correct avatar can be associated with the physical activity being performed, the participant performing the digital activity, and / or the specific movements captured in the digital image. For example, the rigging module 218 can select an anatomically correct avatar that matches general physical characteristics associated with the participant performing the physical activity, stored and / or input into the computer program, such as size, weight, height, etc.

[0066] Once the anatomical landmarks are initially associated with the anatomically correct avatar, the rigging module 218 can reproject the anatomically correct avatar of the living body onto the digital image to ensure that the anatomical landmarks are properly aligned with the avatar (e.g., at operation241805473883PATENTAttorney Docket No. 125847.8032. WO01308). In some embodiments, the motion monitoring platform (e.g., the GUI module 222) prompts a user to verify and / or adjust the association of the anatomical landmarks with the anatomically correct avatar. The verification process can include the annotator reviewing the alignment of the anatomical landmarks with the avatar’s joints, bones, and limbs and making any necessary adjustments to ensure proper mapping, as described further herein.

[0067] The rigging module 218 can then deform the anatomically correct avatar according to the anatomical landmarks to produce a rigged avatar or 3D representation of the living body performing the physical activity in the digital image (e.g., at operation 309). For example, the key points labeled by the annotator can be treated as positional targets for rigging the anatomically correct avatar. The deformation process can include adjusting the joints, bones, and limbs of the avatar to align with the positions indicated by the key points or anatomical landmarks, ensuring that the posture and movement of the avatar correspond to those of the participant in the digital image. The parameters input by the annotator and / or generated by the motion monitoring platform (e.g., the perspective and FoV of the camera used to capture the digital image generated by the modeling module 216) can also be used to rig the anatomically correct avatar. The deformation process can further include adjusting the parameters to ensure that the avatar’s posture and movement accurately reflect those of the participant in the digital image.

[0068] The GUI module 222 can facilitate annotator review and confirmation of the rigged avatar (e.g., at operation 310). As shown in Figure 4C, the computer program interface can include adjustment controls in certain views that allow the annotator to modify various aspects of the representation. The adjustment controls can enable the annotator to modify the scale of the skeleton, change the FoV, adjust the position and orientation of the avatar along the X, Y, and Z axes, scale the target, change the transparency of the FoV, the skeleton, or the digital image, and / or display or hide specific reference features, including truth nodes, lines, targets, avatars, and / or the digital image itself. The computer251805473883PATENTAttorney Docket No. 125847.8032. WO01 program interface provides comprehensive control to the annotator, ensuring that the rigged avatar of the living body accurately reflects a 3D representation of the physical activity being performed in the digital image.

[0069] As described above with reference to operations 307-309, in some embodiments, the annotator reviews and confirms the rigged avatar in a series of steps. The computer program interface can, for example, facilitate the annotator in reviewing and confirming the association of the avatar with the anatomical landmarks, adjusting one or more perspective parameters, deforming the avatar by divorcing the anatomical landmarks using the key points as positional targets for mapping the avatar onto the digital image to create the rigged avatar, and / or performing intermediary tasks. The GUI module 222 can generate one or more views or perspectives of the rigged avatar produced and / or the intermediate steps performed by the rigging module 218 on the computer program interface between steps and / or once modifications are made (e.g., using the adjustment controls, etc.).

[0070] The motion monitoring platform (e.g., the training module 220) can assign the rigged avatar to train a pose estimator (e.g., at operation 311 ). More specifically, the training module 220 can assign the rigged avatar to be part of a labeled dataset used to train the pose estimator. The labeled datasets can be associated with different types of physical activities, various levels of intensity, and diverse anatomical positions. The rigged avatar can be one of multiple rigged avatars in the labeled dataset and each rigged avatar can be associated with different digital images (e.g., different frames in the same video), different living bodies, different targets (e.g., proper form versus improper form), etc., and saved to various labeled datasets to train a pose estimator to accurately predict poses of another user or participant performing a physical activity. The training module 220 can optimize the process of generating training data based on continuously provided inputs (e.g., target positions of anatomical landmarks, parameters of body proportions, parameters of the camera, etc.) to minimize the261805473883PATENTAttorney Docket No. 125847.8032. WO01 error between the 2D spatial locations in an input digital image and the estimated 3D spatial locations for the pose in the avatar.

[0071] Additionally or alternatively, the motion monitoring platform can associate metadata with the rigged avatar and labeled datasets to specify various characteristics. The metadata can include characteristics of the physical activity, such as the type of physical activity, the intensity of the physical activity, or the posture during the physical activity. For example, the metadata can indicate whether the activity is running, jumping, or lifting, the level of exertion involved, or the specific posture adopted during the activity. The metadata can also include characteristics of the living body, such as an identifier of the living body, the proportion of the living body in the digital image, or the anatomical position of the living body. In some embodiments, characteristics of the living body include the identity of the participant, the relative size or dimensions of the participant within the image, or specific anatomical positions (e.g., arm raised, leg bent). Furthermore, the metadata can specify characteristics of the camera used to capture the digital image, such as the type of camera, the FoV of the camera, or the resolution of the camera, including details about the camera model, the angle and scope of the camera’s view, or the quality and clarity of the image captured. The training module 220 can use the metadata to determine which labeled dataset to place the rigged avatar into, ensuring that the data is organized according to relevant characteristics for more effective training of the pose estimator. Similar processes can be applied to optimize the categorization and utilization of the labeled datasets in training the pose estimator.

[0072] It is worth noting that conventional methods for obtaining 3D labels of persons from images or frames may use reprojection (e.g., random reprojection) of inferred 3D skeletons back to 2D images to train a pose estimator. However, these methods do not present the reprojection within the context of an interactive tool, thereby limiting annotator engagement and real-time adjustments. Furthermore, conventional training methods do not incorporate a rigged anatomical model of the body that can support an arbitrary set of anatomical271805473883PATENTAttorney Docket No. 125847.8032. WO01 landmarks. In some conventional training methods, estimated intrinsic and extrinsic camera parameters of images in 3D human pose inference may occur but not within an interactive setting that enables continuous annotator input to improve overall estimation. The processes for associating the avatar with the labeled anatomical landmarks, reprojecting the avatar onto the digital image, and deforming the avatar to produce a rigged avatar are described in more detail with reference to Figures 5-7.A _ Computer Vision Engine

[0073] Figure 5 includes a high-level diagrammatic illustration of a process for labeling the anatomical positions of living bodies in digital images. This process can involve the motion monitoring platform (e.g., the motion monitoring platform 212 of Figure 2) using a computer vision engine (also referred to as a “CV engine”) to facilitate the labeling of digital images to produce perspective- corrected, 3D representations of human bodies for analysis. The 3D representation may further be used to train pose estimators, via labeled datasets, to estimate the poses of a human body that is viewable in digital images within a training module (e.g., the training module 220 of Figure 2).

[0074] The CV engine may be employed by a processing module (e.g., the processing module 214 of Figure 2) for processing input digital images, frames, and / or responses from an annotator interacting with the system described herein. Additionally or alternatively, the CV engine may be applied by a modeling module and / or rigging module (e.g., the modeling module 216 and the rigging module 218 of Figure 2) to enable labeling of the digital images. Moreover, the motion monitoring platform may use (i) a feature extractor to expose relevant features of the present digital image, (ii) a GUI engine (also described as the GUI module 222 of Figure 2) to check the status of annotation prompts (also called “GUI triggers”) against the present digital image, (iii) stored parameters of the image capture settings, and (iv) predefined annotation rules.281805473883PATENTAttorney Docket No. 125847.8032. WO01

[0075] The CV engine may be implemented in software, firmware, hardware, or a combination thereof. For example, the CV engine may be within a system used to perform the processes in Figure 5 and described herein. The system may incorporate, first, a software system or application that performs the proposed algorithm and supports the proposed interactive capabilities (e.g., the CV engine). Examples of CV engines include OpenPose, MediaPipe, Kinect, and proprietary CV engines developed for estimating pose.

[0076] Second, the system may comprise a computing device, such as a computer or tablet, for running the algorithm, processing inputs / outputs (“I / O”), and supporting software and libraries. Third, the system may comprise a memory device, such as system memory, a hard drive, a solid-state drive (“SSD”), or cloud storage, for loading the system software, for loading images or videos to be annotated, and for loading and storing the output annotation files. Fourth, the system can comprise a display device or interface, such as a monitor, and input devices, such as a keyboard, mouse, or touchpad / screen, for the annotator to interface with the system. An example network environment (e.g., the network environment 100 of Figure 1 ) that incorporates these components is described in more detail with reference to Figure 1 .

[0077] When applied to a digital image, the CV engine may extract relevant features of the digital image, the camera used to take the digital image, and / or human bodies included therein, such as contours, edges, and key points, to facilitate the labeling of the digital image and / or generation of perspective- corrected and labeled 3D representations of the human body in the digital image, and / or the like.

[0078] Set forth below is a summary of the approach for employing the system described above for an interactive method enabling 3D annotations of human poses in 2D digital images or video frames. At a high level, the approach may involve an interactive system that renders both an input image or video291805473883PATENTAttorney Docket No. 125847.8032. WO01 frame to be annotated and a 3D representation of a person in that image or video frame, which can be manipulated by the user through a user interface (Figure 5).

[0079] The processes can be interactive, rendering both the input image or video frame to be annotated and a 3D representation of a person in that image that can be manipulated by the user (e.g., via the GUI module 222). Furthermore, a virtual camera may simulate the original camera used to take the image. For example, the virtual camera can be pointed toward and perpendicular to a projection of the image on a 3D plane. The virtual camera can be assumed to be centered on the image and maintain view frustum boundaries, thereby simulating the original real camera that captured the image, as described in more detail with reference to Figures 6A-B. A 3D representation of one or more persons in the image, including a rigged 3D avatar and a set of 3D control targets, can be placed between the camera and the 3D image plane such that the projection of the 3D representations matches the corresponding person in the image, as described in more detail with reference to Figure 7.B. _ Estimating FoV Projections

[0080] Figures 6A-B include illustrations of different FoV projections used for placing a virtual camera onto a 2D digital image of a living body. These examples are provided solely for the purpose of illustration, as embodiments of the field projection employed by the motion monitoring platform could include varying levels of annotator interaction.

[0081] For each digital image (e.g., each frame of a video), the motion monitoring platform (and more specifically, its modeling module) can make the following simplifying assumptions to estimate the FoV of the camera used to capture the image.• Assumption One: The original camera utilized for capturing the image is modeled as a pinhole camera, characterized by the absence of distortion, with intrinsic parameters fully defined by the FoV and focal length.301805473883PATENTAttorney Docket No. 125847.8032. WO01• Assumption Two: The original image is uncropped.

[0082] From the two assumptions, a virtual camera can be reasonably positioned and centered on the original image, with the FoV being the only parameter that requires estimation or annotation. Although the view of each living body, as annotated using the system described herein, can be cropped from the original image, the location of the crop can be known by, for example, referencing the coordinates and dimensions of the cropped area within the original image. Consequently, the relative position of the camera center can be inferred.

[0083] Thus, the system may enable the annotator to tune a parameter (e.g., via a slider in a user interface as shown in Figure 4C), allowing the annotator to adjust the FoV of the camera. The FoV parameter may affect the perspective of the camera and has the following effects:• Effect 1 : Transform the X and Y coordinates of the 3D representation using the appropriate perspective transformation.• Effect 2: Transform the depth axis for each key point on the 3D representation to adhere to the appropriate perspective transformation, ensuring consistency with Effect 1 .

[0084] Since manually tuning the FoV may be challenging for untrained annotators, the proposed system may estimate an initial FoV (e.g., via the modeling module of the motion monitoring platform) and update the estimate as necessary. The update can be achieved by minimizing the reprojection error of the 3D representation when it is projected back onto the 2D pose of the living body in the original image and / or by annotator review and adjustments.C. _ Projecting 2D Image onto 3D Representation

[0085] Figure 7 includes an illustration of a projection of a 2D digital image of a living body onto a 3D representation of the living body. As described in Figures 6A-B, this example is provided solely for the purpose of illustration, as311805473883PATENTAttorney Docket No. 125847.8032. WO01 embodiments of the reprojection approach employed by the motion monitoring platform could include varying levels of annotator interaction.

[0086] The primary inputs (e.g., image data, annotator inputs, etc.), outputs (e.g., 3D representations and associated camera FoVs, 2D key points, 3D key points, etc.), and computational steps of the proposed system are defined above with reference to Figures 3-5. In some embodiments, the annotator specifies an initial camera FoV, manipulates the target depths of key points on the image, and specifies perspective parameters related to, for example, proportions of the anatomical avatar (or skeletal model) to fit the digital image. Using the digital image and labeled 2D key points as input, along with an assumed initial virtual camera position and direction, the system may perform the following steps:• Step 1 : Compute the projected 3D target positions, which are output as key point depths.• Step 2: Deform the 3D avatar to match the specified proportions.• Step 3: Compute the scale of the proportioned avatar.• Step 4: Fit the pose of the proportioned, scaled avatar to the targets using an inverse kinematic (“IK”) system (e.g., Final IK).• Step 5: Deform the anatomical model of the avatar (e.g., using a rigged mesh system such as the rigged mesh systems in Unity or Unreal Engine).• Step 6: Output predefined anatomical locations on that model to 3D key points.• Step 7: Optionally update the FoV to best fit the posed avatar to 3D targets.• Step 8: Optionally update the avatar proportions to best fit the avatar to the 3D targets.

[0087] Additionally or alternatively, the system described herein can run and / or function within or in conjunction with other libraries, frameworks, and321805473883PATENTAttorney Docket No. 125847.8032. WO01 platforms designed for 3D rendering and interactive software, such as Unity, Unreal Engine, and / or the like.D. _ GUI Engine

[0088] The GUI engine that is implemented by the motion monitoring platform (and more specifically, its GUI module) may facilitate the annotation process by providing an intuitive interface for users to label anatomical landmarks on digital images. Furthermore, the GUI engine can transfer input from a user for further processing by the motion monitoring platform (more specifically, its processing module) and / or transfer approved or finalized 3D representations for storage in labeled datasets for training a pose estimator by the motion monitoring platform (more specifically, its training module). The GUI module may generate a set of annotator requests as GUI triggers that are valid for the current frame of digital images being reviewed. Based on which label and / or parameter requests have been fulfilled (e.g., labeling key points, estimating camera perspective) and which annotation prompts have already been displayed to the user (e.g., as cataloged in a history of annotator prompts), the motion monitoring platform may decide which annotator requests to trigger for the current digital image, if any. In some embodiments, the history of annotator requests may only be persisted per session, per series of frames reviewed, and / or per frame reviewed and therefore may be stored locally in memory and then erased from memory following the conclusion of each session, series of frames reviewed, and / or per frame reviewed.

[0089] Finally, when an annotator request is created, an annotator prompt may be generated and then presented to the user via the interface. The motion monitoring platform may generate the annotator prompt based on an annotator prompt template (or simply “prompt template”) that is stored in memory and any relevant parameters (e.g., current frame number, specific anatomical landmarks to be labeled) in order to create a prompt that is relevant and personalized to the frame or frames in question.331805473883PATENTAttorney Docket No. 125847.8032. WO01Methodologies for Facilitating Labeling of Anatomical Positions of Living Bodies in Digital Images with a Computer Program

[0090] Several approaches to facilitating labeling of anatomical positions of living bodies in digital images with a computer program executing on a computing device are set forth below. These approaches are best understood when read in conjunction with the disclosure corresponding to Figures 3-7.

[0091] Figure 8 is a flow diagram of a process 800 for facilitating the labeling of anatomical positions of living bodies in digital images. Initially, the motion monitoring platform 212 can obtain a digital image that includes an individual performing a physical activity (step 802). For example, the processing module 214 can receive image data from a camera associated with the computer program the motion monitoring platform 212 is executing on and process the image data to be in a format that can be displayed and / or annotated, as described herein. The image data can be associated with a 2D image of an individual performing a physical activity. Additionally or alternatively, the 2D image can be one of a series of images taken from a video or a series of snapshots of an individual performing a physical activity (e.g., a recording or reference shots of an individual performing a specific exercise in a specific form).

[0092] The motion monitoring platform 212 can then prompt an annotator to label 2D spatial locations of the anatomical landmarks on the digital image via an interface (step 804). For example, the GUI module 222 can generate an interface that notifies an annotator to input anatomical landmarks of the individual performing the physical activity in the digital image. The anatomical positions can be one or more spatial locations of anatomical landmarks of the living body in the 2D image. The anatomical landmarks can be key points of the digital image labeled by the annotator, and these key points serve as positional targets for the corresponding segments of an anatomically correct avatar. As an illustrative example, the annotator may identify the positions of key points of the living body in the digital image, such as the shoulders, elbows, hips, knees, etc., on one or341805473883PATENTAttorney Docket No. 125847.8032. WO01 both sides of the living body, as described in more detail with reference to Figures 4A-B.

[0093] The motion monitoring platform 212 can then estimate an FoV of a camera used to capture the digital image (step 806). For example, the modeling module 216 can estimate the FoV of the camera based on various data inputs and / or parameters provided by the annotator. The data inputs can include the specifications of the camera used to capture the digital image, such as focal length, sensor size, resolution, etc. Additionally or alternatively, the modeling module 216 can predict information about the distance between the camera and the individual in the image, the angle at which the image was captured, lens distortion characteristics, and / or similar parameters.

[0094] In some embodiments, the GUI module 222 causes the generation of an interface that notifies an annotator to input one or more parameters associated with the digital image captured. The GUI module 222 can prompt an annotator to input one or more parameters associated with (i) the FoV of the camera used to capture the digital image, (ii) the proportion of the individual in the digital image, or (iii) the depth of an anatomical landmark labeled in the digital image via the interface. The modeling module 216 can use the one or more parameters input by the annotator, in addition to the anatomical landmarks labeled by the annotator, to estimate the FoV of the camera. Additionally or alternatively, the modeling module 216 can prepare an initial FoV estimate without annotator input, which the annotator can be prompted to adjust, for example, using one or more sliders on the interface used to adjust a 3D model of the individual, as described in more detail with reference to Figure 4C and steps 810-812 herein.

[0095] The motion monitoring platform 212 can then associate the anatomical landmarks labeled in the digital image with corresponding segments of an anatomically correct avatar (step 808). For example, the rigging module 218 can map the labeled anatomical landmarks to predefined points on the anatomically351805473883PATENTAttorney Docket No. 125847.8032. WO01 correct avatar such that each of the spatial positions of the landmarks labeled in the digital image correspond to a segment on the anatomically correct avatar.

[0096] The motion monitoring platform 212 can project the anatomically correct avatar onto the digital image, based on the FoV, to verify that the anatomical landmarks are properly associated with the corresponding segments of the anatomically correct avatar (step 810). For example, the rigging module 218 can overlay the avatar onto the digital image, aligning the segments of the avatar with the labeled anatomical landmarks. In some embodiments, the GUI module 222 generates an interface to prompt an annotator to verify that the anatomical landmarks labeled in the digital image are properly associated with the corresponding segments of the anatomically correct avatar via the interface. The annotator can make adjustments if necessary, ensuring that the mapping is precise and that the avatar accurately reflects the anatomical structure and movement of the individual. The overlay can allow, for example, an annotator to visually inspect the alignment and ensure that the joints and segments of the avatar correspond accurately to the anatomical landmarks in the digital image. The annotator can change the transparency of the avatar to better visualize the underlying anatomical landmarks, adjust the lines associated with the FoV to enhance clarity, and modify one or more perspective parameters to ensure accurate alignment.

[0097] The motion monitoring platform 212 can then deform the anatomically correct avatar according to the anatomical landmarks to produce a 3D representation of the individual performing the physical activity (step 812). For example, the rigging module 218 can divorce a portion of the anatomical landmarks from the anatomically correct avatar so that the avatar accurately reflects the unique anatomical structure and movement patterns of the individual in the digital image. In some embodiments, the 3D representation is produced based on additional parameters input by the annotator or generated by the modeling module 216. The parameters can include, for example, the FoV of the camera used to capture the digital image, the proportion of the individual in the361805473883PATENTAttorney Docket No. 125847.8032. WO01 digital image, or the depth of an anatomical landmark labeled in the digital image. The annotator can input these parameters as described in more detail with respect to step 806. The annotator can also adjust the parameters via the interface using one or more sliders to deform the anatomically correct avatar to make the 3D representation, as described in more detail with respect to step 810.

[0098] The motion monitoring platform 212 can then assign the 3D representation to be part of a labeled dataset used to train a machine learning model to produce estimated poses of living bodies in digital images (step 814). For example, the training module 220 can assign the 3D representation to train a pose estimator, as described in more detail with reference to Figure 2 and elsewhere herein. The labeled dataset can include a plurality of datasets segregated based on various types of factors and annotations that provide information about the poses in the 3D representation and activities represented in digital images. Annotations can include key points, which are specific anatomical landmarks labeled, such as joints (e.g., elbows, knees, shoulders), to indicate the position of these points in 3D space; bounding boxes used to enclose the entire body or specific parts of the body, providing spatial information about the location and size of the body or body parts within the image; pose labels used to categorize the pose being performed, such as “standing,” “sitting,” “running,” or “jumping”; activity labels used to categorize the activity being performed, such as “walking,” “dancing,” “exercising,” or “playing sports”; and temporal annotations that provide information about the sequence and timing of poses and activities, which can be used to train the pose estimator to understand dynamic movements and transitions between poses.

[0099] Figure 9 is a flow diagram of a process 900 for facilitating labeling of anatomical positions of living bodies in digital images. The process 900 can be generally similar to or identical to one or more of the steps of the process 800 described in more detail with reference to Figure 8 or any of the other processes described herein.371805473883PATENTAttorney Docket No. 125847.8032. WO01

[0100] Initially, the motion monitoring platform 212 can obtain a digital image that includes a living body performing a physical activity (step 902). Step 902 can be generally similar to step 802, as described in more detail with respect to Figure 8. For example, the processing module 214 can receive image data associated with a 2D image of a living body performing a physical activity. Additionally or alternatively, the 2D image can be one of a series of images taken from a video or series of snapshots of a living body performing a physical activity, such as a recording or reference shots of an individual performing a specific exercise in a specific form.

[0101] The motion monitoring platform 212 can then prompt an annotator, via an interface, to input (i) one or more key points associated with one or more anatomical landmarks of the living body in the digital image and (ii) one or more perspective parameters associated with the digital image (step 904). Step 904 can be generally similar to step 804, as described in more detail with respect to Figure 8. For example, the GUI module 222 can generate an interface to input the one or more key points and / or the one or more perspective parameters. The one or more key points can be specific anatomical landmarks such as joints, eyes, or other significant points on the living body in the digital image. The one or more perspective parameters can be associated with a camera used to capture the digital image and can include at least one of (i) an FoV of the camera used to capture the digital image, (ii) a proportion of the living body in the digital image, or (iii) a depth of an anatomical landmark labeled in the digital image. Additionally or alternatively, the motion monitoring platform can generate the one or more perspective parameters. For example, the modeling module 216 can model the FoV of a camera used to capture the digital image based on the one or more key points input by the annotator.

[0102] The motion monitoring platform 212 can then associate the one or more key points labeled in the digital image with one or more corresponding segments of an anatomically correct avatar (step 906). Step 906 can be generally similar to step 808, as described in more detail with respect to Figure 8.381805473883PATENTAttorney Docket No. 125847.8032. WO01For example, the rigging module 218 can map the labeled key points to the corresponding segments of the avatar such that the anatomically accurate avatar can be rigged to reflect the anatomical structure and movement patterns of the living body in the digital image. Additionally or alternatively, the motion monitoring platform 212 can project the anatomically correct avatar onto the digital image to verify (e.g., by annotator confirmation via the interface) that the one or more corresponding segments are properly associated with the anatomically correct avatar.

[0103] The motion monitoring platform 212 can rig the anatomically correct avatar onto the living body in the digital image, based on the one or more key points or the one or more perspective parameters, to produce a rigged avatar (step 908). Step 908 can be generally similar to steps 810-812, as described in more detail with respect to Figure 8. For example, the rigging module 218 and / or the GUI module 222 can be used to adjust the avatar to align with the living body in the digital image according to the one or more key points and / or the one or more perspective parameters by automatically or manually adjusting the parameters on the avatar or changing parameters via sliders on the interface. In some embodiments, the FoV is used to rig the anatomically correct avatar onto the living body in the digital image.

[0104] The motion monitoring platform 212 can then store the rigged avatar to train a machine learning model used to estimate poses of living bodies during various physical activities (step 910). Step 910 can be generally similar to step 81 , as described in more detail with respect to Figure 8. For example, the training module 220 can use the stored rigged avatar to train the machine learning model to estimate poses of living bodies during various physical activities. Additionally or alternatively, the training module 220 can assign the rigged avatar to a dataset based on the one or more key points or the one or more perspective parameters.

[0105] Figure 10 is a flow diagram of a process 1000 for facilitating labeling of391805473883PATENTAttorney Docket No. 125847.8032. WO01 anatomical positions of living bodies in digital images. The process 1000 can be generally similar to or identical to one or more of the steps of the processes 800 and 900 described in more detail with reference to Figures 8 and 9 or any of the other processes described herein.

[0106] Initially, the motion monitoring platform 212 can obtain a digital image of a living body performing a physical activity (step 1002). Step 1002 can be generally similar to steps 802 and 902, as described in more detail with respect to Figures 8 and 9, respectively.

[0107] The motion monitoring platform 212 can then prompt an annotator, via an interface, to identify anatomical positions on the digital image (step 1004). Step 1004 can be generally similar to steps 804 and 904, as described in more detail with respect to Figures 8 and 9, respectively. For example, the GUI module 222 can cause generation of an interface that notifies an annotator to identify anatomical positions of the living body in the digital image. The anatomical positions can be one or more spatial locations of anatomical landmarks of the living body in a 2D digital image. As an illustrative example, the annotator can identify locations of reference points on the living body within the digital image, such as the shoulders, elbows, hips, knees, and other relevant areas on either or both sides of the body, as further detailed with reference to Figures 4A-B.

[0108] The motion monitoring platform 212 can then associate the anatomical positions identified in the digital image with corresponding segments of an anatomically correct avatar (step 1006). Step 1006 can be generally similar to steps 808 and 906, as described in more detail with respect to Figures 8 and 9, respectively. For example, the rigging module 218 can map the input anatomical positions onto corresponding anatomical locations on the anatomically correct avatar.

[0109] The motion monitoring platform 212 can prompt the annotator, via the interface, to confirm that the identified anatomical positions are properly associated with the corresponding segments of the anatomically correct avatar401805473883PATENTAttorney Docket No. 125847.8032. WO01(step 1008). Step 1008 can be generally similar to steps 810 and 906, as described in more detail with respect to Figures 8 and 9, respectively. For example, the GUI module 222 can project the anatomically correct avatar onto the digital image for the annotator to review and / or adjust via the interface.

[0110] The motion monitoring platform 212 can then derive a 3D representation of the living body performing the physical activity by manipulating the anatomically correct avatar according to the anatomical positions identified (step 1010). Step 1010 can be generally similar to steps 812 and 908, as described in more detail with respect to Figures 8 and 9, respectively. For example, the rigging module 218 and / or the GUI module 222 can be used to automatically or manually deform the anatomically correct avatar to align with or diverge from the anatomical positions of the living body input onto the digital image.

[0111] In some embodiments, the motion monitoring platform 212 (e.g., the modeling module 216) estimates the FoV of a camera used to capture the digital image, and this estimated FoV is used to derive the 3D representation.Additionally or alternatively, the motion monitoring platform 212 (e.g., the GUI module 222) prompts the annotator, via the interface, to identify at least one parameter associated with (i) the FoV of the camera used to capture the digital image, (ii) the proportion of the living body in the digital image, or (iii) the depth of an anatomical position identified in the digital image. The parameters can be used by the rigging module 218 to derive the 3D representation.

[0112] The motion monitoring platform 212 can then store the rigged avatar to train a machine learning model to produce, as output, an estimated pose of a living body performing a physical activity (step 1012). Step 1012 can be generally similar to steps 814 and 910, as described in more detail with respect to Figures 8 and 9, respectively. For example, the training module 220 can use the stored rigged avatar to train the machine learning model.

[0113] In some embodiments, the motion monitoring platform 212 associates411805473883PATENTAttorney Docket No. 125847.8032. WO01 metadata with the 3D representation that specifies characteristics of the physical activity, such as the type of physical activity, the intensity of the physical activity, or the posture during the physical activity. Additionally or alternatively, the motion monitoring platform 212 can associate metadata with the 3D representation that specifies characteristics of the living body performing the physical activity, such as an identifier of the living body, the proportion of the living body in the digital image, or the anatomical position of the living body. In some embodiments, the motion monitoring platform 212 associates metadata with the 3D representation that specifies characteristics of the camera used to capture the digital image, such as the type of camera, the FoV of the camera, or the resolution of the camera.Processing System

[0114] Figure 11 includes a block diagram illustrating an example of a processing system 1100 in which at least some operations described herein can be implemented. For example, components of the processing system 1100 may be hosted on a computing device that includes a motion monitoring platform (e.g., motion monitoring platform 102 of Figure 1 or motion monitoring platform 212 of Figure 2).

[0115] The processing system 1100 can include a processor 1102, main memory 1106, non-volatile memory 1110, network adapter 1112, display mechanism 1118, input / output device 1120, control device 1122 (e.g., a keyboard or pointing device such as a computer mouse or trackpad), drive unit 1124 including a storage medium 1126, and signal generation device 1130 that are communicatively connected to a bus 1116. The bus 1116 is illustrated as an abstraction that represents one or more physical buses or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus 1116, therefore, can include a system bus, a Peripheral Component Interconnect (“PCI”) bus or PCI-Express bus, a HyperTransport (“HT”) bus, an Industry Standard Architecture (“ISA”) bus, a Small Computer System Interface421805473883PATENTAttorney Docket No. 125847.8032. WO01(“SCSI”) bus, a Universal Serial Bus (“USB”) data interface, an Inter-Integrated Circuit (“l2C”) bus, or a high-performance serial bus developed in accordance with IEEE 1394.

[0116] While the main memory 1106, non-volatile memory 1110, and storage medium 1126 are shown to be a single medium, the terms “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized / distributed database and / or associated caches and servers) that store one or more sets of instructions 1128. The terms “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the processing system 1100.

[0117] In general, the routines executed to implement the embodiments of the disclosure can be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically comprise one or more instructions (e.g., instructions 1104, 1108, 1128) set at various times in various memory and storage devices in a computing device. When read and executed by the processors 1102, the instruction(s) cause the processing system 1100 to perform operations to execute elements involving the various aspects of the present disclosure.

[0118] Further examples of machine- and computer-readable media include recordable-type media, such as volatile memory devices and non-volatile memory devices 1110, removable disks, hard disk drives, and optical disks (e.g. , Compact Disk Read-Only Memory (“CD-ROMs”) and Digital Versatile Disks (“DVDs”)), and transmission-type media, such as digital and analog communication links.

[0119] The network adapter 1112 enables the processing system 1100 to mediate data in a network 1114 with an entity that is external to the processing system 1100 through any communication protocol supported by the processing431805473883PATENTAttorney Docket No. 125847.8032. WO01 system 1100 and the external entity. The network adapter 1112 can include a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, a repeater, or any combination thereof.Remarks

[0120] The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated.

[0121] Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments can vary considerably in their implementation details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments but also all equivalent ways of practicing or implementing the embodiments.441805473883PATENTAttorney Docket No. 125847.8032. WO01

[0122] The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.451805473883

Claims

PATENTAttorney Docket No. 125847.8032. WO01CLAIMSWhat is claimed is:1 . A computer-implemented method for facilitating labeling of anatomical landmarks of living bodies in digital images, the computer-implemented method comprising: obtaining a digital image that includes an individual performing a physical activity; prompting an annotator to label two-dimensional spatial locations of the anatomical landmarks on the digital image via an interface; estimating a field of view of a camera used to capture the digital image; associating the anatomical landmarks labeled in the digital image with corresponding segments of an anatomically correct avatar; projecting the anatomically correct avatar onto the digital image, based on the field of view, so as to verify that the anatomical landmarks are properly associated with the corresponding segments of the anatomically correct avatar; deforming the anatomically correct avatar according to the anatomical landmarks so as to produce a three-dimensional representation of the individual performing the physical activity; and assigning the three-dimensional representation to be part of a labeled dataset used to train a machine learning model so as to produce estimated poses of living bodies in digital images.

2. The method of claim 1 , wherein the anatomical landmarks are key points of the digital image labeled by the annotator, and wherein the key points are positional targets for the corresponding segments of the anatomically correct avatar.

3. The method of claim 1 , wherein deforming the anatomically correct avatar further comprises:461805473883PATENTAttorney Docket No. 125847.8032. WO01 divorcing a portion of the anatomical landmarks from the anatomically correct avatar.

4. The method of claim 1 , further comprising: prompting the annotator to verify that the anatomical landmarks labeled in the digital image are properly associated with the corresponding segments of the anatomically correct avatar via the interface.

5. The method of claim 1 , further comprising: producing the three-dimensional representation based on at least one of a parameter associated with (i) the field of view of the camera used to capture the digital image, (ii) a proportion of the individual in the digital image, or (iii) a depth of an anatomical landmark labeled in the digital image.

6. The method of claim 1 , further comprising: prompting the annotator to input at least one of a parameter associated with (i) the field of view of the camera used to capture the digital image, (ii) a proportion of the individual in the digital image, or (iii) a depth of an anatomical landmark labeled in the digital image via the interface.

7. The method of claim 1 , further comprising: generating the interface for the annotator to label the two-dimensional spatial locations of the anatomical landmarks on the digital image.

8. The method of claim 1 , further comprising: generating the interface for the annotator to input at least one of a parameter associated with (i) the field of view of the camera used to capture the digital image, (ii) a proportion of the individual in the471805473883PATENTAttorney Docket No. 125847.8032. WO01 digital image, or (iii) a depth of an anatomical landmark labeled in the digital image.

9. An electronic device comprising: a memory that includes instructions for facilitating labeling of anatomical landmarks of living bodies in digital images, wherein the instructions, when executed by a processor, cause the processor to: obtain a digital image that includes a living body performing a physical activity; prompt an annotator, via an interface, to input —(i) one or more key points associated with one or more anatomical landmarks of the living body in the digital image, and(ii) one or more perspective parameters associated with the digital image; associate the one or more key points labeled in the digital image with one or more corresponding segments of an anatomically correct avatar; rig the anatomically correct avatar onto the living body in the digital image, based on the one or more key points or the one or more perspective parameters, to produce a rigged avatar; and store the rigged avatar in the memory to train a machine learning model to produce, as output, an estimated pose of a living body performing a physical activity.

10. The electronic device of claim 9, wherein the one or more perspective parameters are associated with a camera used to capture the digital image and include at least one of (i) a field of view of the camera used to capture the digital481805473883PATENTAttorney Docket No. 125847.8032. WO01 image, (ii) a proportion of the living body in the digital image, or (iii) a depth of an anatomical landmark labeled in the digital image.11 . The electronic device of claim 9, wherein the instructions, when executed by a processor, cause the processor to further: prompt the annotator, via the interface, to input —(i) a field of view of a camera used to capture the digital image,(ii) a proportion of the living body in the digital image, and(iii) a depth of an anatomical position identified in the digital image.

12. The electronic device of claim 9, wherein the instructions, when executed by a processor, cause the processor to further: model a field of view of a camera used to capture the digital image, wherein the field of view is used to rig the anatomically correct avatar onto the living body in the digital image.

13. The electronic device of claim 9, wherein the instructions, when executed by a processor, cause the processor to further: project the anatomically correct avatar onto the digital image to verify the one or more corresponding segments are properly associated with the anatomically correct avatar.

14. The electronic device of claim 9, wherein the instructions, when executed by a processor, cause the processor to further: assign the rigged avatar to a dataset based on the one or more key points or the one or more perspective parameters, wherein the dataset is used to train the machine learning model.491805473883PATENTAttorney Docket No. 125847.8032. WO0115. A non-transitory medium with instructions stored thereon that, when executed by a processor of a computing device, cause the computing device to perform operations comprising: obtaining a digital image of a living body performing a physical activity; prompting an annotator, via an interface, to identify anatomical positions on the digital image; associating the anatomical positions identified in the digital image with corresponding segments of an anatomically correct avatar; prompting the annotator, via the interface, to confirm the anatomical positions identified are properly associated with the corresponding segments of the anatomically correct avatar; deriving a three-dimensional representation of the living body performing the physical activity by manipulating the anatomically correct avatar according to the anatomical positions identified; and in response to determination of a confirmation input into the interface, storing the three-dimensional representation in a dataset used to train a machine learning model to produce, as output, an estimated pose of an individual in a digital image.

16. The non-transitory medium of claim 15, further comprising: estimating a field of view of a camera used to capture the digital image, wherein the field of view is used to derive the three-dimensional representation.

17. The non-transitory medium of claim 15, further comprising: prompting the annotator, via the interface, to identify at least one of a parameter associated with (i) a field of view of a camera used to capture the digital image, (ii) a proportion of the living body in the digital image, or (iii) a depth of an anatomical position identified in the digital image.501805473883PATENTAttorney Docket No. 125847.8032. WO0118. The non-transitory medium of claim 15, associating metadata with the three-dimensional representation that specifies a characteristic of the physical activity, wherein the characteristic is selected from a group consisting of a type of the physical activity, an intensity of the physical activity, or a posture during the physical activity.

19. The non-transitory medium of claim 15, associating metadata with the three-dimensional representation that specifies a characteristic of the living body performing the physical activity, wherein the characteristic is selected from a group consisting of an identifier of the living body, a proportion of the living body in the digital image, or an anatomical position of the living body.

20. The non-transitory medium of claim 15, associating metadata with the three-dimensional representation that specifies a characteristic of a camera used to capture the digital image, wherein the characteristic is selected from a group consisting of a type of the camera used to capture the digital image, a field of view of the camera used to capture the digital image, or a resolution of the camera used to capture the digital image.511805473883

Images