Methods and systems for generating and personalizing visualizations to convey information regarding exercise quality

Abstract

Systems and methods are provided for monitoring the quality of the performance of physical activities by an individual and providing feedback in a personalized manner during the performance of the physical activities. The approach not only can help solve the problem of accurately counting repetitions of physical activities but can also provide useful feedback without requiring that a healthcare professional be present when the repetitions are being performed. The method includes analyzing images of an individual performing a physical activity, generating a skeletal frame, comparing the skeletal frame to a reference skeletal frame, determining a degree of similarity, and displaying a graphical visualization as feedback to the individual.

Description

METHODS AND SYSTEMS FOR GENERATING AND PERSONALIZINGVISUALIZATIONS TO CONVEY INFORMATION REGARDING EXERCISE QUALITYCROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to U.S. App No 63 / 550,157 filed February 6, 2024, which is incorporated herein by reference, in its entirety.TECHNICAL FIELD

[0002] Various embodiments concern computer programs and associated computer-implemented techniques for exercise quality visualization and personalization to promote completion of physical activities.BACKGROUND

[0003] Exercise therapy is an intervention technique that utilizes physical activity as the principal treatment method for addressing the symptoms of musculoskeletal (MSK) conditions, such as acute physical ailments and chronic physical ailments. Exercise therapy programs may involve apian for performing physical activities during exercise therapy sessions that occur on a periodic basis. Exercise therapy platforms can monitor individuals performing the physical activities and provide feedback to the individuals.

[0004] However, these exercise therapy platforms usually use simple counters to count repetitions of the physical activities or are unable to monitor how well an individual performed a repetition. For example, if an individual receives generalized feedback, the individual may be unable to understand whether she is performing the activity well or poorly, causing the individual to become discouraged from doing her exercise therapy sessions. Therefore, a better approach is needed for exercise therapy monitoring and personalized feedback.

[0005] Other systems that facilitate training an individual to perform physical activities may also be unable to provide personalized feedback on whether an individual is properly performing a variety of physical activities, such as dance moves, sporting techniques, exercises, cooking techniques, and the like. For example, if an individual is not informed that she is not using proper form for her forehands, she may not be assuccessful in tennis matches compared to if she were using proper form. In another example, an individual may be penalized in a cooking competition for not cutting her vegetables in a specific manner, and a system could have informed her with personalized feedback on her cutting technique. Thus, these systems need a way to provide personalized feedback for individuals to achieve improved form.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 illustrates a network environment that includes a visualization and personalization platform that is executed by a computing device.

[0007] Figure 2 illustrates an example of a computing device that is able to execute a visualization and personalization platform.

[0008] Figure 3 depicts an example of a communication environment that includes a visualization and personalization platform configured to obtain data from one or more sources.

[0009] Figure 4 illustrates an example of a visualization and personalization platform that is executed by a computing device.

[0010] Figure 5 illustrates examples of digital presentations of graphical visualizations provided on an interface by a visualization and personalization platform.

[0011] Figure 6 illustrates how a template can be captured by estimating a pose in a video where an expert (e.g., a physiotherapist) showcases the ideal movement and, in some embodiments, undesired variations for which feedback is to be provided.

[0012] Figure 7 includes an example of estimated poses being matched against a template prepared for a given physical activity (here, a squat).

[0013] Figure 8 depicts a flow diagram of a process for manipulating an appearance of a graphical element on an interface to reflect how closely a performance of a physical activity matches a reference performance of the physical activity.

[0014] Figure 9 depicts a flow diagram of a process for personalizing a metric to reflect how closely a performance of a physical activity by an individual matches a reference performance based on historical performances by the individual, physical abilities and health of the individual, input received from the individual, or observations of the performance.

[0015] Figure 10 depicts a flow diagram of a process for determining and personalizing a metric that reflects how closely a performance of a physical activity by an individual matches a reference performance of the physical activity.

[0016] Figure 11 includes a block diagram illustrating an example of a processing system in which at least some operations described herein can be implemented.

[0017] Various features of the technology described herein will become more apparent to those skilled in the art from a study of the Detailed Description in conjunction with the drawings. Various embodiments are depicted in the drawings for the purpose of illustration. However, those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technology. Accordingly, although specific embodiments are shown in the drawings, the technology is amenable to various modifications.DETAILED DESCRIPTION

[0018] Over the last several years, significant advances have been made in the field of computer vision. This has resulted in the development of sophisticated exercise therapy programs that are designed to encourage and instruct individuals to perform physical activities as part of an exercise therapy program, as well as monitor completion of performance of the physical activities. For example, an exercise therapy platform can perform pose estimation in either two dimensions or three dimensions. Two- dimensional (“2D"’) pose estimators predict the 2D spatial locations of key points, generally through the analysis of the pixels of a single digital image. Three-dimensional (“3D”) pose estimators predict the 3D spatial arrangement of key points, generally through the analysis of the pixels of multiple digital images, for example, consecutive frames in a video, or a single digital image in combination with another ty pe of data generated by, for example, an inertial measurement unit (“IMU”) or Light Detection and Ranging (“LiDAR”) unit.

[0019] Pose estimators - both 2D and 3D - continue to be applied to different contexts, and as such, continue to be used to help solve different problems. One problem for which pose estimators have proven to be particularly useful is monitoring the performance of physical activities. Consider, for example, a scenario where an individual is instructed or prompted to perform a physical activity by a computerprogram. By applying a pose estimator to digital images of the individual, the computer program can glean insight into performance of the physical activity. Historically, the individual may have instead been asked to summarize her performance of the physical activity7(e.g., in terms of difficulty ); however, this type of manual feedback tends to be inaccurate and inconsistent. Due to their consistent, programmatic nature, pose estimators allow for more accurate monitoring of performances of physical activities.

[0020] Programs generally explain, either audibly or visually, how an individual (also called a “user,” “patient.” or “participant”) should perform physical activities to achieve a therapeutic goal. However, some programs provide only simple “pass / fail” feedback or repetition counts, which may enable an individual to perform an exercise but do not provide enough information to be helpful to the individual. Other programs provide complex visualizations that can include positions of multiple body parts. While there is some benefit to offering very' simple or very detailed feedback - many individuals either are unable to obtain enough information from simple visualizations to incorporate into their exercise practice, or get confused by or are not able to act upon complex feedback. There is therefore a need for personalized visualizations that provide an “optimal” amount of feedback that is readily comprehensible, resonates with the individual, and prompts behavioral change.

[0021] Introduced here is an approach to monitoring the quality of the performance of physical activities by an individual and providing feedback in a personalized manner during the performance of the physical activities. The approach not only can help solve the problem of accurately counting repetitions of physical activities but can also provide useful feedback without requiring that a healthcare professional (e.g., physiotherapist, nurse, or physician) be present when the repetitions are being performed. Simply put, the approach allows individuals to perform high-quality' exercise therapy at home.

[0022] As further discussed below, the approach may rely on real-time personalized analysis of performance of a physical activity' by an individual. The quality (sometimes referred to as “intensity”) of the performance - or indicia that are visually representative thereof- may be presented for display on an interface that is accessible to a computing device. Generally, the computing device is associated with the individual and is responsible for generating the digital images from which the quality' of the performanceis estimated. Given a series of representations of the individual over time, a motion monitoring platform that includes a visualization and personalization module can:• Extract one or more salient features of the individual• Compare the salient features to reference salient features that correspond to a reference performance of the physical activity• Produce a metric that is indicative of similarity between the salient features and the reference salient features• Manipulate the metric to reflect a quality of the performance in view of historical performances by the individual, physical abilities and health of the individual, input received from the individual, and / or observations of the performance• Determine that an individual has completed a number of repetitions of a physical activity• Manipulate an appearance of a graphical element on an interface to reflect the quality of the performance; and• Prioritize, trigger, and present queued feedback in a way that is helpful for the individual.The nature of the representations may depend on the nature of the pose extractor that is applied by the visualization and personalization platform to produce the series of representations. For example, if the pose extractor is a 2D pose extractor, the representations may be 2D skeletal frames that define the 2D spatial locations of key points. If the pose extractor is a 3D pose extractor, the representations may be 3D skeletal frames that define the 3D spatial locations of key points.

[0023] One benefit is that this approach may be generic to a large variety' of physical activities, though specific parameters and templates can be defined per activity. Accordingly, a set of algorithms corresponding to different physical activities could be developed and then released for the visualization and personalization platform, but additional algorithms corresponding to new physical activities could be added to the set or existing algorithms corresponding to existing physical activities could be removed from the set.

[0024] For the purpose of illustration, embodiments may be described with reference to exercises that are performed during sessions as part of a program. However, the visualization and personalization platform could be designed to monitor performance of other physical activities, such as sporting activities, cooking activities, art activities, and the like. Accordingly, the approach described herein could be used to provide personalized feedback regarding performance of nearly any physical activity.

[0025] Moreover, embodiments may be described in the context of computerexecutable instructions for the purpose of illustration. However, aspects of the approach could be implemented via hardware or firmware instead of, or in addition to, software. As an example, the visualization and personalization platform may be embodied as a computer program that offers support for completing exercises during sessions as part of a program, determines which physical activities are appropriate for an individual given performance during past sessions, and enables communication between the individual and one or more coaches. The term “coach” may be used to generally refer to individuals who prompt, encourage, or otherwise facilitate engagement byindividuals with the visualization and personalization platform. Coaches are generally not healthcare professionals but could be in some embodiments.Terminology

[0026] References in the present disclosure to “an embodiment” or “some embodiments” mean that the feature, function, structure, or characteristic being described is included in at least one embodiment. Occurrences of such phrases do not necessarily refer to the same embodiment, nor are they necessarily referring to alternative embodiments that are mutually exclusive of one another.

[0027] Unless the context clearly requires otherwise, the terms “comprise,” “comprising,” and “comprised of' are to be construed in an inclusive sense rather than an exclusive or exhaustive sense. That is, in the sense of “including but not limited to.” The term “based on” is also to be construed in an inclusive sense. Thus, the term “based on” is intended to mean “based at least in part on.”

[0028] The terms “connected,” “coupled,” and variants thereof are intended to include any connection or coupling between two or more elements, either direct or indirect. The connection or coupling can be physical, logical, or a combination thereof.For example, elements may be electrically or communicatively coupled to one another despite not sharing a physical connection.

[0029] The term “module’7may refer broadly to software, firmware, hardware, or combinations thereof. Modules are typically functional components that generate one or more outputs based on one or more inputs. A computer program may include or utilize one or more modules. For example, a computer program may utilize multiple modules that are responsible for completing different tasks, or a computer program may utilize a single module that is responsible for completing all tasks.

[0030] When used in reference to a list of multiple items, the word “or” is intended to cover all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of items in the list.Overview of Visualization and Personalization Platform

[0031] A motion monitoring platform may be responsible for monitoring the motion of an individual (also called a “user,” “patient,” or “participant”) through analysis of digital images that contain her and are captured as she completes a physical activity. A motion monitoring platform can include a visualization and personalization module that determines the quality of a performance of the physical activity by the individual through analysis of the digital images. As an example, the motion monitoring platform may guide the individual through exercise therapy sessions (or simply “sessions”) that are performed as part of an exercise therapy program (or simply “program”). As part of the program, the individual may be requested to engage with the motion monitoring platform on a periodic basis. The frequency with which the individual is requested to engage with the motion monitoring platform may be based on factors such as the anatomical region for which therapy is needed, the MSK condition for which therapy is needed, the difficulty' of the program, the age of the individual, the amount of progress that has been achieved (as determined by the visualization and personalization module), and the like.

[0032] As the individual performs exercises, she may be recorded by a camera of a computing device. Normally, the camera is part of the computing device on which the motion monitoring is executed or accessed. For example, in order to initiate a session, the individual may initiate a mobile application that is stored on, and executable by, hermobile phone or tablet computer, and the mobile application may instruct the individual to position her mobile phone or tablet computer in such a manner that one of its cameras can record her as exercises are performed. Note that, in some embodiments, the camera is part of another computing device. For example, the camera may be included in a peripheral computing device, such as a web camera (also called a “webcam”), that is connected to the computing device. By examining the digital images that are output by the camera, the visualization and personalization module can determine the quality of the performance and provide personalized feedback to the individual over time.

[0033] As mentioned above, the visualization and personalization module could alternatively provide personalized feedback about the quality of performance of activities in contexts that are unrelated to healthcare, for example, to improve technique. As an example, the visualization and personalization module may determine the quality of a performance and provide personalized feedback to an individual while she completes a sporting activity (e.g., performs a dance move, performs a yoga move, shoots a basketball, throws a baseball, swings a golf club), a cooking activity, an art activity, etc. Accordingly, while embodiments may be described in the context of an individual who completes an exercise during a session as part of a program, the features of those embodiments may be similarly applicable to individuals performing other ty pes of physical activities. Individuals whose performances of physical activities are analyzed may be referred to as “individuals” of the motion monitoring platform, even if these individuals have little to no opportunity to interact with the motion monitoring platform.

[0034] Figure 1 illustrates a network environment 100 that includes a visualization and personalization module 102 that is executed by a computing device 104. Individuals can interact with the visualization and personalization module 102 via interfaces 106. For example, individuals may be able to access interfaces that are designed to guide them through physical activities, indicate progress, present personalized feedback, etc. As another example, individuals may be able to access interfaces through which information regarding completed physical activities can be reviewed, personalized feedback can be provided, etc. Thus, interfaces 106 may serve as informative spaces, or the interfaces 106 may serve as collaborative spaces through which individuals and coaches can communicate with one another.

[0035] As show n in Figure 1 , the visualization and personalization module 102 may reside in a network environment 100. Thus, the computing device on which the visualization and personalization module 102 is executing may be connected to one or more networks 108A-B. Depending on its nature, the computing device 104 could be connected to a personal area network (“PAN”), local area network (“LAN”), wide area network (“WAN”), metropolitan area network (“MAN”), or cellular network. For example, if the computing device 104 is a mobile phone, then the computing device 104 may be connected to a computer server of a server system 110 via the Internet. As another example, if the computing device 104 is a computer server, then the computing device 104 may be accessible to individuals via respective computing devices that are connected to the Internet via LANs.

[0036] The interfaces 106 may be accessible via a web browser, desktop application, mobile application, or another form of computer program. For example, to interact with the visualization and personalization module 102, an individual may initiate a web browser on the computing device 104 and then navigate to a web address associated with the visualization and personalization module 102. As another example, an individual may access, via a desktop application or mobile application, interfaces that are generated by the visualization and personalization module 102 through which she can select physical activities to complete, review analyses of her performance of the physical activities, and the like. Accordingly, interfaces generated by the visualization and personalization module 102 may be accessible via various computing devices, including mobile phones, tablet computers, desktop computers, wearable electronic devices (e.g., watches or fitness accessories), virtual reality systems, augmented reality systems, and the like.

[0037] Generally, the visualization and personalization module 102 is hosted, at least partially, on the computing device 104 that is responsible for generating the digital images to be analyzed, as further discussed below. For example, the visualization and personalization module 102 may be embodied as a mobile application executing on a mobile phone or tablet computer. In such embodiments, the instructions that, when executed, implement the visualization and personalization module 102 may reside largely or entirety on the mobile phone or tablet computer. Note, however, that themobile application may be able to access a server system 110 on which other aspects of the visualization and personalization module 102 are hosted.

[0038] In some embodiments, aspects of the visualization and personalization module 102 are executed by a cloud computing service operated by, for example, Amazon Web Services®, Google Cloud Platform™, or Microsoft Azure®. Accordingly, the computing device 104 may be representative of a computer server that is part of a server system 110. Often, the server system 110 is comprised of multiple computer servers. These computer servers can include information regarding different physical activities; computer-implemented models (or simply ‘'models”) that indicate how anatomical regions should move when a given physical activity is performed; computer-implemented templates (or simply “templates”) that indicate how anatomical regions should be positioned when partially or fully engaged in a given physical activity; algorithms for processing image data from which spatial position of anatomical regions can be computed, inferred, or otherwise determined; individual data such as name, age, weight, ailment, enrolled program, duration of enrollment, and number of physical activities completed; and other assets.

[0039] Figure 2 illustrates an example of a computing device 200 that is able to execute a visualization and personalization module 212. As mentioned above, the visualization and personalization module 212 can facilitate the performance of physical activities by an individual, for example, by providing instructions or encouragement. As shown in Figure 2, the computing device 200 can include a processor 202, memory 204, display mechanism 208, communication module 208, image sensor 210A, audio output mechanism 222, and audio input mechanism 224. Each of these components is discussed in greater detail below.

[0040] Those skilled in the art will recognize that different combinations of these components may be present depending on the nature of the computing device 200. For example, if the computing device 200 is a computer server that is part of a server system (e.g., server system 110 of Figure 1), then the computing device 200 may not include the display mechanism 206, image sensor 210A, audio output mechanism 222, or audio input mechanism 224, though the computing device 200 may be communicatively connectable to another computing device that does include a display mechanism, an image sensor, an audio output mechanism, or an audio input mechanism.

[0041] The processor 202 can have generic characteristics similar to general- purpose processors, or the processor 202 may be an application-specific integrated circuit (“ASIC”) that provides control functions to the computing device 200. As shown in Figure 2, the processor 202 can be coupled to all components of the computing device 200. either directly or indirectly, for communication purposes.

[0042] The memory 204 may be comprised of any suitable type of storage medium, such as static random-access memory’ (“SRAM”), dynamic random-access memory (“DRAM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, or registers. In addition to storing instructions that can be executed by the processor 202, the memory 204 can also store data generated by the processor 202 (e.g., when executing the modules of the visualization and personalization module 212) and produced, retrieved, or obtained by the other components of the computing device 200. For example, data received by the communication module 208 from a source external to the computing device 200 (e.g., image sensor 210B) may be stored in the memory 204, or data produced by the image sensor 210A may be stored in the memory 204. Note that the memory 204 is merely an abstract representation of a storage environment. The memory 204 could be comprised of actual integrated circuits (also referred to as “chips”).

[0043] The display mechanism 206 can be any mechanism that is operable to visually convey information to an individual. For example, the display mechanism 206 may be a panel that includes light-emitting diodes (“LEDs”), organic LEDs, liquid crystal elements, or electrophoretic elements. In some embodiments, the display mechanism 206 is touch sensitive. Thus, an individual may be able to provide input to the visualization and personalization module 212 by interacting with the display mechanism 206. Alternatively, the individual may be able to provide input to the visualization and personalization module 212 through some other control mechanism.

[0044] The communication module 208 may be responsible for managing communications external to the computing device 200. For example, the communication module 208 may be responsible for managing communications with other computing devices (e.g., server system 110 of Figure 1, or a camera peripheral such as video camera or w ebcam). The communication module 208 may be wireless communication circuitry that is designed to establish communication channels withother computing devices. Examples of wireless communication circuitry include 2.4 gigahertz (“GHz”) and 5 GHz chipsets compatible with Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 - also referred to as “Wi-Fi chipsets.” Alternatively, the communication module 208 may be representative of a chipset configured for Bluetooth®, Near Field Communication (“NFC”), and the like. Some computing devices - like mobile phones and tablet computers - are able to wirelessly communicate via separate channels. Accordingly, the communication module 208 may be one of multiple communication modules implemented in the computing device 200. As an example, the communication module 208 may initiate and then maintain one communication channel with a camera peripheral (e.g.. via Bluetooth), and the communication module 208 may initiate and then maintain another communication channel with a server system (e g., via the Internet).

[0045] The nature, number, and type of communication channels established by the computing device 200 - and more specifically, the communication module 208 - may depend on the sources from which data is received by the visualization and personalization module 212 and the destinations to which data is transmitted by the visualization and personalization module 212. Assume, for example, that the computing device 200 is representative of a mobile phone or tablet computer that is associated with (e.g., owned by) an individual. In some embodiments the communication module 208 may only externally communicate with a computer server, while in other embodiments the communication module 208 may also externally communicate with a source from which to receive image data. The source could be another computing device (e.g., a mobile phone or camera peripheral that includes an image sensor 210B) to which the mobile device is communicatively connected. Image data could be received from the source even if the mobile phone generates its own image data. Thus, image data could be acquired from multiple sources, and these image data may correspond to different perspectives of the individual performing a physical activity. Regardless of the number of sources, image data - or analyses of the image data - may be transmitted to the computer server for storage in a digital profile that is associated with the individual. The same may be true if the visualization and personalization module 212 only acquires image data generated by the image sensor 210A. The image data may initially be analyzed by the visualization and personalization module 212, and then the image data- or analyses of the image data - may be transmitted to the computer server for storage in the digital profile.

[0046] The image sensor 210A may be any electronic sensor that is able to detect and convey information in order to generate images, generally in the form of image data (also called “pixel data”). Examples of image sensors include charge-coupled device (“CCD”) sensors and complementary metal-oxide semiconductor (“CMOS”) sensors. The image sensor 210A may be part of a camera module (or simply “camera”) that is implemented in the computing device 200. In some embodiments, the image sensor 210A is one of multiple image sensors implemented in the computing device 200. For example, the image sensor 210A could be included in a front- or rear-facing camera on a mobile phone. Alternatively, the image sensor 210A may be externally connected to the computing device 200 such that the image sensor 210A captures image data of an environment and sends the image data to the to the visualization and personalization module 212.

[0047] For convenience, the visualization and personalization module 212 may be referred to as a computer program that resides in the memory 204. However, the visualization and personalization module 212 could be comprised of hardware or firmware in addition to. or instead of, software. In accordance with embodiments described herein, the visualization and personalization module 212 may include a processing module 214, pose estimating module 216, analysis module 218, and graphical user interface (“GUI”) module 220. These modules can be an integral part of the visualization and personalization module 212. Alternatively, these modules can be logically separate from the visualization and personalization module 212 but operate “alongside” it. Together, these modules may enable the visualization and personalization module 212 to programmatically monitor motion of individuals during the performance of physical activities, such as exercises, through analysis of digital images generated by the image sensor 210.

[0048] The processing module 214 can process image data obtained from the image sensor 210A over the course of a session. The image data may be used to infer a spatial position or orientation of one or more anatomical regions as further discussed below. The image data may be representation of a series of digital images. These digital images may be discretely captured by the image sensor 210A over time, such that each digitalimage captured the individual at different stages of performing a physical activity. In some embodiments, these digital images may be representative of frames of a video that is captured by the image sensor 210. In such embodiments, the image data could also be called “video data.”

[0049] The image data may be used to infer a spatial position of one or more anatomical regions as further discussed below. For example, the processing module 214 may perform operations (e.g., filtering noise, changing contrast, reducing size) to ensure that the data can be handled by the other modules of the visualization and personalization module 212. As another example, the processing module 214 may temporally align the data with data obtained from another source (e.g., another image sensor) if multiple data are to be used to establish the spatial position of the anatomical regions of interest.

[0050] Moreover, the processing module 214 may be responsible for processing information input by individuals through interfaces generated by the GUI module 220. For example, the GUI module 220 may be configured to generate a series of interfaces that are presented in succession to an individual as she completes physical activities as part of a session. On some or all of these interfaces, the individual may be prompted to provide input. For example, the individual may be requested to indicate (e.g., via a verbal command or tactile command provided via, for example, the display mechanism 206) that she is ready to proceed with the next physical activity, that she completed the last physical activity, that she would like to temporarily pause the session, etc. These inputs can be examined by the processing module 214 before information indicative of these inputs is forwarded to another module.

[0051] The pose estimating module 216 (or simply “estimating module”) may be responsible for estimating the pose of the individual through analysis of image data, in accordance with the approach further discussed below. Specifically, the estimating module 216 can create, based on a digital image (e g., generated by the image sensor 210A or image sensor 210B), a skeletal frame that specifies a spatial position of each of multiple anatomical regions. For example, the estimating module 216 can apply a computer-implemented model (or simply “model”) referred to as a pose estimator to the digital image, so as to produce the skeletal frame. In some embodiments the pose estimator is designed and trained to identify a predetermined number of joints (e.g., leftand right wrist, left and right elbow, left and right shoulder, left and right hip, left and right knee, left and right ankle, or any combination thereof), while in other embodiments the pose estimator is designed and trained to identify all joints that are visible in the digital image provided as input. The pose estimator could be a neural network that when applied to the digital image, analyzes the pixels to independently identify digital features that are representative of each anatomical region of interest.

[0052] The analysis module 218 may be responsible for establishing the locations of anatomical regions of interest based on the outputs produced by the estimating module 216. Referring again to the aforementioned examples, the analysis module 216 could establish the locations of joints based on an analysis of the skeletal frame. Moreover, the analysis module 218 may be responsible for determining appropriate feedback for the individual based on the outputs produced by the estimating module 216, in accordance with the approach further discussed below. Specifically, the analysis module 218 may determine an appropriate personalized recommendation for the individual based on her current position, and a determination as to how her current position compares to a template that is associated with the physical activity that she has been instructed to perform.

[0053] Other modules could also be included in some embodiments. For example, the visualization and personalization module 212 may include a training module (not shown) that is responsible for training the pose estimator that is employed by the pose estimating module 216. As another example, the visualization and personalization module 212 may include a template generating module (not shown) that is responsible for generating templates that are used by the analysis module 218 to determine which recommendations, if any, are appropriate for an individual given her current position.

[0054] Similarly, other components could be implemented in, or accessible to, the computing device 200 in some embodiments. For example, some embodiments of the computing device 200 include an audio output mechanism 222 and / or an audio input mechanism 224. The audio output mechanism 222 may be any apparatus that is able to convert electrical impulses into sound. One example of an audio output mechanism is a loudspeaker (or simply “speaker”). Meanwhile, the audio input mechanism 224 may be any apparatus that is able to convert sound into electrical impulses. One example of an audio input mechanism is a microphone. Together, the audio output and inputmechanisms 222, 224 may enable feedback, such as personalized recommendations as further discussed below, to be audibly provided to the individual. Assume, for example, that the individual has been instructed to perform a physical activity while being recorded by the image sensor 21 OA. In such a scenario, the individual may be audibly encouraged - in a personalized manner - via the audio output mechanism 222.Monitoring Physical Activities and Providing Personalized Feedback

[0055] Various attempts have been made to improve engagement in programs that require performance of physical activities on a periodic basis. Consider, for example, an exercise therapy program that requires exercises be performed by an individual to achieve a therapeutic goal. The individual may be consistently notified, for example, via text message, email message, or push notification, but the individual may still struggle to adhere to the exercise therapy program. Simply put, because this feedback is not tailored or personalized in any way, the individual may quickly become '‘immune7’ to this feedback.

[0056] Introduced here are approaches to determining the quality of performance of a physical activity by an individual and providing feedback in a personalized manner during the performance of the physical activity. The disclosed approaches are implemented in a visualization and personalization module that can be included in a motion monitoring platform. With this approach, there are several advantages over conventional approaches that rely on generalized feedback.

[0057] First, the visualization and personalization module may implement a generic state machine to model physical activities, and the generic state machine may assume a limited number of states - making computations faster and less computationally intense. For example, the generic state machine may be programmed to assume only (i) a relaxed state, (ii) an engaging state, (iii) an engaged state, and (iv) a relaxing state. As discussed above, the generic state machine could be programmed for various numbers of states, and the number of states may vary for different physical activities. Transitions between states may be defined by generic sets of conditions that can be automatically composed, inferred, or otherwise derived by the visualization and personalization module. This approach enables data-driven definitions of physical activities that can bequickly defined and validated by experts (e.g., healthcare professionals, such as physiotherapists).

[0058] Second, the visualization and personalization module can utilize a templatebased approach by matching skeletal frames extracted from digital images of an individual against different reference skeletal frames, so as to determine a quality of a performance of a physical activity by the individual. As further discussed below, these reference skeletal frames can be captured or determined as part of a template generation operation in which a pose estimator is applied to digital images that capture a reference performance of a given physical activity. If the physical activity is an exercise, for example, the reference performance may be completed by a physiotherapist. This approach to developing and applying templates enables rapid scaling, by allowing an expert to perform a physical activity’ at least once and then having the ideal poses for each state of the physical activity to be extracted and set as criteria for repetition counting in an automated way.

[0059] Third, when an individual performs an exercise repetition using a conventional exercise therapy platform, the exercise therapy platform may provide a repetition count but no indication of the quality of the performance. Using the visualization and personalization module, an individual is informed of how well she performed each repetition. For example, the visualization and personalization module can produce a metric that is indicative of similarity between the performance and a reference performance of the physical activity'. Further, the visualization and personalization module can manipulate an appearance of a graphical element on an interface, such that the appearance varies over the performance to reflect how closely the performance matches the reference performance.

[0060] Fourth, the visualization and personalization module can personalize the monitoring of the quality of a performance by manipulating the metric, such that the metric reflects how closely the performance matches the reference performance in view of historical performances by the individual, physical abilities and health of the individual, or input received from the individual. For example, the individual can dial in a difficulty level for an exercise, and the visualization and personalization module would correspondingly adjust a threshold performance to compare the observed performance against.

[0061] Figure 3 depicts an example of a communication environment 300 that includes a visualization and personalization module 324 configured to obtain data from one or more sources. Here, the visualization and personalization module 324 may obtain data from a therapy system 304 comprised of a tablet computer 312 and one or more sensor units 308 (e.g., image sensors), personal computer 316, or network- accessible server system 320 (collectively referred to as the “networked devices”). For example, the visualization and personalization module 324 may obtain data regarding movement of an individual during a session from the therapy system 304 and other data (e.g., therapy regimen information, models of exercise-induced movements, personalized feedback from coaches, and processing operations) from the personal computer 316 or network-accessible server system 320.

[0062] The networked devices can be connected to the visualization and personalization module 324 via one or more networks. These networks can include PANs, LANs, WANs, MANs, cellular networks, the Internet, etc. Additionally or alternatively, the networked devices may communicate with one another over a short- range wireless connectivity technology. For example, if the visualization and personalization module 324 resides on the tablet computer 312, data may be obtained from the sensor units over a Bluetooth communication channel, while data may be obtained from the network-accessible server system 320 over the Internet via a Wi-Fi communication channel.

[0063] Embodiments of the communication environment 300 may include a subset of the networked devices. For example, some embodiments of the communication environment 300 include a visualization and personalization module 324 that obtains data from the therapy system 304 (and, more specifically, from the sensor units 308) in real time as physical activities as performed during a session and additional data from the network-accessible server system 320. This additional data may be obtained periodically (e.g., on a daily or weekly basis, or when a session is initiated).

[0064] Figure 4 illustrates an example of a visualization and personalization module 400 that is executed by a computing device. As mentioned above, a visualization and personalization module 400 can facilitate the performance of physical activities by an individual, for example, by providing instructions, personalized feedback, or encouragement. The visualization and personalization module 400 can bepart of a motion monitoring platform that monitors the motion of an individual through analysis of digital images that contain her and are captured as she completes a physical activity. As shown in Figure 4, the visualization and personalization module 400 can include an exercise recognition engine 410, a visualization component 420, and a personalization component 430. Each of these components is discussed in greater detail below.

[0065] The visualization and personalization module 400 and the processes depicted by Figures 8, 9, and 10 use a computer vision engine (also called a “CV engine”) to estimate poses of a human body that is viewable in digital images, feature extraction algorithms (also called “feature extractors”) to expose relevant features of the poses corresponding to different states, and an analysis module (e.g., the analysis module 218 of Figure 2) to check the definition of the stored (i.e.. static) template for a physical activity and then initiate personalized feedback based on an analysis of the features of the current state of the human body, stored parameters related to the overall user experience for the physical activity, and events that are propagated to the individual. Figure 6 includes illustrations of different states for an example physical activity (here, a clamshell stretch and squat).

[0066] The CV engine and feature extractors may be employed by the analysis module. The CV engine may be implemented in software, firmware, hardware, or a combination thereof. Examples of CV engines include OpenPose, MediaPipe, Kinect, and proprietary CV engines developed for estimating pose. When applied to a digital image, the CV engine may extract relevant features of human bodies included therein, such as the 2D or 3D positions of key points, 2D shape information, 3D shape information, surface information, and the like.

[0067] Set forth below is a summary of the approach for employing a templatebased approach to monitoring the location of anatomical regions of interest and determining appropriate personalized feedback. At a high level, the approach may involve the identification of physical activities (Figure 7), such as determining repetitions or holds of short exercises, and the identification of appropriate personalized feedback (Figure 5), such as form feedback or encouragement feedback.

[0068] As mentioned above, the processes performed by the personalization module 400 may involve the use of a CV engine that is applied by the analysis module to estimate the poses of an individual included in digital images. Moreover, the analysis module may use (i) a feature extractor to expose relevant features of the current state of the individual, (ii) a feedback engine (also called a “feedback checker” or “condition checker”) to check the definition of the feedback criteria (also called “feedback triggers”) against the current state of the individual, (iii) stored parameters of the overall user experience for the physical activity, and (iv) events that can be propagated to the individual.

[0069] A physical activity state machine (or simply “state machine”) may be defined as a system, implemented in software, firmware, or hardware, that can be in one of a set number of stable conditions - referred to as “states” - depending on its previous state and the present value(s) of its input(s). Specifically, the state machine may contain conditions that model the relevant states of a human body performing a given physical activity. The number of states associated with the given physical activitymay depend on the difficulty of the given physical activity and total range of motion required, among other things. For example, for a simple repetitive physical activity - like an exercise that requires holding a pose - a set of states may include (i) an engaged state that corresponds to the individual having achieved a valid pose for the exercise, (ii) a pre-engagement state that corresponds to the individual moving into the valid pose, and (iii) a post-engagement state that corresponds to the individual moving out of the valid pose. While the number of states included in a set generally is no less than three, a set could include more than three states. A set could include between four and twelve states (and preferably, between four and eight states).

[0070] For example, a set of states may include (i) an engaged state that corresponds to the individual having achieved a valid engaged pose for a physical activity, (ii) an engaged pre-extremum state that corresponds to the individual having achieved the valid engaged pose but not having achieved her personal maximal engagement with the valid engaged pose, (iii) an engaged post-extremum state that corresponds to the individual having achieved her personal maximal engagement and beginning to return to a valid relaxed pose, (iv) a relaxed state that corresponds to the individual beginning to disengage from the valid engaged pose but not yet in whatwould be considered the valid relaxed pose, (v) a relaxed pre-extremum pose that corresponds to the individual having achieved the valid relaxed pose but not having achieved her personal maximum engagement with the valid relaxed pose, (vi) a relaxed post-extremum pose that corresponds to the individual having achieved her personal maximal engagement and holding the valid relaxed pose, (vii) a start state, and (viii) an end state.

[0071] Broadly, the term “engaging” may be used to refer to an individual moving towards the valid engaged pose for the physical activity but not yet having achieved the valid engaged pose (e.g., bending knees and hips in an attempt to perform a squat). Meanwhile, the term “relaxing” may be used to refer to an individual beginning to disengage from a valid engaged pose and return to a valid relaxed pose. Accordingly, whereas the engaged pre-extremum and post-extremum states may be used to describe scenarios where an individual is bending her knees and hips to perform a squat, the related pre-extremum and post-extremum states may be used to describe scenarios where the individual is nearly fully standing and fully standing after having performed the squat. The distinction of the pre-extremum and post-extremum states may be made to facilitate execution or presentation of transition events to the individual that signal whether her maximal poses have been reached while maintaining the ability to recognize repetitions with a permissive set of conditions on the engaged and relaxed states that can generalize to a large population of individuals with different abilities and movement styles.

[0072] The term “transition event” may be used to refer to any event that can be exposed, by the analysis module, to the rest of the motion monitoring platform or directly to the individual to indicate a transition in the state of a physical activity. For example, transition events may be used to identify when the requirements for engagement have been met, identify the maximal engagement of a physical activity, identify when to increment a repetition counter upon relaxing following engagement, record an event notifying the completion of all repetitions for a given physical activity, and the like. Transition events may be strictly internal events to either notify other modules of the motion monitoring platform of state or be associated with audible or visual cues, such as a sound effect that serves as a notification of the completion of aset of repetitions or a visual effect that serves as a visualization of movement through the engaged state.

[0073] A data structure that is called a "‘physical activity definition’7or simply ‘‘definition” may be stored in the memory and contain information about how the physical activity is defined. If the physical activity is an exercise, for example, the definition may include metadata about the type of exercise, preferred user state and placement within the view of the camera, and a list of heuristic conditions that define the conditions for specific state transitions within that exercise. A heuristic condition may contain some description of one or more state features (e.g., a specific pose encoded in a template, a specific joint position or flexion angle, a current state, a previous state, a time in seconds within a given state, a number of complete repetitions of the exercise, a list of other heuristic conditions), a mathematical condition on those state features (e.g., the value of the feature or some metric determined from the feature being less than, equal to, or greater than a threshold, a comparison between the values of two state features), or a score that may be based on the degree of acceptance of the aforementioned mathematical condition and may be used to rank valid conditions.

[0074] Pose features, mentioned above, may be specific poses that are encoded as 2D or 3D coordinates, for example, in a standardized coordinate system with the human body facing towards the +Z direction, with +Y representing the up direction (i.e., such that key points associated with the head will have larger Y-values than key points associated with the feet for standing poses), and +X facing the right direction. For a 3D coordinate system, the coordinates may be relative to an origin defined at the center of the pelvis of the human body (or another part of the human body, the camera, or some arbitrary global origin in space), with measurements scaled by a standardized human template (e.g., at 180 centimeters). In this case, poses of uses may be compared to established template poses for physical activities in a standardized coordinate system.

[0075] Pose features may instead be encoded as joint flexion angles (in degrees or radians) of combinations of key points - also called “key joints” - or joint rotations (in quaternions), for example, relative to the pelvis or some global orientation. Pose features may instead be encoded as bone directions in 2D or 3D space, sets of distances between joints, or as principal components values in an established principal component analysis (“PCA”) based vector space (or vectors in some other featureembedding space that may be statistically or geometrically derived). These kinds of pose features may require the use of training data over which to define a population distribution of valid forms per physical activity and may help the system better generalize to not-yet-observed individuals. Pose features may also represent velocities, accelerations, or other time-dependent quantities that quantify the trajectory or movement of an individual's poses over time.

[0076] Pose features may be used individually, for example, as single values (e.g., coordinate values, joint angle values, etc. ). relative to threshold(s). or pose features may be used in combination, for example, using logical operators (e.g., AND, OR, AND / AND) or as a group to a template pose, which itself may be represented by some pose features (e.g., joint coordinates, angles, velocities, etc.). Templates may represent a single pose of a group of poses, such as a statistical group of poses or a region of poses in an appropriate feature space.

[0077] The exercise recognition engine 410 can be part of the pose estimating module 216, which estimates the pose of the individual through analysis of image data. For example, the exercise recognition engine 410 receives a video stream of an individual performing a physical activity, e.g., a squat or a stretch. For example, the individual has been prompted to perform the physical activity via an interface. The interface can be one of the interfaces 106, image sensor 210A, image sensor 210B, or sensor units 308. The video stream can also be captured by or received from the interface. From the video stream, the exercise recognition engine 410 can extract individual video frames that constitute individual digital images. For example, a camera included in the computing device can generate the digital images. A digital image can be extracted from the video stream, for example, every one tenth of a second, every' one fifth of a second, or every' half second. Each digital image can depict the individual either in one of the example states shown by Figure 4 (relaxed, engaging, engaged, and relaxing) or transitioning from one of the four states to another of the four states.

[0078] From the digital images, the exercise recognition engine 410 can generate skeletal frames that specify spatial positions of anatomical regions of the individual. For example, a skeletal frame can be a three-dimensional (3D) skeletal frame, where the spatial positions of the anatomical regions are 3D spatial positions. A skeletal framecan be encoded using 3D coordinates. In some embodiments, a skeletal frame includes 3D shape information extracted from one or more digital images.

[0079] As part of determining the quality of the individual’s performance of the physical activity, the exercise recognition engine 410 compares a generated skeletal frame to a reference skeletal frame of a template associated with the physical activity. For example, the skeletal frame is compared to a reference skeletal frame that is selected by the exercise recognition engine 410 from among a series of reference skeletal frames. The series of reference skeletal frames correspond to a reference performance (e.g., by a physiotherapist) of the physical activity against which the performance is to be compared. In some embodiments, each skeletal frame is compared to a single reference skeletal frame (e.g., corresponding to one of the states of the physical activity depicted by Figure 4) to determine how close the individual is getting to a desired position (e.g., the engaged state).

[0080] Based on comparing the skeletal frames to the reference skeletal frames, the exercise recognition engine 410 produces a metric 440 that is indicative of a degree of similarity between the skeletal frames and the reference skeletal frames. The metric 440 varies continuously as the individual performs the activity or a repetition. The metric 440 is a measure of the quality of the performance of the physical activity compared to the reference performance. The metric 440 can be a number between 0 and 1, where 0 denotes that the performance is poor, and 1 denotes a perfect performance. The metric 440 can be a number between 1 and 100, a number between 50 and 1000. or a letter between A and Z. The metric 440 changes as the individual physically moves and changes poses during the performance of the physical activity, such that the metric 440 is continuously tracking the quality' of the performance.

[0081] The visualization component 420 causes or generates a display of a graphical visualization (e.g., graphical element 422) based on the metric 440. The visualization component 420 can be part of the GUI module 220 illustrated and described in more detail with reference to Figure 2. For example, the visualization component 420 manipulates an appearance of graphical element 422 on an interface, such that the appearance varies over the performance by the individual. The appearance reflects how closely the performance matches the reference performance across multiple anatomical regions of the individual. In some embodiments, the graphicalvisualization includes a longitudinal indicator (e.g., graphical element 422) or a circular indicator (e.g.. graphical element 424). The graphical visualization can include graphical elements having other shapes, or numbers such as a number between 0 and 100.

[0082] In some implementations, the visualization component 420 adjusts a color of a portion of the graphical visualization to indicate a change in the metric 440. For example, when the metric 440 is between 0 and 33, a portion of a longitudinal indicator can be illuminated or colored red to inform the individual that she should try to improve the quality of her performance. Continuing the example, when the metric 440 is between 33 and 66, another portion of the longitudinal indicator can be illuminated or colored yellow to provide encouragement to the individual or to infonn her that she is getting closer to the reference position. When the metric 440 is between 66 and 100, another portion of the longitudinal indicator can be illuminated or colored green to inform the individual that she is performing the physical activity well or that she is close to a reference performance or physical position.

[0083] The visualization component 420 causes digital presentation of the graphical visualization on an interface. For example, the visualization component 420 causes the graphical visualization to be displayed on an electronic screen. Different portions of the graphical visualization (e.g., a longitudinal indicator or a circular indicator) can light up to indicate different values of the metric 440 as the individual goes through the performance. The graphical visualization is continuously updated during the performance.

[0084] The exercise recognition engine 410 can determine from the digital images that the individual has completed a number of repetitions 412 of the physical activity. For example, the exercise recognition engine 410 counts each completed repetition 412 performed by the individual. Each repetition 412 can be defined by transitions from a relaxed state to an engaging state to an engaged state to a relaxing state, and back to the relaxed state, as depicted by Figure 4. For example, if the individual transitioned from the relaxed state to the engaging state, and back to the relaxed state without going to the engaged state, the exercise recognition engine 410 would not count the motion as a repetition. In some embodiments, the exercise recognition engine 410 counts repetitions of the physical activity using a multi-state machine that is programmed torecognize and classify repetitive movement from the relaxed state to the engaging state to the engaged state to the relaxing state to the relaxed state.

[0085] The visualization component 420 causes or generates a display of a graphical visualization (e.g., graphical element 424) based on the number of repetitions. For example, an unlit segment of the graphical element 424 can light up when a repetition is complete, or a segment of the graphical element 424 that is colored yellow changes to green when a repetition is complete. In some implementations, the color of each segment of the graphical element 424 represents the quality of the performance of a repetition associated with that segment. The visualization component 420 causes digital presentation of a graphical visualization (e.g., graphical element 424) denoting the number of completed repetitions on an interface.

[0086] The personalization component 430 manipulates the generated metric 440 based on information 434 describing the individual, user input 436, and / or observations 432 of the individual's performance. The information 434 can be stored on the visualization and personalization module 400 or retrieved from a network-accessible server system 320. By manipulating the metric 440, the personalization component personalizes the metric 440 and the feedback delivered to the individual based, for example, on the individual’s physical abilities and health characteristics or historical performances of the physical activity by the individual.

[0087] The personalization component 430 can receive information describing historical performances of the physical activity by the individual, for example, from a server that stores data describing the historical performances. The data stored can include metrics determined for previous performances by the individual. The personalization component 430 can receive information describing physical abilities and health of the individual. For example, the information can describe injuries the individual has suffered, medical information, height, weight, age, or agility data for the individual. The personalization component 430 can receive input 436 from the individual via an interface. For example, the input 436 can be entered by an individual through interfaces generated by the GUI module 220. The input 436 can be processed by the processing module 214 illustrated described in more detail with reference to Figure 2. For example, before or during the performance, the individual can dial in a preferred speed or difficulty level for the physical activity. The input 436 can bepresented to the interface using a keyboard (for example on a touchscreen), a smart speaker (e.g.. audio input mechanism 224), or by a hand gesture that is captured by a camera of the interface.

[0088] The personalization component 430 can receive observations 432 either from the exercise recognition engine 410 or from the interface. For example, observations 432 received from the exercise recognition engine 410 can include changes in the metric over time during repetitions, a speed at which each repetition is performed, or the amount of time spent in different states. The observations 432 received from the interface can include video, digital images, or sounds that describe how difficult or easy the individual is finding the physical activity.

[0089] The personalization component 430 can produce adaptation parameters 444 for manipulating of the metric 440 based on the information 434, the input 436, and / or the observations 432. The exercise recognition engine 410 can use the adaptation parameters 444 to manipulate or personalize the metric 440. and send a personalized version of the metric 440 to the visualization component 420. For example, the personalized version of the metric 440 reflects how closely the performance matches the reference performance in view of the historical performances by the individual, the physical abilities and health of the individual, and / or the input 436 received from the individual. The personalized version of the metric 440 can also reflect how closely the performance matches the reference performance in view of the observations 432 (e.g., how difficult or easy the individual is finding the physical activity).

[0090] In some embodiments, to manipulate the metric, the personalization component 430 provides the information 434, the input 436, and / or the observations 432 to a machine learning model so as to obtain a personalized version of the metric 440. The personalized version of the metric 440 is produced by the machine learning model as output. The machine learning model can include one or more model parameters that are received from a source external to the computing device and associated with a generic machine learning model designed and trained to personalize the metric 440 to the individual.

[0091] For example, model parameters for models associated with the personalization component 430 can be trained or generated based on training datapertaining to multiple individuals, for example, in the case of a generic machine learning model. Additionally or alternatively, local versions of the machine learning model can include model parameters that are trained on data pertaining to a particular individual or subset of individuals (either independently, or further trained based on the generic model’s parameters). By personalizing such model parameters, the personalization component 430 can provide improved estimations of the quality of the performance that are more sensitive to an individual’s particular characteristics and / or environment.

[0092] In disclosed embodiments, the personalization component 430 and / or the exercise recognition engine 410 can transmit model parameters to other entities. For example, the visualization and personalization module 400 can transmit sets of model parameters corresponding to a local machine learning model for estimating the quality of the performance to an external server (e.g., server system 108) and / or another device, for further processing. By doing so, the visualization and personalization module 400 enables updating of generic model parameters based on local versions of models, as such local versions can rely on more personalized training data for model fine-tuning. In some cases, this personalized training data may not be available to external sources due to privacy concerns or regulatory constraints. As such, by transmitting updated model parameters to relevant entities, the visualization and personalization module 400 enables such trained model parameters to be leveraged in updating even generic models stored external to individuals’ computing devices.

[0093] The personalization component 430 or the exercise recognition engine 410 can obtain video or digital images depicting the individual completing the performance of the physical activity, and provide the digital images to a machine learning model. The video or digital images are used to tune the machine learning model to reflect how closely the present performance matches the historical performance of the physical activity by the individual. For example, the machine learning model can learn with each repetition or each exercise session whether the individual is performing the physical activity better or worse compared to the reference performance. The personalization component 430 can use the continuously updated machine learning model to generate the adaptation parameters 444 or a personalized version of the metric 440.

[0094] For example, the personalization component 430 can generate the adaptation parameters 444 or a personalized version of the metric 440 using one or more machine learning models designed and trained for feedback personalization, which can include a neural network, artificial intelligence, or computer-based analytical method. A machine learning model can be any software or hardware tool that can leam from data and make predictions, classifications, or inferences based on this data. In some embodiments, the machine learning model can include one or more algorithms, including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, deep learning, neural networks, decision trees, support vector machines, and k-means clustering. For example, the machine learning model can be implemented as a convolutional neural network (or feed forward network, recurrent neural network, random forest, or Xgboost model). The machine learning model can include any model that can accept, for example, the metric 440, the information 434, the input 436, the observations 432, and / or one or more digital images or video frames as input. The machine learning model can produce the adaptation parameters 444 or a personalized version of the metric to reflect how closely the individual’s performance matches the reference performance in view of the individual’s personal characteristics (e.g., physical ability), as described herein.

[0095] The personalization component 430 can manipulate a threshold performance based on the information 434, the input 436, and / or the observations 432 of the present performance. The exercise recognition engine 410 can use the threshold performance in addition to or instead of the reference performance to calculate the metric 440. In some scenarios, the threshold performance is the reference performance. For example, to personalize the metric 440, the exercise recognition engine 410 can compare the present performance to a threshold performance that is different from the reference performance to obtain a metric that is tuned to the individual’s abilities. The threshold performance can be lowered, such that the individual achieves a higher score or metric even when she is not performing at the level of the reference performance. Conversely, the threshold performance can be increased to make the metric calculation more challenging.

[0096] The personalization component 430 can manipulate the threshold performance to reflect a change in the physical abilities and health of the individualover time. For example, different individuals may have widely ranging levels of physical abilities. For some individuals, a shallow squat can be a challenging task, while to other individuals a shallow squat can be easy and not challenging enough. The threshold performance can be used as a factor to influence the difficulty' of the physical activity, for example, to personalize a higher score threshold for individuals having greater physical capabilities.

[0097] The personalization component 430 can manipulate the threshold performance to reflect a change in the physical abilities and health of the individual over time. As physical abilities of the individual improve, the threshold performance can be changed, such that a particular performance by the individual is scored lower. For example, an individual could be recovering from an injury and have limited capability’ or movement range, but gains more movement range with time. The CV methods described herein can be used to detect such changes over time, and automatically adjust the threshold performance to better serve the individual. The personalization component 430 can use the metric 440 to derive information on how well the individual is performing specific exercises. For example, consistent low values of the metric 440 can be used by the personalization component 430 to adjust the threshold performance to render evaluations of the metric 440 more permissive.

[0098] Figure 5 illustrates examples of digital presentations of graphical elements on an interface (e.g., interfaces 106) by the visualization and personalization module 400. For example, to cause display of the graphical elements 512, 518, 552 that provide feedback to the individual depicted by Figure 5, the visualization and personalization module 400 obtains the digital images 510, 550 in temporal order as the individual is performing a physical activity (here, a hold exercise). For example, digital image 510 depicts an individual performing a physical activity’ at a first time, and digital image 550 depicts the same individual performing the same activity at a second time later than the first time.

[0099] The visualization and personalization module 400 generates a skeletal frame for each of the digital images 510, 550. The skeletal frames specify spatial positions of multiple anatomical regions 514, 554 of the individual. For example, the anatomical regions can be joints, limbs, or organs. The visualization and personalization module 400 compares the skeletal frames to one or more reference skeletal frames. Forexample, a reference skeletal frame is selected from among a series of reference skeletal frames that correspond to a reference performance or a threshold performance of the physical activity against which the individual’s performance is to be compared. The visualization and personalization module 400 produces a metric (e.g., metric 440), based on the comparison of the skeleton frames. The metric is indicative of similarity between the skeletal frames and the reference skeletal frames. Based on the metric, the visualization and personalization module 400 manipulates an appearance of the graphical elements 512, 518, 552 on the interface, such that the appearance varies over the performance to reflect how closely the present performance matches the reference performance across the multiple anatomical regions 514, 554.

[0100] The graphical elements 512, 552 are longitudinal indicators including vertical bars that light up from left to right as the metric increases. For example, as shown by Figure 5, the bars in a left portion of graphical element 512 are yellow, the bars in a central portion of graphical element 512 are light green, and the bars in a right portion of graphical element 512 are bright green. When only some or all of the bars in a left portion of graphical element 512 are illuminated, the graphical element 512 may indicate that the performance is not close to the reference performance. When the bars in the central portion of graphical element 512 begin lighting up, the graphical element 512 may indicate that the performance is improving. When the bars in the right portion of graphical element 512 begin lighting up, the graphical element 512 may indicate that the performance is approaching the reference performance. The appearance of the graphical elements 512, 552 on the interface varies continuously over the performance. A comparison of the graphical elements 512, 552 shows that the portion of the performance depicted by digital images 510 is objectively better than the portion of the performance depicted by digital images 550.

[0101] The graphical element 518 indicates a number of completed repetitions of the physical activity recognized by the visualization and personalization module 400. For example, the visualization and personalization module 400 determines, from the digital images, that the individual has completed a number of repetitions, where each repetition is defined by transitions from a relaxed state to an engaging state to an engaged state to a relaxing state, and back to the relaxed state. Display of graphical element 518 is caused based on the number of repetitions counted.

[0102] Figure 6 illustrates how a template including skeletal frames can be captured from poses in a video where an expert (e.g.. a physiotherapist) showcases the ideal movement and, in some embodiments, undesired variations for which feedback is to be provided.

[0103] Figure 7 includes an example of skeletal frames being matched against a template prepared for a given physical activity' (here, a squat). For each digital image (e.g., each frame of a video), a motion monitoring platform (and more specifically, its analysis module) can employ a matching algorithm that goes through the following steps:• Step One: Pose estimator generates a skeletal frame of an individual in the digital image.• Step Two: The skeletal frame is aligned such that the hips are flat in the camera plan.• Step Three: For the skeletal frame, selected feature(s) are computed based on the corresponding definition and features absent from a “key_features” list are masked. o Example:Squat Pose Features = 3D Joint LocationExcluded Features = Elbow L, Elbow R, Wrist L,Wrist R• Step Four: Given a metric function, a distance is computed between the skeletal frame and a corresponding template that is created or captured at exercise creation time. For each digital image, the state with the lowest distance can be selected as shown in Figure 7.• Step Five: The skeletal frame is classified to the state with the lowest template distance.

[0104] Figure 8 depicts a flow diagram of a process for manipulating an appearance of a graphical element on an interface (e.g., interfaces 106) to reflect how closely a performance of a physical activity matches a reference performance of the physical activity (here, an exercise).

[0105] At step 804. the visualization and personalization module 400 obtains digital images depicting an individual who has been prompted to perform a physical activity via an interface of a computing device (e.g., computing device 104). For example, the visualization and personalization module 400 can be hosted, at least partially, on the computing device that is responsible for generating the digital images to be analyzed. The visualization and personalization module 400 may be embodied as a mobile application executing on a mobile phone or tablet computer.

[0106] At step 808, the visualization and personalization module 400 generates a respective skeletal frame (for each digital image) that specifies spatial positions of anatomical regions of the individual. For example, the respective skeletal frame is a 3D skeletal frame, and the spatial positions of the anatomical regions are 3D spatial positions. The visualization and personalization module 400 can create a skeletal frame for each digital image and then compare the skeletal frame to a corresponding reference skeletal frame of a template. The template is representative of a series of reference skeletal frames, in order, that indicate how the physical activity is to be performed.

[0107] At step 812, for each digital image or for each respective skeletal frame, the visualization and personalization module 400 compares the respective skeletal frame to a reference skeletal frame of a template associated with the physical activity. For example, the reference skeletal frame can correspond to a particular state (e.g., an engaged state) of the physical activity, or the reference skeletal frame can be selected from among a series of reference skeletal frames that correspond to a reference performance of the physical activity against which the performance is to be compared.

[0108] At step 816, the visualization and personalization module 400 determines similarity between the respective skeletal frames and the reference skeletal frame. For example, the visualization and personalization module 400 determines a metric for each digital image or each respective skeletal frame that is representative of a degree of similarity between the respective skeletal frame and the reference skeletal frame. The metric can be computed based on a distance between the skeletal frame and the reference skeletal frame based on a relative distance function.

[0109] At step 820. the visualization and personalization module 400 causes display of a first graphical visualization (e.g., graphical element 422 or graphicalelement 512) based on the similarity reflected by the metric. In some embodiments, the graphical visualization includes a longitudinal indicator or a circular indicator, and can also include graphical elements having other shapes, or letters or numbers.

[0110] At step 824, the visualization and personalization module 400 determines, from the digital images, that the individual has completed a number of repetitions of the physical activity. Each repetition is defined by transitions from a relaxed state to an engaging state to an engaged state to a relaxing state, and back to the relaxed state.

[0111] At step 828, the visualization and personalization module 400 causes display of a second graphical visualization (e.g., graphical element 424 or graphical element 518) based on the number of repetitions. A different portion of the second graphical visualization can light up each time a repetition is counted and / or a color of a portion of the second graphical visualization can indicate a quality of the performance for a corresponding repetition.

[0112] At step 832, the visualization and personalization module 400 causes digital presentation of at least one of the graphical visualizations on the interface. The graphical visualizations are continuously updated by the visualization and personalization module 400 during the state transitions and over the course of the performance of the physical activity.

[0113] Figure 9 depicts a flow diagram of a process for personalizing a metric to reflect how closely a performance of a physical activity by an individual matches a reference performance based on historical performances by the individual, physical abilities and health of the individual, input received from the individual, or observations of the performance.

[0114] At step 904, the visualization and personalization module 400 receives or generates a skeletal frame based on a digital image of an individual completing a performance of a physical activity. For example, the estimating module 216 or the exercise recognition engine 410 can apply a computer-implemented model (or simply ■‘model”) referred to as a pose estimator to the digital image, so as to produce the skeletal frame.

[0115] At step 908, the visualization and personalization module 400 can receive information (e.g., information 434) describing historical performances of the physicalactivity by the individual and / or physical abilities and health of the individual. The information describing the historical performances can indicate how well or poorly the individual performed exercises in the past, past values of the metric 440, or skeletal frames generates from digital images of the individual captured during the historical performances. The information describing the physical abilities and health of the individual can describe injuries the individual has suffered, illnesses the individual has, the individual’s weight, or health indicators, such as blood pressure.

[0116] The visualization and personalization module 400 can receive input 436 that the individual has entered on an interface. For example, the individual can select a difficulty level or a speed for an exercise. The visualization and personalization module 400 can receive observations 432 of an ongoing performance by the individual, for example, digital images, a heart rate of the individual captured by sensors that the individual is wearing, or a speed that the individual is performing a repetition at.

[0117] At step 912, the visualization and personalization module 400 produces a metric (e.g., metric 440) that is indicative of similarity between the skeletal frame and a reference skeletal frame that corresponds to a reference performance of the physical activity against which the performance is being compared. For example, the metric varies continuously to indicate a degree of similarity between the skeletal frames being produced as the individual performs the exercise and the reference skeletal frames. The metric is a measure of the quality' of the performance of the physical activity compared to the reference performance.

[0118] At step 916, the visualization and personalization module 400 manipulates or personalizes the metric 440 based on the information 434, the observations 432, and / or the input 436. For example, the personalization component 430 generates parameters (e.g., adaptation parameters 444) based on the information 434, the observations 432, and / or the input 436, and sends the parameters to the exercise recognition engine 410. The exercise recognition engine 410 receives the parameters from the personalization component 430 and uses the parameters to manipulate the metric 440, for example, prior to sending the metric 440 to the visualization component 420 for display via a graphical element.

[0119] The metric 440 is personalized to reflect how closely the performance by the individual matches the reference performance in view of the historical performances by the individual, the physical abilities and health of the individual, the input 436 received from the individual, and / or the observations 432. In some implementations, the metric 440 is personalized by providing the information 434 and / or the observations 432 to a machine learning model so as to obtain the adaptation parameters 444 or a personalized version of the metric that is produced by the machine learning model as output. For example, the machine learning model includes one or more model parameters that are received from a source external to the computing device. The model parameters can be associated with a generic machine learning model designed and trained to personalize the metric 440 to the individual. For example, the visualization and personalization module 400 can provide digital images depicting the individual completing the performance of the physical activity to a machine learning model, so as to tune the machine learning model to reflect how closely the performance matches the historical performance of the physical activity by the individual.

[0120] The visualization and personalization module 400 (e.g., using the visualization component 420) causes digital presentation of a graphical element (e.g.. graphical element 422) based on the metric 440. An appearance of the graphical element reflects a quality of the performance in view of the historical performances, the physical abilities and health data, the input received from the individual, and / or the observations 432.

[0121] Figure 10 depicts a flow diagram of a process for determining and personalizing a metric that reflects how closely a performance of a physical activity matches a reference performance of the physical activity.

[0122] At step 1016, the visualization and personalization module 400 compares a skeletal frame generated from a digital image of an individual performing a physical activity to a reference skeletal frame. For example, the reference skeletal frame is generated from one or more digital images corresponding to a ’relaxed" pose or state of a physical activity. The reference skeletal frame can be created from digital images of a physiotherapist in a relaxed state. For example, the exercise recognition engine 410 determines a (first) distance between the skeletal frame and the reference skeletal frame based on a relative distance function. The implementation of the distancefunction can be varied based on the type of physical activity. The distance can be calculated, for example, based on spatial positions of anatomical regions 514. 554 of the individual in the skeletal frame and the reference skeletal frame.

[0123] At step 1020, the visualization and personalization module 400 compares the skeletal frame generated from the digital image of the individual to another reference skeletal frame, for example, generated from one or more digital images corresponding to an "‘engaged” pose or state of the physical activity. For example, the exercise recognition engine 410 determines a (second) distance between the skeletal frame and the other reference skeletal frame based on the same or another relative distance function.

[0124] At step 1028, the visualization and personalization module 400 compares the first distance to the second distance to determine a difference between the two distances. The difference between the two distances can be represented by a metric (e.g.. metric 440).

[0125] At step 1032. the visualization and personalization module 400 normalizes the metric based on parameters (sometimes referred to as ‘‘normalization parameters”), for example, from a training data set. For example, the difference between the two distances determined at step 1028 or a metric representing the distance can be normalized using the parameters. The normalization performed at step 1032 is to provide that a metric (e.g., metric 440) close to or at a minimum value (e.g.. 0) is obtained when the individual is positioned at or close to the relaxed state. The normalization at step 1032 further provides that the metric or score is determined to be close to or at a maximum value (e.g., 100) when the individual is positioned at or close to the engaged state. The normalization performed at step 1032 further provides a smooth interpolation for the metric or score from the minimum value to the maximum value as the individual transitions from the relaxed state to the engaged state.

[0126] At step 1040, the visualization and personalization module 400 personalizes the normalized metric based on parameters (e.g., the adaptation parameters 444) generated by, for example, the personalization component 430. The parameters used at step 1040 can be generated based on the input 436. the observations 432, and / or the information 434 as described in more detail with reference to Figure 9.Processing System

[0127] Figure 11 includes a block diagram illustrating an example of a processing system 1100 in which at least some operations described herein can be implemented. For example, components of the processing system 1100 may be hosted on a computing device that includes a visualization and personalization platform (e.g., visualization and personalization module 400 of Figure 4).

[0128] The processing system 1100 can include a processor 1102, main memory 1106, non-volatile memoir 1110, network adapter 1112, video display 1118, input / output devices 1120, control device 1122 (e.g., a keyboard or pointing device such as a computer mouse or trackpad), drive unit 1124 including a storage medium 1126, and signal generation device 1130 that are communicatively connected to a bus 1116. The bus 1116 is illustrated as an abstraction that represents one or more physical buses or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus 1116. therefore, can include a system bus, a Peripheral Component Interconnect ('‘PCI’’) bus or PCI-Express bus, a HyperTransport (“HT”) bus, an Industry' Standard Architecture ("ISA") bus, a Small Computer System Interface (‘‘SCSI”) bus, a Universal Serial Bus (“USB”) data interface, an Inter- Integrated Circuit (“I2C”) bus, or a high-performance serial bus developed in accordance with Institute of Electrical and Electronics Engineers (“IEEE”) 1394.

[0129] While the main memory 1106. non-volatile memory 1110. and storage medium 1126 are shown to be a single medium, the terms “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized / distributed database and / or associated caches and servers) that store one or more sets of instructions 1128. The terms “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the processing system 1100.

[0130] In general, the routines executed to implement the embodiments of the disclosure can be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically comprise one or more instructions (e g., instructions 1104, 1108, 1128) set at various times in various memoryand storage devices in a computing device. When read and executed by the processors 1102. the instruction(s) cause the processing system 1100 to perform operations to execute elements involving the various aspects of the present disclosure.

[0131] Further examples of machine- and computer-readable media include recordable-type media, such as volatile memory devices and non-volatile memory devices 1110, removable disks, hard disk drives, and optical disks (e.g., Compact Disk Read-Only Memory ( CD-ROMs ') and Digital Versatile Disks (“DVDs")), and transmission-type media, such as digital and analog communication links.

[0132] The network adapter 1112 enables the processing system 1100 to mediate data in a network 1114 with an entity7that is external to the processing system 1 100 through any communication protocol supported by the processing system 1100 and the external entity. The network adapter 1112 can include a network adaptor card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, bridge router, a hub. a digital media receiver, a repeater, or any combination thereof.Remarks

[0133] The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated.

[0134] Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments can vary considerably in their implementation details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology withwhich that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology' encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the embodiments.

[0135] The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A method performed by a computer program executing on a computing device, the method comprising: obtaining digital images depicting an individual who has been prompted to perform a physical activity via an interface; for each digital image: generating a respective skeletal frame that specifies spatial positions of anatomical regions of the individual; comparing the respective skeletal frame to a reference skeletal frame of a template associated with the physical activity7, wherein the reference skeletal frame corresponds to an engaged state of the physical activity; determining a metric that is representative of a degree of similarity between the respective skeletal frame and the reference skeletal frame; and causing display of a first graphical visualization based on the metric; determining, from the digital images, that the individual has completed a number of repetitions of the physical activity7, wherein a repetition is defined by transitions from a relaxed state to an engaging state to the engaged state to a relaxing state to the relaxed state; causing display7of a second graphical visualization based on the number of repetitions; and causing digital presentation of the first graphical visualization and the second graphical visualization on the interface.

2. The method of claim 1, wherein the respective skeletal frame is a three- dimensional (3D) skeletal frame, and wherein the spatial positions of the anatomical regions are 3D spatial positions.

3. The method of claim 1, wherein the respective skeletal frame comprises 3D shape information extracted from the each digital image.

4. The method of claim 1, wherein the respective skeletal frame is encoded using 3D coordinates.

5. The method of claim 1, wherein the first and the second graphical visualizations are continuously updated during the transitions.

6. The method of claim I . wherein the first graphical visualization comprises a longitudinal indicator or a circular indicator, and wherein the method further comprises: adjusting a color of a portion of the longitudinal indicator or the circular indicator to indicate a change in the metric.

7. The method of claim 1 , wherein determining that the individual has completed the number of repetitions of the physical activity is performed by a multi-state machine that is programmed to recognize and classify repetitive movement from the relaxed state to the engaging state to the engaged state to the relaxing state to the relaxed state.

8. The method of claim 1, wherein the digital images are generated by a camera included in the computing device.

9. A method performed by a computer program executing on a computing device, the method comprising: obtaining digital images depicting an individual who has been prompted to perform a physical activity via an interface; for each digital image: generating a respective skeletal frame that specifies spatial positions of anatomical regions of the individual; comparing the respective skeletal frame to a reference skeletal frame of a template associated with the physical activity,wherein the reference skeletal frame corresponds to an engaged state of the physical activity; determining a metric that is representative of a degree of similarity between the respective skeletal frame and the reference skeletal frame; and causing display of a graphical visualization based on the metric; and causing digital presentation of the graphical visualization on the interface.

10. The method of claim 9. further comprising: determining, from the digital images, that the individual has completed a number of repetitions of the physical activity, wherein a repetition is defined by transitions from a relaxed state to an engaging state to the engaged state to a relaxing state to the relaxed state; causing display of a second graphical visualization based on the number of repetitions; and causing digital presentation of the second graphical visualization on the interface.

11. A non-transitory medium with instructions stored thereon that, when executed by a processor of a computing device, cause the computing device to perform operations comprising: obtaining digital images, in temporal order, of an individual completing a performance of a physical activity; for each of the digital images, generating a skeletal frame that specifies spatial positions of multiple anatomical regions of the individual; comparing the skeletal frame to a reference skeletal frame, selected from among a series of reference skeletal frames that correspond to a reference performance of the physical activity against which the performance is to be compared, to produce a metric that is indicative of similarity between the skeletal frame and the reference skeletal frame; andmanipulating, based on the metric, an appearance of a graphical element on an interface, such that the appearance varies over the performance to reflect how closely the performance matches the reference performance across the multiple anatomical regions.

12. The non-transitory medium of claim 11, wherein the skeletal frame and the series of reference skeletal frames are encoded using 3D coordinates.

13. The non-transitory medium of claim 11, wherein the appearance of the graphical element on the interface varies continuously over the performance.

14. The non-transitory medium of claim 11, wherein the graphical element comprises a longitudinal indicator or a circular indicator, and wherein a color of a portion of the longitudinal indicator or the circular indicator varies to reflect how closely the performance matches the reference performance.

15. A computing device including:(i) one or more processors; and(ii) a non-transitory, computer-readable storage medium storing instructions that, when executed by the one or more processors of the computing device, cause the computing device to perform operations comprising: receiving a skeletal frame generated from a digital image of an individual completing a performance of a physical activity: receiving information describing at least one of: historical performance of the physical activity by the individual, physical abilities and health of the individual, or input received from the individual on an interface; producing a metric that is indicative of similarity between the skeletal frame and a reference skeletal frame that corresponds to areference performance of the physical activity against which the performance is being compared; and manipulating the metric, based on the information, such that the metric reflects how closely the performance matches the reference performance in view of the at least one of the historical performance by the individual, the physical abilities and health of the individual, or the input received from the individual.

16. The computing device of claim 15, wherein the operations further comprise: causing digital presentation of a graphical element based on the metric, such that an appearance of the graphical element reflects a quality of the performance in view of the at least one of the historical performance by the individual, the physical abilities and health of the individual, or the input received from the individual.

17. The computing device of claim 15, wherein said manipulating the metric comprises: providing the information to a machine learning model so as to obtain a personalized version of the metric that is produced by the machine learning model as output, wherein the machine learning model comprises one or more model parameters that are received from a source external to the computing device and associated with a generic machine learning model designed and trained to personalize the metric to the individual.

18. The computing device of claim 15. further comprising manipulating a threshold performance based on observations of the performance received by the computing device, wherein the metric is compared to the threshold performance to determine how closely the performance matches the reference performance.

19. The computing device of claim 18. further comprising manipulating the threshold performance to reflect a change in the physical abilities and health of the individual over time.

20. The computing device of claim 15. wherein said operations further comprise: obtaining digital images depicting the individual completing the performance of the physical activity; and providing the digital images to a machine learning model, so as to tune the machine learning model to reflect how closely the performance matches the historical performance of the physical activity by the individual.

21. The computing device of claim 15, wherein the metric is further indicative of a speed of the performance.

Images