Predicting muscle activations based on estimated poses

Abstract

Introduced here are computer-implemented platforms (also referred to as "muscle activation platforms") that are able to determine activated muscles of individuals performing physical activities. By predicting activated muscles, the muscle activation platforms are able to more accurately determine whether the individuals are performing the physical activities correctly.

Description

PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001PREDICTING MUSCLE ACTIVATIONS BASED ON ESTIMATED POSESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to US Provisional Application No. 63 / 684,273, titled “Method for Interactive Visualization of Musculoskeletal (MSK) Anatomy for Physiotherapy” and filed on August 16, 2024, and US Provisional Application No. 63 / 746,792, titled “Predicting Muscle Activations Based on Estimated Poses” and filed on January 17, 2025, each of which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] Various embodiments concern computer programs and associated computer-implemented techniques for estimating poses of a living body and providing visualizations of muscle activations.BACKGROUND

[0003] Pose estimation (also called “pose detection”) is an active area of study in the field of computer vision. Over the last several years, tens - if not hundreds - of different approaches have been proposed in an effort to solve the problem of pose detection. Many of these approaches rely on machine learning due to its programmatic approach to learning what constitutes a pose.

[0004] As a field of artificial intelligence, computer vision enables machines to perform image processing tasks with the aim of imitating human vision.Pose estimation is an example of a computer vision task that generally includes detecting, associating, and tracking the movements of a person. Insights into positioning and movement can be drawn from analysis of estimated poses.1182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 includes several examples of muscle activation visualizations.

[0006] Figure 2 illustrates a network environment that includes a muscle activation platform that is executed by a computing device.

[0007] Figure 3 illustrates an example of a computing device that is able to execute a muscle activation platform.

[0008] Figure 4 illustrates a flow for predicting muscle activations using a biomechanical model and a muscle activation inference model.

[0009] Figure 5 illustrates a flow for predicting muscle activations based on estimated poses using a combined biomechanical and muscle activation inference model.

[0010] Figure 6 illustrates a flow for predicting muscle activations based on keypoint locations using a combined biomechanical and muscle activation inference model.

[0011] Figure 7 illustrates a flow of applications of muscle activation predictions.

[0012] Figure 8 illustrates avatars of individuals performing physical activities with indications of muscle activations.

[0013] Figure 9 includes a block diagram illustrating an example of a processing system in which at least some operations described herein can be implemented.

[0014] Various features of the technology described herein will become more apparent to those skilled in the art from a study of the Detailed Description in conjunction with the drawings. Various embodiments are depicted in the drawings for the purpose of illustration. However, those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technology. Accordingly, although specific embodiments are shown in the drawings, the technology is amenable to various modifications.2182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001DETAILED DESCRIPTION

[0015] Over the last several years, significant advances have been made in the field of computer vision. This has resulted in the development of sophisticated pose estimation programs (also called "pose estimators” or "pose predictors”) that are designed to perform human pose estimation in either two dimensions or three dimensions. For example, pose estimators may perform pose estimation by inferring locations on the body. Pose estimators continue to be applied to different contexts, and as such, continue to be used to help solve different problems. One problem for which pose estimators have proven to be particularly useful is monitoring the performance of physical activities. Consider, for example, a scenario where an individual is instructed or prompted to perform a physical activity by a computer program. By applying a pose estimator to digital images of the individual, the computer program can glean insight into the performance of the physical activity. Due to their consistent, programmatic nature, pose estimators allow for monitoring of performance of physical activities.

[0016] Traditional pose estimators may rely on an individual’s skeleton (e.g., joint locations) to draw inferences about poses. However, an understanding of muscle activation can aid the system in gleaning additional insights into the individual’s performance. Muscle activation is an integral component of performance of physical activities and is especially important when the system is responsible for monitoring physical activities that have meaningful real-world impact, such as on the health and wellness of the individual responsible for performing the physical activities. Providing individuals with visualizations or feedback regarding muscle activations may allow the individual to correct a pose or improve their performance of a physical activity. Muscle activation can thereby aid individuals in improving their health and wellness.

[0017] Exercise therapy is an intervention technique that utilizes physical activities as the principal treatment for addressing the symptoms of musculoskeletal (“MSK”) conditions, such as acute physical ailments and chronic physical ailments. Exercise therapy programs (or simply “programs”)3182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 generally involve a plan for performing physical activities during exercise therapy sessions (or simply “sessions”) that occur on a periodic basis. Normally, the purpose of a program is to either restore normal MSK functionality or reduce the pain caused by a physical ailment, which may have been caused by injury or disease.

[0018] Introduced here is an approach for predicting muscle activations based on images or videos of an individual performing a physical activity. The system may use a pose estimator to estimate a pose of the individual based on predicted keypoint locations. The system may use a first machine learning model to generate predicted forces on the keypoints based on the locations of the keypoints. The system may then use a second machine learning model to generate predicted muscle activations based on the forces on the keypoints. In some embodiments, the system may use a single machine learning model trained to generate predicted muscle activations directly from the estimated poses or from the predicted keypoint locations. Once the system receives the predicted muscle activations, the system can present the muscle activations to the individual, for example, by highlighting the muscles activated on a digital representation of the user. Figure 1 includes several examples 100 of muscle activation visualizations. The system may determine whether the correct muscles are being activated based on the physical activity, estimated pose, or other information. The system may even differentiate correct versus incorrect muscle activations using different colors or other indicators to highlight different muscles.

[0019] This approach may improve the effectiveness of physical activities and therapeutic exercises. The visualization of incorrect muscle activations may prompt individuals to make corrections to their form. Incorrect muscle activations may negate the effectiveness of such programs or even harm the individual further, so these corrections may improve the effectiveness of the exercises and reduce the risk of injury due to incorrect form. Another benefit is that this approach may be generic to a large variety of physical activities, though specific parameters and templates can be defined per physical activities. Accordingly, a set of models trained to detect muscle activations for different physical activities may be trained and then released, but additional4182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 models corresponding to new physical activities may be added to the set or existing models corresponding to existing physical activities could be removed from the set.

[0020] Moreover, this approach offers a solution for how to convey relevant information about specific parts of the human body in a way that is both anatomically accurate and not too technical for non-experts. Some specific uses might be for digital or in-person physiotherapy sessions when physiotherapists or coaches need to inform or educate their patients about their injuries and the therapies those patients are receiving to facilitate healing of those injuries. By providing selective but accurate anatomical visualization of the most relevant parts of the body in the specific poses and motions that the patient is performing, the information can be provided in a direct and simple way. It may also provide additional motivation for patients performing exercise therapy as they can see how the specific movements are activating the correct parts of their body in a way that contributes to recovery.

[0021] For the purpose of illustration, embodiments may be described with reference to exercises that are performed during sessions as part of a program. However, a system could be designed to accurately detect muscle activations of other physical activities, such as sporting activities, cooking activities, art activities, and the like. Accordingly, the approach described herein could be used to accurately predict muscle activations and provide feedback regarding performance of nearly any physical activity.

[0022] Moreover, embodiments may be described in the context of computer-executable instructions for the purpose of illustration. However, aspects of the approach could be implemented via hardware or firmware instead of, or in addition to, software. As an example, the system may be embodied as a computer program that offers support for completing exercises during sessions as part of a program, determines which physical activities are appropriate for a user given performance during past sessions, and enables feedback for the user based on the estimated poses and muscle activations of the user.Terminology5182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001

[0023] References in the present disclosure to “an embodiment” or “some embodiments” mean that the feature, function, structure, or characteristic being described is included in at least one embodiment. Occurrences of such phrases do not necessarily refer to the same embodiment, nor are they necessarily referring to alternative embodiments that are mutually exclusive of one another.

[0024] Unless the context clearly requires otherwise, the terms “comprise,” “comprising,” and “comprised of” are to be construed in an inclusive sense rather than an exclusive or exhaustive sense. That is, in the sense of “including but not limited to.” The term “based on” is also to be construed in an inclusive sense. Thus, the term “based on” is intended to mean “based at least in part on.”

[0025] The terms “connected,” “coupled,” and variants thereof are intended to include any connection or coupling between two or more elements, either direct or indirect. The connection or coupling can be physical, logical, or a combination thereof. For example, elements may be electrically or communicatively coupled to one another despite not sharing a physical connection.

[0026] The term “module” may refer broadly to software, firmware, hardware, or combinations thereof. Modules are typically functional components that generate one or more outputs based on one or more inputs. A computer program may include or utilize one or more modules. For example, a computer program may utilize multiple modules that are responsible for completing different tasks, or a computer program may utilize a single module that is responsible for completing all tasks.

[0027] When used in reference to a list of multiple items, the word “or” is intended to cover all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of items in the list.Overview of Muscle Activation Platform

[0028] A muscle activation platform may be responsible for monitoring the motion of an individual (also called a “user,” “patient,” or “participant”) through analysis of digital images that contain her and are captured as she completes6182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 a physical activity. As an example, the muscle activation platform may guide the user through exercise therapy sessions (or simply “sessions”) that are performed as part of an exercise therapy program (or simply “program”) by monitoring pose in an ongoing manner. As part of the program, the user may be requested to engage with the muscle activation platform on a periodic basis. The frequency with which the user is requested to engage with the muscle activation platform may be based on factors such as the anatomical region for which therapy is needed, the MSK condition for which therapy is needed, the difficulty of the program, the age of the user, the amount of progress that has been achieved, and the like. Note that because the motion of the user is generally monitored through the continual analysis of pose, the muscle activation platform could also include a pose estimation platform.

[0029] As the user performs exercises, she may be recorded by a camera of a computing device. Normally, the camera is part of the computing device on which the motion monitoring is executed or accessed. For example, in order to initiate a session, the user may initiate a mobile application that is stored on, and executable by, her mobile phone or tablet computer, and the mobile application may instruct the user to position her mobile phone or tablet computer in such a manner that one of its cameras can record her as exercises are performed. Note that, in some embodiments, the camera is part of another computing device. For example, the camera may be included in a peripheral computing device, such as a web camera (also called a “webcam”), that is connected to the computing device. By examining the digital images that are output by the camera, the muscle activation platform can monitor performance of the exercises by estimating the pose of the user over time.

[0030] As mentioned above, the muscle activation platform could alternatively estimate pose in contexts that are unrelated to healthcare, for example, to improve technique. As an example, the muscle activation platform may estimate the pose of an individual while she completes a sporting activity (e.g., performs a dance move, performs a yoga move, shoots a basketball, throws a baseball, swings a golf club), a cooking activity, an art activity, etc. Accordingly, while embodiments may be described in the context of a user who completes an exercise during a session as part of a program, the7182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 features of those embodiments may be similarly applicable to individuals performing other types of physical activities. Individuals whose performances of physical activities are analyzed may be referred to as “users” of the muscle activation platform , even if these individuals have little to no opportunity to interact with the muscle activation platform.

[0031] Figure 2 illustrates a network environment 200 that includes a muscle activation platform 202 that is executed by a computing device 204. Users can interact with the muscle activation platform 202 via interfaces 206. For example, users may be able to access interfaces that are designed to guide them through physical activities, indicate progress, present feedback, etc. As another example, users may be able to access interfaces through which information regarding completed physical activities can be reviewed, feedback can be provided, etc. Thus, interfaces 206 may serve as informative spaces, or the interfaces 206 may serve as collaborative spaces through which users and coaches can communicate with one another.

[0032] As shown in Figure 2, the muscle activation platform 202 may reside in a network environment 200. Thus, the computing device on which the muscle activation platform 202 is executing may be connected to one or more networks 206A-B. Depending on its nature, the computing device 204 could be connected to a personal area network (“PAN”), local area network (“LAN”), wide area network (“WAN”), metropolitan area network (“MAN”), or cellular network. For example, if the computing device 204 is a mobile phone, then the computing device 204 may be connected to a computer server of a server system 210 via the Internet. As another example, if the computing device 204 is a computer server, then the computing device 204 may be accessible to users via respective computing devices that are connected to the Internet via LANs.

[0033] The interfaces 206 may be accessible via a web browser, desktop application, mobile application, or another form of computer program. For example, to interact with the muscle activation platform 202, a user may initiate a web browser on the computing device 204 and then navigate to a web address associated with the muscle activation platform 202. As another8182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 example, a user may access, via a desktop application or mobile application, interfaces that are generated by the muscle activation platform 202 through which she can select physical activities to complete, review analyses of her performance of the physical activities, and the like. Accordingly, interfaces generated by the muscle activation platform 202 may be accessible via various computing devices, including mobile phones, tablet computers, desktop computers, wearable electronic devices (e.g., watches or fitness accessories), virtual reality systems, augmented reality systems, and the like.

[0034] Generally, the muscle activation platform 202 is hosted, at least partially, on the computing device 204 that is responsible for generating the digital images to be analyzed, as further discussed below. For example, the muscle activation platform 202 may be embodied as a mobile application executing on a mobile phone or tablet computer. In such embodiments, the instructions that, when executed, implement the muscle activation platform 202 may reside largely or entirely on the mobile phone or tablet computer. Note, however, that the mobile application may be able to access a server system 210 on which other aspects of the muscle activation platform 202 are hosted.

[0035] In some embodiments, aspects of the muscle activation platform 202 are executed by a cloud computing service operated by, for example, Amazon Web Services®, Google Cloud Platform™, or Microsoft Azure®. Accordingly, the computing device 204 may be representative of a computer server that is part of a server system 210. Often, the server system 210 comprises multiple computer servers. These computer servers can include information regarding different physical activities; computer-implemented models (or simply “models”) that indicate how anatomical regions should move when a given physical activity is performed; computer-implemented templates (or simply “templates”) that indicate how anatomical regions should be positioned when partially or fully engaged in a given physical activity; algorithms for processing image data from which spatial position of anatomical regions can be computed, inferred, or otherwise determined; user data such as name, age, weight, ailment, enrolled program, duration of enrollment, and number of physical activities completed; and other assets.9182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001

[0036] Figure 3 illustrates an example of a computing device 300 that is able to execute a muscle activation platform 312. As mentioned above, the muscle activation platform 312 can facilitate the performance of physical activities by a user, for example, by providing instruction or encouragement. As shown in Figure 3, the computing device 300 can include a processor 302, memory 304, display mechanism 306, communication module 308, image sensor 310A, audio output mechanism 322, and audio input mechanism 324. Each of these components is discussed in greater detail below.

[0037] Those skilled in the art will recognize that different combinations of these components may be present depending on the nature of the computing device 300. For example, if the computing device 300 is a computer server that is part of a server system (e.g., server system 210 of Figure 2), then the computing device 300 may not include the display mechanism 306, image sensor 310A, audio output mechanism 322, or audio input mechanism 324, though the computing device 300 may be communicatively connectable to another computing device that does include a display mechanism, an image sensor, an audio output mechanism, or an audio input mechanism.

[0038] The processor 302 can have generic characteristics similar to general-purpose processors, or the processor 302 may be an applicationspecific integrated circuit (“ASIC”) that provides control functions to the computing device 300. As shown in Figure 3, the processor 302 can be coupled to all components of the computing device 300, either directly or indirectly, for communication purposes.

[0039] The memory 304 may be comprised of any suitable type of storage medium, such as static random-access memory (“SRAM”), dynamic randomaccess memory (“DRAM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory, or registers. In addition to storing instructions that can be executed by the processor 302, the memory 304 can also store data generated by the processor 302 (e.g., when executing the modules of the muscle activation platform 312) and produced, retrieved, or obtained by the other components of the computing device 300. For example, data received by the communication module 308 from a source external to the10182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 computing device 300 (e.g., image sensor 31 OB) may be stored in the memory 304, or data produced by the image sensor 31 OA may be stored in the memory 304. Note that the memory 304 is merely an abstract representation of a storage environment. The memory 304 could be comprised of actual integrated circuits (also referred to as “chips”).

[0040] The display mechanism 306 can be any mechanism that is operable to visually convey information to a user. For example, the display mechanism 306 may be a panel that includes light-emitting diodes (“LEDs”), organic LEDs, liquid crystal elements, or electrophoretic elements. In some embodiments, the display mechanism 306 is touch sensitive. Thus, a user may be able to provide input to the muscle activation platform 312 by interacting with the display mechanism 306. Alternatively, the user may be able to provide input to the muscle activation platform 312 through some other control mechanism.

[0041] The communication module 308 may be responsible for managing communications external to the computing device 300. For example, the communication module 308 may be responsible for managing communications with other computing devices (e.g., server system 210 of Figure 2, or a camera peripheral such as video camera or webcam). The communication module 308 may be wireless communication circuitry that is designed to establish communication channels with other computing devices. Examples of wireless communication circuitry include 2.4 gigahertz (“GHz”) and 5 GHz chipsets compatible with Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 - also referred to as “Wi-Fi chipsets.” Alternatively, the communication module 308 may be representative of a chipset configured for Bluetooth®, Near Field Communication (“NFC”), and the like. Some computing devices - like mobile phones and tablet computers - are able to wirelessly communicate via separate channels. Accordingly, the communication module 308 may be one of multiple communication modules implemented in the computing device 300. As an example, the communication module 308 may initiate and then maintain one communication channel with a camera peripheral (e.g., via Bluetooth), and the communication module 30811182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 may initiate and then maintain another communication channel with a server system (e.g., via the Internet).

[0042] The nature, number, and type of communication channels established by the computing device 300 - and more specifically, the communication module 308 - may depend on the sources from which data is received by the muscle activation platform 312 and the destinations to which data is transmitted by the muscle activation platform 312. Assume, for example, that the computing device 300 is representative of a mobile phone or tablet computer that is associated with (e.g., owned by) a user. In some embodiments the communication module 308 may only externally communicate with a computer server, while in other embodiments the communication module 308 may also externally communicate with a source from which to receive image data. The source could be another computing device (e.g., a mobile phone or camera peripheral that includes an image sensor 31 OB) to which the mobile device is communicatively connected. Image data could be received from the source even if the mobile phone generates its own image data. Thus, image data could be acquired from multiple sources, and these image data may correspond to different perspectives of the user performing a physical activity. Regardless of the number of sources, image data - or analyses of the image data - may be transmitted to the computer server for storage in a digital profile that is associated with the user. The same may be true if the muscle activation platform 312 only acquires image data generated by the image sensor 31 OA. The image data may initially be analyzed by the muscle activation platform 312, and then the image data - or analyses of the image data - may be transmitted to the computer server for storage in the digital profile.

[0043] The image sensor 31 OA may be any electronic sensor that is able to detect and convey information in order to generate images, generally in the form of image data (also called “pixel data”). Examples of image sensors include charge-coupled device (“CCD”) sensors and complementary metal- oxide semiconductor (“CMOS”) sensors. The image sensor 310A may be part of a camera module (or simply “camera”) that is implemented in the computing device 300. In some embodiments, the image sensor 310A is one12182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 of multiple image sensors implemented in the computing device 300. For example, the image sensor 31 OA could be included in a front- or rear-facing camera on a mobile phone. Alternatively, the image sensor 31 OA may be externally connected to the computing device 300 such that the image sensor 31 OA captures image data of an environment and sends the image data to the muscle activation platform 312.

[0044] For convenience, the muscle activation platform 312 may be referred to as a computer program that resides in the memory 304. However, the muscle activation platform 312 could be comprised of hardware or firmware in addition to, or instead of, software. In accordance with embodiments described herein, the muscle activation platform 312 may include a processing module 314, pose estimating module 316, analysis module 318, and graphical user interface (“GUI”) module 320. These modules can be an integral part of the muscle activation platform 312. Alternatively, these modules can be logically separate from the muscle activation platform 312 but operate “alongside” it. Together, these modules may enable the muscle activation platform 312 to programmatically monitor motion of users during the performance of physical activities, such as exercises, through analysis of digital images generated by the image sensor.

[0045] The processing module 314 can process image data obtained from the image sensor 310A over the course of a session. The image data may be used to infer a spatial position or orientation of one or more anatomical regions as further discussed below. The image data may be representative of a series of digital images. These digital images may be discretely captured by the image sensor 31 OA over time, such that each digital image captures the user at different stages of performing a physical activity. In some embodiments, these digital images may be representative of frames of a video that is captured by the image sensor. In such embodiments, the image data could also be called “video data.”

[0046] The image data may be used to infer a spatial position of one or more anatomical regions as further discussed below. For example, the processing module 314 may perform operations (e.g., filtering noise, changing13182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 contrast, reducing size) to ensure that the data can be handled by the other modules of the muscle activation platform 312. As another example, the processing module 314 may temporally align the data with data obtained from another source (e.g., another image sensor) if multiple data are to be used to establish the spatial position of the anatomical regions of interest.

[0047] Moreover, the processing module 314 may be responsible for processing information input by users through interfaces generated by the GUI module 320. For example, the GUI module 320 may be configured to generate a series of interfaces that are presented in succession to a user as she completes physical activities as part of a session. On some or all of these interfaces, the user may be prompted to provide input. For example, the user may be requested to indicate (e.g., via a verbal command or tactile command provided via, for example, the display mechanism 306) that she is ready to proceed with the next physical activity, that she completed the last physical activity, that she would like to temporarily pause the session, etc. These inputs can be examined by the processing module 314 before information indicative of these inputs is forwarded to another module.

[0048] The pose estimating module 316 (or simply “estimating module”) may be responsible for estimating the pose of the user through analysis of image data, in accordance with the approach further discussed below. Specifically, the pose estimating module 316 can create, based on a digital image (e.g., generated by the image sensor 31 OA or image sensor 31 OB), a skeletal frame that specifies a spatial position of each of multiple anatomical regions. For example, the pose estimating module 316 can apply a computer- implemented model (or simply “model”) called a pose estimator to the digital image, so as to produce the skeletal frame. In some embodiments the pose estimator is designed and trained to identify a predetermined number and / or type of anatomical regions (e.g., left and right wrist, left and right elbow, left and right shoulder, left and right hip, left and right knee, left and right ankle, or any combination thereof), while in other embodiments the pose estimator is designed and trained to identify all anatomical regions of a certain type (e.g., all keypoints) that are visible in the digital image provided as input. As discussed in greater detail below, the pose estimator may estimate poses14182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 based on keypoint positions. The pose estimator could be a neural network that when applied to the digital image, analyzes the pixels to independently identify digital features that are representative of each anatomical region of interest.

[0049] The analysis module 318 may be responsible for establishing the locations of anatomical regions of interest based on the outputs produced by the pose estimating module 316. Referring again to the aforementioned examples, the analysis module 318 could establish the locations of keypoints based on an analysis of the skeletal frame. Moreover, the analysis module 318 may be responsible for determining appropriate feedback for the user based on the outputs produced by the pose estimating module 316, in accordance with the approach further discussed below. Specifically, the analysis module 318 may determine an appropriate personalized recommendation for the user based on her current position, and a determination as to how her current position compares to a template that is associated with the physical activity that she has been instructed to perform.

[0050] Other modules could also be included in some embodiments. For example, the muscle activation platform 312 may include a training module (not shown) that is responsible for training the pose estimator that is employed by the pose estimating module 316. As another example, the muscle activation platform 312 may include a template generating module (not shown) that is responsible for generating templates that are used by the analysis module 318 to determine which recommendations, if any, are appropriate for a user given her current position.

[0051] Similarly, other components could be implemented in, or accessible to, the computing device 300 in some embodiments. For example, some embodiments of the computing device 300 include an audio output mechanism 322 and / or an audio input mechanism 324. The audio output mechanism 322 may be any apparatus that is able to convert electrical impulses into sound. One example of an audio output mechanism is a loudspeaker (or simply “speaker”). Meanwhile, the audio input mechanism 324 may be any apparatus that is able to convert sound into electrical15182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 impulses. One example of an audio input mechanism is a microphone. Together, the audio output and input mechanisms 322 and 324 may enable feedback, such as personalized recommendation as further discussed below, to be audibly provided to the user. Assume, for example, that the user has been instructed to perform a physical activity while being recorded by the image sensor 31 OA. In such a scenario, the user may be audibly encouraged - in a personalized manner - via the audio output mechanism 322.Predicting Muscle Activations

[0052] Earlier muscle activation platforms may have limited functionality to provide a user with visualizations of muscle activations in real time. For example, traditional muscle activation platforms may provide visualization of basic skeletal structures as well as certain related information, such as forces and velocities associated with movement of an individual. However, these systems may be unable to infer biomechanics of anatomical elements as a way to convey information about how the human body is functioning. Consequently, such approaches lack meaningful information about which muscles an individual is activating while performing physical activities. Moreover, such systems traditionally require a special studio or advanced equipment in order to estimate poses. Introduced here is an approach for using machine learning models to predict muscle activations based on an image of an individual performing a physical activity.

[0053] Figure 4 illustrates a flow 400 for predicting muscle activations using a biomechanical model and a muscle activation inference model. First, a muscle activation platform may receive a digital image (e.g., from camera(s) 401 ). The digital image may depict an individual who is performing a physical activity. In some embodiments, the digital image may relate to a context for which the muscle activation platform is used. For example, the digital images may relate to a particular physical activity (e.g., yoga) or other activity (e.g., cooking).

[0054] In some embodiments, the camera (e.g., camera(s) 401 ) may include a computer vision module. The computer vision module may take as input frames from a camera and infer the poses or positions or rotations of16182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 key markers on individuals in the frame in 3D such that a 3D pose may be derived or represented by some reasonably expressive set of keypoints on the body. The reference frame for the 3D positions may be a root keypoint on the body (e.g. pelvis or neck), the position of the camera, or some other fixed position inferred in world space, independent of the individual in frame. In all cases, it is important that relative positions and orientations of keypoints on the body may be calculated. Similarly, the coordinates of keypoints may be represented as positions relative to the reference frame (e.g. root keypoint), relative to a parent keypoint in a reasonable hierarchy of limbs and digits, or some other coordinate system such that relationships between keypoints in space and time may be derived. The set of keypoints may have correspondence with every anatomical keypoint on the body (e.g. connecting all 206 anatomical bones) or some simplified model of the body representing key limbs for movement (e.g. 3 keypoints per arm, 4 keypoints per leg and 5 keypoints along the spine). The computer vision module may infer other aspects of persons in frame such as bounding boxes, 2D positions or masks, 2D or 3D shape features, tracking IDs, etc. The computer vision module may infer information about persons in frame in real-time and this processing may occur with some latency (e.g. 1 or more frames of delay). The system may run live on streaming video from the camera or on a pre-captured video stored in memory.

[0055] In some embodiments, the muscle activation platform may use a pose estimation system (e.g., 3D pose estimation system 403) to determine predicted locations of keypoints of the individual. For example, the muscle activation platform may use a model that has been trained on the relationships between keypoints in the body, especially between pairs of keypoints corresponding to adjacent body keypoints (e.g., elbow and wrist). Such keypoints are rigidly connected and tend to move jointly with a specific distance between them. The model may predict the locations of the keypoints of the individual based on the digital image obtained. In some embodiments, the muscle activation platform may determine the muscle activations based on the locations of the keypoints.17182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001

[0056] The muscle activation platform may use a first model to predict the forces on the keypoints. For example, the first model may predict the forces on the keypoints based on the estimated pose. The muscle activation platform may apply, to the predicted locations of the keypoints, a first machine learning model that is trained to predict forces on keypoints based on locations of the keypoints. In some embodiments, the first machine learning model may produce, as output, predicted forces on the keypoints.

[0057] For example, the first model may be a biomechanical model (e.g., biomechanical model 405). The biomechanical model may take the locations of keypoints or an estimated pose and may infer torques or forces on keypoints. In some embodiments, the biomechanical model may produce other physically derived quantities (e.g. stress, strain) as outputs. The biomechanical model may simulate the body as a network of rigid bodies with masses / moments of inertia connected by keypoints that may be constrained with some known or estimated dynamic properties. In some embodiments, the biomechanical model may be represented in some other way, such as a parametric function or learned machine learning model built to infer dynamics of the human body. The model may be updated by inferring forces necessary to match the keypoint locations. In some embodiments, the biomechanical model may estimate mass distributions or moment of inertia tensors based on outputs from the pose estimator. The model may include environmental constraints such as a ground plane or other surfaces. In some embodiments, the muscle activation platform may update the biomechanical model using a classical control theory method (e.g. proportional-integral-derivative (PID) controller), a machine learning based approach, or some other method for updating a dynamic model given targets and constraints.

[0058] In some embodiments, the muscle activation platform may use a second model to predict muscle activations. For example, the muscle activation platform may apply, to the predicted locations of the keypoints and the predicted forces on the keypoints, a second machine learning model that is trained to predict muscle activations based on forces on keypoints. In some embodiments, the second machine learning model may produce, as output, predicted muscle activations.18182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001

[0059] For example, the second model may be a muscle activation inferencing model (e.g., muscle activation inference 407). In some embodiments, the muscle activation inferencing model takes, as input, information about the pose as well as additional information such as movement or dynamics of an individual. The muscle activation inferencing model infers the activation of specific muscles as output. Activation of muscles may be binary (e.g. active vs inactive) or scalar (e.g. as a force enacted by the muscle). The muscle activation inference model may infer activations of all anatomical muscles in the body or on some reasonable subset of large or important keypoints (e.g. task-specific). The muscle activation inference model may consider relative movement of keypoints (e.g. flexion, rotation) and their velocities.

[0060] In some embodiments, the second model may infer muscle activation heuristically (e.g. based on anatomical locations of muscles, keypoints, tendons and the required sum force to cause inferred torques or keypoint accelerations). In the case of heuristic inference of muscle activations given keypoint positions and velocities, a rule based approach may be employed where muscles known to be involved in flexion of each keypoint are activated as flexion is observed and muscles known to be involved in extension are activated as the respective keypoints are extended.

[0061] Alternatively, computation of activation may be based on some other methodology that can reasonably infer muscle activations given the aforementioned inputs. For example, computation of muscle activation may involve the use of electromyography (EMG), which directly measures the electrical impulses generated by muscle fibers during contraction. By placing surface electrodes on the skin or using needle electrodes, EMG may capture real-time data on the intensity and timing of muscle activation. In addition to EMG, biomechanical modeling provides another method to predict muscle activation by analyzing the forces, movements, and keypoint angles involved in physical activity. These models use data from motion capture systems, force plates, and other sensors to estimate how muscles must be activated to achieve specific movements. Devices such as smart garments embedded with EMG sensors, inertial measurement units (IMUs), or force sensors may19182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 enable real-time tracking and prediction of muscle activity during dynamic movements. Finally, machine learning models may learn from complex and non-linear relationships between inputs such as keypoint angles, forces, and EMG signals, allowing for more accurate predictions.

[0062] In some embodiments, the muscle activation platform may cause digital presentation of the estimated pose of the individual, with indications of the predicted muscle activations relative to the estimated pose. For example, the muscle activation platform may present an avatar of the individual. Activated muscles can then be rendered and visualized on the avatar of the individual to convey correct or incorrect muscle activation for the physical activity. In some embodiments, the muscle activation platform may identify (e.g. via text / verbal or other targeted visual cues) key muscle activations for a given activity. As an example, the muscle activation platform may provide inputs to an analysis module (e.g., analysis module 318, as shown in FIG. 3) in order to identify over-exerted or under-utilized muscles for the prescribed exercise and share these insights with the user (e.g., visually, verbally or otherwise). The skeletal proportions and body shape (e.g., muscle and fat mass) may also be inferred via computer vision to establish a biomechanical digital twin (e.g., avatar) that can be employed to create a digital “X-ray” mirror, revealing muscle activation under the skin. For example, Figure 8 illustrates avatars 801 and 805 of individuals 803 and 807, respectively, performing physical activities. As shown in avatar 801 and avatar 805, the activated muscles are indicated.

[0063] In some embodiments, the biomechanical model and muscle activation inferencing model are the same model, in which case muscle activations may be inferred directly from the estimated pose. Figure 5 illustrates a flow 500 for predicting muscle activations based on estimated poses using a combined biomechanical and muscle activation inference model.

[0064] The muscle activation platform may receive a digital image (e.g., from camera(s) 501 ). The digital image may depict an individual who is performing a physical activity. The muscle activation platform may use the20182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 pose estimation system (e.g., 3D pose estimation system 503) to determine an estimated pose of the individual. For example, the muscle activation platform may include a model trained to predict estimated poses by predicting a location for each keypoint, so as to predict a plurality of predicted locations. The muscle activation platform may provide the digital image to the machine learning model as input. For example, the machine learning model may be trained to estimate poses based on locations of keypoints predicted for the digital image input. In some embodiments, inputting the digital image may cause the machine learning model to generate an estimated pose of the person based on an analysis of locations of the keypoints. In some embodiments, the muscle activation platform may determine the muscle activations based on the estimated pose.

[0065] The muscle activation platform may apply, to the estimated pose, a machine learning model that is trained to predict muscle activation based on an analysis of pose, so as to generate a plurality of predicted muscle activations. For example, the model may include both the functions of the biomechanical model and the muscle activation inferencing model. In some embodiments, the model may be trained to predict muscle activation directly based on the estimated pose. In some embodiments, the model may otherwise infer muscle activations based on the estimated pose. The muscle activation platform may process the estimated post or the plurality of predicted muscle activations of the individual to determine a first subset of correct muscle activations and a second subset of incorrect muscle activations of the plurality of predicted muscle activations based on the estimated pose. For example, the muscle activation platform may determine correct and incorrect muscle activations for a given pose using a database of muscle activations associated with various poses. In some embodiments, the muscle activation platform may determine correct and incorrect muscle activations for a given pose using a machine learning model trained to predict correct and incorrect muscle activations for various poses. In some embodiments, the muscle activation platform may determine the correct and incorrect muscle activations using other methods.21182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001

[0066] The muscle activation platform may then transmit an indication of the predicted muscle activations to the individual. In some embodiments, the muscle activation platform may then transmit an indication of correct and incorrect muscle activations to the individual. The muscle activation platform may cause digital presentation of the estimated pose of the individual (e.g., as shown in Figure 8). In some embodiments, the digital presentation may indicate a first subset of correct muscle activations and a second subset of incorrect muscle activations of the plurality of predicted muscle activations based on the estimated pose. In some embodiments, the first subset of correct activations may be presented in green while the second subset of incorrect activations may be presented in red. In some embodiments, the digital presentation may convey additional information, such as flexion, extension, or rotation, using different colors or other visual indicators.

[0067] In some embodiments, muscle activations may be inferred directly from the keypoint locations. Figure 6 illustrates a flow 600 for predicting muscle activations based on keypoint locations using a combined biomechanical and muscle activation inference model. The muscle activation platform may receive a digital image (e.g., from camera(s) 601 ). The digital image may depict an individual who is performing a physical activity. The muscle activation platform may use a computer vision module (e.g., keypoint locator 603) to determine locations of the keypoints based on the digital image. The muscle activation platform may input, into a machine learning model, the predicted locations of the keypoints to cause the machine learning model to generate predicted muscle activations. For example, the machine learning model may be trained to predict muscle activations directly based on locations of keypoints. The muscle activation platform may then cause digital presentation of indications of the predicted muscle activations relative to the predicted locations of the keypoints of the individual (e.g., as shown in Figure 8).

[0068] Figure 7 illustrates a flow 700 of applications of muscle activation predictions. For example, the model 701 may be the second model (e.g., the muscle activation inference model 407, as shown in Figure 7) or a combined biomechanical and muscle activation inferencing model (e.g., model 505, as22182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 shown in Figure 5, or model 605, as shown in Figure 6). The output of the model 701 may be predicted muscle activations of the individual depicted in the digital image. The muscle activation platform may use the output of the model 701 to cause digital presentation of the muscle activations, as discussed above (e.g., visualization 703). In some embodiments, the muscle activation platform may use the output of the model 701 to estimate force and energy required to cause the muscle activations (e.g., force and energy estimation 705). In some embodiments, the muscle activation platform may use the output of the model 701 to analyze stress, strain, or torque caused by the activated muscles (e.g., stress / strain / torque analysis 707). In some embodiments, the muscle activation platform may use the output of the model 701 to determine strength and function of the activation muscles (e.g., strength and function analysis 709). In some embodiments, the muscle activation platform may use the output of the model 701 for other applications.Processing System

[0069] Figure 9 includes a block diagram illustrating an example of a processing system 900 in which at least some operations described herein can be implemented. For example, components of the processing system 900 may be hosted on a computing device that includes a muscle activation platform (e.g., muscle activation platform 202 of Figure 2 or muscle activation platform 312 of Figure 3).

[0070] The processing system 900 can include a processor 902, main memory 906, non-volatile memory 910, network adapter 912, video display 918, input / output devices 920, control device 922 (e.g., a keyboard or pointing device such as a computer mouse or trackpad), drive unit 924 including a storage medium 926, and signal generation device 930 that are communicatively connected to a bus 916. The bus 916 is illustrated as an abstraction that represents one or more physical buses or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus 916, therefore, can include a system bus, a Peripheral Component Interconnect (“PCI”) bus or PCI-Express bus, a HyperTransport (“HT”) bus, an Industry Standard Architecture (“ISA”) bus, a Small Computer23182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001System Interface (“SCSI”) bus, a Universal Serial Bus (“USB”) data interface, an Inter-Integrated Circuit (“l2C”) bus, or a high-performance serial bus developed in accordance with Institute of Electrical and Electronics Engineers (“IEEE”) 1394.

[0071] While the main memory 906, non-volatile memory 910, and storage medium 926 are shown to be a single medium, the terms “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized / distributed database and / or associated caches and servers) that store one or more sets of instructions 928. The terms “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the processing system 900.

[0072] In general, the routines executed to implement the embodiments of the disclosure can be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically comprise one or more instructions (e.g., instructions 904, 908, 928) set at various times in various memory and storage devices in a computing device. When read and executed by the processor 902, the instruction(s) cause the processing system 900 to perform operations to execute elements involving the various aspects of the present disclosure.

[0073] Further examples of machine- and computer-readable media include recordable-type media, such as volatile memory devices and nonvolatile memory devices 910, removable disks, hard disk drives, and optical disks (e.g., Compact Disk Read-Only Memory (“CD-ROMs”) and Digital Versatile Disks (“DVDs”)), and transmission-type media, such as digital and analog communication links.

[0074] The network adapter 912 enables the processing system 900 to mediate data in a network 914 with an entity that is external to the processing system 900 through any communication protocol supported by the processing system 900 and the external entity. The network adapter 912 can include a network adaptor card, a wireless network interface card, a router, an access24182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, bridge router, a hub, a digital media receiver, a repeater, or any combination thereof.Remarks

[0075] The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated.

[0076] Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments can vary considerably in their implementation details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the embodiments.

[0077] The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description, but rather by any claims that issue on an application based25182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.26182561480.1

Claims

PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001CLAIMSWhat is claimed is:1 . A method performed by a computer program executing on a computing device, the method comprising: receiving a digital image of an individual who has been prompted to perform a physical activity; determining an estimated pose of the individual by predicting a location for each keypoint of a plurality of keypoints, so as to predict a plurality of predicted locations; applying, to the plurality of predicted locations of the plurality of keypoints, a first machine learning model that is trained to predict forces on keypoints based on locations of the keypoints, wherein the first machine learning model produces, as output, a plurality of predicted forces on the plurality of keypoints; applying, to the plurality of predicted locations of the plurality of keypoints and the plurality of predicted forces on the plurality of keypoints, a second machine learning model that is trained to predict muscle activations based on forces on keypoints, wherein the second machine learning model produces, as output, a plurality of predicted muscle activations; and causing digital presentation of the estimated pose of the individual, with indications of the plurality of predicted muscle activations relative to the estimated pose.

2. The method of claim 1 , wherein the indications include a first subset of correct muscle activations and a second subset of incorrect muscle activations of the plurality of predicted muscle activations based on the estimated pose.

3. The method of claim 2, wherein the first subset of correct muscle activations is presented in a first color and the second subset of incorrect muscle activations is presented in a second color.27182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W0014. The method of claim 1 , wherein the first machine learning model is a biomechanical model and the second machine learning model is a muscle activation inferencing model.

5. The method of claim 1 , wherein predicting the location for each keypoint of the plurality of keypoints involves inputting the digital image into a computer vision module, so as to cause the computer vision module to infer the location for each keypoint of the plurality of keypoints.

6. The method of claim 1 , wherein determining the estimated pose of the individual involves inputting the location for each keypoint of the plurality of keypoints into a pose estimation model, so as to cause the pose estimation model to output the estimated pose.

7. The method of claim 1 , further comprising: identifying appropriate feedback for the individual based on the plurality of predicted muscle activations; and causing digital presentation of the appropriate feedback via a device associated with the individual.

8. A system for predicting muscle activations based on keypoint locations, the system comprising: one or more processors communicatively coupled to a storage device, wherein the one or more processors execute instructions that are stored in the storage device to cause the system to: obtain a digital image of an individual who has been prompted to perform a physical activity; determine an estimated pose of the individual by predicting a location for each keypoint of a plurality of keypoints; applying, to the estimated pose, a machine learning model that is trained to predict muscle activation based on an analysis of pose, so as to generate a plurality of predicted muscle activations; and28182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 processing the estimated pose and the plurality of predicted muscle activations to determine a first subset of correct muscle activations and a second subset of incorrect muscle activations of the plurality of predicted muscle activations based on the estimated pose.

9. The system of claim 8, wherein the first subset of correct muscle activations is presented in a first color and the second subset of incorrect muscle activations is presented in a second color.

10. The system of claim 8, wherein the machine learning model includes a biomechanical model and a muscle activation inferencing model.1 1 . The system of claim 8, wherein predicting the location for each keypoint of the plurality of keypoints involves inputting the digital image into a computer vision module, so as to cause the computer vision module to infer the location for each keypoint of the plurality of keypoints.

12. The system of claim 8, wherein determining the estimated pose of the individual involves inputting the location for each keypoint of the plurality of keypoints into a pose estimation model, so as to cause the pose estimation model to output the estimated pose.

13. The system of claim 8, wherein the instructions further cause the system to: identify appropriate feedback for the individual based on the plurality of predicted muscle activations; and cause digital presentation of the appropriate feedback via a device associated with the individual.

14. The system of claim 8, wherein the digital image is generated by a camera directed at the individual.29182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W00115. One or more non-transitory media with instructions stored thereon that, when executed by a processor of a computing device, cause the computing device to perform operations comprising: receiving a digital image of an individual who has been prompted to perform a physical activity; determining a plurality of predicted locations of a plurality of keypoints of the individual; inputting, into a machine learning model, the plurality of predicted locations of the plurality of keypoints to cause the machine learning model to generate a plurality of predicted muscle activations, wherein the machine learning model is trained to predict muscle activations based on locations of keypoints; and transmitting, to the individual, an indication of the plurality of predicted muscle activations relative to the plurality of predicted locations of the plurality of keypoints of the individual.

16. The one or more non-transitory media of claim 15, wherein the indications include a first subset of correct muscle activations and a second subset of incorrect muscle activations of the plurality of predicted muscle activations based on the plurality of predicted locations.

17. The one or more non-transitory media of claim 16, wherein the first subset of correct muscle activations is presented in a first color and the second subset of incorrect muscle activations is presented in a second color.

18. The one or more non-transitory media of claim 15, wherein the machine learning model includes a biomechanical model and a muscle activation inferencing model.

19. The one or more non-transitory media of claim 15, wherein determining the plurality of predicted locations of the plurality of keypoints of the individual involves inputting the digital image into a computer vision module, so as to30182561480.1PCT / US25 / 36191 02 July 2025 (02.07.2025)Attorney Docket No. 125847.8044.W001 cause the computer vision module to infer the plurality of predicted locations of the plurality of keypoints.

20. The one or more non-transitory media of claim 15, wherein the instructions cause the computing device to perform further operations comprising: identifying appropriate feedback for the individual based on the plurality of predicted muscle activations; and causing digital presentation of the appropriate feedback via a device associated with the individual.31182561480.1

Images