System, method, and program

Abstract

Exercise instruction systems and methods of use are provided.SOLUTION: A system for exercise instruction comprises a non-transitory computer readable medium configured to store instructions. The system further comprises a processor coupled to the non-transitory computer readable medium. The processor is configured to execute instructions to receive input data from a user, the input data including a plurality of images of the user performing an exercise, and extract pose data from the input data. The processor is configured to execute the instructions to determine whether a difference between the extracted posture data and a reference is less than a predetermined threshold, and in response to the difference being less than the predetermined threshold, determine muscle elongation information from the input data. The processor is configured to execute the instructions to determine feedback to provide to the user to increase a degree of muscle elongation during a subsequent performance of the exercise relative to the determined muscle elongation information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to exercise instruction systems and methods of use. [Background technology]

[0002] Physical therapy is difficult to receive for a variety of reasons. In an effort to provide remote services to assist with physical therapy, some approaches utilize video to assist users in performing exercises. In some approaches, users record videos of the exercises they perform and upload the videos for expert review. Users can receive feedback regarding the positioning of body parts to determine whether their posture is appropriate. In some cases, the feedback includes text or visual depictions of how to adjust their posture. Summary of the Invention [Problem to be solved by the invention]

[0003] Techniques that help account for muscle elongation are desirable. [Means for solving the problem]

[0004] One aspect of the present specification is a system for providing exercise instruction, comprising: a non-transitory computer-readable medium configured to store instructions; and a processor connected to the non-transitory computer-readable medium, the processor configured to execute the instructions to receive input data from a user, the input data including a plurality of images of the user performing an exercise; extract posture data from the input data; determine whether a difference between the extracted posture data and a reference is less than a predetermined threshold; determine muscle stretch information from the input data in response to the difference being less than the predetermined threshold; and determine feedback to provide to the user based on the determined muscle stretch information to increase a degree of muscle stretch during subsequent performance of the exercise.

[0005] One aspect of the present specification is a method for providing exercise instruction, comprising: receiving input data from a user, the input data including a plurality of images of the user performing an exercise; extracting posture data from the input data; determining whether a difference between the extracted posture data and a reference is less than a predetermined threshold; determining muscle elongation information from the input data in response to the difference being less than the predetermined threshold; and determining feedback to provide to the user based on the determined muscle elongation information to increase the degree of muscle elongation during subsequent performance of the exercise.

[0006] One aspect of the present specification is a program for causing a processor to perform operations including receiving input data from a user, the input data including a plurality of images of the user performing an exercise; extracting posture data from the input data; determining whether a difference between the extracted posture data and a reference is less than a predetermined threshold; determining muscle stretch information from the input data in response to the difference being less than the predetermined threshold; and determining feedback to provide to the user based on the determined muscle stretch information to increase the degree of muscle stretch during subsequent performance of the exercise. [Brief explanation of the drawings]

[0007] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be noted that, according to standard industry practice, various features have not been drawn to scale. In fact, the dimensions of various features may be arbitrarily increased or decreased for clarity of illustration. [Figure 1] 1 is a flowchart of a method for providing exercise instruction, according to some embodiments. [Figure 2] FIG. 1 is a diagram of a user interface (UI) for a system for providing exercise instruction, according to some embodiments. [Figure 3] 1 is a schematic diagram of a system for providing exercise instruction, according to some embodiments. [Figure 4] 1 is a flowchart of a method of muscle stretch prediction for determining muscle stretch information, according to some embodiments. [Figure 5] FIG. 1 is a diagram of a solution space for providing exercise coaching recommendations, according to some embodiments. [Figure 6] FIG. 1 is a diagram of a system for providing exercise instruction, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0008] The following disclosure provides many different embodiments or examples for implementing various features of the provided subject matter. Below, specific examples of components, values, operations, materials, arrangements, etc. are described to simplify the disclosure. It should be understood that these are merely examples and are not intended to be limiting. Other components, values, operations, arrangements, etc. are also contemplated. For example, the formation of a first feature above or on a second feature in the following description may include embodiments in which the first and second features are formed in direct contact with each other, and may also include embodiments in which an additional feature may be formed between the first and second features such that the first and second features are not in direct contact with each other. Additionally, the disclosure may repeat reference numerals and / or letters in various examples. This repetition is for purposes of simplicity and clarity and does not, in itself, dictate a relationship between the various embodiments and / or configurations being described.

[0009] It is becoming increasingly difficult to find an available physical therapist. As a result, receiving physical therapy (PT) is becoming increasingly difficult for many people. To help people receive PT, several techniques use at-home exercises to improve physical function. Some techniques use cameras to capture video of a person performing the exercises, and the video is analyzed to determine whether the person's body parts are positioned properly during the exercise. While this type of technique is useful, monitoring only the positioning of a person's body parts, also known as posture, cannot fully consider whether muscles are being stretched as intended. That is, a person can move a body part in an appropriate manner without stretching the correct muscles or without stretching the correct muscles to the appropriate degree.

[0010] To help account for muscle elongation, the present disclosure includes an exercise coaching system and method that uses both posture and muscle elongation monitoring to help improve the outcomes of physical therapists. The system helps support patient decision-making to move body parts in an appropriate manner to achieve a target amount of muscle elongation. The system receives video of a person performing an exercise, analyzes the video to determine both posture and muscle elongation data, and provides advice or corrections to improve the performance of the person performing the exercise. In some embodiments, the system receives additional information, such as input from the user after the exercise, visual input from multiple angles, sensors on the surface of the body, neural network (NN) analysis of the body, or other suitable information. Using this additional information in combination with the video input, the system can determine whether the exercise was performed correctly.

[0011] In some embodiments, the system further includes a trained neural network (NN) that can be used to analyze the movements of a person performing an exercise. The trained NN can plot an analysis of the movement in a solution space. The trained NN can then select feedback to improve the exercise based on the determined muscle stretch information and the target muscle stretch. In some embodiments, the NN is retrained using the analysis of the movement. The system can provide feedback from a live trainer or automated feedback, such as text, text-to-audio, recommended videos, or other suitable automated feedback.

[0012] The system helps improve analysis of a person's performed movements by analyzing both posture and muscle elongation information. Analyzing both types of information improves the system's ability to provide relevant feedback to the person to enhance the benefit of PT to the person. The system also helps reduce the demand for live trainers by providing automated feedback, allowing a person to receive PT without having to wait for an appointment. In some cases, a person may not be able to easily leave their home and travel to a PT center. The system helps allow a person to receive PT in their home or other convenient location without having to travel to a center.

[0013] FIG. 1 is a flowchart of a method 100 for providing exercise instruction, according to some embodiments. Method 100 is performed to provide exercise instruction to a person, also referred to as a patient, performing exercise. In some embodiments, method 100 is performed using system 600 ( FIG. 6 ). In some embodiments, method 100 is performed by a system other than system 600 ( FIG. 6 ). In some embodiments, method 100 is performed using a user interface (UI), such as UI 200 ( FIG. 2 ) or another suitable UI. In some embodiments, method 100 is performed using system 300 ( FIG. 3 ). In some embodiments, method 100 is performed using a trained neural network, for example, using method 400 ( FIG. 4 ). In some embodiments, method 100 is performed using feedback selection using a solution space, for example, solution space 500 ( FIG. 5 ). Those skilled in the art will recognize that the following description of method 100 can be modified within the scope of this application.

[0014] In operation 105, images of the exercise are captured using a camera. The images are images of the patient performing the exercise. In some embodiments, the images are portions of a video. In some embodiments, the images are captured using a single camera. In some embodiments, the images are captured using multiple cameras. In some embodiments, the camera comprises a visible light camera. In some embodiments, the camera comprises a structure light sensor (SLS) camera. In some embodiments, the camera comprises an infrared (IR) camera. In some embodiments involving multiple cameras, different types of cameras are used to capture the images. In some embodiments involving multiple cameras, each of the cameras is of the same type.

[0015] In some embodiments, the camera is part of a device that includes a transmitter, such as a smartphone, tablet, or computer. In some embodiments, the camera is a standalone device that is not capable of wireless transmission.

[0016] In some embodiments, the image is transmitted from the camera to a server located remotely. In some embodiments, the image is transmitted wirelessly. In some embodiments, the image is transmitted via a wired connection. In some embodiments, the image is transmitted to an intermediate device for transfer from the camera to the server. In some embodiments, the image is input into an application or software program, such as an application on a smartphone or tablet, or a computer program running on the program. In some embodiments, the image is captured using an application or software program.

[0017] In operation 110, posture information is extracted from the images and the posture of the user is compared to a reference posture. The posture information indicates the movement of parts of the user, i.e., the patient's body. The posture information is extracted from the images captured in operation 105. In some embodiments, the images are processed to enable extraction of the posture information. For example, in some embodiments, the images are processed to reduce the patient's body to a skeleton.

[0018] The posture information tracks the patient's body movements while performing the exercise. In some embodiments, the posture information relates to the patient's body movements through at least one complete cycle of the exercise. In some embodiments, the posture information relates to the patient's body position in one or more frames during the performance of the exercise. In some embodiments, the one or more frames are selected based on the patient's body position identified as a primary position for stretching a target muscle of the patient's body. In some embodiments, the posture information tracks the patient's body movements through multiple cycles of the exercise. In some embodiments that track posture information through multiple cycles, an average of the posture information is used for subsequent analysis. In some embodiments that track posture information through multiple cycles, a mode of the posture information is used for subsequent analysis. Those skilled in the art will recognize that other criteria, such as standard deviation, can also be used to analyze the posture information across multiple cycles.

[0019] In operation 115, the posture information is compared to a reference image to determine whether the difference between the posture information and the reference image is less than a predetermined threshold. The reference image is an image that shows the proper or correct movement of the body when performing the same exercise performed by the patient. In some embodiments, the reference image is an image of a person. In some embodiments, the reference image is a computer-generated image. In some embodiments, the reference image includes an image of a skeleton.

[0020] The threshold indicates an acceptable difference between the posture information and the reference image to achieve the patient's PT goal. The threshold ranges from a value greater than 0, indicating a perfect match with the reference image, to a value less than 1, indicating a complete difference from the reference image. The threshold is based on the exercise being performed. The threshold is further based on patient information. In some embodiments, the patient information includes the patient's age, the patient's experience with exercise, previous patient performance, or other appropriate patient information. For example, in some embodiments, the patient is relatively new to PT and has little experience performing exercises. The threshold is reduced for patients who are new to PT to account for possible lower flexibility in parts of the patient's body. In some embodiments, the threshold is set by a trainer before the patient begins the exercise. In some embodiments, the threshold is set automatically based on the exercise and patient information. In some embodiments, the threshold changes for a patient as their experience or flexibility increases in future PT sessions. In some embodiments, the threshold is set using a trained neural network that can be used to determine how other patients with similar patient information would perform the current exercise.

[0021] In response to determining that the difference between the pose information and the reference image is greater than or equal to the threshold, method 100 proceeds to operation 120. In response to determining that the difference between the pose information and the reference image is less than the threshold, method 100 proceeds to operation 125.

[0022] In operation 120, feedback is provided to the patient to correct their posture. In some embodiments, the feedback includes a sample animation, such as a reference image. In some embodiments, the feedback includes a graphic, such as a series of still or moving image graphics. In some embodiments, the feedback includes text. In some embodiments, the feedback includes audio. Those skilled in the art will understand that combinations of the above types of feedback or other suitable types of feedback are within the scope of this specification.

[0023] In some embodiments, the feedback is provided using a UI, such as UI 200 (FIG. 2) or another suitable UI. In some embodiments, the feedback is automatically selected by an application or computer program. In some embodiments, the feedback is automatically selected based on a trained NN. In some embodiments, the feedback is automatically selected based on a solution space, such as solution space 500 (FIG. 5). In some embodiments, the feedback is selected by a trainer.

[0024] In some embodiments, the feedback is automatically displayed to the patient. In some embodiments, an alert is generated when feedback is available. In some embodiments, the alert comprises an audio or visual alert. In some embodiments, the remote server is configured to transmit the feedback to a device, such as a smartphone, tablet, or computer, accessible to the patient. In some embodiments, the alert comprises instructions to cause the device to automatically display the alert.

[0025] In operation 125, muscle stretch information is calculated. In some embodiments, the muscle stretch information is calculated based on the images captured in operation 105. In some embodiments, the muscle stretch information is calculated based on images from multiple cameras. In some embodiments, the muscle stretch information is calculated based on a trained neural network, for example, a neural network trained using method 400 (FIG. 4). In some embodiments, the muscle stretch information is determined based on changes in the shape of the patient's body as the patient's body parts move during exercise.

[0026] In some embodiments, the muscle stretch information is obtained from a source other than the image received in act 105. In some embodiments, the muscle stretch information is received by input from the patient. In some embodiments, the input is received from the patient using text. In some embodiments, the input is received from the patient using detection of the patient's voice. In some embodiments, the input is received from the patient via a UI, such as UI 200 (FIG. 2) or another suitable UI. In some embodiments, the muscle stretch information is received from one or more sensors attached to the patient's body during exercise.

[0027] In operation 130, feedback is provided to the patient to improve the patient's muscle lengthening. In some embodiments, the feedback includes a sample animation, such as a reference image. In some embodiments, the feedback includes a graphic, such as a series of still or moving graphic images. In some embodiments, the feedback includes text. In some embodiments, the feedback includes audio. Those skilled in the art will understand that combinations of the above types of feedback or other suitable types of feedback are within the scope of this specification.

[0028] In some embodiments, the feedback is provided using a UI, such as UI 200 (FIG. 2) or another suitable UI. In some embodiments, the feedback is automatically selected by an application or computer program. In some embodiments, the feedback is automatically selected based on a learned NN, for example, a NN trained using method 400 (FIG. 4). In some embodiments, the feedback is automatically selected based on a solution space, such as solution space 500 (FIG. 5). In some embodiments, the feedback is selected by a trainer.

[0029] In some embodiments, the feedback is automatically displayed to the patient. In some embodiments, an alert is generated when feedback is available. In some embodiments, the alert comprises an audio or visual alert. In some embodiments, the remote server is configured to transmit the feedback to a device, such as a smartphone, tablet, or computer, accessible to the patient. In some embodiments, the alert comprises instructions to cause the device to automatically display the alert.

[0030] In some embodiments, feedback is selected based on patient information. For example, in some embodiments where a patient has recently started PT, excessive muscle stretching is not intended. As a result, in some embodiments, feedback is selected to adjust the patient's muscle stretch from a calculated value to a target value, where the target value is less than the maximum muscle stretch. Additional discussion related to the selection of feedback for the target value of muscle stretch is provided below with respect to solution space 500 (FIG. 5).

[0031] In operation 135, feedback is received from the user. The patient provides feedback to the trainer, application, or computer program regarding how the patient feels after the exercise. In some embodiments, the patient provides feedback indicating the amount of stretch the patient felt while performing the exercise. For example, in some embodiments, the patient is asked to enter a value from 1 to 5 indicating the amount of stretch the patient felt in a particular muscle while performing the exercise. In some embodiments, the patient provides feedback regarding the muscle stretch received in operation 130. In some embodiments, the input is received from the patient using text. In some embodiments, the input is received from the patient using detection of the patient's voice. In some embodiments, the input is received from the patient via a UI, such as UI 200 (FIG. 2) or another suitable UI.

[0032] In some embodiments, the feedback received in act 135 causes an alert to automatically display on a device accessible by the trainer, such as a smartphone, tablet, or computer. In some embodiments, the alert is generated when patient feedback is available. In some embodiments, the alert includes an audio or visual alert. In some embodiments, the alert includes instructions to cause the device to automatically display the alert.

[0033] In act 140, the model used to calculate the muscle stretch information is updated based on feedback from the patient. The model is used to help enhance the feedback provided to the patient during future PT sessions. For example, in some embodiments, the patient inputs feedback indicating a larger amount of muscle stretch, but the calculated muscle stretch information in act 125 indicates a smaller amount of muscle stretch. In some embodiments, updating the model can be used to help account for the patient's inexperience in determining the amount of muscle stretch. In some embodiments, updating the model can be used to help calibrate the calculation in act 125 for a particular user to help improve the selection of muscle stretch feedback in future PT sessions.

[0034] In some embodiments, operation 140 is omitted. Omitting operation 140 reduces the processing load on a system, such as system 600 (FIG. 6), used to implement method 100. Retaining operation 140 helps improve the relevance of the feedback provided for muscle stretch in operation 130.

[0035] Those skilled in the art will understand that modifications to method 100 are within the scope of this specification. In some embodiments, additional operations are included in method 100. For example, in some embodiments, data regarding the patient's progress over multiple PT sessions is provided to the patient. In some embodiments, at least one operation is omitted from method 100. For example, in some embodiments, operation 140 is omitted. In some embodiments, the order of operations in method 100 is adjusted. For example, in some embodiments, operation 125 is performed before operation 115.

[0036] Using method 100, improved feedback is provided to the patient to improve the exercises they perform compared to other techniques by including analysis and feedback related to muscle stretch information as well as postural information. Analysis of muscle stretch information helps patients increase the benefits of PT compared to other techniques that rely solely on postural information.

[0037] 2 is a diagram of a user interface (UI) 200 for a system for providing exercise instruction, according to some embodiments. The UI 200 can be used to view a reference video, record a patient's exercise video, and provide feedback between the patient and the system or trainer. In some embodiments, the UI 200 is implemented using system 600 (FIG. 6). In some embodiments, the UI 200 is implemented using a system other than system 600 (FIG. 6). Those skilled in the art will recognize that the following description of the UI 200 can be modified within the scope of this application.

[0038] The UI 200 includes a reference video 205. The reference video 205 includes a video of a trainer or another patient performing an exercise in an appropriate manner. In some embodiments, the reference video 205 includes an animated video on behalf of the trainer or patient. The UI 200 further includes a plurality of reference images 210 that allow scrolling to different portions of the reference video 205. In some embodiments, the reference video 205 or the reference images 210 are usable in the method 100 to perform at least operation 115 or operation 120 (FIG. 1).

[0039] The UI 200 further includes a patient video 215. The patient video 215 is a video of the patient performing an exercise. In some embodiments, the face or other identifying features of the patient video 215 are automatically obscured during recording to protect the patient's privacy. The patient video 215 can be used by the method 100 to calculate posture or muscle extension information. The UI 200 further includes a plurality of patient images 220. The plurality of patient images 220 can be used to provide still images for analysis and comparison with the reference image 210 during analysis of the exercise performed by the patient.

[0040] The UI 200 further includes a feedback panel 225. The feedback panel 225 can be used to display feedback to the patient to improve either their posture or their muscle extensions, for example, using method 100 (FIG. 1). The feedback panel 225 includes a representation of a human form with various muscle contours. The representation of the human form can be used to help the patient visualize the various parts of their body that are being discussed as part of the feedback to help the patient improve their posture or muscle extensions during exercise.

[0041] The UI 200 further includes a feedback description 230. The feedback description 230 includes text describing the type of error made by the patient during exercise and recommendations for correcting the error. The UI 200 includes the feedback description 230 as a text box. In some embodiments, the feedback description 230 includes an audio file. In some embodiments, the feedback description 230 includes both an audio file and a text box. In some embodiments, the UI 200 includes a feedback panel 225 that highlights the muscles discussed in the feedback description 230.

[0042] In some embodiments, the UI 200 further includes fields for receiving input from the patient. In some embodiments, the input includes information such as muscle stretch information, feedback from the patient regarding corrections provided by a trainer or the system, or other suitable input. In some embodiments, the UI 200 includes a button or other item that allows the user to record an audio message as part of the patient input.

[0043] The UI 200 is used to provide the patient with both a reference animation 205 to emulate during the exercise, and a feedback panel 225 and feedback descriptions 230 to instruct the patient on how to improve their performance of the exercise. Using the UI 200 as part of a method for providing both postural and muscle extension feedback, such as method 100 (FIG. 1), helps improve the effectiveness of corrections for the patient. Improving the effectiveness of corrections helps increase the usefulness of PT for the patient.

[0044] FIG. 3 is a schematic diagram of a system 300 for providing exercise instruction, according to some embodiments. System 300 provides exercise instruction to a person, also referred to as a patient, performing exercise. In some embodiments, system 300 is implemented as part of or in conjunction with system 600 (FIG. 6). In some embodiments, system 300 is implemented independently of system 600 (FIG. 6). In some embodiments, system 300 is implemented using a user interface (UI), such as UI 200 (FIG. 2) or another suitable UI. In some embodiments, system 300 implements method 100 (FIG. 1). In some embodiments, system 300 is implemented using a trained neural network, for example, using method 400 (FIG. 4). In some embodiments, system 300 is implemented using feedback selection using a solution space, for example, solution space 500 (FIG. 5). Those skilled in the art will recognize that the following description of system 300 can be modified within the scope of this application.

[0045] The system 300 receives an exercise video as input and determines posture information using a posture processing module 305. In some embodiments, the posture processing module 305 determines posture information similar to that of the motion 110 (FIG. 1).

[0046] System 300 further determines whether to provide posture feedback or muscle stretch feedback based on posture information from posture processing module 305 using modality selector 310. In some embodiments, modality selector 310 determines which type of feedback to provide based on whether a difference between the posture information and the reference information is less than a predetermined threshold. In some embodiments, modality selector 310 determines which type of feedback to provide in a manner similar to operation 115 (FIG. 1). In response to a decision to provide posture feedback, system 300 proceeds to generate feedback using a feedback generator. In response to a decision to provide muscle stretch feedback, system 300 proceeds to determine muscle stretch information using muscle processing module 320.

[0047] The muscle processing module 320 includes a muscle stretch prediction module 322. The muscle stretch prediction module 322 includes a trained NN for predicting the amount of muscle stretch of a patient based on the exercise video. In some embodiments, the muscle stretch prediction module 322 uses the exercise video along with other data, such as additional images, additional video, sensor data, patient input, or other suitable data. The muscle processing module 320 further includes a best posture selector 324 configured to select a recommendation to provide feedback to the patient based on the output of the muscle processing module 320.

[0048] In some embodiments, the muscle stretch prediction module 322 includes a trained neural network (NN) configured to receive the exercise video and some additional data and output the degree to which each monitored muscle of the patient is stretched during exercise. The monitored muscles include gluteal muscles, calf muscles, adductor muscles, etc. In some embodiments, the degree of stretch is measured on a scale of 0 to 5, with 0 being no stretch and 5 being maximum stretch. For example, in some embodiments, the muscle stretch prediction module 322 outputs a solution such as {(gluteal muscles, 4), (calf muscles, 0), (adductor muscles, 1)}. Those skilled in the art will recognize that this output format is merely exemplary and that other output formats are within the scope of this specification.

[0049] The NN of the muscle stretch prediction module 322 is trained using training videos. The training videos include multiple samples of each of the exercises supported by the system 300. In some embodiments, the training videos include videos captured by a trainer during a PT session. The training videos are annotated with feedback for improving muscle stretches for each training video. An example of a training video input may include, for example, (clamshell_1.mp4,{(gluteus, 1),(calf, 0),(adductors, 5)}<lean left side of body forward 10 degrees>. In this sample input, “clamshell” indicates the type of exercise being performed, “mp4” indicates the format of the video file, the data within the curly brackets “{}” indicates the muscle extension being monitored, and the information within the arrows “< >” indicates feedback to improve the muscle extension. Those skilled in the art will recognize that the above example is not intended to be limiting and that different input formats are within the scope of this specification. The degree of muscle extension for the training input is determined by the trainer during initial training of the NN. In addition, feedback is determined by the trainer during initial training of the NN. In some embodiments, once the NN is initially trained, the NN can receive input from a patient or a trainer to further train or retrain the NN, such as using operation 140 (FIG. 1). In some embodiments, training of the NN is performed using multi-label regression, sum of mean squared error loss, or other suitable machine learning algorithms.

[0050] Using the trained neural network, the muscle stretch prediction module 322, in some embodiments, can receive exercise videos along with additional data and determine which muscles of the patient are stretched and to what extent based on a comparison between the exercise videos and data collected from the training videos used to train the neural network. In some embodiments, the muscle stretch prediction module 322 extracts data directly from the exercise videos, for example, using a three-dimensional convoluted neural network (3DCNN) or a CNN combined with a long short-term memory (LSTM) for each exercise performed by the patient. The muscle stretch prediction module 322 collects both skeletal data indicating how the patient's body parts move during exercise and data indicating muscle contraction or movement. Based on the collected data, the muscle stretch prediction module 322 can estimate the amount of stretch of each monitored muscle of the patient.

[0051] Best posture selector 324 is configured to receive the output of muscle stretch prediction module 322 and determine the feedback to provide to the patient. In some embodiments, best posture selector 324 is configured to provide the feedback as text, audio, video, or another appropriate format. In some embodiments, best posture selector 324 is configured to consider patient information, such as age, experience, etc., when determining the feedback to the patient. For example, in some embodiments where the patient is elderly or new to PT, best posture selector 324 does not select feedback intended to maximize muscle stretch. Instead, best posture selector 324 may select feedback to provide the patient with modest improvements in muscle stretch until the patient reaches performance consistent with the patient's current health status. In some embodiments, best posture selector 324 utilizes a solution space, such as solution space 500 (FIG. 5), as a tool for selecting feedback to the patient.

[0052] System 300 further includes a feedback generator 330 configured to receive the output of best posture selector 324 and / or the output of modality selector 310. Feedback generator 330 is configured to process and send feedback to the patient to improve either posture or muscle length associated with the exercise. In some embodiments, feedback generator 330 is configured to output the feedback to a device accessible to the patient, such as a smartphone, tablet, or computer. In some embodiments, upon receiving validation, feedback generator 330 provides the feedback to a trainer for validation before sending the feedback to the patient.

[0053] Those skilled in the art will recognize that modifications to system 300 are within the scope of this specification. Those skilled in the art will further understand that various components of system 300 may be implemented using one or more processors.

[0054] Using system 300, improved feedback is provided to the patient to improve the exercises they perform compared to other techniques by including analysis and feedback related to muscle stretch information as well as posture information. Analysis of muscle stretch information helps patients increase the benefits of PT compared to other techniques that rely solely on posture information.

[0055] FIG. 4 is a flowchart of a method 400 for muscle stretch prediction to determine muscle stretch information, according to some embodiments. Method 400 can be used to provide exercise instruction to a person, also referred to as a patient, performing exercise. In some embodiments, method 400 is a method that implements the functionality of muscle stretch prediction module 322 (FIG. 3). In some embodiments, method 400 is implemented as part of or in conjunction with system 300 (FIG. 3) or system 600 (FIG. 6). In some embodiments, method 400 is implemented independently of system 300 (FIG. 3) and system 600 (FIG. 6). In some embodiments, method 400 is implemented using a user interface (UI), such as UI 200 (FIG. 2) or another suitable UI. In some embodiments, method 400 is implemented in conjunction with method 100 (FIG. 1). In some embodiments, method 400 is implemented independently of method 100 (FIG. 1). In some embodiments, method 400 is implemented using feedback selection using a solution space, e.g., solution space 500 (FIG. 5). Those skilled in the art will recognize that the following description of the method 400 can be modified within the scope of the present application.

[0056] The method 400 includes receiving an exercise video 405. The exercise video includes images of a patient performing an exercise. In some embodiments, the exercise video 405 is accompanied by other data, such as additional images, additional videos, sensor data, patient input, or other suitable data.

[0057] In operation 410, three-dimensional (3D) pose estimation and tracking is performed using the motion video, in some embodiments, along with other data. Pose estimation and tracking converts the motion video, in some embodiments, along with other data, into skeletal features of the patient's body. In some embodiments, the conversion is performed by tracking limbs or joints of the body. In some embodiments, the motion video includes images from multiple angles to aid in estimation and tracking of body parts that are occluded when images from only a single viewing angle are used.

[0058] In operation 415, body movement is captured by tracking limbs and / or joints of the body across a series of images captured at different times. In some embodiments, operation 415 tracks the patient's body movement through at least one complete cycle of motion. In some embodiments, operation 415 tracks the patient's body movement through multiple cycles of motion. Operation 415 is configured to generate posture information. In some embodiments, operation 415 is performed in a manner similar to operation 110 (FIG. 1).

[0059] In operation 420, feature functions associated with the tracked movements are generated. In some embodiments where operation 415 tracks movements over multiple cycles, an average of the posture information is used for subsequent analysis. In some embodiments where posture information is tracked over multiple cycles, the maximum amount of movement in the posture information is used for subsequent analysis. In some embodiments where posture information is tracked over multiple cycles, the minimum amount of movement in the posture information is used for subsequent analysis. Those skilled in the art will recognize that other criteria, such as standard deviation, can also be used to analyze the posture information over multiple cycles. Operation 420 is configured to output skeletal features of the exercise video 405.

[0060] In operation 425, the NN, in some embodiments together with additional data, is used to extract muscle extension data from the exercise video. The trained NN, in some embodiments together with additional data, is used to predict muscle extension information based on the exercise video 405. In some embodiments, the NN is a 3D CNN. Operation 425 is configured to output muscle features of the exercise video 405.

[0061] In operation 430, the muscle features and skeletal features are aggregated. Relationships between the muscle features and the skeletal features are identified. In some embodiments, a trained NN is used to determine the influence of the skeletal features on the muscle features. In some embodiments, operation 430 is performed using the same NN as operation 425. In some embodiments, the feature aggregation includes combining the muscle features and the skeletal features. In some embodiments, the feature aggregation further includes concatenation, average pooling, or some other method of aggregating the muscle features and the skeletal features. Operation 430 outputs the aggregated features.

[0062] In operation 435, the aggregated features are classified to generate muscle stretch information. The muscle stretch information indicates the degree of stretch of each monitored muscle. In some embodiments, operation 435 is performed using the same NN as operation 425. In some embodiments, operation 435 is integrated into operation 425. In some embodiments, the muscle stretch information output from operation 435 is similar to the output of muscle stretch prediction module 322 (FIG. 3).

[0063] Those skilled in the art will recognize that modifications to method 400 are within the scope of this specification. In some embodiments, at least one additional operation is included in method 400. For example, in some embodiments, method 400 includes capturing additional data other than exercise video 405. In some embodiments, at least one operation is omitted from method 400. For example, in some embodiments, operation 430 is integrated into operation 425, and operation 430 is omitted as a separate operation.

[0064] Using method 400, improved feedback is provided to the patient to improve their performed exercises compared to other techniques by including analysis and feedback related to muscle stretch information as well as postural information. Analysis of muscle stretch information helps the patient increase the benefits of PT compared to other techniques that rely solely on postural information.

[0065] FIG. 5 is an illustration of a solution space 500 for providing exercise instruction recommendations, according to some embodiments. The solution space 500 can be used to identify feedback to a patient to improve the patient's performance in performing an exercise. In some embodiments, the solution space 500 is generated and analyzed by system 300 ( FIG. 3 ) or system 600 ( FIG. 6 ). In some embodiments, the solution space 500 is generated or analyzed independently of system 300 ( FIG. 3 ) and system 600 ( FIG. 6 ). In some embodiments, the solution space 500 can be used to generate feedback to the patient that is provided as part of a user interface (UI), such as UI 200 ( FIG. 2 ) or another suitable UI. In some embodiments, the solution space 500 is used by method 100 ( FIG. 1 ) or method 400 ( FIG. 4 ) to generate feedback for the patient. In some embodiments, the solution space 500 is used by a method other than method 100 ( FIG. 1 ) or method 400 ( FIG. 4 ). Those skilled in the art will recognize that the following description of the solution space 500 can be modified within the scope of the present application.

[0066] The solution space 500 plots the patient's exercise performance 505. The location of the performance 505 in the solution space 500 is based on both muscle elongation information and skeletal information obtained by analyzing the patient's exercise video. The solution space 500 is specific to the type of exercise being performed by the patient and the particular muscles monitored while performing the exercise. In the solution space 500, muscles are represented by X muscleThe position of performance 505 in solution space 500 is based on muscle stretch information determined based on data collected during the patient's performance of the exercise. In some embodiments, the data includes exercise videos, such as exercise videos 405 (FIG. 4), or other data, as described above. Solution space 500 is generated using training data for the NN, such as during performance of muscle stretch prediction module 322 (FIG. 3). In some embodiments, solution space 500 is updated based on newly received data, such as by operation 140 (FIG. 1).

[0067] In addition to performance 505, solution space 500 also includes multiple feedback options, such as 510, 515, and 520. For clarity of the drawing, only three feedback options are labeled. The feedback options correspond to feedback that can be provided to a patient to improve muscle extension. In some embodiments, the feedback options correspond to text or audio feedback. In some embodiments, the feedback options correspond to a video of a trainer or another patient performing the same exercise as the current patient.

[0068] Each feedback option, such as 510, 515, or 520, includes an initial stretch value 502a and a revised stretch value 502b. The initial stretch value 502a indicates the degree of muscle stretch during a previous exercise repetition. The revised stretch value 502b indicates the degree of muscle stretch following feedback associated with the feedback option. For example, a previous patient performed an exercise and the NN determined that the muscle stretch had a degree of 3, corresponding to the initial stretch value 502a. The previous patient was then given the feedback option and performed the exercise again. Following the feedback, the NN determined that the muscle stretch had a degree of 5, corresponding to the revised value 502b.

[0069] The solution space 500 can be used to determine the patient's degree of muscle stretch from the most recent repetition of the exercise and to suggest feedback for the patient to reach the target degree of muscle stretch. The feedback option closest to the performance 505 with a modified stretch value closest to the target degree of muscle stretch is selected as the feedback provided to the patient. For example, if the patient has a performance 505 that indicates a degree of muscle stretch as 1 and the target muscle stretch is to maximize the degree of muscle stretch, the solution space 500 can be used to identify feedback options 510. The feedback options 510 include an initial stretch value of 1, which matches the performance 505, and a modified stretch value of 5, which is the maximum degree of muscle stretch. As a result, the solution space 500 is used to identify feedback options 510 to provide to the patient to improve in the next repetition of the exercise. In some instances, the target degree of muscle stretch differs from the maximum degree of muscle stretch for reasons discussed above. In such situations, the solution space 500 can be used to select the feedback option closest to the target degree of muscle stretch. For example, if the patient is relatively new to PT and only a moderate increase in muscle stretch is desired, solution space 500 can be used to select feedback option 520 to provide to the patient. In another example, if the patient has moderate experience with PT and a greater degree of, but less than maximum, muscle stretch is desired, solution space 500 can be used to select feedback option 515.

[0070] Those skilled in the art will understand that modifications to solution space 500 are within the scope of the present specification. For example, in some embodiments, solution space 500 is adjusted to account for the degree of stretching of multiple muscles during a single movement.

[0071] Using the solution space 500 helps improve the exercise instruction provided to the patient by identifying feedback that has helped patients with similar levels of performance improve to reach their target muscle length. As a result, the patient receives feedback that is most relevant to them and most likely to help them progress during PT, reducing their frustration levels compared to approaches that cannot provide customized feedback.

[0072] 6 is a diagram of a system 600 for providing exercise instruction, according to some embodiments. The system 600 comprises a hardware processor 602 and a non-transitory computer-readable storage medium 604 encoded with, i.e., storing, computer program code 606, i.e., a set of executable instructions. The computer-readable storage medium 604 is also encoded with instructions 607 for interfacing with external devices. The processor 602 is electrically coupled to the computer-readable storage medium 604 via a bus 608. The processor 602 is also electrically coupled by the bus 608 to an input / output (I / O) interface 610. A network interface 612 is also electrically connected to the processor 602 via the bus 608. The network interface 612 is connected to a network 614, such that the processor 602 and the computer-readable storage medium 604 can be connected to external elements via the network 614. The processor 602 is configured to execute computer program code 606 encoded on the computer-readable storage medium 604 to enable the system 600 to perform some or all of the operations described in the method 100 (FIG. 1), to implement the UI 200 (FIG. 2), the system 300 (FIG. 3), the method 400 (FIG. 4), or to use the solution space 500 (FIG. 5).

[0073] In some embodiments, the processor 602 is a central processing unit (CPU), a multiprocessor, a distributed processing system, an application specific integrated circuit (ASIC), and / or other suitable processing unit.

[0074] In some embodiments, computer-readable storage medium 604 is an electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor system (or apparatus or device). For example, computer-readable storage medium 604 includes semiconductor or solid-state memory, magnetic tape, removable computer diskette, random access memory (RAM), read-only memory (ROM), rigid magnetic disk, and / or optical disk. In some embodiments using an optical disk, computer-readable storage medium 604 includes a compact disk-read-only memory (CD-ROM), compact disk-read / write (CD-R / W), and / or digital video disk (DVD).

[0075] In some embodiments, the storage medium 604 stores computer program code 606 configured to cause the system 600 to perform the method 100 or the method 400 . In some embodiments, the storage medium 604 also stores information used to perform some or all of the operations described in method 100 (FIG. 1), implement UI 200 (FIG. 2), system 300 (FIG. 3), method 400 (FIG. 4), or use solution space 500 (FIG. 5), as well as information generated while performing some or all of the operations described in method 100 (FIG. 1), implement UI 200 (FIG. 2), system 300 (FIG. 3), method 400 (FIG. 4), or use solution space 500 (FIG. 5), e.g., posture data parameters 616, posture threshold parameters 618, muscle stretch data parameters 620, muscle stretch NN parameters 622, solution space parameters 624, and / or a set of executable instructions for performing some or all of the operations described in method 100 (FIG. 1), implement UI 200 (FIG. 2), system 300 (FIG. 3), method 400 (FIG. 4), or use solution space 500 (FIG. 5).

[0076] In some embodiments, storage medium 604 stores instructions 607 for interfacing with an external device. Instructions 607 enable processor 602 to generate manufacturing instructions readable by an external device to effectively perform some or all of the operations described in method 100 (FIG. 1), implement UI 200 (FIG. 2), system 300 (FIG. 3), method 400 (FIG. 4), or use solution space 500 (FIG. 5).

[0077] System 600 includes an I / O interface 610. I / O interface 610 is coupled to external circuitry. In some embodiments, I / O interface 610 includes a touchscreen, a keyboard, a keypad, a mouse, a trackball, a trackpad, and / or cursor direction keys for communicating information and commands to processor 602.

[0078] System 600 also includes a network interface 612 coupled to processor 602. Network interface 612 enables system 600 to communicate with a network 614 to which one or more other computer systems are connected. Network interface 612 may include a wireless network interface, such as BLUETOOTH, Wi-Fi, WIMAX, GPRS, or WCDMA, or a wired network interface, such as ETHERNET, USB, or IEEE-1394. In some embodiments, some or all of the operations described in method 100 (FIG. 1) and for implementing UI 200 (FIG. 2), system 300 (FIG. 3), method 400 (FIG. 4), or using solution space 500 (FIG. 5) are performed in two or more systems 600, and information, such as posture data, posture thresholds, muscle stretch data, muscle stretch NNs, or solution spaces, is exchanged between the different systems 600 via network 614.

[0079] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0080] (Appendix 1) One aspect of the present disclosure relates to a system for providing exercise instruction. The system includes a non-transitory computer-readable medium configured to store instructions. The system further includes a processor connected to the non-transitory computer-readable medium. The processor is configured to execute instructions to receive input data from a user, the input data including a plurality of images of the user performing an exercise. The processor is configured to execute instructions to extract posture data from the input data. The processor is configured to execute instructions to determine whether a difference between the extracted posture data and a reference is less than a predetermined threshold. The processor is configured to execute instructions to determine muscle stretch information from the input data in response to the difference being less than the predetermined threshold. The processor is configured to execute instructions to determine, in response to the determined muscle stretch information, feedback to provide to the user to increase the degree of muscle stretch during subsequent performance of the exercise.

[0081] (Appendix 2) 10. The system of claim 1, wherein the processor is further configured to execute instructions for determining, in response to the difference being greater than or equal to a predetermined threshold, feedback to provide to the user for correcting the user's posture in subsequent performance of the exercise.

[0082] (Appendix 3) 3. The system of claim 1 or 2, wherein the processor is further configured to execute instructions for determining muscle stretch information using a trained neural network (NN), the trained NN being trained using videos of others performing the exercise.

[0083] (Appendix 4) 4. The system of any one of claims 1 to 3, wherein the processor is further configured to execute instructions for updating the trained NN using the muscle stretch information.

[0084] (Appendix 5) 5. The system of any one of claims 1 to 4, wherein the processor is further configured to execute instructions for determining the feedback using the trained NN to select a feedback option from a solution space.

[0085] (Appendix 6) 6. The system of any one of claims 1 to 5, wherein the processor is configured to execute instructions for selecting a feedback option from a solution space based on an initial muscle stretch value and a target muscle stretch value, the initial muscle stretch value being determined based on muscle stretch information and the target muscle stretch value being determined based on user information of a user performing an exercise, and the feedback being usable to improve the user's decision-making for moving at least one body part of the user.

[0086] (Appendix 7) 7. The system of any one of claims 1 to 6, wherein the predetermined threshold is based on user information of the user performing the exercise.

[0087] (Appendix 8) One aspect of the present disclosure relates to a method for providing exercise instruction, the method including receiving input data from a user, the input data including a plurality of images of the user performing an exercise. The method includes extracting posture data from the input data. The method includes determining whether a difference between the extracted posture data and a reference is less than a predetermined threshold. The method includes determining muscle stretch information from the input data in response to the difference being less than the predetermined threshold. The method includes determining, in response to the determined muscle stretch information, feedback to provide to the user to increase the degree of muscle stretch during subsequent performance of the exercise.

[0088] (Appendix 9) 9. The method of claim 8, further comprising determining, in response to the difference being greater than or equal to a predetermined threshold, feedback to provide to the user to correct the user's posture in subsequent performances of the exercise.

[0089] (Appendix 10) 10. The method of claim 8 or 9, wherein the muscle stretch information is determined using a trained neural network (NN), and the trained NN is trained using videos of others performing the exercise.

[0090] (Appendix 11) 11. The method of any one of claims 8 to 10, further comprising updating the trained neural network using muscle stretch information.

[0091] (Appendix 12) 12. The method of any one of claims 8 to 11, wherein the feedback is determined using a trained neural network to select a feedback option from a solution space.

[0092] (Appendix 13) 13. The method of any one of claims 8 to 12, wherein selecting a feedback option from the solution space is based on an initial muscle stretch value and a target muscle stretch value, the initial muscle stretch value is determined based on muscle stretch information, and the target muscle stretch value is determined based on user information of a user performing the exercise, and the feedback can be used to improve the user's decision-making to move at least one body part of the user.

[0093] (Appendix 14) 14. The method of any one of claims 8 to 13, wherein the predetermined threshold is based on user information of the user performing the exercise.

[0094] (Appendix 15) One aspect of the present disclosure relates to a program for providing exercise instruction. The program causes a processor to perform operations including receiving input data from a user, the input data including a plurality of images of the user performing an exercise. The program causes the processor to perform operations including extracting posture data from the input data. The program causes the processor to perform operations including determining whether a difference between the extracted posture data and a reference is less than a predetermined threshold. The program causes the processor to perform operations including determining muscle stretch information from the input data in response to the difference being less than the predetermined threshold. The program causes the processor to perform operations including determining, based on the determined muscle stretch information, feedback to provide to the user to increase the degree of muscle stretch during subsequent performance of the exercise.

[0095] (Appendix 16) 16. The non-transitory computer-readable medium of claim 15, further causing the processor to perform operations including, in response to the difference being greater than or equal to a predetermined threshold, determining feedback to provide to the user to modify the user's posture in subsequent performances of the exercise.

[0096] (Appendix 17) 17. The program of claim 15 or 16, further causing the processor to perform operations including determining muscle stretch information using a trained neural network (NN), the trained NN being trained using videos of others performing the exercise.

[0097] (Appendix 18) 18. The program of any one of claims 15 to 17, further causing a processor to perform operations including updating the trained neural network using the muscle stretch information.

[0098] (Appendix 19) 19. The program of any one of appendices 15 to 18, further causing a processor to perform operations including determining feedback determined using the trained NN to select a feedback option from a solution space; and selecting a feedback option from the solution space based on an initial muscle stretch value and a target muscle stretch value, wherein the initial muscle stretch value is determined based on muscle stretch information and the target muscle stretch value is determined based on user information of a user performing an exercise, and the feedback is usable to improve the user's decision-making to move at least one body part of the user.

[0099] (Appendix 20) 20. The program of any one of appendices 15 to 19, wherein the predetermined threshold is based on user information of the user performing the exercise.

[0100] The foregoing outlines features of several embodiments so that those skilled in the art may better understand aspects of the present disclosure. Those skilled in the art will readily appreciate that this disclosure may be used as a basis for designing or modifying other processes and structures which carry out the same purposes and / or achieve the same advantages as the embodiments presented herein. Those skilled in the art will also appreciate that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the present disclosure.

[0101] This application claims priority to U.S. Patent Application No. 18 / 749,619, filed June 21, 2025, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0102] 100, 400 ways 105, 110, 115, 120, 125, 130, 135, 140, 410, 415, 420, 425, 430, 435 operation 200 User Interface (UI) 205 Reference Videos 210 Reference Images 215 Patient Videos 220 patient images 225 Feedback Panel 230 Feedback Description 300, 600 systems 305 Attitude Processing Module 310 Modality Selector 320 Muscle Processing Module 322 Muscle elongation prediction module 324 Best Pose Selector 330 Feedback Generator 405 exercise videos 500 solution space 502a Initial elongation value 502b Corrected elongation value 505 Performance 510, 515, 520 Feedback Options 602 Hardware Processor 604 Non-transitory computer-readable storage medium 606 Computer Program Code 607 command 608 Bus 610 Input / Output (I / O) Interface 612 Network Interface 614 Network 616 Attitude Data Parameters 618 Posture Threshold Parameters 620 Muscle stretch data parameters 622 Muscle elongation NN parameters 624 Solution Space Parameters

Claims

1. a non-transitory computer-readable medium configured to store instructions; a processor coupled to the non-transitory computer-readable medium, receiving input data from a user, the input data including a plurality of images of the user performing an exercise; extracting pose data from the input data; determining whether a difference between the extracted posture data and a reference is less than a predetermined threshold; determining muscle stretch information from the input data in response to the difference being less than the predetermined threshold; determining feedback to provide to the user for increasing the degree of muscle elongation during subsequent performance of the exercise in response to the determined muscle elongation information; a processor configured to execute the instructions for: A system for providing exercise instruction, comprising:

2. 2. The system of claim 1, wherein the processor is further configured to execute the instructions for determining, in response to the difference being greater than or equal to the predetermined threshold, feedback to provide to the user for modifying the user's posture in the subsequent performance of the exercise.

3. 2. The system of claim 1, wherein the processor is further configured to execute the instructions to determine the muscle stretch information using a trained neural network (NN), the trained NN being trained using videos of others performing the exercise.

4. The system of claim 3 , wherein the processor is further configured to execute the instructions for updating the trained neural network using the muscle stretch information.

5. The system of claim 3 , wherein the processor is further configured to execute the instructions for determining the feedback using the trained neural network to select a feedback option from a solution space.

6. the processor is configured to execute the instructions to select the feedback option from the solution space based on an initial muscle stretch value and a target muscle stretch value, the initial muscle stretch value being determined based on the muscle stretch information and the target muscle stretch value being determined based on user information of the user performing the exercise; The system of claim 5 , wherein the feedback is usable to improve the user's decision-making for moving at least one body part of the user.

7. The system of claim 1 , wherein the predetermined threshold is based on user information of the user performing the exercise.

8. receiving input data from a user, the input data including a plurality of images of the user performing an exercise; extracting pose data from the input data; determining whether a difference between the extracted posture data and a reference is less than a predetermined threshold; determining muscle stretch information from the input data in response to the difference being less than the predetermined threshold; determining feedback to provide to the user in response to the determined muscle stretch information to increase the degree of muscle stretch during subsequent performance of the exercise; and 12. A method for providing exercise instruction, comprising:

9. 9. The method of claim 8, further comprising: determining feedback to provide to the user to modify the user's posture in the subsequent performance of the exercise in response to the difference being greater than or equal to the predetermined threshold.

10. receiving input data from a user, the input data including a plurality of images of the user performing an exercise; extracting pose data from the input data; determining whether a difference between the extracted posture data and a reference is less than a predetermined threshold; determining muscle stretch information from the input data in response to the difference being less than the predetermined threshold; determining feedback to provide to the user in response to the determined muscle stretch information to increase the degree of muscle stretch during subsequent performance of the exercise; and A program for causing a processor to perform operations including:

Images