Repetition counting with significant frame detection

Abstract

The invention relates to repetition counting with significant frame detection. Determining characteristics of user motion is described. The techniques include capturing a series of frames of a user performing a motion, and determining a progress prediction score and a significance score for each candidate action in a set of candidate actions based on features of the frames. The progress prediction score and the significance score are determined based on characteristics of the current frame and one or more previous frames. Progress predictors are determined and used to track repetitions of user motion. Upon detecting that the repetition has completed, saliency frames are identified based on the saliency scores.

Description

Background Technology

[0001] Current image data analysis techniques provide numerous insights into the scenes depicted in images. For example, object detection can be used to identify objects in a scene or the characteristics of objects within a scene. One application is applying image data to networks to determine human poses.

[0002] There are shortcomings when it comes to predicting the motion of objects. For example, to predict human activity, a sequence of video frames can be fed into the network, and predictions for that sequence can be obtained based on the entire video. However, there are problems in obtaining real-time predictions of user activity. Attached Figure Description

[0003] Figure 1 An example diagram is shown illustrating a technique for predicting user activity according to one or more implementation schemes.

[0004] Figure 2 Example diagrams are shown for determining action, progress, and significance scores according to one or more implementation schemes.

[0005] Figure 3 The flowchart illustrates techniques for detecting salient frames and performing pose analysis according to one or more implementations.

[0006] Figure 4 A flowchart illustrates a technique for performing a repetition count of detected motion categories according to one or more embodiments.

[0007] Figure 5 The diagram illustrates a technique for determining motion, progress, and salience scores for multiple motion categories, according to one or more implementations.

[0008] Figure 6 The flowchart illustrates a technique for determining motion data using a common video encoder, according to one or more implementations.

[0009] Figure 7 The diagram illustrates a technique for determining motion, progress, and salience scores for multiple motion categories, according to one or more implementations.

[0010] Figure 8 An exemplary system diagram of an electronic device according to one or more embodiments is shown.

[0011] Figure 9 A simplified multifunctional device according to one or more embodiments is shown in block diagram form. Detailed Implementation

[0012] This disclosure relates to systems, methods, and computer-readable media for exercise tracking and prediction. Generally, the techniques described herein involve capturing image data of a moving subject and predicting an activity being performed by a user in real time. Furthermore, the techniques described herein involve managing repetition counts for the activity being performed and identifying salient frames from image data.

[0013] The embodiments described herein relate to techniques for determining characteristics of user motion captured in image data on a frame-by-frame basis. Specifically, for each candidate action in a set of candidate actions, action prediction, progress prediction, and saliency prediction are determined. In some embodiments, features can be extracted from each frame corresponding to the user's skeleton. Typically, a network can be trained to ingest image data, determine body pose information, such as the position and / or localization information of various parts of the skeleton. Prediction information can be generated by the network, for example, on a frame-by-frame basis, for each user activity in a set of user activities. When the prediction information stabilizes over time, at least one activity in that set of activities can be identified as the activity being performed in the image data.

[0014] According to one or more implementations, image data of a user performing an activity (such as exercise) can be captured. Although the system may not be aware of the activity, it can predict what activities the user is performing while the activity is in progress. Typically, a network can be trained to ingest image data, determine body posture information, and predict the activity being performed based on the body posture information. The network can be trained to predict the activity being performed based on body postures in the current frame and previous frames. The prediction information can be generated by the network, for example, on a frame-by-frame basis for each user activity in a set of user activities.

[0015] The prediction information may include prediction scores. For example, an action prediction score for each candidate action in a set of candidate actions may indicate the likelihood that the user's current movement belongs to a candidate action. A progress prediction score predicts the degree to which a single repetition of the activity is completed for each candidate action. A salience score may indicate the likelihood that, for each candidate action, a frame includes the salient pose of that particular action and is thus classified as a salient frame. That is, a salient frame may be an image data frame in which the relevant pose of the action is presented. Alternatively, the salience score may indicate a measure of progress toward the next salient frame for each candidate action.

[0016] The technique described in this paper improves user motion understanding by performing online activity detection and repetition tracking efficiently and accurately. This allows for the classification and tracking of user movements in real time. Furthermore, the technique allows for the identification of salient frames based on body posture, which can exist at any stage of the motion process.

[0017] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this specification, some of the accompanying drawings of this disclosure are provided in block diagram form to represent structures and apparatuses in order to avoid obscuring the novel aspects of the disclosed embodiments. In this context, it should be understood that references to numbered drawing elements without associated identifiers (e.g., 100) refer to all instances of drawing elements having identifiers (e.g., 100a and 100b). Additionally, as part of this specification, some of the accompanying drawings of this disclosure are provided in the form of flowcharts. The boxes in any particular flowchart may be presented in a particular order. However, it should be understood that a particular flow in any flowchart is merely illustrative of one embodiment. In other embodiments, any of the various components depicted in the flowchart may be omitted, or the components may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not depicted as part of the flowchart. The language used in this disclosure has been chosen primarily for readability and instruction purposes and may not have been chosen to describe or limit the subject matter disclosed. In this disclosure, reference to “an implementation” or “implementation” means a particular feature, structure or characteristic included in at least one implementation in connection with that implementation, and the repeated references to “an implementation” or “implementation” should not be construed as necessarily referring entirely to the same or different implementations.

[0018] It should be understood that in any actual implementation of development (as in any development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals will vary between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who benefit from the image captures provided in this disclosure.

[0019] refer to Figure 1 The diagram illustrates how image data is processed to predict user activities being performed within the image data. Specifically, the image data is captured in the form of input frames 105, including input frames A 105A, B 105B, C 105C, D 105D, and E 105E. According to one or more embodiments, input frames 105 may be captured by an electronic device. In this invention, the electronic device can be any type of device including a camera or other sensors that can detect human posture information. The electronic device capturing the image data may be the same as or different from the electronic device performing the activity prediction.

[0020] In some implementations, each image frame can be applied to the network to predict the body pose present in the image. For example, the body pose can be predicted in the form of 2D pose, 3D pose, etc. The body pose may include, for example, a pose classification, a representative skeleton of the pose, etc. For instance, the body pose of each input frame in input frame 105 can be determined based on an algorithm that takes image data and / or other sensor data of a moving user and predicts the user's pose in a 2D or 3D manner. The pose may include, for example, a pose classification, a geometric representation of the pose, etc. As an example, the pose may include a representation of the joints and / or segments of the user's skeleton.

[0021] Pose information can be used at each frame to determine a prediction associated with the motion being performed. A prediction can be determined for each candidate motion in a set of candidate motions. Each prediction can be based on features of the current pose and features of poses from one or more previous frames. The prediction can be used at each frame, in the form of a motion prediction score, to determine the probability that the motion being performed belongs to each candidate motion in that set of candidate motions. The prediction can also be used to predict the extent to which a single repetition of each candidate motion has been performed by the user, in the form of a progress prediction score. Finally, the prediction score can include a saliency score, which indicates the probability that, for each candidate motion, the frame includes a salient pose for that particular motion.

[0022] According to one or more embodiments, the predicted score can be used to facilitate data presentation in an output frame, such as that displayed on an electronic device. In some embodiments, output frame 110 may be configured to provide information related to user motion, such as detected actions, repetition counts, etc. Specifically, activity data may be presented in output frame 110, which includes output frame A 110A, output frame B 110B, output frame C 110C, output frame D 110D, and output frame E 110E.

[0023] According to one or more implementations, input frame A 105A corresponds to output frame A. Input frame A 105A shows the user standing. Therefore, based on posture, the system may not be able to determine any specific action. Additionally, for illustrative purposes, no repetition is shown, as reflected in output frame A 110A. At input frame 105B, the user is performing a squat. However, because a squat may be associated with multiple actions (such as squats or burpees), the system may not reflect any detected activity in output frame 110B.

[0024] Moving to input frame 105C, the user is performing a push-up. Based on the fact that a squat follows a push-up in input frame A 105B, the system can determine that the user is performing a burpee, but may not have enough confidence in the burpee, for example, if the user has just awkwardly entered the push-up position. Then, in input frame 105D, where the user is performing a slight knee bend, and in input frame 105E, where the user is performing a jump, the user completes the burpee. Therefore, output frame D110D reflects the detected action "burpee". In output frame 110E, because the burpee ends with a jump, the system can determine that the repetition is complete and can increment the repetition count displayed on the user interface.

[0025] According to one or more implementations, the system may also use each frame to present salient frame predictions of salient poses for a specific action. A trainable network can predict action-specific salient poses, identified based on poses detected in the input frames. Each action may have a different number of salient poses. As an example, such as Figure 1 As shown, salient frames 150 include input frame B 105B in which the user is performing a squat, input frame C 105C in which the user is performing a push-up, and input frame E 105E in which the user is performing a jump. These frames and / or frame-related data (such as posture information, predicted values, etc.) may be stored and / or provided to the user for analyzing the quality level of the movement, determining corrections to the movement, etc.

[0026] Go to Figure 2The network considers four potential exercises: squats, lunges, push-ups, and burpees. For each frame 105, an action score 210, a progress score 215, and a salience score are determined. For example, action score 210A depicts the probability that the posture from frame 105A belongs to each of the candidate actions. Thus, as shown, the system determines that the action is slightly more likely to be a squat or a burpee than a push-up or a lunge. The action score is determined on a peripheral basis and is based on the characteristics of the posture in the current frame as well as features from one or more previous frames. Therefore, frame B105B is associated with action score 210B. Here, the action score indicates that the action is unlikely to be a push-up or a lunge, and is somewhat likely to be a squat or a burpee. Moving to frame C105C, the posture is now in a push-up position. Therefore, the corresponding action score in 210C shows a high probability of a burpee, but still some probability of a push-up. For example, the user might awkwardly enter the push-up position. However, the current frame indicates that the action is unlikely to be a squat. At frame D 105D, the action score 210D indicates a high probability of a burpee, while the probability of other actions has decreased. Therefore, at frame D 105D, the system can determine that the action in a series of frames 105 is a burpee. In some implementations, the difference between the action score for a burpee and the next highest action score may be sufficient to determine that the action is ultimately a burpee. Therefore, returning to the output frame D 110D, the action is now identified as a burpee. Figure 2 In frame E 105E, the final pose is a jump. Therefore, the motion score 210E corresponding to frame E 105E depicts a high probability of a burpee and a low probability of another motion.

[0027] According to one or more implementations, a progress score is also predicted for each frame. Progress score 215 indicates the predicted percentage of a single repetition of the corresponding action that has been completed up to that frame. For example, progress score A 215A depicts the probability that the posture from frame A 105A (the user's legs slightly bent on the ground) is in a very early stage of a push-up or lunge. However, progress scores 215A are higher for both burpees and squats. Notably, the progress score for squats is higher than that for burpees because, although both begin in the same way, squats are a shorter-duration action than burpees. Similarly, frame B 105B is associated with progress score 215B. Here, progress score 215B indicates that the progress scores for squats and burpees continue to rise because both include squats. In contrast, the push-up and lunge scores are both negligible because the squat posture in frame B 105B is not associated with either of these actions. Moving to frame C 105C, the posture is now in a push-up position. Therefore, the corresponding progress score 215C shows that when the movement is a push-up, the progress is 0.5. Similarly, when the movement is a burpee, the progress is 0.5. However, since the current posture is not part of a squat or lunge, those progress scores 215C are negligible.

[0028] At frame D 105D, the pose shows a bent leg rising from a squat. Therefore, the corresponding progress score 215D indicates that, in the case of a squat, the squat progress is 0.6 or close to completion. Similarly, in the case of a burpee, the burpee progress is 0.8, slightly higher than a squat, because the burpee motion lasts longer. However, the current pose is not part of a push-up or lunge, so those progress scores 215D are negligible. Finally, at frame E 105E, the progress score 215E indicates that the burpee motion has been completed. However, the network has determined that the pose is not part of a push-up, squat, or lunge, so these progress scores are negligible. Return to Figure 1 Since the motion score for a series of frames 105 has been identified as a burpee, and the progress score for the burpee indicates that the repetition has been completed, the repetition count is incremented, and the current count is updated at output frame 110E to show a repetition count of 1.

[0029] Return to Figure 2A saliency score can be determined for each frame in a series of 105 frames. The saliency score indicates the probability that the pose in the frame is a salient pose for each candidate action. For example, a network can be trained to detect different salient poses for various candidate actions. Therefore, salient poses are action-specific. Furthermore, each candidate action can be associated with a different number of salient poses. Salient poses can be identified at any time during the repetition of the motion. Additionally, because saliency is determined based on pose rather than progress, salient poses are not limited to the beginning or end of the repetition, or the midpoint defined by the beginning and end.

[0030] For example, salience score A 220A describes the probability that the posture from frame A 105A (user's legs slightly bent on the ground) indicates the likelihood that the posture is considered a salient posture for each of the following: push-ups, squats, lunges, and burpees. In this example, the slight bend in the knees is not associated with a high probability of being a salient posture for any of the candidate movements. However, for squats and burpees, the salience score is slightly higher because a slight bend in the legs is at least part of the movement for squats and burpees. Moving to frame B 105B, the posture is associated with salience score 220B. Here, salience score 220B indicates that the squat posture in frame B 105B is more likely to be a salient posture for squats and burpees than for push-ups and lunges.

[0031] Moving to frame C 105C, the posture is now a push-up. Therefore, the corresponding salience score 220C shows that the salience score is very high when the action is a push-up. Similarly, the salience score is very high when the action is a burpee, because both push-ups and burpees involve push-ups, and the frame shows the object in the lowest position of a push-up. However, the current posture is not part of a squat or lunge, so those salience scores 220C are negligible.

[0032] At frame D 105D, the posture shows a bent leg rising from a squat. Therefore, the corresponding significance score 220D indicates that the significance score is lower when the movement is a squat or burpee. Conversely, those significance scores 220D are negligible when the movement is a push-up or lunge. Finally, at frame E 105E, the posture is part of a jump. Therefore, the significance score 220E is higher for a burpee but lower for other movements because these other movements do not involve jumping.

[0033] According to one or more implementations, saliency scores can be used in conjunction with action scores to determine salient frames for an action. For example, although frame B 105B shows a salient frame for a squat, and frame C 105C shows a salient frame for a push-up, action score 210 indicates that the detected action is a burpee. Therefore, based on the saliency score for the burpee action, a set of salient frames includes frames B 105B, C 105C, and E 105E. Salient frames can be identified based on frames with saliency scores above a predefined saliency threshold, based on the peak saliency score throughout the action, etc. Return to Figure 1 Since the action scores for a series of frames 105 have been identified as burpees, and the salience scores for the burpees indicate that frames 105B, 105C, and 105E are salience frames, those frames are stored or provided as salience frames 150.

[0034] Figure 3 A flowchart illustrates techniques for detecting salient frames and performing pose analysis according to one or more implementations. For illustrative purposes, the steps are described in the context of specific components. However, it should be understood that various actions can be performed by alternative components. As an example, a single system can perform actions relative to… Figure 3 All actions described. Alternatively, individual components can perform functions, and functionality can be distributed across multiple systems or devices. Furthermore, various actions can be performed in different orders. Additionally, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0035] Flowchart 300 begins at box 305, where image data for the current frame of a moving subject is acquired. According to one or more embodiments, the subject can be a user or other person in the environment for whom image data and / or other sensor data are collected. According to one or more embodiments, the image data can be captured by an electronic device. The electronic device can be any type of device including a camera or other sensor that can detect the person's posture information. The electronic device capturing the image data can be the same as or different from the electronic device performing activity prediction.

[0036] Flowchart 300 proceeds to box 310, where pose features are obtained from the image data. In some embodiments, body tracking is performed by an algorithm that takes image data and / or other sensor data of a moving user and predicts the user's pose in a 2D or 3D manner. This pose prediction may include, for example, the type or classification of a particular pose, a geometric representation of the pose, etc. As an example, the pose may include a representation of the joints and / or segments of the user's skeleton. According to one or more embodiments, a spatial transformer may be used to extract features from the pose.

[0037] At box 315, action, progress, and saliency prediction scores are determined for the current frame based on pose features in the current and previous frames. These action, progress, and saliency scores can be determined from one or more networks or other modules configured to predict or provide classification information for frames based on pose features. In some embodiments, an action network can be trained to predict probability scores for specific candidate actions (such as predefined movements, exercises, etc.). Alternatively, in some embodiments, a single network can be trained to predict action scores for multiple candidate actions. Similarly, a progress prediction network can be trained to predict the degree to which a repetition of a specific action has been completed at a given frame. Additionally, a saliency network can be trained to predict saliency scores for each frame of a specific candidate action, or a single network can be trained to predict saliency scores for multiple candidate actions. The saliency network can be trained based on predefined poses for each candidate action, and each candidate action can be associated with a different number of saliency poses. In some embodiments, the action network, progress prediction network, and / or saliency network can be embodied in a computation module configured to provide corresponding outputs based on pose features in the current and previous frames. At box 320, a determination of the repetition progress is made based on the progress score. This may include, for example, predicting the extent to which a single repetition has been completed for one or more candidate action types. At box 325, a determination is made regarding whether the repetition is complete. For example, this determination may occur when a repetition score for one of the candidate actions exceeds a threshold progress score. As another example, this determination may be based first on a threshold high repetition score for a particular action, and then on a threshold low repetition score, which indicates that the action has ended and is being repeated. If the repetition is determined to be complete at box 325, the flowchart proceeds to box 330.

[0038] At box 330, frames are categorized based on actions that are repeatedly completed, triggered by a progress score. That is, actions that meet a threshold are used to categorize frames. Optionally, once an action is identified, the action used for categorization can be provided to the user, for example, as part of an output frame. This can occur before or after the repeated completion.

[0039] Flowchart 300 proceeds to box 335, where the repetition count is incremented for the action. In some implementations, the repetition count may be stored and / or presented to the user. In one example, the repetition score may be presented on the user interface, for example, as part of an output frame, and displayed along with the determined action. The value of the repetition count may be incremented.

[0040] Flowchart 300 continues to box 340, where significant frames are identified for completed repetitions. In some implementations, significant frames may be identified based on the salience score of frames belonging to a specific repetition and classified action. In other words, the salience score associated with the classified action is analyzed for frames associated with a repetition of a classified action. Figure 2 In the example, frame D 105D can be classified as a burpee. Then, the salience scores for burpees in frames 105A, 105B, 105C, and 105D are analyzed to identify salient frames. Therefore, frames B 105B and C 105C can be identified as salient frames based on their high salience scores. Frame E 105E will be similarly classified as a salient frame because its salience score is high once it is captured and identified as part of a burpee repetition. Frames associated with salience scores that meet a threshold for a classified action can be identified as salient frames. In some implementations, progress scores can be used to identify the start and end of a specific action. These frames can also be additionally or alternatively considered as salient frames.

[0041] Return to Figure 3 Optionally, at box 345, pose analysis is performed. According to one or more embodiments, pose analysis may involve comparing the pose of a salient frame with the target pose of the salient frame. For example, the pose in the salient frame may be compared with a predefined salient frame to identify correction actions or other parameters associated with the difference between the two.

[0042] The flowchart continues to box 350, where a determination is made regarding whether an additional frame has been received. Alternatively, it returns to box 325; if no complete repetition is identified, flowchart 300 also proceeds to box 350. If an additional frame is received, flowchart 300 returns to box 310, and pose features are obtained from the additionally received image data. In other words, this process occurs in real-time as new frames are captured.

[0043] Returning to box 350, if no additional frame is received, flowchart 300 terminates at box 355. At box 355, a result related to the action is provided, which is associated with the set of frames. In some embodiments, action data may be performed from output frames that can be presented to the user as data for the interface. According to some embodiments, providing action results may include: at box 360, providing salient frames for the action, such as salient frames identified from each repetition. In some embodiments, salient frames may be provided for display and / or may be stored for later viewing by the user.

[0044] Additionally, optionally, providing motion data at box 365 may include providing pose analysis. In some embodiments, pose analysis may include data determined at box 345. Furthermore, pose analysis may be provided in the form of a user interface that provides data on the user's pose in a salient frame compared to a target pose. Additionally, in some embodiments, pose analysis may be provided in the form of stored raw or filtered pose data for analysis.

[0045] As described above, in some implementations, action scores, schedule scores, and significance scores can be determined simultaneously during runtime. Therefore, Figure 4 A flowchart illustrates a technique for performing a repetition count of detected motion categories according to one or more embodiments. For illustrative purposes, the steps are described in the context of specific components. However, it should be understood that various actions can be performed by alternative components. Furthermore, various actions can be performed in different orders. Additionally, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0046] Flowchart 400 begins at box 405, where pose features are obtained from the current pose and features from previous frames. In some embodiments, body tracking is performed by an algorithm that takes image data and / or other sensor data of a user in motion and predicts the user's pose in a 2D or 3D manner. These pose features may include or indicate, for example, a pose classification, a geometric representation of the pose, etc. As an example, the pose may include a representation of the joints and / or segments of the user's skeleton. In some embodiments, these pose features may be a representation of a pose detected by body tracking and provided in a manner that allows them to be ingested by one or more models for predicting ongoing motion. In some embodiments, additional processing may be performed to incorporate features from one or more previous frames. For example, at least some features from one or more previous frames may be juxtaposed or otherwise incorporated into the pose features. As another example, as will be described below with respect to... Figure 5 In more detail, a gated recurrent unit (GRU) or other mechanism can be configured to augment pose features from the current frame using hidden states or other data from previous frames.

[0047] Flowchart 400 proceeds to box 410, where frame scores are determined. In one or more embodiments, multiple scores are determined for each frame. For example, at box 415, an action score is determined for each candidate action in a set of candidate actions. Action scores can be determined by applying pose features to an action network configured to predict the probability that the pose and the current frame correspond to each candidate action in a set of candidate actions. For example, the action network can provide action scores in the form of percentages or values ​​between zero and one, corresponding to the probability of each candidate action in a set of candidate actions. Determining frame scores may also include, at box 420, a progress prediction score for each action in the set of candidate actions. Progress prediction scores can be determined, for example, by applying pose features to a progress network configured to predict the extent to which a single repetition is performed by the object based on the pose features. Progress prediction scores can be represented in the form of values ​​between zero and one, indicating the percentage of the predicted single repetition of the corresponding action based on the pose features that is completed. Determining frame scores may also include, at box 425, a salience score for each action in a set of candidate actions. As described above, one or more networks (such as a saliency network or other programming modules) can be configured to predict the probability that a given set of pose features corresponds to a salient pose for each candidate action in a set of candidate actions. Therefore, a saliency score is determined for each candidate action and indicates the probability that the current frame presents a salient pose. In some implementations, the saliency network can be trained based on a predefined pose for each candidate action, and each candidate action can be associated with a different number of salient poses.

[0048] Flowchart 400 proceeds to box 430, where a determination is made regarding whether the action score meets a threshold. According to some embodiments, this threshold may be a predefined action score, which, when exceeded, indicates that the associated action corresponds to the set of frames. As another example, the threshold may be a threshold difference between the probability of the most likely action in the set of candidate actions corresponding to the action score and the probability of the second most likely action in the set of candidate actions. If a determination is made at box 430 that the action score does not meet the threshold, flowchart 400 proceeds to box 465, and a determination is made regarding whether an additional frame is received. Alternatively, if a determination is made at box 430 that the action score for a specific action meets the threshold, flowchart 400 proceeds to box 435. At box 435, the motion is classified as a specific action. That is, candidate actions with action scores determined to meet the threshold are identified as the current action being performed by the user's movement. In some embodiments, once the motion is classified as a specific action, a user notification of the action can be provided, as shown in optional box 440. For example, the user interface can be updated, or audio or visual cues can be provided to indicate the recognized actions.

[0049] Flowchart 400 proceeds to box 445, where, for example, based on the progress prediction for a specific action from box 420, a determination is made regarding whether the repetition of the action has been completed. According to one or more embodiments, the completion of the repetition is determined based on the progress prediction value for a specific action. For example, if the progress prediction value is close to or reaches a maximum value, such as 1, and then drops to or near a minimum value, such as 0, the system can detect that the repetition for the specific action has been completed. If it is determined that the repetition has not been completed, the flowchart proceeds to box 465, making a determination regarding whether an additional frame has been received.

[0050] If the repetition of a specific action is completed at box 445, the flowchart proceeds to box 450. At box 450, frames marking the start and end of the repetition are identified. According to some embodiments, frames marking the start and end of the repetition may be determined based on a progress prediction score for the frames. At box 455, salient frames are identified for the specific action. In some embodiments, salient frames may be determined based on: the salient scores of a set of frames between the frames identified as the start and end of the repetition, and the salient scores for that specific action and for those frames (e.g., as determined at box 425). In some embodiments, salient frames may be determined based on the local maximum salient score within the repetition. As another example, salient frames may be determined based on a threshold salient score. In some embodiments, the technique used to determine salient frames may be action-specific. For example, different actions may have different numbers of salient poses. The technique used to identify salient frames may thus involve determining the number of salient frames corresponding to salient poses.

[0051] Flowchart 400 proceeds to box 460 and increments the repetition count for a specific action. If the repetition count is presented to the user, for example, as a user interface overlay, the data presented in that overlay can be updated to reflect the incrementing repetition count. Flowchart 400 then proceeds to box 465. A determination can be made regarding whether any additional frames have been received; if so, the flowchart returns to box 405. At box 405, the pose characteristics are obtained, thereby allowing the application of the procedures described in boxes 410 through 465.

[0052] Figure 5 A flowchart illustrates a technique for determining motion, progress, and salience scores for multiple motion categories, according to one or more implementations. The flowchart depicts a specific technique that can be used for motion prediction and salience frame identification.

[0053] The flowchart 500 begins by collecting frame data 505. In some embodiments, the image data may be 2D or 3D image data capturing an object performing motion. Frame data 505 may be applied to body tracking component 510. In some embodiments, body tracking is performed by an algorithm that takes frame data 505 and predicts the pose of the object in the frame in a 2D or 3D manner. The pose may include, for example, a pose classification, a geometric representation of the pose, etc. As an example, the pose may include a representation of the joints and / or segments of a user's skeleton. In some embodiments, these pose features may be a representation of the pose detected by body tracking and provided in a manner that can be ingested by one or more models for predicting ongoing motion (e.g., as input pose 515). In some embodiments, input pose 515 is applied to spatial transformer 520 to extract pose features (X). T Pose features can be extracted on a per-frame basis.

[0054] According to one or more embodiments, the gated loop unit (GRU) 525 can be configured to select the current feature (X) T ) and past hidden state (H T-1 ) merge to obtain the current hidden state (H) T Therefore, the current hidden state can be derived from the pose features from the current frame and the pose features from one or more previous frames.

[0055] The hidden state can then be passed to three separate networks. These networks can take the form of various types of neural networks. In one example, each network could be a multilayer perceptron (MLP). Therefore, the hidden state can be applied to an action head 530, a progress head 540, and a saliency head 550. The action head can be configured to predict the pose and the probability that the current frame corresponds to each candidate action in a set of candidate actions based on the current hidden state. Therefore, the output of action head 530 can be an action score 535 for each candidate action.

[0056] Progress head 540 can be a progress prediction score determined for each action in the set of candidate actions. The progress prediction score can be determined, for example, by applying hidden states to progress head 540, which is configured to predict the degree to which a single repetition is performed by the object for each candidate action in the set. Therefore, the output of progress head 540 is a progress prediction 545 for each candidate action.

[0057] The saliency head 550 can be configured to predict a saliency score for a given frame for each action in the group of candidate actions. Specifically, the saliency head 550 can be configured to predict the probability that the current frame contains a salient pose for each candidate action in the group of candidate actions. Alternatively, the saliency head 550 can be configured to predict the progress toward the next salient pose based on the pose features of the current frame. Therefore, the output of the saliency head 550 is a saliency score 555 for each candidate action.

[0058] According to one or more embodiments, action scores are used to predict the current action being performed. When determining the current action being performed based on the action score 535 for each candidate action, the current action can be used to select a relevant progress score for a frame via progress selection 560 (e.g., based on a progress score corresponding to the same current action). Similarly, the current action can be used to select a relevant saliency score for a frame via saliency selection 565 (e.g., based on a saliency score corresponding to the same current action).

[0059] According to some implementations, a unified video encoder can be used to generate video features from input image data to determine different predictions, such as motion, progress, and / or salient frames. The unified video encoder can be specifically trained to generate an integrated feature set that satisfies multiple downstream uses. For example, the unified video encoder can be trained to generate a feature set that can be used for predictions related to motion, progress, and / or salient frames, allowing prediction data to be determined in parallel without relying on dependencies between models, thereby introducing robustness across different prediction heads.

[0060] Figure 6 A flowchart illustrates a technique for predicting motion data using a unified video encoder, according to one or more embodiments. For illustrative purposes, the steps are described in the context of specific components. However, it should be understood that various actions can be performed by alternative components. Furthermore, various actions can be performed in different orders. Additionally, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0061] Flowchart 600 begins at box 605, where image data for the current frame of a moving subject is acquired. According to one or more embodiments, the subject can be a user or other person in the environment for whom image data and / or other sensor data are collected. According to one or more embodiments, the image data can be captured by an electronic device. The electronic device can be any type of device including a camera or other sensor that can detect the person's posture information. The electronic device capturing the image data can be the same as or different from the electronic device performing activity prediction.

[0062] At box 610, image data is applied to a unified video encoder configured to acquire video features. The unified video encoder can be pre-trained using a combination of techniques to generate features with different functionalities that can be used downstream. For example, the unified video encoder can be trained using a combination of sparse and dense input information, such that the resulting feature set can be used for predictions that rely on both sparse and dense understanding. In some implementations, the unified video encoder processes streaming video in real time, tokenizes each frame, and passes the tokens through multiple transformer layers to extract rich context-aware features.

[0063] The flowchart proceeds to box 615, where video features are adjusted based on features from previous frames. For example, historical features from previous frames can be combined with features from the current frame to generate adjusted features. As will be described below, a gated recurrent unit (GRU) can be configured to fuse current features with hidden states from past frames to obtain adjusted features for the frame.

[0064] At box 620, motion data is determined based on adjusted video features. In one or more embodiments, multiple scores are determined for each frame. According to one or more embodiments, because adjusted features are generated for processing multiple predictions, various predictions can be performed in parallel or simultaneously. For example, at box 625, a motion score is determined for each candidate motion in a set of candidate motions. Motion scores can be determined by applying adjusted video features to a motion network configured to predict the likelihood of a user performing one or more gestures in the current frame. For example, the motion network can provide motion scores in the form of percentages or values ​​between zero and one, corresponding to the likelihood of each candidate motion in a set of candidate motions. In some embodiments, the motion data may also include a progress score for a specific frame. Determining the motion data may also include, at box 630, a progress prediction score for each motion in the set of candidate motions. Progress prediction scores can be determined, for example, by applying adjusted video features to a progress network configured to predict the degree to which a single repetition is performed by the object. The progress prediction score can be represented as a value from zero to one, indicating the percentage of a single repetition of the corresponding action predicted based on pose features that is completed. Action data determination may also include, at box 635, the saliency score for each action in a set of candidate actions. As described above, one or more networks (such as a saliency network or other programming modules) can be configured to predict the probability that a given set of adjusted video features corresponds to a frame with a salient pose including each candidate action in a set of candidate actions.

[0065] The flowchart proceeds to box 640, where the result of the action data is provided. In some embodiments, the action data may be provided as data for the interface from an output frame that can be presented to the user. According to some embodiments, providing the action result may include providing a salient frame for the action, such as a salient frame identified from each repetition. In some embodiments, the salient frame may be provided for display and / or stored for the user to view later. Alternatively, the action data may be provided to a client application, which may use the action data for further processing. A determination is made at box 645 regarding whether any additional frames have been received. If no additional frames have been received, the flowchart terminates. If additional frames have been received, the flowchart returns to box 605 and processes the next frame.

[0066] Figure 7 An example technique for determining motion, progress, and salience scores for multiple motion categories, according to one or more implementations, is illustrated in flowchart form. The flowchart depicts a specific technique that can be used for motion prediction and salience frame identification, for example, as described above relative to... Figure 6 As described.

[0067] The flowchart 700 begins by collecting frame data 705. In some implementations, the frame data may include image frames capturing objects performing motion. Frame data 705 may be applied to a unified video encoder 710. The unified video encoder 710 may be a self-supervised, vision transformer-based encoder that has been pre-trained using pixel-level view-invariant targets and global cross-modal alignment targets. Therefore, the encoder can provide dense and semantically rich token embeddings in the form of video features 715, which preserve contextual information from the frames, as well as geometric tasks such as 3D pose data. Video features can be extracted on a per-frame basis, which are shown as (X... T ).

[0068] According to one or more embodiments, the gated loop unit (GRU) 725 can be configured to store the current feature (X) T ) and past hidden state (H T-1 ) merge to obtain the current hidden state (H) T Therefore, the current hidden state can be obtained from the video features (X) of the current frame. T ) and video features derived from one or more previous frames.

[0069] Then, the hidden state (H) can be... TThe hidden state is passed to multiple networks or models, such as neural networks. In one example, the networks may each be in the form of a multilayer perceptron (MLP). Therefore, the hidden state can be applied to an action head 730, a progress head 740, and a saliency head 750. The action head can be configured to predict the pose and the probability that the current frame corresponds to each candidate action in a set of candidate actions based on the current hidden state. Therefore, the output of action head 730 can be an action score 735 for each candidate action.

[0070] Progress head 740 can be a progress prediction score determined for each action in the set of candidate actions. The progress prediction score can be determined, for example, by applying hidden states to progress head 740, which is configured to predict the degree to which a single repetition is performed by the object for each candidate action in the set. Therefore, the output of progress head 740 is a progress prediction 745 for each candidate action.

[0071] The saliency head 750 can be configured to predict a saliency score for a given frame for each action in the group of candidate actions. Specifically, the saliency head 750 can be configured to predict the probability that the current frame contains a salient pose for each candidate action in the group of candidate actions. Alternatively, the saliency head 750 can be configured to predict the progress toward the next salient pose based on the pose features of the current frame. Therefore, the output of the saliency head 750 is a saliency score 755 for each candidate action.

[0072] Because all three estimates come from a common feature set, predictions for each of the actions, progress, and salience can be determined without depending on each other. Therefore, in the event that any particular prediction fails, valid prediction data from other models can be obtained.

[0073] See Figure 8 It depicts a simplified block diagram of an electronic device 800 according to one or more embodiments of the present disclosure. The electronic device 800 may be part of a multifunctional device, such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, or any other electronic device including a camera system. Figure 8 A block diagram illustrates an overall view of a system capable of supporting proximity detection and breach control according to one or more embodiments. Electronic device 800 can be connected to other network devices, such as mobile devices, tablets, desktop devices, and network storage devices (such as servers), via a network interface. In some embodiments, electronic device 800 can communicatively connect to other electronic devices via a local network to share sensor data and other information.

[0074] Electronic device 800 may include one or more processors 830, such as a central processing unit (CPU). Processor 830 may be a system-on-a-chip (SoC) such as those present in mobile devices, and may include one or more dedicated graphics processing units (GPUs). Alternatively, processor 830 may include multiple processors of the same or different types. Electronic device 800 may also include memory 840. Memory 840 may include one or more different types of memory that can be used in conjunction with processor 830 to perform device functions. For example, memory 840 may include cache, ROM, and / or RAM. Memory 840 may store various programming modules during execution, including application module 865, body tracking module 870, and motion estimation module 875. According to some embodiments, application 865 may provide activity-based tracking and feedback to a user. As an example, the application may include a health application, a workout application, or other applications that utilize the functionality of predicting and tracking user activity. Body tracking module 870 may utilize data from camera 810 and / or sensor 860 (such as a proximity sensor) used to collect sensor data of a person performing a movement or activity, from which body posture can be derived. For example, body tracking module 870 may utilize a body tracking pipeline to predict the skeleton or other representation of a body in image data. Motion estimation module may utilize a network trained to generate predictions based on characteristics of the outcome of one or more activities, using current and previous pose information. For example, motion estimation module 875 may include functionality for predicting, using body tracking data, the current progress of a set of candidate activities, the duration of the candidate activities, and salient frames for each candidate activity. Electronic device may include one or more storage devices 850 for storing data to facilitate processing of application 865, body tracking module 870, and / or motion estimation module 875.

[0075] Electronic device 800 may also include one or more cameras 810. Each camera 810 may include an image sensor, a lens stack, and other components for capturing images. In one or more embodiments, the cameras may be pointed in different directions within the electronic device. For example, a front-facing camera may be positioned in or on a first surface of the electronic device 800, while a rear-facing camera may be positioned in or on a second surface of the electronic device 800. In some embodiments, camera 810 may include one or more types of cameras, such as RGB cameras, depth cameras, etc. Electronic device 800 may include one or more sensors 860 for detecting physical obstacles in the environment. Examples of sensors 860 include LIDAR, etc.

[0076] In one or more embodiments, the electronic device 800 may further include a display 880. The display 880 may be any type of display device, such as an LCD (liquid crystal display), an LED (light-emitting diode) display, an OLED (organic light-emitting diode) display, etc. Furthermore, the display 880 may be a semi-transparent display, such as a head-up display, a through-screen display, etc. The display 880 may display content associated with the application 865.

[0077] Although the electronic device 800 is described as including the numerous components described above, in one or more embodiments, the various components may be distributed across multiple devices. Additionally, additional components may be used, and any combination of the functionality of any components may be possible.

[0078] Now for reference Figure 9 This diagram illustrates a simplified functional block diagram of an exemplary multi-functional electronic device 900 according to one embodiment. The multi-functional electronic device 900 may include a processor 905, a display 910, a user interface 915, graphics hardware 920, sensors 925 (e.g., proximity / ambient light sensors, accelerometers, and / or gyroscopes), a microphone 930, an audio codec 935, a speaker 940, communication circuitry 945, digital image capture circuitry 950 (e.g., including a camera system), a video codec 955 (e.g., supporting a digital image capture unit), a memory 960, a storage device 965, and a communication bus 970. For example, the multi-functional electronic device 900 may be a digital camera or a personal electronic device, such as a personal media player, mobile phone, head-mounted device, or tablet computer.

[0079] Processor 905 executes instructions necessary for implementing or controlling the operation of many functions performed by device 900 (e.g., the generation and / or processing of images disclosed herein). Processor 905 may, for example, drive display 910 and may receive user input from user interface 915. User interface 915 allows a user to interact with device 900. For example, user interface 915 may take various forms, such as buttons, keypad, dial pad, click wheel, keyboard, display screen, and / or touchscreen. Processor 905 may also be, for example, a system-on-a-chip, such as those present in mobile devices, and includes a dedicated graphics processing unit (GPU). Processor 905 may be based on a Reduced Instruction Set Computer (RISC) or Complex Instruction Set Computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. Graphics hardware 920 may be dedicated computing hardware for processing graphics and / or assisting processor 905 in processing graphics information. In one embodiment, graphics hardware 920 may include a programmable GPU.

[0080] Image capture circuitry 950 may include two (or more) lens assemblies 980A and 980B, each lens assembly having a separate focal length. For example, lens assembly 980A may have a shorter focal length than lens assembly 980B. Each lens assembly may have a separate associated sensor element 990A and associated sensor element 990B. Alternatively, two or more lens assemblies may share a common sensor element. Image capture circuitry 950 may capture still images and / or video images. Output from image capture circuitry 950 may be processed at least in part by a video codec 955, and / or a processor 905 and / or graphics hardware 920, and / or a dedicated image processing unit or pipeline incorporated within circuitry 950. Images thus captured may be stored in memory 960 and / or storage device 965.

[0081] The sensor and camera circuitry 950 can capture still images and video images that can be processed at least in part by the following devices according to this disclosure: a video codec 955, and / or a processor 905 and / or graphics hardware 920, and / or a dedicated image processing unit incorporated within the circuitry 950. Images thus captured can be stored in a memory 960 and / or a storage device 965. The memory 960 may include one or more different types of media used by the processor 905 and graphics hardware 920 to perform device functions. For example, the memory 960 may include a memory cache, read-only memory (ROM), and / or random access memory (RAM). The storage device 965 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device configuration file information, and any other suitable data. Storage device 965 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and magnetic tapes, optical media (such as CD-ROMs and digital video optical discs (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). Memory 960 and storage device 965 can be used to tangibly hold computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 905, such computer program code can implement one or more of the methods described herein.

[0082] The scope of the subject matter disclosed herein should be determined by reference to the appended claims and their equivalents. In the appended claims, the terms “including” and “in which” are used as common English equivalents to the corresponding terms “comprising” and “wherein”.

Claims

1. A method, the method comprising: A series of frames are captured of a user performing motion, the series of frames including a first frame, a second frame, and a third frame, wherein the second frame is captured between the first frame and the third frame; Based on the features of the first frame and the features of the second frame, a first progress prediction score and a first salience score are determined for the second frame; as well as In response to determining that the first progress prediction score for the first candidate action meets the repeatability criterion: Identify a set of repeating frames. Detect one or more salient frames based on the salientity scores for the set of frames, and Based on the features of the first frame and the features of the second frame, a second progress prediction score and a second saliency score are determined for the third frame.

2. The method of claim 1, wherein the saliency score indicates the likelihood that the current frame captures a salient pose of a particular candidate action.

3. The method of claim 2, wherein the first candidate action is a candidate action in a set of candidate actions, wherein the first candidate action is associated with a first number of significant frames in each repetition, and wherein the second candidate action in the set of candidate actions is associated with a second number of significant frames in each repetition, the second number of significant frames being different from the first number of significant frames.

4. The method according to claim 1, further comprising: The characteristics of the motion are determined based on the user's posture in the second frame, since the second frame is identified as a salient frame.

5. The method according to claim 1, further comprising: Based on the features of the first frame and the features of the second frame, determine the action prediction score associated with the first candidate action for the second frame.

6. The method according to claim 1, further comprising: In response to determining the repetition completion of the motion: Increment the repetition count, and A notification is displayed showing the repeat count.

7. The method of claim 1, wherein determining the first action prediction score comprises: The features of the second frame are applied to the gated loop unit to obtain input values ​​for selecting at least one from the group consisting of an action network, a progress network, and a saliency network.

8. A non-transitory computer-readable medium comprising computer-readable code, said computer-readable code being executable by a processor to: A series of frames are captured of a user performing motion, the series of frames including a first frame, a second frame, and a third frame, wherein the second frame is captured between the first frame and the third frame; Based on the features of the first frame and the features of the second frame, a first progress prediction score and a first salience score are determined for the second frame; as well as In response to determining that the first progress prediction score for the first candidate action meets the repeatability criterion: Identify a set of repeating frames. Detect one or more salient frames based on the salientity scores for the set of frames, and Based on the features of the first frame and the features of the second frame, a second progress prediction score and a second saliency score are determined for the third frame.

9. The non-transient computer-readable medium of claim 8, wherein the saliency score indicates the likelihood that a particular candidate action will be captured in the current frame.

10. The non-transitory computer-readable medium of claim 9, wherein the first candidate action is a candidate action in a set of candidate actions, wherein the first candidate action is associated with a first number of significant frames in each repetition, and wherein the second candidate action in the set of candidate actions is associated with a second number of significant frames in each repetition, the second number of significant frames being different from the first number of significant frames.

11. The non-transitory computer-readable medium of claim 10, further comprising computer-readable code for the following operations: The characteristics of the motion are determined based on the user's posture in the second frame, since the second frame is identified as a salient frame.

12. The non-transitory computer-readable medium of claim 10, further comprising computer-readable code for the following operations: Based on the features of the first frame and the features of the second frame, determine the action prediction score associated with the first candidate action for the second frame.

13. The non-transitory computer-readable medium of claim 10, further comprising computer-readable code for performing the following operations in response to determining the repetition completion of the motion: Increment the repetition count, and A notification is displayed showing the repeat count.

14. The non-transitory computer-readable medium of claim 13, wherein the computer-readable code for determining the first action prediction score comprises computer-readable code for: The features of the second frame are applied to the gated loop unit to obtain input values ​​for selecting at least one from the group consisting of an action network, a progress network, and a saliency network.

15. A system comprising: One or more processors; as well as One or more computer-readable media, the one or more computer-readable media including computer-readable code, the computer-readable code being executable by the processor to: A series of frames are captured of a user performing motion, the series of frames including a first frame, a second frame, and a third frame, wherein the second frame is captured between the first frame and the third frame; Based on the features of the first frame and the features of the second frame, a first progress prediction score and a first salience score are determined for the second frame; as well as In response to determining that the first progress prediction score for the first candidate action meets the repeatability criterion: Identify a set of repeating frames. Detect one or more salient frames based on the salientity scores for the set of frames, and Based on the features of the first frame and the features of the second frame, a second progress prediction score and a second saliency score are determined for the third frame.

16. The system of claim 15, wherein the saliency score indicates the likelihood that a particular candidate action will be captured in the current frame.

17. The system of claim 16, wherein the first candidate action is a candidate action in a set of candidate actions, wherein the first candidate action is associated with a first number of significant frames in each repetition, and wherein the second candidate action in the set of candidate actions is associated with a second number of significant frames in each repetition, the second number of significant frames being different from the first number of significant frames.

18. The system of claim 17, further comprising computer-readable code for the following operations: The characteristics of the motion are determined based on the user's posture in the second frame, since the second frame is identified as a salient frame.

19. The system of claim 17, further comprising computer-readable code for the following operations: Based on the features of the first frame and the features of the second frame, determine the action prediction score associated with the first candidate action for the second frame.

20. The system of claim 17, further comprising computer-readable code for performing the following operations in response to determining the repetition completion of the motion: Increment the repetition count, and A notification is displayed showing the repeat count.

Images