Action scoring method and apparatus, action scoring device, and storage medium

Abstract

The application provides a motion scoring method and device, motion scoring equipment and a storage medium. The method comprises: determining the key point coordinates of a target image of a first time frame through a posture recognition algorithm; calculating a first joint angle according to the key point coordinates of the target image of the first time frame; determining whether the first joint angle is within a reasonable range according to the first joint angle and a standard joint angle; if the first joint angle is within the reasonable range, determining whether the matched time frames have reached a preset number; if the matched time frames have reached the preset number, scoring the motion of the first time frame according to the first joint angle and the standard joint angle; the application scores the motion to be scored by judging the joint angle and the number of time frames, and no human intervention is required throughout, thereby reducing labor costs. Furthermore, the motion is scored again when both the above two dimensions are within the reasonable range, so that the motion to be scored is within the reasonable range, and the effectiveness and accuracy of the scoring are improved.

Description

Technical Field

[0001] This application relates to the field of movement training, and more specifically, to a movement scoring method, apparatus, movement scoring device, and storage medium. Background Technology

[0002] With the increasing prevalence of musculoskeletal disorders, the demand for musculoskeletal rehabilitation is also growing. Currently, musculoskeletal pain is mostly relieved and resolved through rehabilitation training. However, rehabilitation training typically relies on the supervision and guidance of a physical therapist, who assesses the trainee's progress through observation and scoring. Human guidance and supervision often depend on experience and are also affected by objective factors, leading to inconsistent scoring standards and low effectiveness and accuracy of the assessments. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a motion scoring method, apparatus, device, and storage medium, which can improve the effectiveness and accuracy of motion scoring.

[0004] In a first aspect, embodiments of this application provide an action scoring method, comprising: determining the key point coordinates of a target image in a first time frame using a pose recognition algorithm, wherein the target image is an image of an action to be scored; calculating a first joint angle based on the key point coordinates of the target image in the first time frame; determining whether the first joint angle is within a reasonable range based on the first joint angle and a standard joint angle; if the first joint angle is within a reasonable range, determining whether the number of frames matched in the target image has reached a preset number; if the number of frames matched in the target image has reached a preset number, scoring the action in the first time frame based on the first joint angle and the standard joint angle; wherein the number of frames matched in the target image is the number of frames in which the first joint angle is within a reasonable range.

[0005] In the above implementation process, the first step is to determine whether the first joint angle of the target image is within a reasonable range to ascertain whether the action to be scored in the target image meets the requirements. Secondly, it is determined whether the number of frames within a reasonable range for the first joint angle is within a preset range to ascertain whether the acquired action information to be scored is sufficient. By judging the action based on both action specifications and the quantity of action information, the objectivity and accuracy of the scoring are ensured, thus improving the accuracy of the scoring.

[0006] In one embodiment, the method further includes: if the number of frames matched in the target image has not reached a preset number, determining whether the distance between the key points of the target image in the second time frame and the key points of the target image in the first time frame is less than a distance threshold; if the distance is less than the distance threshold, calculating a second joint angle based on the key point coordinates of the target image in the second time frame; and scoring the action in the second time frame based on the second joint angle and the standard joint angle.

[0007] In the above implementation process, when the number of matched frames of the target image has not reached the preset number, the stability of the action to be scored is determined by judging the range of key points in two adjacent time frames. When the stability of the action to be scored meets the standard, the action to be scored is scored, ensuring that the action in the scoring time frame is stable and effective, thus improving the effectiveness and accuracy of the scoring.

[0008] In one embodiment, the distance threshold includes a first distance threshold and a second distance threshold. Determining whether the distance between the key points of the target image in the second time frame and the key points of the target image in the first time frame is less than the distance threshold includes: determining whether the distance is less than the second distance threshold; if the distance is less than the second distance threshold, assigning a weight to the second time frame according to the relationship between the distance and the first distance threshold and the second distance threshold; if the distance is greater than the second distance threshold, sending a reminder message for the user to adjust the action.

[0009] In the above implementation process, by dividing the distance threshold into a first distance threshold and a second distance threshold, it is possible not only to determine whether the action to be scored is stable, but also to assign weights to different action amplitudes when the action to be scored is stable. This allows for the calculation and determination of corresponding scores based on different action amplitudes, thus improving the accuracy of the scoring.

[0010] In one embodiment, assigning a weight to the second time frame based on the relationship between the distance and the first and second distance thresholds if the distance is less than the second distance threshold includes: assigning a first weight to the second time frame if the distance is less than the first distance threshold; and assigning a second weight to the second time frame if the distance is between the first and second distance thresholds.

[0011] In the above implementation process, by classifying the actions to be scored into different categories according to the degree of execution of the actions based on different distance thresholds and assigning different weights to each category, the proportion of the actions in each time frame in the actions to be executed can be determined. This can reduce the impact of scores in invalid time frames on the final score and improve the accuracy of the final score.

[0012] In one embodiment, the formula for scoring the action of the first time frame based on the first joint angle and the standard joint angle, or the formula for scoring the action of the second time frame based on the second joint angle and the standard joint angle, is as follows: Among them, S i The motion is scored for the i-th time frame, where m represents the m joints in the motion to be scored that need to be compared. For standard joint angle range, A is the median of the standard joint angles. (i) This refers to the joint angle.

[0013] In the above implementation process, the motion score for each time frame is determined by the relationship between the joint angles of multiple joints in the motion to be scored and the standard joint angle. The score is calculated by the median of the standard joint angle and the average value of multiple key points, which reduces the impact of a few special key points and standard joint angles on the overall result and improves the accuracy of the score.

[0014] In one embodiment, after scoring the action of the second time frame based on the second joint angle and the standard joint angle, the method further includes: calculating a final score for the action to be scored based on the action scores of multiple time frames of the action to be scored; the formula for calculating the final score of the action to be scored is: Where S is the final score, and T is the total number of action scores used to calculate the final score. i For the action score in the i-th time frame, λ (i) The weight value of the i-th time frame.

[0015] In the above implementation process, when determining the final score of the action to be executed, the weight assigned to each time frame is considered in addition to the score of each time frame. By considering the weight of each time frame in the final score, the influence of invalid time frame scores on the final score is reduced, and the authenticity and accuracy of the final score are improved.

[0016] In one embodiment, the method further includes: if the first joint angle is not within a reasonable range, determining adjustment information based on the first joint angle and the standard joint angle; and providing real-time feedback of the adjustment information so that the user can adjust the action based on the adjustment information.

[0017] In the above implementation process, if it is determined that the first joint angle is not within a reasonable range, the deviation of the first joint angle relative to the standard joint angle is calculated based on the first joint angle and the standard joint angle. Corresponding adjustment information is then proposed and sent to the user in real time, allowing the user to make adjustments accordingly. By correcting the movement during the exercise, incorrect movements are prevented from being used during training, ensuring the accuracy of the movements during training and improving training effectiveness.

[0018] Secondly, embodiments of this application also provide an action scoring device, comprising: a first determining module: configured to determine the key point coordinates of a target image of a first time frame using a pose recognition algorithm, wherein the target image is an image of an action to be scored; a calculation module: configured to calculate a first joint angle based on the key point coordinates of the target image of the first time frame; a second determining module: configured to determine whether the first joint angle is within a reasonable range based on the first joint angle and a standard joint angle; a matching module: configured to determine whether a preset number of matched time frames has been reached if the first joint angle is within a reasonable range; and a scoring module: configured to score the action of the first time frame based on the first joint angle and the standard joint angle if the preset number of matched time frames has been reached; wherein the matched time frames are time frames in which the first joint angle is within a reasonable range.

[0019] Thirdly, embodiments of this application also provide a motion scoring device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the motion scoring device is running, the machine-readable instructions are executed by the processor to perform the steps of the method described in the first aspect above, or any possible implementation of the first aspect.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the action scoring method described in the first aspect or any possible implementation of the first aspect.

[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram illustrating the interaction between the image acquisition device, motion scoring device, and feedback device provided in the embodiments of this application;

[0024] Figure 2 This is a block diagram of the motion scoring device provided in an embodiment of this application;

[0025] Figure 3 A flowchart of the action scoring method provided in the embodiments of this application;

[0026] Figure 4 A flowchart illustrating the specific judgment process of the action scoring method provided in this application embodiment;

[0027] Figure 5 This is a schematic diagram of the functional modules of the motion scoring device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0029] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] The hallmarks of musculoskeletal disorders are pain (often persistent) and limitations in mobility, dexterity, and overall function, leading to reduced work capacity. Musculoskeletal disorders include problems with joints, bones, muscles, the spine, or multiple parts or systems of the body. Approximately two-thirds of all adults requiring rehabilitation need musculoskeletal rehabilitation services. Currently, rehabilitation training is often used to alleviate and resolve musculoskeletal pain. However, this training demands high standards of technique and requires real-time monitoring to ensure optimal results. While supervision and guidance from a physical therapist are possible, the therapist's experience and energy levels can influence the scoring of rehabilitation exercises.

[0031] On the other hand, with the rapid development of the times and the continuous improvement of people's quality of life, as their material needs are met, more and more people are beginning to care about their physical and mental health, and fitness exercise has become indispensable. In traditional learning or competitions of fitness exercise, judgment and scoring are mainly based on human experience. This method is highly dependent on people, but its efficiency and accuracy are not high, and it cannot accurately judge whether the movements during training or competition meet the standards.

[0032] In view of this, the inventors of this application propose an action recognition method that scores images of collected actions using image recognition, deep learning algorithms, etc. The entire process does not require human intervention, reducing the need for manual labor. At the same time, all scores are based on a unified standard, unaffected by experience or effort, thus improving the accuracy of action scoring.

[0033] To facilitate understanding of this embodiment, the operating environment for implementing the action scoring method disclosed in this application embodiment will first be described in detail.

[0034] like Figure 1 The diagram illustrates the interaction between the image acquisition device, motion scoring device, and feedback device provided in this embodiment. The motion scoring device communicates with one or more image acquisition devices and feedback devices via a network for data communication or interaction. The motion scoring device can be a web server, database server, personal computer (PC), tablet computer, smartphone, etc. The image acquisition device can be a camera, webcam, camcorder, personal computer (PC), tablet computer, smartphone, personal digital assistant (PDA), etc. The feedback device can be a broadcast device, recorder, smart speaker, personal computer (PC), tablet computer, smartphone, personal digital assistant (PDA), etc.

[0035] The image acquisition device here is used to capture images of the user's actions and send those images to the action scoring device.

[0036] After acquiring the action image, the aforementioned motion scoring device identifies the user's skeletal position information using a posture recognition algorithm. It then calculates joint angles using a preset algorithm and scores the user's action based on these angles and standard joint angles. The score is then sent to a feedback device.

[0037] If the motion scoring device determines that the joint angle is not within the standard joint angle range, it determines adjustment information based on the relationship between the joint angle and the standard joint angle, and sends the adjustment information to the feedback device.

[0038] This feedback device is used to provide feedback on the adjustment information and scores sent by the motion scoring device.

[0039] Understandably, the image acquisition device, motion scoring device, and feedback device can also be different modules of the same device. For example, the image acquisition device, motion scoring device, and feedback device can be the image acquisition module, motion scoring module, and evaluation feedback module in a smartphone, respectively. Specifically, the image acquisition module can be the smartphone's camera, the motion scoring module can be the smartphone's processor, and the evaluation feedback module can be the smartphone's speaker, etc.

[0040] To facilitate understanding of this embodiment, the action scoring device that performs the action scoring method disclosed in this application embodiment will first be described in detail.

[0041] like Figure 2 The diagram shown is a block diagram of a motion scoring device. The motion scoring device 100 may include a memory 111 and a processor 113. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the motion scoring device 100. For example, the motion scoring device 100 may also include a... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0042] The aforementioned memory 111 and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The aforementioned processor 113 is used to execute executable modules stored in the memory.

[0043] The memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 stores programs, and the processor 113 executes these programs upon receiving execution instructions. The method executed by the action scoring device 100, as defined in any embodiment of this application, can be applied to or implemented by the processor 113.

[0044] The aforementioned processor 113 may be an integrated circuit chip with signal processing capabilities. The processor 113 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0045] The motion scoring device 100 in this embodiment can be used to execute various steps in the various methods provided in the embodiments of this application. The implementation process of the motion scoring method is described in detail below through several embodiments.

[0046] Please see Figure 3 This is a flowchart of the action scoring method provided in the embodiments of this application. The following will describe... Figure 3 The specific process shown will be explained in detail.

[0047] Step 201: Determine the coordinates of key points in the target image of the first time frame using a pose recognition algorithm.

[0048] The target image here is the image of the action to be scored. The target image of the first time frame may include one or more.

[0049] The aforementioned pose recognition algorithm can be BlazePose, AlphaPose, pytorch-openpose, OpenPose, etc. The pose recognition algorithm can be adjusted and selected according to the actual situation, and this application does not impose any specific restrictions.

[0050] In some embodiments, step 201 can be implemented by the following steps:

[0051] The target image is input into a convolutional neural network. After passing through each layer of the convolutional neural network, the feature information of the target image is obtained. The human detection module in the pose recognition algorithm detects the position of the human body based on the feature information. Then, the tracking module of the pose recognition algorithm predicts and marks the position of the key points of the human body and returns the coordinate information of each key point.

[0052] Step 202: Calculate the first joint angle based on the key point coordinates of the target image in the first time frame.

[0053] The first joint angle can be calculated using inverse cosine, arctangent, etc., based on the key point coordinates of the target image in the first time frame.

[0054] For example, if the coordinates of key points A, O, and B are respectively (x, y, y) A ,y A ,z A ), (x o ,y o ,z o ), (x B ,y B ,z B These three key points form two line segments AO and BO, connected by point O. The joint angle formed by key points A, O, and B can then be expressed as:

[0055]

[0056] Where arccos is the arccosine, (x A ,y A ,z A (x) represents the coordinates of key point A. o ,y o ,z o (x) represents the coordinates of keypoint O, and (x) represents the coordinates of keypoint O. B ,y B ,z B ), where is the coordinate of key point B.

[0057] Understandably, if the first time frame includes multiple key points, and these multiple key points form multiple first joint angles, then the multiple first joint angles are calculated respectively according to the above joint angle calculation formula.

[0058] Step 203: Determine whether the first joint angle is within a reasonable range based on the first joint angle and the standard joint angle.

[0059] Understandably, each set of movements has a reasonable range of angles for its key joints. This standard joint angle represents the reasonable range of angles for the movement being scored. This standard joint angle can be pre-stored and directly retrieved when determining whether the first joint angle is within the reasonable range.

[0060] For example, if the standard joint angles of key points A, O, and B in a certain movement range from 90° to 120°, then these standard joint angles will be stored in the following form: ∠AOB ranges from 90° to 120°. If the standard joint angles of key points E, F, and G in a certain movement range from 60° to 100°, then these standard joint angles will be stored in the following form: ∠EFG ranges from 60° to 100°.

[0061] In practice, scoring is usually done on a series of consecutive movements. Each movement may involve different key points, thus each movement may involve different key points and first joint angles. Correspondingly, the standard joint angle may correspond to multiple standard joint angles. The number and specific range of these labeled joint angles are adjusted according to the actual movement and requirements; this application does not impose specific limitations.

[0062] Step 204: If the angle of the first joint is within a reasonable range, determine whether the number of frames matched in the target image has reached the preset number.

[0063] The number of frames that have been matched in the target image is the number of frames in the target image where the first joint angle is within a reasonable range.

[0064] The preset quantity here refers to the number of frames within a preset time range.

[0065] For example, taking an image processing speed of 30 frames per second as an example, if the required matching time is 1 second, the preset number is 30 frames. If the recorded number of matched frames has reached 30, it is determined that the number of matched frames of the target image has reached the preset number. If the recorded number of matched frames has not reached 30, it is determined that the number of matched frames of the target image has not reached the preset number.

[0066] Step 205: If the number of frames matched in the target image reaches a preset number, the action of the first time frame is scored according to the first joint angle and the standard joint angle.

[0067] Understandably, when scoring an action in a target image, the following conditions generally need to be met: the action must be within a reasonable range, and a certain amount of image information containing the action within a reasonable range must be obtained in order to ensure that the scoring of the action in the target image is sufficiently objective and accurate.

[0068] In other words, if the first joint angle of the target image in the first time frame is within a reasonable range, it indicates that the action of the target image in the first time frame is standard. If the number of frames in the first time frame where the first joint angle of the target image is within a reasonable range also reaches a preset number, it indicates that the number of images matched by the target image within the first time frame also meets the requirements. If the action to be scored for the target image is standard and the number of images matched by the target image also meets the requirements, it means that the action information of the target image fed back in the first time frame meets the scoring requirements. By scoring this action information, the accuracy of the scoring is ensured.

[0069] In the above implementation process, the first step is to determine whether the first joint angle of the target image is within a reasonable range to ascertain whether the action to be scored in the target image meets the requirements. Secondly, it is determined whether the number of frames within a reasonable range for the first joint angle is within a preset range to ascertain whether the acquired action information to be scored is sufficient. By judging the action based on both action specifications and the quantity of action information, the objectivity and accuracy of the scoring are ensured, thus improving the accuracy of the scoring.

[0070] In one possible implementation, such as Figure 4 As shown, the action scoring method further includes: if the number of frames matched in the target image has not reached a preset number, determining whether the distance between the key points of the target image in the second time frame and the key points of the target image in the first time frame is less than a distance threshold; if the distance is less than the distance threshold, calculating the second joint angle based on the key point coordinates of the target image in the second time frame; and scoring the action in the second time frame based on the second joint angle and the standard joint angle.

[0071] The second time frame here is the next time frame after the first time frame. Both the first and second time frames are within the same action to be scored. For example, if the first time frame is the current time frame, then the second time frame is the next time frame after the first time frame. If the second time frame is the current time frame, then the first time frame is the previous time frame before the second time frame.

[0072] The aforementioned distance threshold is the distance range between a keypoint in the first time frame and a corresponding keypoint in the second time frame. This distance threshold includes a first distance threshold and a second distance threshold.

[0073] Understandably, in two adjacent time frames of the same action to be evaluated, the same key point should move within an appropriate range to determine that the action to be evaluated is relatively stable within the first and second time frames. Therefore, when the distance between the same key point in the first and second time frames is within a distance threshold, it indicates that the action to be evaluated moves within a reasonable range and is relatively stable within the first and second time frames.

[0074] The calculation of the second joint angle based on the key point coordinates of the target image in the second time frame is the same as the calculation of the first joint angle in step 202 above, and will not be repeated here.

[0075] In the above implementation process, when the number of matched frames of the target image has not reached the preset number, the stability of the action to be scored is determined by judging the range of key points in two adjacent time frames. When the stability of the action to be scored meets the standard, the action to be scored is scored, ensuring that the action in the scoring time frame is stable and effective, thus improving the effectiveness and accuracy of the scoring.

[0076] In one possible implementation, determining whether the distance between the key points of the target image in the second time frame and the key points of the target image in the first time frame is less than a distance threshold includes: determining whether the distance is less than a second distance threshold; if the distance is less than the second distance threshold, assigning a weight to the second time frame based on the relationship between the distance and the first and second distance thresholds; if the distance is greater than the second distance threshold, sending a reminder message for the user to adjust their actions.

[0077] In determining the stability of an action, a second distance threshold is typically set, which represents the maximum distance the same keypoint can move between the first and second time frames. If this maximum distance is exceeded, the action being evaluated is considered unstable due to excessive movement between the first and second time frames. Conversely, if the distance between the same keypoint and the second time frame is within the maximum distance, the action is considered relatively stable. In this case, the appropriate weights are determined based on the movement amplitude between the two time frames.

[0078] The second distance threshold here is the maximum movement distance of the same keypoint that remains stable between the first and second time frames. The first distance threshold is the critical range of distances within which the weights of the same keypoint change when the first and second time frames are stable.

[0079] In some embodiments, if the distance is greater than a distance threshold, it indicates that the action to be scored is unstable. In this case, an adjustment message is issued to remind the user to adjust the action. This adjustment message may be: "The action is not standard, please repeat," "Please remain stable," "Please lift your right leg," "Please lift both feet off the ground," etc.

[0080] In the above implementation process, by dividing the distance threshold into a first distance threshold and a second distance threshold, it is possible not only to determine whether the action to be scored is stable, but also to assign weights to different action amplitudes when the action to be scored is stable. This allows for the calculation and determination of corresponding scores based on different action amplitudes, thus improving the accuracy of the scoring.

[0081] In one possible implementation, if the distance is less than a second distance threshold, a weight is assigned to the second time frame based on the relationship between the distance and the first and second distance thresholds, including: if the distance is less than the first distance threshold, assigning a first weight to the second time frame; if the distance is between the first and second distance thresholds, assigning a second weight to the second time frame.

[0082] Understandably, when scoring an action, different action amplitudes should correspond to different action scores. The amplitude of the action to be scored can be determined by the relationship between the distance range of the same key point in the first and second time frames and different distance thresholds, and then the action to be scored can be scored based on the amplitude of the action to be scored.

[0083] For example, if the key point of the first time frame is P s The key point of the second time frame is P. j And the first distance threshold and the second distance threshold are δ1 and δ2, respectively. When the distance between the second time frame and the first time frame is less than the first distance threshold, i.e., |P j -P s If |P < δ1, then assign a weight λ1 to this frame; the distance between the second time frame and the first time frame is between the first distance threshold and the second distance threshold, i.e., δ1 < |P j -P s If |<δ2, then assign a weight λ2 to this frame.

[0084] In some embodiments, the distance threshold may further include a third distance threshold, a fourth distance threshold, etc., which are used to further divide the distance between the first distance threshold and the second distance threshold, and further assign different weights based on the relationship between the distance between the second time frame and the first time frame and the first distance threshold, the second distance threshold, the third distance threshold, and the fourth distance threshold. The selection of the distance threshold here is merely exemplary, and more or fewer distance thresholds may be used depending on the actual situation; this application does not impose specific limitations.

[0085] For example, if the key point of the first time frame is P s The key point of the second time frame is P. j And the first distance threshold, the second distance threshold, the third distance threshold, and the fourth distance threshold are δ1, δ2, δ3, and δ4, respectively. Then the action to be scored can be weighted according to the following rules: If |P j -P s If |<δ1, then assign weight λ1 to this frame; if δ1<|P j -P s If |<δ3, then assign a weight λ2 to this frame; if δ3<|P j-P s If |<δ4, then assign a weight λ3 to this frame; if δ4<|P j -P s If |<δ2, then a weight λ4 is assigned to this frame. It is understood that the rules for assigning weights to the actions to be scored here are merely exemplary, and those skilled in the art can adjust them according to the actual situation; this application does not impose specific limitations.

[0086] In the above implementation process, by classifying the actions to be scored into different categories according to the degree of execution of the actions based on different distance thresholds and assigning different weights to each category, the proportion of the actions in each time frame in the actions to be executed can be determined. This can reduce the impact of scores in invalid time frames on the final score and improve the accuracy of the final score.

[0087] In one possible implementation, the formula for scoring the action of the first time frame based on the first joint angle and the standard joint angle, or for scoring the action of the second time frame based on the second joint angle and the standard joint angle, is as follows:

[0088]

[0089]

[0090] Among them, S i The motion is scored for the i-th time frame, where m represents the m joints in the motion to be scored that need to be compared. For standard joint angle range, A is the median of the standard joint angles. (i) This refers to the joint angle.

[0091] In the above implementation process, the motion score for each time frame is determined by the relationship between the joint angles of multiple joints in the motion to be scored and the standard joint angle. The score is calculated by the median of the standard joint angle and the average value of multiple key points, which reduces the impact of a few special key points and standard joint angles on the overall result and improves the accuracy of the score.

[0092] In one possible implementation, after scoring the action of the second time frame based on the second joint angle and the standard joint angle, the method further includes: calculating the final score of the action to be scored based on the action scores of multiple time frames of the action to be scored.

[0093] The formula for calculating the final score of the action to be scored is as follows:

[0094]

[0095] Where S is the final score, and T is the total number of action scores used to calculate the final score. iFor the action score in the i-th time frame, λ (i) The weight value of the i-th time frame.

[0096] Understandably, in practice, a comprehensive score is usually given for a set of actions. Therefore, after determining the score for each time frame, the final score for the action to be performed is calculated based on the score of each time frame and the corresponding weight of each time frame.

[0097] In the above implementation process, when determining the final score of the action to be executed, the weight assigned to each time frame is considered in addition to the score of each time frame. By considering the weight of each time frame in the final score, the influence of invalid time frame scores on the final score is reduced, and the authenticity and accuracy of the final score are improved.

[0098] In one possible implementation, the method further includes: if the first joint angle is not within a reasonable range, determining adjustment information based on the first joint angle and the standard joint angle; and providing real-time feedback of the adjustment information so that the user can adjust the action based on the adjustment information.

[0099] Understandably, when the angle of the first joint is not within a reasonable range, it indicates that the user's movement is not in place. In this case, adjustment information can be determined based on the angle difference between the angle of the first joint and the standard joint angle.

[0100] For example, if the standard action requires the angle between the thigh and calf to be between R1 and R2, then when the angle between the user's thigh and calf is less than R1, the feedback is "Please lift your calf to increase the angle with your thigh", and when the angle between the user's thigh and calf is greater than R2, the feedback is "Please pull your calf back to decrease the angle with your thigh".

[0101] The prompts here include movement guidance and movement execution information. Movement guidance information may include instructions to correct the user's movements, such as "Please raise your calves," "Please lower your head," "Please open your arms," ​​and "Please lift your feet off the ground." Movement execution information may include instructions to the user regarding whether to perform the movement, such as "Please continue holding the current position," "Please repeat the previous movement," and "Stop training."

[0102] In some embodiments, after determining the user's final score, the next stage of training goals can be determined based on the user's final score. For example, if the user's final score is high, it indicates that the user has achieved good results in training this series of movements. In this case, more intensive training movements can be recommended for the user based on the user's final score. If the user's final score is low, it indicates that the user is finding this series of movements difficult. In this case, more basic training movements can be recommended for the user based on the user's final score. For example, if this scoring method is used for user rehabilitation training, if the user's final score is high, it indicates that the user has recovered. In this case, it can be determined based on the user's final score that the user no longer needs to continue rehabilitation training, etc.

[0103] In the above implementation process, if it is determined that the first joint angle is not within a reasonable range, the deviation of the first joint angle relative to the standard joint angle is calculated based on the first joint angle and the standard joint angle. Corresponding adjustment information is then proposed and sent to the user in real time, allowing the user to make adjustments accordingly. By correcting the movement during the exercise, incorrect movements are prevented from being used during training, ensuring the accuracy of the movements during training and improving training effectiveness.

[0104] Based on the same application concept, this application also provides an action scoring device corresponding to the action scoring method. Since the principle of the device in this application is similar to that of the aforementioned action scoring method, the implementation of the device in this application can refer to the description in the above method embodiments, and the repeated parts will not be described again.

[0105] Please see Figure 5 This is a functional module diagram of the motion scoring device provided in this application embodiment. Each module in the motion scoring device in this embodiment is used to execute the steps in the above method embodiments. The motion scoring device includes a first determining module 301, a calculation module 302, a second determining module 303, a matching module 304, and a scoring module 305; wherein,

[0106] The first determining module 301 is used to determine the key point coordinates of the target image of the first time frame through a pose recognition algorithm, wherein the target image is an image of the action to be scored.

[0107] The calculation module 302 is used to calculate the first joint angle based on the key point coordinates of the target image in the first time frame.

[0108] The second determining module 303 is used to determine whether the first joint angle is within a reasonable range based on the first joint angle and the standard joint angle.

[0109] The matching module 304 is used to determine whether the number of frames matched in the target image has reached a preset number if the angle of the first joint is within a reasonable range.

[0110] The scoring module 305 is used to score the action of the first time frame based on the first joint angle and the standard joint angle if the number of frames matched in the target image reaches a preset number.

[0111] In one possible implementation, the motion scoring device further includes a judgment module, configured to determine whether the distance between the key points of the target image in the second time frame and the key points of the target image in the first time frame is less than a distance threshold if the number of frames matched by the target image has not reached a preset number; if the distance is less than the distance threshold, calculate a second joint angle based on the key point coordinates of the target image in the second time frame; and score the motion in the second time frame based on the second joint angle and the standard joint angle.

[0112] In one possible implementation, the judgment module is further configured to: determine whether the distance is less than a second distance threshold; if the distance is less than the second distance threshold, assign a weight to the second time frame according to the relationship between the distance and the first distance threshold and the second distance threshold; if the distance is greater than the second distance threshold, send a reminder message for the user to adjust the action.

[0113] In one possible implementation, the determination module is specifically used to: assign a first weight to the second time frame if the distance is less than the first distance threshold; and assign a second weight to the second time frame if the distance is between the first distance threshold and the second distance threshold.

[0114] In one possible implementation, the scoring module 305 is further configured to: calculate the final score of the action to be scored based on the action scores of multiple time frames of the action to be scored.

[0115] In one possible implementation, the motion scoring device further includes a feedback module, used to determine adjustment information based on the first joint angle and the standard joint angle if the first joint angle is not within a reasonable range; and to provide real-time feedback of the adjustment information so that the user can adjust the motion according to the adjustment information.

[0116] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the action scoring method described in the above method embodiments.

[0117] The computer program product of the action scoring method provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the action scoring method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0119] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0120] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0121] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of action scoring, characterized by, include: The key point coordinates of the target image in the first time frame are determined by a pose recognition algorithm, and the target image is an image of the action to be scored. Calculate the first joint angle based on the key point coordinates of the target image in the first time frame; Determine whether the first joint angle is within a reasonable range based on the first joint angle and the standard joint angle; If the angle of the first joint is within a reasonable range, determine whether the number of frames matched in the target image has reached a preset number; If the number of frames that have been matched in the target image reaches a preset number, the action of the first time frame is scored according to the first joint angle and the standard joint angle. Wherein, the number of frames in the target image that have been matched is the number of frames in which the first joint angle is within a reasonable range; The method further includes: If the number of frames that have been matched in the target image has not reached the preset number, determine whether the distance between the key points of the target image in the second time frame and the key points of the target image in the first time frame is less than the distance threshold. If the distance is less than the distance threshold, calculate the second joint angle based on the key point coordinates of the target image in the second time frame; The motion in the second time frame is scored based on the second joint angle and the standard joint angle. The distance threshold includes a first distance threshold and a second distance threshold. The step of determining whether the distance between the key points of the target image in the second time frame and the key points of the target image in the first time frame is less than the distance threshold includes: Determine whether the distance is less than a second distance threshold; If the distance is less than the second distance threshold, the second time frame is assigned a weight based on the relationship between the distance and the first distance threshold and the second distance threshold; If the distance is greater than the second distance threshold, a reminder message is sent so that the user can adjust their actions.

2. The method of claim 1, wherein, If the distance is less than the second distance threshold, the second time frame is weighted according to the relationship between the distance and the first distance threshold and the second distance threshold, including: If the distance is less than the first distance threshold, the second time frame is assigned a first weight; If the distance is between the first distance threshold and the second distance threshold, the second time frame is assigned a second weight.

3. The method of claim 2, wherein, The formula for scoring the action in the first time frame based on the first joint angle and the standard joint angle, or the formula for scoring the action in the second time frame based on the second joint angle and the standard joint angle, is as follows: ; ； wherein, is the action score for a time frame, is the joint to compare, is the is the median of the standard joint angle, is the joint angle.

4. The method of claim 3, wherein, After scoring the motion of the second time frame based on the second joint angle and the standard joint angle, the method further includes: The final score of the action to be scored is calculated based on the action scores of multiple time frames of the action to be scored. The formula for calculating the final score of the action to be scored is as follows: ； Where S represents the final score. To calculate the total number of action scores for the final evaluation, For the first Motion scoring for time frames No. Weight values ​​for time frames.

5. The method according to claim 1, characterized in that, The method further includes: If the first joint angle is not within a reasonable range, adjustment information is determined based on the first joint angle and the standard joint angle. The adjustment information is provided in real time so that the user can adjust the action accordingly.

6. A motion scoring device, characterized in that, include: First determining module: used to determine the key point coordinates of the target image in the first time frame through a pose recognition algorithm, wherein the target image is an image of the action to be scored; Calculation module: used to calculate the first joint angle based on the key point coordinates of the target image in the first time frame; The second determining module is used to determine whether the first joint angle is within a reasonable range based on the first joint angle and the standard joint angle. Matching module: used to determine whether the number of frames matched in the target image has reached a preset number if the angle of the first joint is within a reasonable range; Scoring module: used to score the action of the first time frame based on the first joint angle and the standard joint angle if the number of frames matched in the target image reaches a preset number; Wherein, the matched time frame is the time frame in which the first joint angle is within a reasonable range; The action scoring device further includes a judgment module, which is used for: If the number of frames that have been matched in the target image has not reached the preset number, determine whether the distance between the key points of the target image in the second time frame and the key points of the target image in the first time frame is less than the distance threshold. If the distance is less than the distance threshold, calculate the second joint angle based on the key point coordinates of the target image in the second time frame; The motion in the second time frame is scored based on the second joint angle and the standard joint angle. The distance threshold includes a first distance threshold and a second distance threshold. The module for determining whether the distance between key points in the target image of the second time frame and key points in the target image of the first time frame is less than the distance threshold is further configured to: Determine whether the distance is less than a second distance threshold; If the distance is less than the second distance threshold, the second time frame is assigned a weight based on the relationship between the distance and the first distance threshold and the second distance threshold; If the distance is greater than the second distance threshold, a reminder message is sent so that the user can adjust their actions.

7. A motion scoring device, characterized in that, include: The processor and memory, wherein the memory stores machine-readable instructions executable by the processor, and when the action scoring device is running, the machine-readable instructions are executed by the processor to perform the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 5.

Images