Method for correcting and evaluating fitness movements

Abstract

The present invention discloses a method for correcting and evaluating fitness movements. Taking the standard action video constructed by a virtual character as a demonstration, after setting the position and shooting angle of the camera and adjusting it to an angle similar to or the same as the standard action video, through the OpenPose network, after obtaining the skeletal joint point information of the demonstration action and the user's action video and performing registration, calculating the similarity degree of the video based on the registration error, and giving suggestions for the user to correct the action to assist the user in establishing good movement habits. The present invention can compare the similarity degree of the demonstration video and the user's same-action video based on the matching condition of the skeletal joint points, give suggestions for the user to correct the action, and assist the user in establishing good movement habits.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image technology, and relates to a method for correcting and evaluating fitness movements. Background Art

[0002] With the advent of the Internet era, the pace of development of fitness apps has been advancing continuously in the emerging fitness mode. In recent years, in particular, fitness apps such as Keep, Fittime, and Mint have emerged. However, according to market research, the software currently on the market lacks an interface for interacting with users. When users are doing fitness exercises, they can only see the standard movements demonstrated by the coach and do the exercises while watching. Moreover, users themselves cannot know whether the movements they do are correct. If incorrect movements are not corrected, they will instead cause sports injuries to the body. Therefore, a smart e-personal trainer system that can correct movements is a trend to improve the popularization and effective promotion of fitness exercises. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for correcting and evaluating fitness movements, which can compare the similarity between a demonstration video and a user's video of the same movement by matching the skeletal joint points, and then give suggestions for correcting the user's movements to assist the user in establishing good movement habits.

[0004] The technical solution adopted by the present invention is as follows:

[0005] A method for correcting and evaluating fitness movements, comprising the following steps:

[0006] Step 1, obtain a standard movement video and standard skeletal joint points;

[0007] Step 2, collect a real-time user movement video and obtain the user's skeletal joint points;

[0008] Step 3, calculate the intersection over union of the standard skeletal joint points and the user's skeletal joint points, determine whether it is a wrong movement. If so, go to Step 4; otherwise, increment the number of frames of the correct movement by 1 and go to Step 5;

[0009] Step 4, give suggestions for correcting the user's posture;

[0010] Step 5, if the movement is repeated, go to Step 2; otherwise, go to Step 6;

[0011] Step 6, after the movement is completed, give an evaluation of the user's movement.

[0012] The characteristics of the present invention also lie in:

[0013] The specific steps in Step 1 are as follows:

[0014] Step 1.1, obtain a standard movement video, generally shot by a coach wearing tight sports clothes;

[0015] Step 1.2: Input the standard action video obtained in Step 1.1 into the AnimeGAN network to output a virtual character image of the same action.

[0016] Step 1.3: Input the virtual character action video obtained in Step 1.2 into the OpenPose network to output the standard bone joint points of the virtual character image.

[0017] When the user's motion video is collected in real time in Step 2, the user's motion information is a follow-up video consistent with the standard action video, and the camera angle arrangement needs to be consistent with the angle of the standard action video; for the user's motion video, obtain the user's bone joint points through the OpenPose network.

[0018] Step 3 is specifically as follows:

[0019] Step 3.1: Superimpose the standard bone joint points and the user's bone joint points. When superimposing, place the standard action in the central axis part.

[0020] Step 3.2: Calculate the intersection over union (IoU) of the standard bone joint points and the user's bone joint points. Let the superimposed image with size be denoted as , there is:

[0021]

[0022] , (1)

[0023] Calculate the intersection over union of the user's action and the standard action according to the following formula:

[0024] (2)

[0025] Where , ;

[0026] The intersection over union , is the confidence judgment threshold. If it indicates that the user's action exceeds the confidence range, it means the posture is incorrect, and the number of frames of incorrect actions is incremented by 1, and go to Step 4 to give suggestions for correcting the user's posture. Otherwise, the number of frames of correct actions is incremented by 1, and go to Step 5.

[0027] Step 4 is specifically as follows:

[0028] Step 4.1: Describe different limbs linearly.

[0029] Such as Figure 5As shown, for the standard action skeletal joint points obtained in step 1 and the skeletal joint points of the user action obtained in step 2, take the straight lines of the left and right upper arms, lower arms, left and right trunks, thighs, and calves, a total of 10 straight lines, which are marked in sequence as follows: The 10 straight lines representing the standard action limbs are: , and the 10 straight lines representing the user action limbs are , where the superscript marks the standard action, and the superscript marks the user action; the subscript represents the left side, represents the right side, UA and LA respectively represent the upper arm and the lower arm, Tr represents the trunk, and UT and LT respectively represent the thigh and the calf;

[0030] Step 4.2, based on the 10 straight lines of the standard action , calculate the angles with the 10 straight lines of the user action respectively, and record them as: ;

[0031] Step 4.3, give corrections to the user action based on the angle differences, and the correction reminder criteria are as follows:

[0032] a. If , where is the action tolerance parameter, which is an empirical parameter, and this patent preferably takes it as indicating that the left upper arm action is too high, reminding the user to lower the left upper arm, otherwise if , indicating that the left upper arm action is too low, reminding the user to raise the left upper arm;

[0033] b. If , indicating that the right upper arm action is too high, reminding the user to lower the right upper arm, otherwise if , indicating that the right upper arm action is too low, reminding the user to raise the right upper arm;

[0034] c. If , indicating that the left lower arm action is too high, reminding the user to lower the left lower arm, otherwise if , indicating that the left lower arm action is too low, reminding the user to raise the left lower arm;

[0035] d. If , indicating that the right lower arm action is too high, reminding the user to lower the right lower arm, otherwise if , indicating that the movement of the right lower arm is downward, and reminding to raise the right lower arm;

[0036] e. If , indicating that the movement of the left thigh is upward, reminding the user to squat a little lower with the left thigh. Otherwise, if , indicating that the movement of the left thigh is downward, reminding the user to raise the left thigh;

[0037] f. If , indicating that the movement of the right thigh is upward, reminding the user to squat a little lower with the right thigh. Otherwise, if , indicating that the movement of the right thigh is downward, reminding the user to raise the right thigh;

[0038] g. If , indicating that the movement of the left lower leg is upward, reminding the user to lower the left lower leg a little more. Otherwise, if , indicating that the movement of the left lower leg is downward, reminding the user to raise the left lower leg;

[0039] h. If , indicating that the movement of the right lower leg is upward, reminding the user to lower the right lower leg a little more. Otherwise, if , indicating that the movement of the right lower leg is downward, reminding the user to raise the right lower leg.

[0040] In step 6 of [], according to the number of frames of the incorrect movement obtained in step 3.3 and the number of frames of the correct movement , calculate the action score of the user as follows:

[0041] (3)

[0042] So far, all operations are completed.

[0043] The beneficial effects of the present invention are:

[0044] The present invention takes the standard action video constructed by the virtual character as a demonstration, sets the position and shooting angle of the camera in the way indicated by the system, and after adjusting to an angle similar to or the same as the standard action video as much as possible, obtains the skeletal joint point information of the teaching action and the user action video through the OpenPose network, performs registration, calculates the similarity degree of the video based on the registration error, and gives suggestions for the user to correct the action, assisting the user to establish good action habits. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is the flowchart of the method for correcting and evaluating the fitness action described in the present invention

[0046] Figure 2 is the schematic diagram of the AnimeGAN network structure;

[0047] Figure 3 It is a schematic diagram of the OpenPose network structure;

[0048] Figure 4 It is a schematic diagram of the extraction of the skeletal joint points of the user's actions;

[0049] Figure 5 It is a schematic diagram of the superposition of the skeletal joint points of the standard action and the user's action; Specific implementation manner

[0050] The present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0051] The method for correcting and evaluating fitness actions of the present invention, as Figure 1 , includes the following steps:

[0052] Step 1, obtain the standard action video and standard skeletal joint points;

[0053] Step 2, collect the real-time user movement video and obtain the user's skeletal joint points;

[0054] Step 3, calculate the intersection over union of the standard skeletal joint points and the user's skeletal joint points, judge whether it is a wrong action, if yes, go to Step 4, otherwise, increment the number of frames of the correct action by 1 and go to Step 5;

[0055] Step 4, give suggestions for correcting the user's posture;

[0056] Step 5, if the action is repeated, go to Step 2, otherwise go to Step 6;

[0057] Step 6, after the action is completed, give an evaluation of the user's action;

[0058] Among them, the specific steps in Step 1 are:

[0059] Step 1.1, obtain the standard action video, generally filmed by a coach wearing tight sportswear;

[0060] Step 1.2, as Figure 2 , input the standard action video obtained in Step 1.1 into the AnimeGAN network, and output the virtual character image of the same action;

[0061] Step 1.3, as Figure 3 , Figure 4 , input the virtual character action video obtained in Step 1.2 into the OpenPose network, and output the standard skeletal joint points of the virtual character image.

[0062] When the user's motion video is collected in real time in Step 2, the user's motion information is a follow-up video consistent with the standard action video, and the angle arrangement of the camera needs to be consistent with the angle of the standard action video; for the user's motion video, the user's skeletal joint points are obtained in the same way as in Step 1.3.

[0063] Step 3 is specifically as follows:

[0064] Step 3.1: Superimpose the standard skeletal joint points and the user's skeletal joint points. When superimposing, place the standard action in the central axis part;

[0065] Step 3.2: Calculate the intersection over union (IoU) of the standard skeletal joint points and the user's skeletal joint points. Let the superimposed graph with size be denoted as , then there is:

[0066] ,

[0067] , (1)

[0068] Calculate the intersection over union of the user's action and the standard action according to the following formula:

[0069] (2)

[0070] where , ;

[0071] The intersection over union ( is the confidence judgment threshold, which is an empirical value. This patent preferably uses ). If it indicates that the user's action exceeds the confidence range, it means that the posture is incorrect, and the number of frames of the incorrect action is incremented by 1, and then go to Step 4 to give the user posture correction suggestion. Otherwise, the number of frames of the correct action is incremented by 1, and then go to Step 5;

[0072] Step 4: The specific steps are as follows:

[0073] Step 4.1: Describe different limbs linearly;

[0074] As Figure 5 shown, from the standard action skeletal joint points obtained in Step 1.3 (shown as dotted lines) and the user's action skeletal joint points obtained in Step 3.1 (shown as solid lines), take the straight lines of the left and right upper arms, forearms, left and right trunks, thighs, and calves, a total of 10 straight lines, and mark them in sequence as follows: The 10 straight lines representing the standard action limbs are: , the 10 straight lines representing the user's action limbs are , where the superscript marks the standard action, and the superscript marks the user's action; the subscript represents the left side, represents the right side, UA and LA respectively represent the upper arm and the lower arm, Tr represents the torso, and UT and LT respectively represent the thigh and the lower leg;

[0075] Step 4.2, taking the 10 straight lines of the standard action obtained in Step 4.1 as the reference , calculate the angles with the 10 straight lines of the user's action respectively , and denote them as: ;

[0076] Step 4.3, give the correction of the user's action according to the angle difference, and the correction reminder criteria are as follows:

[0077] a. If , where is the action tolerance parameter, an empirical parameter, and this patent preferably takes it as it indicates that the action of the left upper arm is too high, reminding the user to lower the left upper arm, otherwise if , it indicates that the action of the left upper arm is too low, reminding the user to raise the left upper arm;

[0078] b. If , it indicates that the action of the right upper arm is too high, reminding the user to lower the right upper arm, otherwise if , it indicates that the action of the right upper arm is too low, reminding the user to raise the right upper arm;

[0079] c. If , it indicates that the action of the left lower arm is too high, reminding the user to lower the left lower arm, otherwise if , it indicates that the action of the left lower arm is too low, reminding the user to raise the left lower arm;

[0080] d. If , it indicates that the action of the right lower arm is too high, reminding the user to lower the right lower arm, otherwise if , it indicates that the action of the right lower arm is too low, reminding it to raise the right lower arm;

[0081] e. If , it indicates that the action of the left thigh is too high, reminding the user to squat a little lower with the left thigh, otherwise if , indicating that the movement of the left thigh is on the lower side, reminding the user to lift the left thigh a little bit;

[0082] f. If , indicating that the movement of the right thigh is on the upper side, reminding the user to squat down the right thigh a little more. Otherwise, if , indicating that the movement of the right thigh is on the lower side, reminding the user to lift the right thigh a little bit;

[0083] g. If , indicating that the movement of the left calf is on the upper side, reminding the user to lower the left calf a little more. Otherwise, if , indicating that the movement of the left calf is on the lower side, reminding the user to lift the left calf a little bit;

[0084] h. If , indicating that the movement of the right calf is on the upper side, reminding the user to lower the right calf a little more. Otherwise, if , indicating that the movement of the right calf is on the lower side, reminding the user to lift the right calf a little bit;

[0085] Among them, in step 6, according to the number of frames of the wrong movement and the number of frames of the correct movement obtained in step 3.3, calculate the user's movement score as follows:

[0086] (3)

[0087] So far, all operations are completed.

Claims

1. Method for correcting and evaluating fitness movements, Characterized in that, It includes the following steps: Step 1, obtain a standard action video and standard skeletal joint points; Step 2, collect real-time user movement videos and obtain user skeletal joint points; Step 3, calculate the intersection over union of the standard skeletal joint points and the user skeletal joint points, determine whether it is a wrong action, if yes, go to Step 4, otherwise, increment the number of frames of the correct action by 1 and go to Step 5; Step 4, give suggestions for correcting the user's posture; Step 5, if the action is repeated, go to Step 2, otherwise go to Step 6; Step 6, after the action is completed, give an evaluation of the user's action; The specific content of Step 3 is: Step 3.1, superimpose the standard skeletal joint points and the user skeletal joint points. When superimposing, place the standard action in the central axis part; Step 3.2, calculate the intersection over union of the standard skeletal joint points and the user's skeletal joint points. Let the superimposed image with a size of be represented as , then there is: ， （1） Calculate the intersection over union of the user action and the standard action according to the following formula, and the formula is as follows: （2） Among them, , ; The intersection over union calculated in Step 3.3 , is the confidence judgment threshold. If it indicates that the user's action exceeds the confidence range, it means the posture is incorrect, and the number of frames of incorrect actions is incremented by 1, and Step 4 is entered to give suggestions for correcting the user's posture. Otherwise, the number of frames of correct actions is incremented by 1, and Step 5 is entered.

2. The method for correcting and evaluating fitness movements according to claim 1, Characterized in that, The specific steps in Step 1 are: Step 1.1, obtain a standard action video; Step 1.2, input the standard action video obtained in Step 1.1 into the AnimeGAN network to output a virtual character image of the same action; Step 1.3, input the virtual character action video obtained in Step 1.2 into the OpenPose network to output the standard skeletal joint points of the virtual character image.

3. The method for correcting and evaluating fitness movements according to claim 1, Characterized in that, When collecting the user's movement video in real time in Step 2, the user's movement information is a follow-along video consistent with the standard action video, and the angle arrangement of the camera needs to be consistent with the angle of the standard action video; obtain the user's skeletal joint points for the user's movement video through the OpenPose network.

4. The method for correcting and evaluating fitness movements according to claim 1, Characterized in that, The specific content of Step 4 is: Step 4.1, perform linear description on different limbs: For the standard action skeletal joint points obtained in Step 1 and the skeletal joint points of the user action obtained in Step 2, take the straight lines of the left and right upper arms, forearms, left and right trunks, thighs, and calves, a total of 10 straight lines, and mark them in sequence as follows: The 10 straight lines representing the standard action limbs are: , and the 10 straight lines representing the user action limbs are , where the superscript marks the standard action, and the superscript marks the user action; the subscript represents the left side, represents the right side, UA and LA respectively represent the upper arm and forearm, Tr represents the trunk, and UT and LT respectively represent the thigh and calf; Step 4.2, taking the 10 straight lines of the standard action as the reference , calculate the included angles with the 10 straight lines of the user action respectively , denoted as: ; Step 4.3, give corrections to the user's action based on the angle difference, and the correction reminder criteria are as follows: a. If , where is an action fault tolerance parameter, an empirical parameter, indicating that the action of the left upper arm is too high, reminding the user to lower the left upper arm, otherwise if , it indicates that the action of the left upper arm is too low, reminding the user to raise the left upper arm; b. If , it indicates that the movement of the right upper arm is too high, reminding the user to lower the right upper arm. Otherwise, if , it indicates that the movement of the right upper arm is too low, reminding the user to raise the right upper arm; c. If , it indicates that the movement of the lower left arm is too high, reminding the user to lower the lower left arm. Otherwise, if , it indicates that the movement of the lower left arm is too low, reminding the user to raise the lower left arm; d. If , it indicates that the movement of the right lower arm is too high, reminding the user to lower the right lower arm. Otherwise, if , it indicates that the movement of the right lower arm is too low, reminding the user to raise the right lower arm; e. If , it indicates that the movement of the left thigh is on the upper side, reminding the user to squat a little lower with the left thigh. Otherwise, if , it indicates that the movement of the left thigh is on the lower side, reminding the user to lift the left thigh; f. If , it indicates that the movement of the right thigh is on the upper side, reminding the user to squat a little lower with the right thigh. Otherwise, if , it indicates that the movement of the right thigh is on the lower side, reminding the user to lift the right thigh; g. If , it indicates that the movement of the left calf is on the upper side, reminding the user to lower the left calf a little more. Otherwise, if , it indicates that the movement of the left calf is on the lower side, reminding the user to lift the left calf; h. If , it indicates that the movement of the right calf is on the upper side, reminding the user to lower the right calf a little more. Otherwise, if , it indicates that the movement of the right calf is on the lower side, reminding the user to lift the right calf.

5. The method for correcting and evaluating fitness movements according to claim 1, Characterized in that, In step 6, according to the number of frames of incorrect actions obtained in step 3.3 and the number of frames of correct actions , calculate the user's action score as follows: （3） Thus, all operations are completed.

Images