An action assessment method based on monocular RGB video
Through the deep learning technology of monocular RGB video, real-time motion evaluation of sports analysis is realized, high cost and professional problems of the existing technology are solved, suitable for mass sports fitness and popular applications, reducing hardware needs and costs.
Patent Information
- Application Number
- CN202110537068.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-15
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-07-15
AI Technical Summary
Existing sports analysis technologies rely on high-cost depth cameras or binocular cameras, which are difficult to popularize in mass sports fitness and popular applications, and require professional knowledge and dedicated guidance, which is expensive.
Deep learning technology based on monocular RGB video is adopted to realize action evaluation and reduce hardware requirements through static keyframe extraction, skeleton joint node calculation, joint angle evaluation and posture registration.
It realizes real-time and efficient movement evaluation, reduces deployment costs, and is suitable for mass sports and fitness and popular applications, without the need for manual evaluation by experts.
Smart Images

Figure CN113327267B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computer vision and relates to an action evaluation method based on monocular RGB video. Background Art
[0002] Computer vision is now used everywhere in our lives, from face payment to facial identity verification at stations. At the same time, today's applications require computer vision to be able to process large amounts of crowd data without feeling, such as flow monitoring at stations and temperature detection of mobile populations during the COVID-19 pandemic. Motion analysis and evaluation is a critical part of professional sports training, but it requires professional knowledge and personal guidance, and the cost is very high, making it difficult to popularize for mass sports and fitness. Researching computer vision technology for motion analysis can significantly reduce costs and make it popular. In addition, work that requires the participation of professionals, such as rehabilitation training and physical fitness screening for large groups of young people, can also be processed using motion analysis systems.
[0003] Early motion analysis technologies mostly used RGB-D depth cameras such as Kinect to extract human skeleton joints, which had high deployment costs, low market share, and could only be used in professional venues. In recent years, a large number of research results on human posture estimation based on deep learning have emerged, such as OpenPose, AlphaPose, HRNet and other network models, which can accurately obtain 2D human skeleton joints using monocular RGB images, which is expected to bring motion analysis systems to monocular cameras.
[0004] Through the current experiments, it can be concluded that the present invention can meet the operation requirements by using a PC equipped with an Nvidia graphics card with 8G memory and 4G video memory. As the independent NPU units used in mobile terminals become increasingly powerful, and a large number of deep learning frameworks gradually support mobile ARM chips, it is not difficult to migrate the present invention to mobile devices. Using smartphones, tablets, etc. as data collection and calculation units can be used as AI coaches for mass home fitness, and its application prospects are very broad. Summary of the invention
[0005] Purpose of the invention: The present invention provides an action evaluation method based on monocular RGB video, which uses deep learning technology to make the system usage environment highly robust to conditions such as site and lighting. Compared with technical solutions using professional equipment such as depth cameras or binocular cameras, the deployment cost of this system is lower and has great prospects for promotion to mobile devices.
[0006] Technical solution:
[0007] To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a method for action evaluation based on monocular RGB video, the method comprising the following steps:
[0008] (1) Input an action video and extract static key frames;
[0009] (2) extracting skeleton joint points from the static key frame obtained in step (1);
[0010] (3) calculating the joint angles of the skeleton joint points obtained in step (2);
[0011] (4) aligning the input pose and the template pose of the skeleton joint points obtained in step (2);
[0012] (5) performing motion evaluation on the registered posture obtained in step (4), using the coordinates of the registered skeleton joint points and the joint angles obtained in step (3) as evaluation benchmarks;
[0013] Furthermore, in step (1), after inputting the action video, the inter-frame difference method is used to extract the still frame, and the specific method is as follows:
[0014] (1-1) Calculate the inter-frame vector of the video;
[0015] (1-2) Set the flags: static state flag is_static, static start position flag Begin, static end position flag End;
[0016] (1-3) Set the stillness threshold: 0.2 times the maximum inter-frame difference in the video sequence;
[0017] (1-4) Set the starting state is_static=True, Begin=-1 (record the initial action); when the difference value is greater than the threshold, set is_static=False; when the difference value is less than the threshold and is_static=False, mark the current frame as the static starting position, and is_satic=True; when the difference value is greater than the threshold and is_static=True, if Begin!=-1, mark the current frame as the static ending position, and is_static=False, select the frame with the smallest difference in the Begin-End interval as the static frame;
[0018] (1-5) Repeat steps (1-3) and (1-4) until the video ends.
[0019] Furthermore, in step (2), the PyQt submodule interface is called to visually delete any unnecessary frames that may exist.
[0020] Furthermore, in step (1-1), the specific method for calculating the inter-frame difference vector of the video is as follows:
[0021] (1-1-1) Save the current frame and the two frames before the current frame, and calculate the absolute value of the difference between the two adjacent frames to obtain two difference values D 1 , D 2 ;
[0022] (1-1-2) D 1 , D 2 Use the bitwise AND operation to synthesize a difference image D, and convert D into a grayscale image D gray
[0023] (1-1-3) for D gray Binarize and normalize to get 0, 1 represents the difference map D b晦晦l , then D b晦晦l Sum and get the difference value;
[0024] (1-1-4) Calculate the differential value for each frame (except the two frames of the head view) and merge them to obtain the differential vector.
[0025] Furthermore, in step (3), the joint angle calculation is achieved by the following method:
[0026] (3-1) The joint angle is determined by the two connected vectors Angle definition, the vertex is the starting point of the two vectors, the counterclockwise direction is positive, and the value range is
[0027] (3-2) Angle value: Value range (0,π);
[0028] (3-3)Angle direction: Since the image coordinate system is left-handed, we have:
[0029]
[0030] (3-4) Final angle:
[0031]
[0032] Furthermore, in step (3), the joint angles include: two shoulder angles, two elbow angles, two hip angles, two knee angles, and a neck angle.
[0033] Furthermore, anomaly detection is performed on the final angle obtained in step (3-4), and the specific method is as follows:
[0034] (3-4-1) The joint angle group vectors calculated for each frame are concatenated into a joint angle timing matrix in time sequence;
[0035] (3-4-2) Set the current frame angle θ n Angle θ with the previous framen-1 Difference e n The difference between the previous group and e e-1 When the difference exceeds 90°, replace with θ n =θ n- +e n-1 , e n =e n-1 .
[0036] Furthermore, the obtained joint angle timing matrix is decomposed according to the timing axis to obtain the timing change sequence of each joint angle. The curve is smoothed using a Savitzky-Golay filter to eliminate the noise interference in the skeleton joint point extraction process. The filter sliding window width is set to 7 and the smoothing polynomial order is set to 1.
[0037] Furthermore, in step (4), after the skeleton joint points are extracted, the input posture and the template posture are aligned and weak rigid transformation is performed for alignment. The calculation method is as follows:
[0038] (4-1) Select the midpoints of the two shoulders and the two hips as the registration points;
[0039] (4-2) The transformation matrix is defined as follows:
[0040]
[0041] (4-3) The coordinates of two pairs of points before and after the transformation are known: (x 1 ,y 1 ), (x 2 ,y 2 ), Find the parameters: a, θ, p x , p y ; Let: Δx = x 1 -x 2 , Δy=y 1 -y 2 , Solve the expression of the parameter to be found:
[0042]
[0043]
[0044]
[0045]
[0046] The ± in sinθ and cosθ should have the same sign according to the constraint that the sum of squares is equal to 1, and the result obtained by the root should have a + sign.
[0047] Furthermore, in step (5), the action evaluation score is calculated by the following algorithm:
[0048] (5-1) Calculate the position error:
[0049]
[0050] Where N is the number of skeleton joints in the pose, ignoring facial joints (such as eye, ear, and nose to mark the head position), J i (x,y),J i_ref (x, y) is the image coordinate of the joint point;
[0051] (5-2) Calculate the joint angle error:
[0052]
[0053] Where N is the number of joint angles, and in the present invention, N=9; A i , A i_ref It is the joint angle after abnormality detection in claim 7, and the angle unit is rad.
[0054] (5-3) Position error normalization: The human body size (||cShoulder-cHip||) is used as the benchmark for regularization;
[0055] (5-4) Calculate the posture similarity evaluation:
[0056]
[0057] Where ω is the weight of the joint angle error, and the score range is (0, 1). The larger the value, the higher the score.
[0058] (5-5) Scoring the action video: Evaluate the similarity of all key frames and take a weighted average to get the score of the entire video.
[0059] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0060] 1. Simple and convenient deployment, no special requirements for hardware equipment;
[0061] 2. The system can realize real-time evaluation, greatly improving the efficiency of use;
[0062] 3. Introduce an automatic evaluation method based on deep learning computer vision, which does not require manual evaluation by experts. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 is a flow chart of the method of the present invention;
[0064] Figure 2 Demonstrated the system graphical interactive interface based on PyQt;
[0065] Figure 3 The smoothing effect of the joint angle curve is shown;
[0066] Figure 4 The key frame extraction result graph is shown. DETAILED DESCRIPTION
[0067] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0068] like Figure 1-2 An action evaluation method based on monocular RGB video is shown, and the method comprises the following steps:
[0069] (1) The input video is collected in real time by an RGB monocular camera. The template video can be saved in a file in advance. The static key frames are extracted from the motion video. The extraction method is as follows:
[0070] (1-1) Calculate the inter-frame difference vector of the video. The calculation method is as follows:
[0071] (1-1-1) Save the current frame and the two frames before the current frame;
[0072] (1-1-2) Calculate the absolute value of the difference between two adjacent frames and obtain two difference images D 1 , D 2 ;
[0073] (1-1-3) D 1 , D 2 Use the bitwise AND operation of the image to synthesize a difference image D;
[0074] (1-1-4) Convert D into grayscale image D gray ;
[0075] (1-1-5) for D gray Binarize and normalize to get the difference map D represented by 0 and 1 bool ;
[0076] (1-1-6) for D bool Sum and get the difference value;
[0077] (1-1-7) Calculate the differential value for each frame (except the first two frames of the video) and merge them to obtain the differential vector.
[0078] (1-2) Set the identifiers: static state identifier is_static, static start position identifier Begin, and static end position identifier End;
[0079] (1-3) Set the stillness threshold: 0.2 times the maximum value of the inter-frame difference in the video sequence;
[0080] (1-4) The steps of still frame extraction are as follows:
[0081] (1-4-1) Set the starting state is_static = True, Begin = -1 (do not record the initial action);
[0082] (1-4-2) When the difference value is greater than the threshold, set is_static = False;
[0083] (1-4-3) When the difference value is less than the threshold and is_static = False, mark the current frame as the static starting position and is_static = True;
[0084] (1-4-4) When the difference value is greater than the threshold and is_static=True, if Begin!=-1, mark the current frame as the static end position, and is_static=False, select the frame with the smallest difference value in the Begin-End interval as the static frame;
[0085] (1-4-5) Repeat steps (1-4-3) and (1-4-4) until the video ends.
[0086] (2) Using HR-Net to extract skeleton joints from the static keyframes extracted in step (1), a human skeleton joint consisting of 17 points is obtained. In the subsequent processing, the four joints of the eyes and ears, which are relatively fixed and have no effect on the human posture, are discarded. The facial features only retain the nose joint as the identifier of the head position.
[0087] (3) The joint angles of the human skeleton joint points obtained in step (2) are further calculated. The joint angle selection and calculation method are as follows:
[0088] The joint angles used to evaluate the action include: two shoulder angles, two elbow angles, two hip angles, two knee angles, and neck angle. The above angles are selected based on several joints that significantly affect human movements. The specific angle definition can be kept consistent throughout the evaluation process.
[0089] The joint angle is determined by the two connected vectors Angle definition, the vertex is the starting point of the two vectors, the counterclockwise direction is positive, and the value range is
[0090] Angle value: Value range (0, π);
[0091] Angle direction: Since the image coordinate system is left-handed, we have:
[0092]
[0093] Final angle:
[0094]
[0095] (4) Perform anomaly detection on the angle calculated in step (3) as follows:
[0096] The joint angle group vectors calculated for each frame are concatenated into a joint angle timing matrix in time sequence;
[0097] Set the current frame angle θ n Angle θ with the previous frame n-1 Difference e n The difference between the previous group and e e-1 When the difference exceeds 90°, replace with θ n =θ n-1 +e n-1 , e n =e n-1 ;
[0098] (5) The joint angle timing matrix obtained in step (4) is decomposed according to the timing axis to obtain the timing change sequence of each joint angle. The Savitzky-Golay filter is used to smooth the curve to eliminate the noise interference in the skeleton joint point extraction process. The filter sliding window width is set to 7 and the smoothing polynomial order is set to 1, as shown in Figure 3 As described above, a joint curve smoothing effect diagram is obtained.
[0099] (6) The human skeleton joint points obtained in step (2) are further registered with the input posture and the template posture, and a weak rigid transformation is proposed for registration. The calculation method is as follows:
[0100] Select the midpoints of the two shoulders (Shoulder) and the two hips (Hip) as the registration points;
[0101] The transformation matrix is defined as follows:
[0102]
[0103] The coordinates of two pairs of points before and after the transformation are known: (x 1 ,y 1 ), (x 2 ,y 2 ),
[0104] Parameters to be determined: a, θ, p x , p y ;
[0105] Let: Δx = x 1 -x 2 , Δy=y 1 -y 2 ,
[0106] Solve the expression of the parameter to be found:
[0107]
[0108]
[0109]
[0110]
[0111] The ± in sinθ and cosθ should have the same sign according to the constraint that the sum of squares is equal to 1, and should have a + sign according to the experimental results.
[0112] (7) For the posture that has been registered in step (6), the action evaluation score is calculated using the following algorithm:
[0113] Position error:
[0114]
[0115] Where N is the number of skeleton joints in the pose, ignoring facial joints (such as eye, ear, and nose to mark the head position), J i (x,y),J i_ref (x, y) is the image coordinate of the joint point;
[0116] Joint angle error:
[0117]
[0118] Where N is the number of joint angles, and in the present invention, N=9; A i , A i_ref is the joint angle after abnormality detection in claim 7, the angle unit is rad;
[0119] Position error regularization: Regularization is performed based on the human torso size (||cShoulder-cHip||);
[0120] Pose similarity evaluation:
[0121]
[0122] Where ω is the weight of the joint angle error, and the score range is (0, 1). The larger the value, the higher the score.
[0123] Action Video Rating: Figure 4 As shown, all key frames are evaluated for similarity and weighted averaged to obtain the score of the entire video.
[0124] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.
Claims
1. An action evaluation method based on monocular RGB video, characterized by: The method comprises the following steps: (1) Input an action video and extract static key frames; (2) extracting skeleton joint points from the static key frame obtained in step (1); (3) calculating the joint angles of the skeleton joint points obtained in step (2); (4) performing step (2) on the input video and the template video respectively to obtain skeleton joint points, and using the skeleton joint points to align the input posture with the template posture; In step (4), after the skeleton joint points are extracted, the input posture and the template posture are registered by performing weak rigid transformation. The calculation method is as follows: (4-1) Select the midpoints of the two shoulders and the two hips as the registration points; (4-2) The transformation matrix is defined as follows: (4-3) Given the coordinates of two pairs of points before and after the transformation: (x1, y1), (x2, y2), Find parameters: a, θ, p x ,p y ; Let: Δx=x1-x2, Δy=y1-y2, Solve the expression of the parameter to be found: The values of sinθ and cosθ have the same sign according to the constraint that the sum of squares is equal to 1, and are positive numbers according to the experimental results; (5) Perform motion evaluation on the registered posture obtained in step (4), using the coordinates of the registered skeleton joint points and the joint angles obtained in step (3) as evaluation benchmarks.
2. The action evaluation method based on monocular RGB video according to claim 1, characterized in that: In step (1), after inputting the action video, the inter-frame difference method is used to extract the still frame. The specific method is as follows: (1-1) Calculate the inter-frame vector of the video; (1-2) Set the identifiers: static state identifier is_static, static start position identifier Begin, and static end position identifier End; (1-3) Set the stillness threshold: 0.2 times the maximum inter-frame difference in the video sequence; (1-4) Set the starting state is_static=True, Begin=-1. When the difference value is greater than the threshold, set is_static=False; when the difference value is less than the threshold and is_static=False, mark the current frame as the static start position, and is_satic=True; when the difference value is greater than the threshold and is_static=True, if Begin!=-1, mark the current frame as the static end position, and is_static=False, select the frame with the smallest difference between Begin-End as the static frame; (1-5) Repeat steps (1-3) and (1-4) until the video ends.
3. The action evaluation method based on monocular RGB video according to claim 1, characterized in that: In step (2), the PyQt submodule interface is called to visually delete any unnecessary frames.
4. The action evaluation method based on monocular RGB video according to claim 2, characterized in that: In step (1-1), the specific method for calculating the inter-frame difference vector of the video is as follows: (1-1-1) Save the current frame and the two frames before the current frame, calculate the absolute value of the difference between the two adjacent frames, and obtain two difference values D1 and D2; (1-1-2) Use the bitwise AND operation on D1 and D2 to synthesize a difference image D, and convert D into a grayscale image D gray ; (1-1-3) for D gray Binarize and normalize to get 0,1 to represent the difference map D bool , then D bool Sum and get the difference value; (1-1-4) Calculate the differential value for each frame and merge them to obtain the differential vector.
5. The action evaluation method based on monocular RGB video according to claim 1, characterized in that: In step (3), the joint angle calculation is achieved by the following method: (3-1) The joint angle is determined by the two connected vectors Angle definition, the vertex is the starting point of the two vectors, the counterclockwise direction is positive, and the value range is (3-2) Angle value: Value range (0,π); (3-3)Angle direction: Since the image coordinate system is left-handed, we have: (3-4) Final angle:
6. The action evaluation method based on monocular RGB video according to claim 1, characterized in that: In step (3), the joint angles include: two shoulder angles, two elbow angles, two hip angles, two knee angles, and a neck angle.
7. The action evaluation method based on monocular RGB video according to claim 5, characterized in that: The final angle obtained in step (3-4) is subjected to abnormal detection. The specific method is as follows: (3-4-1) The joint angle group vectors calculated for each frame are concatenated into a joint angle timing matrix in time sequence; (3-4-2) Set the current frame angle θ n Angle θ with the previous frame n-1 Difference e n The difference between the previous group and e n-1 When the difference exceeds 90°, replace with θ n =θ n-1 +e n-1 ,e n =e n-1 .
8. The action evaluation method based on monocular RGB video according to claim 7, characterized in that: The obtained joint angle timing matrix is decomposed according to the timing axis to obtain the timing change sequence of each joint angle. The Savitzky-Golay filter is used to smooth the curve to eliminate the noise interference in the skeleton joint point extraction process. The filter sliding window width is set to 7 and the smoothing polynomial order is set to 1.
9. The method for action assessment based on monocular RGB video according to claim 1, characterized in that: In step (5), the action evaluation score is calculated by the following algorithm: (5-1) Calculate the position error: Where N is the number of skeleton joints in the pose, ignoring the facial joints, and J i (x,y),J i_ref (x,y) is the image coordinate of the joint point; (5-2) Calculate the joint angle error: Where N is the number of joint angles, N = 9; A i ,A i_ref is the joint angle after abnormality detection, the angle unit is rad; (5-3) Position error normalization: Regularization is performed using the human body size (||cShoulder-cHip||) as the benchmark; (5-4) Calculate the posture similarity evaluation: Score=e -[(1-w) *err pos +w*err angle ] Where ω is the weight of the joint angle error, and the score range is (0,1). The larger the value, the higher the score. (5-5) Scoring the action video: Evaluate the similarity of all key frames and take a weighted average to get the score of the entire video.
