Automatic evaluation method of sit-up action quality based on posture key points

Abstract

The present invention belongs to the technical field of video action quality assessment, and specifically discloses a sit-up action quality automatic assessment method based on posture key points. The method first integrates the global attention mechanism GAM and the multi-scale attention mechanism EMA into the YOLOv8-Pose network, so that the action posture estimation and key point detection capabilities are greatly improved, and the detection capability of human body posture and the positioning capability of skeleton key points are effectively improved, so that the action quality prediction score of the scoring network is more accurate. The scoring network uses the idea based on skeleton key points to extract the six skeleton key points that best represent the sit-up action in the sit-up video image, and finally uses the weighted scoring method to score the action quality according to the different characteristics of the key angles of the action points in each action stage of the sit-up, which is conducive to improving the accuracy of the automatic scoring of the quality of the sit-up action completion.

Description

Technical Field

[0001] The invention belongs to the technical field of video action quality assessment, and in particular relates to an automatic sit-up action quality assessment method based on posture key points. Background Art

[0002] Action Quality Assessment (AQA) is an automated technology that can monitor and evaluate the quality of human movements in real time, and has obvious advantages in the field of sports testing. There are currently two main processing strategies for action quality assessment based on video analysis and understanding. The first strategy is to directly process and analyze the video stream, and provide a basis for action quality assessment by extracting action features and their spatiotemporal correlations in the video. The second strategy is to analyze and evaluate the completion quality of the action based on the motion trajectory of the key points of the skeleton. The method based on skeleton key point analysis requires the extraction of skeleton features of the video, which may be affected by the performance of skeleton extraction. However, since it can effectively reduce the amount of calculation, it only needs to focus on tracking the motion features such as joint angles and speeds of key points, so it has also attracted much attention in the field of action recognition and action quality assessment.

[0003] Sit-ups are a routine physical test item in primary and secondary school and university physical education. Currently, it mainly relies on physical education teachers to conduct assessments by visual counting. The quality of the subjects' movements depends on the professional experience of the referees, which is highly subjective. It is not only time-consuming and laborious, but also easy to miss irregular movements and it is difficult to make reasonable deductions for irregular movements. At present, although the video-assisted referee movement quality assessment system in large-scale sports competitions has achieved certain application effects. However, in the field of small-scale sports tests for the general public, intelligent evaluation and movement quality analysis and automatic evaluation technologies are not mature enough, and there are problems such as long detection time and insufficient accuracy. At the same time, there is a lack of reasonable scoring standards and basis for deducting points for irregular movements. For example:

[0004] Patent document 1 provides a sit-up action standard detection method, device and storage medium for detecting whether the sit-up action is standard. The sit-up action standard detection method includes collecting a video image of a sit-up; starting from the i-th frame of the video image, calculating the standardization score of the sit-up action from the i-th frame to the i+n1-th frame; when the standardization score>0, the sit-up action is determined to be standard, otherwise the sit-up action is determined to be non-standard. This patent document only determines whether the action is standard, and the non-standard score is directly 0 points, and there is no specific scoring and deduction standard for non-standard actions.

[0005] Patent document 2 provides a comprehensive physical fitness assessment method, system and storable medium based on machine vision, which obtains, processes and judges the site environment data of the tester, obtains the tester's action video data when the first environmental condition is met, obtains the tester's first key point information and first key point coordinate information based on the action video data, and determines the completion status based on the above information; when the second environmental condition is met, the tester's action video data is also obtained, the action video data is analyzed frame by frame, and the corresponding second key point coordinate information is obtained based on the second key point information, and the tester's action state is determined to be pull-ups based on the second key point coordinate information, and the tester's action completion information is determined based on the second preset condition.

[0006] The patent document can detect the tester's sit-ups and pull-ups completion status and display the information in real time.

[0007] However, the patent document 2 requires a large number of key points. Some key points, such as eyes and eyebrows, are almost worthless for judging the sit-up movements and may also slow down the data processing time. In addition, the patent document 2 only conducts a counting study on whether the movements are standard or not, and does not formulate detailed movement scoring and deduction standards.

[0008] References

[0009] Patent Document 1 China Invention Patent Application Publication Number: CN118015706A, Publication Date: 2024.05.10;

[0010] Patent document 2 Chinese invention patent application publication number: CN117877120A, publication date: 2024.04.12. Summary of the invention

[0011] The purpose of the present invention is to propose an automatic evaluation method for sit-up action quality based on posture key points. The method proposes an improved key posture estimation and key point detection network based on the YOLOv8-Pose network, performs action posture estimation and key point detection on key action frames of a video, and performs weighted scoring of the sit-up action quality through key angle features, which is beneficial to improving the accuracy of automatic scoring of the completion quality of a single complete sit-up action.

[0012] In order to achieve the above object, the present invention adopts the following technical scheme:

[0013] The method for automatically evaluating the quality of sit-up movements based on posture key points includes the following steps:

[0014] Step 1. First, collect the sit-up video dataset, extract and annotate the key action frames of the collected videos;

[0015] Step 2. Build a key pose estimation and key point detection network based on the improved YOLOv8-Pose model, and pass the key frames extracted in step 1 into the key pose estimation and key point detection network for pose estimation and key point detection;

[0016] Step 3. Finally, the key point data obtained by the key posture estimation and key point detection network are sent to the score evaluation network for weighted scoring of key action points to complete the quality evaluation of the sit-up action.

[0017] The present invention has the following advantages:

[0018] As described above, the present invention relates to an automatic evaluation method for sit-up action quality based on posture key points. The method develops detailed scoring criteria and weighted scoring schemes according to the characteristics of sit-up actions. It is more detailed than the existing technology that simply counts sit-ups, and can perform detailed, reasonable and effective evaluation of the quality of sit-up actions. Specifically, the present invention first integrates the global attention mechanism (Global Attention Mechanism, GAM) and the multi-scale attention mechanism (Efficient Multi-Scale Attention, EMA) into the YOLOv8-Pose network, which can greatly improve the ability to estimate the action posture and detect key points, effectively improve the ability to detect human posture and the ability to locate skeletal key points, and thus make the action quality prediction score of the scoring network more accurate. The scoring network proposed in the present invention uses the idea based on skeleton key points to extract the six skeleton key points that best represent the sit-up action in the sit-up video image, overcomes the influence of the background environment, and can improve the model speed to a certain extent. Finally, the weighted scoring method is used to score the action quality according to the different characteristics of the key angles of the action points in each action stage of the sit-up, which is conducive to improving the accuracy of the automatic scoring of the completion quality of a single complete sit-up action. The present invention studies the specific physical test item of sit-ups, which is closer to actual needs and can provide a reasonable reference solution for the teaching and testing of sit-ups. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Flow chart of a method for automatically evaluating sit-up action quality based on posture key points in an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of video key frame extraction in an embodiment of the present invention;

[0021] Figure 3 It is a network diagram of key pose estimation and key point detection based on the improved YOLOv8-Pose model in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of a GAM module in an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of an EMA module in an embodiment of the present invention;

[0024] Figure 6 This is a score evaluation network structure diagram in an embodiment of the present invention;

[0025] FIG. 7 is a schematic diagram of four standard postures for completing a sit-up;

[0026] in Figure 7 P1 in the figure is the standard posture at the beginning. Figure 7 P2 in the figure is the standard posture of raising the upper body. Figure 7 P3 in the figure is the standard posture of lifting the abdomen. Figure 7 P4 in the figure is the standard posture at the end of sit-ups;

[0027] Figure 8 A schematic diagram of capturing skeleton key points in an embodiment of the present invention;

[0028] in Figure 8 (a) is a schematic diagram of the six skeleton key points and their coordinates. Figure 8 (b) in the figure is the key frame in the collected sit-up video. The red box marks the area of ​​interest. Figure 8 (c) The 6 skeleton key points captured in this frame. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0030] This embodiment describes a method for automatically evaluating the quality of sit-up movements based on posture key points. Figure 1 As shown, the method for automatically evaluating the quality of sit-up movements comprises the following steps:

[0031] Step 1. First, collect the sit-up video dataset, and extract and annotate the key action frames of the collected videos.

[0032] Since there is no video dataset of sit-up action in the existing public dataset, the present invention builds a video dataset of sit-up action, which can be defined as SDUST-Situp. The SDUST-Situp dataset contains 32 groups of 108 videos, each group contains multiple continuous videos of standard and non-standard sit-up action.

[0033] The sit-up action in the video is annotated with an action quality score, and then the video is subjected to frame extraction.

[0034] The preprocessing operations are as follows:

[0035] Step 1.1. First, use the uniform frame extraction strategy to extract 16 frames from each video; then use the video similarity learning network ViSiL to estimate the intra-frame similarity of the 16 frames of each video, remove redundant frames with high similarity, and finally retain 4 key frames, namely, the start, upper body lifting, abdomen lifting, and end key frames, such as Figure 2 This method not only retains the key features of a single sit-up action, but also reduces the size of the data that needs to be processed.

[0036] The uniform frame extraction strategy is as follows:

[0037] First, read the input sit-up video file, get the total number of frames of each video, and set a fixed threshold of 16 frames for extraction. Calculate the frame extraction interval step according to the total number of video frames (step = total number of frames / 16), then extract a frame of image every step frame for saving. After the loop processing, 16 frames of images are evenly saved for each video.

[0038] For each video, the 16 frames of images saved after uniform frame extraction are used to calculate the inter-frame similarity using the ViSiL similarity learning network, and redundant frames with high similarity are removed. The specific process is as follows:

[0039] First, a convolutional neural network (CNN) is used to sequentially extract the spatial information of 16 frames of each video, including features such as color, texture, and shape. After extracting the spatial features, a recurrent neural network (RNN) is further used to capture the temporal information between frames, such as motion and change features.

[0040] Secondly, the bilinear fusion method is used to fuse the spatiotemporal features, and the cosine similarity between the features of every two frames is calculated.

[0041] Finally, in order to remove redundant frames with high similarity, a similarity threshold is set, for example, 0.8. When the similarity between two frames exceeds the threshold, the previous frame is retained and the next frame is removed. The processing is performed in sequence, and finally 4 key frames are retained.

[0042] Step 1.2. Label the extracted key frames using the lightweight graphics annotation software Labelme.

[0043] The key postures are annotated with the minimum circumscribed rectangle of the target, and the labels are Supine, Lift, Rise, and Achieve, representing the start, upper body lifting, abdomen lifting, and end movements respectively; the key point annotation labels of the human body posture are wrist, elbow, shoulder, hip, knee, and ankle, representing the six posture key points of wrist, elbow, shoulder, hip, knee, and ankle respectively.

[0044] After the above processing, 404 key posture data and 2424 key point data can be obtained. The annotation results are sent to the key posture estimation and key point detection network for learning and training to obtain the key posture and key point detection results.

[0045] Step 2. Build a key pose estimation and key point detection network based on the improved YOLOv8-Pose model, and pass the key frames extracted in step 1 into the key pose estimation and key point detection network for pose estimation and key point detection.

[0046] Since the YOLOv8-Pose model has excellent performance in key point detection, the present invention improves the YOLOv8-Pose model and uses it for posture estimation and key point detection of sit-up actions. The model structure is as follows: Figure 3 shown.

[0047] The improved YOLOv8-Pose model includes a backbone network, a neck network, and a detection head network.

[0048] The backbone network introduces the global attention mechanism GAM, which enables the network to pay more comprehensive attention to the feature areas related to the human body and effectively integrate the contextual information between different channels of the feature map to improve the accuracy and robustness of human posture detection; then, the spatial pyramid pooling layer is used to form aggregated features at multiple scales; and the multi-scale attention mechanism EMA is introduced to adaptively emphasize the importance of features at different scales, thereby improving the detection and positioning capabilities of key points of the human skeleton.

[0049] The output of the backbone network is then used to fuse and enhance the features of the neck network. Finally, the detection head network is used to make decisions and generate the final detection. The YOLOv8-Pose model finally outputs key poses and key point detection results.

[0050] The backbone network includes convolutional modules, C2f modules, global attention mechanism GAM modules, spatial pyramid pooling layers, and multi-scale attention mechanism EMA modules, such as Figure 3 The signal processing flow in the backbone network is as follows:

[0051] The input key frame image sequence first passes through two convolution modules for image feature extraction, and then uses a C2f module to capture the complex features in the image, and then passes through a convolution module for image feature extraction, and then uses a C2f module to capture the complex features in the image, and then passes through a convolution module for image feature extraction, and then enters the global attention mechanism GAM, so that the network pays more attention to the feature areas related to the human body and effectively integrates the contextual information between different channels of the feature map; then the spatial pyramid pooling layer is used to form aggregated features at multiple scales; and then the multi-scale attention mechanism EMA is introduced to adaptively emphasize the importance of features at different scales to improve the detection and positioning capabilities of key points of the human skeleton.

[0052] The specific structure of the global attention mechanism GAM module is as follows Figure 4 It can effectively extract feature information across channels and three dimensions while maintaining the integrity of the original information.

[0053] The global attention mechanism GAM module includes the channel attention module M C and the spatial attention module M S .

[0054] The input feature F1 first passes through the channel attention module M C Further capture its important information F2 in the spatial dimension; then, F2 enters the spatial attention module M S Further processing is performed to highlight highly correlated channel features.

[0055] The processing flow is shown as follows:

[0056] (1)

[0057] in represents element-wise multiplication, , They represent channel attention processing and spatial attention processing respectively.

[0058] By introducing the GAM module, the YOLOv8-Pose network can pay more attention to the feature areas related to the human body and effectively integrate contextual information between different channels of the feature map to improve the accuracy and robustness of human posture detection.

[0059] The specific structure of the multi-scale attention mechanism EMA module is as follows Figure 5It uses a parallel substructure to reduce the network depth, and without reducing the channel dimension, it generates better pixel-level attention for high-level feature maps, improving multi-dimensional perception and multi-scale feature extraction capabilities. The signal processing flow in the multi-scale attention mechanism EMA module is as follows:

[0060] The input aggregated features are first divided into multiple sub-features to form feature groups, and the feature groups are processed through three parallel lines in two branches. The first branch is a 1*1 branch. The first branch contains two parallel lines, which use one-dimensional horizontal global pooling and one-dimensional vertical pooling to encode the feature group along two spatial directions, and then connect the two encoded features and share a 1*1 convolution. Then, the output of the 1*1 convolution is decomposed into two vectors and passed through a nonlinear Sigmoid activation function respectively, and reweighted with the feature group and grouped normalized. Finally, the features are reshaped by average pooling and normalized using the Softmax normalization function; the second branch is a 3*3 branch. The feature group captures local cross-channel interactions through 3*3 convolutions to expand the feature space. , then reshape the features through average pooling and normalize them using the Softmax normalization function, and then use the Matmul function to perform matrix multiplication with the grouped and normalized features of the first branch to obtain the first feature matrix; at the same time, the features after 3*3 convolution use the Matmul function to perform matrix multiplication with the features normalized by the first branch normalization function to obtain the second feature matrix; then add the first feature matrix and the second feature matrix, and then generate the attention weight matrix through the Sigmoid activation function; finally, the input feature group and the attention weight matrix are reweighted to obtain the output features optimized by the EMA attention mechanism.

[0061] By introducing the EMA module, the present invention enables the YOLOv8-Pose network to adaptively emphasize the importance of features at different scales, thereby improving the detection and positioning capabilities of key points of the human skeleton.

[0062] Taking into account the action characteristics and actual measurement environment of sit-ups, the present invention proposes a Situp-PoseNet model based on the YOLOv8-Pose network, and integrates the global attention mechanism (GAM) and the efficient multi-scale attention mechanism (EMA) into the Yolov8-Pose network, which effectively improves the ability to detect human posture and locate skeleton key points, making the evaluation result of the scoring network more accurate.

[0063] Step 3. Finally, the key point data obtained by the key posture estimation and key point detection network are sent to the score evaluation network for weighted scoring of key action points to complete the quality evaluation of the sit-up action.

[0064] The present invention designs a score evaluation network for the quality of sit-up action completion and weighted scoring based on video analysis and understanding. Specifically, four key stages (postures) in the sit-up process are selected as the evaluation objects, and different weights are assigned to the four key postures. Each key posture has three action points, including holding the head with both hands, bending the knees at 90 degrees, and folding the body, and each action point also has its own weight. According to the action points of each stage, the action completion quality and irregular actions are evaluated, thereby completing the weighted scoring of the action quality of sit-ups.

[0065] Compared with the previous manual scoring strategies with strong subjectivity and the methods that only focus on action counting and easily ignore the action quality, the scoring strategy proposed in the present invention pays more attention to the evaluation of the stages, coherence and degree of action standardization of the actions.

[0066] By subdividing key postures and action points and assigning different weights to them, the method of the present invention can more reasonably reflect the accuracy, standardization, and importance of the actions, help learners better understand the correct posture and action points of sit-ups, and make them pay more attention to the quality and standardization of the actions rather than just pursuing quantity, thereby achieving more scientific and fairer scoring.

[0067] The score evaluation network structure is as follows Figure 6 As shown, the overall processing idea is as follows:

[0068] After the key posture estimation and key point detection network obtains the key point positions and corresponding coordinates of each sit-up key frame, they are sent to the scoring network to calculate the key point angles, allocate weight coefficients and judge the deduction items. Finally, the weighted sum is taken to obtain the final action quality prediction score, thereby completing the quality evaluation of the sit-up action.

[0069] According to the 1-minute sit-up test standard, in order to achieve the quality evaluation of the sit-up action, four key stages (postures) in the sit-up process are selected as the evaluation objects, namely:

[0070] The sit-up starting posture P1 has the following movement characteristics: the subject lies on his back on a cushion with his shoulder blades touching the cushion, his knees bent at about 90 degrees, and his hands covering his head; the upper body lifting posture P2 has the following movement characteristics: the subject lifts his upper body off the cushion and keeps his hands covering his head; the abdomen lifting posture P3 has the following movement characteristics: the subject relies on the abdominal strength to complete the sit-up stage and keeps his hands covering his head; the sit-up ending posture P4 has the following movement characteristics: the subject sits up with his elbows touching or exceeding his knees.

[0071] The standard postures from P1 to P4 are shown as follows Figure 7 As shown. P1 is the standard posture at the beginning of the test, P2 is the standard posture of upper body lifting, P3 is the standard posture of abdomen lifting, and P4 is the standard posture at the end of sit-ups.

[0072] To automatically score the quality of sit-up movements based on video analysis and understanding, it is necessary to accurately track the positions of the skeleton key points in the above four key stages and match them with the standard posture to obtain a score.

[0073] like Figure 8 As shown in (a), six key points A to F in the video action skeleton are selected as tracking targets for posture estimation; point A is located at the shoulder joint, with coordinates marked as (x1, y1); point B is located at the hip joint (x2, y2), point C is located at the knee joint (x3, y3), point D is located at the elbow joint (x4, y4), point E is located at the wrist joint (x5, y5), and point F is located at the ankle joint (x6, y6). Figure 8 (b) is an example frame of the P3 stage, and the region of interest is marked with a rectangular box. Figure 8 (c) in the figure shows the 6 key points captured in this frame.

[0074] In order to evaluate the quality of action completion at each stage, the importance of key postures and the quality of action completion of each key posture are judged; first, different weights are assigned to the above four key stages, namely P1 is 0.3, P2 is 0.2, P3 is 0.2, and P4 is 0.3. Then, the quality of action completion and irregular actions are evaluated according to the key points of each stage. The evaluation criteria are as follows: According to the angle matching degree of the key points, the corresponding scores are given to the head holding hands, 90° knee bending, and the degree of body folding.

[0075] The specific weight and score of each action point are shown in Table 1.

[0076] Table 1. Sit-up scoring method based on key point angles

[0077]

[0078] Specifically, the specific process of evaluating the quality of action completion and non-standard actions is as follows:

[0079] The standardization of the action of holding the head with both hands is determined by the folding angle ∠D of the arms; when ∠D is less than or equal to 45°, it is considered a standard action of holding the head with both hands and gets 5 points;

[0080] When ∠D is greater than 45°, the scores are 4, 3, 2, 1, and 0 respectively according to the degree of non-standardness of the movement.

[0081] The calculation formula of ∠D is as follows:

[0082] (2)

[0083] Whether the subject's knee flexion is 90° is determined by the angle ∠C of the legs; when the angle ∠C is greater than 80° and less than 95°, it is considered a standard knee flexion action and is scored 5 points;

[0084] When the angle ∠C is not within the above-mentioned angle range, the scores are 4, 3, 2, 1, and 0 points respectively according to the degree of non-standardness of the action.

[0085] The calculation formula of ∠C is as follows:

[0086] (3)

[0087] Whether the subject lies on his back and completes the sit-up is judged based on the degree of body folding, that is, when the body folding angle ∠B is greater than or equal to 120°, it is considered to be a standard supine action and is scored 5 points;

[0088] When ∠B is less than 120°, the scores are 4, 3, 2, 1, and 0 respectively according to the degree of irregularity of the movement.

[0089] The standard degree of sit-up is determined by ∠B', that is, when the body's folding angle ∠B' is less than or equal to 60° and the elbows touch or exceed the knees, the sit-up is completed. 5 points;

[0090] When ∠B' is greater than 60°, the scores are 4, 3, 2, 1, or 0 depending on the degree of irregularity of the movement.

[0091] The calculation formulas for ∠B and ∠B' are the same, as shown in formula (4):

[0092] (4)

[0093] In summary, based on the 4 key stages of sit-up videos, M , and 3 key action points in each stage MN That is, the standardization of holding the head with both hands, bending knees at 90 degrees, and folding the body. The final prediction score formula is:

[0094] S P = ∑ M = 1 4 [ ∑ N = 1 3 P M N ⋅ α M N ] ⋅ β M (5)

[0095] In the formula, S P is the final prediction score, α MN is the weight of the key action points in each stage, M is the key stage number, N is the key action point number, β M is the weight of each key stage.

[0096] There is currently no public dataset of sit-up action videos. In order to verify the effectiveness of the method of the present invention, the present invention has built a sit-up video dataset SDUST-Situp. The dataset contains 32 groups of 108 .mp4 videos, each video clip is about 2 to 3 seconds long, and the camera equipment uses an iPhone13Pro mobile phone. The sit-up test environment is arranged according to the "1-minute sit-up" test standard. For each volunteer, the complete action of the sit-up from start to finish is captured. Videos with obvious blur, jitter or poor lighting will be eliminated. In addition, static images at the beginning and end will also be eliminated. The resolution and frame rate of the video are standardized, with a resolution of 720×720 and a frame rate of 30.

[0097] The ratio of the training set and the validation set is 4:1, that is, 324 frames for the training set and 80 frames for the validation set.

[0098] For the sit-up posture detection and key point detection tasks, the present invention uses mAP as the evaluation index, which are mAP@0.5 and mAP@0.5-0.95. mAP@0.5 represents the average accuracy when the IoU value is 0.5, and mAP@0.5-0.95 represents the average accuracy of different IoU values ​​(from 0.5 to 0.95, with a step size of 0.05).

[0099] Among them, IoU (Intersection over Union) represents the degree of overlap between the predicted value and the true value.

[0100] The detection accuracy of key points is measured using mAP Pose As an evaluation index, its calculation formula is as follows:

[0101] m A P P o s e = 1 N ∑ i = 1 N [ A P P o s e ] i (6)

[0102] In the formula, N represents the number of key point categories.

[0103] AP Pose It represents the average accuracy of key point detection, which is calculated as follows:

[0104] (7)

[0105] OKS (object keypoint similarity) represents the similarity between the real keypoint and the predicted keypoint, which is calculated by the following formula:

[0106] (8)

[0107] Among them, d pi represents the Euclidean distance between the detected position and the true position of the i-th key point, Sp represents the scale factor of point p, v pi Indicates the visibility of the key point, 0 means unlabeled, 1 means the label is blocked, and 2 means the label is visible.

[0108] δ i represents the normalization factor of key points of type i. pi >0, δ takes 1, when v pi When ≤0, δ is 0.

[0109] T is a given threshold, when OKS p >T, β is OKS p , otherwise β is 0.

[0110] The detection accuracy of action posture is measured using mAP Box As an evaluation index, the calculation formula is as follows:

[0111] m A P B o x = 1 M ∑ i = 1 M [ A P B o x ] i (9)

[0112] Where M represents the number of action posture categories. Box It represents the average accuracy of motion posture detection, which is obtained by calculating the area under the PR curve. The area can be calculated by integration. The calculation formula is:

[0113] (10)

[0114] In the formula, P represents precision and R represents recall. The calculation formula is as follows:

[0115] (11)

[0116] (12)

[0117] Where s represents a certain action posture, non-s represents other states or actions except s; TP s Indicates the number of frames correctly classified as s; FP s Indicates the number of frames misclassified as s; FN S Indicates the number of frames misclassified as non-s.

[0118] The prediction accuracy of action quality score uses mean square error (MSE) and Spearman's rank correlation coefficient ρ as evaluation indicators to reflect the correlation between the predicted score and the true score.

[0119] Let S P Represents the prediction score of the model, SG represents the expert's annotation score (the score range is 0~5), then the calculation formulas for the mean square error MSE and the Spearman rank correlation coefficient ρ are:

[0120] (13)

[0121] (14)

[0122] In the formula, L represents the number of videos, S l P and S l G Represent the predicted score and true score of the lth video respectively. l P ) and R(S l G ) represent the score series ranking of the predicted score and the true score, respectively, and the superscript l represents the video sequence number.

[0123] The value range of ρ is [-1,1]. The higher the value, the better the score prediction performance of the network.

[0124] The operating environment of all experiments in the present invention is a 64-bit Windows 10 operating system, the CPU is an Intel (R) Xeon (R) Silver 4210R, the memory is 128GB, the graphics card is an NVIDIA RTX A6000, and the video memory is 48GB.

[0125] Programming is done in Python 3.8, using Pytorch-GPU 1.9.0 and CUDA 11.1. The input image size is 640×640, the batch size is 32, the initial learning rate is 0.01, the decay coefficient is 0.0005, the epochs is 300, and the AdamW optimizer is used to accelerate training. Other parameters not mentioned are the official default parameters of YOLOv8-pose.

[0126] Based on the YOLOv8-Pose network, this paper proposes an improved key pose estimation and key point detection network, performs action pose estimation and key point detection on key action frames of the video, and performs weighted scoring of sit-up action quality through key angle features. The overall model is named Situp-PoseNet. Table 2 lists the performance indicators of the Situp-PoseNet model of the present invention and other SOTA methods (PEPoseNet, ST-GCN, GDLT) on the SDUST-Situp dataset. Among them, PEPoseNet and ST-GCN are both models based on skeleton key points, while GDLT uses the Transformer framework for video streams.

[0127] For the comparison algorithm, except for modifying its final scoring module to be suitable for sit-up action quality assessment, no other changes were made. Whether it is posture estimation and key point detection performance, or action quality score prediction performance, Situp-PoseNet shows obvious advantages, and its mean square error MSE and Spearman rank correlation coefficient ρ reach 0.017 and 0.933 respectively, indicating that Situp-PoseNet has significantly improved its ability to capture the spatiotemporal correlation of key points in sit-up actions. Compared with the YOLOv8-Pose baseline model, the Spearman rank correlation coefficient increased by 7.6%, and the mean square error of the prediction score decreased by 4.3%.

[0128] Table 2 Action quality evaluation performance of different models on the SDUST-Situp dataset

[0129]

[0130] The PEPoseNet model combines the data set features of the human joint point coordinates with the key point coordinates of the pull-up and the sports equipment during training, but this training method may not be fully applicable to the detection of sit-up action features. Therefore, its key point estimation performance is relatively low, mAP Pose @0.5 / % is only 83.1%, resulting in slightly insufficient performance in motion quality assessment. In contrast, the ST-GCN model has stronger feature expression and generalization capabilities by automatically learning spatial and temporal features from data, making it perform better than PEPoseNet in key point estimation. However, ST-GCN has the problem of key point redundancy, and the scoring network may be affected by redundant key points during the calculation process, resulting in relatively poor score prediction performance. GDLT directly extracts features from video streams, and the motion features are easily affected by background factors such as the environment, and the motion quality score prediction performance is average. From the perspective of the mean square error MSE of the predicted score, GDLT has the worst MSE of 0.439, but its Spearman rank correlation coefficient ρ is not the worst, which is 0.753, indicating that its performance in the ranking of the predicted score series is better than the error performance between the predicted score and the true score. Although PEPoseNet, ST-GCN and GDLT have their own advantages and disadvantages in key point estimation and motion quality assessment, they still have certain reference value in practical applications.

[0131] Tables 3 and 4 list the ablation test results of pose detection and key point estimation and the ablation test results of action quality prediction score respectively. In order to verify the effect of adding GAM attention mechanism and EMA module on the accuracy of YOLOv8-Pose pose detection and key point estimation, the present invention uses ablation test for comparison. It can be seen that the pose detection and key point estimation capabilities of the YOLOv8-Pose benchmark model are the worst, with an average mAP of Box @0.5 / %,mAP Box @0.5-0.95 / % are 94.8% and 91.4% respectively, and the average mAP Pose @0.5 / %,mAP pose @0.5-0.95 / % are 94.8% and 93.5% respectively. This is because it is trained on the COCO dataset with 17 key points of the human body labeled. For the sit-up action, its posture detection will be redundant and the detected key points are relatively chaotic. After adding the GAM attention mechanism to the baseline model, the model pays more attention to the human posture interest area, and the posture detection and key point estimation performance are improved by 0.1%, 2.2%, 0.1%, and 1.4% respectively. After adding the EMA module to the baseline model, the model pays more attention to the skeleton key point features, which improves the key point estimation ability and the posture detection ability, which are improved by 0.6%, 1.7%, 0.6%, and 0.7% respectively. Finally, adding the GAM and EMA modules to the baseline model at the same time enables the model to further improve the key point estimation ability on the basis of extracting the human posture interest area, and the indicators are improved by 1.2%, 3%, 1%, and 2.2% respectively.

[0132] Table 3 Ablation experimental results of pose detection and key point estimation

[0133]

[0134] At the same time, the present invention also compares the impact of the YOLOv8-Pose model with the GAM attention mechanism and the EMA module on the performance of motion quality scoring, and the results are shown in Table 4. It can be seen that after adding the GAM module to the YOLOv8-Pose benchmark model, the model can pay more attention to the human body motion posture area of ​​interest, and the reduction of the feature area also improves the key point extraction ability, and the score prediction result is improved. The MSE decreases by 3.2% and ρ increases by 5.6%. When the EMA module is added to the benchmark model, the model pays more attention to the key point features of the human skeleton, the feature extraction ability becomes stronger, the score prediction results are improved, the MSE decreases by 1.1%, and ρ increases by 6.6%. When the GAM and EMA modules are added to the benchmark model at the same time, the two complement each other, so that the model can further improve the estimation ability of the key points on the basis of extracting the area of ​​interest, effectively improve the accuracy of the score evaluation, and its Spearman rank correlation coefficient ρ can reach 0.933.

[0135] Table 4 Ablation experiment results of action quality prediction scores

[0136]

[0137] The experimental results show that the Situp-PoseNet model proposed in the present invention can accurately and automatically score the quality of a single complete sit-up action, with a Spearman rank correlation coefficient of 0.933 and a mean square error of 0.017. It can be seen that the Situp-PoseNet model proposed in the present invention can effectively improve the accuracy of posture recognition and key point positioning by adding the GAM module and EMA attention mechanism to YOLOv8-Pose, proving the effectiveness of the improved method of the present invention.

[0138] The number of video frames and frame size have a direct impact on computing resources and motion quality assessment performance. Under the premise of ensuring the performance of motion quality assessment, minimizing the number of video frames and frame size can reduce computing costs and facilitate real-time detection on portable devices. Table 5 lists the experimental results of using different number of frames and frame sizes on the SDUST-Situp dataset.

[0139] Table 5 Evaluation performance of SDUST-Situp dataset with different frame numbers and frame sizes

[0140]

[0141] It can be seen from Table 5 above that, in comparison, the impact of frame size is greater than that of frame number. The larger the frame size, the higher the resolution, the more details that can be provided, and the better the score prediction performance of the model. For example, when the number of frames is 4, the MSE and ρ of the frame size of 320×320 are 0.082 and 0.878 respectively, while the MSE and ρ of the frame size of 1024×1024 are 0.016 and 0.937 respectively, and the performance improvement effect is obvious. Similar findings are also found when the number of frames is 8. However, the calculation time increases from 3.92 seconds to 8.02 seconds. Therefore, the selection of frame number and resolution needs to be compromised. From the data in the table, it can be considered that the comprehensive effect of 4 frames and 640×640 frame size is the best.

[0142] The method of the present invention proposes an improved key pose estimation and key point detection network based on the YOLOv8-Pose network, performs action pose estimation and key point detection on key action frames of a video, greatly improves the action pose estimation and key point detection capabilities, effectively improves the detection capabilities of human body poses and the positioning capabilities of skeleton key points, and makes the action quality prediction score of the scoring network more accurate. On this basis, the scoring network proposed in the present invention performs weighted scoring of the sit-up action quality through key angle features, which is beneficial to improving the accuracy of automatic scoring of the completion quality of a single complete sit-up action.

[0143] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. An automatic evaluation method for sit-up action quality based on posture key points, characterized in that: The steps include: Step 1. First, collect the sit-up video dataset, extract and annotate the key action frames of the collected videos; Step 2. Build a key pose estimation and key point detection network based on the improved YOLOv8-Pose model, and pass the key frames extracted in step 1 into the key pose estimation and key point detection network for pose estimation and key point detection; In the step 2, the improved YOLOv8-Pose model includes a backbone network, a neck network and a detection head network; The backbone network introduces the global attention mechanism GAM, which enables the network to pay more attention to the feature areas related to the human body and effectively integrate context information between different channels of the feature map to improve the accuracy and robustness of human posture detection; then the spatial pyramid pooling layer is used to form aggregated features at multiple scales; Then, by introducing the multi-scale attention mechanism EMA, the importance of features at different scales is adaptively emphasized, improving the detection and positioning capabilities of key points of the human skeleton. The output of the backbone network is used to fuse and enhance the features through the neck network. Finally, the detection head network is used to make decisions and generate the final detection. The improved YOLOv8-Pose model finally outputs the key pose and key point detection results. Step 3. Finally, the key point data obtained by the key posture estimation and key point detection network are sent to the score evaluation network for weighted scoring of key action points to complete the quality evaluation of the sit-up action.

2. The method for automatically evaluating the quality of sit-up movements based on posture key points according to claim 1, characterized in that: In step 1, the collected sit-up video data set includes multiple groups of videos; wherein each group of videos includes multiple continuous sit-up standard videos and non-standard action videos, and the following preprocessing operations are performed: Step 1.

1. First, use the uniform frame extraction strategy to extract 16 frames from each video; then use the video similarity learning network ViSiL to estimate the intra-frame similarity of the 16 frames of each video, remove redundant frames with high similarity, and finally retain 4 key frames, namely, the start, upper body lifting, abdomen lifting, and end key frames; Step 1.

2. Label the extracted key frames using the lightweight graphic labeling software Labelme; The key postures are annotated with the minimum circumscribed rectangle of the target, and the labels are Supine, Lift, Rise, and Achieve, representing the start, upper body lifting, abdomen lifting, and end movements respectively; the key point annotation labels of the human body posture are wrist, elbow, shoulder, hip, knee, and ankle, representing the six posture key points of wrist, elbow, shoulder, hip, knee, and ankle respectively.

3. The method for automatically evaluating the quality of sit-up movements based on posture key points according to claim 1, characterized in that: In step 2, the backbone network includes a convolution module, a C2f module, a global attention mechanism GAM module, a spatial pyramid pooling layer, and a multi-scale attention mechanism EMA module; the signal processing flow in the backbone network is as follows: The input key frame image sequence first passes through two convolution modules for image feature extraction, and then uses a C2f module to capture the complex features in the image, and then passes through a convolution module for image feature extraction, and then uses a C2f module to capture the complex features in the image, and then passes through a convolution module for image feature extraction, and then enters the global attention mechanism GAM, so that the network pays more attention to the feature areas related to the human body and effectively integrates the context information between different channels of the feature map; then the spatial pyramid pooling layer is used to form aggregated features at multiple scales; By introducing the multi-scale attention mechanism EMA, the importance of features at different scales is adaptively emphasized to improve the detection and positioning capabilities of key points of the human skeleton.

4. The method for automatically evaluating the quality of sit-up movements based on posture key points according to claim 1, characterized in that: The global attention mechanism GAM module includes the channel attention module M C and the spatial attention module M S ; The input feature F1 first passes through the channel attention module M C Further capture its important information F2 in the spatial dimension; then, F2 enters the spatial attention module M S Further processing to highlight highly correlated channel features; The processing flow is shown as follows: in Indicates element-wise multiplication, M C (F1), M S (F2) represent channel attention processing and spatial attention processing respectively.

5. The method for automatically evaluating the quality of sit-up movements based on posture key points according to claim 1, characterized in that: In step 2, the signal processing flow in the multi-scale attention mechanism EMA module is as follows: The input aggregated features are first divided into multiple sub-features to form feature groups, and the feature groups are processed through three parallel lines in two branches; the first branch is a 1*1 branch, which contains two parallel lines, and uses one-dimensional horizontal global pooling and one-dimensional vertical pooling to encode the feature groups along two spatial directions respectively, and then connects the two encoded features and shares a 1*1 convolution, and then decomposes the output of the 1*1 convolution into two vectors and passes through a nonlinear Sigmoid activation function respectively, and performs a reweighted operation with the feature group and performs group normalization, and finally reshapes the features through average pooling and uses the Softmax normalization function for normalization; the second branch is a 3*3 branch, and the feature group is captured through a 3*3 convolution. The local cross-channel interaction is obtained to expand the feature space, and then the features are reshaped by average pooling and normalized using the Softmax normalization function. Then, the features of the first branch that have been grouped and normalized are matrix multiplied using the Matmul function to obtain the first feature matrix; at the same time, the features that have been 3*3 convolved are matrix multiplied with the features normalized by the first branch normalization function using the Matmul function to obtain the second feature matrix; then the first feature matrix and the second feature matrix are added together, and then the attention weight matrix is ​​generated through the Sigmoid activation function; finally, the input feature group and the attention weight matrix are reweighted to obtain the output features optimized by the EMA attention mechanism.

6. The method for automatically evaluating the quality of sit-up movements based on posture key points according to claim 1, characterized in that: In step 3, after the key posture estimation and key point detection network obtains the key point positions and corresponding coordinates of each sit-up key frame, they are sent to the score evaluation network to calculate the key point angles, allocate weight coefficients and judge the deduction items, and finally the weighted sum is obtained to obtain the final action quality prediction score, thereby completing the quality evaluation of the sit-up action.

7. The method for automatically evaluating the quality of sit-up movements based on posture key points according to claim 1, characterized in that: In step 3, in order to achieve the quality scoring of the sit-up action, four key stages in the sit-up process are selected as the evaluation objects; namely, the sit-up starting posture P1, the action characteristics of which are that the subject lies on his back on the cushion, the two shoulder blades touch the cushion, the knees are bent at about 90 degrees, and the hands are holding the head; the upper body lifting posture P2, the action characteristics of which are that the subject lifts the upper body off the cushion and keeps the hands holding the head; the abdomen lifting posture P3, the action characteristics of which are that the subject relies on the abdominal strength to complete the sit-up stage and keeps the hands holding the head; the sit-up ending posture P4, the action characteristics of which are that the subject sits up and the elbows touch or exceed the knees.

8. The method for automatically evaluating the quality of sit-up movements based on posture key points according to claim 7, characterized in that: In step 3, the quality of sit-up movements is automatically scored based on video analysis and understanding, which requires accurate tracking of the positions of the skeleton key points in the above four key stages and matching them with the standard posture to obtain a score; Select six key points A to F in the video action skeleton as tracking targets for posture estimation; Point A is located at the shoulder joint, with coordinates marked as (x1, y1); Point B is located at the hip joint (x2, y2); Point C is located at the knee joint (x3, y3); Point D is located at the elbow joint (x4, y4); Point E is located at the wrist joint (x5, y5); and Point F is located at the ankle joint (x6, y6); In order to evaluate the quality of action completion at each stage, the importance of key postures and the quality of action completion of each key posture are judged; first, different weights are assigned to the above four key stages; then, the quality of action completion and irregular actions are evaluated according to the key points of each stage. The evaluation criteria are as follows: According to the angle matching degree of key points, corresponding scores are given for holding the head with both hands, bending the knees at 90 degrees, and the degree of body folding.

9. The method for automatically evaluating the quality of sit-up movements based on posture key points according to claim 8, characterized in that: In step 3, the specific process of evaluating the action completion quality and irregular actions is as follows: The standardization of the action of holding the head with both hands is determined by the folding angle ∠D of the arms; when ∠D is less than or equal to 45°, it is considered a standard action of holding the head with both hands and gets 5 points; When ∠D is greater than 45°, the scores are 4, 3, 2, 1, and 0, respectively, according to the degree of non-standardness of the action; The calculation formula of ∠D is as follows: Whether the subject's knee flexion is 90° is determined by the angle ∠C of the legs; when the angle ∠C is greater than 80° and less than 95°, it is considered a standard knee flexion action and is scored 5 points; When the angle ∠C is not within the above range, the scores are 4, 3, 2, 1, and 0 points according to the degree of non-standardness of the action; The calculation formula of ∠C is as follows: Whether the subject lies on his back and completes the sit-up is judged based on the degree of body folding, that is, when the body folding angle ∠B is greater than or equal to 120°, it is considered to be a standard supine action and is scored 5 points; When ∠B is less than 120°, the scores are 4, 3, 2, 1, and 0, respectively, according to the degree of irregularity of the action; The standard degree of the sit-up action is judged based on ∠B', that is, when the folding angle of the body ∠B' is less than or equal to 60° and the elbows touch or exceed the knees, that is, x4 ≥ x3, 5 points are awarded; When ∠B' is greater than 60°, the scores are 4, 3, 2, 1, or 0, depending on the degree of irregularity of the action; The calculation formulas for ∠B and ∠B' are the same and are shown in formula (4): In summary, based on the 4 key stages of sit-up videos, M , and 3 key action points in each stage MN That is, the standardization of holding the head with both hands, bending knees at 90 degrees, and folding the body. The final prediction score formula is: In the formula, S P is the final prediction score, α MN is the weight of the key action points in each stage, M is the key stage number, N is the key action point number, β M is the weight of each key stage.

Images