Human body action recognition method based on machine learning

Abstract

The invention provides a human body action recognition method based on machine learning, and solves the problem of action recognition in fitness training. According to the method, a Mediape framework is used for processing a video, key points of a human body are recognized, and a human body model is constructed; matrix perspective transformation is used for correcting video distortion, and the accuracy of the human body model is ensured; and the human body key points extracted by the Mediape are used as feature pairs to classify actions by using an SVM (Support Vector Machine). In addition, different training objects are distinguished by means of a YOLO-v7 model so as to adapt to the scene that multiple persons exercise at the same time. The high accuracy and performance of the method in action classification prove the potential of the method as a real-time feedback and guidance tool in fitness training. By accurately identifying and evaluating the athletic action, an individual can be helped to improve the athletic form and prevent athletic injury. The accurate motion recognition technology provides better fitness guidance and quality control for individuals.

Description

Technical Field

[0001] The present invention relates to the field of human motion recognition, and more specifically, to a human motion recognition method based on machine learning. Background Art

[0002] At present, the recognition methods of human motion posture can be divided into three categories: statistical-based methods, template-based methods, and grammar-based methods. Hidden Markov model is the most widely used method. Brand et al. (1997) proposed a coupled hidden Markov model (CHMM) and recognized sign language. Duong et al. (2005) proposed the S-HSMM (Switching Hidden-semi Markov Model) model. The recognition performance of S-HSMM is higher than that of HHMMs, but the algorithm complexity is higher. At the same time, HMMs are only suitable for describing the system of a single moving object and cannot effectively fuse information. Some scholars have also combined HMM and dynamic Bayesian networks to efficiently realize the modeling and recognition of human motion posture. In recent years, Yang Jun et al. (2021) used the OpenPose algorithm for data analysis and mining, detected the information of the key points of the human body in the image and calculated the distance, and then calculated the angle, judged the shortcomings of the human body movement and prompted feedback. Cao et al. (2021) proposed a method for effectively detecting the two-dimensional posture of multiple people in an image. The architecture is designed to jointly learn part locations and their associations through two branches of the same sequence prediction process, encoding the global context, allowing a greedy bottom-up parsing step to maintain high accuracy while achieving real-time performance.

[0003] Image recognition technology originated in the 1940s. In the 1990s, the combination of artificial neural networks and support vector machines promoted the development of image recognition technology. Traditional image recognition technology is mainly based on shallow structural models, which requires manual preprocessing of images, resulting in the problem of reduced image recognition accuracy. Researchers began to study deeper network structure models, using the model itself to extract image features to avoid human intervention. During this period, many deep learning models were proposed, such as DBN, DFN, CNN, RNN, etc. At the same time, various algorithm models of deep learning have been successfully applied to business. For example: in China, driverless cars represented by Baidu will apply computer vision, hearing and other recognition technologies based on deep learning to the development of the "Baidu driverless car" system. In foreign countries, companies such as Google, Microsoft, and Facebook regard deep learning technology as one of the important technologies for the development of their companies, and have even developed a series of deep learning frameworks for deep learning technology, such as Tensorflow developed by Goolgle, CNTK developed by Microsoft, Torch developed by Facebook, Keras developed by Fchollet, D, etc. These deep learning technology frameworks are mainly used in image recognition and classification, handwriting recognition, speech recognition, prediction, natural language processing, etc. In recent years, for human image recognition, the University of Florida Computer Vision Research Center has proposed a simple and effective end-to-end deep network (named SSA2D) for action detection in videos. The network performs pixel-level joint action detection in a single shot, where each pixel of the detected participant is assigned an action label. Yuan Yating (2021) proposed a multi-filter dynamic graph convolutional neural network algorithm, which is based on the graph convolutional neural network. It integrates the multi-filter graph convolution structure, the dynamic skeleton graph structure, and the multi-scale time domain convolution structure, optimizes the spatial and temporal feature extraction scheme, realizes a new action recognition model, and improves the accuracy on the data set. Summary of the invention

[0004] The main purpose of the present invention is to provide a human motion recognition method based on machine learning, specifically to solve the problem of motion recognition in fitness training based on machine learning and deep learning methods.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A human action recognition method based on machine learning comprises the following steps:

[0007] S1, collecting video data: collecting video image data of human body movements;

[0008] S2, identifying key points of the human body: using the Mediapipe framework to construct the motion trajectory of the athlete's action in several frames of video images;

[0009] S3, image preprocessing: performing distortion correction on the above motion trajectory based on video image data;

[0010] S4, action classification and evaluation, matches the training parameters with the existing training set through SVM classification technology to identify the athlete's current movement action.

[0011] Preferably, the implementation process of step S2 using the Mediapipe framework to construct the motion trajectory of the athlete's action in a plurality of frames of video images is as follows:

[0012] The Mediapipe framework is based on the Blaze-Pose algorithm, which is a technology that combines detector-tracker, deep convolutional neural network and regression encoder network; that is, the detector-tracker is run first, then the deep convolutional neural network is run, and finally the regression encoder network is used to regress the key point position to obtain accurate key point coordinates, thereby optimizing the accuracy of pose estimation, specifically;

[0013] The detector-tracker is the pre-processing stage of the neural network, which is used to provide initial posture estimation and tracking. If the detector detects a face (if no face is detected, the process ends), it uses the detected face position information to calculate the vector formed by the midline of the hip bone and the face to correct the body's tilt angle and determine the position of other body parts to generate an initial estimation result. The tracker uses historical information and motion models to predict the body's position in the current frame.

[0014] The deep convolutional neural network consists of three parts. The first is the heat map prediction network and the offset network, which are used to output the heat map and offset map for locating the key points of the human body. The offset map represents the small offset of the key point relative to the predicted position in the heat map. By combining the heat map position with the corresponding offset map, the position of the key point can be located more accurately. Finally, the regression encoder network is used to regress the key point position to obtain accurate key point coordinates.

[0015] Preferably, the distortion correction in step S3 is to use matrix perspective transformation to perform distortion correction on the video image, specifically by mapping four corner points in the video image screen to four corner points in the target image to calculate the perspective transformation matrix; based on the perspective transformation matrix, any point in the original image is mapped to the corresponding position in the target image.

[0016] Preferably, the perspective transformation matrix has the following form:

[0017]

[0018] The perspective transformation matrix is ​​a linear transformation matrix used to map a point in a three-dimensional coordinate system to a corresponding point in another three-dimensional coordinate system. X, Y, and Z represent the coordinates of the target three-dimensional coordinate point, and x, y, and z represent the coordinates of the original three-dimensional coordinate point.

[0019] The element [a ij ] reflects the mapping relationship between the two coordinate systems, and the specific transformation method is as follows:

[0020]

[0021] Z=1

[0022] Through this transformation, the original coordinate point can be mapped to the corresponding position in the target coordinate system;

[0023] Each element in the perspective transformation matrix [a ij ] can be obtained by calculating the relationship between the original coordinate system and the target coordinate system;

[0024] For the perspective transformation matrix, calibration methods (such as camera calibration) are used to calculate these parameters to ensure the accuracy of the transformation. When a user creates a personal profile, data such as the length ratio of each torso of the user can be collected to correct the distortion caused by the camera shooting. This can achieve the restoration and accurate positioning of each key point of the human body.

[0025] Preferably, the process of action classification and evaluation in step S4 is: using SVM to classify the action of the target video image, and comparing the posture data of the standard action to determine whether the action of people at a certain moment is standard;

[0026] The implementation method of using the SVM algorithm to classify the actions of the target image is as follows: the joint point coordinates extracted by the Blaze-Pose algorithm are used as the training set, the training set of each action after manual assisted classification and annotation is input into the SVM classifier for training, and the parameters with the minimum loss are obtained through the stochastic gradient descent method to obtain the classifier of the corresponding action, and the action of the target image is classified based on the obtained action classifier.

[0027] SVM is a supervised binary classification model. Its basic idea is to find an optimal hyperplane that meets the classification requirements, so that the hyperplane can ensure accuracy and maximize the distance to the sample closest to the hyperplane. In this invention, we choose SVM with a linear kernel. The decision function of this SVM can be expressed as:

[0028] f(x i )=sign( <w,x i >+b)

[0029] where x i is the input feature vector accepted by the model, w is the weight factor, and b is the intercept value.

[0030] Assume that the sample training set is {(x 1 ,y 1 ),(x 2 ,y 2 ),...,(x l ,y l )}, where x i Data features extracted from the Mediapipe framework, y i is the action classification label, then the optimization model for solving the hyperplane is:

[0031]

[0032] sty i ( <w,x i >+b)≥1-ξ i ,i=1,2,…,l.

[0033] Where C is the penalty parameter, ξ i is the relaxation factor.

[0034] The Sigmoid activation function is mainly used in the activation layer of a neural network to map the input value to a fixed range (usually 0 to 1). Its mathematical form is:

[0035]

[0036] In the present invention, the Sigmoid function result is used to realize the confidence of action classification. The input z of the function is the distance from the sample to the hyperplane. For the sample (x i ,y i ), whose distance to the hyperplane is z i The calculation method is:

[0037]

[0038] When training the classifier, the present invention uses the joint point coordinates extracted by the Blaze-Pose algorithm in the previous text as the training set, and inputs the training set of each action after manual auxiliary classification and annotation into the SVM classifier for training, obtains the parameter with the minimum loss through the stochastic gradient descent method, and obtains the classifier of the corresponding action. In this way, the action of the characters in the video image can be classified, and the functions of motion recognition and counting are realized by identifying the actions of the characters in the video. In addition to the key point training set, the key joint angles calculated by the joint point coordinates can also be used as the training set. Using these angles as the training set to obtain the SVM classifier, accurate action classification can also be achieved.

[0039] Preferably, before step S4, the method further includes: when there are different people in the captured video image, a YOLO-v7 model is used to distinguish and mark the different people in the video image, and the coordinates of the four vertices of the different person identification boxes are output to facilitate cutting and counting, thereby achieving accurate counting of multiple human body scenes.

[0040] Preferably, the implementation process of using the YOLO-v7 model to distinguish and mark different people in the video image is as follows: the target detector in the YOLO-v7 model distinguishes different test objects in the video; first, the video image is input into the YOLO-v7 target detector for processing; the target detector analyzes each frame of the video image and identifies the target object therein; the coordinate information of different objects, that is, the position on the image, is obtained through the results output by the detector; according to the coordinate information of different objects output by the target detector, a specific area in the video is cut; and then statistics are performed on different test objects for each cut image area.

[0041] Preferably, for a single-person sports scene, step S4 also includes outputting a user report, including a standardization analysis report and an action evaluation report; for a multi-person sports scene, step S4 also includes generating separate step S4 for each individual and outputting a user report, including a standardization analysis report and an action evaluation report.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention uses the Mediapipe framework to identify human postures and actions through video to locate key points of the human body, such as shoulders, elbows, knees and other parts, and construct a human body model. Secondly, the matrix perspective transformation is used to correct the distortion of the video image to reduce the error problem caused by the shooting angle. The action recognition model is trained based on the joint standards of different actions in the official standard document. The actions performed by the above-mentioned human body model are classified using the SVM classification technology. And the relevant data of the model positioning and the standard action posture are compared to determine whether people's actions at a certain moment are standard. Provide intelligent action guidance for athletes, prompt them to improve their exercise action posture, so as to avoid the problem of human body injury caused by improper exercise posture. In order to adapt to the scene where multiple objects are exercising at the same time, the present invention also uses the YOLO-v7 model to distinguish different test objects and then run.

[0044] The present invention is used to identify, count and provide corrective suggestions for sports, especially fitness movements, so as to better achieve training movement standards and prevent training injuries to the body caused by non-standard training movements, and effectively achieve standard training, scientific and healthy training. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall flow chart of the present invention.

[0046] Figure 2 This is a schematic diagram of the actual test results of Blazepose key point recognition.

[0047] Figure 3 This is the effect diagram of the distortion correction model.

[0048] Figure 4 Schematic diagram of confidence and angles for standing and squatting squat movements before and after distortion correction.

[0049] Figure 5 This is a test chart for squat action counting in a multi-person state.

[0050] Figure 6 Schematic diagram of the confidence operation count for the squat movement.

[0051] Figure 7 This is a schematic diagram of action judgment for the squat action training set.

[0052] Figure 8 Schematic diagram of the squat action counting scenario.

[0053] Fig. 9 Schematic diagram of the generated user feedback report.

[0054] Fig.10 Schematic diagram of learning curves of different algorithms and feature extraction methods on the training set.

[0055] Fig.11 Schematic diagram of learning curves of different algorithms and feature extraction methods on the test set. DETAILED DESCRIPTION

[0056] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0057] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0058] like Figure 1 The present invention specifically comprises the following steps:

[0059] 1. Human key point recognition - Mediapipe

[0060] In order to quickly and accurately identify the key points of the human body on the mobile terminal, this embodiment uses Google's open source machine learning pipeline framework Mediapipe. This framework is based on the Blaze-Pose algorithm, which is a lightweight neural network that combines detector-tracker technology and uses heatmaps, offsets, and regression methods to build a network architecture for human posture estimation, which mainly includes heatmaps and regression calculation of key point coordinates.

[0061] In the Blaze-Pose algorithm, the detector-tracker is run first, followed by the deep convolutional neural network. The detector-tracker can be considered as the pre-processing stage of the neural network, which is used to provide initial pose estimation and tracking. If the detector detects a face, it uses the detected face position information to calculate the vector formed by the midline of the hip bone and the face to correct the tilt angle of the human body and roughly determine the position of other human body parts to generate an initial estimation result, while the tracker uses historical information and motion models to predict the position of the human body in the current frame. After that, a deep convolutional neural network (CNN) is used to further improve the accuracy and precision of pose estimation. The neural network consists of three parts. The first is the heat map prediction network and the offset network, which are used to output heat maps and offset maps for locating human key points. Among them, the offset map represents the small offset of the key point relative to the predicted position in the heat map. By combining the heat map position with the corresponding offset map, the position of the key point can be more accurately located. Finally, the regression encoder network is used to regress the key point position to obtain accurate key point coordinates, thereby further optimizing the accuracy of pose estimation.

[0062] Compared with the earlier OpenPose and DeepPose algorithms, BlazePose is more suitable for real-time application scenarios. This is mainly reflected in the fact that Blazepose requires fewer computing resources and is more convenient to run on mobile devices. Due to the development advantages of face recognition technology and the fact that faces have more significant features, when recognizing videos, BlazePose will first identify the faces in the video, correct the inclination angle of the body based on the vector formed by the midline of the hip bone and the face, and determine the approximate position of other parts of the body, and then identify the precise position of each key point one by one. After determining the position of each key point, the BlazePose framework will skip the face recognition step, and at the same time abandon the detection of heat maps during reasoning, and directly identify each key point, thereby speeding up the calculation speed of the framework. Through this multi-stage neural network structure, BlazePose can achieve efficient posture estimation and perform well in real-time application scenarios. `

[0063] Compared with other complex algorithms, the BlazePose framework has a computational load of only 6.9MFlop and 3.5M (the lightweight version has 2.7MFlop and 1.3M parameters), and can be deployed directly on mobile terminals without connecting to a server for calculation. It can run at more than 30 frames per second on a Pixel 2 phone, making it particularly suitable for real-time use cases such as fitness tracking and sign language recognition. When running on the GPU of a Pixel 3, BlazePose can reach 112FPS (frames per second), which greatly saves server costs and ensures that users can exercise at any time when offline.

[0064] 2. Image preprocessing - distortion correction

[0065] When using a perspective camera for image acquisition, distortions such as distortion or twisting may occur in the image due to different camera positions and angles. The goal of distortion correction is to make the image more realistic and accurately represent the actual scene by reversing these distortions. In order to solve this problem, the present invention uses matrix perspective transformation. Using the principle of perspective geometry, the perspective transformation matrix is ​​calculated by mapping the four corner points in the image to the four corner points in the target image. The perspective transformation matrix can map any point in the original image to the corresponding position in the target image. This embodiment performs distortion correction based on the geometric figures formed by the key points of the human body when standing with two feet apart.

[0066] 3. Multi-person target detection - YOLO

[0067] YOLO v7 is a single-stage target detection technology. Its network model mainly includes four parts: input layer, backbone network, feature fusion, and detection head. The backbone network backbone module consists of several convolutional layers, batch normalization layers, and activation functions. It is used to extract image features of different scales and is the key part of feature extraction. The feature fusion part combines the advantages of Featurepyramid network (FPN) and Path aggregation network (PANet) to form a PA-FPN structure, which enables efficient fusion of feature maps at different levels. The detection head part generates three layers of feature maps of different sizes and uses reparameterized convolution RepConv to output the prediction results. YOLO-v7 has made many improvements on the previous version of the algorithm. YOLO-v7 has designed several trainable bag-of-freebies methods, including reparameterization, leadhead, and auxiliaryhead, so that real-time target detection can greatly improve detection accuracy without increasing the cost of inference. In PlainNet and ResNet, a planned re-parameterized convolution structure was designed to solve the problem of how the re-parameterized module replaces the original module. In addition, a reliable soft label (SoftLabel) was generated using calculation and optimization methods and a reasonable allocation mechanism was designed to solve the problem of how the dynamic label allocation strategy handles the allocation of different output layers. The "extend" method and the "compound scaling" method were proposed to make better use of parameters and computing resources in real-time target detection tasks. Compared with existing real-time target detectors, YOLO-v7 reduces the number of parameters by about 40%, the amount of calculation by about 50%, and the inference speed and detection accuracy are also improved.

[0068] The present invention uses YOLOv7 target detector to distinguish different test objects in the video. First, the video is input into the YOLOv7 target detector for processing. The target detector analyzes each frame of the video and identifies the target object therein. The coordinate information of different objects, i.e., their positions on the image, can be obtained through the results output by the detector. According to the coordinate information of different objects output by the target detector, a specific area in the video can be cut. Subsequently, different test objects are counted for each cut image area.

[0069] 4. Action Classification and Evaluation - SVM

[0070] SVM is a supervised two-classification model. Its basic idea is to find an optimal hyperplane that meets the classification requirements, so that the hyperplane can ensure accuracy and maximize the distance to the sample closest to the hyperplane.

[0071] When training the classifier, the joint point coordinates extracted by the Blazepose algorithm in the foregoing text of this embodiment are used as a training set. The training set of each action after the manual auxiliary classification annotation is input into the SVM classifier for training, and the parameters with the minimum loss are obtained by the stochastic gradient descent method to obtain the classifier of the corresponding action. In this way, the effect of classifying the actions of the characters in the video can be achieved, and the functions of motion recognition and counting are realized by identifying the actions of the characters in the video. In addition to the key point training set, the key joint angles calculated by the joint point coordinates can also be used as a training set. Using these angles as a training set to obtain an SVM classifier, accurate action classification can also be achieved.

[0072] 5. Output user report - Python program

[0073] In order to facilitate users to obtain effective feedback, this embodiment finally outputs two reports to the users.

[0074] By inputting a video and judging from various angles, it can provide quantitative feedback in the form of a graph to show whether the user's movements are in place, whether the torso is in an appropriate position, whether it is too close to the thighs or too stiff, and whether the speed is appropriate, so that the user can better understand their exercise situation.

[0075] The present invention introduces the complete logic and process by taking the squat action as an example. First, the experimental samples are collected, that is, the video data set containing the squat action is collected. Then, for example, the motion trajectory of the athlete's action is constructed in the video images of several frames by first identifying the key points of the human body. Secondly, after the distortion correction, the training parameters are matched with the existing training set by the SVM classification technology to identify the current movement of the athlete. Finally, the athlete's posture is judged and improved. The selection of key limb angles needs to be combined with the professional knowledge of sports disciplines. For example, for the squat movement, since the video is mostly a side angle, the main positioning is the trunk angle (the angle from the shoulder joint to the hip joint and the vertical axis), the hip angle (the angle of the shoulder joint, the hip joint and the knee joint), the knee angle (the angle of the hip joint, the knee joint and the ankle joint), and the ankle angle (the angle of the knee joint, the ankle joint and the foot joint). By comparing the key angle changes of the user with the standard action, it is judged whether the action is standard and whether it meets the fitness requirements, and provides real-time posture improvement feedback for the athlete, so as to achieve the purpose of protecting the human body and scientific exercise.

[0076] Step 1. Use Google's open source machine learning pipeline framework Mediapipe to build a network architecture for human pose estimation based on the BlazePose algorithm. In the BlazePose algorithm, the detector-tracker is run first, and then the deep convolutional neural network is run. If the detector detects a face, it uses the detected face position information to calculate the vector formed by the midline of the hip bone and the face to correct the tilt angle of the body and roughly determine the position of other body parts to generate an initial estimate. The tracker uses historical information and motion models to predict the position of the body in the current frame. Figure 2 In the video, Mediapipe demonstrated extremely high accuracy and stability in human posture recognition. It not only successfully identified and annotated the key points of the human body in the video, but also performed fast and accurate key point detection and posture estimation in real-time environment, laying the foundation for subsequent action recognition and classification.

[0077] Step 2. Use matrix perspective transformation to correct the image angle. Using the principle of perspective geometry, the perspective transformation matrix is ​​calculated by mapping the four corner points in the image to the four corner points in the target image. The perspective transformation matrix can map any point in the original image to the corresponding position in the target image. Distortion correction is performed based on the geometric figures formed by the key points of the human body when standing with two feet apart. Figure 3 In , the visual effects of distortion correction are demonstrated. Figure 4The figure shows the confidence optimization effect before and after distortion correction. By correcting the distortion of the squat video, this embodiment successfully improves the confidence of the standing and squatting actions, and can more accurately measure the joint angles, and recognize key points in the video more accurately. In summary, perspective transformation distortion correction can effectively support the recognition and analysis of actions.

[0078] Step 3. This embodiment uses the YOLO-v7 target detector to distinguish different test objects in the video (so this step is only performed in a multi-person scenario). First, the video is input into the YOLO-v7 target detector for processing. The target detector analyzes each frame of the video and identifies the target object therein. Through the results output by the detector, the coordinate information of different objects, that is, their positions on the image, can be obtained. According to the coordinate information of different objects output by the target detector, specific areas in the video can be cut. Then, different test objects are counted for each cut image area. Figure 5 It shows that in the test of multi-person scenes, YOLO-v7 successfully distinguishes and marks different people, and outputs the coordinates of the four vertices of the recognition box, which is convenient for cutting and counting separately. The counting output of four different people is successfully realized. This shows that this embodiment successfully applies the YOLO-v7 algorithm, and through its precise human body recognition and position marking functions, it realizes accurate counting of multi-person scenes.

[0079] Step 4. When training the classifier, this embodiment uses the joint point coordinates extracted by the Blaze-pose algorithm in the previous step as the training set. The training set of each action after manual classification and annotation is input into the SVM classifier for training, and the parameters with the minimum loss are obtained through the stochastic gradient descent method to obtain the classifier of the corresponding action. In this way, the action of the characters in the video can be classified, and the functions of motion recognition and counting can be realized by identifying the actions of the characters in the video. Figure 6 In the figure, the two curves of different colors represent the changes in confidence in the standing and squatting states during the squat action. Figure 7 The figure shows the confidence changes for the squat action, where 1 represents squatting and 0 represents standing. The SVM classifier can accurately classify the action and evaluate whether the user's current action is correct. Figure 8 It shows that in actual tests, the confidence calculation and technical accuracy are very high, with almost no errors. This means that the SVM classifier can accurately classify and evaluate the user's current actions, which is quite reliable and effective.

[0080] Step 5. In order to facilitate users to get effective feedback, the present invention uses Python program to generate two reports for users: standard degree analysis and multi-dimensional action evaluation. Input a video, and by judging each angle, it can quantitatively feedback in the form of charts whether the user's action is in place, whether the torso position is appropriate, whether it is too close to the thigh or too stiff, and whether the speed is appropriate, so that users can better understand their own exercise conditions.

[0081] like Fig. 9 In order to facilitate users to get effective feedback, the present invention finally outputs two reports to users. Taking the squat action as an example, in the standard degree analysis report, a radar chart is output; the three aspects of the squat trunk action, squat speed, and squat degree are scored respectively, and the final action score is generated.

[0082] In the multi-dimensional action evaluation report, the knee angle, torso angle, and hip angle are analyzed, and not only the total number of squats is output, but also relevant suggestions are given from the perspectives of key angles of the human body and the speed of the action. For scenes with multiple people, movement feedback reports can be generated separately.

[0083] The above experimental results verify the effectiveness of the method of the present invention in motion recognition. The high accuracy and performance of the process of the present invention in motion classification demonstrate the potential of the method as a real-time feedback and guidance tool in fitness training. By accurately identifying and evaluating sports movements, the method of the present invention can help individuals improve their movement form and prevent sports injuries. This accurate motion recognition technology provides individuals with better fitness guidance and quality control. Future work can further optimize the algorithm, expand the scope of application, and combine other sensors for more comprehensive motion monitoring and analysis.

[0084] For the comparative evaluation of the algorithm model, this embodiment selects logistic regression as the comparison algorithm, and uses the coordinates of key points and the key joint angles of the corresponding actions as features for training and testing, respectively. A comparison chart is drawn based on the learning curve results of the training set and the test set to evaluate the accuracy of the model.

[0085] like Fig.10 In the training set, we observed that the confidence level dropped at the beginning. This may be because the model has not fully learned the features and patterns of the data in the initial stage, resulting in low classification accuracy. As the training progresses, the model gradually learns more features and fits the training data better, and the confidence level begins to increase and gradually reaches a relatively stable level.

[0086] like Fig.11,On the test set, as the training progresses, the confidence of the model system gradually increases and eventually tends to a relatively stable level. This shows that the features and patterns learned by the model system during the training phase can be well generalized to new samples in the test set, with high accuracy.

[0087] In summary, the model system of the present invention has high accuracy and generalization ability, and can accurately identify and classify different samples. In addition, the classifier selection of the present invention is better than the comparison algorithm.

[0088] At present, the algorithm of the present invention has been stably extended to multiple movements such as sit-ups, pull-ups, push-ups, etc. in the experimental stage, and a complete operation logic method is also available for other movements.

[0089] (The experimental environment of this invention is Windows 10, the development language is Python 3.9, the framework is Scikit-learn+mediapipe, and the experimental data set comes from the relevant action data set in the HMDB51 action recognition data set and the standard videos of fitness actions collected in mainstream fitness websites and software.)

[0090] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A human action recognition method based on machine learning, characterized in that: The following steps are involved: S1, collecting video data: collecting video image data of human body movements; S2, identifying key points of the human body: using the Mediapipe framework to construct the motion trajectory of the athlete's action in several frames of video images; S3, image preprocessing: performing distortion correction on the above motion trajectory based on video image data; S4, action classification and evaluation, matches the training parameters with the existing training set through SVM classification technology to identify the athlete's current movement action.

2. The method according to claim 1, characterized in that The implementation process of step S2 using the Mediapipe framework to construct the motion trajectory of the athlete's action in a number of video frames is as follows: The Mediapipe framework is based on the Blaze-Pose algorithm, which is a technology that combines detector-tracker, deep convolutional neural network and regression encoder network; that is, the detector-tracker is run first, then the deep convolutional neural network is run, and finally the regression encoder network is used to regress the key point position to obtain accurate key point coordinates, thereby optimizing the accuracy of pose estimation, specifically; The detector-tracker is the pre-processing stage of the neural network, which is used to provide initial posture estimation and tracking. If the detector detects a face, it uses the detected face position information to calculate the vector formed by the midline of the hip bone and the face to correct the body's tilt angle and determine the position of other body parts to generate an initial estimation result. The tracker uses historical information and motion models to predict the body's position in the current frame. The deep convolutional neural network consists of three parts. The first is the heat map prediction network and the offset network, which are used to output the heat map and offset map for locating the key points of the human body. The offset map represents the small offset of the key point relative to the predicted position in the heat map. By combining the heat map position with the corresponding offset map, the position of the key point can be located more accurately. Finally, the regression encoder network is used to regress the key point position to obtain accurate key point coordinates.

3. The method according to claim 2, characterized in that The distortion correction in step S3 is to use matrix perspective transformation to correct the distortion of the video image, specifically, by mapping the four corner points in the video image screen to the four corner points in the target image to calculate the perspective transformation matrix; based on the perspective transformation matrix, any point in the original image is mapped to the corresponding position in the target image.

4. The method according to claim 3, characterized in that The perspective transformation matrix has the following form: The perspective transformation matrix is ​​a linear transformation matrix used to map a point in a three-dimensional coordinate system to a corresponding point in another three-dimensional coordinate system. X, Y, and Z represent the coordinates of the target three-dimensional coordinate point, and x, y, and z represent the coordinates of the original three-dimensional coordinate point. The element [a ij ] reflects the mapping relationship between the two coordinate systems, and the specific transformation method is as follows: Z＝1 Through this transformation, the original coordinate point can be mapped to the corresponding position in the target coordinate system; Each element in the perspective transformation matrix [a ij ] can be obtained by calculating the relationship between the original coordinate system and the target coordinate system; For the perspective transformation matrix, a calibration method is used to calculate these parameters to ensure the accuracy of the transformation.

5. The method according to claim 4, characterized in that The process of action classification and evaluation in step S4 is: using SVM to classify the action of the target video image, and comparing the posture data of the standard action to determine whether the action of people at a certain moment is standard; The implementation method of using the SVM algorithm to classify the actions of the target image is as follows: the joint point coordinates extracted by the Blaze-Pose algorithm are used as the training set, the training set of each action after manual assisted classification and annotation is input into the SVM classifier for training, and the parameters with the minimum loss are obtained through the stochastic gradient descent method to obtain the classifier of the corresponding action, and the action of the target image is classified based on the obtained action classifier.

6. The method according to claim 5, characterized in that Before step S4, the method also includes: when there are different people in the captured video image, the YOLO-v7 model is used to distinguish and mark the different people in the video image, and the coordinates of the four vertices of the different person identification boxes are output to facilitate cutting and counting, thereby achieving accurate counting of multiple human body scenes.

7. The method according to claim 6, characterized in that The implementation process of using the YOLO-v7 model to distinguish and mark different people in the video image is as follows: the target detector in the YOLO-v7 model distinguishes different test objects in the video; first, the video image is input into the YOLO-v7 target detector for processing; the target detector analyzes each frame of the video image and identifies the target object therein; the coordinate information of different objects, that is, the position on the image, is obtained through the results output by the detector; and the specific area in the video is cut according to the coordinate information of different objects output by the target detector; Statistics of different test objects are then taken for each cut image region.

8. The method according to claim 6 or 7, characterized in that: For a single-person sports scene, step S4 also includes outputting a user report, including a standardization analysis report and an action evaluation report; for a multi-person sports scene, step S4 also includes generating separate step S4 for each individual and also includes outputting a user report, including a standardization analysis report and an action evaluation report.

Images