A method and system for FMS motion classification based on human joints

Abstract

The present invention relates to a FMS action classification method and system based on human joints. The front image and the side image at the current moment are input into a human joint recognition model to obtain a joint data sequence at the current moment; the target joint data is determined according to the confidence of each joint data in the joint data sequence; the target joint data is input into an action classification model to obtain a classification result corresponding to the image; the classification result corresponding to the image is input into a buffer queue, and it is determined whether the buffer queue is full; if yes, the mode in the buffer queue is determined as the final classification result of the target image, and a dequeue operation is performed on the buffer queue, and then the current moment is updated and the image of the tester at the current moment is returned to be obtained; if no, the final classification result of the image at the current moment is determined according to the number of elements in the buffer queue, and the current moment is updated and returned to obtain the image of the tester at the current moment. The present invention can improve the accuracy of the FMS action classification result.

Description

Technical Field

[0001] The present invention relates to the technical field of video understanding of computer vision, and in particular to a FMS action classification method and system based on human joint points. Background Art

[0002] FMS stands for Functional Movement Screen, which is a simple and effective way to test athletes' coordination and flexibility. It includes seven movements: squats, hurdle steps, straight lunges, shoulder flexibility, active straight knee leg raises, trunk stability push-ups, and rotational stability. Some of these movements have symmetrical postures and different levels of difficulty. During the test, the tester needs to complete each movement with the maximum amplitude. After the test, the referee will give each scoring item a score of 1 to 3 points based on the quality and difficulty of the tester's completed movements. However, if the tester feels any pain, the movement will be recorded as 0 points. A total score of 14 points is generally considered qualified. In this test, the higher the score means that the tester will perform better in the sport and the less likely he or she will be injured in the sport.

[0003] Although there are abundant datasets and a series of solutions for existing motion recognition tasks, there are few datasets specifically used for FMS action classification and evaluation, and their solutions are relatively few. In addition, most methods are performed on data from a single perspective, and many actions do not appear in common action recognition datasets, such as straight lunges, active straight knee leg raises, etc., which makes the classification results inaccurate. Therefore, a method specifically used for FMS action classification is needed to improve the accuracy of FMS action classification. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for FMS action classification based on human joints, which is specially used for FMS action classification and can improve the accuracy of FMS action classification results.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for classifying FMS actions based on human joints, comprising:

[0007] Acquire the image of the tester at the current moment, wherein the image includes a front image and a side image;

[0008] Inputting the image at the current moment into a human joint point recognition model to obtain a joint point data sequence at the current moment;

[0009] Determine the target joint point data at the current moment according to the confidence of each joint point data in the joint point data sequence at the current moment; the target joint point data is the joint point data corresponding to the top two confidences ranked from large to small;

[0010] Inputting the target joint point data at the current moment into the action classification model to obtain the classification result corresponding to the image at the current moment;

[0011] Inputting the classification result corresponding to the image at the current moment into the buffer queue at the current moment, and determining whether the buffer queue at the current moment is full, to obtain a first determination result;

[0012] If the first judgment result is yes, the majority in the buffer queue at the current moment is determined as the final classification result of the target image, and a dequeue operation is performed on the buffer queue at the current moment, and then the current moment is updated and the process returns to the step of "obtaining the image of the tester at the current moment"; the target image is the image corresponding to the classification result at the middle position of the buffer queue at the current moment;

[0013] If the first judgment result is no, the final classification result of the image at the current moment is determined according to the number of elements in the buffer queue at the current moment, and the current moment is updated and then returns to the step of "obtaining the image of the tester at the current moment".

[0014] Optionally, the step of inputting the image at the current moment into a human joint point recognition model to obtain a joint point data sequence at the current moment specifically includes:

[0015] splicing the front image and the side image at the current moment into one image to obtain a spliced ​​image;

[0016] The spliced ​​image is input into a human joint point recognition model to obtain a joint point data sequence at the current moment.

[0017] Optionally, determining the target joint point data at the current moment according to the confidence of each joint point data in the joint point data sequence at the current moment specifically includes:

[0018] Calculate the confidence of each joint point data in the joint point data sequence at the current moment to obtain the confidence sequence at the current moment;

[0019] Traversing the confidence sequence at the current moment, determining the position of the joint point corresponding to each confidence in the confidence sequence at the current moment in the spliced ​​image, wherein the position includes the left side and the right side;

[0020] Determining the maximum confidence on the left side of the spliced ​​image and the maximum confidence on the right side of the spliced ​​image according to the positions of the joint points corresponding to the respective confidences in the spliced ​​image;

[0021] Determine that the joint point data corresponding to the maximum confidence on the left side of the stitched image and the maximum confidence on the right side of the stitched image are the target joint point data at the current moment.

[0022] Optionally, the training process of the human joint point recognition model specifically includes:

[0023] The human body joint point recognition model is trained using a human body posture estimation data set to obtain a first human body joint point recognition model;

[0024] The first human joint point recognition model is trained using the human joint point FMS data set to obtain a trained human joint point recognition model; the human joint point FMS data set includes an image of a trainer and joint point data corresponding to the image of the trainer.

[0025] Optionally, determining the final classification result of the image at the current moment according to the number of elements in the buffer queue at the current moment, updating the current moment and then returning to the step of “obtaining the image of the tester at the current moment” specifically includes:

[0026] When the number of elements in the buffer queue at the current moment is less than or equal to half of the preset length of the buffer queue, the classification result corresponding to the image at the current moment is determined to be the final classification result of the image at the current moment, and the current moment is updated and then returns to the step of "obtaining the image of the tester at the current moment";

[0027] When the number of elements in the buffer queue at the current moment is greater than half of the preset length of the buffer queue, the current moment is updated and then the process returns to the step of "obtaining the image of the tester at the current moment".

[0028] An FMS motion classification system based on human joints, comprising:

[0029] An acquisition module, used to acquire an image of the tester at a current moment, the image including a front image and a side image;

[0030] A joint point data sequence determination module, used for inputting the image at the current moment into a human joint point recognition model to obtain a joint point data sequence at the current moment;

[0031] A target joint point data determination module is used to determine the target joint point data at the current moment according to the confidence of each joint point data in the joint point data sequence at the current moment; the target joint point data is the joint point data corresponding to the top two confidences ranked from large to small;

[0032] A classification result determination module, used for inputting the target joint point data at the current moment into the action classification model to obtain the classification result corresponding to the image at the current moment;

[0033] A judgment module, used for inputting the classification result corresponding to the image at the current moment into the buffer queue at the current moment, and judging whether the buffer queue at the current moment is full, so as to obtain a first judgment result;

[0034] The final classification result determination module is used for, if the first judgment result is yes, determining the majority in the buffer queue at the current moment as the final classification result of the target image, and performing a dequeue operation on the buffer queue at the current moment, and then updating the current moment and returning to the step of "obtaining the image of the tester at the current moment"; the target image is the image corresponding to the classification result at the middle position of the buffer queue at the current moment;

[0035] The return module is used to determine the final classification result of the image at the current moment according to the number of elements in the buffer queue at the current moment if the first judgment result is no, and update the current moment and then return to the step of "obtaining the image of the tester at the current moment".

[0036] Optionally, the joint point data sequence determination module specifically includes:

[0037] A stitching unit, used for stitching the front image and the side image at a current moment into one image to obtain a stitched image;

[0038] The joint point data sequence determination unit is used to input the spliced ​​image into a human joint point recognition model to obtain a joint point data sequence at the current moment.

[0039] Optionally, the target joint point data determination module specifically includes:

[0040] A confidence sequence determination unit, used to calculate the confidence of each joint point data in the joint point data sequence at the current moment to obtain the confidence sequence at the current moment;

[0041] A position determination unit, used to traverse the confidence sequence at the current moment, and determine the position of the joint point corresponding to each confidence in the confidence sequence at the current moment in the spliced ​​image, wherein the position includes the left side and the right side;

[0042] a maximum confidence determination unit, configured to determine the maximum confidence on the left side of the spliced ​​image and the maximum confidence on the right side of the spliced ​​image according to the positions of the joint points corresponding to the respective confidences in the spliced ​​image;

[0043] The target joint point data determining unit is used to determine the joint point data corresponding to the maximum confidence on the left side of the spliced ​​image and the maximum confidence on the right side of the spliced ​​image as the target joint point data at the current moment.

[0044] Optionally, the training process of the human joint point recognition model specifically includes:

[0045] A pre-training module, used to train a human joint point recognition model using a human posture estimation data set to obtain a first human joint point recognition model;

[0046] The human joint point recognition model training module is used to train the first human joint point recognition model using the human joint point FMS data set to obtain a trained human joint point recognition model; the human joint point FMS data set includes an image of a trainer and joint point data corresponding to the image of the trainer.

[0047] Optionally, the return module specifically includes:

[0048] A result determination unit, configured to determine, when the number of elements in the buffer queue at the current moment is less than or equal to one half of the preset length of the buffer queue, that the classification result corresponding to the image at the current moment is the final classification result of the image at the current moment, update the current moment, and then return to the step of “obtaining the image of the tester at the current moment”;

[0049] The returning unit is used to update the current moment and then return to the step of "obtaining the image of the tester at the current moment" when the number of elements in the buffer queue at the current moment is greater than half of the preset length of the buffer queue.

[0050] According to the specific embodiment provided by the present invention, the present invention discloses the following technical effects: the present invention inputs the image at the current moment into the human joint point recognition model to obtain the joint point data sequence at the current moment; the image includes a front image and a side image; the target joint point data at the current moment is determined according to the confidence of each joint point data in the joint point data sequence at the current moment; the target joint point data is the joint point data corresponding to the top two confidences ranked from large to small; the target joint point data at the current moment is input into the action classification model to obtain the classification result corresponding to the image at the current moment; the classification result corresponding to the image at the current moment is input into the buffer queue, and it is judged whether the buffer queue is full to obtain the first judgment result; If the first judgment result is yes, the majority in the buffer queue is determined as the final classification result of the target image, and the dequeue operation is performed on the buffer queue, and then the current time is updated and the image of the tester at the current time is returned; the target image is the image corresponding to the classification result of the middle position of the buffer queue; if the first judgment result is no, the final classification result of the image at the current time is determined according to the number of elements in the buffer queue at the current time, and the current time is updated and then returned to obtain the image of the tester at the current time, and observed from two angles: front and side, which completely simulates the observation process of FMS action scoring in real situations, so as to comprehensively judge the tester's action quality and difficulty to improve the accuracy of the classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A flow chart of a method for classifying FMS actions based on human joints provided by an embodiment of the present invention;

[0053] Figure 2 A more specific flow chart of the FMS action classification method based on human joint points provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of the numbering of human joints;

[0055] Figure 4 Designed scenes for FMS test video acquisition;

[0056] Figure 5 This is the flow chart of the algorithm for normalizing human joint data;

[0057] Figure 6 Flowchart of the algorithm for correcting the FMS action classification results;

[0058] Figure 7 The following is an example diagram of the corresponding joint points for 14 types of FMS actions. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] The present invention provides a FMS action classification method based on human joint points, such as Figure 1 As shown, including:

[0062] Step 101: Obtain the image of the tester at the current moment, which includes a front image and a side image. The optical axes of the two cameras used to capture the front image and the side image of the tester during the test are perpendicular to each other and are equidistant from the tester, and the entire body of the tester is within the field of view of the two cameras.

[0063] Step 102: input the image at the current moment into a human joint point recognition model to obtain a joint point data sequence at the current moment.

[0064] Step 103: Determine the target joint point data at the current moment according to the confidence of each joint point data in the joint point data sequence at the current moment. The target joint point data is the joint point data corresponding to the top two confidences ranked from large to small.

[0065] Step 104: input the target joint point data at the current moment into the action classification model to obtain the classification result corresponding to the image at the current moment.

[0066] Step 105: input the classification result corresponding to the image at the current moment into the buffer queue at the current moment, and determine whether the buffer queue at the current moment is full to obtain a first determination result.

[0067] Step 106: If the first judgment result is yes, the majority in the buffer queue at the current moment is determined as the final classification result of the target image, and the dequeue operation is performed on the buffer queue at the current moment, and then the current moment is updated and the step "obtaining the image of the tester at the current moment" is returned. The target image is the image corresponding to the classification result in the middle position of the buffer queue at the current moment. The element value in the queue remains unchanged, and only the classification result is modified. The purpose of this is to prevent the asymmetric distribution of results in the queue before and after the modification, which triggers a domino effect and causes the subsequent results to be "corrected" incorrectly.

[0068] Step 107: If the first judgment result is no, the final classification result of the image at the current moment is determined according to the number of elements in the buffer queue at the current moment, and the current moment is updated and then returns to the step of "obtaining the image of the tester at the current moment".

[0069] In practical applications, the step of inputting the image at the current moment into a human joint point recognition model to obtain a joint point data sequence at the current moment specifically includes:

[0070] The front image and the side image at the current moment are spliced ​​into one image to obtain a spliced ​​image.

[0071] The spliced ​​image is input into a human joint point recognition model to obtain a joint point data sequence at the current moment.

[0072] In practical applications, the spliced ​​image is input into the human joint point recognition model to obtain the joint point data sequence at the current moment, which includes two stages: the human body recognition stage and the single person joint point recognition stage. The human body recognition stage is: the model recognizes the human body in the image and gives the position and size of the human body in the image. The single person joint point recognition stage is: according to the recognition results of the human body recognition stage, the model recognizes the single person joint point of each human target.

[0073] In practical applications, determining the target joint point data at the current moment according to the confidence of each joint point data in the joint point data sequence at the current moment specifically includes:

[0074] The confidence of each joint point data in the joint point data sequence at the current moment is calculated to obtain the confidence sequence at the current moment.

[0075] The confidence sequence at the current moment is traversed to determine the position of the joint point corresponding to each confidence in the confidence sequence at the current moment in the spliced ​​image, where the position includes the left side and the right side.

[0076] According to the positions of the joint points corresponding to the respective confidence levels in the stitched image, the maximum confidence level on the left side of the stitched image and the maximum confidence level on the right side of the stitched image are determined.

[0077] Determine that the joint point data corresponding to the maximum confidence on the left side of the stitched image and the maximum confidence on the right side of the stitched image are the target joint point data at the current moment.

[0078] In practical applications, the training process of the human joint point recognition model specifically includes:

[0079] The human body joint point recognition model is trained using a human body posture estimation dataset to obtain a first human body joint point recognition model.

[0080] The first human joint point recognition model is trained using the human joint point FMS data set to obtain a trained human joint point recognition model; the human joint point FMS data set includes an image of a trainer and joint point data corresponding to the image of the trainer.

[0081] In actual applications, the data annotation standards used for training the action classification model are as follows: Squat: The tester faces the front-view camera. Both wrists are over the head in the front view; Right leg hurdle step: The tester faces the front-view camera. The right foot leaves the horizontal board; Left leg hurdle step: The tester faces the front-view camera. The left foot leaves the horizontal board; Right leg front straight lunge: The tester faces the side-view camera. The feet stand forward and backward with no gap left and right, the lower wrist enters the torso boundary line, the upper wrist is above the shoulder and the elbow is at an acute angle; Left leg front straight lunge: The tester faces the side-view camera. The feet stand forward and backward with no gap left and right, the lower wrist enters the torso boundary line, the upper wrist is above the shoulder and the elbow is at an acute angle; Right hand upper shoulder flexibility: The tester faces the front-view camera. The upper hand touches the head behind the head or the lower wrist passes the torso boundary line; Left hand upper shoulder flexibility: The tester faces the front-view camera. The upper hand touches the head behind the head or the lower wrist crosses the trunk boundary line; the left leg is actively raised with the knee straight: the tester's feet are facing the side-view camera. Lie flat on the back with the head close to the ground, lift the left leg so that the heel is higher than the leg; the right leg is actively raised with the knee straight: the tester's feet are facing the side-view camera. Lie flat on the back with the head close to the ground, lift the right leg so that the heel is higher than the leg; Trunk stable push-ups: the tester's feet are facing the side-view camera. Both hands and feet are on the ground, both knees and abdomen are off the ground at the same time, and both legs are parallel; Unilateral left hand trunk rotation stability: the tester's feet are facing the side-view camera. The right hand, foot, and knee are on the ground and touch the horizontal board, and the left hand, foot, and knee are lifted off the ground; Unilateral right hand trunk rotation stability: the tester's feet are facing the side-view camera. The left hand, foot, and knee are on the ground and touch the horizontal board, and the right hand, foot, and knee are lifted off the ground; Contralateral left hand trunk rotation stability: the tester's feet are facing the side-view camera. The right hand, left foot, and knee land on the ground and touch the horizontal board, and the left hand, right foot, and knee are lifted off the ground; the opposite right torso rotation stability: the tester's feet face the side view camera. The left hand, right foot, and knee land on the ground and touch the horizontal board, and the right hand, left foot, and knee are lifted off the ground. Invalid actions: All actions except the above standard actions.

[0082] In practical applications, determining the final classification result of the image at the current moment according to the number of elements in the buffer queue at the current moment, updating the current moment and then returning to the step of "obtaining the image of the tester at the current moment" specifically includes:

[0083] When the number of elements in the buffer queue at the current moment is less than or equal to half of the preset length of the buffer queue, the classification result corresponding to the image at the current moment is determined to be the final classification result of the image at the current moment, and the current moment is updated and then returns to the step of "obtaining the image of the tester at the current moment";

[0084] When the number of elements in the buffer queue at the current moment is greater than half of the preset length of the buffer queue, the current moment is updated and then the process returns to the step of "obtaining the image of the tester at the current moment".

[0085] In view of the above method, the present invention also provides a FMS action classification system based on human joints, comprising:

[0086] The acquisition module is used to acquire the image of the tester at the current moment, wherein the image includes a front image and a side image.

[0087] The joint point data sequence determination module is used to input the image at the current moment into the human joint point recognition model to obtain the joint point data sequence at the current moment.

[0088] The target joint point data determination module is used to determine the target joint point data at the current moment according to the confidence of each joint point data in the joint point data sequence at the current moment; the target joint point data is the joint point data corresponding to the top two confidences ranked from large to small.

[0089] The classification result determination module is used to input the target joint point data at the current moment into the action classification model to obtain the classification result corresponding to the image at the current moment.

[0090] The judgment module is used to input the classification result corresponding to the image at the current moment into the buffer queue at the current moment, and judge whether the buffer queue at the current moment is full to obtain a first judgment result.

[0091] The final classification result determination module is used to determine the majority in the buffer queue at the current moment as the final classification result of the target image if the first judgment result is yes, and perform a dequeue operation on the buffer queue at the current moment, and then update the current moment and return to the step of "obtaining the image of the tester at the current moment"; the target image is the image corresponding to the classification result in the middle position of the buffer queue at the current moment.

[0092] The return module is used to determine the final classification result of the image at the current moment according to the number of elements in the buffer queue at the current moment if the first judgment result is no, and update the current moment and then return to the step of "obtaining the image of the tester at the current moment".

[0093] As an optional implementation, the joint point data sequence determination module specifically includes:

[0094] The splicing unit is used to splice the front image and the side image at the current moment into one image to obtain a spliced ​​image.

[0095] The joint point data sequence determination unit is used to input the spliced ​​image into a human joint point recognition model to obtain a joint point data sequence at the current moment.

[0096] As an optional implementation, the target joint point data determination module specifically includes:

[0097] The confidence sequence determination unit is used to calculate the confidence of each joint point data in the joint point data sequence at the current moment to obtain the confidence sequence at the current moment.

[0098] The position determination unit is used to traverse the confidence sequence at the current moment and determine the position of the joint point corresponding to each confidence in the confidence sequence at the current moment in the spliced ​​image, wherein the position includes the left side and the right side.

[0099] The maximum confidence determination unit is used to determine the maximum confidence on the left side of the spliced ​​image and the maximum confidence on the right side of the spliced ​​image according to the positions of the joint points corresponding to the respective confidences in the spliced ​​image.

[0100] The target joint point data determining unit is used to determine the joint point data corresponding to the maximum confidence on the left side of the spliced ​​image and the maximum confidence on the right side of the spliced ​​image as the target joint point data at the current moment.

[0101] As an optional implementation, the training process of the human joint point recognition model specifically includes:

[0102] The pre-training module is used to train the human joint point recognition model using the human posture estimation data set to obtain a first human joint point recognition model.

[0103] The human joint point recognition model training module is used to train the first human joint point recognition model using the human joint point FMS data set to obtain a trained human joint point recognition model; the human joint point FMS data set includes an image of a trainer and joint point data corresponding to the image of the trainer.

[0104] As an optional implementation, the return module specifically includes:

[0105] A result determination unit, configured to determine, when the number of elements in the buffer queue at the current moment is less than or equal to one half of the preset length of the buffer queue, that the classification result corresponding to the image at the current moment is the final classification result of the image at the current moment, update the current moment, and then return to the step of “obtaining the image of the tester at the current moment”;

[0106] The returning unit is used to update the current moment and then return to the step of "obtaining the image of the tester at the current moment" when the number of elements in the buffer queue at the current moment is greater than half of the preset length of the buffer queue.

[0107] The embodiment of the present invention provides a more specific method for real-time classification of FMS test actions based on human joint point detection, such as Figure 2 As shown, firstly, it is necessary to train a targeted model, then use two cameras to record in real time and call the joint point recognition model, then normalize the data according to the detection results of the joint point recognition model, perform action classification according to the normalization results and the action classification model, and finally use the buffer queue to correct the classification results, as described in detail below.

[0108] Step 1: Use the general posture estimation dataset to train a multi-person human joint recognition model, and then use the FMS dataset with annotated human joints to continue training it on the basis of this model to obtain a human joint recognition model dedicated to FMS actions; this model can simultaneously output the joint data of the tester from the front and side angles in the FMS test image. In addition, Figure 3 As shown in the figure, the human joint point recognition model dedicated to FMS actions needs to recognize 26 key points of the tester's body, including nose 0, left eye 1, right eye 2, left ear 3, right ear 4, left shoulder 5, right shoulder 6, left elbow 7, right elbow 8, left wrist 9, right wrist 10, left hip 11, right hip 12, left knee 13, right knee 14, left ankle 15, right ankle 16, head 17, neck 18, hip 19, left big toe 20, right big toe 21, left little toe 22, right little toe 23, left heel 23, right heel 25, etc., and output them in a unified format.

[0109] Step 2: If Figure 4 As shown, two cameras with the same parameters, orthogonal optical axes (optical axis 1 is perpendicular to optical axis 2) and equal distance from the tester (d1=d2) are used to synchronously collect data from the front and side of the tester during the test, and the two video images collected at the same time are spliced ​​into an image, which is then input into the human joint point recognition model in sequence to obtain the joint point data of the tester at two angles: front and side.

[0110] Step 3: If Figure 5As shown, the obtained joint point data is normalized. First, the confidence of all joint points between two human bodies in each image (after splicing) is compared, and two sets of human joint point data with the highest joint point detection confidence are retained for each image and the remaining results are discarded. Specifically, for the joint point data output by the joint point recognition model, each frame (each picture) is obtained by traversing the human body, and the pixel position corresponding to the currently traversed joint point data is determined. When the position is in the left part of the image, the confidence is compared with the detection result of the left part. When the confidence is less than or equal to the current detection result of the left part, the currently traversed joint point data is discarded. When it is greater than, the detection result of the left part is replaced. The right side is also based on the above method and will not be repeated here. After the traversal, the normalized result of the image is returned, and the FMS action classification model is trained using the above normalized joint point data and the training data labeled according to the unified standard. During the test, the real-time normalized joint point data is input into the classification model to obtain the original result of the tester's action recognition. The normalization process of joint point recognition results is to traverse the joint point recognition results according to the joint point confidence, screen them by human body, and retain the best two groups of human joint points as the basis for classification.

[0111] Step 4: Set a fixed-length queue with an odd length as the buffer queue for the classification results, and change the classification result corresponding to the middle position of the queue to the mode of the queue classification results. Figure 6 As shown, the corrected FMS action classification result is obtained. The correction of the classification result is to correct the action classification result of each frame, and take the mode of the two adjacent classification result sequences of equal length and its own classification result as the corrected value. That is, in a classification result buffer queue with an odd length, the classification result corresponding to the middle position is changed to the mode of the entire queue, but the queue element value is not changed, specifically: the classification result is input into the buffer queue, and then it is determined whether the queue length (the current number of elements in the queue) is equal to L (L is the preset length of the buffer queue). If it is equal to L, the queue mode is output, the mode in the queue is determined as the classification result corresponding to the target image, and the head element is dequeued to obtain the classification result of the target image; the target image is the image corresponding to the classification result of the middle position of the buffer queue at the current moment. If it is not equal to L, it is determined whether the queue length is greater than L / 2. If it is greater than L / 2, the image is returned to be acquired. If it is less than L / 2, the tail element is output, and the tail element is determined to be the final classification result of the image at the current moment.

[0112] In practical applications, the multi-person human joint recognition model is first trained using large datasets such as COCO and Halpe as a pre-training model, and then transfer training is performed on the FMS dataset.

[0113] In actual applications, the synchronized images of the two cameras are scaled in equal proportions and stitched together horizontally, and the subsequent steps are performed based on this stitched image as a unit.

[0114] In practical applications, action classification models use classic machine models such as random forest, XGBoost, support vector machine, etc.

[0115] In practical applications, the annotation standard for FMS action classification is developed for action classification tasks, not for the definition of standard actions in FMS testing.

[0116] In practical applications, action categories such as Figure 7 From left to right, from top to bottom: squat, right leg hurdle step, left leg hurdle step, right leg front straight lunge, left leg front straight lunge, right hand upper shoulder flexibility, left hand upper shoulder flexibility, left leg active straight knee leg raise, right leg active straight knee leg raise, trunk stability push-up, unilateral left hand trunk rotation stability, unilateral right hand trunk rotation stability, contralateral left hand trunk rotation stability, contralateral right hand trunk rotation stability.

[0117] In practical applications, the buffer queue length is generally an odd number between 31 and 45.

[0118] The embodiment of the present invention fills the gap in the prior art and provides a real-time classification method for FMS actions based on human joints. The method uses two cameras with orthogonal optical axes and a depth model to detect the human joints of each frame, and annotates the actions through details such as the angle and relative position of the joints to train the FMS action classification model. The original data, joint data and classification results are processed in real time to provide users with feedback on the current FMS action category.

[0119] The present invention has the following technical effects:

[0120] 1. The present invention discloses a standard method for real-time classification of FMS actions based on human joints, including training of human joint recognition model; training of FMS action classification model; using two cameras with orthogonal optical axes and equal distance from the tester to collect test data from the front and side perspectives of the tester respectively; normalizing the joint recognition results; correcting the classification results through a buffer queue, etc., which conforms to the actual FMS test scene and scoring process, can accurately recognize the actions of the FMS test process in real time, and fills the gap in FMS action automatic recognition technology. The method is simple and easy to implement, with high accuracy, and suitable for promotion.

[0121] 2. Real-time and accurate identification of FMS actions is achieved through innovative artificial intelligence means, data collection methods and data processing algorithms.

[0122] 3. The result correction algorithm of the present invention can not only make the result sequence more accurate, but also maintain the integrity of the single test action in the classification sequence, greatly ensuring the integrity and accuracy of the FMS action evaluation.

[0123] 4. The present invention adopts an innovative data collection method, which completely simulates the observation process of FMS action scoring under real conditions, that is, observing from two angles, front and side, so as to comprehensively judge the quality and difficulty of the tester's action.

[0124] 5. The present invention uses public and general data sets to train the model, and further adjusts the model in combination with FMS actions, so that the model can achieve good accuracy under the condition of using a small amount of dedicated data.

[0125] 6. The present invention establishes a classification standard for FMS actions, specifies the labeling specifications for each FMS test action, provides a unified specification and template for subsequent work, and contributes to the development of FMS automatic evaluation work.

[0126] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0127] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A FMS action classification method based on human joints, characterized in that: include: Acquire the image of the tester at the current moment, wherein the image includes a front image and a side image; Inputting the image at the current moment into a human joint point recognition model to obtain a joint point data sequence at the current moment; Determining the target joint point data at the current moment according to the confidence of each joint point data in the joint point data sequence at the current moment; The target joint point data are the joint point data corresponding to the top two confidence levels ranked from largest to smallest; Inputting the target joint point data at the current moment into the action classification model to obtain the classification result corresponding to the image at the current moment; Inputting the classification result corresponding to the image at the current moment into the buffer queue at the current moment, and determining whether the buffer queue at the current moment is full, to obtain a first determination result; If the first judgment result is yes, the majority in the buffer queue at the current moment is determined as the final classification result of the target image, and a dequeue operation is performed on the buffer queue at the current moment, and then the current moment is updated and the process returns to the step of "obtaining the image of the tester at the current moment"; the target image is the image corresponding to the classification result at the middle position of the buffer queue at the current moment; If the first judgment result is no, then determine the final classification result of the image at the current moment according to the number of elements in the buffer queue at the current moment, update the current moment and then return to the step of "obtaining the image of the tester at the current moment"; Determining the final classification result of the image at the current moment according to the number of elements in the buffer queue at the current moment, updating the current moment and then returning to the step of "obtaining the image of the tester at the current moment" specifically includes: When the number of elements in the buffer queue at the current moment is less than or equal to one half of the preset length of the buffer queue, the classification result corresponding to the image at the current moment is determined to be the final classification result of the image at the current moment, and the current moment is updated and then returns to the step of "obtaining the image of the tester at the current moment"; When the number of elements in the buffer queue at the current moment is greater than half of the preset length of the buffer queue, the current moment is updated and then the process returns to the step of "obtaining the image of the tester at the current moment".

2. The FMS action classification method based on human joint points according to claim 1 is characterized in that: The step of inputting the image at the current moment into a human joint point recognition model to obtain a joint point data sequence at the current moment specifically includes: splicing the front image and the side image at the current moment into one image to obtain a spliced ​​image; The spliced ​​image is input into a human joint point recognition model to obtain a joint point data sequence at the current moment.

3. The FMS action classification method based on human joint points according to claim 2 is characterized in that: Determining the target joint point data at the current moment according to the confidence of each joint point data in the joint point data sequence at the current moment specifically includes: Calculate the confidence of each joint point data in the joint point data sequence at the current moment to obtain the confidence sequence at the current moment; Traversing the confidence sequence at the current moment, determining the position of the joint point corresponding to each confidence in the confidence sequence at the current moment in the spliced ​​image, wherein the position includes the left side and the right side; Determining the maximum confidence on the left side of the spliced ​​image and the maximum confidence on the right side of the spliced ​​image according to the positions of the joint points corresponding to the respective confidences in the spliced ​​image; Determine that the joint point data corresponding to the maximum confidence on the left side of the stitched image and the maximum confidence on the right side of the stitched image are the target joint point data at the current moment.

4. The FMS action classification method based on human joint points according to claim 1 is characterized in that: The training process of the human joint point recognition model specifically includes: The human body joint point recognition model is trained using a human body posture estimation data set to obtain a first human body joint point recognition model; The first human joint point recognition model is trained using the human joint point FMS data set to obtain a trained human joint point recognition model; the human joint point FMS data set includes an image of a trainer and joint point data corresponding to the image of the trainer.

5. A FMS motion classification system based on human joints, characterized in that: include: An acquisition module, used to acquire an image of the tester at a current moment, the image including a front image and a side image; A joint point data sequence determination module, used for inputting the image at the current moment into a human joint point recognition model to obtain a joint point data sequence at the current moment; A target joint point data determination module is used to determine the target joint point data at the current moment according to the confidence of each joint point data in the joint point data sequence at the current moment; The target joint point data are the joint point data corresponding to the top two confidence levels ranked from largest to smallest; A classification result determination module, used for inputting the target joint point data at the current moment into the action classification model to obtain the classification result corresponding to the image at the current moment; A judgment module, used for inputting the classification result corresponding to the image at the current moment into the buffer queue at the current moment, and judging whether the buffer queue at the current moment is full, so as to obtain a first judgment result; a final classification result determination module, for determining the mode in the buffer queue at the current moment as the final classification result of the target image if the first judgment result is yes, and performing a dequeue operation on the buffer queue at the current moment, and then updating the current moment and returning to the step of "obtaining the image of the tester at the current moment"; The target image is an image corresponding to the classification result at the middle position of the buffer queue at the current moment; A return module, for determining the final classification result of the image at the current moment according to the number of elements in the buffer queue at the current moment if the first judgment result is no, and updating the current moment and then returning to the step of "obtaining the image of the tester at the current moment"; The return module specifically includes: A result determination unit, configured to determine, when the number of elements in the buffer queue at the current moment is less than or equal to one half of the preset length of the buffer queue, that the classification result corresponding to the image at the current moment is the final classification result of the image at the current moment, and update the current moment and then return to the step of "obtaining the image of the tester at the current moment"; The returning unit is used to update the current moment and then return to the step of "obtaining the image of the tester at the current moment" when the number of elements in the buffer queue at the current moment is greater than half of the preset length of the buffer queue.

6. The FMS motion classification system based on human joint points according to claim 5 is characterized in that: The joint point data sequence determination module specifically includes: A stitching unit, used for stitching the front image and the side image at a current moment into one image to obtain a stitched image; The joint point data sequence determination unit is used to input the spliced ​​image into a human joint point recognition model to obtain a joint point data sequence at the current moment.

7. The FMS motion classification system based on human joints according to claim 6, characterized in that: The target joint point data determination module specifically includes: A confidence sequence determination unit, used to calculate the confidence of each joint point data in the joint point data sequence at the current moment to obtain the confidence sequence at the current moment; A position determination unit, used to traverse the confidence sequence at the current moment, and determine the position of the joint point corresponding to each confidence in the confidence sequence at the current moment in the spliced ​​image, wherein the position includes the left side and the right side; a maximum confidence determination unit, configured to determine the maximum confidence on the left side of the spliced ​​image and the maximum confidence on the right side of the spliced ​​image according to the positions of the joint points corresponding to the respective confidences in the spliced ​​image; The target joint point data determining unit is used to determine the joint point data corresponding to the maximum confidence on the left side of the spliced ​​image and the maximum confidence on the right side of the spliced ​​image as the target joint point data at the current moment.

8. The FMS motion classification system based on human joints according to claim 5, characterized in that: The training process of the human joint point recognition model specifically includes: A pre-training module, used to train a human joint point recognition model using a human posture estimation data set to obtain a first human joint point recognition model; The human joint point recognition model training module is used to train the first human joint point recognition model using the human joint point FMS data set to obtain a trained human joint point recognition model; the human joint point FMS data set includes an image of a trainer and joint point data corresponding to the image of the trainer.

Images