A method for counting sports items based on a deep learning network

Abstract

This invention discloses a method for counting sports events based on a deep learning network, comprising the following steps: Step 1, data annotation; Step 2, training data preparation; Step 3, training data preprocessing; Step 4, model building; Step 5, model training; Step 6, model inference. This invention uses a deep learning network structure to replace logical rules for counting sports events. Users only need to use annotation tools to quickly and easily annotate the data, and then feed the annotated data into the network to train the model to achieve sports event counting. Furthermore, this solution is simple to iterate later; only new data needs to be added and the model continued to be trained to adapt to new rules, resulting in a short development cycle and low cost.

Description

Technical Field

[0001] This invention belongs to the field of motion counting technology, specifically relating to a motion counting method based on deep learning networks. Background Technology

[0002] For algorithms related to counting human movements, such as rope skipping and pull-ups, current technologies mostly involve first determining the current state of the human body and then adding logical rules. Taking rope skipping as an example: the current state of the human body can be determined first through infrared laser sensing or visual key point detection, and then each rise and fall of the shoulder or foot is counted according to the human body's state. The counting logic rules are then continuously refined to improve the accuracy of the counting.

[0003] Counting schemes based on logical rules are complex and cumbersome to design for complex rules, and difficult to optimize and iterate later. Development and modification are time-consuming and costly. Take rope skipping as an example: simply counting by the number of steps taken up and down cannot handle tripping or movement during jumps. Adding position and jump frequency still fails to handle skipping patterns where both feet land alternately. Adding rope detection still requires additional handling for multiple interruptions in the skipping process. It is clear that considering and handling all possibilities for complex rules is extremely complex and difficult to implement. Summary of the Invention

[0004] The purpose of this invention is to solve the aforementioned technical problems in the prior art and provide a sports counting method based on deep learning networks. This method uses a deep learning network structure to replace logical rules for counting each sports event. Users only need to use annotation tools to quickly and easily annotate the data, and then feed the annotated data into the network to train the model to achieve sports event counting. Furthermore, this solution is easy to iterate later; simply adding new data and continuing to train the model is sufficient to adapt to new rules, resulting in a short development cycle and low cost.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for counting sports events based on deep learning networks, characterized by the following steps:

[0007] Step 1: Data annotation: Users import motion videos into the annotation tool, view the motion status frame by frame, and annotate the data on frames that meet the counting conditions;

[0008] Step 2: Training data preparation: Use the human keypoint detection algorithm to obtain the coordinates of each key point on the athlete's body;

[0009] Step 3: Training data preprocessing: Processing the data based on the relationship between the sport and the surrounding environment;

[0010] Step 4: Model Building: The model consists of multiple fully connected layers with an input dimension of 1x60, denoted as X1-X60, and an output dimension of 1x10, denoted as Y1-Y10; all layers use the ReLU activation function.

[0011] Step 5: Model Training

[0012] (1) Take n frames from the labeled data each time, where n is the number of consecutive frames between each labeling and the previous labeling;

[0013] (2) Model input: Flatten the result of each frame into 1X51, put it into X1-X51, and splice it with the model output Y2-Y10 of the previous frame to form the input of 1X60. When inputting the first frame, X52-X60 is 0.

[0014] (3) Model output true value: Input the above structure sequentially from frame t1 to frame tn. The true value of the output result is: Y1 of frames from t1 to tn-1 is 0, while the output of frame tn is Y1 equal to 1 and Y2-Y10 equal to 0.

[0015] (4) Loss calculation: Take the output Y1 from t1 to tn-1 and the output Y1 to Y10 from tn, and calculate the loss based on their difference from the true value;

[0016] (5) Model iteration: Inputting frames t1 to tn into the model sequentially is considered as one iteration. All parameters in the process are recorded. Based on the backpropagation of the model loss, the correction values ​​of each model parameter from t1 to tn are calculated, and the average of the correction values ​​at all the same positions is used to update the model.

[0017] Step 6, Model Inference: After training the model, use the same data preprocessing method as during training to concatenate the results of each frame with the output Y2-Y10 of the previous frame to form 1X60 data, input it into the model, and obtain the model's output Y1-Y10, where Y1 represents the counting result: if it is 1, it means the count is incremented by one. Y2-Y10 will be used as X52-X60 in the input of the next frame.

[0018] Furthermore, the annotation tool annotates the data related to the motion counting algorithm. It automatically splits the video into temporally continuous single-frame images. The default annotation result for each frame is 0. When the user browses each frame image and clicks the "increment count" button on the frame that meets the counting condition, the truth value corresponding to that frame is changed to 1.

[0019] Furthermore, in the human keypoint detection algorithm, the position estimate of the human body's pose in the image is obtained through a human image. Single-person pose estimation is adopted. By inputting a single-person image into the trained deep learning network, the prediction results of the coordinate positions of each human key point on the image are obtained.

[0020] Furthermore, deep learning networks include MoveNet or HRNet.

[0021] Furthermore, in obtaining the coordinates of various body key points of the athlete, taking the COCO human key point data format as an example, there are a total of 17 human key points. Each key point has three parameters, namely the X coordinate, Y coordinate, and confidence level of the key point.

[0022] Furthermore, in the training data preprocessing, for sports such as rope skipping and sit-ups that are unrelated to the surrounding environment, in order to improve detection accuracy, the origin of the coordinate system can be moved to the upper left corner of the human body, and the maximum value of the X and Y directions can be taken as five times the distance between the knee and the hip. Then, the X and Y directions can be normalized to 0-1. For sports such as pull-ups and shuttle runs that require environmental props, the maximum length and width of the screen can be used to normalize them to 0-1.

[0023] The present invention, by adopting the above-described technical solution, has the following beneficial effects:

[0024] 1. The present invention uses a self-developed deep learning network and dynamically iterates the model in combination with different training data, so that the model can handle continuous actions and has a significant effect on counting motion algorithms; and for motion algorithms with high complexity, it has a more significant improvement in counting accuracy compared with traditional logical rule counting schemes.

[0025] 2. This solution only requires users to provide the original motion video and perform simple annotations based on the video content. Once training is complete, the entire algorithm development is finished. No specialized knowledge is required, resulting in low development costs and a short development cycle. Furthermore, if other scenarios are considered later, only the corresponding videos need to be provided, annotated, and the model trained again to complete the algorithm iteration. The cost of iterative updates is also significantly reduced. Attached Figure Description

[0026] The present invention will be further described below with reference to the accompanying drawings:

[0027] Figure 1 This is a schematic diagram of the model establishment in this invention;

[0028] Figure 2 This is a schematic diagram of model training in this invention;

[0029] Figure 3 This is a map showing the key points of COCO's human body in this invention. Detailed Implementation

[0030] like Figures 1 to 3As shown, this invention uses a deep learning network structure to replace logical rules for counting various sports events. Users only need to use the developed annotation tool to quickly and easily annotate the data, and then feed the annotated data into the network to train the model to achieve the counting of sports events. Moreover, this solution is simple to iterate in the later stages; simply adding new data and continuing to train the model can complete the adaptation to new rules, resulting in a short development cycle and low cost.

[0031] A method for counting sports events based on deep learning networks includes the following steps:

[0032] Step 1: Data Labeling: For the counting algorithm, users only need to import the motion video into the labeling tool to view the motion status frame by frame, and click "increment count" on the frame that meets the counting conditions to quickly complete the data labeling work.

[0033] The annotation tool of this invention enables rapid annotation of data related to motion counting algorithms. Taking rope skipping as an example, simply import the rope skipping video into the annotation tool, which will automatically split the video into continuous single-frame images in the time domain. The default annotation result for each frame is 0. Users can quickly browse each frame and click the "increment count" button on the frame that meets the counting condition, changing the corresponding truth value to 1. The annotation work can be completed quickly by simply browsing the video content once.

[0034] Step 2: Training Data Preparation: First, a human keypoint detection algorithm is used to obtain the coordinates of each body keypoint of the moving person. Human keypoint detection, also known as human pose estimation, is a fundamental task in computer vision. It involves obtaining the estimated position of the human pose (keypoints, such as head, left hand, right foot, etc.) from an image of a person. Here, only single-person pose estimation is used. A single-person image is input into a trained deep learning network (such as MoveNet or HRNet) to obtain the predicted coordinate positions of each human keypoint in the image. Taking the COCO human keypoint data format as an example, there are 17 human keypoints. Each keypoint has three parameters: the X-coordinate, Y-coordinate, and confidence score. Figure 3 As shown, the 17 key points of the human body are: 0 for the nose; 1 for the left eye; 2 for the right eye; 3 for the left ear; 4 for the right ear; 5 for the left shoulder; 6 for the right shoulder; 7 for the left elbow; 8 for the right elbow; 9 for the left wrist; 10 for the right wrist; 11 for the left hip; 12 for the right hip; 13 for the left knee; 14 for the right knee; 15 for the left ankle; and 16 for the right ankle.

[0035] Step 3: Training Data Preprocessing: For exercises like rope skipping and sit-ups, which are independent of the surrounding environment, to improve detection accuracy, the origin of the coordinate system can be moved to the upper left corner of the body, and the maximum values ​​in the X and Y directions can be set to five times the distance between the knee and hip. Then, the X and Y values ​​can be normalized to 0-1. For exercises like pull-ups and shuttle runs, which require environmental props, the maximum width and height of the screen can be used to normalize them to 0-1.

[0036] Step 4: Model Building: The model consists of multiple fully connected layers with an input dimension of 1x60, denoted as X1-X60, and an output dimension of 1x10, denoted as Y1-Y10; all layers use the ReLU activation function.

[0037] Step 5: Model Training

[0038] (1) Take n frames from the labeled data each time, where n is the number of all consecutive frames between each labeling of "count incremented" and the previous labeling of "count incremented".

[0039] (2) Model input: Flatten the result of each frame into 1X51, put it into X1-X51, and splice it with the model output Y2-Y10 of the previous frame to form the input of 1X60. When inputting the first frame, X52-X60 is 0.

[0040] (3) Model output true value: Input the above structure sequentially from frame t1 to frame tn. The true value of the output result is: Y1 of frames from t1 to tn-1 is 0, while the output of frame tn is Y1 equal to 1 and Y2-Y10 equal to 0.

[0041] (4) Loss calculation: Take the output Y1 from t1 to tn-1 and the output Y1 to Y10 from tn, and calculate the loss based on their difference from the true value.

[0042] (5) Model iteration: Inputting frames t1 to tn into the model sequentially is considered as one iteration. All parameters in the intermediate process are recorded. Based on the backpropagation of the model loss, the correction values ​​of each model parameter from time t1 to tn are calculated, and the average of the correction values ​​at all the same positions is used to update the model.

[0043] Step 6, Model Inference: After training the model, use the same data preprocessing method as during training to concatenate the results of each frame with the output Y2-Y10 of the previous frame to form 1X60 data, input it into the model, and obtain the model's output Y1-Y10, where Y1 represents the counting result: if it is 1, it means the count is incremented by one. Y2-Y10 will be used as X52-X60 in the input of the next frame.

[0044] The above are merely specific embodiments of the present invention, but the technical features of the present invention are not limited thereto. Any simple changes, equivalent substitutions, or modifications made based on the present invention to solve essentially the same technical problems and achieve essentially the same technical effects are all covered within the protection scope of the present invention.

Claims

1. A method for counting sports events based on deep learning networks, characterized in that, Includes the following steps: Step 1: Data annotation: Users import motion videos into the annotation tool, view the motion status frame by frame, and annotate the data on frames that meet the critical points of the motion cycle defined in advance according to the target motion project as the counting condition. Step 2: Training Data Preparation: Using a human keypoint detection algorithm, obtain the coordinates of each key point on the athlete's body; the human keypoint detection algorithm outputs data containing multiple key points, each key point including the x-coordinate, y-coordinate, and confidence parameter; Step 3: Training data preprocessing: Processing the data based on the relationship between the sport and the surrounding environment; Step 4: Model Building: The model consists of multiple fully connected layers; the input dimension of the model is 60, corresponding to X1 to X60, where X1 to X51 are used to carry the key point data of the current frame, and X52 to X60 are used to carry the output Y2 to Y10 of the model in the previous frame; the output dimension of the model is 10, corresponding to Y1 to Y10; all of them use ReLU activation functions. Step 5: Model Training (1) Each time, a counting segment is taken from the labeled data. The segment consists of n consecutive frames of images from the first frame after the previous counting labeled frame to the current counting labeled frame. The current counting labeled frame is the nth frame of the segment. (2) Model input: The key point data obtained in each frame is flattened into 51 parameters, which are used as the first 51 parameters of the model input. Then, the 9 parameters Y2 to Y10 from the previous frame model output are spliced ​​together to form a model input of 60 parameters. When inputting the first frame, since there is no previous frame output, the next 9 parameters are set to 0. (3) Model output true value: Input the above model with frames t1 to tn in sequence. The true value of the output result is: Y1 of frames t1 to tn-1 is 0, while the output of frame tn is Y1 equal to 1 and Y2-Y10 equal to 0. (4) Loss calculation: Take the output Y1 from t1 to tn-1 and the output Y1 to Y10 from tn, and calculate the loss based on their difference from the true value; (5) Model iteration: Inputting frames t1 to tn into the model sequentially is considered as one iteration. All parameters in the process are recorded. Based on the backpropagation of the model loss, the correction values ​​of each model parameter from t1 to tn are calculated, and the average of the correction values ​​at all the same positions is used to update the model. Step Six, Model Inference: After training the model, use the same data preprocessing method as during training to flatten the 51 parameters of the keypoint data obtained in each frame and concatenate them with the 9 parameters Y2 to Y10 output by the model in the previous frame to form 60 parameters of input data. Input the input data into the model and obtain the model outputs Y1 to Y10. Here, Y1 is the counting result, and if it is 1, it means that the count is incremented by one. Y2 to Y10 will be used as X52 to X60 in the model input of the next frame.

2. The method for counting sports events based on deep learning networks according to claim 1, characterized in that: The annotation tool annotates the data related to the motion counting algorithm. It automatically splits the video into temporally continuous single-frame images. The default annotation result for each frame is 0. When the user browses each frame image and clicks the "increase count" button on the frame that meets the counting condition, the truth value corresponding to that frame is changed to 1.

3. The method for counting sports events based on deep learning networks according to claim 1, characterized in that: In the human keypoint detection algorithm, the position estimate of the human body's posture in the image is obtained from a human image. Single-person posture estimation is used. A single-person image is input into a trained deep learning network to obtain the predicted coordinate position of each human key in the image.

4. The method for counting sports events based on deep learning networks according to claim 3, characterized in that: The deep learning network includes MoveNet or HRNet.

5. The method for counting sports events based on deep learning networks according to claim 2, characterized in that: In obtaining the coordinates of various body key points of athletes, taking the COCO human key point data format as an example, there are a total of 17 human key points. Each key point has three parameters, namely the X coordinate, Y coordinate, and confidence level of the key point.

6. The method for counting sports events based on deep learning networks according to claim 1, characterized in that: In training data preprocessing, for sports such as rope skipping and sit-ups that are unrelated to the surrounding environment, in order to improve detection accuracy, the origin of the coordinate system is moved to the upper left corner of the human body, and the maximum value of the X and Y directions is taken as five times the distance between the knee and hip. Then, X and Y are normalized to 0-1. For sports such as pull-ups and shuttle runs that require environmental props, the maximum length and width of the screen are used to normalize X and Y to 0-1.

Images