A method for 2D human pose estimation key point jittering

Abstract

The present application relates to human posture estimation technical field, provide a kind of 2D human posture estimation key point jitter method, comprising: first, according to the distance between the human head and human hip coordinate points of model output, as the radius of human circumscribed circle, then according to the radius determines a human circumscribed rectangle, as the boundary box of human. As long as the IOU of subsequent frame and the boundary box of human is greater than the set threshold, target detection is not carried out, and the range determined by the boundary box is cut out to the image size of the posture estimation model input of next frame. For the case that the maximum value of heat map is unstable due to light, hand or camera shaking, the moving speed of the human body is calculated within the sliding window range, a weight is determined according to the moving speed, and a weighted average is obtained by using the current frame coordinates and the previously tracked saved coordinates as the final coordinates. Through the method of the present application, the running track of human key points can be accurately obtained, and collision can be more accurately detected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human pose estimation, and in particular to a method for key point jitter of 2D human pose estimation. BACKGROUND

[0002] The algorithm model of 2D human skeleton key point detection generally has two implementation modes of top-down (TopDown) and bottom-up (BottomUp). Currently, there are mainly two ways to obtain human key points: the first way is to train a heatmap (heat map) of key points, and the index position of the maximum value in the heatmap is the coordinate of the current key point; the second way is to obtain the key point coordinates by model regression based on the heatmap (heat map). Due to light, hand (or camera) shaking, and frame detection box coordinate deviation, input images are unstable, so that the key point coordinates finally output by the model are unstable (shaking). The reason is that most heat maps are generated by a Gaussian blur function (a self-defined Gaussian distribution function), and as long as the external environment is slightly unstable, the maximum value in the heat map will be unstable.

[0003] When the model is deployed on a mobile terminal, in addition to the need to ensure high accuracy, it also needs to be lightweight. In some real person interactive entertainment projects, the positions of human body organs (such as hands and feet) need to be accurately positioned, and then it is determined whether a collision occurs with objects in a virtual scene. To more accurately determine and better game experience, the human key point jitter problem must be solved.

[0004] A common method is to take the average value of multiple previous frame coordinates as the current frame coordinate. This method can achieve certain effect when the human body is relatively stationary. However, the frame number (sliding window size) for averaging is manually set according to experience, and it is more difficult to set a suitable value when the human body moves quickly. If the value is set too large, it will appear to be not coherent and slow. SUMMARY

[0005] The present application mainly solves the technical problem that the existing technology is more difficult to set a suitable value when the human body moves quickly, and sets too large to appear not coherent and slow. A method for key point jitter of 2D human pose estimation is provided to solve the problem of frame detection box coordinate deviation caused by the rapid movement of the human body, and to prevent the key point jitter of human pose estimation.

[0006] The present application provides a method for key point jitter of 2D human pose estimation, comprising:

[0007] S1, when the IOU of the current frame and the previous frame is less than a first threshold value, using the first human body bounding box output by target detection to crop the image as the input data of the human pose estimation model, and using the human pose estimation model to obtain a heat map;

[0008] S2, obtaining the distance between the coordinate points of two key points of the human body in the heat map as the radius of the human body circumscribed circle, and determining a human body circumscribed rectangle according to the radius of the human body circumscribed circle as a second human body bounding box;

[0009] S3, calculating the IOU of the second human body bounding box of the current frame and the previous frame, if less than the first threshold, re-executing step S1; if greater than the first threshold, continuing to judge whether greater than the second threshold;

[0010] S4, if greater than the second threshold, using the first human body bounding box; if less than the second threshold, using the second human body bounding box;

[0011] S5, the current frame human body key point coordinates in the heat map obtained by using the first human body bounding box / second human body bounding box and the coordinate points saved in step S2 are weighted averaged to obtain the final coordinate points of the more accurate human body key points.

[0012] Further, when the IOU of the current frame and the previous frame is less than the first threshold, the first human body bounding box output by the target detection is used to crop the image as the input data of the human pose estimation model, including: setting the first human body bounding box as bbox1, the common format of bbox1 output by the target detection is (p x , p y , w, h); wherein p x , p y are the upper left corner coordinates of bbox1, w and h are the bounding box width and height of bbox1 respectively; F(x), G(x), H(x), I(x) represent the inference composite function of the model, which is represented as: F(x)=p x , G(x)=p y , H(x)=w, I(x)=h.

[0013] Further, the distance between the coordinate points of two key points of the human body in the heat map is obtained as the radius of the human body circumscribed circle, and a human body circumscribed rectangle is determined according to the radius of the human body circumscribed circle as a second human body bounding box, including:

[0014] Select the top head tophead and the hip hip of the human body as the support points, obtain the distance between the two points tophead and hip as the radius r of the human body circumscribed circle, and then determine a circumscribed circle according to the radius r, and take the circumscribed rectangle of the circumscribed circle as the second human body bounding box, set as bbox2, the above process is represented by formulas (1)-(5):

[0015] Let p(x1, y1) and p(x2, y2) be the coordinates of the top head and hip, respectively;

[0016]

[0017] Bbox2 is:

[0018] x2-r=p x (2)

[0019] y2-r=p y (3)

[0020] 2r=w (4)

[0021] 2r=h (5).

[0022] Further, the IOU of the second human body bounding box of the current frame and the previous frame is calculated, comprising:

[0023] Let the bbox2 of the current frame be A, and the bbox2 of the previous frame be B, then the calculation method of IOU is:

[0024]

[0025] Further, the current frame human body key point coordinates in the heat map obtained by the first human body bounding box / second human body bounding box and the coordinate points saved in step S2 are weightedly averaged to obtain more accurate final coordinate points of the human body key points, comprising:

[0026] The final coordinate points are calculated by formula (7):

[0027] P update =wP old +(1-w)P new (7)

[0028] In the formula, p update is the final coordinate point, p old is the coordinate point saved in step S2, p new is the current frame human body key point coordinate, and w is the weight.

[0029] Further, the weight w is calculated by formula (8) and (9):

[0030]

[0031]

[0032] In the formula, v is the coordinate point moving speed, d k is the coordinate point moving distance of the kth frame relative to the k-1th frame, and t kis the time consumption of the kth frame, n is the number of sliding windows or the number of calculated frames.

[0033] The application provides a 2D human posture estimation key point jittering method, which comprises the following steps: firstly, calculating the distance between the head and the hip of a human body according to the model output, taking the distance as the radius of the human body circumscribed circle, and then determining a human body circumscribed rectangle according to the radius, taking the human body circumscribed rectangle as the bounding box of the human body. As long as the IOU of the subsequent frame and the bounding box is greater than a set threshold, target detection is not performed, and the range determined by the bounding box is used to crop the image size input into the posture estimation model of the next frame. For the case that the maximum value of the heatmap is unstable due to light, hand (or camera) shaking, the moving speed of the human body is calculated in the sliding window range, and a weight is determined according to the moving speed. The faster the moving speed, the greater the weight of the current frame coordinates, and vice versa. The current frame coordinates and the previously tracked and saved coordinates are weighted and averaged to obtain the final coordinates. Through optimization and adjustment, the human key point coordinates deployed on the mobile terminal are more stable, the jittering problem is obviously improved, the coordinate running track is smoother, the running track of the human key point can be accurately obtained, and collision can be more accurately detected.

[0034] The application has been applied to the product (Kungfu chain) of the applicant company, the Kungfu chain is an AI Kungfu product, which interacts and entertains through recognition of human key points, such as squatting, punching in horseback, shadow foot and Huashan Lunsword. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flow chart of the 2D human posture estimation key point jittering method provided by the application;

[0036] Figure 2 is a logic diagram of the 2D human posture estimation key point jittering method provided by the application;

[0037] Figure 3 is a human skeleton diagram in the embodiment. DETAILED DESCRIPTION

[0038] In order to make the technical problems solved by the application, the technical solutions adopted and the technical effects reached more clear, the application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the application, and not to limit the application. In addition, it should be noted that, in order to facilitate description, only the parts related to the application are shown in the drawings, not all the contents.

[0039] As Figure 1 and Figure 2As shown, the method for key point jitter of 2D human posture estimation provided by the embodiment of the application comprises:

[0040] S1, when the IOU of the current frame and the previous frame is less than a first threshold, using the first human body bounding box output by the target detection to crop the image as input data of the human posture estimation model, and using the human posture estimation model to obtain a heat map;

[0041] Specifically, when the IOU of the current frame and the previous frame is less than a first threshold, using the first human body bounding box output by the target detection to crop the image as input data of the human posture estimation model, comprising: setting the first human body bounding box as bbox1, the bbox1 output by the target detection is commonly in the format (p x , p y , w, h); wherein p x , p y are the upper left corner coordinates of bbox1, w and h are the width and height of the bounding box of bbox1, respectively;

[0042] F(x), G(x), H(x), I(x) represent the inference composite function of the model, and are expressed as: F(x) = p x , G(x) = p y , H(x) = w, I(x) = h.

[0043] S2, obtaining the distance between the coordinate points of two key points of the human body in the heat map as the radius of the human body circumscribed circle, and determining the human body circumscribed rectangle according to the radius of the human body circumscribed circle as a second human body bounding box;

[0044] Specifically, obtaining the distance between the coordinate points of two key points of the human body in the cropped image as the radius of the human body circumscribed circle, and determining the human body circumscribed rectangle according to the radius of the human body circumscribed circle as a second human body bounding box, comprising:

[0045] Selecting the top head tophead and the human body hip hip as support points, obtaining the distance between tophead and hip as the radius r of the human body circumscribed circle, and then determining a circumscribed circle according to the radius r, taking the circumscribed rectangle of the circumscribed circle as the second human body bounding box, and setting it as bbox2, the above process is expressed by formulas (1)-(5):

[0046] Let p(x1, y1) and p(x2, y2) be the coordinates of tophead and hip, respectively;

[0047]

[0048] bbox2 is:

[0049] x2-r = p x(2)

[0050] y2-r=p y (3)

[0051] 2r=w (4)

[0052] 2r=h (5).

[0053] S3, calculate the IOU of the second human body bounding box of the current frame and the previous frame. If it is less than the first threshold, re-execute step S1; if it is greater than the first threshold, continue to determine whether it is greater than the second threshold;

[0054] Assume that the bbox2 of the current frame is A and the bbox2 of the previous frame is B, then the IOU is calculated as follows:

[0055]

[0056] S4. If the value is greater than the second threshold, the first human body bounding box is used; if the value is less than the second threshold, the second human body bounding box is used;

[0057] S5. Take the weighted average of the coordinates of the human key points in the current frame in the heat map obtained using the first human body bounding box / the second human body bounding box and the coordinate points saved in step S2 to obtain a more accurate final coordinate point of the human key points.

[0058] Specifically, for the unstable maximum value of the heatmap caused by the shaking of light or human hands (or cameras), the final coordinate point is calculated by formula (7):

[0059] P update =wP old +(1-w)P new (7)

[0060] Where p update is the final coordinate point, p old is the coordinate point saved in step S2, p new is the coordinate of the human body key point in the current frame, and w is the weight.

[0061] The weight w is calculated by equations (8) and (9):

[0062]

[0063]

[0064] Where v is the moving speed of the coordinate point, d k t is the distance the kth frame moves relative to the k-1 frame coordinate point. k is the time taken for the kth frame, and n is the number of sliding windows or the number of frames calculated.

[0065] 2D case, the coordinate average can be calculated as follows, n is the sample size (sliding window size):

[0066]

[0067]

[0068] The average of the current frame and the previous frame can also be calculated, as follows (k is the current frame, k-1 is the previous frame):

[0069] x = (x k-1 +x k )

[0070] y = (y k-1 +y k )

[0071] The bounding box (object bounding box) is generally obtained by target detection, and common deep learning algorithms include yolo, faster-rcnn, mask-rcnn, etc. The general format is the left upper corner coordinate and the target width and height.

[0072] In Figure 2 ,

[0073] Image preprocessing: image scaling, cropping to target size;

[0074] Human target detection model inference: use the target detection model (such as yolov3) for forward inference to obtain the human bounding box (bbox1)

[0075] According to the bbox, crop the image: crop a person's image from the source input image according to the bounding box (bounding box)

[0076] Human pose estimation model inference: use the pose estimation model for forward inference to obtain the heat map (or directly regress to obtain the key points)

[0077] Generate head and hip feature points: according to the heat map, calculate the key point coordinates to determine the head and hip coordinate points

[0078] Calculate r and generate the circumscribed rectangle box bbox2: as formulas (1)-(5)

[0079] Calculate the iou of the current frame and the previous frame bbox2: as formula (6)

[0080] Iou>threshold1: such as threshold1=0.7

[0081] iou > threshold2: as threshold2 = 0.9

[0082] In practical applications, it is also necessary to set the key point id and name of the skeleton graph, such as Figure 3 The correspondence between the key point id and name is as follows:

[0083] (0, "Nose"), (1, "LEye"), (2, "REye"), (3, "LEar"), (4, "REar"),

[0084] (5, "LShoulder"), (6, "RShoulder"), (7, "LElbow"), (8, "RElbow"), (9, "LWrist"),

[0085] (10, "RWrist"), (11, "LHip"), (12, "RHip"), (13, "LKnee"), (14, "Rknee"),

[0086] (15, "LAnkle"), (16, "RAnkle"), (17, "Head"), (18, "Neck"), (19, "Hip"),

[0087] (20, "LBigToe"), (21, "RBigToe"), (22, "LSmallToe"), (23, "RSmallToe"), (24, "LHeel"),

[0088] (25, "RHeel"), (26, "LThumb"), (27, "RThumb"), (28, "LMiddle"), (29, "RMiddle")

[0089] The id corresponds to the Chinese name:

[0090] (0, "Nose"), (1, "Left Eye"), (2, "Right Eye"), (3, "Left Ear"), (4, "Right Ear"),

[0091] (5, "Left Shoulder"), (6, "Right Shoulder"), (7, "Left Elbow"), (8, "Right Elbow"), (9, "Left Wrist"),

[0092] (10, "Right Wrist"), (11, "Left Hip"), (12, "Right Hip"), (13, "Left Knee"), (14, "Right Knee"),

[0093] (15, "Left Ankle"), (16, "Right Ankle"), (17, "Head"), (18, "Neck"), (19, "Hip"),

[0094] (20, "left big toe"), (21, "right big toe"), (22, "left little toe"), (23, "right little toe"), (24, "left heel"),

[0095] (25, "right heel"), (26, "left thumb"), (27, "right thumb"), (28, "left middle finger"), (29, "right middle finger")

[0096] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: the modification of the technical solutions recorded in the foregoing embodiments, or the equivalent replacement of part or all of the technical features, does not make the essence of the corresponding technical solution deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for 2D human pose estimation keypoint jittering, characterized in that, The method comprises: S1, when the IOU of the current frame and the previous frame is less than a first threshold value, using a first human body bounding box output by target detection to crop an image as input data of a human body posture estimation model, and using the human body posture estimation model to obtain a heat map; S2, obtaining the distance between the coordinate points of two key points of the human body in the heat map as the radius of a human body circumscribed circle, and determining a human body circumscribed rectangle according to the radius of the human body circumscribed circle as a second human body bounding box; S3, calculating the IOU of the second human body bounding box of the current frame and the previous frame, if less than the first threshold value, re-executing step S1; if greater than the first threshold value, continuing to judge whether it is greater than a second threshold value; S4, if greater than the second threshold value, using the first human body bounding box; if less than the second threshold value, using the second human body bounding box; S5, using the current frame human body key point coordinates in the heat map obtained by the first human body bounding box / second human body bounding box and the coordinate points saved in step S2 to perform weighted averaging to obtain the final coordinate points of more accurate human body key points.

2. The method of 2D human pose estimation keypoint jittering according to claim 1, wherein, The first human body bounding box output by the target detection is used to crop the image as input data of the human pose estimation model when the IOU of the current frame and the previous frame is less than the first threshold, comprising: setting the first human body bounding box as bbox1, the common format of the bbox1 output by the target detection is (p x , p y , w, h); wherein, p x , p y are the upper left corner coordinates of the bbox1, and w and h are the width and height of the bounding box of the bbox1, respectively. F(x), G(x), H(x), I(x) represent the inference composite functions of the model, expressed as: F(x) = p x , G(x) = p y , H(x) = w, I(x) = h.

3. The method of 2D human pose estimation keypoint jittering according to claim 2, wherein, The method comprises: Selecting the top head tophead and the human body hip hip as support points, obtaining the distance between the two points as the radius r of the human body circumscribed circle, and then determining a circumscribed circle according to the radius r, taking the circumscribed rectangle of the circumscribed circle as the second human body bounding box, and setting it as bbox2, the above process is represented by formulas (1)-(5): Let p(x1, y1) and p(x2, y2) be the coordinates of tophead and hip respectively; The IOU of the current frame and the previous frame is calculated as follows: x2-r = p x (2) y2-r = p y (3) 2r＝w (4) 2r＝h (5)。 4. The method of 2D human pose estimation keypoint jittering according to claim 3, characterized in that, The IOU of the current frame and the previous frame is calculated as follows: The IOU of the current frame and the previous frame is calculated as follows:

5. The method of 2D human pose estimation keypoint jittering according to claim 4, characterized in that, The IOU of the current frame and the previous frame is calculated as follows: The IOU of the current frame and the previous frame is calculated as follows: P update = wP old + (1 - w)P new (7) In the formula, p update is the final coordinate point, p old is the coordinate point saved in step S2, p new is the current frame human body key point coordinate, and w is a weight.

6. The method of 2D human pose estimation keypoint jittering according to claim 5, wherein, The IOU of the current frame and the previous frame is calculated as follows: The IOU of the current frame and the previous frame is calculated as follows: In the formula, v is the coordinate point moving speed, d k is the coordinate point moving distance of the kth frame relative to the k-1th frame, t k is the time consumption of the kth frame, and n is the number of sliding windows or the number of calculated frames.

Images