A multi-human pose tracking method based on human key points

Abstract

The application discloses a multi-human posture tracking method based on human key points, detects human bodies in a video picture; carries out target correlation matching on the detected human bodies and tracked targets, carries out human key point tracking screening on initial new targets which are not successfully matched, deletes false targets, and adopts human key point tracking on real human bodies and human bodies which are successfully matched. The human tracking area is expanded, human key point coordinates and confidence are obtained, human crossing and shielding are judged to exclude repeated tracking, the tracking target is deleted, and position information and key point information of the human tracking target are output; the cycle is executed until human target tracking is completed. The application is based on a light-weight human key point MOT tracking framework, can effectively track multi-human targets, and has the advantages of fast model reasoning speed, small human tracking frame fluctuation, strong tracking stability, output of human posture estimation information and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a multi-human pose tracking method based on human key points. BACKGROUND

[0002] Multi-human target tracking is to detect multiple targets in a video picture and track the trajectory of each target and assign an ID. It has wide application value in TV scene AI home mobile frame detection, body sensing game interaction, DPTZ stability improvement, virtual reality and augmented reality, etc. The existing multi-human target tracking MOT framework mainly includes: (1) tracking by detection, such as Sort / DeepSort; (2) joint detection and tracking, such as JDE, CenterTrack, etc.; (3) based on attention mechanism, such as TransTrack, TrackFormer, etc. The processing process mainly includes four steps: (1) analyze the input video by frame number; (2) obtain the target detection box of the original frame of the video through the target detection network; (3) extract the features (motion or semantic features) of the detected target box and calculate the similarity between the front and back video frames; (4) data association, match the target box and the corresponding trajectory and ID. Although the above methods can realize multi-human target tracking, they have the following shortcomings: 1. They are heavily dependent on human body detection algorithms, the human body tracking speed is slow, the human body tracking box fluctuates greatly, the tracking stability is poor, and the target is easy to be lost. 2. The multi-human target tracking framework implementation process is complex, and only the human body target position can be output, and the human body pose estimation information cannot be obtained. SUMMARY

[0003] The present application aims to provide a multi-human pose tracking method based on human key points, which solves the problems of slow speed, poor tracking stability, easy to lose targets, and only outputting human body target position in the prior art.

[0004] The present application solves the above problems by the following technical solutions:

[0005] A multi-human pose tracking method based on human key points, comprising:

[0006] Step S1, detecting the human body in the video picture;

[0007] Step S2, matching the detected human body with the tracked target, if the matching is successful, jumping to step S4, if the matching is not successful, an initial new target appears, jumping to step S3;

[0008] Step S3, human key point tracking screening is performed on the initial new target. If it is determined that the initial new target is a false target, the initial new target is deleted. If it is determined that the initial new target is a real human body, the process jumps to step S4.

[0009] Step S4, the human body tracking area is expanded.

[0010] Step S5, image data of each human body tracking area is obtained. After scaling processing, the image data is input into a lightweight human key point model for inference operation. According to an inference operation result of the lightweight human key point model, human key point coordinate positions and human key point confidence are obtained. A human key point minimum circumscribed rectangle, a human body tracking expansion area and human body confidence information are calculated according to the human key point coordinate positions. The human body confidence is an average value of the human key point confidence.

[0011] According to the human key point minimum circumscribed rectangle, the size of the human body is calculated to determine whether the cross occlusion between human bodies exceeds a first threshold value. If the cross occlusion exceeds the first threshold value, a human body area larger target is retained to avoid a repeated tracking human body situation.

[0012] Step S6, whether the human body target exceeds a video picture boundary, whether the human body target area is smaller than a preset value, and whether a continuous multiple frame human body confidence is lower than a second threshold value are determined according to the human body tracking expansion area. If one of the conditions is met, it is determined that the tracking target is lost. The tracking target is deleted. Otherwise, position information and key point information of the tracked human body target are output. If a video frame interval is greater than a third threshold value, steps S1-S6 are cycled. Otherwise, steps S4-S6 are cycled until the human body target tracking is completed.

[0013] Further, in step S2, IOU calculation and Hungarian algorithm matching are performed between the M human body targets detected and the N human body targets tracked.

[0014] Further, step S3 specifically includes the following steps.

[0015] Step S31, the human body tracking area is expanded.

[0016] Step S32, image data of each human body tracking area is obtained. After scaling processing, the image data is input into a lightweight human key point model for inference operation. According to an inference operation result of the lightweight human key point model, human key point coordinate positions and human body confidence are obtained. If a continuous K frame human body confidence is greater than a fourth threshold value, the initial new target is a real human body. Otherwise, the initial new target is determined to be a false target.

[0017] Further, the expansion of the human body tracking area specifically includes the following steps.

[0018] A human body detection rectangle [x d ,y d ,wd ,h d The initial center and radius of the human body are converted into the tracking radius, and the tracking radius is expanded by a factor of 1 to ensure that the human body tracking area can cover the entire human body ROI region. t ,cy t ,scale t *cr t The extended calculation formula is as follows:

[0019] cx t =x d +w d / 2, cy t =y d +h d / 2

[0020] cr t =sqrt((w d / 2) 2 +(h d / 2) 2 )

[0021] scale t =1.25

[0022] Where x d The x-axis starting coordinate of the top-left corner of the human detection rectangle; y d The starting y-coordinate of the top-left corner of the human detection rectangle; w d h is the width of the human body detection rectangle. d Detects the height of a rectangle for the human body; cx t The x-coordinate of the extended center for human body tracking; cy t The y-coordinate of the extended center for human body tracking; cr t To extend the radius for human body tracking; scale t The radius of the human body tracking extension is increased by a factor of 1.

[0023] Furthermore, based on the model's inference results, the coordinates of each key point on the human body [kx, ky, ks] are obtained. 14 and human confidence score t Calculate the minimum bounding rectangle [x] of the human body tracking target based on the coordinates of key human body points. t ,y t ,w t ,h t ] m To ensure a stable output of the minimum bounding rectangle of the human body, the confidence level (ks) of the human keypoints involved in the calculation is required. i >0.5, and simultaneously calculate the extended region of interest (ROI) for human tracking based on the coordinates of key human points [cx]. t ,cyt scale t *cr t ] and human body confidence score t , whose calculation formula is as follows:

[0024] sx t =kx tophead , sy t =ky tophead

[0025] cx t =(kx leftshoulder +kx rightshoulder +4*kx lefthip +4*kx righthip ) / 10

[0026] cy t =(ky leftshoulder +ky rightshoulder +4*ky lefthip +4*ky righthip ) / 10

[0027] cr t =sqrt((cx t -sx t ) 2 +(cy t -sy t ) 2 )

[0028] scale t =1.25

[0029] score t =∑ks i / 14

[0030] Wherein, kx tophead is the head top key point x coordinate; ky tophead is the head top key point y coordinate; sx t is the head top key point x coordinate; sy t is the head top key point y coordinate; kx leftshoulder is the left shoulder key point x coordinate; kx rightshoulder is the right shoulder key point x coordinate; kx lefthip is the left hip key point x coordinate; kx righthip is the right hip key point x coordinate; ky leftshoulder is the left shoulder key point y coordinate; ky leftshoulder is the right shoulder key point y coordinate; ky lefthip is the left hip key point y coordinate; ky righthip is the right hip key point y coordinate.

[0031] Further, the step S6 specifically comprises:

[0032] According to the human body tracking extended region ROI[cx t ,cy t ,scale t *cr t ], it is judged whether the human body center exceeds the video picture boundary, that is, cx t <0, cy t <0, cx t >fw, cy t >fh, or whether the human body area satisfies π*cr t *cr t <300, or whether the continuous K frame human body confidence score t <0.5, if one of the conditions is met, it is judged that the tracking target is lost, and the corresponding tracking target is deleted, otherwise the minimum circumscribed rectangle frame [x t ,y t ,w t ,h t ] of the human body tracking target and the human body key point coordinates [kx, ky, ks] m are output. 14 Wherein, fw is the video frame image width, and fh is the video frame image height.

[0033] If the video frame interval is greater than a threshold Th=20, the steps S1-S6 are cycled to re-perform human body detection, otherwise the steps S4-S6 are cycled.

[0034] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0035] The present application is based on a lightweight human body key point MOT tracking framework, can effectively track multiple human body targets, the algorithm framework relies on human body detection to a lower extent, and is mainly based on lightweight human body key point tracking; has the advantages of fast model reasoning speed, small human body tracking frame fluctuation, strong tracking stability, and can output human body posture estimation information. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0037] The present application will be further described in detail below in conjunction with embodiments, but the embodiments of the present application are not limited thereto.

[0038] Embodiment:

[0039] In conjunction with the accompanying Figure 1 Fig. 1 shows a human body key point based multi-human posture tracking method, comprising the following steps:

[0040] Step 1: Detect human body in video frame by human detection algorithm.

[0041] Step 2: Perform target association matching on the detected human body in the video frame, that is, detect M human body targets [x, y, w, h] i and N tracked human body targets [x t ,y t ,w t ,h t ] j Perform IOU i,j calculation and Hungarian algorithm matching. The human body detection target matching result has two cases: successful matching or unsuccessful matching. If the matching is unsuccessful, it is judged as an initial new target. If the matching is successful, perform human key point tracking.

[0042] Step 3: If an initial new target is determined to exist, perform human key point tracking screening on the initial new target to exclude false targets that may be caused by human body misdetection. The screening condition is that the human body confidence score is greater than a threshold value for consecutive multiple frames, for example, set the human body confidence score score t > 0.5 for consecutive K >= 5 frames, then determine that it is a real human body target, and continue to perform human key point tracking on the real human body target. Otherwise, determine that it is a false target and perform deletion processing, where the human body confidence is the average value of the human key point confidence.

[0043] The steps of the lightweight human key point tracking method are as follows:

[0044] Step a: First, convert the human detection rectangle frame [x d ,y d ,w d ,h d ] to the initial center and radius of human tracking, and simultaneously perform tracking radius expansion to make the expanded area cover the entire human body ROI area ROI [cx t ,cy t ,scale t *cr t ]. The expansion calculation formula is as follows:

[0045] cx t = x d + w d / 2, cy t = y d + h d / 2, cr t = sqrt((w d / 2) 2 + (h d / 2) 2 ), scalet =1.25

[0046] Step b: Next, obtain the ROI [cx] for each human body. t ,cy t ,scale t *cr t The acquired image data is scaled to the model input size and then input into a lightweight human keypoint model for inference calculation, outputting 14*3 dimensional human keypoint information.

[0047] Step c: Based on the model inference results, obtain the coordinates and confidence scores [kx, ky, ks] of each human body keypoint. 14 Calculate the minimum bounding rectangle of the human body based on key points of the human body [x t ,y t ,w t ,h t ] m To ensure a stable human tracking bounding box output, the confidence level ks of the human key points is required during the calculation of the minimum bounding rectangle. i >0.5. Simultaneously, the ROI [cx] of the human body tracking extension region is calculated based on the human body key point information. t ,cy t ,scale t *cr t and human confidence score t The calculation formula is as follows:

[0048] sx t =kx tophead sy t =ky tophead

[0049] cx t =(kx leftshoulder +kx rightshoulder +4*kx lefthip +4*kx righthip ) / 10

[0050] cy t =(ky) leftshoulder +ky rightshoulder +4*ky lefthip +4*ky righthip ) / 10

[0051] cr t =sqrt((cx t -sx t ) 2 +(cy t -sy t ) 2 )

[0052] scale t =1.25

[0053] score t =∑ks i / 14

[0054] Step d: Based on the minimum bounding rectangle of each human body key point [x t ,y t ,w t ,h t ] m Calculate the IOU between human bodies i,j Size is used to determine if there is severe cross-occlusion between human bodies; if IOU i,j A value greater than 0.75 indicates severe cross-occlusion between human bodies.

[0055] Step e: If it is determined that there is serious cross occlusion between human bodies, then retain the target with the larger human body area to avoid repeated tracking of human bodies.

[0056] Step f: Expand the ROI based on human body tracking [cx] t ,cy t ,scale t *cr t [This determines whether the center of the human body exceeds the boundary of the video frame, i.e., cx] t <0, cy t <0, cx t >fw、cy t >fh, or whether the human body area satisfies π*cr t *cr t <300, or consecutive K-frame human confidence score t If the value is less than 0.5, and one of the following conditions is met, the tracking target is considered lost and the corresponding tracking target is deleted; otherwise, the minimum bounding rectangle of the human tracking target [x] is output. t ,y t ,w t ,h t ] m And the coordinates of key points on the human body [kx,ky,ks] 14 , where fw is the width of the video frame image and fh is the height of the video frame image.

[0057] Step g: If the video frame interval is greater than the threshold Th = 20, return to step 1 and repeat the human detection; otherwise, repeat step ag.

[0058] Although the present application has been described with reference to the explanatory embodiments thereof, the above embodiments are merely the preferred embodiments of the present application, and the embodiments of the present application are not limited to the above embodiments, and it should be understood that many other modifications and embodiments can be devised by those skilled in the art, and such modifications and embodiments will fall within the scope and spirit of the principles disclosed in the present application.

Claims

1. A multi-human pose tracking method based on human key points, characterized in that, The method comprises the following steps: Step S1, detecting a human body in a video frame; Step S2, performing target association matching on the detected human body and a tracked target, if the matching is successful, jumping to step S4, if the matching is not successful, an initial new target appears, and jumping to step S3; Step S3, performing human key point tracking screening on the initial new target, if the initial new target is determined to be a false target, the initial new target is deleted, if the initial new target is determined to be a real human body, jumping to step S4; Step S4, expanding a human body tracking area; Step S5, obtaining image data of each human body tracking area, inputting the image data into a lightweight human key point model after scaling processing for inference operation, obtaining human key point coordinate positions and human key point confidence according to an inference operation result of the lightweight human key point model, and calculating a human key point minimum bounding box, a human body tracking expansion area, and human body confidence information according to the human key point coordinate positions, wherein the human body confidence is an average value of the human key point confidence; According to the human key point minimum bounding box, the size of the human body is calculated to determine whether the cross occlusion between the human bodies exceeds a first threshold value, if the cross occlusion exceeds the first threshold value, a human body area larger target is retained to avoid a repeated tracking human body situation; Step S6, determining whether a human body target exceeds a video frame boundary, whether a human body target area is smaller than a preset value, and whether a human body confidence of continuous multiple frames is lower than a second threshold value according to the human body tracking expansion area, if one of the conditions is met, it is determined that the tracked target is lost, and the tracked target is deleted, otherwise, position information and key point information of the tracked human body target are output; if a video frame interval is greater than a third threshold value, steps S1-S6 are repeated, otherwise, steps S4-S6 are repeated, until the human body target tracking is completed.

2. The multi-person pose tracking method based on human key points according to claim 1, characterized in that, In step S2, IOU calculation and Hungarian algorithm matching are performed on M detected human body targets and N tracked human body targets.

3. The multi-person pose tracking method based on human key points according to claim 1 or 2, characterized in that, Step S3 specifically comprises the following steps: Step S31, expanding a human body tracking area; Step S32, obtaining image data of each human body tracking area, inputting the image data into a lightweight human key point model after scaling processing for inference operation, and obtaining human key point coordinate positions and human body confidence according to an inference operation result of the lightweight human key point model; if human body confidence of continuous K frames is greater than a fourth threshold value, the initial new target is a real human body, otherwise, the initial new target is determined to be a false target.

4. The multi-person pose tracking method based on human key points according to claim 3, characterized in that, The expanded human body tracking area specifically comprises: A human body detection rectangle frame The initial center and radius of the human body tracking target are converted, and a tracking radius multiple expansion is performed to enable the human body tracking expansion area to cover the entire human body ROI region The expansion calculation formula is as follows: ， ； ； ； wherein is a starting coordinate of the x-axis of the upper left corner of the human detection rectangular frame; is a starting coordinate of the y-axis of the upper left corner of the human detection rectangular frame; is the width of the human detection rectangular frame; is the height of the human detection rectangular frame; is the x-coordinate of the human tracking expansion center; is the y-coordinate of the human tracking expansion center; is the human tracking expansion radius; is the human tracking expansion radius multiple.

5. The multi-person pose tracking method based on human key points according to claim 4, characterized in that, In the step S5, the model inference operation result is obtained to acquire the human key point coordinate position and the human body confidence , the human body key point coordinates are calculated to obtain the minimum bounding rectangle frame of the human body tracking target To ensure the output of stable human body minimum bounding rectangle frame, the human body key point confidence participating in the calculation is required , and the human body tracking extended area is calculated according to the human body key point coordinates and the human body confidence , and the calculation formula is as follows: ， ； ； ； ； ； ； wherein, is the head top keypoint x coordinate; is the head top keypoint y coordinate; is the head top keypoint x coordinate; is the head top keypoint y coordinate; is the left shoulder keypoint x coordinate; is the right shoulder keypoint x coordinate; is the left hip keypoint x coordinate; is the right hip keypoint x coordinate; is the left shoulder keypoint y coordinate; is the right shoulder keypoint y coordinate; is the left hip keypoint y coordinate; is the right hip keypoint y coordinate.

6. The multi-person pose tracking method based on human key points according to claim 5, characterized in that, Step S6 specifically comprises: According to human body tracking extended area , whether the human body center exceeds the video frame boundary, i.e. , , , , or whether the human body area meets , or the human body confidence in consecutive frames meets , one of the conditions is met, it is judged that the tracking target is lost, then the corresponding tracking target is deleted, otherwise the minimum external rectangle frame of the human body tracking target and the human body key point coordinates are output, wherein is the video frame image width, is the video frame image height; If the video frame interval is greater than the threshold The steps S1-S6 are repeated for human detection, otherwise the steps S4-S6 are repeated.

Images