A posture tracking method based on key point screening

Abstract

The present invention discloses a posture tracking method based on key point screening, constructs a multi-person posture tracking model based on key point screening, and includes a target detection module, a posture estimation and key point screening module, a matching module, and a pedestrian re-identification module. Through the technical solution of the present invention, it is possible to solve the problem that the threshold of human key points in multi-person posture tracking tasks is difficult to adjust, and the problem of low multi-target tracking accuracy (MOTA) index caused by low key point recognition can be solved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a posture tracking method based on key point screening. Background Art

[0002] The multi-person posture tracking task in the field of computer vision refers to processing the input video images, detecting the posture of each pedestrian in each frame of the video, and then calculating and analyzing the appearance characteristics, position, motion state and other information of the target to correctly record the continuous posture trajectory of each person over time. As a hot issue in the field of computer vision, human posture tracking is the basis for almost all human-related research in computer vision. It has a wide range of applications in human-computer interaction, video surveillance, activity recognition, and sports video analysis. These practical needs have aroused great interest in this topic.

[0003] Existing multi-person posture tracking methods usually adopt a top-down framework. First, the bounding box information of each human body is detected through the multi-person posture estimation method, and then the posture information of each human body bounding box is estimated. Next, the information of the human body to be tracked in the current frame of the video is matched with the information obtained in the next frame.

[0004] Although multi-person pose tracking technology has been widely used, multi-person pose estimation and tracking not only need to consider solving the common challenging problems in multi-target tracking, such as scale changes, plane rotation, and illumination changes, but also the complex changes in people's poses in the video, which makes multi-person pose estimation and tracking difficult. In addition, in the Pose Track Challenge dataset, key points below a certain threshold need to be cleared. All previous methods adjusted the threshold based on experience, and the choice of threshold often affects the final performance of the model. These problems make the model's multi-target tracking accuracy (MOTA) indicator still need to be improved. Summary of the invention

[0005] In order to solve the shortcomings of the above-mentioned existing technologies, the present invention proposes a posture tracking method based on key point screening, which can solve the problem that the threshold of human key points is difficult to adjust in multi-person posture tracking tasks, and the problem that the multi-target tracking accuracy (MOTA) index is low due to low key point recognition. The specific technical solution of the present invention is as follows:

[0006] A posture tracking method based on key point screening includes the following steps:

[0007] S1: Construct dataset;

[0008] S1-1: Use the COCO 2017 dataset for detection and pose estimation, and the PoseTrack 2018 dataset for pose estimation and tracking;

[0009] S1-2: Based on the PoseTrack 2018 dataset, a person re-identification dataset is constructed. All human frames in the PoseTrack 2018 dataset are cropped and saved as images. The same labels are given to human frames with IDs in the same video. Finally, a person re-identification dataset containing 4613 labels and a total of 119656 images is generated.

[0010] S2: Construct a multi-person pose tracking model based on a top-down framework and a general keypoint screening network;

[0011] S3: Model training;

[0012] S4: Input the video to be tested into the model to obtain the detection result.

[0013] Furthermore, the multi-person posture tracking model includes a target detection module, a posture estimation and key point screening module, a matching module and a pedestrian re-identification module, wherein:

[0014] The target detection module is based on a multi-task multi-stage hybrid cascade structure model, namely the HTC model. The backbone network of the HTC model uses a 101-layer ResNeXt network, and adopts a feature graph pyramid network, namely the FPN network, and a non-maximum suppression algorithm based on the PCK indicator;

[0015] The posture estimation and key point screening module adopts a universal posture estimation network, and obtains the heat map and existence probability of each key point through the key point prediction module and the key point screening module, and finally obtains the human body posture;

[0016] The matching module performs data association between the human body posture of the current frame and the existing human body posture trajectory, and uses the Hungarian algorithm based on the intersection-over-union distance of the bounding box to associate; for the human body with lost trajectory, it is added to the pedestrian re-identification module; when a new human body appears, the pedestrian re-identification module is used to retrieve the lost human body trajectory.

[0017] Furthermore, the non-maximum suppression algorithm based on the PCK indicator uses the PCK distance as a similarity measure between two human body frames, and when it is confirmed that one human body frame is retained, other human body frames whose PCK distance is greater than a set threshold are deleted.

[0018] Furthermore, the key point prediction module and the key point screening module adopt the same network structure, which is composed of three layers of transposed convolution with a convolution kernel of 3x3. The key point prediction module finally generates a heat map for each key point, and the key point screening module obtains the binary classification probability of each key point through a fully connected layer for the feature map of each key point generated.

[0019] Furthermore, the pedestrian re-identification module uses the posture estimation network HRNet as the backbone network to extract features to obtain the feature map of each person, and associates the existing posture trajectory with the posture of the current frame each time, and saves the lost trajectory; for the posture that fails to match successfully, the pedestrian re-identification module is used to compare with the lost pedestrian trajectory. If the comparison is successful, the new posture is merged with the lost posture trajectory. If the comparison fails, a new trajectory is generated;

[0020] In the training phase, each person with the same label is set as a category. After obtaining the feature map of each person, it is connected to the classification network for classification, and multi-classification cross entropy is used as its loss function;

[0021] During the posture tracking process, the feature map of each person is extracted and matched with the lost pedestrians. The L1 distance, i.e., Hamiltonian distance, is used to compare the distance between the two people. When the distance is less than the threshold, they are considered to be the same person.

[0022] Furthermore, the heat map output by the key point prediction module will be smoothed by a second-order Gaussian filter.

[0023] Furthermore, the step S3 is to train the models separately, specifically:

[0024] S3-1: The object detection module uses the HTC detector, and the mAP reaches 0.346 on the PoseTrack 2018 validation set. The object detection module is only trained on the COCO 2017 dataset and is not fine-tuned on the PoseTrack 2018 dataset.

[0025] S3-2: The pose estimation and key point screening module is trained on COCO2017, and the trained model is fine-tuned on the PoseTrack 2018 dataset. Since only key points are annotated, the bounding box of all its key points is extended by 20% to obtain the basic truth box of a person instance. By extending the bounding box in height or width, the human body bounding box is made into a fixed aspect ratio;

[0026] In the training of the pose estimation and key point screening modules, pose data is enhanced, including random scaling, random rotation, and random flipping, using Adam as the optimizer;

[0027] S3-3: The pedestrian re-identification module crops pedestrians with the same label from the original image of the PoseTrack 2018 dataset to form a new dataset. The HRNet network pre-trained with the PoseTrack 2018 dataset is used as the backbone network. During training, the feature maps output by the HRNet network are classified through the fully connected layer. During verification, only the feature maps output by the HRNet network are used, and no longer through the fully connected layer.

[0028] The beneficial effect of the present invention is that: on the basis of the existing top-down multi-person posture estimation method, the present invention adds a universal key point screening network to the existing posture estimation network, thereby improving the accuracy of the model and reducing the influence of the key point threshold on the tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0030] Figure 1 is a network flow chart of the present invention;

[0031] Figure 2 It is the posture estimation and key point screening module of the present invention;

[0032] Figure 3 It is the HRNet network structure;

[0033] Figure 4 It is the network structure of the key point screening module;

[0034] Figure 5 It is the visualization result of posture estimation;

[0035] Figure 6 It is the posture tracking visualization result. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0038] The present invention can add a universal key point screening model to the existing top-down multi-person posture estimation method, thereby improving the accuracy of the model and reducing the influence of the key point threshold on the tracking accuracy.

[0039] A posture tracking method based on key point screening includes the following steps:

[0040] S1: Construct dataset;

[0041] S1-1: Use the COCO 2017 dataset for detection and pose estimation, and the PoseTrack 2018 dataset for pose estimation and tracking;

[0042] S1-2: Based on the PoseTrack 2018 dataset, a person re-identification dataset is constructed. All human frames in the PoseTrack 2018 dataset are cropped and saved as images. The same labels are given to human frames with IDs in the same video. Finally, a person re-identification dataset containing 4613 labels and a total of 119656 images is generated.

[0043] S2: Construct a multi-person pose tracking model based on a top-down framework and a general keypoint screening network;

[0044] S3: Model training;

[0045] S4: Input the video to be tested into the model to obtain the detection result.

[0046] like Figure 1-4 As shown, the present invention constructs a multi-person posture tracking model based on a top-down framework and a universal key point screening network, including a target detection module, a posture estimation and key point screening module, a matching module, and a pedestrian re-identification module. The target detection module is a hybrid cascade structure model based on multi-task and multi-stage, namely, the HTC model. The backbone network of the HTC model uses a 101-layer ResNeXt network, and adopts a feature map pyramid network, namely, the FPN network, and a non-maximum suppression algorithm based on the PCK indicator;

[0047] The posture estimation and key point screening module uses a general posture estimation network to obtain the heat map and existence probability of each key point through the key point prediction module and the key point screening module, and finally obtains the human body posture;

[0048] The matching module associates the human body posture of the current frame with the existing human body posture trajectory using the Hungarian algorithm based on the intersection-over-union distance of the bounding box. For people whose trajectories are lost, they are added to the pedestrian re-identification module. When a new person appears, the pedestrian re-identification module is used to retrieve the lost human trajectory.

[0049] (1) The object detection module is based on a multi-task and multi-stage hybrid cascade structure (HTC) model. It uses ResNeXt as the backbone network for feature extraction, uses the feature pyramid network (FPN) for multi-scale testing, and finally uses the PCK distance to perform a non-maximum suppression algorithm to filter out redundant human proposal boxes, thereby obtaining the final human bounding box.

[0050] Each stage of the HTC network combines cascading and multi-tasking to improve information flow, and uses spatial context to further improve accuracy. The backbone network of the HTC model uses a 101-layer ResNeXt network. ResNeXt is a combination of a deep residual network (ResNet) and Inception. Unlike Inception v4, ResNext does not require manual design of complex Inception structural details, but each branch uses the same topology. The essence of ResNeXt is grouped convolution, which controls the number of groups by variable cardinality.

[0051] The FPN network solves the multi-scale problem in human body detection. By simply changing the network connection, the performance of small human body detection is greatly improved without increasing the computational complexity of the original model. In addition, the non-maximum suppression algorithm based on the PCK indicator is used to eliminate redundant human body proposal frames. PCK is used to calculate the ratio of the normalized distance between the key points of the detection that is less than the set threshold. In the case of interactive occlusion of the human body, the traditional IOU distance is prone to incorrectly filter the interactively occluded human body, while the PCK distance can better retain the human body proposal frame in the case of interactive occlusion through posture information, which is more suitable for human body detection than the IOU indicator used in the traditional detection model. The PCK distance formula is:

[0052]

[0053] Among them, i represents the key point with ID i, and k represents the kth threshold T k , p represents the pth pedestrian, d pqi The Euclidean distance between the predicted value of the key point with id i in the pth person and the predicted value of the key point with id i in the qth person, represents the scale factor of the pth person, T k This is a manually set threshold.

[0054] (2) The backbone network of the pose estimation and key point screening module is as follows Figure 2 As shown in Figure 1, its backbone network uses a high-resolution sub-network (HRNet). Figure 3 As shown in the figure, HRNet is achieved by gradually adding low-resolution feature map sub-networks in parallel to the high-resolution feature map main network, and different networks achieve multi-scale fusion and feature extraction.

[0055] The present invention uses the third layer output in the multi-resolution output of the HRNet network as the feature extraction part of the posture estimation model to obtain the human body posture feature map, and then obtains the heat map and existence probability of each key point through the posture estimation module and the key point screening module respectively.

[0056] The key point prediction module and the key point screening module use the same network structure, which consists of three layers of transposed convolution with a convolution kernel of 3x3. The posture estimation module finally generates a heat map for each key point, and the key point screening module uses the fully connected layer to obtain the binary classification probability of each key point through the feature map of each key point generated on the basis of the transposed convolution. In addition, the heat map output by the key point prediction module will also be smoothed by a second-order Gaussian filter to obtain a more accurate key point position.

[0057] Specifically, the output of the key point prediction module is a heat map of 15 key points. The labels of the key points are probability maps of Gaussian distribution. The loss function of the training is:

[0058]

[0059] Among them, i represents the i-th key point, j represents the probability that the j-th position in the heat map is a key point, represents the probability of the i-th key point in the label at the j-th position, represents the probability of the i-th key point predicted by the network at the j-th position, v i Indicates whether the i-th key point is visible, v i =0 means the key point is invisible and is not included in the network loss.

[0060] The output of the key point screening module is the binary classification probability of 15 key points. The labels of the key points are one-hot binary distribution, and the loss function is:

[0061]

[0062] Among them, y i Indicates whether the i-th key point is visible, z i represents the visibility of the i-th keypoint prediction.

[0063] (3) The matching module associates the human body posture of the current frame with the existing posture trajectory of the previous frame using the Hungarian algorithm based on the intersection-over-union (IOU) distance of the bounding box. The calculation formula of the IOU distance is as follows:

[0064]

[0065] Among them, box1 is the bounding box corresponding to the human body posture in the current frame and box2 is the bounding box corresponding to the existing posture trajectory in the previous frame, and area is the area of ​​the calculated box.

[0066] (4) A person re-identification module uses the posture estimation network HRNet as the backbone network to extract features and obtain feature maps for each person. Each time, the existing posture trajectory is associated with the posture of the current frame, and the lost trajectory is saved. For postures that fail to match successfully, the person re-identification module is used to compare with the lost pedestrian trajectory. If the comparison is successful, the new posture is merged with the lost posture trajectory. If the comparison fails, a new trajectory is generated.

[0067] In the training phase, each person with the same label is set as a category. After obtaining the feature map of each person, it is connected to the classification network for classification, and multi-classification cross entropy is used as its loss function;

[0068] During the posture tracking process, the feature map of each person is extracted and matched with the lost pedestrians. The L1 distance, i.e., Hamiltonian distance, is used to compare the distance between the two people. When the distance is less than the threshold, they are considered to be the same person.

[0069] In addition, the network model of the present invention is trained in modules:

[0070] S3-1: The object detection module uses the HTC detector, and the mAP reaches 0.346 on the PoseTrack 2018 validation set. The object detection module is only trained on the COCO 2017 dataset and is not fine-tuned on the PoseTrack 2018 dataset.

[0071] S3-2: The pose estimation and key point screening module is trained on COCO 2017, and the trained model is fine-tuned on the PoseTrack 2018 dataset. Since only key points are annotated, the human body bounding box is obtained by extending the bounding box corresponding to all its key points by 20%. The human body bounding box is made into a fixed aspect ratio by extending the bounding box in height or width; for example, height: width = 4:3, the resolution adopted is 384:288, and the size of the heat map is one-fourth of the resolution, that is, 96:72.

[0072] In the training of the pose estimation and key point screening modules, the pose data is enhanced, including random scaling (30%), random rotation (40 degrees) and random flipping, and Adam is used as the optimizer.

[0073] S3-3: The pedestrian re-identification module crops pedestrians with the same label from the original image of the PoseTrack 2018 dataset to form a new dataset, which contains a total of 4613 pedestrian labels and 119656 images. It uses the HRNet network pre-trained with the PoseTrack2018 dataset as the backbone network. During training, the feature maps output by the HRNet network are classified through the fully connected layer. During verification, only the feature maps output by the HRNet network are used, and no longer through the fully connected layer.

[0074] Example 1

[0075] In order to verify the effectiveness of each module of the model, ablation experiments are performed on key modules. As shown in Table 1, the experimental process includes a benchmark experiment and the continuous addition of two modules on this basis:

[0076] Benchmark experiments: HTC detector, HRNet pose estimation network, and IOU matching are used as benchmark methods.

[0077] Benchmark + key point screening module: Based on the benchmark method, a key point screening module (checkmodel) is added.

[0078] Benchmark + Keypoint Filtering Module + Person Re-ID: Based on the benchmark method, keypoints are added

[0079] Screening module (check model) and pedestrian re-identification (re-id).

[0080] It can be found from the table that each module in the network of the present invention has a certain contribution to the improvement of performance, especially the key point screening module has the greatest contribution to the performance improvement.

[0081] Table 1 Ablation experiment of the method of the present invention on the PoseTrack 2018 dataset

[0082]

[0083] This embodiment also verifies the effectiveness of the key point screening module in different top-down posture estimation networks. As shown in Table 2, HRNet, Hourglass and pose_resnet are tested for the multi-target tracking accuracy (MOTA) index with and without the key point screening module. The experimental results show that the key point screening module can effectively improve the performance of the posture tracking network, which also shows that the method of the present invention has good generalization ability.

[0084] Table 2 Ablation experiments of key point screening module in different pose estimation networks

[0085]

[0086] As shown in Table 3, in the framework of the top-down posture estimation network, the threshold setting of the human key points greatly affects the tracking accuracy MOTA index. However, when the key point screening module is used, the impact of the human key point threshold on the tracking accuracy MOTA index will be significantly reduced, alleviating the problem of difficulty in adjusting the human key point threshold in multi-person posture tracking tasks.

[0087] Table 3 Performance of key point screening module under different thresholds

[0088]

[0089] Table 4 reports the comparison of the results of the method of the present invention with other methods, including: LightTrack, Miracle+, OpenSV AI, and STAF. It can be seen from the table that the method of the present invention shows excellent results on the PoseTrack 2018 dataset, among which, Figure 5 Visualize the pose estimation results on this dataset. Figure 6 Visualize the results for pose tracking.

[0090] Table 4 Comparison of the method of the present invention with other methods

[0091]

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A posture tracking method based on key point screening, characterized in that: The following steps are involved: S1: Construct dataset; S1-1: Use the COCO 2017 dataset for detection and pose estimation, and the PoseTrack 2018 dataset for pose estimation and tracking; S1-2: Based on the PoseTrack 2018 dataset, a person re-identification dataset is constructed. All human frames in the PoseTrack2018 dataset are cropped and saved as images. The same labels are given to human frames with ids in the same video. Finally, a person re-identification dataset containing 4613 labels and a total of 119656 images is generated. S2: Construct a multi-person pose tracking model based on a top-down framework and a general keypoint screening network; S3: Model training; S4: Input the video to be tested into the model to obtain the detection result; The multi-person posture tracking model includes a target detection module, a posture estimation and key point screening module, a matching module and a pedestrian re-identification module, wherein: The target detection module is based on a multi-task multi-stage hybrid cascade structure model, namely the HTC model. The backbone network of the HTC model uses a 101-layer ResNeXt network, and adopts a feature graph pyramid network, namely the FPN network, and a non-maximum suppression algorithm based on the PCK indicator; The posture estimation and key point screening module adopts a universal posture estimation network, and obtains the heat map and existence probability of each key point through the key point prediction module and the key point screening module, and finally obtains the human body posture; The matching module associates the human posture of the current frame with the existing human posture trajectory, and uses the Hungarian algorithm based on the intersection-over-union distance of the bounding box to associate; for the human body whose trajectory is lost, it is added to the pedestrian re-identification module; when a new human body appears, the pedestrian re-identification module is used to retrieve the lost human body trajectory; The non-maximum suppression algorithm based on the PCK index uses the PCK distance as a similarity measure between two human frames, and when it is confirmed that a human frame is retained, other human frames whose PCK distance is greater than a set threshold are deleted; The key point prediction module and the key point screening module adopt the same network structure, which is composed of three layers of transposed convolution with a convolution kernel of 3x3. The key point prediction module finally generates a heat map for each key point, and the key point screening module obtains the binary classification probability of each key point through a fully connected layer for the feature map of each key point generated.

2. The posture tracking method based on key point screening according to claim 1, characterized in that: The pedestrian re-identification module uses the posture estimation network HRNet as the backbone network to extract features to obtain the feature map of each person, and associates the existing posture trajectory with the posture of the current frame each time, and saves the lost trajectory; for postures that fail to match successfully, the pedestrian re-identification module is used to compare with the lost pedestrian trajectory. If the comparison is successful, the new posture is merged with the lost posture trajectory. If the comparison fails, a new trajectory is generated; In the training phase, each person with the same label is set as a category. After obtaining the feature map of each person, it is connected to the classification network for classification, and multi-classification cross entropy is used as its loss function; During the posture tracking process, the feature map of each person is extracted and matched with the lost pedestrians. The L1 distance, i.e., Hamiltonian distance, is used to compare the distance between the two people. When the distance is less than the threshold, they are considered to be the same person.

3. The posture tracking method based on key point screening according to claim 1, characterized in that: The heat map output by the key point prediction module will also be smoothed by a second-order Gaussian filter.

4. The posture tracking method based on key point screening according to claim 1, characterized in that: The step S3 is to train the models separately, specifically: S3-1: The object detection module uses the HTC detector, and the mAP reaches 0.346 on the PoseTrack 2018 validation set. The object detection module is only trained on the COCO 2017 dataset and is not fine-tuned on the PoseTrack 2018 dataset. S3-2: The pose estimation and key point screening module is trained on COCO2017, and the trained model is fine-tuned on the PoseTrack 2018 dataset. Since only key points are annotated, the bounding box of all its key points is extended by 20% to obtain the basic truth box of a person instance. By extending the bounding box in height or width, the human body bounding box is made into a fixed aspect ratio; In the training of the pose estimation and key point screening modules, pose data is enhanced, including random scaling, random rotation, and random flipping, using Adam as the optimizer; S3-3: The pedestrian re-identification module crops pedestrians with the same label from the original image of the PoseTrack 2018 dataset to form a new dataset. The HRNet network pre-trained with the PoseTrack2018 dataset is used as the backbone network. During training, the feature maps output by the HRNet network are classified through the fully connected layer. During verification, only the feature maps output by the HRNet network are used, and no longer through the fully connected layer.

Images