A multi-frame trajectory tracking system based on pedestrian posture estimation and a method thereof

Abstract

The present application relates to a kind of multi-frame trajectory tracking system and method based on pedestrian pose estimation, which is based on Tracking by Object Detection tracking framework implementation, on the basis of single frame object detection, increase pose detection, simultaneously introduce the information of pose in tracking, to track by target and its pose detection paradigm;The method comprises: after single frame image is input into detector after feature extraction, the coordinate of the output detection confidence and detection frame is output;Respectively, pose prediction is carried out to pedestrian in different detection frame;According to the data of corresponding single frame model output in the first frame image of video, initialize tracker;On the basis of matching based on detection frame and matching based on pose information, tracking optimization is carried out based on the reference point of detector, and target association is established between two frame images using the relevant judgment of pose and the reference point of detector.Compared with prior art, the present application can optimize overall tracking effect, improve detection and association performance in the scene with occlusion and motion, effectively improve tracking effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a multi-frame trajectory tracking system based on pedestrian pose estimation and a method thereof. BACKGROUND

[0002] As one of the main trends in the development of the automobile industry in recent years, the corresponding detection and control technology of automatic driving has become a current research hotspot.

[0003] In addition to vehicles in common traffic environment, a considerable number of pedestrians are indispensable, and pedestrian detection naturally becomes an inevitable link in automatic driving technology. How to use the vehicle-mounted visual perception system to detect and track the human body posture in the automatic driving environment has become an important proposition in this direction. Traditional algorithms use handcrafted features and complex human body models to obtain local representation and global pose structure. Considering the complexity of the human body, more models start to use deep learning methods to extract relevant features.

[0004] In recent years, deep learning algorithms have developed rapidly, and a large number of efficient models and perfect data sets have been published, which makes it possible to use human pose estimation for multi-target understanding and tracking. However, most existing methods directly use the results of the detector for tracking trajectory analysis. Although many powerful detector models have excellent performance, simply using the results of the detector for tracking will reduce the overall association performance in scenes with occlusion and motion, resulting in poor actual tracking effect. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a multi-frame trajectory tracking method based on pedestrian pose estimation, which integrates human pose information based on existing detectors to effectively improve the tracking effect.

[0006] The purpose of the present application can be achieved by the following technical scheme: a multi-frame trajectory tracking system based on pedestrian pose estimation, realized based on the Tracking by Object Detection tracking framework, increasing pose detection on the basis of single-frame object detection, and introducing pose information into tracking to track through the target and its posture detection paradigm, including a single-frame model and a tracker connected in turn, the single-frame model being connected with a vehicle-mounted camera to obtain single-frame images from video data collected by the vehicle-mounted camera, the single-frame model being provided with a detector and a multi-person pose estimation module, the detector being used to output detection confidence and detection boxes corresponding to all pedestrian targets in the single-frame image;

[0007] The multi-person pose estimation module is configured to perform 2D human pose estimation on the pedestrian target, and output corresponding 2D pose key point coordinates.

[0008] The tracker is configured to track and match the pedestrian target in the current frame image and the previous frame image according to the output data of the single-frame model, and update the parameters of the tracker synchronously.

[0009] Further, the detector specifically adopts a Transformer-based Deformable DETR framework.

[0010] A multi-frame trajectory tracking method based on pedestrian pose estimation, comprising the following steps:

[0011] S1, extracting a single frame image from video data collected by a vehicle-mounted camera and inputting the single frame model;

[0012] S2, the single frame model processes the input single frame image, and outputs the detection confidence, detection box and 2D pose key point coordinates of all pedestrian targets in the single frame image;

[0013] S3, initializing the parameters of the tracker according to the output data of the single frame model corresponding to the first frame image in the video data, and then updating the output data of the single frame model to the tracker;

[0014] S4, the tracker tracks and matches the pedestrian target in the current frame image and the previous frame image, outputs the tracking result, and updates the parameters of the tracker synchronously.

[0015] Further, the step S2 specifically comprises the following steps:

[0016] S21, the single frame image is input into the detector of the single frame model, and the detection confidence and detection box corresponding to all pedestrian targets in the single frame image are output;

[0017] S22, according to the data output by the detector, the multi-person pose estimation module of the single frame model performs 2D pose estimation on each pedestrian target in the single frame image, and outputs the corresponding 2D pose key point coordinates.

[0018] Further, the step S3 of initializing the parameters of the tracker specifically initializes the following parameters of the tracker:

[0019] The detection confidence, the detection box, the 2D pose key point and the track ID, wherein the detection confidence, the detection box and the 2D pose key point correspond to the output data of the single frame model corresponding to the first frame image in the video data, and the track ID is a non-repetitive mark with a value starting from 0.

[0020] Further, the specific process of the tracker tracking and matching the pedestrian target in the current frame image and the previous frame image in the step S4 is:

[0021] S41, according to the single frame model data corresponding to the current frame image and the single frame model data corresponding to the previous frame image, respectively calculating the detection box matching degree, the key point similarity and the matching score based on the reference point, and accumulating the three calculation results to obtain the final matching score matrix;

[0022] S42, according to the final matching score matrix, determining the trajectory similarity between the current frame image and the previous frame image, if the trajectory similarity exceeds the corresponding preset threshold, judging that the target in the previous frame image finds a matched object in the current frame image, that is, the matching is successful;

[0023] Otherwise, it is judged that the target in the previous frame image does not have a matched object in the current frame image, that is, the matching fails.

[0024] Further, the detection box matching degree is specifically:

[0025]

[0026] Wherein, A, B respectively correspond to the area occupied by two detection boxes, and C is the area of the minimum rectangle circumscribed by AB.

[0027] Further, the key point similarity is specifically:

[0028]

[0029]

[0030]

[0031] Wherein, d is the Euclidean distance between the corresponding key points, S is the size of the object, and x and y are the coordinate values of the key points, The diagonal line of the object true value box.

[0032] Further, the calculation process of the matching score based on the reference point includes:

[0033] 1) According to the feature and the detection box, the reference point is initially matched;

[0034] 2) According to the order of the reference point initial matching result, rearrange the embedded features to obtain the reference point order of the current frame;

[0035] 3) Using a set of multilayer perceptron to obtain the offset of the corresponding reference point, obtaining the reference point coordinates required by the tracking branch;

[0036] 4) The reference point sequence of the previous frame is rearranged according to the current frame, and is sent to the decoder of the current frame, so as to obtain the matching score based on the reference point.

[0037] Further, the specific process of updating the tracker itself parameters in the step S4 is as follows:

[0038] If the matching is successful, the single-frame model data corresponding to the current frame is used to update the confidence, the detection frame, and the 2D key point parameters currently saved in the tracker, while keeping the active state of the matching successful target;

[0039] If the matching fails, the state corresponding to the target is changed to suspended, and the suspended counter is increased by one. When the matching object of the target is found in the subsequent frame image, the suspended counter is cleared. If the suspended counter exceeds a preset threshold, the tracking of the target is closed.

[0040] Compared with the prior art, the present application has the following advantages:

[0041] Firstly, the present application is based on the Tracking by Detection tracking framework, and adds pose detection on the basis of single-frame object detection. Meanwhile, the information of the pose is introduced into the tracking, thereby constructing a multi-frame tracking system, including a single-frame model and a tracker. The detector is arranged in the single-frame model, and is used to predict the detection frame of the pedestrian. The multi-person 2D pose prediction module is used to perform pose prediction on a single person respectively, and outputs the 2D key point coordinates of the human body. The single-frame model of the first frame image of the video is used to initialize the tracker. The tracker is used to establish the connection between two frames of images, and the detection frame and the key point coordinate information of the pedestrian are predicted simultaneously to update the tracking trajectory, so as to optimize the tracking result and effectively improve the tracking effect.

[0042] Secondly, in the present application, the detector in the single-frame model adopts the Deformable DETR framework based on the Transformer, which is used to output the detection confidence and the detection frame of the single person. The detection confidence and the detection frame of the single person can fully utilize the feature extraction capability of the Deformable DETR, so as to mine the prior structural information of the human body based on the sampled points in the Deformable DETR, realize a tracking optimization structure based on the reference points, and improve the overall detection and association performance of the tracker in the scene with occlusion and motion.

[0043] Thirdly, in the present application, the tracker outputs data according to the single-frame model, on one hand, the tracker matches the target in the current frame and the last frame by calculating the matching degree of the detection frame, the similarity of the key points, and the matching score based on the reference points, on the other hand, the tracker synchronously updates the parameters based on the matching result, thereby ensuring the real-time optimization of the tracker parameters and fully guaranteeing the accuracy of the trajectory tracking.

[0044] Fourthly, in the present application, the tracker further optimizes the tracking based on the reference points of the detector based on the matching of the detection frame and the matching of the posture information, and establishes the target association between two frames by calculating the matching score based on the reference points, thereby using the related judgment of the posture and the reference points of the detector. Compared with the method of directly using the results of the detector for trajectory tracking, the overall tracking effect can be greatly optimized, and the detection and association performance in the scene with occlusion and motion can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The figure is a schematic diagram of the method of the present application;

[0046] Figure 2 The figure is a schematic diagram of the process of calculating the matching score based on the reference points of the tracker. DETAILED DESCRIPTION

[0047] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0048] Embodiment

[0049] A multi-frame trajectory tracking system based on pedestrian posture estimation is based on a Tracking by Object Detection tracking framework, increases pose detection on the basis of single-frame object detection, and introduces the information of the pose into tracking, that is, constructs a Tracking by Object and Pose Detection tracking framework, and tracks through the target and the posture detection paradigm, specifically includes a single-frame model and a tracker connected in sequence, wherein the single-frame model is connected with the vehicle-mounted camera to acquire single-frame images from the video data collected by the vehicle-mounted camera.

[0050] The single-frame model is provided with a detector and a multi-person posture estimation module, the detector is used to output the detection confidence and the detection frame corresponding to all pedestrian targets in the single-frame image, and the multi-person posture estimation module is used to perform 2D human body posture estimation on the pedestrian target and output the corresponding 2D posture key point coordinates.

[0051] The tracker is used to track and match the pedestrian targets in the current frame image and the previous frame image according to the output data of the single-frame model, and to update the parameters of the tracker synchronously.

[0052] In this embodiment, the detector specifically adopts a Deformable DETR framework based on a Transformer, which is used to output the detection confidence and the detection frame of a single person. The Deformable DETR itself can provide a human detection frame and corresponding feature extraction for the tracking task, but does not consider the prior structural information of the human body. Therefore, the technical solution adds a multi-person pose estimation module for processing the human pose information. The multi-person pose estimation module adopts a top-down scheme, performs 2D human pose estimation on a single person according to the human detection frame predicted by the detector, and outputs 2D key point coordinates.

[0053] Since the related annotations of the human pose need to be used, the tracker based on pose estimation is trained in advance on a multi-frame image dataset PoseTrack related to human skeleton key points, so as to integrate the detection frame and the joint point for multi-frame tracking.

[0054] The above system is applied to realize a multi-frame trajectory tracking method based on pedestrian pose estimation, as shown in Figure 1 The method comprises the following steps:

[0055] S1, extracting a single frame image from video data collected by a vehicle-mounted camera and inputting the single frame model;

[0056] S2, the single frame model processes the input single frame image, and outputs the detection confidence, the detection frame and the 2D pose key point coordinates of all pedestrian targets in the single frame image;

[0057] S3, initializing the parameters of the tracker according to the output data of the single frame model corresponding to the first frame image in the video data, and then updating the output data of the single frame model to the tracker. The parameters of the initialized tracker include the detection confidence, the detection frame, the 2D pose key point and the track ID. The track ID is a non-repetitive mark valued from 0, and the detection confidence, the detection frame and the 2D pose key point correspond to the output parameters of the single frame model corresponding to the first frame image;

[0058] S4, the tracker tracks and matches the pedestrian targets in the current frame image and the previous frame image (specifically, the tracking and matching are based on the matching degree of the detection frame, the key point similarity and the matching score of the reference point-based tracking optimization module), and updates the parameters of the tracker synchronously;

[0059] The matching degree score based on the detection frame is calculated as follows:

[0060]

[0061] In the formula, A and B correspond to the area occupied by the two detection boxes, and C is the area of the minimum rectangle circumscribed by AB;

[0062] The key point-based similarity score calculation is as follows:

[0063]

[0064]

[0065]

[0066] In the formula, d represents the Euclidean distance between the corresponding joints, S represents the size of the object, and x and y correspond to the coordinate values of the joints. The coordinates of the two vertices on the diagonal of the object ground truth box are

[0067] As shown in Figure 2 The multi-frame tracking optimization module based on reference points has the following process:

[0068] 1) According to the features and detection boxes, the reference points are initially matched;

[0069] 2) The embedding features are rearranged according to the order of the initial matching results of the reference points to obtain the reference point order of the current frame;

[0070] 3) A set of MLPs are calculated to obtain the offset of the corresponding reference points, and the reference point coordinates required by the tracking branch are obtained;

[0071] 4) The reference point order of the previous frame is rearranged according to the current frame and sent to the decoder of the current frame, and the matching result based on the reference points can be obtained;

[0072] The initial matching result of the reference points is as follows:

[0073] For the given i-th detection result of the T-th frame, represents the corresponding embedding feature, is the reference point output by the DETR, is the predicted detection box size;

[0074]

[0075]

[0076] By weighting the sum of the above two distance matrices, the reference point matching result between the two consecutive frames of images can be obtained using the Hungarian algorithm.

[0077] The three calculation results are accumulated to obtain a final matching score matrix, and according to the final matching score matrix, the trajectory similarity between the current frame image and the previous frame image is determined, if the trajectory similarity exceeds the corresponding preset threshold, it is judged that the target in the previous frame image finds a matched object in the current frame image, that is, the matching is successful;

[0078] Otherwise, it is judged that the target in the previous frame image does not have a matched object in the current frame image, that is, the matching fails.

[0079] In addition, when updating the parameters of the tracker itself, the updating is mainly based on the following principles:

[0080] 1) For the trajectories that are matched successfully in the current frame: the confidence, the detection box, the 2D key points and other data saved in the tracker are updated using the results of the current frame, and the active state is maintained.

[0081] 2) For the trajectories that have no matched object in the current frame: the state is changed to suspended, and the suspended counter is increased by one, if a matched object is found, the suspended counter can be cleared, and if the suspended counter exceeds the threshold, the trajectory is closed.

[0082] From the above, it can be seen that the technical scheme considers that multi-target tracking can usually be divided into two sub-tasks, namely detection and association, and the existing method directly uses the results of the detector to analyze the tracking trajectory, which will reduce the overall detection and association performance in scenes with occlusion and motion. Deformable DETR itself can provide human detection boxes and corresponding feature extraction for the tracking task, but it does not consider the prior structural information of the human body, therefore, the multi-person pose estimation module is added to the improved Deformable DETR to focus on processing the human pose information, so that the improved Deformable DETR can detect pedestrians and pedestrian pose key points at the same time, realize the integration and integration of human pose information, and use the pose-related judgment and the reference point of DETR to establish a certain relationship between two frames, rather than isolated detection, thereby effectively optimizing the tracking result.

Claims

1.A method for multi-frame trajectory tracking based on pedestrian pose estimation, applied to a multi-frame trajectory tracking system based on pedestrian pose estimation, characterized in that, The multi-frame trajectory tracking system is implemented based on a Tracking by Object Detection tracking framework, and pose detection is added on the basis of single-frame object detection, and the information of the pose is introduced into tracking, so as to track through the target and its posture detection paradigm, including a single-frame model and a tracker connected in sequence, the single-frame model is connected with the vehicle-mounted camera to obtain a single-frame image from the video data collected by the vehicle-mounted camera, a detector and a multi-person pose estimation module are arranged in the single-frame model, the detector is used to output the detection confidence and the detection box corresponding to all pedestrian targets in the single-frame image; The multi-person pose estimation module is used for 2D human body pose estimation of the pedestrian target, and the corresponding 2D pose key point coordinates are output; The tracker is used to track and match the pedestrian targets in the current frame image and the last frame image according to the output data of the single-frame model, and synchronously update the parameters of the tracker itself; The multi-frame trajectory tracking method comprises the following steps: S1, a single-frame image is extracted from the video data collected by the vehicle-mounted camera and input into the single-frame model; S2, the single-frame model processes the input single-frame image, and outputs the detection confidence, the detection box and the 2D pose key point coordinates of all pedestrian targets in the single-frame image; S3, the parameters of the tracker are initialized according to the single-frame model output data corresponding to the first frame image in the video data, and then the single-frame model updates the output data to the tracker; S4, the tracker tracks and matches the pedestrian targets in the current frame image and the last frame image, outputs the tracking result, and synchronously updates the parameters of the tracker itself; The specific process of tracking and matching the pedestrian targets in the current frame image and the last frame image by the tracker in step S4 is as follows: S41, the detection box matching degree, the key point similarity and the matching score based on the reference point are calculated respectively according to the single-frame model data corresponding to the current frame image and the single-frame model data corresponding to the last frame image, and the three calculation results are accumulated to obtain a final matching score matrix; S42, the trajectory similarity between the current frame image and the last frame image is determined according to the final matching score matrix, if the trajectory similarity exceeds the corresponding preset threshold, it is judged that the target in the last frame image finds a matched object in the current frame image, that is, the matching is successful; Otherwise, it is judged that the target in the last frame image does not have a matched object in the current frame image, that is, the matching fails; The detection box matching degree is specifically: ， wherein, respectively correspond to areas occupied by the two bounding boxes, is an area of a minimum enclosing rectangle of 2.The multi-frame trajectory tracking method based on pedestrian pose estimation of claim 1, wherein, The detector specifically adopts a Deformable DETR framework based on Transformer. 3.The multi-frame trajectory tracking method based on pedestrian pose estimation of claim 1, wherein, Step S2 specifically comprises the following steps: S21, the single-frame image is input into the detector of the single-frame model, and the detection confidence and the detection box corresponding to all pedestrian targets in the single-frame image are output; S22, according to the data output by the detector, the multi-person pose estimation module of the single-frame model performs 2D pose estimation on each pedestrian target in the single-frame image, and outputs the corresponding 2D pose key point coordinates. 4.The multi-frame trajectory tracking method based on pedestrian pose estimation of claim 1, wherein, The parameter initialization of the tracker in the step S3 is specifically initialization of the following parameters of the tracker: detection confidence, detection box, 2D pose key points and track ID, wherein the detection confidence, the detection box and the 2D pose key points correspond to single-frame model output data of a first frame image in the video data respectively, and the track ID is a non-repetitive mark with a value starting from 0. 5.The multi-frame trajectory tracking method based on pedestrian pose estimation of claim 1, wherein, The key point similarity is specifically: ， ， ， wherein, is the Euclidean distance between the corresponding key points, is the size of the object, is the coordinate value of the key point, is the coordinates of two vertices on the diagonal of the object ground truth box. 6.The multi-frame trajectory tracking method based on pedestrian pose estimation of claim 1, wherein, The calculation process of the matching score based on the reference points includes: 1) performing primary matching on the reference points according to the features and the detection box; 2) rearranging the embedded features according to the order of the primary matching result of the reference points to obtain the reference point order of the current frame; 3) using a set of multilayer perceptrons to obtain the offset of the corresponding reference points to obtain the reference point coordinates required by the tracking branch; 4) inputting the reference point order of the previous frame into the decoder of the current frame after rearranging the reference point order of the previous frame according to the current frame to obtain the matching score based on the reference points. 7.The multi-frame trajectory tracking method based on pedestrian pose estimation of claim 1, wherein, The specific process of updating the parameters of the tracker itself in the step S4 is: if the matching is successful, using the single-frame model data of the current frame to update the confidence, the detection box and the 2D key points currently saved in the tracker, and keeping the active state of the matching successful target; if the matching fails, converting the state of the target into suspended and adding one to the suspended counter, and clearing the suspended counter when the target matching object is found in the subsequent frame image; if the suspended counter exceeds a preset threshold, closing the track of the target.

Images