A multi-target tracking method for modeling inter-target relationships

Abstract

The application discloses a multi-target tracking method for target interrelation modeling. The method comprises the following steps: constructing an intra-frame graph by using the position, appearance feature and topological relation among targets of the target output by a current frame detector; then updating the vertex and edge features of the intra-frame graph by using a message passing network to further fuse the features among the targets; then constructing an inter-frame graph by combining the trajectory graph of a past frame and the intra-frame graph, wherein the edges of the inter-frame graph represent the similarity between the trajectories and the detected features; performing a message passing process on the inter-frame graph to further fuse; then calculating the score of the matching relation represented by the edges by using a fully connected layer network, and recovering the low confidence detection prone to missing and the trajectories lost due to occlusion and the like. The method can realize stable association in a scene with nonlinear camera motion by modeling the topological relation among the targets, and can assist in recovering the detection of the occluded target by using the information of the neighboring targets. To some extent, the method overcomes the defects of the mainstream multi-target tracking algorithm which separately processes the targets, and can realize stable tracking in the scene with nonlinear motion and occlusion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and relates to a multi-target tracking method for modeling inter-target relationship. BACKGROUND

[0002] Multi-target tracking is an important computer vision task, and has important value in security, autonomous driving, crowd density monitoring, behavior analysis, and video content understanding tasks. It analyzes videos, assigns different identities to different targets, and reasons out the specific positions of each target in each frame, thereby forming the trajectory corresponding to each target.

[0003] In recent years, the multi-target tracking field has developed rapidly and good results have been achieved on MOT, KITTI, DanceTrack and other datasets. However, due to the uncertainty of camera motion, situations such as target nonlinear motion, inter-target occlusion, and target and background occlusion are prone to occur, which will cause the target to be assigned with a wrong identity, and thus lead to unstable tracking.

[0004] In response to target occlusion, the following solutions are mainly used: the first type is to optimize the matching strategy, that is, in the association stage of detection and trajectory, a double-threshold matching method is used, and the representation of the association matrix is optimized. The former can restore the low-confidence target caused by occlusion, and the latter can better combine appearance features and motion features, so that new features are more biased towards high-confidence motion features or appearance features. The second type is to establish an additional network branch to directly predict the occlusion of the target, thereby making up for the missed detection. The third type is to establish a memory bank of target features, and through attention calculation and other forms, the features of the target in the past several frames are optimized into more robust features to cope with the situation of occlusion.

[0005] In response to target nonlinear motion, the following solutions are mainly used: the first type is to use image registration before the association stage, and update the motion coordinates of the trajectory through the obtained affine matrix, thereby to a certain extent, offsetting the sudden change of motion coordinates caused by camera motion. The second type is to fit the nonlinear motion in the form of trajectory smoothing. The third type is to abandon the Kalman filter used in the classical algorithm when it is determined that the target has a large degree of nonlinear motion, and instead use motion features with rotation invariance for matching.

[0006] However, the above algorithms are all single target processing, that is, the relationship between targets is not considered, which leads to the difficulty of maintaining stable tracking in complex scenes. Specifically, since the same type of target has a similar appearance, the single target feature processing has certain limitations; for the target crowded scene, frequent mutual occlusion between targets is prone to occur, and the single target processing method is prone to confuse the target when occluded; for the situation of target nonlinear motion caused by camera motion, the single processing trajectory or the calculation of affine matrix has a large amount of calculation, and the relative position between targets during the camera motion process can be considered to remain basically unchanged, so modeling the relationship between targets can also deal with this situation well.

[0007] Therefore, there is an urgent need for a multi-target tracking method for modeling the relationship between targets to deal with occlusion and nonlinear motion and other problems by processing the relationship between targets, so as to achieve stable tracking in complex scenes. SUMMARY

[0008] Therefore, the present application provides a multi-target tracking method for modeling the relationship between targets, which can utilize the appearance and topological relationship between targets to achieve stable tracking in occlusion and nonlinear motion scenes.

[0009] To achieve the above-mentioned application purposes, the technical scheme of the present application is as follows:

[0010] A multi-target tracking method for modeling the relationship between targets, the specific steps comprising:

[0011] S1, constructing a high-performance detector and an appearance feature extraction network. The input of the detector is a video frame, and the output is the position and category of each target. The input of the appearance feature extraction network is the cropped target region, and the output is the appearance feature vector of the target.

[0012] S2, for the current frame, to avoid the missing detection phenomenon caused by occlusion, a lower confidence threshold is used to screen out the detection candidate box. Then, non-maximum suppression is used to remove the highly repeated detection box. The appearance feature extraction network is used to extract the appearance feature of each target.

[0013] S3, constructing an intra-frame graph for the target detected in the current frame, the vertex of the graph represents the encoding of the appearance feature and the motion feature of the target, and the edge represents the topological relationship between the two targets connected by the edge. Specifically, the vertex feature encoding consists of two parts: one is the position information and the appearance feature of the target, and the other is the angle and normalized distance of the target and its several nearest neighbor targets. The edge feature encoding consists of three parts: the fusion vertex feature between the two targets connected by the edge, the geometric distance and the bounding box similarity. Then, the message passing step is performed on the constructed intra-frame graph to fuse the features of the edge and the vertex. If the current frame is not the first frame, S4 is performed, otherwise S5 is performed.

[0014] S4, construct a matching graph for the current frame and the track graph. Since the targets in the frame cannot match each other, the matching graph is a bipartite graph. One end of the bipartite graph is the vertex of the active track and the temporary lost track features, and the other end of the bipartite graph is the vertex of the target features in the current frame; the edge is the similarity measurement of the connected track and the current target. Similar to the intra-frame graph, the edge feature coding of the matching graph is composed of the feature vector similarity of the track and the current target, the geometric distance and the bounding box similarity. The vertex feature coding of the matching graph is composed of the feature vector of the track and the target in the vertex. Figure One to reduce the complexity. Then, a message passing step is performed on the constructed matching graph, and the score of the matching relationship represented by each edge is calculated by inputting the edge to the edge classifier (composed of a fully connected layer network). The score matrix is composed of the scores between the tracks and the targets, and the optimal solution of the cost matrix is obtained by calculating the linear assignment problem, that is, the matching relationship.

[0015] S5, arrange the current matching relationship. Specifically, the following cases can be divided:

[0016] The active track matched successfully and the high-confidence target: the matching relationship of this case is often reliable, the target is classified into the corresponding track, and the target current feature and the track feature are exponentially smoothed as the new track feature.

[0017] The active track matched successfully and the low-confidence target: this case indicates that the target may be occluded, the target is recovered and classified into the corresponding track, and since the current target feature is not reliable, the track feature is not updated.

[0018] The non-active track matched successfully and the high-confidence target: this case indicates that the target disappears for a period of time and then reappears, the target is classified into the corresponding track, and the current target feature is used as the new track feature.

[0019] The track that is not matched successfully: this case indicates that the target exits the field of view or is severely occluded. If the state of the track is an active track, it is marked as inactive and the inactive duration is recorded; if the state of the track is a non-active track, it is determined that if the inactive state duration is greater than the maximum retention time, it is deleted from the track set.

[0020] The target that is not matched successfully: if the confidence of the target is greater than a certain value, it indicates that the target is likely to be a new target, and it is initialized as a new track.

[0021] After the above steps, the current track graph is updated, and S2 is returned.

[0022] After processing all the frames in the video to be processed, the process is ended.

[0023] Further, the number of vertices and edges of the intra-frame graph is related to the specific detection result. Since the target density is indefinite, geometric distance is used to find the nearest neighbor target of the target, and the range is 0.1 times the smaller of the image height and width with the target as the center and the radius. The target appearance feature is a 64-dimensional vector, and the angle with the nearest neighbor target and the normalized distance are each preset as a 20-dimensional vector. If there are less than 20 nearest neighbor targets, fill with 0; if there are more than 20, take the nearest 20.

[0024] Further, in the edge feature encoding of the intra-frame graph, the vertex features are fused by attention calculation, the geometric distance uses Euclidean distance, and the bounding box similarity includes center point difference, width-height ratio logarithm value, and Wasserstein distance.

[0025] Further, the number of iterations of message passing of the intra-frame graph and the matching graph is 3, and the message passing mode is average aggregation. The edge classifier of the matching graph has 2 full connection layer, and the activation function uses Sigmoid to normalize the output.

[0026] Further, the high confidence threshold is 0.5, the low confidence threshold is 0.2, the target initialization threshold is 0.6. After calculating the edge score of the matching graph, first filter the edges with a score greater than or equal to 0.4, and when constructing the cost matrix, set the cost of edges with a score less than 0.4 to a large positive number (e.g. 1E5). The maximum retention time of the lost track is 30 frames.

[0027] Beneficial effects:

[0028] 1. The application provides a multi-target tracking method based on inter-target relationship modeling, which achieves good results in different tracking scenes. The application uses a graph structure to model the inter-target relationship by calculating the appearance and topological information of the neighboring targets, and constructs a matching graph in the matching stage, representing the matching relationship between the trajectory and the target with an edge, and calculating the matching score by extracting the semantic features of the edge. When considering the inter-target relationship, the application combines the appearance features and topological information (angle and normalized distance) of the target and its neighboring targets. This feature coding method associates the target with its surrounding environment, enhancing the semantic information richness of the target and benefiting in non-linear motion and occlusion scenes. Specifically, considering the situation where the absolute position of the target in the picture changes abruptly due to non-linear camera motion, current mainstream algorithms only use discrete absolute position information for matching. Since the absolute position of the target changes too much between adjacent frames, current algorithms can cause the target identity to change abruptly. However, the relative position of the target remains consistent during non-linear motion, so the feature coding of the application can remain stable in most non-linear motion scenes, thereby achieving stable tracking. In addition, considering the target occlusion problem, the current mainstream algorithm handles targets separately, which can cause missed detection when the target confidence is low. Simply reducing the detector confidence can cause false detection. Therefore, the inter-target relationship modeling method of the application can restore low-confidence detection by the consistency of the target and its surrounding environment, and can also suppress incorrect and fragmented false detections.

[0029] 2. In addition, in order to realize long-time tracking, the application proposes a robust trajectory management mechanism, which fully considers the problem of lost trajectory recovery. The mainstream algorithm still uses absolute position information or appearance information for trajectory recovery. When the target appears again, the position often changes greatly, and relying solely on the appearance information of a single target is unstable, but the target and its surrounding neighboring targets often still maintain the features before the trajectory is lost. Therefore, the inter-target relationship modeling method proposed by the application can fully consider the semantic information of the target and its surrounding environment, thereby accurately recovering the trajectory. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The application is an implementation block diagram.

[0031] Figure 2 The application is an intra-frame graph construction process diagram.

[0032] Figure 3 The application is a matching graph association process diagram. DETAILED DESCRIPTION

[0033] The application will be described in detail below with reference to the accompanying drawings and examples.

[0034] ATTACHMENT Figure 1The flow of the whole tracking algorithm is as follows: firstly, the current video frame image is read, the detection result of the frame is obtained by using a high-performance detector network, including the category and position information of each target. Then, the intra-frame graph is constructed, and the topological and appearance relationship between targets is used to form the vertex and edge feature encoding of the intra-frame graph. Then, the intra-frame graph is iterated through the message passing network, and the vertex and feature relationship are fused to increase the receptive field of the feature. Then, the current track graph is constructed to form a matching graph, and the edge of the matching graph represents the appearance and topological relationship between the connected targets and tracks. Similarly, the matching graph is input into the message passing network for iteration, and a cost matrix is constructed according to the classification score of the edge. The matching is completed by solving the cost matrix, and finally the track is updated and managed.

[0035] The specific steps will be described in detail below.

[0036] Step one, constructing a high-performance detector and appearance feature extraction network.

[0037] Step two, the detection result obtained in the video frame image detection network is defined, high and low confidence thresholds are defined, and the detection result is screened according to the thresholds. The target greater than the high confidence threshold is marked as a high confidence target, and the target greater than the low confidence threshold and less than or equal to the high confidence threshold is marked as a low confidence target. The low confidence target is likely to have a situation such as occlusion. Then, non-maximum suppression is used to remove the detection box. Then, the target region is input into the appearance feature extraction network to obtain the appearance feature of each target and save it.

[0038] In the example of the present application, the detector network can be selected from YOLO series or CenterNet, the appearance feature extraction network is OSNet, the high confidence threshold is selected as 0.5, the low confidence threshold is selected as 0.2, and the dimension of each target appearance feature vector is 64.

[0039] Step three, after obtaining the position and appearance feature of the target of the current frame, the intra-frame graph representing the relationship between the targets is constructed. Figure 2 The feature encoding mode of the vertex and edge of the intra-frame graph is embodied. Each vertex of the intra-frame graph represents the feature of the target of the current frame, and the edge represents the similarity feature of the connected two targets.

[0040] For vertex feature encoding, first, a distance is selected, which is 0.1 times the smaller of the height and width of the image. Only consider the target within the circle with the current vertex as the center and the distance as the radius. These targets are called "near neighbor targets", and the rest are called "non-near neighbor targets". The distance between the target and all near neighbor targets is calculated and normalized. The specific way of normalization is to divide each distance by the maximum distance. Then the angle between the near neighbor targets of the target is calculated. Therefore, the distance and the angle describe a topological relationship between the target and the near neighbor targets. In order to maintain the time arrangement, the distance and the angle features are sorted when stored. The angle with the near neighbor target and the normalized distance are each preset to a 20-dimensional vector. If there are less than 20 near neighbor targets, fill in 0; if there are more than 20, take the nearest 20. Then the appearance feature of the target, the position information of the target (center point and boundary box height and width), and the topological relationship obtained with the near neighbor targets are combined to form the feature encoding of the vertex. Therefore, the vertex feature dimension is 108.

[0041] For edge feature encoding, first, the appearance features of the two connected targets are fused. The specific way of fusion is to perform attention calculation to obtain a vector of the same dimension. Then the similarity of the position information is calculated, and the fused vertex feature and the position information similarity are combined to form the edge feature encoding. The position information similarity is composed of the difference between the center points of the boundary boxes, the logarithmic ratio of the boundary box height and width, and the Wasserstein distance of the two targets. The use of Wasserstein distance is more conducive to describing the overlap degree of the boundary boxes of small targets compared with the mainstream algorithm using IoU distance. Therefore, the dimension of the edge feature encoding is 69.

[0042] Then the constructed intra-frame graph is updated through the message passing network. The message passing network can fuse the features between vertices and edges. After multiple iterations, each vertex can be fused into larger range features, which is more conducive to modeling target relationships. To balance speed and accuracy, the number of layers of the message passing network is 3, and the way of updating the vertex feature is average aggregation, and the edge does not update the feature. That is, the features of the vertices and the connected edges are mapped to new features through a multi-layer perceptron.

[0043] Step four, after obtaining the intra-frame graph, a matching graph is constructed with the trajectory graph to complete the tracking step. If the current frame is the first frame, there is no trajectory graph, so the current intra-frame graph is taken as the trajectory graph. Otherwise, as shown in the accompanying drawings Figure 3 The first to fourth steps show that a matching graph is constructed.

[0044] The matching graph is a bipartite graph composed of the trajectory graph and the intra-frame graph, one side of which is the vertex of the trajectory graph and the other side of which is the vertex of the intra-frame graph. The edge represents the feature metric of the connected trajectory and vertex matching, specifically, similar to the edge feature of the intra-frame graph, the fusion appearance feature of the trajectory and the target is calculated, and the similarity of the position information of the two is encoded as the edge feature. The edge feature vector dimension of the matching graph is also 69. Then the matching graph is passed through the message passing network, the number of layers is 3, and the edge and vertex features are updated in the form of average aggregation. A fully connected layer with 2 layers is used to calculate each edge to obtain a score between 0 and 1, representing the possibility score of the corresponding trajectory and the target being connected. According to the possibility score, a matching cost matrix is constructed, and the edges with a score less than or equal to 0.4 are set to 1E5 at the corresponding position of the cost matrix to reject matching, and the rest are set to 1 minus the possibility score at the corresponding position of the cost matrix. Then the Sinkhorn algorithm or the Hungarian algorithm is used to obtain the optimal solution of the matching matrix, that is, the corresponding relationship between the trajectory and the target.

[0045] Step five, as shown in the accompanying drawings Figure 3 As shown in the fifth step, trajectory management and trajectory graph updating are performed. The matching relationship solved in step four is checked one by one. If it is the matching of an active trajectory and a high-confidence target, such matching is often reliable, then the target feature and the trajectory feature are exponentially smoothed as the new trajectory feature; if it is the matching of a non-active trajectory and a high-confidence target, it means that the target reappears after disappearing for a period of time, the state of the trajectory is restored to active, and the original trajectory feature is discarded, and the current target feature is used as the trajectory feature; if it is the matching of an active trajectory and a low-confidence target, the target is restored, and the trajectory feature is not updated. For the remaining trajectories that are not matched successfully, set their state to non-active, and check if the duration of non-activity exceeds 30 frames, then delete the trajectory. For the remaining targets that are not matched successfully, if their confidence is greater than 0.6, it means that it is a newly appeared target, which is initialized as a new trajectory for future frame matching. After all the frames in the video to be processed are processed, the process ends.

[0046] From now on, the entire process design of the multi-target tracking algorithm for modeling the relationship between targets is completed.

[0047] To sum up, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-target tracking method of inter-target relationship modeling, characterized in that, The method comprises the following steps: Step one, constructing a detector and an appearance feature extraction network, the input of the detector being a video frame, the output being the position and category of each target, the input of the appearance feature extraction network being a target region cropped out, and the output being an appearance feature vector of the target; Step two, for the current frame, screening out detection candidate boxes with a low confidence threshold, then removing highly repeated detection boxes by non-maximum suppression, and extracting the appearance features of each target by using the appearance feature extraction network; Step three, constructing an intra-frame graph for the targets detected in the current frame, the vertex of the graph representing the encoding of the appearance feature and the motion feature of the target, the edge representing the topological relationship of the features of the two targets connected, and the intra-frame graph features being updated through a message passing network; Step four, constructing a matching graph for the current frame and the trajectory graph, one end of the vertex of the matching graph being the features of the active trajectory and the temporarily lost trajectory, the other end of the vertex being the features of the targets in the current frame, the edge being the similarity measurement of the connected trajectory and the current target, and the matching graph features being updated through a message passing network; The edge is input to an edge classifier composed of a fully connected layer network, the score of the matching relationship represented by the edge is calculated, the score between the trajectory and the target constitutes a cost matrix, and the optimal solution of the cost matrix is obtained by calculating a linear assignment problem, that is, the matching relationship; Step five, arranging the current matching relationship; The step one further comprises: the detector network selects a YOLO series or CenterNet, the appearance feature extraction network is an OSNet, the high confidence threshold is selected as 0.5, the low confidence threshold is selected as 0.2, and the dimension of the appearance feature vector of each target is 64; The specific method for constructing the intra-frame graph in the step three is that the number of the vertices and the edges of the intra-frame graph is related to the specific detection result, the geometric distance is used to find the near neighbor targets of the target, and the range is 0.1 times of the smaller one of the image height and width as the radius with the target as the center; the target and the near neighbor target angle and the normalized distance are each preset as a 20-dimensional vector, if the near neighbor target is less than 20, the 0 is filled; if it is more than 20, the nearest 20 are taken; in the edge feature coding of the intra-frame graph, the vertex feature is fused to obtain the attention calculation, the geometric distance adopts the Euclidean distance, and the bounding box similarity includes the center point difference, the logarithmic value of the width-height ratio and the Wasserstein distance; The specific method for constructing the matching graph in the step four is that the vertex features of the trajectory graph and the vertex features of the intra-frame graph are fused, and the position information similarity is added. The specific method of calculating the matching relationship in step four from the matching graph is: inputting the matching relationship represented by the edge of the matching graph into the edge classifier to calculate the confidence, the edge classifier has 2 full connection layer numbers, and the activation function adopts Sigmoid to normalize the output; according to the edge score, a matching cost matrix is constructed, the edges with scores less than or equal to 0.4 are set to 1E5 at the corresponding positions of the cost matrix to reject matching, and the rest are set to 1 minus the possibility score at the corresponding positions of the cost matrix, and then the optimal solution of the matching matrix, that is, the corresponding relationship between the trajectory and the target, is obtained by using the Sinkhorn algorithm or the Hungarian algorithm; The updating mode of the message passing network in steps three and four is that the iteration number of the message passing of the intra-frame graph and the matching graph is 3, and the message passing mode is the average aggregation mode, that is, the updated feature of each vertex is connected by the average feature of the neighbor vertices, the average feature of the in-edge and the out-edge and the residual structure of the last layer feature of itself; The specific method of trajectory arrangement in step five is: Matching successful activity trajectory and high confidence target: the target is classified into the corresponding trajectory, and the target current feature and the trajectory feature are exponentially smoothed as the new trajectory feature; Matching successful activity trajectory and low confidence target: the target is restored and classified into the corresponding trajectory, and since the current target feature is not reliable, the trajectory feature is not updated; Matching successful non-activity trajectory and high confidence target: the target is classified into the corresponding trajectory, and the current target feature is taken as the new trajectory feature; Matching unsuccessful trajectory: if the state of the trajectory is an activity trajectory, it is marked as non-activity, and the non-activity duration is recorded; if the state of the trajectory is a non-activity trajectory, it is determined whether the non-activity state duration is greater than the maximum retention time, and if yes, it is deleted from the trajectory set; Matching unsuccessful target: if the confidence of the target is greater than a certain value, it is initialized as a new trajectory.

Images