Two-stage multi-target tracking method and product based on sparse optical flow motion compensation

Abstract

The application provides a two-stage multi-target tracking method and product based on sparse optical flow motion compensation, and relates to the technical field of multi-target tracking, and comprises: processing multi-target information output by a target detector in stages, in a first stage, candidate targets with high confidence are matched, motion similarity and appearance similarity of a target to be tracked and a track target are considered, and the two similarities are fused into tracking similarity in a reasonable and reliable manner, and a track is assigned to the candidate target based on the tracking similarity; in a second stage, candidate targets with low confidence are matched, only the motion similarity of the target to be tracked and the track target is considered, and a track is assigned to the candidate target based on the motion similarity; the matching of the two stages is repeated until the last picture of a tracking sequence, and a track set is output. The multi-target tracking method provided by the application achieves more robust performance in multi-target tracking in complex scenes such as camera motion and fast target motion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-target tracking, in particular to a two-stage multi-target tracking method based on sparse optical flow motion compensation and a product. BACKGROUND

[0002] The task definition of multi-target tracking is to detect and track all targets in the picture frame for the picture sequence, generate the spatiotemporal trajectory of the target in the video stream, and each target keeps its own unique identifier (ID). As a computer vision task, multi-target tracking plays a huge role in the innovation of automatic driving, video monitoring and other fields. At present, in the field of multi-target tracking, the detection-based multi-target tracking is a very efficient paradigm, which mainly includes two steps: (1) motion modeling and estimating the bounding box of the target in the current frame, (2) combining the target detection result of the current frame with the historical trajectory to generate the target trajectory of the current frame.

[0003] For estimating the bounding box of the target in the current frame, a detector in the target detection field is usually used to generate reliable target bounding box estimation and target confidence result. The candidate target with the highest similarity to the target in the trajectory is usually selected to join the current trajectory, and the target in the current frame is joined into the historical trajectory according to such a strategy, until the last frame of the picture sequence, and the multiple target trajectories of tracking are output. However, the current detection-based multi-target tracking algorithm has the following shortcomings: most of the detection-based multi-target algorithms apply Kalman filter as the motion model of the target to predict the position of the historical trajectory target in the current frame. However, Kalman filter has a model assumption that the target moves at a constant speed, which is not suitable for the challenging properties of multi-target tracking such as fast target motion and camera motion, and secondly, the prediction of Kalman filter for the trajectory target in the current frame is the prediction of the proportion of the bounding box, not the direct prediction of the width and height of the bounding box. The prediction estimation method for the proportion of the bounding box introduces errors to the position estimation of the generated bounding box.

[0004] In addition, the performance of the candidate target combined with the trajectory based on the position similarity value depends on the prediction of the bounding box position of the trajectory target in the current frame. The challenging properties of multi-target tracking such as fast target motion and camera motion make the position predicted by Kalman filter inaccurate, which directly affects the tracking efficiency. SUMMARY

[0005] The present application provides a two-stage multi-target tracking method based on sparse optical flow motion compensation and a product to solve the problems of motion model estimation defects and inaccurate target bounding box generation in the multi-target trajectory in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a two-stage multi-target tracking method based on sparse optical flow motion compensation, which comprises:

[0007] Step 1, using the first frame picture in the tracking sequence to initialize the track set of multi-target tracking and Kalman filter parameters;

[0008] Step 2, using the n-th frame picture of the tracking sequence to start the two-stage multi-target tracking matching process, n is a positive integer greater than or equal to 2;

[0009] Step 3, using the detector to obtain the candidate target information of the n-th frame picture in step 2, and dividing the candidate targets into a high-confidence candidate target set and a low-confidence candidate target set based on the confidence of the candidate targets;

[0010] Step 4, in the first stage, based on the high-confidence candidate target set in step 3, calculating the motion similarity and appearance similarity of the candidate targets and the track targets and fusing to obtain the tracking similarity of the candidate targets and the track targets, and using the Hungarian algorithm to optimize the matching;

[0011] Step 5, in the second stage, based on the low-confidence candidate target set in step 3, calculating the motion similarity of the candidate targets and the track targets, and using the Hungarian algorithm to optimize the matching;

[0012] Step 6, if a high-confidence target that has not been matched appears continuously for more than two frames in the middle, it is considered as a newly appearing target in the middle, and is added to the current track set, and the same track of the same target in the track set is deleted;

[0013] Step 7, repeating steps 3 to 6 until the last frame picture of the tracking sequence, and outputting the track set.

[0014] Based on the first aspect, the step 4 comprises:

[0015] Step 401, using a convolution network to extract the appearance features of the candidate targets, and calculating the cosine distance of the appearance features of the candidate targets and the track targets, wherein the cosine distance is the appearance similarity of the candidate targets and the track targets;

[0016] Step 402, calculating the sparse optical flow of the track target corner points;

[0017] Step 403, calculating the prediction result of the Kalman filter of the track target in the current frame picture;

[0018] Step 404, modeling the motion of the track target in the previous and subsequent frame pictures;

[0019] Step 405, calculating the intersection over union distance of the candidate targets and the track targets;

[0020] Step 406, fusing the appearance similarity and the intersection over union distance of the candidate targets and the track targets;

[0021] Step 407: Update the appearance features of the trajectory target and the motion noise formula of the trajectory.

[0022] Based on the first aspect, the cosine distance in step 401 The definition is as follows:

[0023]

[0024] In the formula, Represents the appearance features of candidate targets in the k-th frame image. Appearance features of the i-th trajectory target cosine distance, Let represent the appearance features of the j-th candidate target detected in the k-th frame, and let || denote the L2 norm of the vector. Let w represent the list of appearance features of the i-th trajectory target in the k-th frame image. There are a total of σ features, where w represents the w-th appearance feature and σ represents the number of appearance features.

[0025] Based on the first aspect, in step 402, calculating the sparse optical flow at the corner points of the target trajectory includes:

[0026] Modeling is based on the consistency of sparse optical flow at the corner points of the trajectory target, that is, assuming that the brightness of the previous frame image is the same as that of the current frame image.

[0027] Calculate the optimal sparse optical flow value at the corner point of the trajectory target;

[0028] The modeling is defined as follows:

[0029] I(x,y,t)=I(x+u,y+v,t+1),

[0030] In the formula, I(·) represents the brightness at point (x,y), (u,v) represents the set of moving distances of the target corner point in the x and y directions, and t represents the time of the current frame;

[0031] The calculation of the sparse optical flow value is defined as follows:

[0032] I x u x +I y v y +I t =0,

[0033] In the formula, I x and I y I represents the derivative of I(x,y) in space, i.e., the gradient of the image. t The time derivative, u x and u ydenotes the moving distance of the corner point in x and y directions, u x and u y denotes the sparse optical flow value of the trajectory target corner point.

[0034] Based on the first aspect, in step 404, the motion of the trajectory target in the front and rear frame pictures is modeled, including:

[0035] The moving distance of the motion modeling of the trajectory target corner point in step 402 and the prediction result of the Kalman filtering of the current frame picture in step 403 are fused in the form of a weighted sum to obtain the motion modeling of the trajectory target current frame picture.

[0036] Wherein, the motion modeling is defined as follows:

[0037]

[0038] In the formula, M k denotes the set of bounding boxes of all trajectory targets of the kth frame based on the estimation of the previous frame, λ is a weight factor, is the optimal state estimation of the current frame, (u, v) denotes the set of moving distances of the trajectory target corner point.

[0039] Based on the first aspect, in step 406, the appearance similarity and the intersection-over-union distance of the candidate target and the trajectory target are fused, and the definition is as follows:

[0040]

[0041] In the formula, is the fusion result of the appearance similarity and the intersection-over-union distance of the candidate target and the trajectory target, ω is a weight factor, is the intersection-over-union distance of the candidate target and the trajectory target, In the kth frame, denotes the intersection-over-union distance between the target frame of the ith trajectory target and the jth candidate target, denotes the bounding box position of the ith trajectory target, denotes the bounding box position of the jth candidate target.

[0042] Based on the first aspect, in step 407, the appearance feature and the motion noise formula of the trajectory target are updated, including:

[0043] The appearance feature of the trajectory target in the trajectory set is updated, and the definition is as follows:

[0044]

[0045] In the formula, denotes the appearance feature of the trajectory target in the previous frame, fi k is the appearance feature of the current frame candidate target matching the trajectory, and a is a weight factor;

[0046] updating the trajectory motion noise formula of the trajectory target, defined as follows:

[0047] P k|k =(1-K k H k )P k|k-1 ,

[0048] wherein P k|k is the updated noise estimation of the trajectory target state of the current frame, H k represents an observation matrix, K k represents the Kalman coefficient of the current frame, P k|k-1 represents the transmission relationship of the trajectory target state uncertainty from the k-1th frame to the kth frame of discrete time, F k represents the target state transition matrix from the k-1th frame to the kth frame of discrete time, R k represents the measurement noise covariance, T represents the matrix transpose, and the covariance matrix Q k represents the noise of the prediction model itself.

[0049] In a second aspect, an embodiment of the present application provides a two-stage multi-target tracking system based on sparse optical flow motion compensation, which comprises:

[0050] An initialization module is configured to initialize the trajectory set and the Kalman filter parameters for multi-target tracking in step 1 by using the first frame of pictures in the tracking sequence.

[0051] A multi-target tracking matching module is configured to perform the two-stage multi-target tracking matching process in step 2 by using the nth frame of pictures in the tracking sequence, wherein n is an integer greater than or equal to 2.

[0052] A target set division module is configured to obtain the candidate target information of the nth frame of pictures in step 2 by using a detector in step 3, and divide the candidate targets into a high-confidence candidate target set and a low-confidence candidate target set based on the confidence of the candidate targets.

[0053] A first-stage matching module is configured to calculate the motion similarity and the appearance similarity between the candidate targets and the trajectory targets and fuse them to obtain the tracking similarity between the candidate targets and the trajectory targets based on the high-confidence candidate target set in step 3 in the first stage, and optimize the matching by using the Hungarian algorithm.

[0054] A second stage matching module is configured to, in step 5, in the second stage, calculate the motion similarity of the candidate targets and the track targets based on the low-confidence candidate target set in step 3, and optimize the matching by using the Hungarian algorithm.

[0055] An unmatched target searching module is configured to, in step 6, add the high-confidence targets that have not been matched and that appear for more than two frames in the middle to the current track set, and delete the same track of the same target in the track set.

[0056] A track set output module is configured to, in step 7, repeat steps 3 to 6 until the last frame of the tracking sequence, and output the track set.

[0057] In a third aspect, an electronic device is provided, and the electronic device comprises:

[0058] A memory is configured to store one or more programs;

[0059] A processor;

[0060] When the one or more programs are executed by the processor, the two-stage multi-target tracking method based on sparse optical flow motion compensation is implemented.

[0061] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, which, when executed by a processor, implements the two-stage multi-target tracking method based on sparse optical flow motion compensation.

[0062] The present application has the following advantages:

[0063] The present application fully utilizes the sparse optical flow and the Kalman filtering algorithm to model the motion of the target in the picture, the sparse optical flow generates the moving distance of the track target based on the corner points between the picture frames, the Kalman filtering algorithm predicts the positioning of the track target in the current frame of picture based on the target position of the track target in the last frame, the moving distance of the track target and the positioning of the track target in the current frame of picture are fused by the weighted sum method, the sparse optical flow compensates for the inaccurate prediction problem of the target caused by the uniform speed assumption in the Kalman filtering, and the multi-target tracking method provided by the present application achieves more robust performance in the multi-target tracking in complex scenes such as camera motion and fast target motion. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A flowchart of the two-stage multi-target tracking method based on sparse optical flow motion compensation provided by the present application is shown in the figure;

[0065] Figure 2 A flowchart of the first-stage multi-target tracking method provided by the present application is shown in the figure;

[0066] Figure 3 A schematic diagram of a two-stage multi-target tracking method based on sparse optical flow motion compensation is provided for an embodiment of the present application.

[0067] Figure 4 A two-stage multi-target tracking system architecture diagram based on sparse optical flow motion compensation is provided for an embodiment of the present application.

[0068] Figure 5 A schematic structural block diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0069] Currently, the focus of improvement of the multi-target tracking method is the joint task of target bounding box estimation and historical target trajectory, that is, how to associate the target estimation generated by the target detector in the current frame with the target trajectory in the historical frame, and ensure that each trajectory corresponds to the same target. The conventional idea for completing the above-mentioned joint task is to compare the similarity of the current frame target estimation (i.e. candidate target) and the target in the trajectory from two aspects: the position similarity of the bounding box of the candidate target and the trajectory and the appearance similarity. The position similarity of the bounding box refers to the intersection over union value between the candidate target and the bounding box of the trajectory. The appearance similarity refers to the feature vector similarity of the candidate target and the target in the trajectory. Both of the two similarities can be used to measure whether the current target belongs to a certain target trajectory. Based on the above-mentioned idea, the present application provides a two-stage multi-target tracking method based on sparse optical flow motion compensation and product, and according to the structural characteristics of the method, a high-efficiency multi-target tracking framework is designed, which solves the problems of motion model estimation defects and inaccurate target bounding box generation in the multi-target trajectory in the prior art.

[0070] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0071] In the embodiment of the present application, in the first aspect, the present application provides a two-stage multi-target tracking method based on sparse optical flow motion compensation, referring to Figure 1 , Figure 1 A flowchart of a two-stage multi-target tracking method based on sparse optical flow motion compensation is provided for an embodiment of the present application, and the method comprises the following steps:

[0072] Step 1, the track set of multi-target tracking in the first frame picture in the tracking sequence and Kalman filter parameter initialization are performed;

[0073] In the embodiment of the present application, the multi-target tracking public data set is a tracking sequence in the form of a picture sequence; all targets in the first frame picture in the tracking sequence are tracked targets, i.e., track targets, target detection is performed on the first frame picture by using an existing detector, high-confidence targets in the target detection result are taken as tracked targets, and these targets are put into a target track set; in the embodiment of the present application, Kalman filtering is a basic motion model used in the present application, and the estimation of the current motion state depends on the state transition from the last frame to the current frame, and the positions of all targets in the detection result are directly taken as the estimation result of the current state based on all targets in the first frame.

[0074] Step 2: Starting a two-stage multi-target tracking matching process from the nth frame picture of the tracking sequence, n is an integer greater than or equal to 2;

[0075] In the first frame picture, all high-confidence targets appearing are taken as tracked targets, and starting from the nth frame picture, similarity calculation is performed on the detected targets and the targets in the historical frames to associate the same targets in different frames. The tracking process starts from the first frame, but the track association between the candidate targets and the track targets starts from the nth frame. For example, after the first frame picture is initialized, a two-stage multi-target tracking matching process is performed by using the second frame picture.

[0076] Step 3: Obtaining candidate target information of the nth frame picture in step 2 by using a detector, and dividing the candidate targets into a high-confidence candidate target set and a low-confidence candidate target set based on the confidence of the candidate targets;

[0077] The detector can be YOLOX; the confidence is the uncertainty of the conclusion when the sample is randomly sampled to estimate the population parameter, the span of the confidence interval is a positive function of the confidence level, that is, the greater the required degree of certainty, the wider the confidence interval, which accordingly reduces the accuracy of the estimation. The confidence refers to the probability that the object contained in the bounding box is a target, and the high and low of the confidence is used to determine the high and low of the probability that the object contained in the bounding box is a target; when the confidence is high, it means that the target in the bounding box is more likely to be a candidate target and less likely to be a background; when the confidence is low, it means that the target in the bounding box is less likely to be a candidate target and more likely to be a background; the high-confidence target set refers to the probability that the object contained in the bounding box is a candidate target, and the low-confidence target set refers to the probability that the object contained in the bounding box is a candidate target.

[0078] Step 4, in the first stage, based on the high-confidence candidate target set in step 3, the motion similarity and appearance similarity of the candidate target and the trajectory target are calculated and fused to obtain the tracking similarity of the candidate target and the trajectory target, and the Hungarian algorithm is used to optimize the matching;

[0079] Wherein, the motion similarity is calculated according to the current frame picture detection result and the prediction result of the last frame picture; the appearance similarity is calculated according to the candidate target appearance of the current frame picture and the high-confidence target appearance in the historical frame, but the historical frame picture is not necessarily the last frame picture; the Hungarian algorithm is an algorithm for solving the minimum weight matching problem of bipartite graph, which can realize the optimal matching between targets, and the Hungarian algorithm can be used to match the trajectories of the candidate target and the trajectory target, and complete the association of the candidate target and the trajectory target.

[0080] Step 5, in the second stage, based on the low-confidence candidate target set in step 3, the motion similarity of the candidate target and the trajectory target is calculated, and the Hungarian algorithm is used to optimize the matching;

[0081] Wherein, in the second stage, the motion similarity of the candidate target in the low-confidence candidate target set and the trajectory target is calculated, which is the same as the calculation method of the motion similarity of the candidate target and the trajectory target in the high-confidence candidate target set, and is also calculated according to the current frame picture detection result and the prediction result of the last frame picture.

[0082] Step 6, the high-confidence target that has not been matched appears more than two frames in the middle, which is regarded as a new target appearing in the middle, and is added to the current trajectory set, and the same trajectory of the same target in the trajectory set is deleted;

[0083] Wherein, in the process of two-stage multi-target tracking matching, there may be a high-confidence target that has not been matched, when the high-confidence target that has not been matched appears in two continuous frames, the high-confidence target that has not been matched is regarded as a new target appearing in the middle, and the high-confidence target that has not been matched is added to the trajectory set; in addition, in the matching process, the same trajectory of the same target in the trajectory set may be repeatedly added, therefore, the same trajectory of the same target in the trajectory set needs to be deleted.

[0084] Step 7, repeat steps 3 to 6 until the last frame of the tracking sequence, and output the trajectory set.

[0085] Based on the two-stage multi-target tracking method provided in the above first aspect, the embodiment of the application further provides a first-stage multi-target tracking method, and the matching in the first stage is for high-confidence candidate targets, that is, conf k[i]>τ, representing the confidence level conf of the i-th detected target in the k-th frame. k [i] is greater than the threshold τ. See also... Figure 2 , Figure 2 A flowchart of a first-stage multi-target tracking method provided in an embodiment of the present invention, wherein step 4 above includes the following steps:

[0086] Step 401: Use a convolutional network to extract the appearance features of the candidate target, and calculate the cosine distance between the appearance features of the candidate target and the appearance features of the trajectory target, wherein the cosine distance is the appearance similarity between the candidate target and the trajectory target;

[0087] Among them, the convolutional network is an efficient convolutional network, such as ResNet and ResNeSt; in order to mitigate the interference of similar targets, the appearance features of the i-th trajectory target are... It consists of σ high-confidence historical frame targets, and multiple templates are spliced ​​together for appearance feature similarity calculation.

[0088] The cosine distance in step 401 The definition is as follows:

[0089]

[0090] In the formula, Represents the appearance features of candidate targets in the k-th frame image. Appearance features of the i-th trajectory target cosine distance, Let represent the appearance features of the j-th candidate target detected in the k-th frame, and let || denote the L2 norm of the vector. Let w represent the list of appearance features of the i-th trajectory target in the k-th frame image. There are a total of σ features, where w represents the w-th appearance feature and σ represents the number of appearance features.

[0091] Among them, the appearance features of candidate targets In the case of 1≤j≤N target Let j represent the j-th target detected in the k-th frame, and the k-th frame contains N targets. target There are N targets to track, and N target It is an integer.

[0092] Step 402: Calculate the sparse optical flow at the corner points of the trajectory target;

[0093] In step 402, calculating the sparse optical flow at the corner points of the trajectory target includes:

[0094] sparse optical flow of the trajectory target corner point is consistent, that is, modeling is performed assuming that the brightness of a previous frame of picture and a current frame of picture is consistent; and an optimal sparse optical flow value of the trajectory target corner point is calculated;

[0095] wherein the modeling is defined as follows:

[0096] I(x, y, t) = I(x + u, y + v, t + 1),

[0097] wherein I(·) represents the brightness at point (x, y), (u, v) represents a set of moving distances of the trajectory target corner point in x and y directions, and t represents the time of a current frame; wherein the calculation of the sparse optical flow value is defined as follows:

[0098] I x u x +I y v y +I t = 0,

[0099] wherein I x and I y represent the derivatives of I(x, y) in space, that is, the gradient of the image, I t represents the derivative in time, u x and u y represent the moving distances of the corner point in x and y directions, u x and u y represent the sparse optical flow value of the trajectory target corner point; wherein I x and I y can be calculated by a Sobel filtering algorithm, I t can be calculated by subtraction of two frames of images, and u x and u y can be calculated by a Lucas-Kanade algorithm.

[0100] In modeling the corner point of the trajectory target, first, the brightness in two frames of pictures is assumed to be consistent for modeling, which means that the corner points in two pictures are located at the same position. However, in the actual picture scene, the brightness of two frames of pictures is inconsistent. Therefore, the position of the corner point on the second picture needs to be calculated to obtain the offset of the same corner point in the x and y directions, that is, the moving distance. Exemplarily, the point (1, 1) (referred to as point one) in the first frame of picture is selected, and the brightness value of this position is 100 (the brightness range of the gray image is 0-255); the brightness value of the (1, 1) position in the second frame of picture becomes 50, which means that the point at the (1, 1) position in the second frame of picture is not point one. Therefore, the point with a brightness value of 100 around the position of the (1, 1) point needs to be found. It is known that the brightness value of the (9, 9) point (referred to as point two) is also 100, which means that point one and point two belong to the same point. The offset distance of point two relative to point one in the x direction is 8, and the offset distance of point two relative to point one in the y direction is 8. Therefore, the offset of point two relative to point one in the x and y directions is (8, 8), which is the calculated optical flow offset.

[0101] Step 403, calculating the prediction result of the Kalman filter of the trajectory target in the current frame of picture;

[0102] In order to utilize the trajectory target estimation state of the previous frame of picture (the k-1 frame) and the trajectory target observation state of the current frame of picture (the k frame) to predict the trajectory target estimation state of the current frame of picture, the trajectory target estimation state of the previous frame of picture (the k-1 frame) is represented as x , the trajectory target observation state of the current frame of picture (the k frame) is represented as z , the prediction of the trajectory target estimation state of the current frame of picture is represented as x , and the fitting of the real state of the trajectory target in the current frame of picture is represented as x , wherein

[0103] The prediction formula of the known state is defined as follows:

[0104]

[0105] In the formula, x represents the estimation target state of the k-1 frame of picture, F k represents the target state transition matrix from the discrete time k-1 frame of picture to the k frame of picture, represents the prediction of the estimation target state of the k frame of picture based on the estimation target state of the k-1 frame of picture, n k is a random variable, representing the random noise in the prediction process.

[0106] The relationship formula of the real state and the observation state of the trajectory target in the k frame of picture is defined as follows:

[0107] z k = H k x k+v k ,

[0108] wherein x k represents the real state of the kth picture, z k represents the observation state of the kth picture, H k represents an observation matrix, and v k is a random variable representing random noise of the observation process.

[0109] The state uncertainty formula represents the transmission relationship of the uncertainty of the picture at each time. Since the method needs to estimate the real value of the trajectory target, the influence of noise on the state of the trajectory target in the picture is considered, and the formula of the state of the trajectory target is defined as follows:

[0110]

[0111] wherein P k|k-1 represents the transmission relationship of the state uncertainty of the trajectory target from the (k-1)th picture to the kth picture, F k represents the transition matrix of the state of the trajectory target from the (k-1)th picture to the kth picture, T represents the transpose of the transition matrix F k , and Q k is a covariance matrix. The introduction of the covariance matrix represents the noise of the prediction model itself, that is, the uncertainty of the noise in the transmission process of the trajectory target from the (k-1)th picture to the kth picture.

[0112] Based on the above definitions, the trajectory target state prediction estimation and the trajectory target observation state z k , the observation matrix H k are used to generate the optimal state estimation

[0113]

[0114] wherein K k represents the Kalman coefficient of the current picture, represents the residual error between the actual observation state and the predicted state of the trajectory target state, H k represents the observation matrix, and R k represents the measurement noise covariance. In the actual implementation of the Kalman filter, the measurement noise covariance R can be generally observed and is a known condition of the Kalman filter.

[0115] In step 404, the motion of the trajectory target in the front and rear pictures is modeled.

[0116] In step 404, modeling the motion of the trajectory target in the preceding and following frames includes:

[0117] The motion modeling distance of the trajectory target corner point in step 402 and the prediction result of the Kalman filter of the current frame image in step 403 are fused in a weighted sum manner to obtain the motion modeling of the trajectory target in the current frame.

[0118] The motion modeling is defined as follows:

[0119]

[0120] In the formula, M k Let λ represent the set of bounding boxes for all trajectory targets in the k-th frame estimated based on the previous frame, where λ is the weighting factor. For the optimal state estimate of the current frame, (u,v) represents the set of movement distances of the target corner points on the trajectory.

[0121] Because the Kalman filtering method mentioned above assumes uniform velocity, this assumption makes the Kalman filtering algorithm inaccurate in modeling the distance of the trajectory target's motion in the preceding and following frames. Therefore, by fusing these two models through a weighted sum method, the sparse optical flow compensates for the inaccurate target prediction caused by the uniform velocity assumption in the Kalman filtering. In step 404, the movement distance (u) of the trajectory target corner point obtained in step 402 is used to model the motion. j ,v j ), 1≤j≤N target The trajectory target bounding box obtained in step 403 is fused with the Kalman filter prediction result in the current frame in the form of a weighted sum to form the motion model of the trajectory target in the current frame, where j represents the j-th target detected in the k-th frame image, and there are N targets in the k-th frame image. target There are N targets to track, and N target λ is an integer; considering that the motion modeling of the target corner points of the trajectory starts from the corner features of the image, there is noise in the motion modeling of invisible trajectory targets (e.g., targets that are occluded), so λ is usually set to 0.9.

[0122] Step 405: Calculate the intersection-union ratio (IU / U) distance between the candidate target and the trajectory target;

[0123] In step 405, the intersection-union distance (IU) between the current frame localization of the candidate target and the current frame localization of the trajectory target prediction is defined as follows:

[0124]

[0125] In the formula, in the k-th frame This represents the intersection-over-union distance (IoU) between the bounding boxes of the i-th trajectory target and the j-th candidate target. a bounding box position of the i-th track target, a bounding box position of the j-th candidate target.

[0126] Step 406, fusing the appearance similarity and the IoU distance between the candidate target and the track target;

[0127] In step 406, the appearance similarity and the IoU distance between the candidate target and the track target are fused, and the definition is as follows:

[0128]

[0129] In the formula, is the fusion result of the appearance similarity and the IoU distance between the candidate target and the track target, ω is a weight factor, is the IoU distance between the candidate target and the track target, In the k-th frame, denotes the IoU distance between the target frame of the i-th track target and the j-th candidate target, denotes a bounding box position of the i-th track target, denotes a bounding box position of the j-th candidate target.

[0130] The cosine distance between the appearance features of the candidate target in step 401 and the appearance features of the track target is fused with the IoU distance between the candidate target and the track target in step 405 to obtain the fusion result of the appearance similarity and the IoU distance between the candidate target and the track target. Considering that the targets processed at this stage are all targets in the high-confidence candidate target set, the reliability of the appearance features of the target is high, ω is set to ω=0.6; the threshold θ is set to θ=0.8, and the similarity matrix is used to complete the association between the candidate target and the track target by combining the Hungarian matching algorithm. match

[0131] Step 407, updating the appearance features of the track target and the motion noise formula of the track.

[0132] In step 407, the updating of the appearance features of the track target and the motion noise formula of the track includes:

[0133] The appearance features of the track target in the track set are updated, and the definition is as follows:

[0134]

[0135] In the formula, denotes the appearance features of the track target in the previous frame, f i k ​is the appearance feature of the current frame candidate target matched with the track, and a is a weight factor; wherein a is set as a=0.9;

[0136] updating the track motion noise formula of the track target, defined as follows:

[0137] P k|k =(1-K k H k )P k|k-1 ,

[0138] In the formula, the updated noise estimation of the track target state of the current frame, H k represents an observation matrix, K k represents the Kalman coefficient of the current frame, P k|k-1 represents the transfer relationship of the track target state uncertainty from the k-1 frame to the k frame of discrete time, F k represents the target state transition matrix from the k-1 frame to the k frame of discrete time, R k represents the measurement noise covariance, T represents the matrix transpose, and the covariance matrix Q k represents the noise of the prediction model itself; wherein P k|k is equivalent to updating the noise distribution after completing the multi-target tracking in the current frame picture, adapting to the changes of the target in the current frame. In order to facilitate the noise estimation of the next frame, P k|k is a frame-by-frame updated variable.

[0139] Exemplarily, refer to Figure 3 , Figure 3 is a schematic diagram of a two-stage multi-target tracking method based on sparse optical flow motion compensation provided by an embodiment of the present application. As an optional embodiment, Figure 3The flow in the method is to start from the t-th frame of the tracking sequence, and to complete the target trajectory generation in the t-th frame by using the information in the t-1-th frame. The specific steps are as follows: applying a target detector (such as YOLOX) to the t-th frame of the picture for target detection, and the result of the target detector is the candidate target in the t-th frame of the picture, including the target detection frame and the corresponding confidence. According to the relationship between the confidence and the threshold (detection threshold) described above, the target detection frame is divided into high-confidence target detection frame and low-confidence target detection frame; then, for the high-confidence detection frame, the appearance similarity is calculated: the picture block feature in the extracted candidate target confidence frame, that is, the candidate target feature, is used to calculate the cosine similarity with the trajectory target feature; the motion similarity is calculated: the predicted position of the trajectory target in the current frame is calculated based on the historical trajectory of the trajectory target by using Kalman filtering, and the predicted position of the trajectory target in the current frame is obtained by combining the sparse optical flow offset calculated from the t-1-th frame of the picture and the t-th frame of the picture; the appearance similarity and the motion similarity are fused, and the similarity between the predicted value of the candidate target and the detection value of the current frame is used to determine whether the target matches the trajectory; then, for the low-confidence detection frame, the motion similarity is calculated: the predicted position of the trajectory target in the current frame is calculated based on the historical trajectory of the trajectory target by using Kalman filtering, and the predicted position of the trajectory target in the current frame is obtained by combining the sparse optical flow offset calculated from the t-1-th frame of the picture and the t-th frame of the picture; then, for the high-confidence target that has not been matched to the trajectory, it is judged whether it is continuously present for more than two frames in the middle, if so, the high-confidence target that has not been matched to the trajectory is regarded as a newly appearing target in the middle, and is added to the current trajectory set Γ, and the repeated trajectories of the same target in the trajectory set are deleted; finally, the trajectory set of the target from the 1st frame to the t-th frame is output; wherein, Figure 3 The first association and the second association in the method have no chronological relationship, and in the code implementation level, the high threshold can be processed first and then the low threshold. Conversely, the low threshold candidate target in the pre-processing can be processed first, and then the high threshold target is processed. The corresponding settings are subject to the code sequence. In another optional embodiment, the first association and the second association can be processed in parallel.

[0140] The application provides a two-stage multi-target tracking method based on sparse optical flow motion compensation, which comprises the following steps: step 1, a first frame of picture in a tracking sequence is used to initialize a track set and Kalman filter parameters for multi-target tracking; step 2, a two-stage multi-target tracking matching process is started from the n-th frame of picture in the tracking sequence, wherein n is a positive integer greater than or equal to 2; step 3, a detector is used to obtain candidate target information of the n-th frame of picture in step 2, and the candidate targets are divided into a high-confidence candidate target set and a low-confidence candidate target set based on the confidence of the candidate targets; step 4, in the first stage, the high-confidence candidate target set in step 3 is used to calculate the motion similarity and appearance similarity of the candidate targets and the track targets and to fuse the two similarities to obtain the tracking similarity of the candidate targets and the track targets, and the Hungarian algorithm is used to optimize the matching; step 5, in the second stage, the low-confidence candidate target set in step 3 is used to calculate the motion similarity of the candidate targets and the track targets, and the Hungarian algorithm is used to optimize the matching; step 6, if a high-confidence target that has not been matched appears in more than two frames in the middle, the target is regarded as a newly appearing target in the middle and is added to the current track set, and the same track of the same target in the track set is deleted; and step 7, steps 3 to 6 are repeated until the last frame of picture in the tracking sequence, and a track set is output. The two-stage multi-target tracking method based on sparse optical flow motion compensation provided by the application processes the multi-target information output by a target detector in stages, in the multi-target matching process in the first stage, the motion similarity and appearance similarity of the track targets and the to-be-tracked targets are considered for the high-confidence candidate targets, and the two similarities are fused into tracking similarity in a reasonable and reliable manner, and the track targets are assigned to the candidate targets based on the tracking similarity; in the multi-target matching process in the second stage, only the motion similarity of the track targets and the to-be-tracked targets is considered for the low-confidence candidate targets, and the track targets are assigned to the candidate targets based on the motion similarity. The application makes full use of the sparse optical flow and Kalman filter algorithm to model the motion of the track targets in the picture, the sparse optical flow generates the moving distance of the track targets between picture frames based on the corner points, the Kalman filter algorithm predicts the positioning of the track targets in the current frame of picture based on the positioning of the track targets in the last frame of picture, and the two modeling methods are fused in a weighted manner, so that the sparse optical flow compensates for the inaccurate prediction of the track targets caused by the uniform speed assumption in the Kalman filter, and the tracker of the application achieves more robust performance in multi-target tracking in complex scenes such as camera motion and fast target motion.

[0141] Based on the two-stage multi-target tracking method based on sparse optical flow motion compensation provided in the first aspect, the application further provides a two-stage multi-target tracking system based on sparse optical flow motion compensation, which comprises the following steps: Figure 4 , Figure 4A two-stage multi-target tracking system architecture diagram based on sparse optical flow motion compensation is provided for an embodiment of the present application, and the system comprises the following modules:

[0142] An initialization module 110 is configured to initialize the track set and Kalman filter parameters for multi-target tracking in step 1 using the first frame of the tracking sequence.

[0143] A multi-target tracking matching module 120 is configured to start the two-stage multi-target tracking matching process in step 2 using the nth frame of the tracking sequence, where n is an integer greater than or equal to 2.

[0144] A target set division module 130 is configured to obtain candidate target information of the nth frame in step 2 using a detector in step 3, and divide the candidate targets into a high-confidence candidate target set and a low-confidence candidate target set based on the confidence of the candidate targets.

[0145] A first-stage matching module 140 is configured to calculate the motion similarity and appearance similarity between the candidate targets and the track targets and fuse them to obtain the tracking similarity between the candidate targets and the track targets in the first stage in step 4 based on the high-confidence candidate target set in step 3, and optimize the matching using the Hungarian algorithm.

[0146] A second-stage matching module 150 is configured to calculate the motion similarity between the candidate targets and the track targets in the second stage in step 5 based on the low-confidence candidate target set in step 3, and optimize the matching using the Hungarian algorithm.

[0147] An unmatched target searching module 160 is configured to add the high-confidence targets that have not been matched and have appeared for more than two frames in the middle to the current track set as newly appeared targets in step 6, and delete the same tracks of the same target in the track set.

[0148] A track set output module 170 is configured to repeat steps 3 to 6 until the last frame of the tracking sequence in step 7, and output the track set.

[0149] The application provides a two-stage multi-target tracking system based on sparse optical flow motion compensation. An initialization module 110 is used for step 1 to initialize a track set and Kalman filter parameters for multi-target tracking in a first frame of a tracking sequence. A multi-target tracking matching module 120 is used for step 2 to start a two-stage multi-target tracking matching process in the nth frame of the tracking sequence, where n is an integer greater than or equal to 2. A target set division module 130 is used for step 3 to obtain candidate target information of the nth frame in step 2 by using a detector, and divide the candidate targets into a high-confidence candidate target set and a low-confidence candidate target set based on the confidence of the candidate targets. A first-stage matching module 140 is used for step 4 to calculate the motion similarity and appearance similarity between the candidate targets and the track targets in the first stage based on the high-confidence candidate target set in step 3, fuse the motion similarity and the appearance similarity to obtain the tracking similarity between the candidate targets and the track targets, and optimize the matching by using the Hungarian algorithm. A second-stage matching module 150 is used for step 5 to calculate the motion similarity between the candidate targets and the track targets in the second stage based on the low-confidence candidate target set in step 3, and optimize the matching by using the Hungarian algorithm. An unmatched target searching module 160 is used for step 6 to regard a high-confidence target that has not been matched and appears continuously for more than two frames in the middle as a newly appearing target in the middle, add the target to the current track set, and delete the same track of the same target in the track set. A track set output module is used for step 7 to repeat steps 3 to 6 until the last frame of the tracking sequence, and output the track set. The application fully utilizes the sparse optical flow and the Kalman filter algorithm to model the motion of the track targets in the picture. The sparse optical flow generates the target movement distance between the picture frames based on the corner points, the Kalman filter algorithm predicts the target position in the current frame based on the target position in the last frame of the track target, and the two modeling methods are fused by using the weighted sum method. The sparse optical flow compensates for the inaccurate target prediction caused by the uniform speed assumption in the Kalman filter, and the tracker of the application achieves more robust performance in multi-target tracking in complex scenes such as camera motion and fast target motion.

[0150] Please refer to Figure 5 , Figure 5A schematic structural block diagram of an electronic device is provided in an embodiment of the present application. The electronic device includes a memory 101, a processor 102 and a communication interface 103, which are electrically connected to each other directly or indirectly to realize the transmission or interaction of data. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, such as the program instructions / modules of the class incremental learning system based on the dynamic class prototype generation mechanism provided in the embodiment of the present application. The processor 102 performs various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.

[0151] The memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0152] The processor 102 can be an integrated circuit chip with signal processing capability. The processor 102 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0153] It can be understood that, Figure 5 The structure shown is only schematic, and the electronic device can include more or fewer components than those shown in the figures, or have a different configuration from that shown in the figures. Figure 5 The components shown in the figures can be realized in hardware, software or a combination thereof. Figure 5 The components shown in the figures can be realized in hardware, software or a combination thereof. Figure 5 The components shown in the figures can be realized in hardware, software or a combination thereof.

[0154] In the embodiments of the present application, it should be understood that the disclosed system and method can also be implemented in other manners. The embodiments described above are merely exemplary for describing the present application and should not be used in a limiting manner. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the embodiments of the present application. In this regard, each block in the flowcharts and block diagrams can represent a module, a program segment or a part of code, which includes one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order from that shown in the accompanying drawings. For example, two consecutive blocks can actually be executed in a substantially parallel manner, and sometimes they can be executed in a reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for implementing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0155] In addition, the functional modules in the various embodiments of the present application can be integrated together to form a separate part, or each module can exist independently, or two or more modules can be integrated to form a separate part.

[0156] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0157] The above only describes the preferred embodiments of the present application and should not be used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0158] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims with respect to the figures of the patent document.

Claims

1. A two-stage multi-target tracking method based on sparse optical flow motion compensation, characterized in that, The method comprises: Step 1, using the first frame picture in the tracking sequence to initialize the track set of multi-target tracking and Kalman filter parameters; Step 2, starting from the n-th frame picture of the tracking sequence, a two-stage multi-target tracking matching process is adopted, and n is a positive integer greater than or equal to 2; Step 3, using a detector to obtain candidate target information of the n-th frame picture in step 2, and dividing the candidate targets into a high-confidence candidate target set and a low-confidence candidate target set based on the confidence of the candidate targets; Step 4, in the first stage, based on the high-confidence candidate target set in step 3, the motion similarity and appearance similarity of the candidate targets and the track targets are calculated and fused to obtain the tracking similarity of the candidate targets and the track targets, and the Hungarian algorithm is used for optimization matching; Step 5, in the second stage, based on the low-confidence candidate target set in step 3, the motion similarity of the candidate targets and the track targets is calculated, and the Hungarian algorithm is used for optimization matching; Step 6, if a high-confidence target that has not been matched appears continuously for more than two frames in the middle, it is considered as a newly appearing target in the middle, and is added to the current track set, and the same track of the same target in the track set is deleted; Step 7, repeating steps 3 to 6 until the last frame picture of the tracking sequence, and outputting the track set.

2. The method of claim 1, wherein, The step 4 comprises: Step 401, using a convolution network to extract the appearance features of the candidate targets, and calculating the cosine distance of the appearance features of the candidate targets and the track targets, wherein the cosine distance is the appearance similarity of the candidate targets and the track targets; Step 402, calculating the sparse optical flow of the track target corner point; Step 403, calculating the prediction result of the Kalman filter of the track target in the current frame picture; Step 404, modeling the motion of the track target in the previous and subsequent frame pictures; Step 405, calculating the intersection over union distance of the candidate target and the track target; Step 406, fusing the appearance similarity and the intersection over union distance of the candidate target and the track target; Step 407, updating the appearance features of the track target and the motion noise formula of the track.

3. The method of claim 2, wherein, The cosine distance in step 401 is defined as follows: wherein, denotes the appearance feature of the candidate target in the k-th frame of picture denotes the appearance feature of the i-th tracked target denotes the cosine distance between the appearance feature of the i-th tracked target and the appearance feature of the j-th candidate target detected in the k-th frame of picture, denotes the appearance feature of the j-th candidate target detected in the k-th frame of picture, and ||·||2denotes the L2norm of a vector, denotes the appearance feature of the i-th tracked target in the k-th frame of picture, in total there are σ appearance features, w denotes the w-th appearance feature, and σ denotes the number of appearance features.

4. The method of claim 2, wherein, In step 402, the sparse optical flow of the track target corner point comprises: Based on the consistency of the sparse optical flow of the track target corner point, that is, assuming that the brightness of the previous frame picture and the current frame picture is consistent for modeling; Calculating the optimal sparse optical flow value of the track target corner point; Wherein, the modeling is defined as follows: I(x,y,t)=I(x+u,y+v,t+1), In the formula, I(·) represents the brightness at point (x, y), (u, v) represents a set of movement distances of the track target corner point in x and y directions, and t represents the time of the current frame; Wherein, the calculation of the sparse optical flow value is defined as follows: I x u x +I y v y +I t =0, where I x and I y denotes the derivative of I(x,y) in space, i.e. the gradient of the image, I t denotes the derivative in time, u x and u y denotes the movement distance of the corner point in x and y direction, u x and u y denotes the sparse optical flow value of the trajectory target corner point.

5. The method of claim 2, wherein, In step 404, the motion of the track target in the previous and subsequent frame pictures is modeled, comprising: The motion modeling distance of the trajectory target corner point in step 402 and the prediction result of the Kalman filtering of the current frame picture in step 403 are fused in a weighted sum manner to obtain the motion modeling of the trajectory target current frame picture; The motion modeling is defined as follows: where M k represents the set of bounding boxes of all trajectory targets of the k-th frame estimated based on the previous frame, and λ is a weight factor, is the optimal state estimation of the current frame, and (u, v) represents the set of moving distances of the trajectory target corner points.

6. The method of claim 2, wherein, In step 406, the appearance similarity and the intersection-over-union distance of the candidate target and the trajectory target are fused, and the definition is as follows: In the formula, is the fusion result of the appearance similarity and the IoU distance between the candidate target and the trajectory target, and ω is a weight factor, is the IoU distance between the candidate target and the trajectory target, is the appearance similarity between the candidate target and the trajectory target in the kth frame, represents the IoU distance between the target frame of the ith trajectory target and the jth candidate target, represents the boundary frame position of the ith trajectory target, represents the boundary frame position of the jth candidate target.

7. The method of claim 2, wherein, In step 407, the appearance feature and the trajectory motion noise formula of the trajectory target are updated, including: The appearance feature of the trajectory target in the trajectory set is updated, and the definition is as follows: wherein represents the appearance feature of the trajectory target in the previous frame, f i k is the appearance feature of the current frame candidate target matched to the trajectory, and a is a weight factor. The trajectory motion noise formula of the trajectory target is updated, and the definition is as follows: P k|k = (1 - K k H k )P k|k-1 , where P k|k an updated noise estimate of the trajectory target state of the current frame, H k denotes an observation matrix, K k denotes the Kalman coefficient of the current frame, P k|k-1 denotes the transfer relation of the trajectory target state uncertainty from the k-1th frame to the kth frame of discrete time, F k denotes the target state transition matrix from the k-1th frame to the kth frame of discrete time, R k denotes the measurement noise covariance, T denotes the matrix transpose, the covariance matrix Q k denotes the noise of the prediction model itself.

8. A two-stage multi-target tracking system based on sparse optical flow motion compensation, characterized in that, The system comprises: An initialization module is configured to perform step 1, and initialize the trajectory set for multi-target tracking and the Kalman filtering parameters by using the first frame picture in the tracking sequence; A multi-target tracking matching module is configured to perform step 2, and start a two-stage multi-target tracking matching process by using the nth frame picture in the tracking sequence, where n is an integer greater than or equal to 2; A target set division module is configured to perform step 3, and obtain the candidate target information of the nth frame picture in step 2 by using the detector, and divide the candidate targets into a high-confidence candidate target set and a low-confidence candidate target set based on the confidence of the candidate targets; A first-stage matching module is configured to perform step 4, and in the first stage, calculate the motion similarity and the appearance similarity of the candidate targets and the trajectory targets based on the high-confidence candidate target set in step 3, fuse the motion similarity and the appearance similarity to obtain the tracking similarity of the candidate targets and the trajectory targets, and optimize the matching by using the Hungarian algorithm; A second-stage matching module is configured to perform step 5, and in the second stage, calculate the motion similarity of the candidate targets and the trajectory targets based on the low-confidence candidate target set in step 3, and optimize the matching by using the Hungarian algorithm; An unmatched target searching module is configured to perform step 6, and the high-confidence target that has not been matched appears for more than two frames in the middle, is regarded as a newly appearing target in the middle, is added to the current trajectory set, and the same trajectory of the same target in the trajectory set is deleted; A trajectory set output module is configured to perform step 7, and repeat steps 3 to 6 until the last frame picture in the tracking sequence, and output the trajectory set.

9. An electronic device, comprising: Comprise: A memory for storing one or more programs; A processor; When the one or more programs are executed by the processor, the two-stage multi-target tracking method based on sparse optical flow motion compensation is realized.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the two-stage multi-target tracking method based on sparse optical flow motion compensation. The computer program is executed by the processor to realize the two-stage multi-target tracking method based on sparse optical flow motion compensation.

Images