Multi-target tracking method based on weak clue and trajectory prediction

Abstract

The invention discloses a multi-target tracking method based on weak clue and trajectory prediction, and belongs to the technical field of deep learning multi-target tracking. The method comprises the following steps: firstly, setting a target detection model, then setting a target tracking algorithm, predicting a future trajectory of a target by using a position velocity-LSTM model when shielding occurs, and calculating the movement direction cost; if not, track prediction is carried out by adopting Kalman filtering; when the position cost is calculated, appearance features are extracted by using an appearance feature model on the basis of a designed mixed height intersection-to-union ratio using a weak clue, namely a height state, and after each clue is obtained, trajectory distribution is carried out on a target by using strong and weak clues through a Hungary algorithm. In order to balance and utilize each cost, multi-target tracking is realized through three times of association based on a configured cost allocation strategy. According to the method, the problems of shielding, similar objects, appearance degradation and the like are solved to a certain extent, and the model generalization ability and the target tracking ability to complex scenes are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning multi-target tracking, and in particular to a multi-target tracking method based on weak clues and trajectory prediction. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, especially the widespread application of deep learning algorithms in the field of computer vision, multi-target tracking technology has been greatly improved. It has important application value in the fields of intelligent monitoring, autonomous driving, drone navigation, etc. Its core goal is to track multiple targets simultaneously in a video sequence and assign a unique identity to each target. Generally, multi-target tracking algorithms need to handle complex scenes, including target occlusion, changes in the number of targets, dynamic changes in the background, etc., which puts extremely high demands on the robustness and accuracy of the algorithm.

[0003] At present, there are three mainstream models for video multi-target tracking:

[0004] The first one is detection-based multi-target tracking, which is to first detect the target in each frame of the video sequence, crop the target according to the bounding box, and obtain all the targets in the image. Then, it is transformed into the target association problem between the previous and next frames, and the similarity matrix is ​​constructed through IoU, appearance, etc., and solved by methods such as Hungarian algorithm and greedy algorithm.

[0005] The second type is multi-target tracking based on joint detection and tracking, which is to add a parallel feature extraction branch to the detection network, so as to output the location and appearance features of the target simultaneously in a single network. By making the two tasks share features, some repeated calculations are effectively avoided, and the tracking speed of the model is improved.

[0006] The third type is multi-target tracking based on the attention mechanism, which takes multiple adjacent frames as input and predicts the current position offset or appearance features based on the target's previous motion or appearance information, thereby associating the target. Currently, the algorithms based on the JDT paradigm are mainly divided into methods based on twin networks and methods based on Transformer.

[0007] The common drawbacks of the above existing video multi-target tracking methods are mainly reflected in:

[0008] (1) During the tracking process, weak clues such as height feature clues, historical trajectory clues, and motion speed clues are not properly utilized when associating data.

[0009] (2) Appearance degradation and target loss caused by occlusion often result in identity exchange and trajectory fragmentation, making data association more difficult.

[0010] (3) When the costs of strong cues such as position cues and appearance cues and weak cues such as trajectory cues, height cues, and motion speed cues are introduced, it is difficult to balance the various costs. Summary of the invention

[0011] The present invention provides a multi-target tracking method based on weak cues and trajectory prediction, aiming to solve the problems of insufficient utilization of weak cues, association failure caused by occlusion, and difficulty in balancing multiple association costs in existing target tracking methods.

[0012] The technical solution adopted by the present invention is:

[0013] A multi-target tracking method based on weak clues and trajectory prediction, the method comprising the following steps:

[0014] Step 1, setting a target detection model for obtaining target detection results of frame images, and training the target detection model based on a set training image set, and obtaining a target detector based on the trained target detection model; wherein the target detection results include target categories, target detection boxes (target positions and bounding boxes) and their confidence levels;

[0015] Step 2: Obtain the target detection result of the frame image based on the target detector, and perform target tracking based on the target detection frame of the frame image:

[0016] The target detection frame of the frame image is used as an observation and input into the tracker for state estimation. If the current frame is the first frame of the video, the initial trajectory is directly assigned to the detected target; otherwise, state estimation is performed based on the Kalman filter and the position-speed-LSTM model. When occlusion is detected, the position-speed-LSTM model is used for trajectory prediction; otherwise, the Kalman filter is used for trajectory prediction; the trajectory assignment of multiple detected targets is performed through multiple Hungarian algorithms, the detected target in the current frame is matched with the tracked target in the previous frame, and the trajectory is assigned to the detected target in the current frame to update the detection result of successful data association, that is, update the tracker corresponding to the detected target; and create a new tracker and assign an initial trajectory for the unmatched detected target (new target). In order to further reduce unnecessary resource usage, trackers whose update time exceeds the set threshold can also be cleaned up;

[0017] Among them, the trajectory prediction of the position-speed-LSTM model specifically includes: in the encoding stage, the model encodes the position and speed information of the historical target detection frame into a hidden state; in the decoding stage, the speed decoder predicts the speed change of the future target detection frame of the detection target based on the encoded hidden state and the recently observed speed information, and then calculates the position and size of the future target detection frame of the detection target based on the speed change conversion output by the speed decoder, and outputs the prediction result of the target detection frame of a specified number of future frames.

[0018] Furthermore, the object detection model includes: backbone network, feature pyramid network, decoupling head, anchor-free detection head and classification and regression module in sequence;

[0019] The backbone network is used to extract multi-scale image features of the input image, that is, based on the input image, to extract the underlying features of the image at multiple scales; preferably, Modified CSP v5 can be used as the backbone network, so as to effectively extract the spatial features and semantic information of the input image;

[0020] Furthermore, the backbone network Modified CSP v5 network includes CSP Bottleneck, Darknet53, cross-stage partial connection, downsampling layer and residual connection convolution layer in sequence; among them, CSP Bottleneck is used to divide the feature map into two parts, one of which is processed by multiple convolution layers, and the other part remains unchanged, and finally the outputs of the two parts are merged. This design allows deeper networks to learn features more efficiently. Darknet53 contains multiple convolution layers and residual blocks. The residual block avoids gradient disappearance and accelerates training through jump connections, so that the network still maintains good performance at deeper levels. In the cross-stage partial connection network, its input feature map is divided into two parts, one of which is processed by one stage of convolution, and the other is unchanged, and finally the feature maps of the two parts are merged. This can effectively reduce the computational complexity of the network and improve training efficiency. The downsampling layer implements downsampling through convolution layers and pooling layers (such as maximum pooling). These layers reduce the size of the input feature map while maintaining important feature information. Residual connected convolutional layers, each residual block consists of multiple convolutional layers and a skip connection, which directly adds the input to the convolutional output.

[0021] The feature pyramid network takes the image features extracted by the backbone network as input, and is used to enhance and fuse the multi-scale image features extracted by the backbone network, outputting enhanced feature maps of three different scales, thereby improving the detection performance of objects of different sizes.

[0022] Decoupling head: For the FPN features at each scale level, a 1×1 convolution layer is first used to reduce the feature channels to the target number of channels, such as 256, and then two parallel branches are added, each with two 3×3 convolution layers for classification and regression tasks, respectively, where the IoU branch is added to the regression branch.

[0023] Anchor-free detection head, which means that the model no longer relies on predefined anchor boxes for object localization. Instead, the detection head directly regresses the object's bounding box (i.e., object detection box) position and classification results on the feature map (i.e., the feature map output by the regression task branch of the decoupled head). This design simplifies the model and reduces the complexity of hyperparameter tuning.

[0024] The classification and regression module is used to further process the features output by the detection head to achieve object classification and bounding box regression. Through the decoupled head design, the classification and regression tasks are separated to improve the detection performance.

[0025] Furthermore, model training of the target detection model based on the set training image set is specifically as follows: deep learning training is performed on the target detection model based on the training image set, and when the preset convergence conditions are met (the number of training times reaches an upper limit or the set loss function converges), a target detector for each frame image in the detection video sequence for multi-target tracking is obtained; and the total loss function used in the deep learning training of the target detection model is the weighted sum of the prediction box regression loss function, the classification loss function and the confidence loss function.

[0026] Furthermore, the prediction box regression function is specifically set as:

[0027] L reg =1-IoU

[0028] Among them, L reg It represents the prediction box regression function, and IoU is the intersection over union ratio between the real box and the prediction box (i.e., the target detection box output by the model).

[0029] The classification loss function and confidence loss function are specifically set as:

[0030]

[0031] Among them, L obj represents the classification loss function, L obj represents the confidence loss function, N is the total number of training samples, y i is the true label of the i-th training sample, which takes a value of 0 or 1. is the predicted value of the i-th training sample, is the Sigmoid function, defined as:

[0032] Furthermore, the expression of the total loss function used in deep learning training of the target detection model is:

[0033] L=λ reg L reg +λ cls L cls +λobj L obj

[0034] Among them, L represents the total loss function, λ reg , cls , obj They are the weights of the prediction box regression loss function, the classification loss function, and the confidence loss function, respectively.

[0035] Furthermore, the position-speed-LSTM model includes a speed encoder, a position encoder and a speed decoder based on an LSTM network (Long Short-Term Memory Network);

[0036] The speed encoder is used to encode the speed information of the target detection frame to obtain its corresponding hidden state, and the position encoder is used to encode the position information of the target detection frame to obtain its corresponding hidden state; the two are then connected into a fused hidden state as the initial input of the speed decoder;

[0037] For the speed encoder, the input data is: based on the center coordinates of the target detection frame (x t ,y t ), width W t and height H t Get the target detection box information B of the tth frame t =(x t ,y t ,W t ,H t ), and then based on the target detection box information B of the specified sequence length t Get the detection frame sequence, and extract the speed information of the tth frame of the detection frame sequence based on the position and size changes Based on the speed information of each frame of the detection frame sequence Get the input data of the speed encoder;

[0038] Speed ​​encoder based on and the velocity hidden state v of the previous frame t-1 Calculate the speed hidden state of the current frame in, represents the output of the LSTM-based velocity encoder, W v is the weight matrix of the speed encoder, which is a trainable parameter;

[0039] For the position encoder, its input data is the target detection box information B t The detection frame sequence is composed of the position encoder based on B t and the position hidden state p of the previous frame t-1 Calculate the position hidden state of the current frame in, represents the output of the LSTM-based position encoder, W p is the weight matrix of the position encoder, which is a trainable parameter;

[0040] The speed decoder is based on the fused hidden state is the initial hidden state, and the speed information v obtained from the most recent observation Bt Perform the first prediction as input and get the next hidden state in, represents the output of the LSTM-based decoder, W dv is the weight matrix of the speed decoder, which can be trained parameters; then the predicted hidden state Feed to the fully connected layer to predict the next speed information Where W ov 、b ov They are the weight matrix and bias vector of the output layer (fully connected layer), both of which are trainable parameters; multiple rounds of operations are performed iteratively to save the speed information of each round of fully connected layer output The subscript t′ represents the preset number of iteration rounds;

[0041] Finally, the position and size of the future target detection frame are calculated based on the predicted speed information and the corresponding target detection frame information: Among them, the number of iterations j = 1,…,t′.

[0042] Furthermore, when performing trajectory assignment of multiple detection targets through multiple Hungarian algorithms, the calculated assignment costs include: mixed height position cost, appearance cost and motion direction cost.

[0043] Furthermore, the cubic Hungarian algorithm is used to perform trajectory assignment of multiple detection targets to achieve cubic data association of trajectory assignment, specifically:

[0044] During the first data association, only the confidence level is higher than or equal to the preset first confidence threshold τ high The target detection box uses the Hungarian algorithm to assign trajectories. The allocation cost used includes the mixed height position cost, appearance cost, and motion direction cost.

[0045] After the first data association is completed, the target detection frames whose appearance feature cost and position cost exceed the specified threshold are filtered out, and the Hungarian algorithm performs the second data association on the remaining unassociated target detection frames, and the allocation cost during the second data association includes the mixed height position cost and appearance cost;

[0046] After the second data association is completed, the target detection frames whose appearance feature cost exceeds the specified threshold or the position cost exceeds the specified threshold are filtered out, and the Hungarian algorithm performs the third data association on the remaining unassociated target detection frames, and the allocation cost during the third data association is the mixed height position cost;

[0047] After the third data association is completed, the target detection frames whose location cost exceeds the specified threshold are filtered out, so that the new target can be obtained from the remaining unassociated target detection frames.

[0048] Furthermore, in step 2, the following steps are used to perform occlusion detection:

[0049] The image features of the image content of the target detection frame are extracted as its appearance feature vector, and occlusion detection is performed based on the intersection-over-union ratio of the target detection frames of adjacent frames, the appearance feature vector, and the confidence of the target detection frame of the current frame:

[0050] (1) The intersection-and-union ratio of the target detection frame of the current frame and the previous frame is greater than or equal to the preset intersection-and-union ratio threshold τ loU ;

[0051] (2) The number of continuous tracking times is greater than or equal to the preset continuous tracking times threshold τ hs ;

[0052] (3) The cosine similarity between the appearance feature vector i of the target detection frame in the previous frame and the appearance feature vector j of the target detection frame in the current frame Less than or equal to the preset cosine similarity threshold τ d , or the confidence of the target detection frame of the current frame is greater than or equal to the preset threshold τ conf ;

[0053] When the above three conditions are met at the same time, it is determined that there is occlusion in the current frame.

[0054] Furthermore, a re-identification model may be used to extract the appearance feature vector of the target detection frame, wherein the re-identification model includes a feature extraction network (for example, a backbone network of ResNet-50 is used as the feature extraction network), a global average pooling layer, a normalization layer, and a fully connected layer in sequence, and the input data of the re-identification model is the image data corresponding to the target detection frame; and then the measurement value of the re-identification model for the target detection frame, i.e., the appearance cost, may be obtained based on the cosine similarity between the appearance feature vectors of the target detection frame of the same target in the previous and next frames extracted;

[0055] Furthermore, the mixed height position cost, appearance cost and motion direction cost in the allocation cost are specifically set as:

[0056] (1) Mixed height position cost

[0057] The mixed height intersection-over-union ratio is calculated according to the formula MHIoU=αHMIoU+βIoU, where MHIoU is the mixed height intersection-over-union ratio, HMIoU is the height modulation intersection-over-union ratio, and IoU is the intersection-over-union ratio of the target detection frame of the current frame and the previous frame; α and β are preset weight coefficients respectively;

[0058] in,

[0059] A and B are the areas of the two target detection boxes;

[0060] The calculation expression of the highly modulated intersection-over-union ratio HMIoU is:

[0061]

[0062] HMIoU=HIoU·IoU

[0063] Among them, HIoU is the height coefficient, and the two target detection boxes are defined as and Where x1 and y1 represent the upper left corner, and x2 and y2 represent the lower right corner.

[0064] The mixed height position cost C is obtained according to the mixed height intersection-over-union ratio MHIoU MHIoU = -MHIoU;

[0065] (2) Appearance cost

[0066] The re-identification model is used to extract the appearance feature vector of the target detection frame, and the cosine similarity between the appearance feature vector i of the target detection frame of the previous frame and the appearance feature vector j of the target detection frame of the current frame is used as the appearance cost C A ppr, the appearance feature similarity cost, is expressed as:

[0067] (3) Movement direction cost (also called trajectory direction cost or speed direction cost)

[0068] The velocity direction is calculated using the four corners of the detection box. Given two points (u1, v1) and (u2, v2), the velocity direction is calculated as follows.

[0069]

[0070] In the present invention, the cost metric used is the absolute difference between the trajectory velocity direction θt and the trajectory to the new detection frame velocity direction θd, expressed in radian format, expressed as Δθ=|θt-θd|. The trajectory velocity direction is obtained from the centers of two detection frames in the trajectory at a time interval Δt, and the trajectory to the new detection frame velocity direction is obtained from the centers of the trajectory history detection frame and the new detection frame.

[0071] The expression for the velocity direction cost is:

[0072]

[0073] Among them, C Vel is the total cost in the velocity direction, The velocity direction costs are the upper left corner, upper right corner, lower left corner, and lower right corner respectively.

[0074] The trajectory speed direction θt is obtained based on the speed direction between the center points of the two target detection frames in the previous and next frames, or based on different time intervals Δt, the speed direction between the center points of the two target detection frames with a time interval of Δt in the currently obtained trajectory is calculated, and then the trajectory speed direction θt is obtained based on the cumulative sum of the speed directions of all the set time intervals Δt; and the speed direction θd of the trajectory to the new detection frame is obtained based on the speed direction between the trajectory and the center point of the target detection frame of the current frame, and the absolute error between θt and θd is used as the motion direction cost; the motion direction cost can also be obtained based on the cumulative sum of the average values ​​of the speed direction θ between the four vertices of the target detection frames of the two frames with a specified interval before and after, that is, the total speed direction cost C V el.

[0075] In this application, when performing trajectory allocation of multiple detection targets based on the Hungarian algorithm, the allocation cost considered includes strong and weak clues. Usually, position clues, appearance clues, etc. can be regarded as strong clues, and trajectory clues, height clues, and motion speed clues can be regarded as weak clues. After obtaining the mixed height position cost, appearance cost, and motion direction cost, the cost is properly managed based on the Hungarian allocation based on the cost management strategy involved. It is specifically embodied in the use of cost filtering before Hungarian allocation and trajectory rejection after Hungarian allocation. Cost management can be completed through three data associations.

[0076] Furthermore, when the target moves in the x direction, due to the camera perspective principle, the target height changes slightly, and when the target moves in the y direction, the object height changes significantly. To cope with this situation, the weight coefficient α of HMIoU in the mixed height position cost can be further adjusted in the following way:

[0077]

[0078] Among them, α′ represents the weight coefficient of the adjusted HMIoU, that is, when the longitudinal speed v y Too large (exceeding the preset threshold ), the coefficient α of HMIoU is gradually reduced by the preset adjustment step Δα. Each time α is reduced (α = α - Δα), the longitudinal velocity v needs to be detected. yIs it back to normal? When it is detected that the longitudinal speed vy returns to normal, the reduction adjustment is stopped; or even if the longitudinal speed v y If the current longitudinal speed is still too large, the adjustment will stop when the coefficient α decreases to 0; if the current longitudinal speed is too small, that is, when When the coefficient α is gradually increased by the set adjustment step, each time α is increased (α=α+Δα), the longitudinal velocity v needs to be detected. y Is it too large? When it is too large, stop increasing the adjustment, or when it is detected that the value of the coefficient α increases to the preset upper limit α max Stop adjusting it when the longitudinal speed v y The coefficient α is adjusted in real time according to the state: Check the longitudinal speed v of the target detection frame y Whether it exceeds If so, α is gradually reduced with the set step size until the longitudinal velocity v y Less than or equal to or until the value of α decreases to a preset lower limit (preferably set to 0); if v y Less than Then gradually increase α with the set step size until the longitudinal velocity v y Greater than Stop increasing the adjustment when the value of α increases to the preset upper limit α max Stop increasing the adjustment.

[0079] In the process of three-way data association, the present invention adopts a specific cost management strategy to balance the cost of each clue, which is:

[0080] In the first data association, only high confidence detection boxes (i.e., those with confidence higher than or equal to the preset first confidence threshold τ) are included. high The target detection box) uses Hungarian matching, that is, the expression of the allocation cost used in the first data association is:

[0081] C1=C MHIoU +λ1C Vel +λ2C Appr

[0082] Among them, C1 is the allocation cost of the first data association, λ1 is the speed cost coefficient, and λ2 is the appearance feature cost coefficient, all of which are preset values;

[0083] The expression of the allocation cost during the second data association is: C2 = C MHIoU +λ3C Appr ;

[0084] Among them, C2 is the allocation cost of the second data association, and λ3 is the appearance feature cost coefficient during the second data association;

[0085] Correspondingly, the allocation cost used in the third data association can be directly expressed as: C3 = C MHIoU .

[0086] Furthermore, a filtering function is used to solve the problem that when the MH-IOU score is significantly higher than the appearance similarity score between two different objects, they may only overlap but are not identical. Its expression is as follows:

[0087]

[0088] Where C is the cost and τ1 is C MHIoU threshold, τ2 is the appearance cost threshold.

[0089] Furthermore, in the three data association processes, after each data association is completed, the filtering strategy for the target detection frame is specifically set as follows:

[0090] After the first data association is completed, the filtering strategy for the target detection box is: if the target detection box satisfies C MHIoU >τ3 and C Appr >τ4, the current target detection box is set as the filtering object, where τ3 is the rejection threshold of the MHIoU cost of the first data association, and τ4 is the rejection threshold of the appearance feature cost of the first data association;

[0091] After the second data association is completed, the filtering strategy for the target detection frame is: if the target detection frame satisfies C MHIoU >τ5 or C Appr >τ6, the current target detection box is set as the filtering object, where τ5 is the rejection threshold of the MHIoU cost of the second data association, and τ6 is the rejection threshold of the appearance feature cost of the second data association;

[0092] After the third data association is completed, the filtering strategy for the target detection frame is: if the target detection frame satisfies C MHIoU >τ7, the current target detection box is set as the filtering object, where τ7 is the rejection threshold of the MHIoU cost of the third data association.

[0093] The present invention completes the trajectory allocation of detected targets and realizes multi-target tracking through the above three-time data association trajectory allocation.

[0094] The technical solution provided by the present invention brings at least the following beneficial effects:

[0095] The present invention can perform efficient multi-target tracking after inputting the video frame, and obtain the target position and target tracking results. Based on the existing multi-target tracking model based on deep learning, the present invention introduces a trajectory prediction model, and uses the position speed-LSTM to predict the position and movement direction of the target in the next frame, which can better judge the target identity and reduce the impact of occlusion on target identity recognition. The introduction of MH-IoU can reasonably use the information of the target's height when associating objects, making the association more robust. The ReID-CCM (Cost Control Module) is introduced to obtain the target appearance features, and the CCM cost management strategy is designed to balance the costs of position clues, appearance clues, trajectory clues and height clues. The characteristics of simplicity, online and real-time are achieved, and the performance of multi-target tracking is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0097] Figure 1 A schematic diagram of a network architecture of a multi-target tracking method based on weak clues and trajectory prediction provided by an embodiment of the present invention;

[0098] Figure 2 This is a diagram of the position speed-LSTM trajectory prediction network architecture used in an embodiment of the present invention.

[0099] Figure 3 This is a structural diagram of the re-identification module used in an embodiment of the present invention.

[0100] Figure 4 The data association flow chart used in the embodiment of the present invention. DETAILED DESCRIPTION

[0101] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0102] Video multi-object tracking (MOT) technology is a computer vision technology that aims to detect and track multiple targets simultaneously from a video. Its core task is to assign a unique identifier to each target and maintain the continuity of the target in each frame of the video. MOT technology usually involves two key steps: target detection and data association. Target detection is responsible for identifying the target in each frame, while data association matches the detected target with the tracked targets in the previous frames to ensure the continuity of the target's trajectory.

[0103] Multi-target tracking technology is of great significance in many fields. For example, in intelligent transportation systems, multi-target tracking can be used to monitor vehicles and pedestrians on the road, help analyze traffic flow and prevent accidents; in security monitoring, multi-target tracking can be used to track suspicious people and improve public safety; in sports game analysis, multi-target tracking technology can be used to track athletes' performance and provide data support for coaches and analysts. In addition, multi-target tracking is also widely used in autonomous driving, smart retail, drone monitoring and other fields. The research and development of multi-target tracking technology will help improve the intelligence level and application breadth of machine vision systems.

[0104] The embodiment of the present invention provides a multi-target tracking method based on weak cues and trajectory prediction to solve the problems in the prior art multi-target tracking method based on deep learning recognition, that is, the existing algorithm causes appearance degradation and target loss caused by occlusion, which often causes identity exchange and trajectory fragmentation; and the weak cues such as height and speed are not fully utilized. The method is based on the SORT (Simple Online and Realtime Tracking) paradigm, performs tracking based on detection, and makes improvements in trajectory prediction after occlusion, utilization of weak cues such as height and speed, and cost balance of strong and weak cues, thereby achieving simple, online and real-time characteristics and improving the performance of multi-target tracking.

[0105] like Figure 1 As shown, in one embodiment, a multi-target tracking method based on weak clues and trajectory prediction provided by the present invention specifically includes the following steps:

[0106] Step 1: Build a target detection model to obtain the target detection information of the frame image (i.e. the position, size, confidence and target category of the target detection box (also called bounding box, detection box, predicted detection box, etc.)):

[0107] Specifically, in this embodiment, Modified CSP v5 is used as the backbone network. The backbone network used is composed of multiple layers of basic convolutional layers, residual layers, and a spatial pyramid pooling module, and each layer extracts and processes features step by step. The input is an RGB image with a shape of 3×H×W. Among them, "3" is the image RGB channel, "H" is the height of the image, and "W" is the width of the image. After the basic convolutional layer, the residual layer and a spatial pyramid pooling module, the FPN feature maps of the shapes are: (256, H / 8, W / 8), (512, H / 16, W / 16), (1024, H / 32, W / 32).

[0108] The obtained FPN feature map then enters the decoupling head. For each level of FPN features, a 1×1 convolution layer is first used to reduce the feature channels to 256, and then two parallel branches are added, each with two 3×3 convolution layers for classification and regression tasks respectively. The IoU branch is added to the regression branch. Finally, the classification prediction with a shape of C×H×W, the regression prediction with a shape of 4×H×W, and the IoU prediction with a shape of 1×H×W are obtained.

[0109] Step 2: Use the SORT paradigm target tracking algorithm to perform target tracking, including:

[0110] The target detection frame obtained by the target detection model is input into the tracking model. If this frame is the first frame of the video, the initial trajectory is directly assigned to the detected object. Otherwise, calculations are started including position and height clues, appearance clues, trajectory clues, and speed clues.

[0111] like Figure 2 As shown in the figure, the trajectory clues of the target are calculated through the trajectory prediction model. When the tracking is good, the Kalman filter is continued to be used for prediction. However, when occlusion occurs, the trajectory prediction module is used to predict the trajectory based on the position speed-LSTM. In the Kalman filter prediction detection frame, the state transfer matrix F is used to predict the target state in the next time step. That is, the position of the target is predicted based on the current state and speed, and the uncertainty of the state in the next time step is predicted through covariance prediction.

[0112] The trajectory prediction in this embodiment can be expressed by the formula:

[0113]

[0114] in, is the predicted detection box, B t is the detection box at time t, KF() is the Kalman filter, PV-LSTM() is the trajectory prediction based on position velocity-LSTM, and f oIs whether occlusion occurs, if the value is 1, it means occlusion occurs, if the value is 0, it means no occlusion. Further, the judgment expression of occlusion perception in this embodiment can be expressed as follows:

[0115]

[0116] in is the cosine similarity between the appearance feature vector i of the trajectory and the appearance feature vector j of the new detection box. conf is the confidence, hs is the number of consecutive tracking times, τ loU , τ hs , τ conf , τ d They are the preset IoU threshold, the threshold of continuous tracking times, the confidence threshold and the cosine similarity threshold.

[0117] In this embodiment, the trajectory prediction based on position speed-LSTM is specifically as follows: in the encoding stage, the model (the trajectory pre-model based on position speed-LSTM) encodes the position and speed information of the historical bounding box into a hidden state, and selects to use the trajectory of the historical 8 frames. In the decoding stage, the speed decoder predicts the speed change of the object's future detection box based on the encoded hidden state and the last observed speed information. These speed predictions are then converted into the position and size of the future detection box, which is achieved by cumulative addition update. The detection box of the next 8 frames is selected for output.

[0118] like Figure 3 As shown in the figure, the appearance clues are calculated and the FastReID re-identification model is used to extract the appearance features. The detected target image with a shape of 3×H×W is firstly subjected to feature extraction through the ResNet-50 backbone network. After four residual layers, a feature map with a shape size of (2048, H / 32, W / 32) is obtained. Then, a feature vector V with a shape size of (2048, 1, 1) is obtained through global average pooling. The feature vector V after global average pooling is then subjected to the normalization layer to perform L2 normalization on the features to obtain the feature vector V, where the L2 norm is the Euclidean length of the vector. After obtaining the normalized vector, the final appearance feature Z is obtained through the fully connected layer FC. Then, the cosine similarity of the appearance features of image A and image B is calculated to obtain the appearance similarity score.

[0119] After obtaining the predicted detection frame, the current detection frame and the predicted detection frame of the previous frame can be used to calculate the position clues and speed clues. The position clues use the mixed height IoU to obtain the position clues with height information, and then use the absolute difference between the trajectory speed direction θt and the trajectory to detection speed direction θd, expressed in radian format, expressed as Δθ=|θt-θd| to obtain the speed clues.

[0120] Step 3: Through three Hungarian allocations, the obtained clues are used to calculate the cost required for allocation and assign tracks to the detected objects. This includes: distinguishing high and low confidence detection boxes, tracker state estimation, strong and weak clue calculation, first data association, second data association, third data association, creating new trackers, and cleaning up unupdated trackers.

[0121] like Figure 4 As shown in the figure, when the detection value enters the tracking algorithm, the detection frame is first divided into high and low confidence detection frames according to the confidence:

[0122] τ low <conf low <τ high <conf high

[0123] Among them, τ low is the lower threshold of low confidence, τ high is the upper threshold of low confidence. low 、conf high are low confidence and high confidence respectively.

[0124] After obtaining the mixed height position cost, appearance cost and motion direction cost through calculation, the cost management module designed in this embodiment is used to properly manage the cost in Hungarian allocation. Specifically, cost management can be completed through three data associations by using cost filtering before Hungarian allocation and trajectory rejection after Hungarian allocation.

[0125] In the first association, Hungarian matching is only used for high confidence detection boxes, and the cost used is the MH-IoU position cost (C MHI o U ), appearance feature similarity cost (C A ppr), the weighted sum of the speed direction cost (C V el), and use the filtering function to filter out objects that may only overlap but are not identical before association, and use the rejection function to reject objects whose appearance feature cost is too high and whose position cost is too high after association.

[0126] During the second association, Hungarian matching is used for the remaining detection boxes. The cost used is the weighted sum of the MH-IoU position cost and the appearance feature similarity cost. After the second association, a rejection function is used to reject objects with too high appearance feature cost or too high position cost.

[0127] In the third association, Hungarian matching is used for the remaining detection boxes, and the cost used is only the MH-IoU position cost. After association, associations with too high position costs are rejected. Finally, new trackers are created for unmatched new targets, new tracks are assigned, and trackers that have not been updated for a long time are cleaned up.

[0128] In the embodiment of the present invention, in order to better deal with occlusion and utilize historical trajectories, the embodiment of the present invention designs a trajectory prediction module, which utilizes the position speed-LSTM module to predict the position and movement direction of the target in the next frame, better judges the target identity and reduces the impact of occlusion on target identity recognition; in order to reasonably utilize the information of the height of the target, the embodiment of the present invention also designs MH-IoU, which can reasonably utilize the information of the height of the target when associating objects, making the association more robust; in addition, in order to obtain the target appearance features and balance the various costs in the Hungarian allocation, the embodiment of the present invention also designs a CCM (cost control module) cost management module to balance the costs of position clues, appearance clues, trajectory clues and height clues.

[0129] In order to further verify the performance of the method proposed in the embodiment of the present invention, the multi-target tracking model constructed in the embodiment of the present invention (which includes a target detection model and a target tracking module based on the target tracking algorithm provided in the embodiment of the present invention) is used for multi-target tracking of pedestrians, wherein the target detection model adopts an action recognition model, which is trained and evaluated based on the public datasets MOT17 and MOT20. The MOT17 dataset consists of 7 video sequences, including 11,235 frames of images, 292,733 detection frames, and 1,342 trajectories. The MOT20 dataset consists of 8 video sequences, including 1,652,040 frames of images, 1,022,941 detection frames, and 3,457 trajectories.

[0130] After the method proposed in the embodiment of the present invention is tested using MOT17, the test evaluation results are shown in Table 1. After the method is tested on the MOT20 dataset, the test evaluation results are shown in Table 2. The comparison results compared with the same type of algorithms on the MOT17 dataset are shown in Table 3, and the comparison results on the MOT20 dataset are shown in Table 4. In order to verify the effectiveness of each module, the ablation experiment results of each module are shown in Table 5. Under the baseline model, the method proposed in the embodiment of the present invention has certain improvements in the indicators: HOTA and IDF1, and has achieved excellent performance. It can be seen that the multi-target tracking method based on weak clues and trajectory prediction provided by the embodiment of the present invention can track multiple targets more accurately while being efficient.

[0131] The calculation formula of the evaluation index used is as follows:

[0132]

[0133] Among them, TP represents the number of targets correctly tracked by the tracker, FN represents the number of missed targets, FP represents the number of false alarm targets, TPA(c) refers to the number of correctly associated targets under category c, FNA(c) refers to the number of true targets that are not correctly associated under category c, and FPA(c) refers to the number of incorrectly associated targets under category c.

[0134]

[0135] Among them, IDTP is the number of targets correctly identified and tracked, IDFN is the number of targets that were not correctly tracked or identified, and IDFP is the number of newly created targets by mistake.

[0136]

[0137] Among them, FN is the number of missed detections, FP is the number of false detections, IDSW is the number of identity switches, and gtDet is the object in the real box.

[0138] Table 1 MOT17 test evaluation results

[0139] sequence HOTA IDF1 MOTA TP FP FN IDSW MOT17-01 49.70 58.91 56.33 4826 1177 1624 16 MOT17-03 72.00 88.55 92.94 101295 3960 3380 49 MOT17-06 52.55 65.26 64.59 8684 1040 3100 33 MOT17-07 54.34 65.55 72.84 13067 691 3826 71 MOT17-08 49.95 58.29 64.46 14885 1136 6239 133 MOT17-12 59.93 72.78 57.22 6994 2011 1673 24 MOT17-14 50.65 68.90 58.37 11561 710 6922 63 overall 64.03 78.60 79.86 483936 32175 80292 1167

[0140] Table 2 MOT20 test evaluation results

[0141] sequence MOTA IDF1 HOTA TP FP FN IDSW MOT20-04 86.08 84.88 69.87 256263 20000 17821 328 MOT20-06 65.02 65.38 53.49 94469 7686 38288 462 MOT20-07 80.38 74.82 65.01 29473 2745 3628 120 MOT20-08 60.28 64.57 50.25 52632 5681 24852 241 overall 76.45 76.77 63.08 432837 36112 84589 1151

[0142] Table 3 Comparison results of MOT17 test evaluation

[0143]

[0144]

[0145] Table 4 Comparison results of MOT20 test evaluation

[0146]

[0147] Table 5 Ablation experiment table of each module

[0148] HM-IoU TPM ReID-CCM HOTA MOTA IDF1 FPS 67.8 76.6 79.3 9.7 √ 68.4 77.9 80.4 9.6 √ √ 69.3 78.8 82.2 6.2 √ √ √ 70.1 79.6 82.0 4.5

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0150] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A multi-target tracking method based on weak clues and trajectory prediction, characterized in that: The following steps are involved: Step 1, setting a target detection model for obtaining target detection results of frame images, and training the target detection model based on a set training image set, and obtaining a target detector based on the trained target detection model; wherein the target detection results include target categories, target detection frames and their confidence levels; Step 2: Obtain the target detection result of the frame image based on the target detector, and perform target tracking based on the target detection frame of the frame image: The target detection frame of the frame image is used as an observation and input into the tracker for state estimation. If the current frame is the first frame of the video, the initial trajectory is directly assigned to the detection target; otherwise, state estimation is performed based on the Kalman filter and the position-speed-LSTM model. When occlusion is detected, the position-speed-LSTM model is used for trajectory prediction; otherwise, the Kalman filter is used for trajectory prediction; the trajectory assignment of multiple detection targets is performed through multiple Hungarian algorithms, the detection target in the current frame is matched with the tracked target in the previous frame, and the trajectory is assigned to the detection target in the current frame to update the tracker corresponding to the detection target; and a new tracker is created for the unmatched detection target and the initial trajectory is assigned; Among them, the trajectory prediction of the position-speed-LSTM model specifically includes: in the encoding stage, the model encodes the position and speed information of the historical target detection frame into a hidden state; in the decoding stage, the speed decoder predicts the speed change of the future target detection frame of the detection target based on the encoded hidden state and the recently observed speed information, and then obtains the position and size of the future target detection frame of the detection target based on the speed change conversion output by the speed decoder, and outputs the prediction result of the target detection frame of a specified number of future frames.

2. The method according to claim 1, characterized in that Step 2 also includes: cleaning up the trackers whose update time exceeds a set threshold.

3. The method according to claim 1, characterized in that The position-speed-LSTM model includes a speed encoder, a position encoder and a speed decoder based on a long short-term memory network LSTM; The speed encoder is used to encode the speed information of the target detection frame to obtain its corresponding hidden state, and the position encoder is used to encode the position information of the target detection frame to obtain its corresponding hidden state; the two are then connected into a fused hidden state as the initial input of the speed decoder; For the speed encoder, the input data is: based on the center coordinates of the target detection frame (x t ,y t ), width Wt and height Ht to obtain the target detection box information B of the tth frame t =(x t ,y t ,W t ,H t ), and then obtain the detection frame sequence based on the target detection frame information Bt of the specified sequence length, and extract the speed information of the tth frame of the detection frame sequence based on the position and size changes Based on the speed information of each frame of the detection frame sequence Get the input data of the speed encoder; Speed ​​encoder based on and the velocity hidden state v of the previous frame t-1 Calculate the speed hidden state of the current frame in, represents the output of the LSTM-based velocity encoder, W v is the weight matrix of the velocity encoder; For the position encoder, its input data is the target detection box information B t The detection frame sequence is composed of the position encoder based on B t and the position hidden state p of the previous frame t-1 Calculate the position hidden state of the current frame in, represents the output of the LSTM-based position encoder, W p is the weight matrix of the position encoder; The speed decoder is based on the fused hidden state is the initial hidden state, and the speed information obtained from the most recent observation Perform the first prediction as input and get the next hidden state in, represents the output of the LSTM-based decoder, W dv is the weight matrix of the speed decoder; then the predicted hidden state Feed to the fully connected layer to predict the next speed information Where W ov 、b ov They are the weight matrix and bias vector of the fully connected layer respectively; perform multiple rounds of operations iteratively and save the speed information of each round of fully connected layer output The subscript t′ represents the preset number of iterations; Calculate the position and size of the future target detection frame based on the predicted speed information and the calculated corresponding target detection frame information: Among them, the number of iterations j = 1,…,t′.

4. The method according to claim 1, characterized in that When performing trajectory assignment of multiple detected targets through multiple Hungarian algorithms, the calculated assignment costs include: mixed height position cost, appearance cost, and motion direction cost.

5. The method according to claim 1, characterized in that The cubic Hungarian algorithm is used to perform trajectory assignment for multiple detection targets to achieve cubic data association for trajectory assignment: During the first data association, only the confidence level is higher than or equal to the preset first confidence threshold τ high The target detection box uses the Hungarian algorithm to assign trajectories. The allocation cost used includes the mixed height position cost, appearance cost, and motion direction cost. After the first data association is completed, the target detection frames whose appearance feature cost and position cost exceed the specified threshold are filtered out, and the Hungarian algorithm performs the second data association on the remaining unassociated target detection frames, and the allocation cost during the second data association includes the mixed height position cost and appearance cost; After the second data association is completed, the target detection frames whose appearance feature cost exceeds the specified threshold or the position cost exceeds the specified threshold are filtered out, and the Hungarian algorithm performs the third data association on the remaining unassociated target detection frames, and the allocation cost during the third data association is the mixed height position cost; After the third data association is completed, the target detection frames whose location cost exceeds the specified threshold are filtered out, so that the new target can be obtained from the remaining unassociated target detection frames.

6. The method according to claim 1, characterized in that In step 2, occlusion detection is performed using the following steps: The image features of the image content of the target detection frame are extracted as its appearance feature vector, and occlusion detection is performed based on the intersection-over-union ratio of the target detection frames of adjacent frames, the appearance feature vector, and the confidence of the target detection frame of the current frame: (1) The intersection-and-union ratio of the target detection frame of the current frame and the previous frame is greater than or equal to the preset intersection-and-union ratio threshold τ loU ; (2) The number of continuous tracking times is greater than or equal to the preset continuous tracking times threshold τ hs ; (3) The cosine similarity between the appearance feature vector i of the target detection frame in the previous frame and the appearance feature vector j of the target detection frame in the current frame Less than or equal to the preset cosine similarity threshold τ d , or the confidence of the target detection frame of the current frame is greater than or equal to the preset threshold τ conf ; When the above three conditions are met at the same time, it is determined that there is occlusion in the current frame.

7. The method according to claim 6, characterized in that A re-identification model is used to extract the appearance feature vector of the target detection box, where the re-identification model includes a feature extraction network, a global average pooling layer, a normalization layer and a fully connected layer in sequence, and the input data of the re-identification model is the image data corresponding to the target detection box.

8. The method according to claim 1, characterized in that The mixed height position cost, appearance cost and motion direction cost in the allocation cost are specifically set as: (1) Mixed height position cost The mixed height intersection-over-union ratio is calculated according to the formula MHIoU=αHMIoU+βIoU, where MHIoU is the mixed height intersection-over-union ratio, HMIoU is the height modulation intersection-over-union ratio, and IoU is the intersection-over-union ratio of the target detection frame of the current frame and the previous frame; α and β are preset weight coefficients respectively; where the height modulation intersection-over-union ratio HMIoU=HIoU·IoU, the height coefficient Respectively represent the upper left corners of the two target detection boxes, They represent the lower right corners of the two target detection boxes, and the superscripts are target detection box distinguishers; According to MHIoU, the mixed height position cost C is obtained MHIoU = -MHIoU; (2) Appearance cost Extract the appearance feature vector of the target detection frame, and use the cosine similarity between the appearance feature vector i of the target detection frame in the previous frame and the appearance feature vector j of the target detection frame in the current frame as the appearance cost C A ppr; (3) Movement direction cost The trajectory speed direction θt is obtained based on the speed direction between the center points of the two target detection frames in the previous and next frames, or based on different time intervals Δt, the speed direction between the center points of the two target detection frames with a time interval of Δt in the currently obtained trajectory is calculated, and then the trajectory speed direction θt is obtained based on the cumulative sum of the speed directions of all the set time intervals Δt; and the speed direction θd from the trajectory to the new detection frame is obtained based on the speed direction from the trajectory to the center point of the target detection frame of the current frame, and the absolute error between θt and θd is used as the motion direction cost.

9. The method according to claim 8, characterized in that Replace the movement direction cost with: Given multiple time intervals Δt, for each time interval Δt, calculate the velocity direction between the four vertices of the two target detection boxes with a time interval of Δt in the currently obtained trajectory, and then obtain the velocity direction cost of the current time interval Δt based on the average value of the velocity direction between the four vertices; then accumulate the velocity direction costs of all time intervals Δt to obtain the motion direction cost; For two target detection frames with a time interval of Δt, the velocity direction between the vertices is expressed as: Among them, (u1, v1) and (u2, v2) represent the position coordinates of a vertex of the front and rear target detection boxes in the two target detection boxes respectively.

10. The method according to claim 1, characterized in that When calculating the hybrid height position cost, based on the longitudinal velocity v y The weight coefficient α of HMIoU is adjusted in real time based on the state of Check the vertical speed v of the target detection box y Whether the speed threshold is exceeded If so, α is gradually reduced with the set step size until the longitudinal velocity v y Less than or equal to or when the value of α decreases to a preset lower limit, the adjustment is stopped; If the longitudinal velocity v y Less than speed threshold Then gradually increase α with the set step size until the longitudinal velocity v y Greater than or stop increasing the adjustment until the value of α increases to the preset upper limit.

Images