A multi-target tracking method based on weak cues and trajectory prediction

Abstract

The present invention discloses a multi-target tracking method based on weak clues and trajectory prediction, which belongs to the technical field of deep learning multi-target tracking. The present invention first sets a target detection model, and then sets a target tracking algorithm. When occlusion occurs, the position-speed-LSTM model is used to predict the future trajectory of the target and calculate the motion direction cost; when there is no occlusion, the Kalman filter is used to predict the trajectory; when calculating the position cost, the appearance feature model is used to extract the appearance features based on the designed mixed height intersection-union ratio using the altitude state as a weak clue. After obtaining each clue, the trajectory of the target is assigned using the strong and weak clues through the Hungarian algorithm. In order to balance and utilize each cost, the present invention realizes multi-target tracking through three-dimensional association based on the configured allocation cost strategy. The present invention solves the problems of occlusion, similar objects, appearance degradation, etc. to a certain extent, and enhances the model generalization ability and target tracking ability for complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning multi-target tracking, and in particular to a multi-target tracking method based on weak clues and trajectory prediction. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI), particularly the widespread application of deep learning algorithms in computer vision, multi-target tracking technology has significantly advanced. It holds significant application value in intelligent surveillance, autonomous driving, and drone navigation. Its core goal is to simultaneously track multiple targets within a video sequence and assign unique identifiers to each target. Multi-target tracking algorithms typically need to handle complex scenarios, including target occlusion, changes in the number of targets, and dynamic background changes. This places extremely high demands on the robustness and accuracy of the algorithms.

[0003] Currently, there are three mainstream models for video multi-target tracking:

[0004] The first method, detection-based multi-object tracking, involves detecting objects in each frame of a video sequence and cropping them based on bounding boxes to obtain all objects in the image. This problem is then transformed into an object association problem between the previous and next frames. A similarity matrix is ​​constructed using metrics such as Intersection over Union (IoU) and appearance, and solved using methods such as the Hungarian algorithm and the greedy algorithm.

[0005] The second method, multi-target tracking based on joint detection and tracking, adds a parallel feature extraction branch to the detection network, thereby simultaneously outputting the target's location and appearance features within a single network. By sharing features between the two tasks, some duplicate computation is effectively avoided, improving the model's tracking speed.

[0006] The third type is multi-target tracking based on an attention mechanism. This uses multiple adjacent frames as input and predicts the current position offset or appearance features based on the target's previous motion or appearance information, thereby associating the targets. Current algorithms based on the JDT paradigm are mainly divided into methods based on Siamese networks and Transformer-based methods.

[0007] The common drawbacks of the above existing video multi-target tracking methods are mainly reflected in:

[0008] (1) During the tracking process, weak clues such as height feature clues, historical trajectory clues, and motion speed clues are not properly utilized when associating data.

[0009] (2) Appearance degradation and target loss caused by occlusion often lead to identity exchange and trajectory fragmentation, making data association more difficult.

[0010] (3) When the costs of strong cues such as position cues and appearance cues and weak cues such as trajectory cues, height cues, and motion speed cues are introduced, it is difficult to balance the various costs. Summary of the Invention

[0011] The present invention provides a multi-target tracking method based on weak cues and trajectory prediction, aiming to solve the problems of existing target tracking methods such as insufficient utilization of weak cues, association failure caused by occlusion, and difficulty in balancing multiple association costs.

[0012] The technical solution adopted in the present invention is:

[0013] A multi-target tracking method based on weak clues and trajectory prediction, the method comprising the following steps:

[0014] Step 1: Set a target detection model for obtaining target detection results of frame images, train the target detection model based on a set training image set, and obtain a target detector based on the trained target detection model; wherein the target detection results include target category, target detection box (target location and bounding box) and its confidence;

[0015] Step 2: Obtain the target detection result of the frame image based on the target detector, and perform target tracking based on the target detection frame of the frame image:

[0016] The target detection frame of the frame image is used as an observation and input into the tracker for state estimation. If the current frame is the first frame of the video, the initial trajectory is directly assigned to the detection target; otherwise, state estimation is performed based on the Kalman filter and the position-velocity-LSTM model. When occlusion is detected, the position-velocity-LSTM model is used for trajectory prediction; otherwise, the Kalman filter is used for trajectory prediction; the trajectory assignment of multiple detection targets is performed through multiple Hungarian algorithms, matching the detection target in the current frame with the tracked target in the previous frame, and assigning a trajectory to the detection target in the current frame to achieve an update of the detection result of successful data association, that is, updating the tracker corresponding to the detection target; and creating a new tracker and assigning an initial trajectory to the unmatched detection target (new target). In order to further reduce unnecessary resource usage, trackers whose update time exceeds the set threshold can also be cleaned up;

[0017] Among them, the position-speed-LSTM model performs trajectory prediction specifically as follows: in the encoding stage, the model encodes the position and speed information of the historical target detection frame into a hidden state; in the decoding stage, the speed decoder predicts the speed change of the future target detection frame of the detection target based on the encoded hidden state and the recently observed speed information, and then calculates the position and size of the future target detection frame of the detection target based on the speed change conversion output by the speed decoder, and outputs the prediction result of the target detection frame for a specified number of frames in the future.

[0018] Furthermore, the object detection model includes: backbone network, feature pyramid network, decoupling head, anchor-free detection head and classification and regression module;

[0019] The backbone network is used to extract multi-scale image features of the input image, that is, based on the input image, extract the underlying features of the image at multiple scales; preferably, Modified CSP v5 can be used as the backbone network, so as to effectively extract the spatial features and semantic information of the input image;

[0020] Furthermore, the Modified CSP v5 backbone network consists of a CSP Bottleneck, Darknet53, cross-stage partial connections, downsampling layers, and convolutional layers with residual connections. The CSP Bottleneck splits the feature map into two parts, one of which is processed by multiple convolutional layers while the other remains unchanged. The outputs of the two parts are then merged. This design allows deeper networks to learn features more efficiently. Darknet53 contains multiple convolutional layers and residual blocks. Residual blocks use skip connections to prevent gradient vanishing and accelerate training, allowing the network to maintain good performance at deeper layers. In the cross-stage partial connection network, the input feature map is split into two parts, one of which undergoes a single convolutional stage while the other remains unchanged. The two feature maps are then merged. This effectively reduces the network's computational complexity and improves training efficiency. The downsampling layer achieves downsampling through convolutional layers and pooling layers (such as max pooling). These layers reduce the size of the input feature map while preserving important feature information. Residual connection convolution layer, each residual block consists of multiple convolution layers and a skip connection, which directly adds the input to the convolution output.

[0021] The feature pyramid network takes the image features extracted by the backbone network as input, and is used to enhance and fuse the multi-scale image features extracted by the backbone network, outputting enhanced feature maps of three different scales, thereby improving the detection performance of targets of different sizes.

[0022] Decoupling head: For the FPN features at each scale level, a 1×1 convolutional layer is first used to reduce the feature channels to the target number of channels, such as 256. Then two parallel branches are added, each with two 3×3 convolutional layers for classification and regression tasks respectively. The IoU branch is added to the regression branch.

[0023] The anchor-free detection head means that the model no longer relies on predefined anchor boxes for object localization. Instead, the detection head directly regresses the object's bounding box (i.e., object detection box) location and classification results from the feature map (i.e., the feature map output by the regression task branch of the decoupled head). This design simplifies the model and reduces the complexity of hyperparameter tuning.

[0024] The classification and regression module further processes the features output by the detection head to achieve object classification and bounding box regression. By decoupling the head design, the classification and regression tasks are separated to improve detection performance.

[0025] Furthermore, the target detection model is trained based on the set training image set as follows: deep learning training is performed on the target detection model based on the training image set, and when the preset convergence conditions are met (the number of training times reaches the upper limit or the set loss function converges), a target detector for each frame image in the detection video sequence for multi-target tracking is obtained; and the total loss function used in the deep learning training of the target detection model is the weighted sum of the prediction box regression loss function, the classification loss function and the confidence loss function.

[0026] Furthermore, the prediction box regression function is specifically set as:

[0027] L reg =1-IoU

[0028] Among them, L reg Represents the prediction box regression function, and IoU is the intersection over union ratio of the real box and the prediction box (i.e., the target detection box output by the model).

[0029] The classification loss function and confidence loss function are specifically set as:

[0030]

[0031] Among them, L obj represents the classification loss function, L obj represents the confidence loss function, N is the total number of training samples, y i is the true label of the i-th training sample, which takes a value of 0 or 1. is the predicted value of the i-th training sample, is the Sigmoid function, defined as:

[0032] Furthermore, the expression of the total loss function used in deep learning training of the target detection model is:

[0033] L=λ reg L reg +λ cls L cls +λobj L obj

[0034] Among them, L represents the total loss function, λ reg ,λ cls ,λ obj They are the weights of the prediction box regression loss function, the classification loss function, and the confidence loss function, respectively.

[0035] Furthermore, the position-speed-LSTM model includes a speed encoder, a position encoder, and a speed decoder based on an LSTM network (long short-term memory network);

[0036] The speed encoder is used to encode the speed information of the target detection frame to obtain its corresponding hidden state, and the position encoder is used to encode the position information of the target detection frame to obtain its corresponding hidden state. The two are then connected to form a fused hidden state as the initial input of the speed decoder.

[0037] For the speed encoder, the input data is: based on the center coordinates of the target detection frame (x t ,y t ), width W t and height H t Get the target detection box information B of the tth frame t =(x t ,y t ,W t ,H t ), and then based on the target detection frame information B of the specified sequence length t Get the detection frame sequence and extract the speed information of the tth frame of the detection frame sequence based on the position and size changes Based on the speed information of each frame of the detection frame sequence Get the input data of the speed encoder;

[0038] Speed ​​encoder based on and the velocity hidden state v of the previous frame t-1 Calculate the speed hidden state of the current frame in, represents the output of the LSTM-based velocity encoder, W v is the weight matrix of the velocity encoder, which is a trainable parameter;

[0039] For the position encoder, its input data is the target detection box information B t The detection frame sequence is composed of the position encoder based on B t and the position hidden state p of the previous frame t-1 Calculate the position hidden state of the current frame in, represents the output of the LSTM-based position encoder, W p is the weight matrix of the position encoder, which is a trainable parameter;

[0040] The speed decoder is based on the fused hidden state is the initial hidden state, and the speed information v obtained by the most recent observation Bt Perform the first prediction as input and get the next hidden state in, represents the output of the LSTM-based decoder, W dv is the weight matrix of the speed decoder, which can be trained parameters; then the predicted hidden state Feed to the fully connected layer to predict the next speed information Where W ov 、b ov They are the weight matrix and bias vector of the output layer (fully connected layer), both of which are trainable parameters. Multiple rounds of operations are performed iteratively to save the speed information of each round of fully connected layer output. The subscript t′ represents the preset number of iterations;

[0041] Finally, the position and size of the future target detection frame are calculated based on the predicted speed information and the calculated corresponding target detection frame information: Wherein, the number of iterations j = 1,…,t′.

[0042] Furthermore, when performing trajectory assignment of multiple detection targets through multiple Hungarian algorithms, the calculated assignment costs include: mixed height position cost, appearance cost and motion direction cost.

[0043] Furthermore, the cubic Hungarian algorithm is used to perform trajectory assignment of multiple detection targets to achieve cubic data association of trajectory assignment, specifically:

[0044] During the first data association, only the confidence level is higher than or equal to the preset first confidence threshold τ. high The target detection box uses the Hungarian algorithm to assign trajectories. The allocation cost used includes the mixed height position cost, appearance cost, and motion direction cost.

[0045] After the first data association is completed, the target detection frames whose appearance feature cost and position cost exceed the specified threshold are filtered out, and the Hungarian algorithm performs a second data association on the remaining unassociated target detection frames. The allocation cost during the second data association includes the mixed height position cost and appearance cost.

[0046] After the second data association is completed, the target detection frames whose appearance feature cost exceeds the specified threshold or the position cost exceeds the specified threshold are filtered out, and the Hungarian algorithm performs the third data association on the remaining unassociated target detection frames. The allocation cost during the third data association is the mixed height position cost.

[0047] After the third data association is completed, the target detection frames whose location cost exceeds the specified threshold are filtered out, so that the new target can be obtained from the remaining unassociated target detection frames.

[0048] Furthermore, in step 2, occlusion detection is performed using the following steps:

[0049] The image features of the image content of the target detection frame are extracted as its appearance feature vector, and occlusion detection is performed based on the intersection-over-union ratio of the target detection frames of adjacent frames, the appearance feature vector, and the confidence of the target detection frame of the current frame:

[0050] (1) The intersection-and-union ratio of the target detection frame of the current frame and the previous frame is greater than or equal to the preset intersection-and-union ratio threshold τ loU ;

[0051] (2) The number of continuous tracking times is greater than or equal to the preset continuous tracking times threshold τ hs ;

[0052] (3) The cosine similarity between the appearance feature vector i of the target detection frame of the previous frame and the appearance feature vector j of the target detection frame of the current frame Less than or equal to the preset cosine similarity threshold τ d , or the confidence of the target detection frame of the current frame is greater than or equal to the preset threshold τ conf ;

[0053] When the above three conditions are met at the same time, it is determined that there is occlusion in the current frame.

[0054] Furthermore, a re-identification model can be used to extract the appearance feature vector of the target detection frame, wherein the re-identification model sequentially includes a feature extraction network (for example, a ResNet-50 backbone network is used as the feature extraction network), a global average pooling layer, a normalization layer, and a fully connected layer. The input data of the re-identification model is the image data corresponding to the target detection frame. Then, based on the cosine similarity between the extracted appearance feature vectors of the target detection frame of the same target in the previous and next frames, the metric value of the re-identification model for the target detection frame, i.e., the appearance cost, can be obtained.

[0055] Furthermore, the mixed height position cost, appearance cost and motion direction cost in the allocation cost are specifically set as:

[0056] (1) Mixed height position cost

[0057] The mixed height intersection over union (IoU) is calculated according to the formula MHIoU = αHMIoU + βIoU, where MHIoU is the mixed height intersection over union, HMIoU is the height modulated intersection over union, and IoU is the intersection over union of the target detection box between the current frame and the previous frame; α and β are preset weight coefficients respectively.

[0058] in,

[0059] A and B are the areas of the two target detection boxes;

[0060] The calculation expression of the highly modulated intersection-over-union ratio HMIoU is:

[0061]

[0062] HMIoU=HIoU·IoU

[0063] Among them, HIoU is the height coefficient, and the two target detection boxes are defined as and Where x1 and y1 represent the upper left corner, and x2 and y2 represent the lower right corner.

[0064] The mixed height position cost C is obtained according to the mixed height intersection-union ratio MHIoU MHIoU = -MHIoU;

[0065] (2) Appearance cost

[0066] The re-identification model is used to extract the appearance feature vector of the target detection frame, and the cosine similarity between the appearance feature vector i of the target detection frame of the previous frame and the appearance feature vector j of the target detection frame of the current frame is used as the appearance cost C A ppr, namely the appearance feature similarity cost, is expressed as:

[0067] (3) Movement direction cost (also known as trajectory direction cost or speed direction cost)

[0068] The velocity direction is calculated using the four corners of the detection box. Given two points (u1, v1) and (u2, v2), the velocity direction is calculated as follows.

[0069]

[0070] In this paper, the cost metric used is the absolute difference between the trajectory velocity direction θt and the trajectory-to-new detection box velocity direction θd, expressed in radians, as Δθ = |θt - θd|. The trajectory velocity direction is obtained from the centers of two detection boxes in the trajectory at a time interval Δt, and the trajectory-to-new detection box velocity direction is obtained from the centers of the trajectory's previous detection box and the new detection box.

[0071] The expression for the velocity direction cost is:

[0072]

[0073] Among them, C Vel is the total cost in the velocity direction, The velocity direction costs are the upper left corner, upper right corner, lower left corner, and lower right corner respectively.

[0074] The trajectory speed direction θt is obtained based on the speed direction between the center points of the two target detection frames of the two frames before and after, or based on different time intervals Δt, the speed direction between the center points of the two target detection frames with a time interval of Δt in the currently obtained trajectory is calculated, and then the trajectory speed direction θt is obtained based on the cumulative sum of the speed directions of all the set time intervals Δt; and the speed direction θd of the trajectory to the new detection frame is obtained based on the speed direction between the trajectory and the center point of the target detection frame of the current frame, and the absolute error between θt and θd is used as the motion direction cost; the motion direction cost can also be obtained based on the cumulative sum of the average values ​​of the speed direction θ between the four vertices of the target detection frames of the two frames with a specified interval before and after, that is, the total speed direction cost C V el.

[0075] In this application, when performing trajectory allocation for multiple detection targets based on the Hungarian algorithm, the allocation cost considered includes strong and weak clues. Generally, position clues, appearance clues, etc. can be regarded as strong clues, and trajectory clues, height clues, and motion speed clues can be regarded as weak clues. After obtaining the mixed height position cost, appearance cost, and motion direction cost, the cost is properly managed based on the Hungarian allocation based on the cost management strategy involved. This is specifically reflected in the use of cost filtering before Hungarian allocation and trajectory rejection after Hungarian allocation. Cost management can be completed through three data associations.

[0076] Furthermore, when the target moves in the x-direction, the target height changes slightly due to the camera perspective principle. When the target moves in the y-direction, the object height changes significantly. To address this situation, the weight coefficient α of the HMIoU in the hybrid height position cost can be further adjusted as follows:

[0077]

[0078] Among them, α′ represents the weight coefficient of the adjusted HMIoU, that is, when the longitudinal speed v y Too large (exceeding the preset threshold ), the HMIoU coefficient α is gradually reduced by the preset adjustment step Δα. Each time α is reduced (α=α-Δα), the longitudinal velocity v yIs it back to normal? When it is detected that the longitudinal speed vy returns to normal, the reduction adjustment is stopped; or even if the longitudinal speed v y If the current longitudinal speed is still too large, the adjustment will stop when the coefficient α decreases to 0. When the coefficient α is gradually increased by the set adjustment step, each time α is increased (α=α+Δα), the longitudinal velocity v y Is it too large? When it is too large, stop increasing the adjustment, or when it is detected that the value of the coefficient α increases to the preset upper limit α max Stop increasing the adjustment when the longitudinal speed v y The state of the coefficient α is adjusted in real time: check the longitudinal speed v of the target detection frame y Whether it exceeds If so, α is gradually reduced with the set step size until the longitudinal velocity v y Less than or equal to Stop reducing the adjustment when the value of α is reduced to a preset lower limit (preferably set to 0); if v y Less than Then gradually increase α with the set step size until the longitudinal velocity v y Greater than Stop increasing the adjustment when α reaches the preset upper limit α. max Stop increasing adjustments.

[0079] During the three data association processes, the present invention adopts a specific cost management strategy to balance the cost of each clue:

[0080] In the first data association, only high confidence detection boxes (i.e., those with confidence higher than or equal to the preset first confidence threshold τ) are included. high The target detection box) uses Hungarian matching, that is, the expression of the allocation cost used in the first data association is:

[0081] C1=C MHIoU +λ1C Vel +λ2C Appr

[0082] Among them, C1 is the allocation cost of the first data association, λ1 is the speed cost coefficient, and λ2 is the appearance feature cost coefficient, all of which are preset values;

[0083] The expression of the allocation cost during the second data association is: C2=C MHIoU +λ3C Appr ;

[0084] Among them, C2 is the allocation cost of the second data association, and λ3 is the appearance feature cost coefficient during the second data association;

[0085] Correspondingly, the allocation cost used in the third data association can be directly expressed as: C3 = C MHIoU .

[0086] Furthermore, a filtering function is used to solve the problem that when the MH-IOU score is significantly higher than the appearance similarity score between two different objects, they may only overlap but not be identical. Its expression is as follows:

[0087]

[0088] Where C is the cost, τ1 is C MHIoU threshold, τ2 is the appearance cost threshold.

[0089] Furthermore, in the three data association processes, the filtering strategy for the target detection frame after each data association is completed is specifically set as follows:

[0090] After the first data association is completed, the filtering strategy for the target detection frame is: if the target detection frame meets C MHIoU >τ3 and C Appr >τ4, the current target detection box is set as the filtering object, where τ3 is the rejection threshold of the MHIoU cost of the first data association, and τ4 is the rejection threshold of the appearance feature cost of the first data association;

[0091] After the second data association is completed, the filtering strategy for the target detection frame is: if the target detection frame meets C MHIoU >τ5 or C Appr >τ6, the current target detection box is set as the filtering object, where τ5 is the rejection threshold of the MHIoU cost of the second data association, and τ6 is the rejection threshold of the appearance feature cost of the second data association;

[0092] After the third data association is completed, the filtering strategy for the target detection frame is: if the target detection frame meets C MHIoU >τ7, the current target detection box is set as the filtering object, where τ7 is the rejection threshold of the MHIoU cost of the third data association.

[0093] The present invention completes the trajectory allocation of detected targets and realizes multi-target tracking through the above three-time data association trajectory allocation.

[0094] The technical solution provided by the present invention brings at least the following beneficial effects:

[0095] The present invention can perform efficient multi-target tracking after inputting the video frame, and obtain the target position and target tracking results. Based on the existing multi-target tracking model based on deep learning, the present invention introduces a trajectory prediction model, and uses the position speed-LSTM to predict the position and movement direction of the target in the next frame, which can better judge the target identity and reduce the impact of occlusion on target identity recognition. The introduction of MH-IoU can reasonably utilize the target height information when associating objects, making the association more robust. The ReID-CCM (Cost Control Module) is introduced to obtain the target appearance features, and the CCM cost management strategy is designed to balance the costs of position clues, appearance clues, trajectory clues and height clues. It achieves the characteristics of simplicity, online and real-time, and improves the performance of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0097] Figure 1 A schematic diagram of the network architecture of a multi-target tracking method based on weak cues and trajectory prediction provided by an embodiment of the present invention;

[0098] Figure 2 This is a diagram of the position-speed-LSTM trajectory prediction network architecture used in an embodiment of the present invention.

[0099] Figure 3 This is a structural diagram of the re-identification module used in an embodiment of the present invention.

[0100] Figure 4 This is a data association flow chart used in an embodiment of the present invention. DETAILED DESCRIPTION

[0101] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0102] Multi-object tracking (MOT) in video is a computer vision technique designed to simultaneously detect and track multiple objects in a video. Its core task is to assign a unique identifier to each object and maintain the continuity of the objects across frames of the video. MOT technology typically involves two key steps: object detection and data association. Object detection identifies the objects in each frame, while data association matches the detected objects with previously tracked objects in previous frames to ensure the continuity of the object's trajectory.

[0103] Multi-target tracking technology is crucial in many fields. For example, in intelligent transportation systems, it can be used to monitor vehicles and pedestrians on the road, helping to analyze traffic flow and prevent accidents. In security monitoring, it can be used to track suspicious individuals and improve public safety. In sports analysis, it can track athlete performance and provide data support for coaches and analysts. Furthermore, multi-target tracking has widespread applications in autonomous driving, smart retail, drone monitoring, and other fields. The research and development of multi-target tracking technology will help enhance the intelligence and application breadth of machine vision systems.

[0104] To address the problems of appearance degradation and target loss caused by occlusion, which often result in identity swapping and trajectory fragmentation, as well as insufficient utilization of weak cues such as height and speed, in existing multi-target tracking methods based on deep learning recognition, the present invention provides a multi-target tracking method based on weak cues and trajectory prediction. The method is based on the SORT (Simple Online and Realtime Tracking) paradigm, performs tracking based on detection, and makes improvements in trajectory prediction after occlusion, utilization of weak cues such as height and speed, and cost balancing of strong and weak cues, thereby achieving simplicity, onlineness, and real-timeness, and improving the performance of multi-target tracking.

[0105] like Figure 1 As shown, in one embodiment, the present invention provides a multi-target tracking method based on weak clues and trajectory prediction, which specifically includes the following steps:

[0106] Step 1: Build a target detection model to obtain target detection information of the frame image (i.e., the position, size, confidence, and target category of the target detection box (also known as bounding box, detection box, predicted detection box, etc.)):

[0107] Specifically, in this embodiment, Modified CSP v5 is used as the backbone network. The backbone network used consists of multiple layers of basic convolutional layers, residual layers, and a spatial pyramid pooling module, and each layer extracts and processes features step by step. The input is an RGB image with a shape of 3×H×W. Among them, "3" is the image RGB channel, "H" is the height of the image, and "W" is the width of the image. After the basic convolutional layer, the residual layer and a spatial pyramid pooling module, the FPN feature maps of the shapes are: (256, H / 8, W / 8), (512, H / 16, W / 16), and (1024, H / 32, W / 32).

[0108] The resulting FPN feature map then enters the decoupling head. For each level of FPN features, a 1×1 convolutional layer is first used to reduce the feature channels to 256. Two parallel branches are then added, each with two 3×3 convolutional layers, for classification and regression respectively. The IoU branch is added to the regression branch. The resulting predictions are of shape C×H×W for classification, 4×H×W for regression, and 1×H×W for IoU.

[0109] Step 2: Target tracking is performed using the SORT paradigm target tracking algorithm, which includes:

[0110] The target detection frame obtained by the target detection model is input into the tracking model. If this frame is the first frame of the video, the initial trajectory is directly assigned to the detected object. Otherwise, calculations are started including position and height clues, appearance clues, trajectory clues, and speed clues.

[0111] like Figure 2 As shown, the trajectory prediction model calculates the target's trajectory cues. When tracking is good, the Kalman filter continues to be used for prediction. However, when occlusion occurs, the trajectory prediction module uses the position-velocity LSTM to perform trajectory prediction. Within the Kalman filter prediction detection framework, the state transition matrix F is used to predict the target state at the next time step. Specifically, the target's position is predicted based on the current state and velocity, and the uncertainty of the state at the next time step is predicted using covariance prediction.

[0112] The trajectory prediction in this embodiment can be expressed as:

[0113]

[0114] in, is the predicted detection box, B t is the detection frame at time t, KF() is the Kalman filter, PV-LSTM() is based on the trajectory prediction of position velocity-LSTM, f oIs whether occlusion occurs, if the value is 1, it means occlusion occurs, if the value is 0, it means no occlusion. Further, the judgment expression of occlusion perception in this embodiment can be expressed as follows:

[0115]

[0116] in is the cosine similarity between the appearance feature vector i of the trajectory and the appearance feature vector j of the new detection box. conf is the confidence level, hs is the number of consecutive tracking times, τ loU , τ hs , τ conf , τ d They are the preset IoU threshold, the threshold of continuous tracking times, the confidence threshold and the cosine similarity threshold.

[0117] In this embodiment, trajectory prediction based on position-velocity LSTM is specifically as follows: During the encoding phase, the model (the trajectory pre-model based on position-velocity LSTM) encodes the position and velocity information of the historical bounding box into hidden states and selects to use the trajectory of the historical 8 frames. During the decoding phase, the velocity decoder predicts the velocity changes of the object's future detection box based on the encoded hidden states and the last observed velocity information. These velocity predictions are then converted into the position and size of the future detection box through cumulative addition updates. The detection box of the next 8 frames is selected for output.

[0118] like Figure 3 As shown in the figure, appearance cues are calculated and the FastReID re-identification model is used for appearance feature extraction. The detected target image with a shape of 3×H×W is first passed through the ResNet-50 backbone network for feature extraction. After passing through four residual layers, a feature map with a shape of (2048, H / 32, W / 32) is obtained. Then, a feature vector V with a shape of (2048, 1, 1) is obtained through global average pooling. The feature vector V after global average pooling is then passed through a normalization layer to perform L2 normalization on the features to obtain the feature vector V, where the L2 norm is the Euclidean length of the vector. The obtained normalized vector is then passed through the fully connected layer FC to obtain the final appearance feature Z. The cosine similarity of the appearance features of image A and image B is then calculated to obtain the appearance similarity score.

[0119] After obtaining the predicted detection frame, the current detection frame and the predicted detection frame of the previous frame can be used to calculate position cues and velocity cues. The position cues use mixed height IoU to obtain position cues with height information. The absolute difference between the trajectory velocity direction θt and the trajectory-to-detection velocity direction θd, expressed in radians as Δθ = |θt - θd|, is then used to obtain velocity cues.

[0120] Step 3 uses the cues obtained through three Hungarian allocations to calculate the allocation cost and assign tracks to the detected objects. This includes distinguishing high- and low-confidence detection boxes, estimating the tracker state, calculating strong and weak cues, performing the first, second, and third data associations, creating a new tracker, and cleaning up unupdated trackers.

[0121] like Figure 4 As shown in the figure, when the detection value enters the tracking algorithm, the detection frame is first divided into high-confidence and low-confidence detection frames according to the confidence level:

[0122] τ low <conf low <τ high <conf high

[0123] Among them, τ low is the lower threshold of low confidence, τ high is the upper threshold of low confidence. low 、conf high are low confidence and high confidence respectively.

[0124] After calculating the mixed height position cost, appearance cost, and motion direction cost, the cost management module designed in this embodiment is then used to properly manage these costs during Hungarian allocation. Specifically, cost management is achieved through cost filtering before Hungarian allocation and trajectory rejection after Hungarian allocation, using three data associations.

[0125] In the first association, Hungarian matching is only used for high confidence detection boxes, and the cost used is the MH-IoU position cost (C MHI o U ), appearance feature similarity cost (C A ppr), the weighted sum of the speed direction cost (C V el), and use the filtering function to filter out objects that may only overlap but are not the same before association, and use the rejection function to reject objects whose appearance feature cost is too high and whose position cost is too high after association.

[0126] During the second association, Hungarian matching is used for the remaining detection boxes. The cost used is the weighted sum of the MH-IoU position cost and the appearance feature similarity cost. After the second association, a rejection function is used to reject objects with too high appearance feature cost or too high position cost.

[0127] In the third association, Hungarian matching is applied to the remaining detection boxes, using only the MH-IoU position cost. After association, associations with excessively high position costs are rejected. Finally, new trackers are created for new unmatched objects, new tracks are assigned, and trackers that have not been updated for a long time are cleaned up.

[0128] In an embodiment of the present invention, in order to better cope with occlusion and utilize historical trajectories, a trajectory prediction module is designed in the embodiment of the present invention. The module uses a position-speed-LSTM module to predict the position and motion direction of the target in the next frame, thereby better judging the target identity and reducing the impact of occlusion on target identity recognition. In order to rationally utilize the target height information, the embodiment of the present invention also designs MH-IoU, which can rationally utilize the target height information when associating objects, making the association more robust. In addition, in order to obtain the target appearance features and balance the various costs in Hungarian allocation, the embodiment of the present invention also designs a CCM (cost control module) cost management module to balance the costs of position cues, appearance cues, trajectory cues, and height cues.

[0129] To further verify the performance of the method proposed in the embodiment of the present invention, the multi-target tracking model constructed in the embodiment of the present invention (which includes a target detection model and a target tracking module based on the target tracking algorithm provided in the embodiment of the present invention) is used for multi-target tracking of pedestrians. The target detection model adopts an action recognition model, which is trained and evaluated based on the public datasets MOT17 and MOT20. The MOT17 dataset consists of 7 video sequences, including 11,235 frames of images, 292,733 detection boxes, and 1,342 trajectories. The MOT20 dataset consists of 8 video sequences, including 1,652,040 frames of images, 1,022,941 detection boxes, and 3,457 trajectories.

[0130] After the method proposed in the embodiment of the present invention was tested using MOT17, the test evaluation results are shown in Table 1. After the method was tested on the MOT20 dataset, the test evaluation results are shown in Table 2. The comparison results with the same type of algorithms on the MOT17 dataset are shown in Table 3, and the comparison results on the MOT20 dataset are shown in Table 4. In order to verify the effectiveness of each module, the ablation experiment results of each module are shown in Table 5. Under the baseline model, the method proposed in the embodiment of the present invention has certain improvements in the indicators: HOTA and IDF1, and has achieved excellent performance. It can be seen that the multi-target tracking method based on weak clues and trajectory prediction provided by the embodiment of the present invention can track multiple targets more accurately while being efficient.

[0131] The calculation formula of the evaluation index used is as follows:

[0132]

[0133] Where TP represents the number of targets correctly tracked by the tracker, FN represents the number of missed targets, FP represents the number of false alarm targets, TPA(c) refers to the number of correctly associated targets under category c, FNA(c) refers to the number of true targets that were not correctly associated under category c, and FPA(c) refers to the number of incorrectly associated targets under category c.

[0134]

[0135] Among them, IDTP is the number of targets correctly identified and tracked, IDFN is the number of targets not correctly tracked or identified, and IDFP is the number of newly created targets by mistake.

[0136]

[0137] Among them, FN is the number of missed detections, FP is the number of false detections, IDSW is the number of identity switches, and gtDet is the object in the real box.

[0138] Table 1 MOT17 test evaluation results

[0139] sequence HOTA IDF1 MOTA TP FP FN IDSW MOT17-01 49.70 58.91 56.33 4826 1177 1624 16 MOT17-03 72.00 88.55 92.94 101295 3960 3380 49 MOT17-06 52.55 65.26 64.59 8684 1040 3100 33 MOT17-07 54.34 65.55 72.84 13067 691 3826 71 MOT17-08 49.95 58.29 64.46 14885 1136 6239 133 MOT17-12 59.93 72.78 57.22 6994 2011 1673 24 MOT17-14 50.65 68.90 58.37 11561 710 6922 63 overall 64.03 78.60 79.86 483936 32175 80292 1167

[0140] Table 2 MOT20 test evaluation results

[0141] sequence MOTA IDF1 HOTA TP FP FN IDSW MOT20-04 86.08 84.88 69.87 256263 20000 17821 328 MOT20-06 65.02 65.38 53.49 94469 7686 38288 462 MOT20-07 80.38 74.82 65.01 29473 2745 3628 120 MOT20-08 60.28 64.57 50.25 52632 5681 24852 241 overall 76.45 76.77 63.08 432837 36112 84589 1151

[0142] Table 3 Comparison results of MOT17 test evaluation

[0143]

[0144]

[0145] Table 4 Comparison results of MOT20 test evaluation

[0146]

[0147] Table 5 Ablation experiment table of each module

[0148] HM-IoU TPM ReID-CCM HOTA MOTA IDF1 FPS 67.8 76.6 79.3 9.7 √ 68.4 77.9 80.4 9.6 √ √ 69.3 78.8 82.2 6.2 √ √ √ 70.1 79.6 82.0 4.5

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0150] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A multi-target tracking method based on weak clues and trajectory prediction, characterized in that: The following steps are involved: Step 1: Setting a target detection model for obtaining target detection results of frame images, training the target detection model based on a set of training images, and obtaining a target detector based on the trained target detection model; wherein the target detection results include target categories, target detection boxes, and their confidence levels; Step 2: Obtain the target detection result of the frame image based on the target detector, and perform target tracking based on the target detection frame of the frame image: The target detection frame of the frame image is used as an observation and input into the tracker for state estimation. If the current frame is the first frame of the video, the initial trajectory is directly assigned to the detection target; otherwise, state estimation is performed based on the Kalman filter and the position-velocity-LSTM model. When occlusion is detected, the position-velocity-LSTM model is used for trajectory prediction; otherwise, the Kalman filter is used for trajectory prediction; the trajectory assignment of multiple detection targets is performed through multiple Hungarian algorithms, matching the detection target in the current frame with the tracked target in the previous frame, and assigning a trajectory to the detection target in the current frame to update the tracker corresponding to the detection target; and creating a new tracker and assigning an initial trajectory to the unmatched detection target; The position-speed-LSTM model performs trajectory prediction in the following ways: in the encoding phase, the model encodes the position and speed information of the historical target detection frame into a hidden state; in the decoding phase, the speed decoder predicts the speed change of the future target detection frame based on the encoded hidden state and the recently observed speed information. The speed change output by the speed decoder is then used to convert the position and size of the future target detection frame of the detected target, and the prediction result of the target detection frame for a specified number of frames in the future is output; Among them, when performing trajectory assignment of multiple detection targets through multiple Hungarian algorithms, the calculated assignment cost includes: mixed height position cost, appearance cost and motion direction cost; The cubic Hungarian algorithm is used to perform trajectory assignment for multiple detection targets to achieve cubic data association for trajectory assignment: During the first data association, only the data with confidence higher than or equal to the preset first confidence threshold are The target detection box uses the Hungarian algorithm to assign trajectories. The allocation cost used includes the mixed height position cost, appearance cost, and motion direction cost. After the first data association is completed, the target detection frames whose appearance feature cost and position cost exceed the specified threshold are filtered out, and the Hungarian algorithm performs a second data association on the remaining unassociated target detection frames. The allocation cost during the second data association includes the mixed height position cost and appearance cost. After the second data association is completed, the target detection frames whose appearance feature cost exceeds the specified threshold or the position cost exceeds the specified threshold are filtered out, and the Hungarian algorithm performs the third data association on the remaining unassociated target detection frames. The allocation cost during the third data association is the mixed height position cost. After the third data association is completed, the target detection boxes whose location costs exceed the specified threshold are filtered out.

2. The method according to claim 1, wherein Step 2 also includes: cleaning up the trackers whose update time exceeds a set threshold.

3. The method according to claim 1, wherein The position-speed-LSTM model includes a speed encoder, a position encoder, and a speed decoder based on the long short-term memory network LSTM; The speed encoder is used to encode the speed information of the target detection frame to obtain its corresponding hidden state, and the position encoder is used to encode the position information of the target detection frame to obtain its corresponding hidden state. The two are then connected to form a fused hidden state as the initial input of the speed decoder. For the speed encoder, the input data is: based on the center coordinates of the target detection frame ,width and height Get the target detection box information of the tth frame , and then based on the target detection box information of the specified sequence length Get the detection frame sequence and extract the speed information of the tth frame of the detection frame sequence based on the position and size changes ; Then based on the speed information of each frame of the detection frame sequence Get the input data of the speed encoder; Speed ​​encoder based on and the speed hidden state of the previous frame Calculate the speed hidden state of the current frame ,in, represents the output of the LSTM-based velocity encoder, is the weight matrix of the velocity encoder; For the position encoder, its input data is the target detection box information The detection frame sequence is composed of the position encoder based on and the position hidden state of the previous frame Calculate the position hidden state of the current frame ,in, represents the output of the LSTM-based positional encoder, is the weight matrix of the position encoder; The speed decoder is based on the fused hidden state is the initial hidden state, and the speed information obtained from the most recent observation Perform the first prediction as input and get the next hidden state ,in, represents the output of the LSTM-based decoder, is the weight matrix of the speed decoder; then the predicted hidden state Feed to the fully connected layer to predict the next speed information ,in 、 They are the weight matrix and bias vector of the fully connected layer respectively; perform multiple rounds of operations iteratively and save the speed information of the output of each round of the fully connected layer , where the subscript Indicates the preset number of iteration rounds; Calculate the position and size of the future target detection frame based on the predicted speed information and the calculated corresponding target detection frame information: , where the number of iterations .

4. The method according to claim 1, wherein In step 2, occlusion detection is performed using the following steps: The image features of the image content of the target detection frame are extracted as its appearance feature vector, and occlusion detection is performed based on the intersection-over-union ratio of the target detection frames of adjacent frames, the appearance feature vector, and the confidence of the target detection frame of the current frame: (1) The intersection-and-union ratio of the target detection frame of the current frame and the previous frame is greater than or equal to the preset intersection-and-union ratio threshold ; (2) The number of continuous tracking times is greater than or equal to the preset continuous tracking times threshold ; (3) The cosine similarity between the appearance feature vector i of the target detection frame of the previous frame and the appearance feature vector j of the target detection frame of the current frame Less than or equal to the preset cosine similarity threshold , or the confidence of the target detection frame of the current frame is greater than or equal to the preset threshold ; When the above three conditions are met at the same time, it is determined that there is occlusion in the current frame.

5. The method according to claim 4, wherein A re-identification model is used to extract the appearance feature vector of the target detection box. The re-identification model includes a feature extraction network, a global average pooling layer, a normalization layer, and a fully connected layer in sequence. The input data of the re-identification model is the image data corresponding to the target detection box.

6. The method according to claim 1, wherein The mixed height position cost, appearance cost and motion direction cost in the allocation cost are specifically set as: (1) Mixed height position cost According to the formula Calculate the intersection-over-union ratio of the mixed height, where is the mixed height intersection-merge ratio, is the highly modulated intersection-to-combination ratio, is the intersection-over-union ratio of the target detection frame of the current frame and the previous frame; and are preset weight coefficients respectively; among them, the height modulation intersection ratio , height coefficient , 、 Respectively represent the upper left corners of the two target detection boxes, 、 They represent the lower right corners of the two target detection frames, and the superscript is the target detection frame distinguisher; according to Get the mixed height position cost ; (2) Appearance cost Extract the appearance feature vector of the target detection frame, and use the cosine similarity between the appearance feature vector i of the target detection frame of the previous frame and the appearance feature vector j of the target detection frame of the current frame as the appearance cost ; (3) Movement direction cost The trajectory velocity direction θt is obtained based on the velocity direction between the center points of the two target detection frames in the previous and next frames, or based on different time intervals ∆t, the velocity direction between the center points of the two target detection frames with a time interval of ∆t in the current trajectory is calculated, and then the trajectory velocity direction θt is obtained based on the cumulative sum of the velocity directions of all set time intervals ∆t; and the velocity direction θd of the trajectory to the new detection frame is obtained based on the velocity direction from the trajectory to the center point of the target detection frame in the current frame, and the absolute error between θt and θd is used as the motion direction cost.

7. The method according to claim 6, wherein Replace the movement direction cost with: Given multiple time intervals ∆t, for each time interval ∆t, calculate the velocity direction between the four vertices of the two target detection boxes with a time interval of ∆t in the current trajectory, and then calculate the velocity direction cost of the current time interval ∆t based on the average value of the velocity direction between the four vertices; then accumulate the velocity direction costs of all time intervals ∆t to get the motion direction cost; For two target detection frames with a time interval of ∆t, the velocity direction between the vertices is expressed as: ,in 、 Represents the position coordinates of a vertex of the front and back target detection frames in the two target detection frames respectively.

8. The method according to claim 1, wherein When calculating the hybrid height position cost, based on the longitudinal velocity Status of The weight coefficient Make real-time adjustments, including: Check the vertical speed of the target detection box Whether the speed threshold is exceeded If so, then use the set step size to Make gradual reduction adjustments until the longitudinal speed Less than or equal to Stop reducing adjustments when When the value is reduced to the preset lower limit, the adjustment stops; If the longitudinal speed Less than the speed threshold , then the set step size is used to Make incremental adjustments until the longitudinal speed Greater than Stop increasing the adjustment when When the value reaches the preset upper limit, the adjustment stops.

Images