Multi-target tracking method based on deep learning and multi-task joint training
Through the joint training and feature extraction strategy of deep neural network model, the occlusion overlap problem in multi-object tracking is solved, the tracking speed and accuracy are improved, and the efficient multi-object tracking effect is achieved.
Patent Information
- Application Number
- CN202210107826.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-01-28
AI Technical Summary
When existing multi-objective tracking technology deals with multiple targets to be tracked, it is difficult to distinguish the real category of the target, resulting in the problem of detection error matching and slow tracking speed.
A deep neural network model was designed to realize the joint training of object detection and feature extraction through the combination of backbone feature extraction subnet, multi-objective complete feature extraction subnet, multi-objective occlusion feature extraction subnet and multi-task joint module. The splicing fusion strategy and multi-task joint loss function improve the optimization capability and tracking speed of feature extraction branches.
Real-time multi-object tracking is realized, the number of error matches is reduced, the overall performance and stability of the tracking network is improved, and the dual-task joint training effect of object detection and feature extraction is improved.
Smart Images

Figure CN114529581B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision and image processing, and further relates to a multi-target tracking method, which can be applied to security monitoring, automatic driving and human-computer interaction. Background Art
[0002] Target tracking is an important task in the field of computer vision and is widely used in security monitoring, autonomous driving, human-computer interaction and other scenarios. Single target tracking mainly involves searching and locating a target of interest manually selected in any frame of a video clip in subsequent video frames frame by frame. Single target tracking generally includes modules such as motion modeling and appearance modeling, in which motion modeling determines the search area, target detection is performed within the search area, and the detected target is located based on the appearance model, that is, it is determined whether the detected target is the target to be searched. The detection-based multi-target tracking method is similar to the above-mentioned single target tracking method. It detects and tracks multiple targets of interest at the same time based on the single target tracking technology. The difficulty lies in the following: in the single target tracking task, except for the target to be tracked, other image parts in the video frame are treated as background information. Therefore, the appearance modeling part is relatively simple, and the target positioning is completed by only identifying the area containing the target to be tracked; in the multi-target tracking task, since multiple targets to be tracked may overlap with each other, some information of other targets to be tracked may exist in the detected target area to be tracked. Therefore, it is necessary to design and implement an appearance model with occlusion overlap robustness, so that it can distinguish the true categories of each detected target when there is overlap of multiple target areas.
[0003] In recent years, under the background of deep learning, the detection-based tracking paradigm has been widely used in multi-target tracking tasks. This paradigm mainly includes three parts: target detection, feature extraction and data association. Among them, deep convolutional networks are mainly used in the two stages of target detection and feature extraction. Yu et al. proposed to use deep learning to replace traditional manual design in the paper "Poi: Multiple object tracking with high performance detection and appearance feature" (European Conference on Computer Vision, 2016, pp.36-42.) to more simply realize the extraction of target appearance features, and the deep feature extraction network is more universal. However, this method requires the training of the target detection network and the feature extraction network separately, so there are two problems: 1) The feature extraction network can only realize the appearance modeling of the target, but cannot effectively learn the target relationship between frames, and cannot learn the motion model of the target; 2) The tracking model cannot be trained end-to-end, and the tracking speed of the two stages of target detection and feature extraction is too slow. In the paper "Towards real-time multi-object tracking" (European Conference on Computer Vision, 2020, pp.107-122.), Wang et al. first unified the two modules of target detection and feature extraction into a tracking network called joint detection and feature extraction (JDE) for training. They designed a multi-task joint training method, which combines the detection branch loss function and the feature extraction branch loss function in the JDE network to form a joint loss function as the loss function of the JDE model, so that the detection branch and the feature extraction branch can be optimized synchronously during training, realizing end-to-end training of the tracking network. Because the JDE model combines the target detection and feature extraction tasks in the target tracking task, it can achieve near real-time tracking speed. However, due to the overly simple design of the feature extraction branch of the JDE model, the feature extraction branch cannot be fully optimized, and the JDE model cannot fully learn the visual saliency features of the target, which in turn causes a large number of target drifts and mismatches in the data association stage. Summary of the invention
[0004] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a multi-target tracking method based on deep learning and multi-task joint training to reduce the number of incorrect matching of the tracking network and improve the overall performance of the tracking network.
[0005] The technical solution of the present invention is to construct a deep neural network model that has the ability to extract strong visual saliency features of targets and can unify the two tasks of target detection and feature extraction in one network model for joint training; and perform multi-target tracking based on the network model. The specific implementation includes the following:
[0006] (1) Construct a deep neural network model consisting of a backbone feature extraction subnetwork, a multi-target complete feature extraction subnetwork, a multi-target occlusion feature extraction subnetwork, and a cascade of multi-task joint modules;
[0007] (2) Get the tth frame image F t ∈R W×H×3 And the target bounding box label set B∈R n×4 , normalize the set B to obtain the normalized target bounding box label set B′, where R represents the real number domain symbol, W, H represent the width and height of the image respectively, and n represents F t The number of targets in ;
[0008] (3) The image F obtained in (2) t and the normalized target bounding box label set B′ are input into the deep neural network model in (1), and the target detection set D∈R is output. n×4 and the target complete feature vector set E g ∈R n×a and the target occlusion feature vector set E p ∈R n×b , where a represents the vector dimension of the target complete feature vector, and b represents the vector dimension of the target occlusion feature vector;
[0009] (4) The target complete feature vector set E in (3) is fused by the concatenation strategy g ∈R n×a and the target occlusion feature vector set E p ∈R n×b Fusion is performed to obtain the target fusion feature vector set gather The n target fusion feature vectors of the target detection set D correspond one-to-one with the n targets in the target detection set D according to the set subscript, that is, the set The i-th target fusion feature vector in belongs to the i-th target in the set D, i∈[1,n];
[0010] (5) Target fusion feature vector set for target detection set D The target fusion feature vector set with the target trajectory T Calculate the cosine distance and get the distance matrix C n×m, where the target trajectory T represents the set of m targets tracked in the t-1th frame. Each target in the target trajectory T contains the identity attribute and the target fusion feature vector. represents the set of target fusion feature vectors of all targets in the target trajectory T,
[0011] (6) Using the distance matrix C n×m Perform correlation matching between the detected target and the track target to achieve multi-target tracking.
[0012] Compared with the prior art, the present invention has the following advantages:
[0013] First, the present invention unifies the target detection subtask and feature extraction subtask in the tracking task into a deep neural network model for multi-task joint training, which effectively utilizes the feature extraction capability of the deep neural network, so that the tracking network constructed by the present invention achieves real-time tracking speed and improves the overall performance of the tracking network;
[0014] Second, since the feature extraction subtask of the present invention is designed as a dual-branch network structure consisting of a target complete feature extraction branch and a target occlusion feature extraction branch, the deep neural network model can obtain more model losses of the feature extraction subtask when jointly training the target detection subtask and the feature extraction subtask, thereby balancing the two branch tasks of target detection and feature extraction, solving the problem of insufficient optimization of the feature extraction branch in the training stage of the tracking network for previous tasks that jointly detects target detection and extracts features, and reducing the number of incorrect matches of the tracking network. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is the overall flow chart of the implementation of the present invention;
[0016] Figure 2 It is a deep neural network model diagram in the present invention;
[0017] Figure 3 It is a sub-flow chart for obtaining a target detection set, a target complete feature vector set and a target occlusion feature vector set in the present invention;
[0018] Figure 4 It is a schematic diagram of a feature aggregation module in a deep neural network model of the present invention;
[0019] Figure 5 It is a schematic diagram of a multi-task joint module in the deep neural network model of the present invention. DETAILED DESCRIPTION
[0020] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.
[0021] Reference Figure 1 , the implementation steps of this embodiment are as follows:
[0022] Step 1: Build a deep neural network model.
[0023] Reference Figure 2 , the implementation of this step is as follows:
[0024] 1.1) The network structure before level 3 in the deep aggregation network DLA34 model improved based on ResNet34 is selected as the backbone feature extraction subnetwork, and its structure is: dla34_base_layer→dla34_layer0→dla34_layerl→dla34_layer2;
[0025] 1.2) A multi-target complete feature extraction subnetwork is formed by the structure of level3 to level5 in the DLA34 model, and its structure is: dla34_layer3→dla34_layer4→dla34_layer5; the output of the above backbone feature extraction subnetwork is used as the input of the multi-target complete feature extraction subnetwork;
[0026] 1.3) Add a random occlusion network layer rand_block before the multi-target complete feature extraction sub-network to form a multi-target occlusion feature extraction sub-network, whose structure is: rand_block→dla34_layer3→dla34_layer4→dla34_layer5;
[0027] The input of the random occlusion network layer is the response map output by the backbone feature extraction subnetwork and the normalized target bounding box label set, and its output is an occlusion response map of the same size as the input response map. In the model training stage, the occlusion parameters of the random occlusion network layer in this embodiment have a maximum occlusion ratio of width and height of w_drop_r and h_drop_r of 0.33. In the model testing stage, since the test data does not have target bounding box data, the random occlusion network layer does not process the input response map and directly outputs it.
[0028] 1.4) Define the common prediction convolution structure as: first two-dimensional convolution → activation function layer → second two-dimensional convolution, where the convolution kernel size of the first two-dimensional convolution layer is 3x3, the convolution kernel size of the second two-dimensional convolution layer is 1x1, the convolution step size of the two convolution layers is 1, and the activation function layer uses the ReLU activation function;
[0029] 1.5) Construct a target detection branch consisting of a heat map prediction convolution block, a target center offset prediction convolution block, and a target bounding box width and height prediction convolution block, and construct a feature extraction branch consisting of a target complete feature vector map prediction convolution block and a target occlusion feature vector map prediction convolution block; and use the target detection branch and the feature extraction branch to form a multi-task joint module; wherein:
[0030] The heat map prediction convolution block, the target center offset prediction convolution block, and the target bounding box width and height prediction convolution block are ordinary prediction convolution structures with output channel numbers of 1, 2, and 2 respectively;
[0031] The target complete feature vector map prediction convolution block and the target occlusion feature vector map prediction convolution block are respectively common prediction convolution structures with output channel numbers a and b. In this embodiment, a and b are 128 and 256 respectively.
[0032] 1.6) The multi-target complete feature extraction subnetwork and the multi-target occlusion feature extraction subnetwork are connected in parallel to form a multi-target fusion feature extraction module, and then the backbone feature extraction subnetwork, the multi-target fusion feature extraction module and the multi-task joint module are cascaded to form a deep neural network model.
[0033] Step 2: Get the image to be tracked and its label, and normalize the bounding box label.
[0034] 2.1) Sequentially read the tth frame image F of the video to be tracked t ∈R W×H×3 And the target bounding box label set B∈R n×4 , where R represents the real number domain symbol, W, H represent the width and height of the image respectively, 3 represents the number of RGB channels, and n represents the number of targets in Ft; the target bounding box is denoted as (x, y, w, h), where x∈[0, W], y∈[0, H], w∈(0, W], h∈(0, H);
[0035] 2.2) The bounding box label set B is normalized as follows:
[0036]
[0037] The normalized bounding box center point coordinates x∈[0,1], y∈[0,1] and bounding box width and height w∈(0,1], h∈(0,1], the normalized target bounding box label set is denoted as B′.
[0038] Step 3: The network outputs a target detection set, a target complete feature vector set, and a target occlusion feature vector set.
[0039] Reference Figure 3 , the implementation of this step is as follows:
[0040] 3.1) Transform the image F t The normalized target bounding box label set B′ is input into the backbone feature extraction subnetwork, and the response map X is output;
[0041] 3.2) Input the response map X into the multi-target complete feature extraction subnetwork to obtain intermediate feature maps X1, X2 and X3 with different downsampling degrees output by different convolutional layers. The intermediate feature maps X1, X2 and X3 are downsampled by 2, 4 and 8 times respectively relative to the response map X;
[0042] 3.3) Aggregate the response map X with the intermediate feature maps X1, X2 and X3 obtained in 3.2) to obtain the multi-target complete feature map X g ;
[0043] Reference Figure 4 , the steps of multi-layer feature map aggregation are as follows:
[0044] 3.3.1) Assume is the feature map σ downsampled by u times u And the feature map σ downsampled by v times v The function of the aggregation method is expressed as follows:
[0045]
[0046] Among them, σ u represents the feature map downsampled u times relative to the original input image, σ v represents a feature map that is downsampled v times relative to the original input image, and u>v; τ(·) represents scale-invariant convolution; It means that the feature map downsampled by v times is upsampled to the feature map downsampled by u times, and the upsampling method is deconvolution; Indicates the bitwise addition operation of feature maps of the same scale; σ′ u Represents the aggregated feature map, whose scale is the same as the feature map σ u same;
[0047] 3.3.2) Cluster the response map X and the intermediate feature maps X1, X2 and X3:
[0048] In the first step, the response map X and the intermediate feature maps X1, X2 and X3 are denoted as σ k , σ k / 2 , σ k / 4 , σ k / 8 ;
[0049] In the second step, the feature maps with a downsampling multiple of 2 times are respectively Functions are aggregated pairwise to obtain an aggregated result and
[0050] Step 3: For the aggregation result σ′ k ,σ′ k / 2 ,σ′ k / 4 Repeat step 2 to get the secondary aggregation result. and
[0051] Step 4: The secondary aggregation result σ″ k and σ″ k / 2 pass Function aggregation, get three aggregation results
[0052] Step 5: The three-time aggregation result σ″′ k and σ″ in the secondary aggregation result k / 2 pass Function aggregation, four aggregation results
[0053] Step 6: Gather the results of the four and σ′ in the result of agglomeration k / 2 pass Function aggregation, five aggregation results
[0054] Step 7: Gather the results of the five aggregations And the intermediate feature map X3 of step 1, that is, σ k / 8 ,pass Function aggregation, six aggregation results Should That is, the multi-layer aggregation result X of the response map X and the intermediate feature maps X1, X2 and X3 g , whose scale is the same as the feature map X;
[0055] 3.4) Input the response map X and the normalized target bounding box label set B′ into the random occlusion network layer of the multi-target occlusion feature extraction subnetwork to obtain the target occlusion response map X′. The specific steps are as follows;
[0056] 3.4.1) Restore the center coordinates (x, y) and width and height (w, h) of all bounding boxes in the normalized target bounding box label set B′ on the response map X to obtain the target bounding box set B on the response map X. X , the reduction method is x=x×W X , y=y×H X , w=w×W X , h=h×H X . Where W X and H X Respectively represent the width and height of the response graph X;
[0057] 3.4.2) Create a two-dimensional matrix M with the same width and height as the response map X, and fill all the values of the matrix M with 1; for the target bounding box set B on the response map X X For each bounding box in M, randomly mask the area mapped on M, that is, randomly take a continuous area in the bounding box and fill it with 0. The selection rule of this continuous area is that the width does not exceed w×w_drop_r, and the height does not exceed h×h_drop_r, where w and h represent the width and height of the bounding box, w_drop_r∈[0,1), h_drop_r∈[0,1); use the processed M as a mask;
[0058] 3.4.3) Multiply the mask M by the response map X bitwise to obtain the target occlusion response map X′;
[0059] 3.5) The target occlusion response map X′ is further processed by the remaining part of the multi-target occlusion feature extraction subnetwork to obtain intermediate feature maps X′1, X′2 and X′3 with different downsampling degrees output by different convolutional layers. These intermediate feature maps X′1, X′2 and X′3 are downsampled by 2, 4 and 8 times respectively relative to the target occlusion response map X′;
[0060] 3.6) Perform multi-layer feature map aggregation on the target occlusion response map X′ and the intermediate feature maps X′1, X′2 and X′3 obtained in 3.5) to obtain the multi-target occlusion feature map X p ;
[0061] 3.7) Multi-target complete feature map X g and multi-target occlusion feature map X p Output to the multi-task joint module, output the target detection branch results and feature extraction branch results:
[0062] Reference Figure 5 , the specific steps are as follows:
[0063] 3.7.1) Multi-target complete feature map X g ∈R C×W×H After the heat map prediction convolution transformation output heat map X ht ∈R 1×W×H , after the target center offset is predicted, the convolution transform outputs the offset X off ∈R 2×W×H , after the target bounding box width and height prediction convolution transformation output target bounding box width and height X wh ∈R 2×W×H , heat map prediction value X ht , target center offset prediction value X off And the target bounding box width and height prediction value X wh Together as the target detection branch result, where C represents X gThe number of channels, W, H represent the multi-target complete feature map X g width and height;
[0064] 3.7.2) Multi-target complete feature map X g and multi-target occlusion feature map X p The target complete feature vector graph X is output through a 2D convolution block respectively ge ∈R a×W×H and target occlusion feature vector map X pe ∈R b×W×H , target complete feature vector graph X ge and target occlusion feature vector map X pe Together they serve as the result of the feature extraction branch, where a represents the vector dimension of the target complete feature vector, and b represents the vector dimension of the target occlusion feature vector;
[0065] 3.8) Heat map X ht ∈R 1×W×H As a two-dimensional matrix A of size W×H, and the x-th i Row, No. x j The value at the column position is recorded as Similarly, the offset X off ∈R 2×W×H 、Width and height of target bounding box X wh ∈R 2×W×H 、Target complete feature vector graph X ge ∈R a×W×H and target occlusion feature vector map X pe ∈R b×W×H They are also regarded as two-dimensional matrices of size W×H, and are denoted as O, S, G, and P respectively, then we have in Respectively represent the xth matrix O, S, G, P i Row, No. x j The value at the column position;
[0066] The coordinates of the two-dimensional space position with a size of W×H are filtered by matrix A:
[0067] if Then the coordinate (x i , x j ) satisfies the condition, and the coordinate (x i , x j ) is added to the coordinate set L to obtain a coordinate set that satisfies the condition: L = {(x i1 , x j1 ), (x i2 , x j2 ),…,(x in , xjn )}, where n is the number of positions that meet the coordinate screening condition, ε is the screening threshold, and in this embodiment, ε=0.4;
[0068] Otherwise, no processing is done;
[0069] 3.9) According to the selected coordinate set L, the values at the corresponding positions are taken from the matrices O and S in 3.2) to obtain the target center offset set O′ and the target bounding box width and height set S′:
[0070]
[0071]
[0072] Perform vector addition on the elements in the coordinate set L and the target center offset set O′ to obtain the center coordinate set L′ of the n detected target bounding boxes, and then form the image F with the center coordinate set L′ of the target bounding box and the width and height set S′ of the target bounding box. t The target detection set D on
[0073] 3.10) According to the selected position L, the values at the corresponding positions are taken from the matrices G and P in 3.2) to obtain the target complete feature vector set E g and the target occlusion feature vector set E p :
[0074]
[0075]
[0076] Among them, a represents the vector dimension of the target complete feature vector, and b represents the vector dimension of the target occlusion feature vector.
[0077] Step 4: Use a multi-task joint training method to perform supervised training on the network model constructed in step 1.
[0078] 4.1) For the target detection set D, the loss function designed by the target detection model CenterNet is used to calculate the loss L of the target detection branch det ,Among them, the method of generating heat map labels through bounding box labels also follows the method designed by CenterNet;
[0079] 4.2) For the target complete feature vector set E g Each vector in After classifier FC g Calculate its classification prediction results
[0080]
[0081] According to The cross entropy loss function of the multi-classification task is used to calculate the target complete feature vector classification loss L id_g :
[0082]
[0083] Where n is the target complete feature vector set E g The number of elements in ; considering the identity attribute of the target as a category, K is the total number of categories of the target in the training data; Represents the target complete feature vector set E g The i-th vector in Whether the corresponding target category real label is category l, For vector After classifier FC g Output prediction results The l-th element value in , l∈[1,K];
[0084] 4.3) Target occlusion feature vector set E p Each vector in After classifier FC p Calculate its classification prediction results
[0085]
[0086] According to The cross entropy loss function of the multi-classification task is used to calculate the classification loss L of the target occlusion feature vector id_p :
[0087]
[0088] Where n is the target complete feature vector set E p The number of elements in ; considering the identity attribute of the target as a category, K is the total number of categories of the target in the training data; Represents the target occlusion feature vector set E p The i-th vector in Whether the corresponding target category real label is category l, For vector After classifier FC p Output prediction results The l-th element value in , l∈[1,K];
[0089] 4.4) In order to achieve joint training of target detection task and feature extraction task, it is necessary to merge the loss of detection branch and feature extraction branch in the deep neural network into the loss L of the whole deep neural network. totalThe specific design is as follows:
[0090]
[0091] L id =γL id_g +(1-γ)L id_p
[0092] Among them, L det To detect the branch loss, L id is the feature extraction branch loss, L id_g and L id_p are the classification loss of the target complete feature vector and the classification loss of the target occluded feature vector, respectively, α and β is the network self-learning weight, γ is the classification loss L of the target complete feature vector id_g The proportion of the entire classification loss. In this embodiment, γ is set to 0.5.
[0093] Step 5: Use the trained network model to perform video target tracking test.
[0094] 5.1) Input the first frame image F1 of the video to be tracked into the deep convolutional network model to obtain the target detection set D∈R m×4 , target complete feature vector set E g ∈R m×a and the target occlusion feature vector set E p ∈R m×b , where m is the number of detected targets, a is the dimension of the target complete feature vector, and b is the dimension of the target occlusion feature vector;
[0095] 5.2) Using the first frame image to obtain the target detection set D and target complete feature vector set E g and the target occlusion feature vector set E p , initialize the target trajectory T:
[0096] 5.2.1) For each target bounding box D in the target detection set j ∈D, with its subscript j as its initial identity ID (D j ) = j;
[0097] 5.2.2) The target complete feature vector set E is merged by splicing and fusion strategy g and the target occlusion feature vector set E p Fusion is performed to obtain the target fusion feature vector set
[0098] 5.2.3) The target bounding box D of each detected target j The corresponding identity value ID (Dj ), target fusion feature vector Elements that make up the target trajectory Then we get the target trajectory T = {T1, T2, ..., T j …, T m};
[0099] 5.3) The deep convolutional network model reads the next frame image F t , get the target detection set D∈R n×4 , target complete feature vector set E g ∈R n×a and the target occlusion feature vector set E p ∈R n×b , and the target complete feature vector set E g and the target occlusion feature vector set E p Splice and fuse to get the target fusion feature vector set
[0100] 5.4) Associate each target in the target detection set D obtained in 5.3) with the target in the target trajectory T to determine the identity of each target in the target detection set D:
[0101] 5.4.1) For each detected object, the object bounding box D i ∈D, using its corresponding target fusion feature vector Calculate the target fusion feature vector corresponding to this vector and each target in the target trajectory T The cosine distance of n×m , where n is the image F t The number of elements in the target detection set D, m is the number of elements in the target trajectory T, i∈[1,n], j∈[1,m];
[0102] 5.4.2) For the distance matrix C m×n Using the Hungarian algorithm, we get the target matching pair set P∈R r×2 ,in:
[0103] The elements in the P set are two-tuples consisting of the element subscripts in the target detection set D and the element subscripts in the target trajectory T, r∈[0, min(m, n)];
[0104] 5.4.3) According to the target matching pair set P, set the t-th frame image F t The identity of the target detected in the target detection set D, that is, if any target matching pair in P is (i, j), then ID(D i )=ID(T j ), to achieve the matching of the detection target and the trajectory target in the tth frame, where D iis the i-th target in the target detection set D, T j is the jth target in the target trajectory T, and ID(·) represents the identity of the target ·;
[0105] 5.5) Repeat 5.3) to 5.4) until the last frame of image is processed and the video multi-target tracking task is completed.
[0106] The effect of the present invention can be further illustrated by the following simulation results:
[0107] 1. Simulation Experiment Conditions
[0108] The hardware platform used in the present invention is: the CPU uses an Intel Core i7-9700k with eight cores and eight threads, with a main frequency of 3.6GHz and a memory of 64GB; the GPU uses two Nvidia RTX 3090s with a video memory of 24GB. The software platform used is: the operating system uses Ubuntu16.04LTS, the deep learning computing framework uses PyTorch 1.4, and the programming language uses Python3.6.
[0109] This simulation experiment uses three evaluation indicators: multi-target tracking accuracy MOTA, identification F value IDF1 and ID transition number IDS. Among them:
[0110] Multi-target tracking accuracy (MOTA) is the proportion of samples in the test set that are correctly detected and matched. The closer it is to 100, the better the tracker performance.
[0111] Identification F value IDF1 is the F value of target matching in all detection boxes on the test set. Its value is between 0 and 100%. The larger the value, the better the classification effect.
[0112] The ID transition number IDS is the number of times the target ID changes in the tracking trajectory. It is used to reflect the stability of tracking. The smaller the value, the better.
[0113] 2. Simulation content and result analysis
[0114] The pedestrian multi-target tracking dataset MOT17 was tested under the above simulation conditions using the present invention and the two existing methods JDE and FairMOT. The tracking results were evaluated using the above three evaluation indicators. The results are shown in Table 1.
[0115] Table 1 Comparison of tracking effects between the prior art and the present invention on MOT17
[0116] Method name\Indicator MOTA IDF1 IDS JDE 63.9 57.9 4434 FairMOT 73.2 72.8 2964 Method of the present invention 73.9 73.7 2640
[0117] As can be seen from Table 1, the multi-target tracking accuracy MOTA, identification F value IDF1 and ID transition number IDS of the present invention are higher than the two existing methods, indicating that the overall tracking effect and target recognition ability of the present invention are better than the two existing methods. This is because the present invention extracts the complete features and occlusion features of the target at the same time by designing a feature extraction branch, thereby enhancing the loss ratio of the feature extraction task in the joint training of the dual tasks of target detection and feature extraction, and then the model can fully train the feature extraction branch, so that the joint training model can achieve high-performance target detection while also having strong visual saliency feature extraction capabilities, so that the accuracy of the data association stage in the tracking process is effectively improved. From the IDS index, it can be seen that the present invention significantly reduces the number of misaligned matches during the tracking process and greatly improves the stability of the tracking model.
[0118] In summary, the method proposed in the present invention can realize joint training of target detection and strong feature extraction tasks directly in one tracking network without using a separate visual saliency feature extraction model, and enable the joint training model to achieve high-performance target detection while also having strong visual saliency feature extraction capabilities. The data association accuracy is improved through strong visual saliency features, thereby improving the overall performance of the tracking model.
Claims
1. A multi-target tracking method based on deep learning and multi-task joint training, It is characterized in that include: (1) Construct a deep neural network model consisting of a backbone feature extraction subnetwork, a multi-target complete feature extraction subnetwork, a multi-target occlusion feature extraction subnetwork, and a cascade of multi-task joint modules; (2) Get the tth frame image F t ∈R W×H×3 And the target bounding box label set B∈R n×4 , normalize set B to obtain the normalized target bounding box label set B ′ , where R represents the real number domain symbol, W, H represent the width and height of the image respectively, and n represents F t The number of targets in ; (3) The image F obtained in (2) t and the normalized target bounding box label set B ′ Input to the deep neural network model in (1), output target detection set D∈R n×4 and the target complete feature vector set E g ∈R n×a and the target occlusion feature vector set E p ∈R n×b , where a represents the vector dimension of the target complete feature vector and b represents the vector dimension of the target occlusion feature vector; the implementation is as follows: 3a) Transform image F t Input the backbone feature extraction subnetwork to obtain the response graph X; 3b) Extracting the multi-target complete feature map X of the response map X through the multi-target complete feature extraction sub-network g ; 3c) Process the response map X and the normalized target bounding box label set B through the multi-target occlusion feature extraction sub-network ′ , get the multi-target occlusion feature map X p ; The implementation is as follows: 3c1) Combine the response map X and the normalized target bounding box label set B ′ Input to the random occlusion network layer of the multi-target occlusion feature extraction subnetwork to obtain the target occlusion response map X ′ ; 3c2) The target occlusion response map X ′ Continue to process the rest of the multi-target occlusion feature extraction subnetwork to obtain the intermediate feature maps X with different downsampling degrees output by different convolutional layers. 1 ′ , X 2 ′ and X 3 ′ , these intermediate feature maps X 1 ′ , X 2 ′ and X 3 ′ Respectively relative to the target occlusion response map X ′ Downsampling by 2, 4, or 8 times; 3c3) The target occlusion response map X ′ And the intermediate feature map X obtained in 3c4) 1 ′ , X 2 ′ and X 3 ′ Perform multi-layer feature map aggregation to obtain a multi-target occlusion feature map X p ; 3d) Processing multi-target complete feature map X through multi-task joint module g and multi-target occlusion feature map X p , get the target detection set D∈R n×4 and the target complete feature vector set E g ∈R n×a and the target occlusion feature vector set E p ∈R n×b ; (4) The target complete feature vector set E in (3) is fused by the concatenation strategy g ∈R n×a and the target occlusion feature vector set E p ∈R n×b Fusion is performed to obtain the target fusion feature vector set gather The n target fusion feature vectors of the target detection set D correspond one-to-one with the n targets in the target detection set D according to the set subscript, that is, the set The i-th target fusion feature vector in belongs to the i-th target in the set D, i∈[1,n]; (5) Target fusion feature vector set for target detection set D The target fusion feature vector set with the target trajectory T Calculate the cosine distance and get the distance matrix C n×m , where the target trajectory T represents the set of m targets tracked in the t-1th frame. Each target in the target trajectory T contains the identity attribute and the target fusion feature vector. represents the set of target fusion feature vectors of all targets in the target trajectory T, (6) Using the distance matrix C n×m Perform correlation matching between the detected target and the track target to achieve multi-target tracking.
2. The method according to claim 1, It is characterized in that (2) The target bounding box label set B is normalized using the following formula: in, Represents image F t The width, height, x, and y represent the center pixel position of the bounding box, image F t The pixel position of the upper left corner is (0,0), w and h represent the width and height of the bounding box.
3. The method according to claim 1, It is characterized in that 3b) The multi-target complete feature map X of the response map X is extracted by the multi-target complete feature extraction sub-network g , implemented as follows: 3b1) Input the response map X into the multi-target complete feature extraction sub-network to obtain the intermediate feature maps X with different downsampling degrees output by different convolutional layers. 1 , X 2 and X 3 , the intermediate feature map X 1 , X 2 and X 3 Downsampled by 2, 4, and 8 times relative to the response map X; 3b2) Compare the response map X with the intermediate feature map X obtained in 3b1) 1 , X 2 and X 3 Perform multi-layer feature map aggregation to obtain a multi-target complete feature map X g .
4. The method according to claim 3, It is characterized in that 3b2) Compare the response map X with the intermediate feature map X 1 , X 2 and X 3 Perform multi-layer feature map aggregation, which is implemented as follows: 3b2.1) Assume is the feature map σ downsampled by u times u And the feature map σ downsampled by v times v The function of the aggregation method is expressed as follows: Among them, σ u represents the feature map downsampled u times relative to the original input image, σ v represents a feature map that is downsampled v times relative to the original input image, and u>v; τ(·) represents scale-invariant convolution; It means that the feature map downsampled by v times is upsampled to the feature map downsampled by u times, and the upsampling method is deconvolution; Indicates the bitwise addition operation of feature maps of the same scale; σ u ′ Represents the aggregated feature map, whose scale is the same as the feature map σ u same; 3b2.2) Aggregate multi-layer feature maps: Step 1: Combine the response map X and the intermediate feature map X 1 , X 2 and X 3 Denoted as σ k , σ k / 2 , σ k / 4 , σ k / 8 ; In the second step, the feature maps with a downsampling multiple of 2 times are respectively Functions are aggregated pairwise to obtain an aggregated result and Step 3: For the aggregation result σ k ′ , σ k ′ / 2 , σ k ′ / 4 Repeat step 2 to get the secondary aggregation result. and Step 4: The secondary aggregation result σ″ k and σ″ k / 2 pass Function aggregation, get three aggregation results Step 5: The three-time aggregation result σ″′ k and σ″ in the secondary aggregation result k / 2 pass Function aggregation, four aggregation results Step 6: Gather the results of the four and σ in the result of agglomeration k ′ / 2 pass Function aggregation, five aggregation results Step 7: Gather the results of the five aggregations and the intermediate feature map X in step 1 3 , that is, k / 8 ,pass Function aggregation, six aggregation results Should That is, the response map X and the intermediate feature map X 1 , X 2 and X 3 The multi-layer aggregation result X g , whose scale is the same as the feature map X.
5. The method according to claim 1, It is characterized in that 3c1) to obtain the target occlusion response map X ′ , implemented as follows: 3c1.1) The normalized target bounding box label set B ′ The center coordinates (x, y) and width and height (w, h) of all bounding boxes in the response map X are restored to obtain the target bounding box set B on the response map X. X , the reduction method is x=x×W X , y=y×H X , w=w×W X , h=h×H X ;W X and H X Respectively represent the width and height of the response graph X; 3c2) Create a two-dimensional matrix M with the same width and height as the response map X, and fill all the values of the matrix M with 1; for the target bounding box set B on the response map X X For each bounding box in M, randomly mask the area mapped on M, that is, randomly take a continuous area in the bounding box and fill it with 0. The selection rule of this continuous area is that the width does not exceed w×w_drop_r, and the height does not exceed h×h_drop_r, where w and h represent the width and height of the bounding box, w_drop_r∈[0,1), h_drop_r∈[0,1); use the processed M as a mask; 3c3) Multiply the mask M by the response map X bitwise to obtain the target occlusion response map X ′ .
6. The method according to claim 1, It is characterized in that 3d) In the multi-task joint module, the multi-target complete feature map X is processed g and multi-target occlusion feature map X p , implemented as follows: 3d1) Multi-target complete feature map X g ∈R C×W×H After the heat map prediction convolution transformation output heat map X ht ∈R 1×W×H , after the target center offset is predicted, the convolution transform outputs the offset X off ∈R 2×W×H , after the target bounding box width and height prediction convolution transformation output target bounding box width and height X wh ∈R 2×W×H , where C represents X g The number of channels, W, H represent the multi-target complete feature map X g width and height; 3d2) Multi-target complete feature map X g ∈R C×W×H After the 2D convolution block, the target complete feature vector image X is output ge ∈R a ×W×H , where a represents the vector dimension of the target complete feature vector; 3d3) Multi-target occlusion feature map X p ∈R C×W×H After the 2D convolution block, the target occlusion feature vector map X is output pe ∈R b ×W×H , where b represents the vector dimension of the target occlusion feature vector; 3d4) Heat map X hT ∈R 1×W×H As a two-dimensional matrix A of size W×H, and the x-th i Row, No. x j The value at the column position is recorded as Similarly, the offset X off ∈R 2×W×H 、Width and height of target bounding box X wh ∈R 2×W×H 、Target complete feature vector graph X ge ∈R a×W×H and target occlusion feature vector map X pe ∈R b×W×H They are also regarded as two-dimensional matrices of size W×H, and are denoted as O, S, G, and P respectively, then we have in Respectively represent the xth matrix O, S, G, P i Row, No. x j The value at the column position; The coordinates of the two-dimensional space position with size W×H are screened by matrix A: if Then the coordinate (x i ,x j ) satisfies the condition, and the coordinate (x i ,x j ) is added to the coordinate set L to obtain a coordinate set that satisfies the condition: L = {(x i1 ,x j1 ),(x i2 ,x j2 ),…,(x in ,x jn )}, where n is the number of locations that meet the coordinate screening conditions and ε is the screening threshold; 3d5) According to the selected coordinate set L, the values at the corresponding positions are taken from the matrices O and S in 3d4) to obtain the target center offset set O ′ And the target bounding box width and height set S ′ , and then the coordinate set L and the target center offset set O ′ Perform vector addition on the elements in to obtain the center coordinate set L of the detected n target bounding boxes ′ , and then the center coordinate set L of the target bounding box ′ And the target bounding box width and height set S ′ Combined, we get image F t The target detection set D on 3d6) According to the selected position L, the values at the corresponding positions are taken from the matrices G and P in 3d4) to obtain the target complete feature vector set E g and the target occlusion feature vector set E p ,in, Where a represents the vector dimension of the target complete feature vector, and b represents the vector dimension of the target occlusion feature vector.
7. The method according to claim 1, It is characterized in that (6) using the distance matrix C n×m Perform association matching between the detected target and the track target to achieve multi-target tracking, which is implemented as follows: 6a) For the distance matrix C m×n Using the Hungarian algorithm, we get the target matching pair set P∈R r×2 , where the elements in the P set are two-tuples consisting of the element subscripts in the target detection set D and the element subscripts in the target trajectory T, r∈[0,min(m,n)]; 6b) According to the target matching pair set P, set the identity of the target detected in the target detection set D, that is, if any target matching pair in P is (i, j), then ID(D i )=ID(T j ), where D i is the i-th target in the target detection set D, T j is the jth target in the target tracking set T, ID(·) represents the identity of the target ·, and the detection target in the tth frame is matched with the track target to complete the tracking task of the current frame.
