A lightweight multi-target detection and tracking method for road video stream vehicle monitoring

Abstract

The application discloses a lightweight multi-target detection and tracking method for road video stream vehicle monitoring, and belongs to the field of computer vision. In the lightweight multi-target detection and tracking algorithm model, a lightweight backbone network is used to reduce the complexity and parameter quantity of the model. In order to make up for the precision decline caused by the lightweight backbone network, the application introduces a self-adaptive activation function to improve the C3 module and designs a Meta-C3 module. In addition, the application introduces an Adaface loss to act on a feature extraction branch to enhance the network's ability to distinguish different vehicle targets, and further improves the performance of the network. The application designs a lightweight multi-target detection and tracking algorithm according to the characteristics that targets are easy to identify in a road video monitoring scene, reduces the requirement of a traditional algorithm on device computing power, realizes the deployment of an algorithm on an edge node, and has good practical application value.

Description

TECHNICAL FIELD

[0001] The application discloses a lightweight multi-target detection and tracking algorithm for road video stream vehicle monitoring and belongs to the field of computer vision. BACKGROUND

[0002] In recent years, intelligent devices have become an indispensable part in people's life, especially in the field of intelligent transportation. After the application of intelligent monitoring, management and control devices, the operation of urban transportation systems can save a lot of manpower, material resources and time. Road monitoring collection, as the basis of intelligent transportation, is one of the current hot fields of concern. Road traffic analysis and congestion determination rely on the collection and processing of road monitoring information, and the related collection technology scheme has become the focus of research.

[0003] Unlike the general multi-target detection and tracking technology applied to the scene of tracking pedestrians, the tracking target in the road monitoring scene is mainly vehicles and the perspective is a fixed overhead perspective. Compared with pedestrians, vehicle targets are larger, and the road background of the monitoring perspective is relatively simple compared to the background in the pedestrian tracking application scene, which makes the accuracy requirement of the detector lower than that in the previous two application scenes. In this case, designing a more simplified neural network can achieve a higher speed and accuracy ratio, and a smaller network is also more conducive to edge deployment. The high-precision pedestrian tracking algorithm uses a complex model and has a large amount of calculation, and it is difficult to be put into practice in the face of massive road monitoring and real-time processing. However, compared with pedestrian tracking, vehicle tracking in the road monitoring scene also has other challenging difficulties: the target speed is relatively fast, the difference between two frames is large, and when the vehicle enters or exits the picture, there is a large appearance gap for the same target, such as the difference between half the vehicle body and the whole vehicle body. These difficulties will lead to the need for additional adjustment of the matching strategy to better adapt to the target application environment when associating and matching.

[0004] This paper aims to complete the multi-target detection and tracking task by using a lightweight network in the road monitoring scene, and to reduce the network size as much as possible and improve the network inference speed while maintaining similar accuracy. SUMMARY

[0005] Aiming at vehicle monitoring of road video stream, the purpose of the present application is to propose a lightweight network to complete the detection and tracking task, which is composed of a shared feature extraction network, a target detection branch and a target re-identification feature extraction branch. Among them, the basic structure of ShuffleNetV2 is used in the shared feature extraction network part to reduce the network complexity, and ACON series activation function is applied to improve the network accuracy. In addition, for the target re-identification feature extraction branch, the Adaface loss is introduced to enhance the distinguishability of the features between different vehicles, which can increase the cosine distance between the features of two vehicle targets. Therefore, the lightweight target detection and tracking network can accurately and efficiently realize the detection and tracking of vehicle targets in road video monitoring.

[0006] The technical scheme of the present application is a lightweight multi-target detection and tracking method for road video stream vehicle monitoring, which comprises:

[0007] Step 1: Constructing training and test dataset samples; based on the UA-DETRAC dataset, 80 videos are selected as the training set and 20 videos are selected as the test set;

[0008] Step 1.1: Read the original picture file and the corresponding label file of each video, modify the directory structure to adapt to the data loader used in this paper, and store it in the cache for subsequent steps;

[0009] Step 1.2: Parse the xml label file corresponding to each video, according to the ignored area marked in the label file, shield the frame by frame for each video, and store the preprocessed picture in the dataset folder after renaming;

[0010] Step 1.3: For the xml label file corresponding to each video, parse the target information contained in each frame, and store an independent label file for each frame according to the specified format;

[0011] Step 1.4: Traverse the folder where the label file is located and the folder where the picture file is located, compare the number of files in the two folders, and delete the empty video frames without recognized targets according to the label file;

[0012] Step 1.5: For the test set, special true value label files are needed for tracking performance index determination; use the original label file to generate true value files for evaluation for each test set video;

[0013] Step 2: Constructing a multi-target detection and tracking neural network;

[0014] The multi-target detection and tracking neural network comprises a CBRM module, a backbone network, a feature fusion network and an output head, the CBRM module is a feature extraction module comprising a 3x3 convolution layer, a BN layer, a ReLU layer and a MaxPool layer;

[0015] The backbone network comprises a first shuffle-block2 module, a first shuffle-block1 module, a second shuffle-block2 module, a second shuffle-block1 module, a third shuffle-block1 module and a third shuffle-block1 module in sequence;

[0016] The feature fusion network comprises a first Meta-C3 module, a first concat module, a second Meta-C3 module, a first Upsample module, a second concat module, a third Meta-C3 module, a third concat module and a fourth Meta-C3 module; the output of the third shuffle-block1 module in the backbone network is input into the first Meta-C3 module, the outputs of the second shuffle-block1 module and the first Meta-C3 module are jointly input into the first concat module, the output of the first concat module is sequentially subjected to the second Meta-C3 module and the first Upsample module; then the outputs of the first shuffle-block1 module and the first Upsample module are jointly input into the second concat module, the output of the second concat module is input into the third Meta-C3 module, the outputs of the first Upsample module and the third Meta-C3 module are jointly input into the third concat module, the output of the third concat module is input into the fourth Meta-C3 module, and the output of the fourth Meta-C3 module is connected to the output head;

[0017] The output head comprises four output heads, each output head comprising a 3x3 convolution layer, a ReLU layer and a 1x1 convolution layer;

[0018] Wherein, the CBRM module is sequentially connected by 3x3 convolution layer, batch normalization layer, activation function layer and maximum pooling layer, the Shuffle-block is derived from ShuffleNetV2 network, wherein the Shuffle-block1 module is composed of channel splitting layer, depth separable convolution layer and channel scrambling layer, and the Shuffle-block2 module is composed of depth separable convolution layer and channel scrambling layer, the Meta-C3 module is improved based on C3 module structure using ACON series activation function, the Concat module outputs after splicing operation of input features, and the Upsample module outputs after up-sampling operation of input features;

[0019] Step 3: training the multi-target detection and tracking network and introducing Adaface loss;

[0020] Step 3.1: the loss function of the multi-target detection and tracking network is composed of four parts, which are respectively:

[0021] Focal loss L of the heat map loss heatmap , regular reconstruction loss L of the center paranoia loss offset , regular reconstruction loss L of the target width and height loss wh , and Adaface loss L introduced for target recognition feature branch ReID ; therefore, combined with the branch weight coefficient τ of each branch during multi-task training, the final loss function of the framework is:

[0022] L total =L heatmap +τ offset L offset +τ wh L wh +τ ReID L ReID

[0023] Step 3.2: in order to obtain the optimal network parameters of the multi-target detection and tracking network, a batch of training data sets are randomly selected and input into the network; calculate the loss function L total , and use Adam optimizer to optimize the multi-task learning network parameters;

[0024] Step 4: using the trained multi-target detection and tracking network to test the test set.

[0025] Further, the calculation method of the Focal loss L of the heat map loss heatmap is as follows:

[0026] Let the center coordinates of the detected target i in the image be The coordinates of the top-left corner and the bottom-right corner of the target frame are wherein And The output of the heat map has d times down-sampling, so the center coordinates of the target i correspond to the mapping coordinates on the heat map Finally, the response at the heat map (x, y) coordinate point is:

[0027]

[0028] Where σ p is the standard deviation, and finally the heat map loss formula is as follows:

[0029]

[0030] Where is the estimated heat map, and α and β are hyperparameters.

[0031] Further, the calculation method of the regular reconstruction loss L offset of the center paranoia loss is:

[0032] The center paranoia O obtained from the true value label can be calculated, and its expression is as follows:

[0033]

[0034] Where p is the coordinate of the key target center point in the true value, is the coordinate mapping of the center point obtained after down-sampling, and L1 loss is used for training, so the final center paranoia loss expression is as follows:

[0035]

[0036] Where is the center paranoia output by the prediction head.

[0037] Further, the calculation method of the regular reconstruction loss L wh of the target width and height loss is:

[0038] The target width and height loss uses L1 loss for training, and the expression is as follows:

[0039]

[0040] Where is the target box width and height output by the prediction head, and S i is the width and height of the target box in the true value.

[0041] Further, the calculation method of the Adaface loss L ReID introduced by the target re-identification feature branch is:

[0042]

[0043] where s is a hyper parameter set manually, and θ j is the angle between the weight of the jth classification and the feature vector; the model-related quantity The image quality vector expression in a batch training can be obtained as follows:

[0044]

[0045] where μ z is the mean of z i in a training batch, and σ z is the standard deviation of ||z i || in a training batch, and the expression is as follows; h is a manually set parameter, and the purpose is to limit the final value of μ between ±1; at the same time, an exponential moving average processing method is also used to avoid the unstable cases of μ z and σ z due to too small a training batch;

[0046] μ z = α μ z (k) + (1-α) μ z (k-1)

[0047] σ z = α σ z (k) + (1-α) σ z (k-1)

[0048] where k is the order number that needs to be stabilized, and α is a manually set parameter, and after the definition of μ is completed, the image quality adaptive function g angle and g add about the manually set parameter m can be obtained, and the expression is as follows:

[0049]

[0050]

[0051] In summary, due to the adoption of the above technical solutions, the beneficial effects of the present application are:

[0052] 1. The lightweight network model proposed in the present application is specially optimized for the application field of vehicle target detection and tracking in road video monitoring, so that the algorithm can accurately and efficiently run in the application scenario.

[0053] 2. The application introduces an adaptive activation function to improve the feature extraction module, and the improved feature extraction model can effectively improve the network precision.

[0054] 3. The application introduces Adaface loss in the re-identification feature extraction branch, which can enhance the distinguishability of features between different vehicles, increase the cosine distance between the features of two vehicle targets, and further improve the accuracy of the tracking task.

[0055] 4. The application can achieve an inference speed of 17.7FPS in the unoptimized acceleration of the edge development board JETSON NX. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 The figure is a schematic diagram of the overall structure of the multi-target detection and tracking algorithm network of the application.

[0057] Figure 2 The figure is a road information statistics application result graph of the application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical method and advantages of the application more clear, the application will be further described in detail below in combination with embodiments and drawings.

[0059] The application proposes a lightweight target detection and tracking network for the problems of low running efficiency of algorithm complexity and high requirement for processing device computing power in road video stream vehicle monitoring application. The network model is composed of a shared feature extraction network, a target detection branch and a target re-identification feature extraction branch. In the shared feature extraction network part, the basic structure of ShuffleNetV2 is used to reduce the network complexity, and ACON series activation function is applied to improve the network precision. In addition, for the target re-identification feature extraction branch, Adaface loss is introduced to enhance the distinguishability of features between different vehicles, which can increase the cosine distance between the features of two vehicle targets. Therefore, the lightweight target detection and tracking network can accurately and efficiently realize the detection and tracking of vehicle targets in road video monitoring.

[0060] The lightweight target detection and tracking network of the application has the following specific implementation steps:

[0061] Step 1: Construct the data set sample for training and testing. Based on the UA-DETRAC data set, select 80 videos as the training set and 20 videos as the test set

[0062] Step 1.1: Read the original picture file and the label file corresponding to each video, modify the directory structure to adapt to the data loader used in this paper, and store it in the cache for subsequent steps.

[0063] Step 1.2: Parse the xml tag file corresponding to each video segment, and according to the ignored area marked in the tag file, perform occlusion shielding on the video segment frame by frame, and store the preprocessed image in the dataset folder after renaming.

[0064] Step 1.3: For each video segment corresponding to the xml tag file, parse the target information contained in each frame, and store an independent tag file for each frame according to the specified format.

[0065] Step 1.4: Traverse the folder where the tag file is located and the folder where the image file is located, compare the number of files in the two folders, and delete the empty video frames without the target to be recognized according to the tag file.

[0066] Step 1.5: For the test set, special true value tag files are needed to track performance indicators. Use the original tag file to generate true value files for evaluation for each test set video.

[0067] Step 2: Construct a multi-target detection and tracking neural network. The backbone of the shared feature extraction network has multiple ShuffleNetV2 basic structures, and the specific network structure parameters are shown in Table 1. In the feature fusion part, the shallow and deep features of the backbone network are fused to fully exploit the potential features of the target. The Mate-C3 module improved using the ACON series of activation functions is an important component of this part. For the target detection branch and the target re-identification feature extraction branch, they are composed of 3x3 size convolution and activation function and a 1x1 size convolution module, respectively.

[0068] Step 3: Introduce Adaface loss and train the multi-target detection and tracking network.

[0069] Step 3.1: The loss function of the multi-target detection and tracking network is composed of four parts. The focal loss of the heat map loss L heatmap , the L1 regular reconstruction loss of the center paranoia loss L offset , the L1 regular reconstruction loss of the target width and height loss L wh , and the Adaface loss L ReID introduced for the target re-identification feature branch.

[0070] Step 3.2 Heat map loss. Let the center coordinates of the detected target i in the image be The coordinates of the top left corner and the bottom right corner of the target box are where and The output of the heat map has d times down-sampling, so the center coordinates of target i correspond to the mapping coordinates on the heat map Finally, the response of the heat map (x, y) coordinate point can be obtained as follows:

[0071]

[0072] where σ p is the standard deviation. Finally, the heat map loss formula can be obtained as follows:

[0073]

[0074] where is the estimated heat map, and α and β are manually set hyperparameters.

[0075] Step 3.3: Center bias loss. The center bias O can be calculated from the true value label, and its expression is as follows:

[0076]

[0077] where p is the coordinate of the key target center point in the true value, is the coordinate mapping of the center point after down-sampling. In this paper, L1 loss is used for training, so the final center bias loss expression is as follows:

[0078]

[0079] where is the center bias output by the prediction head.

[0080] Step 3.4: Target width and height loss. The target width and height loss uses L1 loss for training, and the expression is as follows:

[0081]

[0082] where is the target box width and height output by the prediction head, and S i is the width and height of the target box in the true value.

[0083] Step 3.5: Target re-identification feature loss. The target re-identification feature problem is converted into a classification problem, i.e., targets with the same ID are a class. By using the cross-entropy loss function, the cosine distance between vehicle features with the same ID is smaller, while the cosine distance between vehicles with different IDs is larger, so as to improve the accuracy of target association matching results. In this paper, the original loss function is replaced by AdaFace loss and several similar loss functions for experiments. And finally verify the improvement of this method on the multi-target tracking task. The loss of this branch is as follows:

[0084]

[0085] Therefore, combined with the branch weight coefficient τ of the multi-task training, the final loss function of the framework is:

[0086] L total = L heatmap + τ offset L offset + τ wh L wh + τ ReID L ReID

[0087] Step 3.6: In order to obtain the optimal network parameters of the multi-target detection and tracking network, a batch of training data sets are randomly selected to input the network. The loss function L total is calculated. The Adam optimizer is used to optimize the multi-task learning network parameters. After 30 rounds of training, the parameters of the multi-task learning network gradually converge and approach the optimal value.

[0088] Step 4: Test the network performance using the test data set. The network is the neural network processing part in the multi-target detection and tracking task, and the post-processing can be completed in multiple ways. In this place, the post-processing in FairMOT is used for detection and tracking to facilitate comparison with other mainstream methods. The test samples are input into the network frame by frame and post-processed to generate the driving trajectory of each vehicle target. The multi-target tracking accuracy (MOTA), multi-target tracking precision (MOTP), ID total switching number (IDs), model frame processing rate (FPS), and model parameter quantity and calculation amount commonly used in MOT challenge are used as evaluation indexes to summarize the tracking results.

[0089] The specific tracking results of the application are described using the pre-divided UA-DETRAC test data set.

[0090] Network parameter setting: The lightweight network is essentially a multi-task learning framework, in which the branch loss balancing constant is set as follows: the coefficient of the center paranoid loss is set to 1, the coefficient of the target width and height loss is set to 0.5, and the coefficient of the target re-identification feature loss is set to 0.001. In the training process, the initial learning rate is 0.0005, the Adam optimizer is used, the learning rate is reduced by 1 / 10 every 10 Epoch, the Batch Size is set to 16 and the network is trained for 30 Epoch.

[0091] Example identification results:

[0092] For the test data set of 20 videos, tracking processing is performed respectively, and the tracking results of the 20 videos are summarized and averaged to obtain the final result. The recognition result is shown in Table 1. The lightweight multi-target detection and tracking network in the application achieves an inference speed of 17.7 FPS in the edge development board JETSON NX. The actual processing screen of the application algorithm applied to the vehicle flow counting is as shown in Figure 2

[0093] Table 1 is a backbone network structure table of the application.

[0094]

[0095] Table 2 is a measured speed and accuracy result table of the method of the application.

[0096] ​

Claims

1. A lightweight multi-target detection and tracking method for road video stream vehicle monitoring, the method comprising: Step 1: constructing a training and testing dataset sample; Based on the UA-DETRAC dataset, 80 videos are selected as the training set and 20 videos are selected as the testing set; Step 1.1: read the original picture file and the corresponding label file of each video, modify the directory structure to adapt to the data loader used in this paper, and store it in the cache for subsequent steps; Step 1.2: parse the xml label file corresponding to each video, according to the ignore area marked in the label file, shield the occlusion for each frame of the video, and store the preprocessed picture in the dataset folder after renaming; Step 1.3: for the xml label file corresponding to each video, parse the target information contained in each frame, and store an independent label file for each frame according to the specified format; Step 1.4: traverse the folder where the label file is located and the folder where the picture file is located, compare the number of files in the two folders, and according to the label file, determine whether the corresponding video frame contains the target to be recognized, and delete the empty video frame without the target to be recognized; Step 1.5: for the testing set, the judgment of tracking performance indicators requires special true value label files; use the original label file to generate true value files for evaluation for each test set video; Step 2: constructing a multi-target detection and tracking neural network; The multi-target detection and tracking neural network comprises a CBRM module, a backbone network, a feature fusion network, and an output head, wherein the CBRM module is a feature extraction module comprising a 3x3 convolution layer, a BN layer, a ReLU layer, and a MaxPool layer; The backbone network comprises a first shuffle-block2 module, a first shuffle-block1 module, a second shuffle-block2 module, a second shuffle-block1 module, a third shuffle-block1 module, and a third shuffle-block1 module in sequence; The feature fusion network comprises a first Meta-C3 module, a first concat module, a second Meta-C3 module, a first Upsample module, a second concat module, a third Meta-C3 module, a third concat module and a fourth Meta-C3 module; the output of the third shuffle-block1 module in the backbone network is taken as the input of the first Meta-C3 module, the outputs of the second shuffle-block1 module and the first Meta-C3 module are taken as the input of the first concat module, and the output of the first concat module sequentially passes through the second Meta-C3 module and the first Upsample module; then the outputs of the first shuffle-block1 module and the first Upsample module are taken as the input of the second concat module, the output of the second concat module is taken as the input of the third Meta-C3 module, the outputs of the first Upsample module and the third Meta-C3 module are taken as the input of the third concat module, the output of the third concat module is taken as the input of the fourth Meta-C3 module, and the output of the fourth Meta-C3 module is connected to the output head; The output head comprises four output heads, and each output head comprises a 3x3 convolution layer, a ReLU layer and a 1x1 convolution layer. The CBRM module is sequentially connected by a 3x3 convolution layer, a batch normalization layer, an activation function layer and a maximum pooling layer, the Shuffle-block is derived from a ShuffleNetV2 network, the Shuffle-block1 module is composed of a channel splitting layer, a depth separable convolution layer and a channel scrambling layer, the Shuffle-block2 module is composed of a depth separable convolution layer and a channel scrambling layer, the Meta-C3 module is based on a C3 module structure and is improved by using an ACON series activation function, the Concat module outputs after performing a splicing operation on input features, and the Upsample module outputs after performing an up-sampling operation on input features; Step 3: training the multi-target detection and tracking network and introducing an Adaface loss; Step 3.1: the loss function of the multi-target detection and tracking network comprises four parts, which are respectively: Focal loss L of the heat map loss heatmap Regular reconstruction loss L of the center bias loss offset Regular reconstruction loss L of the target width-height loss wh Adaface loss L introduced for the target re-identification feature branch ReID Therefore, combined with the branch weight coefficient τ when training multiple tasks, the final loss function of the framework is: L total = L heatmap + τ offset L offset + τ wh L wh + τ ReID L ReID Step 3.2: In order to obtain the optimal network parameters of the multi-target detection and tracking network, a batch of training data sets are randomly selected to input the network; the loss function L is calculated total , and the multi-task learning network parameters are optimized using the Adam optimizer; Step 4: testing the test set by using the trained multi-target detection and tracking network.

2. The lightweight multi-target detection and tracking method for road video stream vehicle monitoring of claim 1, wherein, The focal loss L of the heat map loss heatmap The calculation method is: Let the center coordinate of the detected target i in the image be The coordinates of the top-left and bottom-right points of the target frame are located as where And The output of the heat map is down-sampled by d times, so the corresponding mapping coordinates of the center coordinate of the target i on the heat map are The response at the heat map (x, y) coordinate point is finally obtained as where σ p is the standard deviation, and the final heat map loss formula is as follows: wherein are hyperparameters for the estimated heat map.

3. The lightweight multi-target detection and tracking method for road video stream vehicle monitoring of claim 1, wherein, the regularized reconstruction loss L of the center paranoia loss offset The calculation method is: The center bias quantity O obtained from the true value label can be calculated, and the expression is as follows: Wherein, p is the coordinate of the key target center point in the true value, The coordinate mapping of the center point obtained after downsampling is trained using L1 loss, so the final center bias loss expression is as follows: wherein is the center bias of the prediction head output.

4. The lightweight multi-target detection and tracking method for road video stream vehicle monitoring of claim 1, wherein, the regularized reconstruction loss L of the target width-height loss wh The calculation method is: The target width-height loss is trained by using an L1 loss, and the expression is as follows: wherein S are the width and height of the target box in the ground truth. i S are the width and height of the target box in the ground truth.

5. The lightweight multi-target detection and tracking method for road video stream vehicle monitoring of claim 1, wherein, The target re-identification feature branch introduces an Adaface loss L ReID The calculation method is: where s is a hyperparameter set manually, θ j is the angle between the weight of the jth category and the feature vector; the model-related quantity The image quality vector expression in a batch training can be obtained as follows: where μ z is the mean of z i in a training batch, σ z is the standard deviation of ||z i || in a training batch, and the expression is shown below; h is a manually set parameter to limit the final value of μ to be between ±1; and an exponential moving average is used to avoid the instability of μ z and σ z when the training batch is too small. μ z = a μ z (k) + (1 - a) μ z (k-1) σ z = ασ z (k) + (1 - α)σ z (k-1) where k is the order of stabilization required, and a is a manually set parameter, which is obtained by completing the definition of the image quality adaptive function g angle and g add with respect to the manually set parameter m, as follows:

Images