A real-time multi-category multi-target tracking method in tunnels

Abstract

The present invention relates to the field of computer vision technology, and in particular to a real-time multi-category multi-target tracking method in a tunnel: the method comprises step 1: training after fusing a detection network with a feature extraction network to obtain a trained fusion network; step 2: inputting a lane image in the tunnel into the fusion network, outputting a prediction result of a target in an area of ​​interest and saving a global feature map, wherein the prediction result includes target category, confidence and location information; step 3: performing non-maximum suppression on the prediction result to screen out the detection target; step 4: intercepting the target feature in the global feature map according to the location information of the detection target; step 5: obtaining a target feature vector corresponding to each target through an SPP network for each target feature map; and step 6: inputting the prediction result after maximum suppression and the target feature vector into a tracker for target tracking. The present invention effectively improves the efficiency and accuracy of multi-target tracking in complex tunnel scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a real-time multi-category multi-target tracking method in a tunnel. Background Art

[0002] Multi-target tracking is one of the current research hotspots. It is the basis for in-depth mining of vehicle driving conditions (such as vehicle speed) and analysis of abnormal vehicle behaviors (such as abnormal parking and crossing of solid lines) in traffic surveillance videos. However, the current video multi-target tracking technology still has some difficulties, such as the long tracking time and low target tracking accuracy in complex scenes. Tunnel scenes are complex, with serious glare interference and small distant targets, which puts higher requirements on the accuracy and real-time performance of vehicle multi-target tracking. The existing multi-target tracking algorithms either have insufficient tracking accuracy, or do not meet the real-time performance of engineering practice, or can only be applied to single-category multi-target tracking, and are difficult to apply to tunnel video surveillance intelligent processing systems and meet the requirements of tunnel scenes. Summary of the invention

[0003] The purpose of the present invention is to provide a real-time multi-category multi-target tracking method in a tunnel, which effectively improves the processing efficiency of vehicle multi-target tracking, realizes simultaneous tracking of multiple targets of multiple categories, and is suitable for complex tunnel scenes.

[0004] In order to solve the above technical problems, the technical solution of the present invention is: a real-time multi-category multi-target tracking method in a tunnel, comprising:

[0005] Step 1: Fuse the detection network and the feature extraction network and then train them to obtain a trained fusion network;

[0006] Step 2: After the tunnel lane image is input into the fusion network, the prediction results of the targets in the area of ​​interest are output and the global feature map is saved. The prediction results include target category, confidence and location information;

[0007] Step 3: Perform non-maximum suppression on the prediction results to screen out the detection targets;

[0008] Step 4: Extract target features from the global feature map based on the location information of the detected target;

[0009] Step 5: Each target feature map is used to obtain the target feature vector corresponding to each target through the SPP network;

[0010] Step 6: Input the prediction result after maximum value suppression and the target feature vector into the tracker for target tracking.

[0011] Furthermore, the detection network is SSD or YOLO series.

[0012] Furthermore, when training the fusion network, a corresponding training set is constructed, and the training set includes target categories, ids, and coordinates.

[0013] Furthermore, the tracker is a tracker based on the DeepSORT algorithm.

[0014] Furthermore, step 1 is specifically as follows:

[0015] Step 1.1: Input the image into the fusion network to obtain the FPN global feature map of the input image;

[0016] Step 1.2: Obtain the category, confidence and location information of the target prediction;

[0017] Step 1.3: Determine whether to train. If not, execute step 2; if training, execute the training steps of step 1.4; Step 1.4: Train the fusion network, specifically:

[0018] Extract target features in the global feature map according to the predicted coordinates;

[0019] Obtain the target feature vector corresponding to each target through the SPP network;

[0020] Calculate the loss and train the network by backpropagation, that is, modify the network parameters according to the loss value.

[0021] Furthermore, the training steps of step 1.4 are specifically as follows:

[0022] Step 1.41: The input image is passed through the detection network to obtain the predicted feature map, thereby obtaining the prediction result, namely the category, confidence and location information;

[0023] Step 1.42: Perform 2X upsampling on the 8X downsampled feature map C4 of the detection network to obtain C4'; perform 1*1 convolution on the 4X downsampled feature map C3 of the detection network to obtain C3', and concatenate C3' and C4' to obtain the feature map P3;

[0024] Step 1.43: The feature map P3 is upsampled by 2X to obtain P3', the feature map C2 is convolved by 1*1 to obtain C2', P3' and C2' are concatenated to obtain the feature map P2, the size of P2 is w*h*c, where w represents width, h represents height, and c represents channel;

[0025] Step 1.44: Perform convolution operation on the feature map P2 to obtain the global feature map F2;

[0026] Step 1.45: Map the predicted coordinates of the n targets of the detection network to the global feature map F2;

[0027] Step 1.46: According to the position of the target in the global feature map, the target is cut out from the global feature map to obtain the feature map F_Xi of the i-th target;

[0028] Step 1.47: Each target feature map is passed through the SPP network to obtain the feature vector F_Vi;

[0029] Step 1.48: Repeat step 1.47 to obtain the feature vectors F_Vi corresponding to n targets, perform concatenation operation, and then perform BN operation to achieve normalization to obtain the feature vector F_Vn;

[0030] Step 1.49: Connect the feature vector F_Vn to a fully connected layer. The output size of the fully connected layer is the maximum value of the training set id id_max+2, and the size is n*(id_max+2);

[0031] Step 1.410: According to formula (1), the cross entropy loss is calculated for n*(id_max+2) and the real id label values ​​corresponding to the n targets, where no loss is calculated when id is -1, and L is obtained. id ;

[0032]

[0033] In the formula, M is the number of categories, N is the number of samples, and y ic is the symbolic function, p ic is the predicted probability that the observed sample i belongs to category c;

[0034] Step 1.411: Use multi-task loss training to compare the original loss function of the detection network with L id Carry out training; the specific formula is as follows:

[0035]

[0036] In the formula, α j is the weight value of the j-th loss, is the jth loss value of the ith sample, including prediction box loss, confidence loss, classification loss and id label value loss, L total is the total loss.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention firstly integrates the detection network with the feature extraction network, which greatly reduces the tracking time and can track multiple targets of multiple categories at the same time. Secondly, it also optimizes the feature extraction branch of the fusion network by means of FPN, SPP, cross entropy loss training, etc., so that the network extracts features more accurately, which can effectively improve the processing efficiency of the multi-target tracking algorithm and realize the simultaneous tracking of multiple targets of multiple categories, which is suitable for complex tunnel scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the real-time multi-category multi-target tracking method in a tunnel of the present invention;

[0040] Figure 2 This is a structural diagram of the detection and feature extraction fusion network of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Please refer to Figure 1 and Figure 2 The present invention is a real-time multi-category multi-target tracking method in a tunnel, which comprises:

[0043] Step 1: Fuse the detection network and the feature extraction network and train them to obtain a trained fusion network; the selected detection network is SSD or YOLO series; Step 1 is specifically as follows:

[0044] Step 1.1: Input the image into the fusion network to obtain the FPN global feature map of the input image;

[0045] Step 1.2: Obtain the category, confidence and location information of the target prediction;

[0046] Step 1.3: Determine whether to train. If not, execute step 2; if training, execute the training step of step 1.4;

[0047] Step 1.4: Train the fusion network: intercept the target features in the global feature map according to the predicted coordinates; obtain the target feature vector corresponding to each target through the SPP network; calculate the loss, back-propagate the training network, that is, modify the network parameters according to the loss value; when training the fusion network, construct the corresponding training set, which includes the target category, id and coordinates; Step 1.4 specifically includes:

[0048] Step 1.41: The input image is passed through the detection network to obtain the predicted feature map, thereby obtaining the prediction result, namely the category, confidence and location information;

[0049] Step 1.42: Perform 2X upsampling on the 8X downsampled feature map C4 of the detection network to obtain C4'; perform 1*1 convolution on the 4X downsampled feature map C3 of the detection network to obtain C3', and concatenate C3' and C4' to obtain the feature map P3;

[0050] Among them, the downsampling operation can be understood as compressing the image. The original image is wide w and high h. After 2X downsampling, the width and height are reduced by half; the 4X downsampling feature map is the original image width and height divided by 4; the 8X downsampling feature map is the original image width and height divided by 8; the upsampling operation can be understood as filling the image. The original feature map is wide w and high h. After 2X upsampling, the feature map with a width of 2w and a height of 2h is obtained;

[0051] Among them, the size of C3 is w / 4, h / 4, and 1*1 convolution is performed, which means that its width and height remain unchanged, but the channel dimension changes, for example, from 2c to c. Channel can be understood as thickness, so that C3' and C4'channel dimensions are the same, both are c, and then spliced ​​to obtain w / 4, h / 4, channel 2c feature map;

[0052] Step 1.43: The feature map P3 is upsampled by 2X to obtain P3', the feature map C2 is convolved by 1*1 to obtain C2', P3' and C2' are concatenated to obtain the feature map P2, the size of P2 is w*h*c, where w represents width, h represents height, and c represents channel;

[0053] Step 1.44: Perform a convolution operation on the feature map P2, the convolution kernel input feature map size is 3*3, padding is 1, step size is 1, input channel is c, output channel is 128, and a global feature map F2 of size w*h*128 is obtained;

[0054] Step 1.45: Map the predicted coordinates of the n targets of the detection network to the global feature map F2;

[0055] Step 1.46: According to the position of the target in the global feature map, the target is cut out from the global feature map to obtain the feature map F_Xi of the i-th target, and the size of F_Xi is wi*hi*128;

[0056] Step 1.47: Each target feature map passes through the SPP network (in this embodiment, the feature map is directly divided into 1*1 feature blocks and a maximum pooling operation is performed) to obtain a 1*128 feature vector F_Vi;

[0057] The SPP itself is divided into the following steps:

[0058] (1) The feature map is divided into 1*1 feature blocks, and each block is pooled to obtain a 1*128 feature vector;

[0059] (2) The feature map is divided into 2*2 feature blocks, and each block is pooled to obtain a 4*128 feature vector;

[0060] (3) The feature map is divided into 4*4 feature blocks, and each block is pooled to obtain a 16*128 feature vector;

[0061] (4) Concatenate all the feature vectors of 1*128, 4*128, and 16*128 to obtain a feature vector of 21*128;

[0062] In this embodiment, for convenience, the feature map is only divided into 1*1 feature blocks to obtain a 1*128 feature vector;

[0063] Step 1.48: Repeat step 1.47 to obtain the feature vectors F_Vi corresponding to n targets, perform concatenation operation, and then perform BN operation to achieve normalization, and obtain the feature vector F_Vn of size n*128;

[0064] Step 1.49: Connect the feature vector F_Vn to a fully connected layer. The input size of the fully connected layer is 128, and the output size is the maximum value of the training set id id_max+2, resulting in a size of n*(id_max+2);

[0065] Step 1.410: According to formula (1), the cross entropy loss is calculated for n*(id_max+2) and the real id label values ​​corresponding to the n targets, where no loss is calculated when id is -1, and L is obtained. id ;

[0066]

[0067] In the formula, M is the number of categories, N is the number of samples, and y ic is a sign function (1 if the true category of the observed sample i is c, otherwise 0), p ic is the predicted probability that the observed sample i belongs to category c;

[0068] Step 1.411: Use multi-task loss training to compare the original loss function of the detection network with L id Carry out training; the specific formula is as follows:

[0069]

[0070] In the formula, α j is the weight value of the j-th loss, is the jth loss value of the ith sample, including prediction box loss, confidence loss, classification loss and id label value loss, L total is the total loss.

[0071] Step 2: After the lane image in the tunnel is input into the fusion network, the prediction results of the targets in the area of ​​interest are output and the global feature map is saved, where the prediction results include target category, confidence and location information; the lane image in the tunnel is obtained through the video monitoring of the camera in the tunnel;

[0072] Step 3: Perform non-maximum suppression (NMS) on the position coordinates in the prediction results to screen out accurate detection targets;

[0073] Step 4: Map the position coordinate information of the detected target to the global feature map, and extract the target feature F_Xi from the global feature map;

[0074] Step 5: Each target feature map is passed through the SPP network to obtain the target feature vector F_Vi corresponding to each target; repeat this step to obtain the feature vector F_Vi corresponding to n targets;

[0075] Step 6: Input the prediction result (category, confidence, location information) after maximum value suppression and the target feature vector into the tracker for target tracking; in this embodiment, the tracker is a tracker based on the DeepSORT algorithm.

[0076] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A real-time multi-category multi-target tracking method in a tunnel, characterized by: include Step 1: Fuse the detection network and the feature extraction network and train them to obtain a trained fusion network; specifically: Step 1.1: Input the image into the fusion network to obtain the FPN global feature map of the input image; Step 1.2: Obtain the category, confidence and location information of the target prediction; Step 1.3: Determine whether to train. If not, execute step 2; if training, execute the training steps of step 1.4; Step 1.4: Train the fusion network, specifically: Extract target features in the global feature map according to the predicted coordinates; Obtain the target feature vector corresponding to each target through the SPP network; Calculate the loss and back-propagate the training network, that is, modify the network parameters according to the loss value; Step 2: After the tunnel lane image is input into the fusion network, the prediction results of the targets in the area of ​​interest are output and the global feature map is saved. The prediction results include target category, confidence and location information; Step 3: Perform non-maximum suppression on the prediction results to screen out the detection targets; Step 4: Extract target features from the global feature map based on the location information of the detected target; Step 5: Each target feature map is used to obtain the target feature vector corresponding to each target through the SPP network; Step 6: Input the prediction result after maximum value suppression and the target feature vector into the tracker for target tracking.

2. The real-time multi-category multi-target tracking method in a tunnel according to claim 1, characterized in that: The detection network is SSD or YOLO series.

3. The real-time multi-category multi-target tracking method in a tunnel according to claim 1, characterized in that: When training the fusion network, a corresponding training set is constructed, which includes target categories, ids, and coordinates.

4. The real-time multi-category multi-target tracking method in a tunnel according to claim 1, characterized in that: The tracker is a tracker based on the DeepSORT algorithm.

5. The real-time multi-category multi-target tracking method in a tunnel according to claim 1, characterized in that: The training steps of step 1.4 are specifically as follows: Step 1.41: The input image is passed through the detection network to obtain the predicted feature map, thereby obtaining the prediction result, namely the category, confidence and location information; Step 1.42: Perform 2X upsampling on the 8X downsampled feature map C4 of the detection network to obtain C4'; perform 1*1 convolution on the 4X downsampled feature map C3 of the detection network to obtain C3', and concatenate C3' and C4' to obtain the feature map P3; Step 1.43: Perform 2X upsampling on feature map P3 to obtain P3', perform 1*1 convolution on feature map C2 to obtain C2', concatenate P3' and C2' to obtain feature map P2, the size of P2 is w*h*c, where w represents width, h represents height, and c represents channel; Step 1.44: Perform convolution operation on feature map P2 to obtain global feature map F2; Step 1.45: Map the predicted coordinates of the n targets of the detection network to the global feature map F2; Step 1.46: According to the position of the target in the global feature map, the target is cut out from the global feature map to obtain the feature map F_Xi of the i-th target; Step 1.47: Each target feature map is passed through the SPP network to obtain the feature vector F_Vi; Step 1.48: Repeat step 1.47 to obtain the feature vectors F_Vi corresponding to n targets, perform concatenation operation, and then perform BN operation to achieve normalization to obtain the feature vector F_Vn; Step 1.49: Connect the feature vector F_Vn to a fully connected layer. The output size of the fully connected layer is the maximum value of the training set id id_max+2, and the size is n*(id_max+2); Step 1.410: According to formula (1), the cross entropy loss is calculated for n*(id_max+2) and the real id label values ​​corresponding to the n targets, where no loss is calculated when id is -1, and L is obtained. id ; In the formula, M is the number of categories, N is the number of samples, and y ic is the symbolic function, p ic is the predicted probability that the observed sample i belongs to category c; Step 1.411: Use multi-task loss training to compare the original loss function of the detection network with L id Carry out training; the specific formula is as follows: In the formula, α j is the weight value of the j-th loss, is the jth loss value of the ith sample, including prediction box loss, confidence loss, classification loss and id label value loss, L total is the total loss.

Images