Urban road target detection and tracking identification method based on Deepsort algorithm

Abstract

The invention provides an urban road target detection and tracking identification method based on a Deepsort algorithm, and the method comprises the steps: obtaining an urban road target image, carrying out the preprocessing of the obtained image, constructing an urban road target detection data set, constructing an urban road target detection model and an urban road target detection and tracking model, and training the urban road target detection model based on the urban road target detection data set, training the urban road target detection tracking model based on a preset urban road target detection tracking data set to obtain a model, and inputting a to-be-detected video into the final urban road target detection model and the urban road target detection tracking model. And performing urban road target detection through the urban road target detection model, tracking a detection result through the urban road target detection tracking model based on the detection result, and outputting a tracking result. According to the method, the target tracking precision is improved by optimizing the target detection and tracking process.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a method for detecting, tracking and identifying urban road targets based on a Deepsort algorithm. Background Art

[0002] In Intelligent Transportation Systems (ITS), accurate identification and tracking of vehicles, pedestrians and non-motor vehicles on the road is a key technology for achieving road safety and traffic management.

[0003] At present, the existing urban road target detection and tracking recognition methods have poor target detection and tracking performance due to factors such as complex road environment, illumination changes, and mutual occlusion of targets. There are problems such as large amount of calculation and low precision, which are very inconvenient to use. Therefore, it is very necessary to design an urban road target detection and tracking recognition method based on Deepsort algorithm. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a method for urban road target detection and tracking and identification based on the Deepsort algorithm.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for detecting, tracking and identifying urban road targets based on Deepsort algorithm, comprising:

[0007] Acquire urban road target images;

[0008] Preprocess the acquired images and construct a data set for urban road target detection;

[0009] Construct urban road target detection model and urban road target detection and tracking model;

[0010] The urban road target detection model is trained based on the urban road target detection data set, and the urban road target detection and tracking model is trained based on the preset urban road target detection and tracking data set to obtain the final urban road target detection model and urban road target detection and tracking model;

[0011] The video to be detected is input into the final urban road target detection model and the urban road target detection and tracking model. The urban road target detection model is used to detect the urban road target. Based on the detection result, the detection result is tracked by the urban road target detection and tracking model, and the tracking result is output.

[0012] Preferably, obtaining the urban road target image is specifically as follows:

[0013] The target images of urban roads are obtained by taking photos, downloading from the Internet and obtaining public data sets, and the images are filtered according to six categories: car, person, bus, bicycle, train and motorcycle, covering multiple weather scenes such as sunny, cloudy, rainy and foggy days.

[0014] Preferably, the acquired images are preprocessed to construct a data set for urban road target detection, specifically:

[0015] Obtain low-light images from urban road target images and perform image enhancement processing on them;

[0016] Randomly extract some images from the urban road target image, and perform image noise addition and image blurring processing on them;

[0017] Based on the LabelImg annotation tool, the original urban road target images, the low-light images after image enhancement, and some images after image noise addition and image blurring are annotated to construct an urban road target detection dataset.

[0018] Preferably, a low-light image in the urban road target image is obtained and image enhancement processing is performed on it, specifically as follows:

[0019] Build an Image-to-patch enhancement model and pre-train it based on a preset dataset;

[0020] Obtain low-light images of urban road targets and input them into the pre-trained Image-to-patch enhancement model to obtain processed low-light images.

[0021] Preferably, the urban road target detection model is an improved YOLOv4 model, and the specific improvements include:

[0022] Replace the standard convolution of the traditional YOLOv4 model with depth-wise separable convolution;

[0023] Based on the three output layers of the traditional YOLOv4 model, an output layer with a grid size of 104×104 is added;

[0024] Part of the Inceptionv3 structure is introduced into the Head part of the traditional YOLOv4 model.

[0025] Preferably, the urban road object detection and tracking model is a DeepSort model.

[0026] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0027] The present invention provides a method for detecting, tracking and identifying urban road targets based on a Deepsort algorithm, the method comprising: acquiring an urban road target image, preprocessing the acquired image, constructing an urban road target detection data set, constructing an urban road target detection and tracking model, training the urban road target detection and tracking model based on the urban road target detection data set, obtaining a final urban road target detection and tracking model, inputting a video to be detected into the final urban road target detection and tracking model, and obtaining a detection result. The present invention improves the accuracy of target tracking by optimizing the target detection and tracking process, and is easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of the Image-to-patch enhancement model training strategy;

[0031] Figure 3 This is a schematic diagram of the Patch-net network structure;

[0032] Figure 4 This is a schematic diagram of the comparison of Image-to-patch enhancement models;

[0033] Figure 5 Schematic diagram of image noise addition (a is the original image, b is the image after adding salt and pepper noise);

[0034] Figure 6 Schematic diagram of image blur (a is the original image, b is the image after adding Gaussian blur);

[0035] Figure 7 This is a schematic diagram of the standard convolution process;

[0036] Figure 8 Schematic diagram of the depth-wise separable convolution process;

[0037] Fig. 9 This is a schematic diagram of the YOLO Head structure;

[0038] Fig.10 This is a schematic diagram of the Inceptionv3 Head structure;

[0039] Fig.11 To improve the YOLOv4 network framework diagram;

[0040] Figure 12 is a schematic diagram of the effect of the improved YOLOv4 model (a is the original image, b is a schematic diagram of the effect of the YOLOv4 model, and c is a schematic diagram of the effect of the improved YOLOv4 model);

[0041] Fig.13 Schematic diagram of the target tracking framework based on improved YOLOv4 and DeepSort;

[0042] Fig.14 The following is a schematic diagram of the tracking process for Case 1 and Case 2;

[0043] Fig.15 This is a schematic diagram of the tracking process for situation three;

[0044] Fig.16 This is a schematic diagram of the tracking process for Case 4 and Case 5. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] The purpose of the present invention is to provide a method for urban road target detection and tracking and identification based on the Deepsort algorithm, which can improve the accuracy of target tracking and facilitate use by optimizing the target detection and tracking process.

[0047] Figure 1 A flow chart of a method provided by an embodiment of the present invention, such as Figure 1 As shown, the present invention provides a method for detecting and tracking urban road targets based on a Deepsort algorithm, comprising:

[0048] Step 100: Acquire a city road target image;

[0049] Step 200: preprocessing the acquired image to construct an urban road target detection dataset;

[0050] Step 300: constructing a city road target detection model and a city road target detection and tracking model;

[0051] Step 400: training an urban road target detection model based on an urban road target detection data set, and training an urban road target detection and tracking model based on a preset urban road target detection and tracking data set, to obtain a final urban road target detection model and an urban road target detection and tracking model;

[0052] Step 500: Input the video to be detected into the final urban road target detection model and the urban road target detection and tracking model, perform urban road target detection through the urban road target detection model, and based on the detection result, track the detection result through the urban road target detection and tracking model, and output the tracking result.

[0053] In step 100, a city road target image is obtained, specifically:

[0054] The target images of urban roads are obtained by taking photos, downloading from the Internet and obtaining public data sets, and the images are filtered according to six categories: car, person, bus, bicycle, train and motorcycle, covering multiple weather scenes such as sunny, cloudy, rainy and foggy days.

[0055] In step 200, the acquired image is preprocessed to construct an urban road target detection dataset, specifically:

[0056] Step 201: Acquire a low-light image in a city road target image and perform image enhancement processing on it;

[0057] Step 202: randomly extracting a portion of the image from the urban road target image, and performing image noise addition and image blurring processing on the portion;

[0058] Step 203: Based on the LabelImg annotation tool, the original urban road target image, the low-light image after image enhancement processing, and the partial image after image noise addition processing and image blur processing are annotated to construct an urban road target detection dataset.

[0059] In step 201, a low-light image in the urban road target image is obtained and image enhancement processing is performed on it, specifically:

[0060] Build an Image-to-patch enhancement model and pre-train it based on a preset dataset;

[0061] Obtain low-light images of urban road targets and input them into the pre-trained Image-to-patch enhancement model to obtain processed low-light images;

[0062] A detailed introduction to the Image-to-patch enhancement model:

[0063] The image-to-patch enhancement model is mathematically expressed as:

[0064] H(x,y)=L(x,y)+I×P(x,y)

[0065] Among them, L(x, y) is the input low-light image, P(x, y) is the patch map, which is the illumination estimation map. The intermediate process map in the dark area mainly contains the incremental information of the low-light image restored to the normal image, and the normal area is mainly to maintain the harmony between the dark area and the normal area. Therefore, the pixel values ​​corresponding to the normal area in the patch map are mainly close to 0 or directly equal to 0, which can well protect the highlight information in the original image.

[0066] By fusing with the original image, the image details in the dark are restored. I is the enhancement level, which constrains the strength of the patch image, mainly to constrain the degree of enhancement of dark information, and prevent excessive enhancement of dark parts from causing overall image disharmony. In addition, the introduced enhancement level I is an adjustable parameter, which can effectively improve the generalization ability of the enhancement network. The existing end-to-end enhancement method can often only process low-light images with brightness similar to that of the training set. Under the image-to-patch model, by adjusting the overall strength of the patch image, the brightness enhancement is made more flexible to meet the image enhancement needs under different exposure conditions.

[0067] Under the image-to-patch enhancement model, the convolutional neural network needs to learn the mapping from the low-light image to the patched image. The optimization goal of the network is mathematically described as:

[0068] argmin[Y(x,y)-L(x,y)-I×P(x,y)]

[0069] Among them, Y(x, y) is the reference image, L(x, y) is the input low-light image, P(x, y) is the patch image converted by the convolutional neural network, and I is the enhancement level. It is set to 1 in the training stage and the maximum value of this value is 1 in the test stage. This value can be quantized into 10 levels. By setting this value, the brightness enhancement level can be constrained. Under the image-to-patch enhancement model, the network training strategy is as follows Figure 2 shown.

[0070] The strategy in the network training stage can be understood as residual learning. By learning the residuals of normal images and low-light images, the network can speed up the convergence of the network and reduce the focus on normal areas. Therefore, a lighter network structure can be designed on this basis.

[0071] In order to reduce resources, the present invention proposes that the Patch-net network consists of only 6 layers of convolution, and its structure is as follows Figure 3 As shown;

[0072] The network is mainly composed of three parts: feature extraction, feature mapping and feature reconstruction. The specific network parameters of the network are shown in Table 1. First, a feature extraction layer composed of three 3×3 convolutions + PReLU is used to extract the input image features, and the dimension of the feature map is increased to 64. Then, a 1×1 convolution is used to perform spatial transformation, and the extracted 64 feature maps are converted into 32, and the image feature domain is converted into a residual domain. Then, two 3×3 convolutions are used to reconstruct the patch map to predict the generated patch map. For the generation of the patch map, the present invention does not limit the value range of pixel values. By using PReLU instead of ReLU activation function, the patch map may also have negative values. The final enhanced image is obtained by fusing the patch map with the input image.

[0073] Table 1 Patch-net network parameter setting table

[0074]

[0075]

[0076] In order to better generate a high-quality normally exposed image by fusing the patch map generated by the network constraints with the input image, the L1 loss function and the SSIM loss function are used in the training stage, where the L1 loss function is expressed as:

[0077]

[0078] Among them, F(X org ,θ) is the predicted output graph of the network, X org is the low illumination image, Y ref is a reference image, and ||·||1 is the first norm. The main reason for using L1 loss instead of L2 loss is that the network mainly focuses on the detail information of the dark area. After merging with the dark part of the input image, the details need to be restored. If L2 loss is used, the detail information of the dark area will be too smooth, making the restored dark area blurred. In addition to L1 loss, the present invention also uses SSIM loss function to constrain the overall structural information of the image. SSIM can be expressed as:

[0079]

[0080] Among them, z is the enhanced image after the fusion of the patch image and the input image, y is the reference image, μ z is the average pixel value of the enhanced image, μ y is the pixel average of the reference image, and are the variances of the enhanced image and the reference image, σ z,yrepresents the covariance of the enhanced image and the reference image, C1 and C2 are two constants to avoid division by 0, and the SSIM loss function is expressed as:

[0081]

[0082] Among them, z i is the i-th output image, y i is the i-th reference image. After many experiments, the present invention sets the weight of the SSIM loss function to 0.1 to achieve the best effect. The loss function of the entire network is expressed as:

[0083] Loss = L1 + 0.1 × L SSIM

[0084] The present invention provides an embodiment, by comparing with existing excellent algorithms, these algorithms include Dong, LIME, SRIE, GLADNet, NPE and BIMEF, the comparison diagram is as follows Figure 4 As shown, the effect achieved by the method described in the present invention is better.

[0085] In step 202, a portion of the image in the urban road target image is randomly selected, and image noise and image blur processing are performed on it, specifically:

[0086] Randomly extract some images from the urban road target image, and perform image noise processing on half of them, such as adding salt and pepper noise. The detailed introduction is as follows: Salt and pepper noise is a mixed noise containing both pepper noise and salt noise, where pepper noise appears as black spots in the image and salt noise appears as white spots in the image. The addition of salt and pepper noise is achieved by randomly obtaining image pixels and setting them as high brightness points and low grayscale points. The image with added salt and pepper noise has a certain similarity with the image affected by noise, and can well simulate the effect of noisy images, thereby enhancing the robustness of deep learning models for noisy images and blurred images. The results of adding salt and pepper noise to the images in the dataset are shown in the figure. Figure 5 As shown;

[0087] The other half of the image is blurred, such as Gaussian blur. The detailed introduction is as follows: Gaussian blur is a commonly used data smoothing method in image processing, which can make the image smoother and more blurred. The specific steps are to first scan each pixel in the image, and use the Gaussian curve to perform weighted average processing on the pixels around the image, and then remove the details in the image to make the image blurred. The image obtained by Gaussian blur has a certain similarity with the image obtained due to vehicle shaking and excessive speed, and can simulate the blurred image well, thereby enhancing the robustness of the deep learning model for blurred images. The results of the images in the dataset after Gaussian blur are as follows Figure 6 As shown;

[0088] In step 203, the original urban road target image, the low-light image after image enhancement processing, and the partial image after image noise addition processing and image blur processing are labeled based on the LabelImg labeling tool to construct an urban road target detection dataset, specifically:

[0089] All original urban road target images (including low-light images before processing and images without image noise and image blur processing) as well as low-light images after image enhancement and some images after image noise and image blur processing are obtained for annotation, and annotated based on the LabelImg annotation tool to generate an urban road target detection dataset.

[0090] The urban road target detection model is an improved YOLOv4 model;

[0091] First, the improved YOLOv4 model is introduced. The specific improvements include:

[0092] (I) Replace the standard convolution of the traditional YOLOv4 model with a depthwise separable convolution, specifically:

[0093] The YOLOv4 model uses a lot of standard 3×3 convolutions, which results in a large number of parameters and a large amount of calculation. It is not suitable for detection on devices with limited computing power and memory resources. Therefore, the depthwise separable convolution is introduced to replace the standard convolution, which greatly reduces the number of model parameters and calculation, and improves the model detection speed.

[0094] The process of standard convolution is as follows Figure 7 As shown. For example, when the input feature map is D x ×D y ×3, the feature map is convolved with N 3×3 convolution kernels respectively, and finally a new feature map with N channels is obtained.

[0095] The process of depthwise separable convolution is as follows Figure 8 As shown. When its input is also D x ×D y ×3 feature map, the depthwise separable convolution first convolves the input feature map with three 3×3 convolution kernels layer by layer to obtain a feature map with the same number of channels as the input channels, and then convolves this feature map with N 1×1 convolution kernels point by point, and finally obtains a new feature map with N channels.

[0096] The calculation formula for the convolution parameter when considering the bias is as follows:

[0097] P conv =(W×H×S+1)×N

[0098] In the formula, the parameters are:

[0099] P conv is the number of convolution parameters;

[0100] The size of the W×H convolution kernel;

[0101] S is the number of input channels of the convolution;

[0102] N is the number of output channels of the convolution.

[0103] When the input is D x ×D y ×S feature map, convolved with N 3×3 convolution kernels, and the final output is D h ×D w ×N, the number of parameters of the standard convolution is P conv The calculation formula is as follows:

[0104] P conv1 =(3×3×S+1)×N=(9S+1)×N

[0105] The depthwise separable convolution first convolves the input feature map with three 3×3 convolution kernels layer by layer, and then convolves the obtained new feature map with N 1×1 convolution kernels point by point. The parameter quantity P of the depthwise separable convolution is conv2 As shown below:

[0106] P conv2 =P1+P2=9S+(S+1)×N

[0107] In the formula, the parameters are:

[0108] P1 is the parameter amount of layer-by-layer convolution;

[0109] P2 is the parameter amount for point-by-point convolution.

[0110] Comparing the above formulas, we can get the following formula:

[0111]

[0112] Among them, k1 is the ratio of the number of parameters used in the depthwise separable convolution to the number of parameters used in the standard convolution. Since the number of input channels S and the number of output channels N in YOLOv4 are both large, k1 is much smaller than 1; the calculation formula of the convolution is as follows:

[0113] C conv =W×H×S×N×D h ×D w

[0114] In the formula, the parameters are:

[0115] C conv is the computational amount of convolution;

[0116] D h ×D w is the size of the output feature map.

[0117] When the input is D x ×D y ×S feature map, convolved with N 3×3 convolution kernels, and the final output is D h ×D w ×N, the computational cost of standard convolution Ccony1 is as follows:

[0118] C conv1 =3×3×S×N×D h ×D w =9×S×N×D h ×D

[0119] The depth-wise separable convolution first performs layer-by-layer convolution and then performs point-by-point convolution. The computational complexity of the depth-wise separable convolution is C. conv2 As shown below:

[0120] C conv2 =C1+C2=9S×D h ×D w +S×N×D h ×D w

[0121] In the formula, the parameters are:

[0122] C1 is the amount of computation during layer-by-layer convolution;

[0123] C2 is the amount of computation required for point-by-point convolution.

[0124] Comparing the above formulas, we can get the following formula:

[0125]

[0126] Among them, k2 is the ratio of the amount of computation used by the depthwise separable convolution to the amount of computation used by the standard convolution. Since the number of output channels N in YOLOv4 is large, k2 is much less than 1.

[0127] In summary, after using depthwise separable convolution, we can get the same effect as standard convolution while greatly reducing the number of parameters and calculations used for convolution. Therefore, depthwise separable convolution is introduced to replace standard convolution.

[0128] (II) Based on the three output layers of the traditional YOLOv4 model, an output layer with a grid of 104×104 is added, specifically:

[0129] In the YOLOv4 network structure, the YOLO Head part finally outputs feature maps with grids of 13×13, 26×26, and 52×52 scales. In the deep learning network, the size of the receptive field (RF) for feature maps of different scales is also different. The receptive field refers to the area size corresponding to the pixel points on the output feature map obtained by the network after the convolutional neural network on the input image. Therefore, with the difference of the receptive field, the extracted features are also relatively different. The calculation formula of the receptive field is as follows:

[0130] F i =S i (F i-1 -1)+K i

[0131] In the formula, the parameters are:

[0132] F i is the receptive field on the feature map of the i-th layer;

[0133] S i is the convolution step size of the i-th convolution layer;

[0134] F i-1 is the receptive field on the feature map of the i-1th layer;

[0135] K i is the convolution kernel size of the i-th convolution layer.

[0136] It can be seen from the formula that as the model network continues to deepen, the receptive field corresponding to its feature map is also constantly increasing. On urban traffic roads, since the acquired images often have a larger number of pixels occupied by targets closer to the camera, and targets farther away from the camera occupy a relatively smaller area in the image, in a deeper network model, more information about small target vehicles in the distance is lost, which is prone to missed detection during vehicle detection, reducing the detection performance of the model.

[0137] The head part responsible for detection in the YOLOv4 network only uses features of three scales, resulting in a limited number of grids divided, and insufficient use of shallow information in the network structure. At the same time, the YOLOv4 model has a deeper network structure. When detecting, the information of small targets in the distance is lost, which is prone to missed detection, reducing the detection performance of the YOLOv4 network model. Therefore, based on the original three output layers, an output layer with a grid of 104×104 is added. The feature layer with a grid of 52×52 in the path aggregation network PANet network is further upsampled to obtain a feature layer with a grid of 104×104, and is stacked with the feature layer with a shallow grid of 104×104 and convolved 5 times to obtain a new feature layer. On the one hand, the new feature layer is input into the detection head part for detection to obtain an output layer with a grid of 104×104, and on the other hand, it continues to downsample and stack convolution operations with the deep feature layer to provide more feature information for the detection network and improve the model's detection performance for the target.

[0138] (III) Introduce some Inceptionv3 structures into the Head part of the traditional YOLOv4 model, specifically:

[0139] In the process of head recognition by the YOLOv4 network, the detection accuracy is often related to the feature information obtained after the feature fusion network. The finer the input features, the greater the possibility of detecting the target. In general, in order to improve the detection accuracy of the model, it is usually necessary to increase the depth of the network model to obtain finer features, thereby achieving the purpose of improving the detection accuracy of the network model. However, increasing the depth of the network model will also lead to a large increase in the number of network parameters and the amount of calculation, which will lead to an increase in the model size and a decrease in the detection speed.

[0140] The Inceptionv3 network has a sparse structure and high performance of dense matrices. The Inceptionv3 network is mainly composed of different modules. The most important module uses convolution kernel sizes of 1×1, 3×3, and 5×5 to perform multiple convolutions on the input, and finally stacks and fuses the outputs containing feature information of different scales. This structure can improve the detection performance of the model without significantly increasing the amount of calculation.

[0141] When the YOLOv4 network detects the head part, it first performs a convolution with a kernel size of 3×3 on the feature map input to the head part, then normalizes the result and performs a Leaky ReLU activation function operation, and finally uses a convolution with a kernel size of 1×1 to integrate the features. The YOLO Head structure is as follows: Fig. 9 shown.

[0142] In order to further enhance the detection performance of the YOLOv4 network, the Head part of the YOLOv4 network model is improved by introducing some Inceptionv3 structures into the Head part, so that the Head part can perform convolution operations on the input through convolution kernels of different sizes during detection, thereby obtaining receptive fields of different sizes. The newly obtained Head part is denoted as Inceptionv3 Head, and its structure is as follows: Fig.10 as shown.

[0143] Depend on Fig.10 It can be seen that the Inceptionv3 Head structure has four parallel routes, in which the first route only uses a 1×1 convolution kernel to perform a convolution operation on the input once, and then normalizes the result and performs a LeakyReLU activation function operation; the second route first uses a 1×1 convolution kernel to perform a convolution operation, normalization, and a Leaky ReLU activation function operation on the input, and then uses a 5×5 convolution kernel to perform a convolution operation, normalization, and a Leaky ReLU activation function operation on the result; the third route first uses a 1×1 convolution kernel to perform a convolution operation, normalization, and a Leaky ReLU activation function operation on the input, and then uses a 3×3 convolution kernel to perform two consecutive convolution operations, normalization, and a Leaky ReLU activation function operation on the result; the fourth route first performs an average pooling operation with a pooling size of 3×3 on the input, and then uses a 1×1 convolution kernel to perform a convolution operation, normalization, and a Leaky ReLU activation function operation on the pooled result. ReLU activation function operation; finally, the outputs of these four routes are stacked and a convolution with a kernel size of 1×1 is used for feature integration. By introducing the Inceptionv3 structure in the head part, the head part can obtain receptive fields of different sizes when performing vehicle detection, and can further improve the model's detection performance on targets without significantly increasing the amount of calculation.

[0144] After the above three improvements, the network framework of the YOLOv4 model is finally improved as follows Fig.11 As shown. The depth-wise separable convolution is introduced as Fig.11 As shown in the dashed box improvement 1, the depth-separable convolution is represented by convolution_DW in the figure, and an output layer with a grid of 104×104 is added as shown in Fig.11 As shown in the dashed box improvement 2, the Inceptionv3 structure is introduced in the Head part. Fig.11 Improvement 3 is shown in the dashed box.

[0145] In the training process of the model, in a large convolutional neural network model, due to the overly complex network and too many sample data sets, such as the ImageNet data set with more than 10 million pictures, if the model is trained from scratch, a lot of time will be wasted, and if the training data set is too small, the model will easily produce overfitting when trained from scratch, reducing the training effect of the model. Therefore, the present invention uses the weights of the YOLOv4 model as the pre-training weights of the improved YOLOv4 model, which can effectively improve the training efficiency of the improved YOLOv4 model, while reducing the training time of the model, and obtaining a better training effect.

[0146] The training environment of the improved YOLOv4 model is configured. The present invention uses PyCharm to train and test the improved YOLOv4 model on a computer with Windows 10 system, where the deep learning framework is Pytorch. Other computer configurations and the required training environment are shown in Table 2.

[0147] Table 2 Experimental environment configuration table

[0148]

[0149]

[0150] The present invention provides an embodiment as shown in FIG12, wherein: Fig.12a For the original picture, Figure 12b This is a schematic diagram of the detection effect of the YOLOv4 model. Fig.12c This is a schematic diagram of the detection effect of the improved YOLOv4 model. The improved YOLOv4 model has better detection effect.

[0151] The urban road target detection and tracking model is a DeepSort model, which is introduced in detail:

[0152] The present invention mainly detects the target in the input video based on the improved YOLOv4 model, then uses the DeepSort model to track the target in the video, and finally outputs the tracking result. Fig.13 As shown;

[0153] Taking vehicle detection and tracking as an example, Fig.13 It can be seen that the tracking algorithm based on improved YOLOv4 and DeepSort has the following five situations when tracking the target:

[0154] (1) Case 1: The Kalman filter prediction result is determined and the cascade matching is successful. For the input vehicle video, the improved YOLOv4 model is first used to detect the vehicle, and the Kalman filter algorithm is used to predict the tracking frame of the target vehicle in the next frame of the input video. There are two states here: a certain state and an uncertain state. The target vehicle is predicted to be a certain state, and vice versa. When the Kalman filter prediction result is determined, the detection frame obtained by the improved YOLOv4 model is data-associated with the tracking frame predicted by the Kalman filter. The data association method is cascade matching. When the detection frame and the tracking frame are successfully matched, the Kalman filter algorithm is used to update the predicted tracking frame to reduce the error. Finally, the prediction, matching and update operations are repeated until the input video tracking is completed.

[0155] (2) Case 2: The Kalman filter prediction result is confirmed but the cascade matching result fails, and the IOU matching is successful.

[0156] In the prediction stage, the prediction result of the Kalman filter algorithm is still certain, but when the detection frame and the tracking frame are cascade matched, the match fails. The reasons for the failure of cascade matching are tracking matching failure and detection matching failure. The reason for the failure of tracking matching may be that there is a missed detection in the detection stage, resulting in the predicted tracking frame having no matching detection frame, and the reason for the failure of detection matching may be that a new target vehicle has entered the lens, resulting in no tracking frame that can be cascade matched with the detection frame. When the cascade matching fails, IOU matching will be performed. If the detection frame and the tracking frame are matched successfully, the Kalman filter algorithm will be used to update the tracking frame to reduce the error. Finally, the prediction, matching and update operations are looped until the input video tracking is completed.

[0157] The tracking process for Case 1 and Case 2 is as follows: Fig.14 shown.

[0158] (3) Case three: The Kalman filter prediction result is certain, but the cascade matching result fails, and the IOU matching result also fails. As in case two, in the prediction stage, the Kalman filter algorithm predicts a certain result; in the cascade matching stage, the match fails, and then enters the IOU matching stage. If the IOU matching also fails, determine whether it is a tracking match failure or a detection match failure. If the detection match fails, a new tracking frame is established and set to an uncertain state, and then enters the investigation stage. If three consecutive frames are hit, the uncertain state is changed to a certain state, and prediction, matching and update operations are performed; if the tracking match fails, determine whether the tracking frame is in a certain state or an uncertain state. If it is an uncertain state, the tracking frame will be deleted. If it is a certain state, see whether it exceeds the set time threshold. If it exceeds the time threshold, the tracking frame will also be deleted. If it does not exceed the time threshold, enter the three investigation stages. If the investigation is successful, prediction, matching and update operations are performed. The tracking process for case three is as follows Fig.15 shown.

[0159] (4) Case 4: Kalman filter prediction result is uncertain but IOU matching is successful. In the prediction stage, if the prediction result of the Kalman filter algorithm is uncertain, it will directly enter the IOU matching. When the detection frame and the tracking frame are successfully matched, the Kalman filter algorithm is used to update the predicted tracking frame to reduce the error. Similarly, the prediction, matching and update operations are looped until the input video tracking is completed.

[0160] (5) Case 5: Kalman filter prediction result is uncertain and IOU matching result is unsuccessful. In the prediction stage, the prediction result of the Kalman filter algorithm is uncertain, and when performing IOU matching, the detection frame and the tracking frame fail to match. In this case, as in case 3, it is determined whether the tracking match fails or the detection match fails.

[0161] The tracking process for Case 4 and Case 5 is as follows: Fig.16 shown.

[0162] The present invention provides an embodiment for comparing the tracking performance using different models. The specific comparison results are shown in Table 3.

[0163] Table 3 Comparison of tracking performance of different models

[0164]

[0165] It can be seen from Table 3 that the vehicle tracking performance of the improved YOLOv4+DeepSort model proposed in the present invention is better than that of other models.

[0166] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0167] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for detecting and tracking urban road targets based on Deepsort algorithm, characterized in that: include: Acquire urban road target images; Preprocess the acquired images and construct a data set for urban road target detection; Construct urban road target detection model and urban road target detection and tracking model; The urban road target detection model is trained based on the urban road target detection data set, and the urban road target detection and tracking model is trained based on the preset urban road target detection and tracking data set to obtain the final urban road target detection model and urban road target detection and tracking model; The video to be detected is input into the final urban road target detection model and the urban road target detection and tracking model. The urban road target detection model is used to detect the urban road target. Based on the detection result, the detection result is tracked by the urban road target detection and tracking model, and the tracking result is output.

2. The urban road target detection and tracking identification method based on Deepsort algorithm according to claim 1 is characterized in that: Get the urban road target image, specifically: The target images of urban roads are obtained by taking photos, downloading from the Internet and obtaining public data sets, and are filtered according to six categories: car, person, bus, bicycle, train and motorcycle. The images cover multiple weather scenes such as sunny, cloudy, rainy and foggy days.

3. The urban road target detection and tracking identification method based on Deepsort algorithm according to claim 1 is characterized in that: The acquired images are preprocessed to construct a data set for urban road target detection, specifically: Obtain low-light images from urban road target images and perform image enhancement processing on them; Randomly extract some images from the urban road target image, and perform image noise addition and image blurring processing on them; Based on the LabelImg annotation tool, the original urban road target images, the low-light images after image enhancement, and some images after image noise addition and image blurring are annotated to construct an urban road target detection dataset.

4. The urban road target detection and tracking identification method based on Deepsort algorithm according to claim 3 is characterized in that: Obtain the low-light image in the urban road target image and perform image enhancement processing on it, specifically: Build an Image-to-patch enhancement model and pre-train it based on a preset dataset; Obtain low-light images of urban road targets and input them into the pre-trained Image-to-patch enhancement model to obtain processed low-light images.

5. The urban road target detection and tracking identification method based on Deepsort algorithm according to claim 1 is characterized in that: The urban road target detection model is an improved YOLOv4 model, and the specific improvements include: Replace the standard convolution of the traditional YOLOv4 model with depth-wise separable convolution; Based on the three output layers of the traditional YOLOv4 model, an output layer with a grid size of 104×104 is added; Part of the Inceptionv3 structure is introduced into the Head part of the traditional YOLOv4 model.

6. The urban road target detection and tracking identification method based on Deepsort algorithm according to claim 1 is characterized in that: The urban road target detection and tracking model is a DeepSort model.

Images