Vehicle trajectory extraction method suitable for highway surveillance video

Abstract

The invention discloses a vehicle driving trajectory extraction method suitable for highway monitoring videos, including improving the network structure of the joint detection and tracking model CenterTrack, adding a Centerness branch; improving the network structure of CenterTrack, adding a vehicle re-identification output branch; designing a cascaded vehicle target matching algorithm; training the improved joint detection and tracking model through a data set; using the training model to detect and track vehicle targets in highway monitoring videos, thereby extracting the driving trajectory of the vehicle target. The vehicle driving trajectory extraction method suitable for highway monitoring videos has accurate detection and fast reasoning speed, and solves the problem that the existing vehicle driving trajectory extraction method cannot be applied to practice.

Description

Technical Field

[0001] The invention belongs to the field of intelligent transportation technology, and in particular relates to a vehicle driving trajectory extraction method suitable for highway monitoring videos. Background Art

[0002] At present, vehicle target trajectory extraction has important practical value for studying the behavioral characteristics of highway vehicle targets. In the highway environment, the application of vehicle trajectory extraction technology and automatic analysis can improve the intelligent level of highway management, reduce the accident rate of highway vehicles, and further ensure the safe operation of highways. However, since the field of intelligent transportation involves many disciplines and the research objects are complex, even after years of development, there are still many problems that need to be solved.

[0003] Most of the deep learning vehicle target trajectory extraction methods currently used are detection-based tracking models. The idea of ​​the detection-based tracking method is to first detect the vehicle target in the current frame, and then match the detected vehicle target with the previous vehicle trajectory. The characteristics of this type of method are that the target tracking effect is heavily dependent on the detection effect, and due to the lack of shared computing between the detection task and the tracking task, the model processing speed is slow, making it difficult to meet the requirements of real-time vehicle tracking.

[0004] When existing detection-based tracking methods are actually applied to highway environments, it is difficult to obtain accurate and continuous vehicle driving trajectories due to factors such as frequent occlusions between vehicle targets and complex and changeable environments. The above problems have always been a research difficulty, and many existing technologies and solutions are not perfect enough and are far from being practical. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a vehicle driving trajectory extraction method suitable for highway monitoring video. The present invention aims to solve the problems that the existing vehicle driving trajectory extraction method has slow processing speed, is difficult to track vehicles in real time, and cannot be applied in practice.

[0006] To achieve the above object, the present invention provides a method for extracting vehicle driving trajectories from highway surveillance videos, comprising the following steps:

[0007] S1. Improve the network structure of the joint detection and tracking model CenterTrack and add a Centerness branch to the CenterTrack model;

[0008] CenterTrack is a point-based joint detection and tracking framework, and is a MOT framework that can unify target detection and data association.

[0009] S2. Further improve the CenterTrack model improved in step S1 by adding a vehicle re-identification output branch to the CenterTrack model;

[0010] S3. Design a cascaded vehicle target matching algorithm;

[0011] S4. Select a training set from the collected data set and train the detection and tracking model improved in step S2;

[0012] S5. Use the training model to detect and track vehicle targets in highway surveillance videos, thereby extracting the driving trajectory of the vehicle targets.

[0013] Further, the step S1 includes the following sub-steps:

[0014] S1.1 inputs the current frame image, the previous frame image and the heat map of the previous frame image into the vehicle target tracking network CenterTrack to obtain a shared feature map;

[0015] S1.2 performs 3×3 convolution on the shared feature map obtained in step S1.1, and then performs batch normalization and ReLU activation to finally obtain the feature map output by the bounding box prediction branch and the feature map output by the Centerness branch;

[0016] S1.3 performs a dot multiplication on the feature map output by the bounding box prediction branch and the feature map output by the Centerness branch to obtain a weighted bounding box prediction feature map.

[0017] Further, the step S2 includes the following sub-steps:

[0018] S2.1 cancels the original "Displacement" branch of the CenterTrack joint detection and tracking framework for predicting the target center point offset value;

[0019] S2.2 Design a re-identification network. The re-identification network is a linear connection layer. The input dimension of the re-identification network is the total number of non-repeated vehicle targets that appear in the training set.

[0020] S2.3 uses the re-identification network designed in step S2.2 as an output branch of the tracking model, and jointly trains the detection branch and the re-identification branch of the tracking model.

[0021] Further, the step S3 includes the following sub-steps:

[0022] S3.1 inputs the detection result of the current frame, the tracking trajectory of the previous frame and the trajectory lost in the previous frame, and uses the re-identification features of the vehicle target for matching;

[0023] If the track is matched successfully, the re-identification features and bounding box information of the track are updated;

[0024] If the track matching is unsuccessful, the IoU matching algorithm is used for secondary matching. After the secondary matching is successful, the re-identification features and bounding box information of these tracks are updated;

[0025] S3.2 performs trajectory initialization for the detection results of unsuccessful secondary matching; for the trajectories that failed secondary matching, the trajectories whose tracking failure length exceeds the threshold are discarded, and the remaining trajectories are the trajectories lost in the current frame tracking.

[0026] Further, the step S4 includes the following sub-steps:

[0027] S4.1 generating a current frame image, a previous frame image and a heat map of the previous frame image through the training set, wherein the current frame image, the previous frame image and the heat map of the previous frame image are inputs of the vehicle target tracking network improved in step S2;

[0028] S4.2 generates a heat map, an offset feature map, a tracking vector, and a bounding box regression feature map of the current frame image by training the set, wherein the heat map, the offset feature map, the tracking vector, and the bounding box regression feature map of the current frame image are the true outputs of the improved tracking model in step S2;

[0029] S4.3 calculates the loss value of the model based on the predicted output and the actual output of the improved tracking model in step S2, uses the Adam optimizer for training, and then obtains and saves the weight of the tracking model.

[0030] Further, the step S5 includes the following sub-steps:

[0031] S5.1 uses the vehicle tracking model improved in step S2, loads the model weights obtained in step S4 into the vehicle tracking model improved in step 2, and builds a final network model for detection and tracking;

[0032] S5.2 uses the final network model built in step S5.1 to perform vehicle target detection and tracking on the input monitoring video stream data.

[0033] The beneficial effects of the present invention are:

[0034] The present invention provides a vehicle driving trajectory extraction method suitable for highway monitoring videos. The method of the present invention starts from the actual environment of the highway field. On the basis of a joint detection and tracking model CenterTrack, the method uses a redesigned backbone network to solve the problem that the original model backbone network has a large amount of calculation and is difficult to realize reasoning, thereby improving the reasoning speed while maintaining the feature extraction capability; in view of the problem that the original tracking model CenterTrack is prone to false detection, a Centerness branch is designed to weight the predicted vehicle target bounding box, thereby improving the quality of the bounding box and reducing false detection; in view of the problem of ID jump and tracking drift caused by occlusion between vehicles in highway scenes, the present invention adds a vehicle re-identification output branch to the original tracking model, and designs a cascaded target matching algorithm; finally, a complete set of vehicle driving trajectory extraction methods is formed.

[0035] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the overall flow chart of the present invention;

[0037] Figure 2 Design a flow chart for the Centerness branch of the improved CenterTrack model of the present invention;

[0038] Figure 3 This is a branch flow chart of the re-identification output of the improved CenterTrack model of the present invention;

[0039] Figure 4 This is a flow chart of the cascade matching algorithm of the present invention. DETAILED DESCRIPTION

[0040] In order to make the technical solutions, advantages and purposes of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of this application.

[0041] Example 1

[0042] like Figure 1As shown, the present invention provides a method for extracting vehicle driving trajectories in highway monitoring videos, comprising the following steps:

[0043] Step S1. Improve the network structure of the joint detection and tracking model CenterTrack and design the Centerness branch for the CenterTrack model. Figure 2 As shown, the predicted target bounding box is weighted to suppress the low confidence bounding box, thereby reducing false positives and improving tracking accuracy. Step S1 mainly includes the following three parts:

[0044] S1.1 inputs the current frame image, the previous frame image and the heat map of the previous frame image recorded in the video into the vehicle target tracking network to obtain a shared feature map;

[0045] The size of the shared feature map in this embodiment is The size of the output shared feature map is related to the width W and height H of the input image and the parameters set by the tracking model. The "64" in the formula indicates that the number of channels in this embodiment is 64.

[0046] S1.2 is of size The feature map is convolved 3×3, then batch normalized and ReLU activated, and the final feature map and size of the bounding box prediction branch output are Centerness feature map;

[0047] Batch normalization is an optional unit in convolutional neural networks. Using BN can ensure faster training and also have some regularization functions.

[0048] ReLU is a piecewise linear function that outputs a positive value directly if the input is positive, otherwise it outputs zero. It has become the default activation function for many types of neural networks, and models using it are easier to train and can achieve better performance.

[0049] S1.3 outputs the bounding box prediction branch output Perform point multiplication with the feature map output by the Centerness branch to obtain the weighted bounding box prediction feature map.

[0050] Step S2. Design a vehicle target re-identification branch for the improved joint detection and tracking model CenterTrack in step S1 to enhance the tracking model's ability to identify different vehicle targets, such as Figure 4 As shown, step S2 mainly includes the following three parts:

[0051] S2.1 ID branch output of the tracking model Feature map, for The feature map is mapped to obtain a 128-dimensional vector tracking_id. The vector tracking_id is passed through a linear classifier to obtain an nID-dimensional vector tracking_output, where nID is the number of different vehicle targets in the training set.

[0052] S2.2 calculates the nID-dimensional vector target_id based on the true label of the training set;

[0053] S2.3 uses the cross entropy loss function to calculate the training loss of the re-identification branch;

[0054] Loss reid =CrossEntropyLosstracking_id,target_id

[0055] CrossEntropyLoss(x,y)=l 1 ,…,l N} T

[0056]

[0057] In the formula, Loss reid It is the loss function of the re-identification branch, which is obtained by performing cross entropy on the model output ID (tracking_id) and the real ID (target_id); x is the input, y is the target, ω is the weight, and N is the minimum batch value set.

[0058] Step S3. Design a cascaded vehicle target matching algorithm to improve the tracking success rate and trajectory continuity of the vehicle target, such as Figure 4 As shown, step S3 mainly includes the following three parts:

[0059] S3.1 inputs the detection result of the current frame, the tracking trajectory of the previous frame and the trajectory lost in the previous frame, and uses the re-identification features of the vehicle target for matching;

[0060] If the track is matched successfully, the re-identification features and bounding box information of the track are updated;

[0061] If the track matching is unsuccessful, the IoU matching algorithm is used for secondary matching. After the secondary matching is successful, the re-identification features and bounding box information of these tracks are updated;

[0062] The calculation formula of IoU is:

[0063] Where a is the bounding box of the unmatched detection result, b is the bounding box recorded by the unmatched trajectory, and Area(a) and Area(b) are the areas of a and b respectively.

[0064] S3.2 performs trajectory initialization for detection results that have not been successfully matched after the second matching; for trajectories that have not been successfully matched after the second matching, the trajectories whose tracking failure length exceeds the threshold are discarded, and the remaining trajectories are the trajectories lost in the current frame tracking.

[0065] Step S4. Select a training set from the collected data set, and perform joint training on the improved joint detection and tracking model until the loss function converges. Step S4 mainly includes the following three parts:

[0066] S4.1 generates a current frame image, a previous frame image, and a heat map of the previous frame image respectively through the training set, and uses the generated current frame image, previous frame image, and the heat map of the previous frame image as inputs of the improved vehicle target tracking network;

[0067] S4.2 generates a heat map, an offset feature map, a tracking vector, and a bounding box regression feature map of the current frame image through the training set, and uses the generated heat map, offset feature map, tracking vector, and bounding box regression feature map of the current frame image as the true output of the improved tracking model;

[0068] S4.3 calculates the loss value of the model based on the predicted output and true output of the improved tracking model, uses the Adam optimizer to train 70 epochs (training rounds) on the training set, and then obtains the weight of the improved tracking model and saves it.

[0069] S5. Use the training model to detect and track the vehicle targets in the highway monitoring video, so as to extract the driving trajectory of the vehicle targets. Step S5 mainly includes the following two parts:

[0070] S5.1 uses the vehicle tracking model improved in step S2, loads the model weights obtained in step S4 into the vehicle tracking model improved in step 2, and builds a final network model for detection and tracking;

[0071] S5.2 uses the final network model built in step S5.1 to perform vehicle target detection and tracking on the input monitoring video stream data.

[0072] This embodiment uses the features extracted by the backbone network to accelerate the vehicle target tracking speed by jointly training the detection and tracking models; by designing the Centerness branch, the predicted target bounding box is weighted to suppress the bounding box with low confidence, thereby improving the tracking accuracy; by adding the vehicle re-identification output branch to the tracking model, the model's recognition ability for different vehicle targets is improved, thereby improving the model's tracking accuracy; by designing a matching algorithm that cascades re-identification feature matching and IOU matching, the success rate of vehicle target matching is improved, thereby improving the continuity of vehicle trajectories. Afterwards, the designed model is trained on the prepared highway vehicle tracking data set until the model converges, and the trained model weights are saved; based on the tracking model and the trained weights, vehicle target detection and tracking in the surveillance video under the highway can be achieved, thereby extracting the driving trajectory of the vehicle target.

[0073] Starting from the actual environment of highway field, the present invention adds Centerness branch and vehicle weight recognition output branch on the basis of CenterTrack, designs a cascade vehicle target matching algorithm, and proposes a vehicle driving trajectory extraction method suitable for highway field environment. The method of the present invention can effectively reduce vehicle target tracking drift and tracking loss caused by factors such as target occlusion, target loss, target similarity, target transformation, etc., improve tracking accuracy and meet real-time requirements.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the protection scope of the present invention.

Claims

1. A method for extracting vehicle driving trajectories under highway surveillance videos, characterized in that, it includes the following steps: S1. Improve the network structure of the joint detection and tracking model CenterTrack, and add a Centerness branch to the CenterTrack model; S1.1 Input the current frame image, the previous frame image, and the heat map of the previous frame image into the vehicle target tracking network CenterTrack to obtain a shared feature map; S1.2 Perform 3×3 convolution on the shared feature map obtained in step S1.1, then perform batch normalization processing and ReLU activation, and finally obtain the feature map output by the bounding box prediction branch and the feature map output by the Centerness branch; S1.3 Multiply the feature map output by the bounding box prediction branch and the feature map output by the Centerness branch point by point to obtain a weighted bounding box prediction feature map; S2. Further improve the CenterTrack model improved in step S1, and add a vehicle re-identification output branch to the CenterTrack model; S2.1 Cancel the original "Displacement" branch of the CenterTrack joint detection and tracking framework for predicting the offset value of the target center point; S2.2 Design a re-identification network; S2.3 Use the re-identification network designed in step S2.2 as an output branch of the tracking model, and perform joint training on the detection branch and the re-identification branch of the tracking model; S3. Design a cascaded vehicle target matching algorithm; S4. Select a training set from the collected data set, and train the joint detection and tracking model improved in step S2; S5. Use the trained model to detect and track vehicle targets under highway surveillance videos, so as to extract the driving trajectories of vehicle targets.

2. A method for extracting vehicle driving trajectories under highway surveillance videos according to claim 1, characterized in that, the step S3 includes the following sub-steps: S3.1 Input the detection results of the current frame, the tracking trajectories of the previous frame, and the trajectories lost in the previous frame tracking, and use the re-identification features of vehicle targets for matching; if the trajectory matching is successful, update the re-identification features and bounding box information of the trajectory; if the trajectory matching is unsuccessful, use the IoU matching algorithm for secondary matching, and after the secondary matching is successful, update the re-identification features and bounding box information of these trajectories; S3.2 Initialize the trajectories for the detection results that are not successfully matched in the secondary matching; for the trajectories that are not successfully matched in the secondary matching, discard the trajectories whose tracking failure length exceeds the threshold, and the remaining trajectories are the trajectories lost in the current frame tracking.

3. A method for extracting vehicle driving trajectories under highway surveillance videos according to claim 1, characterized in that, the step S4 includes the following sub-steps: S4.1 Generate the current frame image, the previous frame image, and the heat map of the previous frame image through the training set, and the current frame image, the previous frame image, and the heat map of the previous frame image are the inputs of the vehicle target tracking network improved in step S2; S4.2 Generate the heat map, offset feature map, tracking vector, and bounding box regression feature map of the current frame image through the training set. The heat map, offset feature map, tracking vector, and bounding box regression feature map of the current frame image are the true outputs of the improved tracking model in step S2. S4.3 Calculate the loss value of the model based on the predicted output and the true output of the improved tracking model in step S2, and use the Adam optimizer for training. Then obtain and save the weights of the tracking model.

4. A method for extracting vehicle driving trajectories suitable for highway surveillance videos according to claim 1, characterized in that the step S5 includes the following sub-steps: S5.1 Use the improved vehicle tracking model in step S2, load the model weights obtained in step S4 into the improved vehicle tracking model in step 2, and build the final network model for detection and tracking. S5.2 Use the final network model built in step S5.1 to perform vehicle target detection and tracking on the input surveillance video stream data.

Images