Aerial video multi-target tracking method based on irregular quadrilateral frame

Abstract

The present invention is a method for tracking multiple targets in aerial videos based on irregular quadrilateral frames. The present invention includes the following three steps: first, a dataset of aerial video frames annotated with irregular quadrilaterals is compiled and produced, including two parts: a training set and a test set; then, improvements are made on the basis of a general deep learning target detection model, a target detection model based on an irregular quadrilateral frame is created and trained to convergence; finally, a Kalman filter algorithm and a combinatorial optimization algorithm are combined to complete the multi-target tracking task, and visual interaction of the tracking process is realized. The present invention fully considers the multi-angle distribution and irregular shape of target objects in aerial video frames. Using an irregular quadrilateral frame for target tracking can effectively solve the problems of inaccurate positioning and excessive background noise introduced by the horizontal rectangular frame target tracking method, and can improve the tracking accuracy, with higher practical value.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an aerial video multi-target tracking method based on an irregular quadrilateral frame. Background Art

[0002] In recent years, drones have played an important role in traffic control, logistics and resource exploration. Most drones are equipped with high-definition cameras, which bring people a new perspective on the world and provide a large amount of high-resolution aerial video data for research in the field of computer vision. Target tracking technology based on aerial video is a research direction with high application value in the field of computer vision. It can be used in intelligent security monitoring, traffic vehicle monitoring, etc. It is also one of the current research hotspots.

[0003] Due to the special shooting angle of aerial photography, there are many types and numbers of objects in the video. In view of this feature, the research object of this invention is multi-target tracking. The multi-target tracking task includes locating and detecting the target object in a given video frame, determining the motion trajectory of the target object and keeping the number of the same target object consistent in different video frames.

[0004] At present, supported by massive data and computer computing performance, deep learning algorithms have become the mainstream method for multi-target tracking with their good nonlinear learning ability. Usually, relevant tracking algorithms are embedded in the deep neural network structure, and the detection results are screened and matched to determine the number of each tracked target. In the paper "Online multi-object tracking using CNN-based single object tracker with spatial-temporalattention mechanism", Chu et al. established a tracker and classifier for a specific target, identified the optimal matching box in the detection results of the image neighbors, and added a spatial attention mechanism to the structure to weight the features, thereby improving the classification accuracy. Li Muyu of the University of the Chinese Academy of Sciences proposed a twin network structure under the input data fusion method in his 2020 doctoral thesis "Research on Key Technologies for Real-time Multi-target Tracking Based on Deep Learning". This method adopts a rectangular border regression model and a globally optimized loss function, making full use of the global information of a single video frame to improve the problem of mutual occlusion between objects and achieve a high multi-target tracking accuracy.

[0005] However, the above methods all have the problem of inaccurate description of the shape and orientation of the target object by the tracking frame in multi-target tracking of aerial videos. Aerial photography is different from horizontal shooting. It is usually a low-altitude perspective. Objects in this perspective are distributed at multiple angles and irregularly, and the shapes and scales of objects vary greatly, which brings challenges to the tracking task. Existing multi-target tracking algorithms usually use a horizontal rectangular frame annotation method, which introduces too much background noise and makes it difficult to obtain the shape information, orientation information and accurate position information of the target object. Summary of the invention

[0006] The purpose of the present invention is to provide a method for tracking multiple targets in aerial video based on irregular quadrilateral frames. The method uses irregular quadrilateral frames to mark, train and detect aerial video frame data, and then uses Kalman filtering algorithm and combined optimization algorithm to match the same targets in adjacent frames to complete the multi-target tracking task.

[0007] The technical solution to achieve the purpose of the present invention is: a method for tracking multiple targets in aerial video based on an irregular quadrilateral frame, comprising the following steps:

[0008] Step (1): Organize and create a dataset of aerial video frames annotated with irregular quadrilateral boxes, including a training set and a test set;

[0009] Step (2): Based on the general deep learning target detection model, improve it, design the calculation method of the intersection-over-union ratio of the quadrilateral box, improve the loss function, create a target detection model based on the irregular quadrilateral box and train it until convergence;

[0010] Step (3): Combine the Kalman filter algorithm and the combinatorial optimization algorithm to complete the multi-target tracking task and realize the visual interaction of the tracking process.

[0011] Furthermore, the method of "arranging and producing an aerial video frame dataset annotated with irregular quadrilateral frames" in step (1) is as follows:

[0012] Step (11): Select the video frame of the drone as image data, and annotate each frame of the image data with a quadrilateral box;

[0013] Step (12): Process the labels: process the labeling results in step (11) so that each frame of image corresponds to a txt document with the same name as the label of the frame of image;

[0014] Step (13): Divide the data set processed in step (12) into a training set and a test set.

[0015] Furthermore, the annotation format of the document obtained by annotating each frame of the image in step (11) is [[picture_address,[x1,y1,x2,y2,x3,y3,x4,y4,class_name],...],...], where picture_address represents the storage location of the image, (x1,y1), (x2,y2), (x3,y3) and (x4,y4) represent the coordinates of the upper left corner, upper right corner, lower right corner and lower left corner of each object in the annotated image, respectively, and class_name represents the category name of the object.

[0016] Furthermore, the label format after step (12) is [[class_name,x1,y1,x2,y2,x3,y3,x4,y4],...].

[0017] Furthermore, in step (13), the entire data set is divided into two parts, a training set and a test set, in a ratio of 8:2 by random allocation.

[0018] Furthermore, step (2) specifically includes the following steps:

[0019] Step (21): Select the YOLOv3 detection model with the backbone network structure of DarkNet53, and modify the object category name and category number in the model parameters to be consistent with the object category name and category number of the data set in step (1); for a data set with n object categories, modify the number of output channels to (8+1+n)*3, where 8 represents the coordinates of the four key points of the output quadrilateral box, 1 represents the confidence, n represents the number of object categories, and 3 represents the number of anchor boxes;

[0020] Step (22): Design a method for calculating the intersection over union (IOU) of quadrilateral boxes;

[0021] Step (23): Improve the loss function: The loss function includes the coordinate loss L coord , confidence loss L conf and classification loss L cls Three parts, among which the confidence loss L conf and classification loss L cls The calculation method remains unchanged, and the coordinate loss L coord Modified to the smooth L1 loss function SmoothL1Loss between the predicted value and the label value of the 4 key coordinate points:

[0022]

[0023] Among them, (xi,yi) and (xi',yi') represent the true coordinate value and predicted coordinate value of the i-th vertex respectively, and the calculation formula of SmoothL1Loss is as follows:

[0024]

[0025] Step (24): Train the improved quadrilateral box target model on the training set until convergence and save the trained weights. The improved model can complete the quadrilateral box target detection task for the input video frame by loading the trained weights.

[0026] Furthermore, step (22) designs a method for calculating the intersection-union ratio of the quadrilateral box as follows: given two quadrilaterals, each quadrilateral is represented by four vertex coordinates, the intersection and union are initialized to 0, and the IOU calculation method is as follows:

[0027] Step (221): Determine the upper, lower, left and right boundaries of the two quadrilaterals; wherein the upper boundary is the minimum value of the ordinates of all vertices, the lower boundary is the maximum value of all ordinates, the left boundary is the minimum value of all abscissas, and the right boundary is the maximum value of all abscissas;

[0028] Step (222): traverse inside the boundary; if the traversal point is in one of the quadrilaterals, the union is incremented by 1; if the traversal point is in two quadrilaterals, the intersection and the union are incremented by 1; if the traversal point is not in any quadrilateral, continue traversing;

[0029] Step (223): Calculate the IOU; the ratio of the intersection to the union is the IOU of the two quadrilaterals.

[0030] Furthermore, step (3) "combining the Kalman filter algorithm and the combined optimization algorithm to complete the multi-target tracking task and realize the visual interaction of the tracking process" is specifically as follows:

[0031] Step (31): Design a matching algorithm between the detection box and the tracking box: first use the Kalman filter algorithm for prediction, then use the IOU value as the matching basis, and use the Hungarian algorithm to perform the optimal linear allocation between the quadrilateral boxes;

[0032] Step (32): Combine the detection model and the target matching algorithm to complete multi-target tracking: Combine the quadrilateral box target detection model trained and converged in step (2) with the Hungarian algorithm in step (31), match the detection result of the current frame with the tracking result of the previous frame, and after matching, the same object in adjacent frames keeps the same number and the detection frame that successfully matches in the current frame will be updated to the tracking frame, and then perform detection and matching of the next frame in sequence to complete multi-target tracking;

[0033] Step (33): Use the matplotlib visualization drawing tool to display the tracking process in a dynamic drawing format, and add a single target tracking function to track a single target selected by the left mouse button.

[0034] Compared with the prior art, the present invention has the following significant advantages:

[0035] The present invention fully considers the multi-angle distribution and irregular shape of the target objects in the aerial video frames. The use of irregular quadrilateral frames for target tracking can effectively solve the problems of inaccurate positioning and excessive background noise introduced by the horizontal rectangular frame target tracking method, thereby improving the tracking accuracy and having higher practical value.

[0036] This method marks the object with four key points: the upper left corner, upper right corner, lower right corner and lower left corner of the object, and uses an irregular quadrilateral box to detect and track the object to reduce the proportion of background noise in the tracking box, thereby obtaining accurate target object position information, orientation information and shape information. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the overall block diagram of the aerial video multi-target tracking method based on irregular quadrilateral box.

[0038] Figure 2 Schematic diagram of the calculation method of irregular quadrilateral IOU.

[0039] Figure 3 This is the result of irregular trapezoid detection.

[0040] Figure 4 Figure 2 is the tracking effect diagram; (a) is the target selection, (b) is the tracking effect I, (c) is the tracking effect II, and (d) is the tracking effect III. DETAILED DESCRIPTION

[0041] In order to make the technical means and creative features implemented by the present invention easy to understand, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0042] (1) Organize and create a dataset of aerial video frames annotated with irregular quadrilaterals, including a training set and a test set;

[0043] (1.1) Taking vehicle tracking as an example, the VisDrone2019-MOT dataset collected by the drone platform can be downloaded and decompressed, and each road video frame can be selected as the original image data. Use CasiaLabeler software to label the four vertices of the vehicle in the selected image. After the labeling is completed, a document with the labeling format of [[picture_address,[x1,y1,x2,y2,x3,y3,x4,y4,class_name],...],...] can be obtained, where picture_address indicates the storage location of the image, (x1,y1), (x2,y2), (x3,y3) and (x4,y4) respectively represent the coordinates of the upper left corner, upper right corner, lower right corner and lower left corner of each object in the labeled image, class_name represents the class name of the object, the number of rows in the file represents the number of frames in the video, and each column contains the coordinates of the four vertices of all vehicles in the video frame.

[0044] (1.2) Process the labels. The annotation results of (1.1) are segmented and cleaned, and each frame of the image corresponds to a txt file with the same name as the label of the frame of the image. The processed label format is [[class_name,x1,y1,x2,y2,x3,y3,x4,y4],...], and the number of rows of the label represents the number of vehicles in the frame of the image.

[0045] (1.3) Divide the data set into a training set and a test set. The data set is divided into a training set and a test set by random allocation in a ratio of 8:2.

[0046] (2) Improve the general deep learning target detection model, create a target detection model based on irregular quadrilateral box and train it to convergence;

[0047] (2.1) Select a general deep learning target detection model and make corresponding adjustments. For example, select the YOLOv3 target detection model, whose backbone network structure is DarkNet53, and modify the model parameters such as object category name, number of categories and number of output channels in the model to match the data set in step (1).

[0048] (2.2) Design a calculation method for the intersection-over-union (IOU) of quadrilateral boxes. The original IOU calculation method for horizontal rectangular boxes is no longer applicable, and the IOU calculation method between irregular quadrilateral boxes needs to be redesigned. Given two quadrilaterals A and B, each quadrilateral can be represented by 4 vertex coordinates, and the intersection and union are initialized to 0. The schematic diagram of the IOU calculation method is as follows: Figure 2 As shown, the specific calculation steps are as follows:

[0049] The first step is to determine the upper, lower, left, and right boundaries of the two quadrilaterals. The upper boundary is the minimum value of the ordinates of all vertices, the lower boundary is the maximum value of all ordinates, the left boundary is the minimum value of all abscissas, and the right boundary is the maximum value of all abscissas, so that all vertices of the two quadrilaterals are exactly within the four boundaries. Figure 2 As shown by the red dotted line,

[0050] The second step is to calculate the intersection and union. Traverse inside the boundary. If the traversal point is only in one of the quadrilaterals, the union is increased by 1; if the traversal point is in both quadrilaterals, the intersection and union are increased by 1; if the traversal point is not in any quadrilateral, continue traversing until the traversal is completed. The intersection is Figure 2 The A∩B part of the union can be represented by A+BA∩B, which is Figure 2 There are colored parts in it.

[0051] To determine whether a point is inside a quadrilateral, the cross product of the vector can be used: since the points inside a convex quadrilateral are all on the same side of the vector of the convex polygon's edge, the point is cross-producted with the vectors composed of the quadrilateral's four vertices and the vectors composed of the quadrilateral's four edges. If the four values ​​obtained have the same sign, the point is inside the quadrilateral, otherwise it is outside the quadrilateral.

[0052] The third step is to calculate the IOU. The ratio of the intersection to the union is the IOU of the two quadrilaterals.

[0053] (2.3) Improved loss function. The loss function of YOLOv3 includes coordinate loss L coord , confidence loss L conf and classification loss L cls Three parts, can be expressed as: Loss = L coord +L conf +L cls , where the confidence loss L conf and classification loss L cls The calculation method remains the same, which is:

[0054] L conf =BCELoss(pred_conf,true_conf)

[0055] L cls =CrossEntropyLoss(pred_cls,true_cls)

[0056] Among them, pred_conf and true_conf represent the predicted confidence and true confidence respectively, and pred_cls and true_cls represent the predicted target category and true target category respectively. Since the output detection box is in the form of four vertices of a quadrilateral, the coordinate loss L is coord Modified to SmoothL1Loss between the predicted value and the label value of the 4 key coordinate points:

[0057]

[0058] Among them, (xi,yi) and (xi',yi') represent the true coordinate value and predicted coordinate value of the i-th vertex of the quadrilateral box respectively. The SmoothL1Loss calculation formula is as follows:

[0059]

[0060] (2.4) Train the improved quadrilateral box target model until convergence and save the trained weights. Train the quadrilateral box target model on the training set until convergence, and then save the trained weights as a .pt file. By loading the weight file, the quadrilateral box target detection can be completed for the input video frame. The irregular quadrilateral detection results are as follows: Figure 3 shown.

[0061] (3) Combine the Kalman filter algorithm and the combinatorial optimization algorithm to complete the multi-target tracking task and realize the visual interaction of the tracking process.

[0062] (3.1) Design a matching algorithm between the detection box and the tracking box. The matching algorithm consists of two parts: the Kalman filter algorithm and the Hungarian algorithm. The Kalman filter algorithm is used for prediction first, and then the IOU value is used as the matching basis, and the Hungarian algorithm is used for optimal linear allocation between quadrilateral boxes. The Kalman filter algorithm can be implemented by calling the kalman class in the FilterPy library; the Hungarian algorithm can call the lapjv function in the lap library to achieve the best match between the IOU matrix between the detection box and the prediction box.

[0063] (3.2) Combine the detection model and target matching algorithm to complete multi-target tracking. The quadrilateral box target detection model trained and converged in step (2) is combined with the matching algorithm between the detection box and the tracking box in step (3.1). The detection result of the current frame is matched with the tracking result of the previous frame. After matching, the same object in the adjacent frames keeps the same number and the detection box that is successfully matched in the current frame will be updated to the tracking box. Then, the detection and matching of the next frame are performed in sequence to complete multi-target tracking.

[0064] (3.3) Design a visual interface interaction to complete the tracking of a single target of interest. Use the matplotlib visual drawing tool to display the tracking process in a dynamic drawing format, and add interactive functions to track a single target selected by the left mouse button. The tracking effect is as follows: Figure 4 As shown, the green box in Figure (a) is the area selected by the left mouse button, the blue box is the matching tracking box, labeled 1, and Figures (b), (c) and (d) are the tracking effects in different frames respectively.

Claims

1. A method for tracking multiple targets in aerial video based on irregular quadrilateral frame, characterized in that: The steps include: Step (1): Organize and create a dataset of aerial video frames annotated with irregular quadrilateral boxes, including a training set and a test set; Step (2): Based on the general deep learning target detection model, improve it, design the calculation method of the intersection-over-union ratio of the quadrilateral box, improve the loss function, create a target detection model based on the irregular quadrilateral box and train it until convergence; Step (3): Combine the Kalman filter algorithm and the combinatorial optimization algorithm to complete the multi-target tracking task and realize the visual interaction of the tracking process; Step (2) specifically includes the following steps: Step (21): Select the YOLOv3 detection model with the backbone network structure of DarkNet53, and modify the object category name and category number in the model parameters to be consistent with the object category name and category number of the data set in step (1); For a dataset with n object categories, the number of output channels is modified to (8+1+n)*3, where 8 represents the coordinates of the four key points of the output quadrilateral box, 1 represents the confidence, n represents the number of object categories, and 3 represents the number of anchor boxes; Step (22): Design a method for calculating the intersection over union (IOU) of quadrilateral boxes; Step (23): Improve the loss function: The loss function includes the coordinate loss L coord , confidence loss L conf and classification loss L cls Three parts, among which the confidence loss L conf and classification loss L cls The calculation method remains unchanged, and the coordinate loss L coord Modified to the smooth L1 loss function SmoothL1Loss between the predicted value and the label value of the 4 key coordinate points: Among them, (xi,yi) and (xi',yi') represent the true coordinate value and predicted coordinate value of the i-th vertex respectively, and the calculation formula of SmoothL1Loss is as follows: Step (24): Train the improved quadrilateral box target model on the training set until convergence and save the trained weights. The improved model can complete the quadrilateral box target detection task for the input video frame by loading the trained weights.

2. The method according to claim 1, characterized in that The method of "arranging and producing an aerial video frame dataset annotated with irregular quadrilateral frames" in step (1) is as follows: Step (11): Select the video frame of the drone as image data, and annotate each frame of the image data with a quadrilateral box; Step (12): Process the labels: process the labeling results in step (11) so that each frame of image corresponds to a txt document with the same name as the label of the frame of image; Step (13): Divide the data set processed in step (12) into a training set and a test set.

3. The method according to claim 2, characterized in that The annotation format of the document obtained by annotating each frame of the image in step (11) is [[picture_address,[x1,y1,x2,y2,x3,y3,x4,y4,class_name],...],...], where picture_address represents the storage location of the image, (x1,y1), (x2,y2), (x3,y3) and (x4,y4) represent the coordinates of the upper left corner, upper right corner, lower right corner and lower left corner of each object in the annotated image, respectively, and class_name represents the class name of the object.

4. The method according to claim 3, characterized in that The label format after step (12) is [[class_name,x1,y1,x2,y2,x3,y3,x4,y4],...].

5. The method according to claim 4, characterized in that In step (13), the entire data set is divided into a training set and a test set in a ratio of 8:2 by random allocation.

6. The method according to claim 5, characterized in that Step (22) designs a method for calculating the intersection-union ratio of a quadrilateral box as follows: given two quadrilaterals, each quadrilateral is represented by four vertex coordinates, the intersection and union are initialized to 0, and the IOU calculation method is as follows: Step (221): Determine the upper, lower, left and right boundaries of the two quadrilaterals; wherein the upper boundary is the minimum value of the ordinates of all vertices, the lower boundary is the maximum value of all ordinates, the left boundary is the minimum value of all abscissas, and the right boundary is the maximum value of all abscissas; Step (222): traverse inside the boundary; if the traversed point is inside one of the quadrilaterals, the union is incremented by 1; If the traversal point is inside two quadrilaterals, the intersection and union are both increased by 1; If the traversal point is not within any quadrilateral, continue traversing; Step (223): Calculate the IOU; the ratio of the intersection to the union is the IOU of the two quadrilaterals.

7. The method according to claim 6, characterized in that Step (3) "combining the Kalman filter algorithm and the combined optimization algorithm to complete the multi-target tracking task and realize the visual interaction of the tracking process" is specifically as follows: Step (31): Design a matching algorithm between the detection box and the tracking box: first use the Kalman filter algorithm for prediction, then use the IOU value as the matching basis, and use the Hungarian algorithm to perform the optimal linear allocation between the quadrilateral boxes; Step (32): Combine the detection model and the target matching algorithm to complete multi-target tracking: Combine the quadrilateral box target detection model trained and converged in step (2) with the Hungarian algorithm in step (31), match the detection result of the current frame with the tracking result of the previous frame, and after matching, the same object in adjacent frames keeps the same number and the detection frame that successfully matches in the current frame will be updated to the tracking frame, and then perform detection and matching of the next frame in sequence to complete multi-target tracking; Step (33): Use the matplotlib visualization drawing tool to display the tracking process in a dynamic drawing format, and add a single target tracking function to track a single target selected by the left mouse button.

Images