A multi-target vehicle unmanned aerial vehicle tracking detection method based on RYolov7

Abstract

The present application relates to a kind of multi-target vehicle unmanned aerial vehicle tracking detection method based on RYolov7, belong to unmanned aerial vehicle tracking detection technical field.The present application changes yolov7 baseline algorithm, replaces Mish activation function with Silu activation function;Secondly, join DCL angle prediction method and GWDIoU rotating frame regression loss function;Finally, the state of rotating vehicle target is predicted by Kalman filter algorithm, the cost matrix constructed by predicted result and feature result is associated matched using hungarian algorithm, the matching degree between the two frames before and after is calculated, determine detection tracking result, assign ID to vehicle target, mark vehicle driving track.The present application not only solves the problem that yolov7 is not optimal in the detection performance on remote sensing image, but also makes the algorithm more accurate in target positioning on unmanned aerial vehicle remote sensing image vertical detection.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) target detection technology, and relates to a multi-target vehicle UAV tracking and detection method based on RYolov7. Background Technology

[0002] With the continuous maturation of drone technology, aerial remote sensing images acquired by drones offer a wide field of view, providing valuable information on large open areas in a short time. High-resolution aerial remote sensing images are becoming increasingly readily available. Simultaneously, target detection technology in the field of computer vision is maturing and being widely applied in various aspects of production and daily life. Due to the rapid increase in the number of vehicles, traffic monitoring and management have become extremely complex, especially in urban areas. Traffic-related issues, such as air pollution, time loss due to traffic congestion, and health problems, have increased the demand for developing new automated algorithms and comprehensive traffic data. Aerial remote sensing image-based vehicle detection algorithms have proven to provide frequent and cost-effective information on the location, quantity, and type of vehicles in various traffic scenarios, such as congestion caused by infrastructure bottlenecks, accidents, and even a lack of parking spaces. Due to the dynamic nature of traffic, obtaining large-scale information through aerial imagery allows traffic management to better adapt to constantly changing traffic conditions and helps predict infrastructure bottlenecks. In disaster management, vehicle detection based on drone aerial imagery can quickly locate traffic congestion and abandoned vehicles, determining routes for effective search and rescue operations. Furthermore, during natural disasters such as floods and earthquakes, aerial remote sensing imagery is the most effective means of detecting affected vehicles.

[0003] Many existing patents utilize YOLOv5 for efficient vehicle target detection and tracking. For example, patent application CN114882393A (published August 9, 2022) discloses a method for detecting road mis-driving and traffic accident events based on target detection; and patent application CN115588126A (published January 10, 2023) discloses a vehicle target detection method integrating GAM, CARAFE, and SnIoU. However, these methods are only suitable for horizontal target detection in natural scenes, and their algorithm performance is not optimal. Vehicle targets in aerial images exhibit characteristics such as rotation, occlusion, and shadows, requiring a target detection method that can address these issues. However, the aforementioned models can only output horizontal bounding boxes for vehicle target detection in UAV remote sensing images, exhibiting poor coverage of rotating vehicle targets and resulting in unsatisfactory detection performance. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a multi-target vehicle UAV tracking and detection method based on RYolov7. The method uses the DCL angle prediction classification method, incorporates rotation angle loss, and uses the GWDIoU rotation box regression loss function to make the model converge faster and the localization more accurate. Finally, by adding the deepsort multi-target tracking algorithm, the method is combined with the detection model and applied to UAV equipment to track and detect multi-target vehicles, so as to quickly obtain traffic conditions and track and locate dangerous vehicles.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A multi-target vehicle UAV tracking and detection method based on RYolov7 includes the following steps:

[0007] S1. Replace the Silu activation function of the yolov7 model with the Mish activation function to improve the performance of the yolov7 model in target vehicle detection;

[0008] S2. Add the DCL angle prediction method to the YOLOv7 model;

[0009] S3. Add the GWDIoU rotating box regression loss function to the YOLOv7 model to improve the accuracy of vehicle target detection in UAV images;

[0010] S4. Detect the video frames captured by the drone using the RYolov7 detection network to obtain the correctly detected target vehicle boxes;

[0011] S5. Predict the state of the rotating vehicle target using Kalman filtering to obtain the vehicle target tracking boxes;

[0012] S6. The Hungarian algorithm is used to perform correlation matching on the cost matrix constructed by the detection boxes and track boxes, calculate the matching degree between the two frames, determine the detection and tracking results, assign IDs to the target vehicles, and mark the vehicle's driving trajectory.

[0013] Further, in step S1, the Mish activation function is:

[0014] Mish(x)=x×tanh(ln(1+exp(x)))

[0015] Add the Mish activation function to each convolutional layer of the YOLOv7 model.

[0016] Furthermore, step S2 specifically includes:

[0017] The four-parameter loss function of the YOLOv7 model is changed to a five-parameter loss function by adding an angle parameter θ; the loss for the angle parameter θ is calculated using the angle classification loss based on the DCL algorithm.

[0018] L dcl (θ gt ,logits)=FL(Encode dcl (θ gt ),logits)×W ADARSW (×θ)

[0019] In the formula, FL represents the localization loss function Encode. dcl (g) is encoded in binary, where Δθ represents the predicted angle θ. prred and angle label θ gt The angle difference between them, where logits represents the predicted angle vector; W ADARSW The loss weights are represented as follows:

[0020] W ADARSW (Δθ)=|sin(α(Δθ))|=|sin(α(θ gt -θ prred ))|

[0021]

[0022] In the formula, h gt and w gt represents the long side and short side of the actual value, respectively, and r represents the aspect ratio threshold.

[0023] Furthermore, in step S3, the GWDIoU rotating box regression loss function is specifically as follows:

[0024]

[0025] In the formula, f(d) 2 ) represents a nonlinear function used to transform the Wasserstein distance d 2 To make the loss smoother and more expressive, τ represents a hyperparameter.

[0026] Furthermore, step S4 specifically includes:

[0027] S41. Use the image segmentation module ImgSpilt to segment the dataset, and input the segmented dataset into the RYolov7 detection network for training to obtain the optimal weight file best.pt and the weight file last.pt for the last epoch.

[0028] S42. Use the result fusion module ResuleMerge to merge the detection results of the segmented dataset, and use the target location information detected in the segmented image and the cropping location information in the image name to restore the position of the target in the original unsegmented image.

[0029] The results evaluation module is used to compare the original image gt_poly and the original image detection_poly to obtain the AP and mAP for each category in order to evaluate the model performance.

[0030] S43. Test the vehicle target video frames captured by the UAV according to the optimal weight file best.pt to obtain detection boxes.

[0031] Furthermore, step S6 specifically includes:

[0032] S61. Obtain the target motion model based on Mahalanobis distance and perform IoU matching;

[0033] First, similarity measurement is calculated:

[0034]

[0035] In the formula, d (1) (i,j) represents the minimum cosine distance (i.e., Mahalanobis distance) of the motion characteristics, d j y represents the position of the j-th detection bounding box. i This represents the predicted position of the target by the i-th tracker. This represents the covariance matrix between the detection bounding box and the track bounding box.

[0036] Then compare the thresholds of the Mahalanobis distance and the chi-square distribution:

[0037]

[0038] In the formula, This represents an indicator that compares the Mahalanobis distance with the threshold of the chi-square distribution, t. (1) This represents the threshold of the chi-square distribution; the threshold is used to measure the matching degree between detection boxes and track boxes. If the Mahalanobis distance is less than t... (1) This indicates a successful match;

[0039] S62. Use cosine distance to obtain the target appearance model for feature matching:

[0040]

[0041] In the formula, d (2)(i,j) represents the minimum cosine distance of the appearance features, r j Let r represent the feature vector of the j-th detection. k Let R represent the feature vector of the k-th tracked target. i Represents a set of tracking feature vectors;

[0042] S63. Based on steps S61 and S62, obtain the comprehensive cost matrix:

[0043] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0044] In the formula, λ represents the weighting coefficient;

[0045] Data association is performed based on the integrated cost matrix, the matching degree between two consecutive frames is calculated, the detection and tracking results are determined, an ID is assigned to the vehicle target, and the vehicle's driving trajectory is marked.

[0046] The beneficial effects of this invention are as follows:

[0047] (1) By using Mish as the activation function, this invention achieves higher accuracy in natural scenes compared to the original YOLOv7 detection algorithm;

[0048] (2) By adding the DCL angle classification prediction method, this invention constructs the YOLOv7 five-parameter regression loss model, enabling YOLOv7 to have the ability to predict the rotation target angle. It can be applied to targets with rotation angle characteristics, solving the problem of YOLOv7's poor detection performance in remote sensing images. It has excellent detection performance in vertical detection of UAV remote sensing images.

[0049] (3) By using the GWDIoU rotation box regression localization loss function, the present invention enables YOLOv7 to locate targets more accurately on remote sensing images, thereby improving the performance of rotation box regression.

[0050] (4) The method model of the present invention is small in scale and easy to deploy on embedded platforms such as UAVs, and can be widely used in the field of UAV detection and tracking of multi-target vehicles.

[0051] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0053] Figure 1 This is a flowchart of the overall process of the present invention;

[0054] Figure 2 This is a schematic diagram of the multi-target vehicle detection algorithm model.

[0055] Figure 3 This is a schematic diagram of the multi-target vehicle tracking algorithm model.

[0056] Figure 4 This is a schematic diagram of the cascade matching process of the Hungarian algorithm. Detailed Implementation

[0057] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0058] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0059] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0060] like Figures 1-3The image shows a multi-target vehicle UAV tracking and detection method based on RYolov7. This embodiment uses the Tianjin University vehicle target public dataset as a basis and details the specific implementation steps of the invention:

[0061] First, the YOLOv7 model is improved, as shown in steps S1 to S3:

[0062] S1. Modify the YOLOv7 baseline algorithm by replacing the Silu activation function with the Mish activation function, as follows:

[0063] S11. Obtain the publicly available dataset of vehicle targets at Tianjin University captured by drones, and process the dataset.

[0064] The dataset images are 840*712 pixels. The Poly annotation format of the dataset was converted to the original YOLOv7 training format. Due to the characteristics of vehicle targets in the dataset, such as occlusion, low lighting, and boundary blurring, the Mosic data augmentation method was used to process the dataset. The Mosic processing method involves randomly flipping, cropping, scaling, and mirroring four training images, and then stitching them together in sequence before inputting them into the network model.

[0065] S12. Add the Mish activation function to each network convolutional layer; the formula for the Mish activation function is as follows:

[0066] Mish(x)=x×tanh(ln(1+exp(x)))

[0067] The Mish activation function is input into each network convolutional layer.

[0068] S2. The DCL angle prediction method is added, making it suitable for vehicle target detection in UAV remote sensing images with rotation angles, and it performs better than horizontal target detection algorithms in natural scenes; details are as follows:

[0069] S21. Add the angle parameter θ to the four-parameter (x,y,w,h) loss of yolov7 to change it to a five-parameter (x,y,w,h,θ) loss.

[0070] The XML label files from the Tianjin University Vehicle Target Public Dataset were converted into a YOLO format of [classid, x, y, w, h] using a dataset segmentation tool. Here, classid represents the class ID (there are five classes in the dataset: car, bus, freight car, truck, and van), x and y are the coordinates of the target's center point, and w and h are the target's width and height, respectively. An angle parameter θ was added to change the format to [classid, x, y, w, h, θ]. The data was then normalized and input into the network.

[0071] S22. Calculate the angle classification prediction loss according to the DCL algorithm. The loss for (x, y, w, h) can be calculated using the original loss calculation formula. The loss for θ is the angle classification loss based on DCL, and its formula is as follows:

[0072] L dcl (θ gt ,logits)=FL(Encode dcl (θ gt ),logits)×W ADARSW (Δθ)

[0073] Where FL is the localization loss function, and Encode... dcl (·) represents binary encoding, and Δθ represents the predicted angle θ. prred and angle label θ gt The angle difference between them, logits represents the predicted angle vector. W ADARSW The loss weights are calculated using the following formula:

[0074] W ADARSW (Δθ)=|sin(α(Δθ))|=|sin(α(θ gt -θ prred ))|

[0075]

[0076] Where h gt and w gt These are the longer and shorter sides of the actual value. r is the aspect ratio threshold, with a default value of 1.5.

[0077] S3. Add the GWDIoU rotating frame regression loss function to improve the accuracy of vehicle target detection in UAV images;

[0078] The loss function of GWDIoU is as follows:

[0079]

[0080] Where f(d) 2 ) represents a nonlinear function used to transform the Wasserstein distance d. 2 To make the loss smoother and more expressive, the hyperparameter τ is used to adjust the overall loss.

[0081] This completes the improvement of the YOLOv7 model, resulting in the Ryolov7 detection network model.

[0082] S4. Detect the video frames captured by the drone using the RYolov7 detection network to obtain the correctly detected target vehicle boxes;

[0083] S41. Use a dataset splitting tool to split the Tianjin University vehicle target dataset;

[0084] The ImgSplit image segmentation module in the tool was used to segment the dataset. The 840*712 training image in the vehicle dataset was divided into 4*4 images of size 640*640. The overlap region (gap) was set to 80%. Then, multiple predictions were made on the segmented images, and the prediction results were stitched together.

[0085] The segmented dataset was input into the network for training for 200 epochs with a batch size of 4. The detection confidence threshold conf-thres was set to 0.25 and the IOU threshold iou-thres for NMS was set to 0.4, resulting in the optimal weight file best.pt and the weight file last.pt for the last epoch.

[0086] S42. Evaluate network parameters, perform inference calculations, and obtain various indicators of network training;

[0087] The ResuleMerge module is used to merge the detection results of the segmented dataset. It uses the target location information detected in the segmented image and the cropping location information in the image name to restore the position of the target in the original unsegmented image.

[0088] The results evaluation module is used to compare the original image gt_poly and the original image detection_poly to obtain the AP and mAP for each category in order to evaluate the model performance.

[0089] S43. Test the vehicle target video frames captured by the drone according to the optimal weight file best.pt to obtain the required detection boxes.

[0090] S5. Predict the state of the rotating vehicle target using the Kalman filter algorithm to obtain the vehicle target tracking boxes.

[0091] S51. The Kalman filter algorithm consists of a prediction process and an update process.

[0092] The prediction process is as follows:

[0093]

[0094]

[0095] in, Let A be the mean of track at time k-1, and let A be the state transition matrix. Let be the covariance of the track at time k-1, and Q be the noise matrix of the system, representing the reliability of the entire system, which is generally initialized to a very small value. The prediction provides the predicted value, also known as the prior box state estimate;

[0096] The update process is as follows:

[0097]

[0098]

[0099]

[0100] The update process yields the optimal estimate of the state, which is called the posterior state estimate.

[0101] The prediction formula is implemented in the deep sort code through the following five processes:

[0102] a. The measurement matrix H represents the mean vector of the track boxes. Mapping to the detection space:

[0103]

[0104] b. Map the covariance matrix P to the detection space, and add the noise matrix R:

[0105] S = HPH T +R

[0106] c. Calculate the Kalman gain K to estimate the significance of the error:

[0107] K = PH T S -1

[0108] d. Calculate the updated mean vector And covariance matrix P':

[0109]

[0110] P'=(I-KH)P

[0111] Then use the updated mean vector. The covariance matrix P' is correlated in step S6.

[0112] S6. The Hungarian algorithm is used to perform correlation matching on the cost matrix constructed from detection boxes and track boxes, calculate the matching degree between consecutive frames, determine the detection and tracking results, assign IDs to vehicle targets, and mark vehicle trajectories, such as... Figure 4 As shown; including the following steps:

[0113] S61. Obtain the target motion model based on Mahalanobis distance and perform IoU matching, as follows:

[0114] The covariance matrix [cx, cy, r, h] required for Mahalanobis distance is obtained from the Kalman filter in step S5, and its similarity metric is calculated using the following formula:

[0115]

[0116] In the formula, d j This represents the position of the j-th detection bounding box; y i This represents the predicted position of the target by the i-th tracker; This represents the covariance matrix between the detection bounding box and the track bounding box.

[0117]

[0118] The comparison is between the thresholds of the Mahalanobis distance and the chi-square distribution, where This represents an indicator that compares the Mahalanobis distance with the threshold of the chi-square distribution, t. (1) The threshold representing the chi-square distribution is set to t. (1) = 9.4877; The threshold is used to measure the matching degree between detection boxes and track boxes. If the Mahalanobis distance is less than the threshold t, the result is positive. (1) This indicates a successful match;

[0119] S62. Use cosine distance to obtain the target appearance model for feature matching:

[0120]

[0121] Where d (2) (i,j) represents the minimum cosine distance of the appearance features, r j For the feature vector corresponding to the j-th detection, r k Let R represent the feature vector of the k-th tracked target. i This represents the set of tracking feature vectors, retaining the features from the k successful tracking attempts in the past;

[0122] S63. The combined cost matrix:

[0123] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0124] Data association is performed based on the integrated cost matrix, the matching degree between two consecutive frames is calculated, the detection and tracking results are determined, an ID is assigned to the vehicle target, and the vehicle's driving trajectory is marked.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-target vehicle UAV tracking and detection method based on RYolov7, characterized in that: Includes the following steps: S1. Replace the Silu activation function of the yolov7 model with the Mish activation function to improve the performance of the yolov7 model in target vehicle detection; S2. Incorporate the DCL angle prediction method into the YOLOv7 model. Specifically, this is done by adding angle parameters. The four-parameter loss function of the YOLOv7 model was changed to a five-parameter loss function; the angle parameters were calculated using the angle classification loss based on the DCL algorithm. Losses: In the formula, FL Represents the localization loss function binary encoding, Indicates the prediction angle and angle tags The angle difference between them logits A prediction vector representing the angle; The loss weights are represented as follows: In the formula, and These represent the long and short sides of the actual value, respectively. r Indicates the aspect ratio threshold; S3. Add the GWDIoU rotating box regression loss function to the YOLOv7 model to improve the accuracy of vehicle target detection in UAV images; S4. Detect the video frames captured by the drone using the RYolov7 detection network to obtain the correctly detected target vehicle boxes; S5. Predict the state of the rotating vehicle target using Kalman filtering to obtain the vehicle target tracking boxes; S6. The Hungarian algorithm is used to perform correlation matching on the cost matrix constructed by the detection boxes and track boxes, calculate the matching degree between the two frames, determine the detection and tracking results, assign IDs to the target vehicles, and mark the vehicle's driving trajectory.

2. The multi-target vehicle UAV tracking and detection method according to claim 1, characterized in that: In step S1, the Mish activation function is: Add the Mish activation function to each convolutional layer of the YOLOv7 model.

3. The multi-target vehicle UAV tracking and detection method according to claim 1, characterized in that: In step S3, the GWDIoU rotating frame regression loss function is specifically as follows: In the formula, This represents a nonlinear function used to transform the Wasserstein distance. To make the loss smoother and more expressive, This represents hyperparameters.

4. The multi-target vehicle UAV tracking and detection method according to claim 1, characterized in that: Step S4 is as follows: S41. Use the image segmentation module ImgSpilt to segment the dataset, and input the segmented dataset into the RYolov7 detection network for training to obtain the optimal weight file best.pt and the weight file last.pt for the last epoch. S42. Use the result fusion module ResuleMerge to merge the detection results of the segmented dataset, and use the target location information detected in the segmented image and the cropping location information in the image name to restore the position of the target in the original unsegmented image. The results evaluation module is used to compare the original image gt_poly and the original image detection_poly to obtain the AP and mAP for each category in order to evaluate the model performance. S43. Test the vehicle target video frames captured by the UAV according to the optimal weight file best.pt to obtain detection boxes.

5. The multi-target vehicle UAV tracking and detection method according to claim 1, characterized in that: Step S6 is as follows: S61. Obtain the target motion model based on Mahalanobis distance and perform IoU matching; First, similarity measurement is calculated: In the formula, The minimum cosine distance representing the motion characteristics. Indicates the first j The position of each detection bounding box Indicates the first i The tracker predicts the target's location. This represents the covariance matrix between the detection bounding box and the track bounding box. Then compare the thresholds of the Mahalanobis distance and the chi-square distribution: In the formula, This represents an indicator that compares the Mahalanobis distance with the threshold of the chi-square distribution. This represents the threshold of the chi-square distribution; the threshold is used to measure the matching degree between detection boxes and track boxes. If the Mahalanobis distance is less than 1, the matching is considered successful. This indicates a successful match; S62. Use cosine distance to obtain the target appearance model for feature matching: In the formula, The minimum cosine distance representing the appearance feature. Indicates the first j Each detected feature vector Indicates the first k The feature vector of the tracked target Represents a set of tracking feature vectors; S63. Based on steps S61 and S62, obtain the comprehensive cost matrix: In the formula, Indicates the weighting coefficient; Data association is performed based on the integrated cost matrix, the matching degree between two consecutive frames is calculated, the detection and tracking results are determined, an ID is assigned to the vehicle target, and the vehicle's driving trajectory is marked.

Images