A road vehicle trajectory extraction method based on multi-UAV images
Through multi-drone image stitching and video stabilization processing, combined with slice reasoning and Deep OC-SORT algorithm, the problem of data deviation of axis quasi-frame in the traditional method is solved, and high-precision vehicle trajectory extraction is achieved to obtain rotary frame form The detection results support a wider detection range and higher accuracy.
Patent Information
- Application Number
- CN202410262131.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-07
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-03-07
AI Technical Summary
Most of the detection results obtained by traditional drone trajectory extraction methods are axle frame data. In scenarios such as vehicle lane change or lane tilt, there is a large deviation between the detection frame and the actual trajectory data, making it difficult to obtain accurate vehicle size and heading. Angle information.
Multi-UAV image stitching, video stabilization processing, slice reasoning and Deep OC-SORT algorithm are used, combined with GPS point mapping method, and vehicle trajectory data are extracted and calibrated to obtain vehicle detection results in the form of a rotating box.
It realizes the acquisition of vehicle trajectory information on a longer section of road on the basis of ensuring accuracy, significantly optimizes the detection effect of the target detection model, provides more accurate vehicle size and heading angle information, supports a wider detection range and a higher Detection accuracy.
Smart Images

Figure CN118015377B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of road vehicle trajectory tracking, and in particular relates to a road vehicle trajectory extraction method based on multi-UAV images. Background Art
[0002] In recent years, with the rapid progress of drone technology and the rapid development of computer vision technology, drones have become a popular tool for road traffic data collection due to their flexibility, efficiency, and convenient deployment. In comparison, the road traffic data collection method based on drones has the advantages of convenient deployment, wide coverage, relatively low cost, and no obstruction to traffic compared to traditional traffic data collection methods such as induction coils, video surveillance, and radar. The video data collected by drones can extract information such as vehicle trajectories, vehicle lanes, and vehicle heading angles, providing important basic analysis data for research directions such as traffic flow characteristics analysis, driving behavior simulation modeling, autonomous driving testing, and traffic safety impact.
[0003] During the process of drone traffic flow data collection, multiple internal and external factors such as environmental weather, motor vibration, and human control errors will cause random disturbances in the captured drone data, which manifests as slight rotation, jitter, and scaling between video image frames. At the same time, since the vehicle targets in drone images are smaller than those in video images, this disturbance will cause obvious interference to the high-precision vehicle trajectory data collection.
[0004] On the other hand, most of the current patent methods are for trajectory extraction from video data collected by a single drone. The road space covered by a single drone is limited, and a larger road range can only be covered by increasing the flight altitude of a single drone. Increasing the flight altitude of the drone will further reduce the target pixels occupied by the detected vehicle, significantly reducing the detection effect of vehicle trajectory data. It is difficult to extract vehicle trajectory data on a sufficiently long road space while ensuring the detection accuracy of drone aerial photography data.
[0005] Most of the detection results obtained by traditional trajectory extraction methods are axis-accurate frame data (the detection frame is parallel to the x-axis and y-axis of the image). In scenarios such as vehicle lane changes or lane tilt, there is a large deviation between the detection frame and the actual trajectory data, making it difficult to obtain accurate vehicle size and heading angle information, which restricts further research on road traffic behavior.
[0006] The existing invention patent, CN105488484A, is a vehicle trajectory extraction method based on drone aerial images, which aims to extract macro traffic status data and improve traffic monitoring efficiency. The basic steps are as follows:
[0007] Step 1: For the UAV low-altitude aerial image sequence, use the Haar classifier to detect the vehicle and obtain the pixel coordinates, size (length, width) and frame number of the vehicle in the aerial image;
[0008] Step 2: Screen suspected candidate vehicles based on position prediction, where the screening range is a rectangle and the prediction area size is twice the size of the target vehicle. If the vehicle ID number is not tracked for three consecutive frames, it is considered that the tracking of the vehicle ID number has failed;
[0009] Step 3: Determine the target vehicle based on the similarity of the 4-channel color histogram of the detection image;
[0010] Step 4: Output the position and frame number of the vehicle in different frames and save the obtained vehicle trajectory data.
[0011] The disadvantages of this prior art are:
[0012] (1) This method uses the traditional Haar classifier, which has lower detection accuracy than neural network algorithms such as YOLO;
[0013] (2) This method gives the vehicle trajectory in the form of an axial frame. Compared with the rotation frame form, the practicality of the trajectory data given is relatively poor;
[0014] (3) The target tracking method of this method is to define a rectangular prediction area with the position of the vehicle as the center, and perform tracking and matching calculations on the vehicle within the rectangular prediction area. However, this method only considers spatial motion information, which may result in omissions and tracking errors.
[0015] The second existing invention patent, CN116434133A, is a method for real-time image feature recognition and trajectory extraction of unmanned aerial vehicles. Its purpose is to monitor the vehicle position and extract the vehicle trajectory in real time and has better monitoring and extraction effects. The working steps of this method are as follows:
[0016] Step 1: For the UAV low-altitude aerial image sequence, use the Haar classifier to detect the vehicle and obtain the pixel coordinates, size (length, width) and frame number of the vehicle in the aerial image;
[0017] Step 2: Filter suspected candidate vehicles based on position prediction, where the screening range is circular and the size of the screening area is three times the size of the target vehicle. If there is no match in the current frame, the screening range will be enlarged to five times the size of the target vehicle in the next frame. After successful tracking, it will be restored to the original screening area size. If there is no tracking in three consecutive frames, it is considered that the vehicle ID number tracking has failed;
[0018] Step 3: Determine the target vehicle based on the 4-channel color histogram similarity;
[0019] Step 4: Complete vehicle trajectory extraction based on the obtained vehicle position and frame number in different frames.
[0020] The disadvantages of this prior art are:
[0021] (1) This method uses the traditional Haar classifier, which has lower detection accuracy than neural network algorithms such as YOLO;
[0022] (2) This method gives the vehicle trajectory in the form of an axial frame. Compared with the rotation frame form, the practicality of the trajectory data given is relatively poor;
[0023] (3) The multi-target tracking method used in this method has a higher error rate than tracking algorithms such as SORT.
[0024] Existing invention patent three, CN114972767A, a vehicle trajectory and heading angle extraction method based on high-altitude drone video, the method aims to reduce the error of vehicle trajectory information extraction caused by random disturbances of drone photography, and obtain high-precision vehicle trajectory data and vehicle heading angle data. The technical solution steps are as follows:
[0025] Step 1: Perform image stabilization preprocessing on the traffic flow video shot by the drone at high altitude to remove the jitter caused by the drone flight and the high altitude environment;
[0026] Step 2: (1) Classify the vehicle targets to be identified; (2) Create or download a dataset for training the YOLO v5 target detection model; (3) Train the rotation target detection model YOLOv5R.
[0027] Step 3: Perform target detection and tracking (YOLO v5R+DeepSortR) on each frame of the drone traffic flow video data after video stabilization processing in step 1.
[0028] Step 4: Perform data cleaning, smoothing and noise reduction preprocessing on the acquired vehicle trajectory and heading angle data;
[0029] Step 5: Extract the background image after stabilization processing, and convert the image coordinates of the acquired vehicle trajectory into corresponding geographic coordinates.
[0030] The disadvantages of this prior art are:
[0031] (1) Due to the limitation of a single UAV collecting images, the acquired trajectory data is short;
[0032] (2) This method directly performs target detection on images collected by drones. However, when the drone is at high altitude, the detection accuracy of the drone will be poor because it occupies too small a portion of the entire image.
[0033] Existing patent four, invention patent CN113421289A is a high-precision vehicle trajectory data extraction method that overcomes the disturbance caused by drone photography. This method aims to overcome the disturbance caused by drone photography and combines the existing image processing functions and RTS smoothing algorithm in the OpenCV library to obtain high-precision vehicle trajectory data. The technical solution steps are as follows.
[0034] Step 1: Obtain drone traffic flow video data and determine the type of vehicle to be identified according to the requirements, and extract a certain amount of images from the video data for labeling. After labeling, divide the labeled data into training set, verification set and test set, input the corresponding data into the darknet-based deep learning framework for YOLO v4 model training, and obtain the corresponding model weight file;
[0035] Step 2: Perform target detection (YOLO v4) and tracking (Deepsort) on each frame of the drone traffic flow video data to obtain vehicle target detection information, including vehicle category information, body size information, and vehicle number information;
[0036] Step 3: Perform anchor point detection and tracking on each frame of the UAV traffic flow video data to obtain the image coordinates of the anchor points in each frame;
[0037] Step 4: Based on the image coordinates and actual ground coordinates of the anchor points, a homography transformation model is used to calculate the coordinate transformation parameters between the image coordinate system of each frame image and the unique ground coordinate system;
[0038] Step 5: Calculate the ground coordinates corresponding to the vehicle target using the coordinate conversion parameters obtained from the anchor point and the image coordinates of the vehicle target obtained by the target detection model;
[0039] Step 6: Smoothing the ground coordinate sequence data constituting the vehicle motion trajectory using the RTS smoothing algorithm;
[0040] Step 7: Calculate microscopic traffic parameters based on vehicle trajectory data.
[0041] The disadvantages of this prior art are:
[0042] (1) Due to the limitation of a single UAV collecting images, the acquired trajectory data is short;
[0043] (2) This method directly performs target detection on images collected by drones. However, when the drone is at high altitude, the detection accuracy of the drone will be poor because it occupies too small a portion of the entire image.
[0044] (3) The one-way response transformation matrix calculated by the anchor segment only transforms the results obtained by the target detection model and cannot quantify and correct the deviation of the detection results.
[0045] Existing patent five, CN110751099A A high-precision extraction method for drone aerial video trajectories based on deep learning, which aims to automatically and accurately identify vehicles in drone aerial videos, extract and denoise the vehicle driving behavior trajectories, and extract some necessary traffic flow parameters, thereby forming a key data set for traffic flow research and analysis. The technical solution is as follows:
[0046] Step 1: Obtain target feature information, create a data set, divide the data set into training / test sets, and train the YOLO v3 and RCNN models simultaneously;
[0047] Step 2: Read the drone aerial video stream and obtain the current video frame;
[0048] Step 3: Use the trained YOLO v3 to perform an initial inspection on the current video frame to obtain the initial inspection result; use the RCNN model to perform a local re-inspection and verification on the initial inspection result to obtain the re-inspection result. Finally, obtain the pixel coordinates and length and width of the detection frame;
[0049] Step 4: Detect all frames in the video stream to obtain target detection results for all frames;
[0050] Step 5: Using the speed-displacement relationship of a single vehicle, a time series model is constructed to predict the re-inspection results, find the coordinates of the associated and unassociated targets, perform coordinate association, and generate the initial trajectory;
[0051] Step 6: De-noise, splice and smooth the initial trajectory to obtain a high-precision vehicle trajectory.
[0052] The disadvantages of this prior art are:
[0053] (1) The video is not stabilized, and the negative impact of the rotation and offset of the video image over time on the vehicle trajectory coordinates is not considered;
[0054] (2) The trajectory data are given in an axial frame rather than a rotation frame which is more convenient for analysis;
[0055] (3) The higher flying altitude of the UAV will cause the detection effect of each detection model to drop significantly, and it is impossible to improve the detection effect by mixing multiple models.
[0056] Existing patent six, CN117078717A, a method for extracting road vehicle trajectories based on a drone monocular camera, which aims to extract vehicle trajectories using computer vision methods by shooting driving video data of vehicles on mountain roads with a drone monocular camera, and can obtain high-definition vehicle trajectories. Its basic workflow is as follows:
[0057] Step 1: Calibrate the drone monocular camera based on the video stream containing the landmarks taken by the drone to obtain the camera internal parameters;
[0058] Step 2: Based on the video stream containing the landmarks taken by the drone while hovering, obtain the drone's posture and adjust the drone's attitude so that the parallelism between the drone and the road plane meets the requirements, and then calculate the drone's posture to obtain the height of the drone relative to the road plane;
[0059] Step 3: Based on the video stream of the road vehicle taken by the drone, the pixel coordinates of the center point of the vehicle and the pixel coordinates of the edges on both sides of the road are obtained;
[0060] Step 4: Establish a first world coordinate system with the drone's hovering position as the base point. Based on the camera's intrinsic parameters, the drone's height relative to the road plane, and the conversion relationship between the camera coordinate system and the pixel coordinate system, convert the vehicle center point pixel coordinates and the road edge pixel coordinates into world coordinates.
[0061] Step 5: Arrange the world coordinates in order to obtain a vehicle trajectory map.
[0062] The disadvantages of this prior art are:
[0063] (1) This method imposes requirements on the position (flight altitude should not exceed 1.5 meters) and attitude (parallel to the road surface and covering landmarks) of the drone data collection, which has strong collection restrictions;
[0064] (2) This method only extracts the pixel coordinates of the center point of the vehicle and cannot provide vehicle frame information, which limits its application to vehicle trajectory data;
[0065] (3) This method will compress the size of the captured video image to 1024*1024px 2 , will cause strong information loss and high misrecognition rate.
[0066] Existing patent seven, CN111341097A A traffic data collection method based on drone video is designed to give full play to the advantages of drone mobility and flexibility while ensuring the accuracy and reliability of traffic parameter extraction. Its technical solution is as follows:
[0067] Step 1: Select and measure road feature points: Select K road feature points in a chessboard format and measure the world coordinates of each feature point;
[0068] Step 2: According to the monitoring range, the drone takes video at high altitude;
[0069] Step 3: Perform homography transformation between world coordinates and image coordinates;
[0070] Step 4: Perform image matching on the video taken by the drone;
[0071] Step 5: Use the YOLO v3 vehicle detection model to obtain the detection target node set and target trajectory set of the input image;
[0072] Step 6: Use the DeepSort multi-target tracking algorithm to achieve data association and obtain vehicle trajectory data;
[0073] Step 7: Obtain the vehicle's traffic data based on the vehicle's position and time interval in different frames.
[0074] The disadvantages of this prior art are:
[0075] (1) Implementing image matching and then using the YOLO v3 algorithm to detect vehicle targets will reduce its target detection accuracy;
[0076] (2) The patented method requires obtaining the actual ground (world) coordinates of the feature points through field measurement, which results in significant manpower loss and safety risks;
[0077] (3) The vehicle detection information obtained by this method is in the form of an axis-accurate frame, which only contains the center point coordinates, height, width and reliability, but lacks vehicle type information. Summary of the invention
[0078] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art and to provide a road vehicle trajectory extraction method based on multi-UAV images, so as to solve the problem that most of the detection results obtained by traditional trajectory extraction methods are axis-accurate frame data. In scenarios such as vehicle lane change or lane inclination, there is a large deviation between the detection frame and the actual trajectory data, making it difficult to obtain accurate vehicle size and heading angle information.
[0079] In order to achieve the above object, the technical solution adopted by the present invention is:
[0080] A method for extracting road vehicle trajectories based on multi-UAV images, comprising the following steps:
[0081] S1. Multiple drones take off and control them to hover at the same height above the collection area;
[0082] S2, multiple drones simultaneously collect original road vehicle video data;
[0083] S3, performing video stabilization processing on the original road vehicle video data to obtain stabilized road vehicle video data;
[0084] S4, performing background extraction processing on the stabilized road vehicle video data to obtain road vehicle video data without road signs and markings;
[0085] S5, splicing the road vehicle video data corresponding to the multiple drones to obtain spliced road vehicle video data;
[0086] S6. Sampling and labeling the original road vehicle video data, and training the target detection model based on the sampled and labeled data;
[0087] S7, based on the target detection model, using a slice inference method to perform target detection on the road vehicle video data spliced in step S5, and obtain a vehicle detection result for each frame image in the road vehicle video data;
[0088] S8, using the Deep OC-SORT algorithm to match the vehicle detection results of the front and rear frame images to obtain vehicle image trajectory data;
[0089] S9. Calibrate the vehicle image trajectory data based on the active learning strategy, and convert the calibrated vehicle image trajectory data into vehicle road trajectory data using a GPS point mapping method.
[0090] Furthermore, in step S1, the images of the adjacent areas taken by the two drones contain 20% overlap.
[0091] Further, step S3 includes:
[0092] The SIFT feature detection algorithm with scale and rotation invariance is used to extract image features from the first frame and the last frame of the original road vehicle video data to obtain shared features across frames;
[0093] Based on the extracted feature points, other frames of the drone-collected video are sampled and stabilized. When the road feature points disappear due to the swaying of vegetation or vehicle occlusion in the video, the feature point positions obtained by the channel and spatial reliability tracker are used to match the cross-frame situation.
[0094] The findHomography method is used to calculate the homography matrix H, and the result of each frame after perspective transformation is output in combination with the homography matrix H. Finally, each processed frame is merged to obtain the stabilized road vehicle video data.
[0095] Furthermore, in step S4, an adaptive mixed Gaussian background modeling method is used to separate the foreground objects in the stabilized road vehicle video data from the background, and a repair algorithm is used to remove road surface markings to obtain road vehicle video data with road surface markings removed.
[0096] Furthermore, step S5 includes: for multiple road vehicle video data collected by multiple drones, reading the timestamps of the drone collected data to perform time alignment and cropping of the road vehicle video data, then using histogram color matching to process the multiple road vehicle video data to be spliced, and then performing feature matching processing on each frame of the multiple road vehicle video data to be spliced to obtain the spliced road vehicle video data.
[0097] Furthermore, the homography matrix H is calculated as:
[0098]
[0099] Among them, the two cameras on the drones are taken by pic1 and pic2 respectively; [x, y, 1] T is the pixel homogeneous coordinate of pic1; [x′, y′, 1] T are the homogeneous coordinates of the same point in pic2; λ is the scaling factor.
[0100] Further, step S6 includes:
[0101] S6.1. extracting a plurality of video images from the spliced road vehicle video data at fixed frame intervals, and annotating the plurality of video images to obtain labeled data;
[0102] S6.2, segmenting the marked data to obtain basic data;
[0103] S6.3, augmenting the basic data, and dividing the augmented basic data into a training set and a test set;
[0104] S6.4. Use the training set and test set to train the target detection model.
[0105] Further, step S7 includes:
[0106] The slice inference method is used to crop each frame of the road vehicle video data after splicing in step S5 into multiple 640*640px 2 sub-images, and 25% overlapping area is reserved between adjacent sub-images;
[0107] Detect each cropped sub-image based on the target detection model, adjust the coordinates of the detection result of each frame according to the sub-image position, and then obtain the full detection result of the video frame;
[0108] The morphological method is used to summarize the full detection results to obtain the vehicle detection results of each frame image in the road vehicle video data.
[0109] Further, step S8 includes:
[0110] S8.1. Use the Deep OC-SORT algorithm to match the detection results of the previous and next frame images to obtain the vehicle tracking ID number. If a single ID number is not matched for three consecutive frames, it is determined that the ID number has lost detection, and a new ID number is used for frame selection;
[0111] S8.2. Match the detection results of all frame images according to the method in S8.1 to obtain vehicle image trajectory data.
[0112] Further, step S9 includes:
[0113] Based on the active learning trajectory calibration strategy, the Data Fix Tool is used to manually check and calibrate the vehicle image trajectory data to obtain the corrected vehicle image trajectory data, and the corrected vehicle image trajectory data is used to optimize the target detection model;
[0114] Based on the perspective transformation technology, the GPS point mapping method is used to convert the calibrated vehicle image trajectory data into the GPS point spatial position to obtain the vehicle's real geographic coordinate distance, that is, the vehicle road trajectory data.
[0115] The road vehicle trajectory extraction method based on multi-UAV images provided by the present invention has the following beneficial effects:
[0116] 1. The present invention combines image stabilization, video stitching, target detection, minimum rectangle estimation, multi-target tracking and other methods to obtain high-precision traffic flow data, which has positive practical significance for future research on traffic flow analysis, traffic safety simulation and other aspects using vehicle trajectory data.
[0117] 2. The present invention adopts a multi-UAV splicing mechanism, through which the vehicle trajectory information of a longer road section can be extracted while ensuring accuracy.
[0118] 3. The present invention adopts a road vehicle detection mechanism and can significantly optimize the detection effect of the target detection model through the slice reasoning method, and can obtain the vehicle detection result in the form of a rotating box by combining the minimum rectangle estimation method;
[0119] 4. The present invention adopts a trajectory data calibration and verification mechanism, combined with a self-built data verification tool, to verify the detection results and further ensure the accuracy of the data used.
[0120] 5. The extraction of road vehicle trajectory information is achieved through multiple drones. Compared with traditional methods of obtaining vehicle trajectory data such as induction coils and vehicle-mounted GPS positioning, the present invention uses drones to extract road vehicle trajectory information, which has the advantages of easy deployment, low cost, and high detection accuracy.
[0121] 6. Compared with other inventive methods that use drones to extract road vehicle trajectory information, the method of the present invention supports the extraction of road vehicle trajectories under the coordination of multiple drones; considering the problem that the direct detection effect of multi-drone video data is poor, the method of the present invention integrates slice reasoning and manual calibration to extract a set of more universal, higher-precision, and wider-ranging detection methods for extracted road vehicle trajectories.
[0122] 7. From the perspective of the model, the present invention uses YOLO v8n seg and Deep OC-SORT, combined with the minimum rectangle estimation method, to support the acquisition of vehicle trajectory data of the rotating frame, providing more powerful data support for related research on vehicle driving behavior.
[0123] 8. The present invention can obtain high-precision vehicle trajectory data over a longer range, which is of great value to future research on road traffic planning, road traffic safety, driver behavior, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0124] Figure 1 The present invention is a flow chart of a method for extracting road vehicle trajectories based on multi-UAV images.
[0125] Figure 2 This is a road image collection diagram of multiple drones in the present invention.
[0126] Figure 3 The perspective transformation map of the image collected by the drone of the present invention.
[0127] Figure 4 FIG. 4 is a schematic diagram of processed video data according to the present invention.
[0128] Figure 5 This is a marking line diagram for eliminating the video image in the present invention.
[0129] Figure 6 This is the difference diagram of the axis alignment frame and the rotation frame estimation of the present invention.
[0130] Figure 7 This is a difference diagram before and after data augmentation of the present invention.
[0131] Figure 8 This is a schematic diagram of slice reasoning of the present invention.
[0132] Fig. 9 It is the detection effect enhancement diagram before and after the slice reasoning of the present invention, where Fig. 9 The left picture shows the direct recognition of the original drone-collected image; the right picture shows the recognition effect after using the slice inference method.
[0133] Fig.10 This is a calibration diagram of the detection results based on active learning of the present invention.
[0134] Fig.11 This is a corresponding diagram between the video image coordinates and the GPS coordinates of the present invention. DETAILED DESCRIPTION
[0135] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0136] Example 1
[0137] This embodiment provides a method for extracting road vehicle trajectories based on multi-UAV images. This embodiment is based on multiple quad-rotor UAVs. By shooting the driving video data of road vehicles captured by the UAVs, the vehicle trajectories are extracted by combining video stitching technology, computer vision technology and other methods to obtain high-precision vehicle trajectories. This has positive practical significance for future research on using vehicle trajectory data for traffic flow analysis, traffic safety simulation and other aspects. Figure 1 , which specifically includes the following steps:
[0138] Step S1: multiple drones take off and control them to hover at the same height above the collection area;
[0139] As a preferred embodiment of the present invention, during specific shooting, it is necessary to ensure that the pictures taken by the drone in adjacent areas contain 20% overlapping parts, and the overlapping parts can be used for subsequent video stitching.
[0140] Specifically, refer to Figure 2 ,To ensure the accuracy of shooting, it is necessary to ensure that the heights of multiple drones are basically consistent before and after ,collection.
[0141] After field testing, when the drone is hovering at a vertical distance of 100m from the road, with light wind or no wind, within 25 minutes of shooting time (the safe recording time supported by a single battery of a civilian drone), the vertical drift error of the drone is within 1m when it is hovering, and the drone's shooting picture changes little before and after. The overall pixel length and width changes of each captured vehicle are less than 1%, which can be used for subsequent multi-drone video image matching.
[0142] Step S2, multiple drones simultaneously collect original road vehicle video data;
[0143] The original road vehicle video data collected by multiple drones are defined as Video1, Video2, … Videoon respectively.
[0144] Step S3, performing video stabilization processing on the original road vehicle video data to obtain stabilized road vehicle video data, which specifically includes the following contents:
[0145] Firstly, the SIFT feature detection algorithm with scale and rotation invariance is used to extract image features from the first and last frames of the original road vehicle video data, and the shared features across frames are collected.
[0146] The extracted feature points are then used to sample and stabilize other frames of the drone video. When the road feature points disappear due to factors such as vegetation swaying and vehicle occlusion in the video, the feature point positions obtained by the Channel and Spatial Reliability Tracker (CSRT) are used to match the cross-frame situations.
[0147] Based on the extracted feature points, the findHomography method is used to calculate the homography matrix H. The result of each frame after perspective transformation is output in combination with the homography matrix H. Finally, each processed frame is merged to obtain stabilized video data.
[0148] Step S4, performing background extraction processing on the stabilized road vehicle video data to obtain road vehicle video data with road signs and markings removed;
[0149] Specifically, this embodiment uses an adaptive mixed Gaussian background modeling method to separate the foreground object from the background in the stabilized road vehicle video data, and uses a repair algorithm to remove road signs and markings to obtain road vehicle video data with road signs and markings removed;
[0150] In this embodiment, road markings are removed, and background modeling includes the following steps:
[0151] Step S4.1, reading an image;
[0152] Step S4.2, based on the adaptive mixed Gaussian background modeling, select a period of continuous frames as background modeling frames, establish an initial background model with the pixel average value of these frames, and then distinguish the background and foreground images;
[0153] Step S4.3, combining the background model, using the restoration algorithm (FMM) to fill the road surface area after the sign and marking area is removed and the affected foreground image;
[0154] Step S4.4: continuously update the background model as the video image evolves.
[0155] refer to Figure 5 , is the image difference before and after repair, to ensure that there is no interference of road markings on the target detection results after repair.
[0156] Step S5: splicing the road vehicle video data corresponding to the multiple drones to obtain spliced road vehicle video data, which specifically includes the following contents:
[0157] For multiple road vehicle video data collected by multiple drones, the timestamps of the drone collection data are read to perform time alignment and cropping of the road vehicle video data, and then the histogram color matching is used to process the multiple road vehicle video data to be spliced, and then feature matching processing is performed on each frame of the multiple road vehicle video data to be spliced to obtain the spliced road vehicle video data.
[0158] As a preference of this embodiment, the splicing of two road vehicle video data Video1 and Video2 is taken as an example for explanation, where (Video1, frame1) and (Video2, frame2) are two videos to be spliced and their corresponding video frames, wherein frame1 and frame2 are the same time frame. Since a 20% overlapping area is reserved during the shooting process, the SIFT features of the scale and rotation invariance can be calculated from the overlapping area, and the BFMatcher method feature matching is used to calculate the result of splicing frame1 and frame2. By performing the above operation on each frame of the two videos, the results are combined to obtain the video data after the splicing of Video1 and Video2.
[0159] In step S3, step S4 and step S5 of this embodiment, the stability calibration of the collected drone road image and the splicing of video data collected by multiple drones are all performed, and the SIFT algorithm is used to obtain its scale and rotation invariant features and calculate the corresponding homography matrix; then, the perspective transformation and mapping can be used in combination with its homography matrix to perform perspective transformation on the collected drone image. Figure 3 The schematic diagram of perspective transformation of images collected by drone is shown.
[0160] In this embodiment, the homography matrix can be understood as a mapping matrix of the imaging results of two cameras on the same plane in space, and its calculation relationship is:
[0161]
[0162] The two drones’ cameras take pictures respectively as pic1 and pic2; [x, y, 1] T is the pixel homogeneous coordinate of pic1; [x′, y′, 1] Tare the homogeneous coordinates of the same point in pic2; λ is the scaling factor; the entire homography matrix has 8 degrees of freedom, so at least 4 feature points are required to calculate the homography matrix. In order to ensure sufficient effectiveness of the stitching result, this embodiment adopts the findHomography method in Opencv, and repeatedly calculates by inputting multiple feature points, and selects the optimal solution in combination with the optimization algorithm. Through this step, the video data collected by the drone is stabilized and the multiple drone images are stitched. For the drone images in the boundary area, consider using the image blurring method to synthesize the results. Finally, the video data to be detected is obtained as follows: Figure 4 shown.
[0163] Step S6: sampling and labeling the original road vehicle video data, and training the target detection model based on the sampled and labeled data, which specifically includes the following steps:
[0164] Step S6.1, extracting a plurality of video images from the spliced road vehicle video data at fixed frame intervals, and annotating the plurality of video images to obtain labeled data;
[0165] This embodiment preferably extracts 400 video images, divides the annotated data categories according to the characteristics of the video data and the traffic flow requirements, and the annotation format is polygon annotation;
[0166] Step S6.2: Segment the marked data and divide the marked data into 640*640px 2 Perform segmentation to obtain basic data;
[0167] Step S6.3, augmenting the basic data, and dividing the augmented basic data into a training set and a test set;
[0168] In this embodiment, 1x data augmentation is performed on the basic data. Data augmentation includes random rotation, loss of image pixels, random flipping, random stretching, etc. The difference before and after data augmentation is as follows: Figure 7 As shown, at the same time, the annotation of the augmented image can be obtained by the contour estimation method in Opencv; then all the obtained labeled data are divided into training set and test set in a ratio of 80:20 for target detection model training;
[0169] Step S6.4: Use the training set and the test set to train the target detection model. The target detection model trained in this embodiment is YOLO v8n seg, and the set training rounds are 500 times.
[0170] The YOLO v8n seg model of this embodiment has the main feature of being able to output a polygonal area of the detection result, thereby performing minimum rectangle estimation on this area. Figure 6, it can be seen that the results of these two estimations are different. This embodiment can obtain data such as vehicle heading angle and steering status through rotating frame estimation, which is sufficient to obtain more fine-grained trajectory data.
[0171] Step S7: Based on the target detection model, a slice inference method is used to perform target detection on the road vehicle video data spliced in step S5 to obtain a vehicle detection result for each frame image in the road vehicle video data, which specifically includes the following contents:
[0172] The slice inference method is used to crop each frame of the road vehicle video data after splicing in step S5 into multiple 640*640px 2 sub-images, and 25% overlapping area is reserved between adjacent sub-images;
[0173] Each cropped sub-image is detected based on the target detection model. The confidence of the detection model is 0.5. The detection result of each frame is adjusted according to the position of the sub-image to obtain the full detection result of the video frame.
[0174] Finally, the detection results are processed according to the following principles: 1) morphological merging operation is performed on connected detection results of the same type;
[0175] 2) The detection results of different classes that intersect have a larger area;
[0176] 3) Take the minimum rectangle estimate (rotation box) of the detection result;
[0177] In this way, the vehicle detection result of each frame image in the road vehicle video data can be obtained;
[0178] refer to Figure 8 , the slice inference method of this embodiment, where the solid line frame is the actual predicted image size of the input model, the dotted line frame is each step of the prediction process, and finally all the detections are combined and summarized in combination with morphological calculations. The result is as follows Fig. 9 As shown, it can be seen that the detection results have been significantly improved.
[0179] Step S8: For the vehicle detection result of each frame image in step S7, the Deep OC-SORT algorithm is used to match the vehicle detection results of the previous and next frame images to obtain vehicle image trajectory data, which specifically includes the following steps:
[0180] Step S8.1, using the Deep OC-SORT algorithm to match the detection results of the previous and next frame images, obtain the vehicle tracking ID number, if a single ID number is not matched for three consecutive frames, it is determined that the ID number has lost detection, and a new ID number is used for frame selection;
[0181] Step S8.2, matching the detection results of all frame images according to the method of S8.1 to obtain vehicle image trajectory data;
[0182] The results can be organized into the following basic format: frame, vehicle_id, vehicle_type, p1x, p1y, p2x, p2y, p3x, p3y, p4x, p4y, which respectively represent the frame number (frame), vehicle id number (vehicle_id), vehicle type (vehicle_type) and the coordinates of the four corner points (p1x, p1y), (p2x, p2y), (p3x, p3y), (p4x, p4y) describing the detection box.
[0183] Step S9: calibrating the vehicle image trajectory data based on the active learning strategy, and converting the calibrated vehicle image trajectory data into vehicle road trajectory data using the GPS point mapping method, which specifically includes the following contents:
[0184] In this embodiment, the obtained vehicle image trajectory data may still have some errors, so a trajectory calibration strategy based on active learning is used. The self-built data repair tool Data Fix Tool is used to manually check and calibrate part of the detection data, and the trajectory data is corrected. The corrected vehicle image trajectory data is then used for target detection model optimization, and the process is repeated on this basis to ensure the accuracy of the detection results.
[0185] Then, based on the detected data, the perspective transformation technology is used to query the GPS values of multiple specific points in the image on the satellite base map, and the homography matrix that maps the image coordinates to the GPS point values is calculated. Based on this calculation result, the video image data can be converted into the GPS point spatial position, and then its real geographic coordinate distance can be obtained.
[0186] As a preferred embodiment of the present invention, the upper left corner of the video image is used as the origin to calculate the real geographical distance of the detected video image. The vehicle road trajectory data finally obtained includes the following data: frame, vehicle_id, vehicle_type, p1x_px, p1y_px, p2x_px, p2y_px, p3x_px, p3y_px, p4x_px, p4y_px, p1x_ft, p1y_ft, p2x_ft, p2y_ft, p3x_ft, p3y_ft, p4x_ft, p4y_ft, which respectively represent the frame number, vehicle id, vehicle type, and the image coordinate values of each point of the vehicle frame, with the upper left corner of the image as the origin. The upper corner pixel point (0,0) is the origin, and the units of each value along the x-axis and y-axis directions of the image are px (p1x_px, p1y_px, p2x_px, p2y_px, p3x_px, p3y_px, p4x_px, p4y_px), the relative geographical distance value between each point in the vehicle frame and the upper left corner pixel point (0,0) of the image, and the units of each value along the x-axis and y-axis directions of the image are ft (p1x_ft, p1y_ft, p2x_ft, p2y_ft, p3x_ft, p3y_ft, p4x_ft, p4y_ft).
[0187] The acquisition of geographic coordinates in step S9 is completed based on the perspective conversion of the image coordinates and GPS coordinates of the corresponding points in the video data and the satellite base map, such as Fig.11 As shown in the figure, by manually marking multiple (>10) road centerline endpoints, sign line endpoints, etc. in the video data, the GPS values of the corresponding points are obtained in the satellite base map, avoiding the danger of actually measuring the geographic location coordinates. The homography matrix is calculated by combining the image coordinates and the corresponding GPS coordinates, and then the video image coordinates can be mapped to GPS coordinates. The spatial distance between each point in the video image can be calculated using the GPS coordinate values, such as Fig.11 The embodiment shown takes the upper left point of the video image as the origin, calculates the actual geographical distance of each detection frame in the direction of the image coordinate system, and the basic distance unit is feet (ft).
[0188] Aiming at the range limitation of video data collected by a single UAV in the current prior art, the present invention makes full use of the characteristics of UAVs, such as fast speed, low cost, large detection range and flexible mobility, deploys multiple UAVs to collect data on road sections at the same time, and obtains the trajectory information of road vehicles on long sections.
[0189] In view of the limited detection effects of traditional algorithms or target detection models used in the current prior art, the present invention adopts the currently advanced YOLO v8nseg model and Deep OC-Sort model in road vehicle target detection and multi-target tracking, which have higher accuracy and smaller error than the basic models of other patents. In terms of detection strategy, it is proposed to use a unified slice inference model to adapt to the acquisition scale of each drone and retain the acquired video frame image information as much as possible, avoiding the problem of poor detection accuracy caused by small target detection.
[0190] In view of the fact that the results obtained in the current prior art are axis alignment frame detection results, and there is a lack of manual verification of the detection results, the present invention provides a rotation frame detection result, which can obtain more accurate vehicle size information and heading angle information for each vehicle. At the same time, a self-built data repair tool is used for manual verification to further ensure the detection accuracy.
[0191] Although the specific implementation of the invention is described in detail in conjunction with the drawings, it should not be understood as limiting the scope of protection of this patent. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this patent.
Claims
1. A road vehicle trajectory extraction method based on multi-UAV images, characterized in that: The following steps are involved: S1. Multiple drones take off and control them to hover at the same height above the collection area; S2, multiple drones simultaneously collect original road vehicle video data; S3, performing video stabilization processing on the original road vehicle video data to obtain stabilized road vehicle video data; S4, performing background extraction processing on the stabilized road vehicle video data to obtain road vehicle video data without road signs and markings; S5, splicing the road vehicle video data corresponding to the multiple drones to obtain spliced road vehicle video data; S6. Sampling and labeling the original road vehicle video data, and training the target detection model based on the sampled and labeled data; S7, based on the target detection model, using a slice inference method to perform target detection on the road vehicle video data spliced in step S5, and obtain a vehicle detection result for each frame image in the road vehicle video data; S8, using the Deep OC-SORT algorithm to match the vehicle detection results of the front and rear frame images to obtain vehicle image trajectory data; S9. Calibrate the vehicle image trajectory data based on the active learning strategy, and convert the calibrated vehicle image trajectory data into vehicle road trajectory data using a GPS point mapping method.
2. The method for extracting road vehicle trajectories based on multi-UAV images according to claim 1, characterized in that: In step S1, the images of the adjacent areas taken by the two drones contain 20% overlap.
3. The method for extracting road vehicle trajectories based on multi-UAV images according to claim 1, characterized in that: The step S3 comprises: The SIFT feature detection algorithm with scale and rotation invariance is used to extract image features from the first frame and the last frame of the original road vehicle video data to obtain shared features across frames; Based on the extracted feature points, other frames of the drone-collected video are sampled and stabilized. When the road feature points disappear due to the swaying of vegetation or vehicle occlusion in the video, the feature point positions obtained by the channel and spatial reliability tracker are used to match the cross-frame situation. The findHomography method is used to calculate the homography matrix H, and the result of each frame after perspective transformation is output in combination with the homography matrix H. Finally, each processed frame is merged to obtain the stabilized road vehicle video data.
4. The method for extracting road vehicle trajectories based on multi-UAV images according to claim 3 is characterized in that: In step S4, an adaptive mixed Gaussian background modeling method is used to separate the foreground object from the background in the stabilized road vehicle video data, and a repair algorithm is used to remove road surface markings to obtain road vehicle video data with road surface markings removed.
5. The method for extracting road vehicle trajectories based on multi-UAV images according to claim 4 is characterized in that: The step S5 includes: for multiple road vehicle video data collected by multiple drones, reading the timestamps of the drone collected data to perform time alignment and cropping of the road vehicle video data, then using histogram color matching to process the multiple road vehicle video data to be spliced, and then performing feature matching processing on each frame of the multiple road vehicle video data to be spliced to obtain the spliced road vehicle video data.
6. The method for extracting road vehicle trajectories based on multi-UAV images according to claim 5, characterized in that: The calculation of the homography matrix H is: Among them, the two cameras on the drones are taken by pic1 and pic2 respectively; [x, y, 1] T is the pixel homogeneous coordinate of pic1; [x′, y′, 1] T are the homogeneous coordinates of the same point in pic2; λ is the scaling factor.
7. The method for extracting road vehicle trajectories based on multi-UAV images according to claim 1, characterized in that: The step S6 comprises: S6.
1. extracting a plurality of video images from the spliced road vehicle video data at fixed frame intervals, and annotating the plurality of video images to obtain labeled data; S6.2, segmenting the marked data to obtain basic data; S6.3, augmenting the basic data, and dividing the augmented basic data into a training set and a test set; S6.
4. Use the training set and test set to train the target detection model.
8. The method for extracting road vehicle trajectories based on multi-UAV images according to claim 1, characterized in that: The step S7 comprises: The slice inference method is used to crop each frame of the road vehicle video data after splicing in step S5 into multiple 640*640px 2 sub-images, and 25% overlapping area is reserved between adjacent sub-images; Detect each cropped sub-image based on the target detection model, adjust the coordinates of the detection result of each frame according to the sub-image position, and then obtain the full detection result of the video frame; The morphological method is used to summarize the full detection results to obtain the vehicle detection results of each frame image in the road vehicle video data.
9. The method for extracting road vehicle trajectories based on multi-UAV images according to claim 1, characterized in that: The step S8 comprises: S8.
1. Use the Deep OC-SORT algorithm to match the detection results of the previous and next frame images to obtain the vehicle tracking ID number. If a single ID number is not matched for three consecutive frames, it is determined that the ID number has lost detection, and a new ID number is used for frame selection; S8.
2. Match the detection results of all frame images according to the method in S8.1 to obtain vehicle image trajectory data.
10. The method for extracting road vehicle trajectories based on multi-UAV images according to claim 1, characterized in that: The step S9 comprises: Based on the active learning trajectory calibration strategy, the Data Fix Tool is used to manually check and calibrate the vehicle image trajectory data to obtain the corrected vehicle image trajectory data, and the corrected vehicle image trajectory data is used to optimize the target detection model; Based on the perspective transformation technology, the GPS point mapping method is used to convert the calibrated vehicle image trajectory data into the GPS point spatial position to obtain the vehicle's real geographic coordinate distance, that is, the vehicle road trajectory data.
Citation Information
Patent Citations
Vehicle track extracting method based on unmanned aerial vehicle image
CN105488484A
Unmanned aerial vehicle aerial video track high-precision extraction method based on deep learning
CN110751099A
Traffic data acquisition method based on unmanned aerial vehicle video
CN111341097A
Road vehicle trajectory extraction method based on unmanned aerial vehicle monocular camera
CN117078717A
High-precision vehicle trajectory data extraction method capable of overcoming shooting disturbance of unmanned aerial vehicle
CN113421289A
