A real-time traffic congestion status assessment method based on drone aerial video

Abstract

A real-time traffic congestion status assessment method based on drone aerial video is divided into a vehicle flow statistics module and a road time occupancy calculation module. The vehicle flow statistics module records the tracked vehicle ID and detection frame coordinate value, and when the change in the detection frame coordinate value of the vehicle with the same ID is greater than the threshold, the vehicle flow is increased by 1; the road time occupancy calculation module sets N virtual coils with dynamically changing positions, records the number of times each virtual coil is occupied by a vehicle in each frame image, and calculates the road time occupancy. The traffic congestion status is assessed by the vehicle flow and road time occupancy within a specified time period. The present invention solves the problems that the existing methods require a lot of manpower and material resources, the equipment is prone to aging and damage, and the implementation is relatively complicated. It takes into account accuracy, real-time and economy, and is of great significance for realizing traffic congestion status judgment and traffic scheduling management.

Description

Technical Field

[0001] The present invention belongs to the field of video image processing and intelligent transportation, and relates to the recognition, positioning and tracking of targets in drone aerial videos. It is a real-time traffic congestion status assessment method based on drone aerial videos. Background Art

[0002] With the progress of society and the improvement of people's living standards, the number of cars in my country is increasing, and the resulting traffic congestion problem is becoming more and more serious. In traffic management, it is of great significance to accurately assess the state of road congestion, take timely diversion measures, and improve road traffic capacity.

[0003] There are some existing methods for evaluating traffic congestion. Among them, manual detection and evaluation requires a lot of manpower and material resources; hardware facilities such as ground sensor coils are complex to construct, easy to damage, and difficult to repair; methods based on GPS data are difficult to obtain original data, and data loss has a greater impact on the results; image processing methods based on fixed surveillance camera videos do not need to cause damage to the road surface and the video acquisition equipment is simple to deploy, and have gradually become a popular research direction, but this method has a small coverage range and poor flexibility. UAVs have the advantages of wide field of view, small size, and the ability to hover in the air, providing a new way to monitor traffic conditions.

[0004] Road monitoring cameras are fixed in position and easy to obtain calibration information. They can calculate the actual distance and the vehicle speed by the ratio of each vehicle's driving distance and time. Compared with road cameras fixed on a certain section of road, drones can achieve more flexible monitoring configurations. However, the flight altitude and angle of drones are not fixed and are affected by the hovering accuracy. Even when hovering, the shooting scene is not completely fixed. Therefore, it is difficult to obtain distance information by calibrating the absolute fixed objects in the video. Therefore, in the drone monitoring scenario, it is difficult to accurately assess the traffic congestion status by calculating the vehicle speed. Summary of the invention

[0005] The problem to be solved by the present invention is that the existing traffic congestion assessment methods are time-consuming and labor-intensive, the related equipment is easily damaged, difficult to repair, data acquisition is difficult, and the assessment results are inaccurate. UAVs have the advantages of wide field of view, small size, and hovering in the air, but the height and angle of drone aerial photography are not fixed, and distance information cannot be obtained through calibration. The method of evaluating the road congestion state based on video calculation of vehicle speed is limited. In addition, a single indicator is difficult to accurately evaluate the road congestion state, and an assessment method that can better balance accuracy, real-time and economy is needed.

[0006] The technical solution of the present invention is: a real-time traffic congestion status assessment method based on drone aerial video, comprising the following steps:

[0007] Step 1: Build a dataset of drone aerial videos, train the target detection network and target tracking network, and obtain weights;

[0008] Step 2: The drone hovers in the air to take aerial photos of the road, and the drone RTMP address is read to obtain the video stream;

[0009] Step 3: Set a tracking area for the aerial video frame, define the range for tracking the vehicle in the video stream read in step 2, and set N virtual coils in the tracking area, N>1, the virtual coils are used to count when the tracking vehicle passes through the coils, and the setting position of the virtual coils changes dynamically with the size of the vehicle detection frame to ensure that the distance between two adjacent virtual coils is less than the length of the vehicle detection frame;

[0010] Step 4: Configure and initialize the uncounted traffic flow ID list and the counted traffic flow ID list. The uncounted traffic flow ID list is used to save the vehicle information that is not counted in the traffic flow, including the vehicle's detection box coordinates and ID value. The counted traffic flow ID list is used to save the vehicle ID value that is counted in the traffic flow.

[0011] Step 5: Use the target detection network to detect all vehicles in the aerial video frame image in real time and track the vehicles entering the tracking area;

[0012] Step 6: Traffic flow statistics and road time occupancy calculation:

[0013] Step 6.1: Traffic flow statistics: Set the initial value of traffic flow F to 0, count the traffic flow within a specified time period, and use the target tracking network to track the vehicle when it enters the tracking area and assign an id value. For the vehicle id value and the detection frame coordinates configured in the list of uncounted traffic flow ids, determine whether the change of the vehicle's detection frame coordinates in the driving direction exceeds the set threshold. If it exceeds the threshold, the traffic flow F is increased by 1, and the vehicle id value and the detection frame coordinates are deleted from the list of uncounted traffic flow ids, and then the vehicle id value is saved in the list of counted traffic flow ids. If the vehicle id information cannot be found in both lists and the vehicle is at the boundary of the tracking area, the vehicle id information is saved in the list of uncounted traffic flow ids, indicating that the vehicle has just entered the tracking area;

[0014] Step 6.2: Calculation of road time occupancy rate: Update the position of the virtual coil, for each frame of the drone aerial video, record the number of times each virtual coil is occupied by a vehicle, and calculate the road time occupancy rate Occ;

[0015] Step 7: Use the traffic flow and road time occupancy rate within the specified time period to comprehensively evaluate the traffic congestion status, and set the traffic flow threshold within the specified time period during congestion as T F , road occupancy threshold T Occ , when F is less than TF , Occ is greater than T Occ , it indicates that the road traffic congestion is serious and relevant measures should be taken in time.

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] The evaluation method of the present invention does not require a large amount of manpower. Manual detection and evaluation requires a large amount of manpower and has a certain degree of subjectivity. The present invention automatically evaluates the road congestion status by comparing with a set threshold.

[0018] The evaluation method of the present invention does not require a lot of material resources, and the required equipment is easy to maintain and manage. Vehicle detection is a key step in evaluating traffic congestion, and the vehicle detection method uses more ground sensor coils, microwave vehicle detectors and video detection technologies. Ground sensor coils are buried underground, and when they are aged and damaged, they are difficult to install and repair; microwave vehicle detectors have large detection errors for congested sections and sections with uneven vehicle model distribution, and are expensive; technologies based on video detection mostly read the video stream of installed fixed cameras and identify vehicles through target detection networks, but fixed cameras have a small coverage range and poor flexibility. Cameras must be installed on each congestion assessment section, and the equipment cost is too high. Drones have the advantages of a wide field of view, small size, and the ability to hover in the air. They are equivalent to cameras that can be moved and adjusted in height and angle, providing a new way to monitor traffic conditions.

[0019] The present invention does not require calibration and is easy to implement. The existing traffic congestion assessment method uses the average speed of vehicles as an indicator, but the calculation of speed requires prior calibration, converting the pixel information in the image into actual distance information, obtaining the time information through the frame number, and calculating the vehicle speed by the ratio of distance to time. However, the route, altitude, and angle of each flight of the drone are not fixed and cannot be calibrated, making it difficult to obtain accurate distance information. In addition, reading the video stream of the drone aerial photography may cause frame loss due to poor network conditions, and the time converted by the frame number is inaccurate. Therefore, when the distance and time errors are large, the speed calculation will also be extremely inaccurate. The present invention uses two indicators, vehicle flow and road time occupancy rate, to comprehensively evaluate the road congestion situation. If only the vehicle flow rate is used as an indicator, when the vehicle flow rate is small, it is impossible to know whether there is traffic congestion or there are few vehicles on the road section. The above situation can be avoided by combining the road time occupancy rate. When the vehicle flow rate is small and the road time occupancy rate is also small, it means that there are few vehicles on the road section. When the vehicle flow rate is small but the road time occupancy rate is large, it means that there are many vehicles on the road section but few vehicles pass through. Traffic congestion has occurred, and effective measures should be taken in time to evacuate vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1The present invention is a flow chart of a method for evaluating real-time traffic congestion status based on drone aerial video.

[0021] Figure 2 The present invention is applied to the calculation results of traffic flow and road time occupancy rate of drone aerial video. DETAILED DESCRIPTION

[0022] The present invention proposes a real-time traffic congestion status assessment method based on drone aerial video, which uses two indicators, vehicle flow and road time occupancy, to comprehensively assess the traffic congestion status, which can save a lot of manpower and material resources and is easy to implement and maintain.

[0023] The present invention is further described below with reference to the accompanying drawings and examples. Figure 1 The specific steps of the method of the present invention are as follows.

[0024] Step 1: Construct a training data set, collect drone aerial video images, and construct a picture library with annotations and labels as a training sample set for the network model. The present invention uses the YOLOv5 network model for target detection and DeepSORT for tracking. The training data set is used for training to obtain the weight W.

[0025] Step 2: Select the road section where you want to evaluate the traffic congestion status, fly the drone to this section, adjust the altitude and angle of the drone so that there is a clear outline of the vehicle in the picture, and obtain the video stream by reading the drone's RTMP address.

[0026] Step 3: Set up the tracking area and virtual coil

[0027] Step 3.1: Set up the tracking area

[0028] In the video stream read in step 2, define the range for tracking the vehicle. The significance of setting the tracking area is that only vehicles located in the tracking area will be tracked and assigned an ID. The tracking area is generally slightly smaller than the image. For example, for a 1920×1080 image, the vertical coordinates are 0-1080, and the tracking area can be set to 200-900. This is because when the car is at the top or bottom, the entire body may not be exposed. At this time, the detection effect may be unstable, which will lead to unstable tracking. If the 200-900 area is selected for tracking, the cars located in this area can basically expose the entire body, the detection is more stable, and the tracking effect is better.

[0029] Step 3.2: Set up the virtual coil

[0030] N virtual coils are set in the tracking area, N>1, and the virtual coils are used to count when tracking vehicles passing through the coils. The setting position of the virtual coils changes dynamically with the size of the vehicle detection frame to ensure that the spacing between two adjacent virtual coils is less than the length of the vehicle detection frame. The present invention sets a detection line that runs through the lane as a virtual coil, and the spacing of the virtual coils is dynamically adjusted with the size of the vehicle detection frame. First, the position of a virtual coil is fixed, and then the spacing between the virtual coils is set according to the average length of the vehicle detection frame, and is dynamically adjusted with the change of the average length of the vehicle detection frame. The present invention sets multiple virtual coils. When a traffic jam occurs on the road, if there is only one virtual coil, a certain distance is maintained between the vehicles, and if this virtual coil is just between the two vehicles, the road time occupancy rate will tend to 0, which is too far away from the actual situation; if the spacing between the virtual coils is fixed, it is also possible that each coil is located between the two vehicles, resulting in the road time occupancy rate tending to 0, which is too far away from the actual situation.

[0031] An embodiment is used to illustrate the setting of the virtual coil of the present invention: Figure 2 As shown, three coils are set, that is, three detection lines running through the lane, wherein the position of the first coil in the drone aerial video frame is fixed, the spacing between adjacent coils is proportional to the longitudinal average size of the vehicle detection frame in the video frame, the longitudinal average size of the vehicle detection frame obtained by the target detection network is set as h, and the ratio is set as a, a<1, for example, set to a number between 0.7-0.9, if a>1, the vehicle may be located between two adjacent coils, that is, both coils are not occupied, and the coils do not play a role, when a<1, it can be ensured that the above situation will not occur, in this embodiment, the spacing between adjacent coils d=a×h, the remaining two virtual coils are set with the fixed first coil as the starting point, and are set downwardly in increments according to the spacing d. Among them, N, a and the starting position of the fixed coil are all pre-set, h is related to the video being shot, different videos have different vehicle sizes due to shooting height and angle, so the coil spacing is also different, and the present invention designs a solution for the coil position to dynamically change with the size of the detection frame.

[0032] Step 4: Initialize the uncounted traffic flow ID list and the counted traffic flow ID list. The uncounted traffic flow ID list is used to save the vehicle information that is not counted in the traffic flow, including the detection box coordinates and ID values. The counted traffic flow ID list is used to save the vehicle ID values ​​that are counted in the traffic flow.

[0033] Step 5: The read drone aerial video stream is detected by the YOLOv5 network, and the vehicle in the image is detected using the weight W in step 1. In the present invention, the vehicle detected by the YOLOv5 network in the tracking area is tracked using the DeepSORT algorithm, and the tracking accuracy is closely related to the detection accuracy.

[0034] Step 6: Carry out traffic flow statistics and calculate road time occupancy rate;

[0035] Step 6.1: Traffic statistics

[0036] 1) Set the initial value of the traffic flow F to 0, and count the traffic flow within a specified time period. When a vehicle enters the tracking area, track it and assign an ID value. The same vehicle will exist in multiple frames of images, and can only be calculated once when counting the traffic flow F. The tracking algorithm ensures that the same vehicle is only counted once.

[0037] 2) When the vehicle enters the tracking area for the first time, that is, when it passes through the boundary, the relevant information of the id cannot be found in the list of uncounted traffic ids and the list of counted traffic ids, and the id and detection frame coordinate information at this time are saved in the list of uncounted traffic ids. For each vehicle detected by each frame image, if it has entered the tracking area, check whether there is the same id value in the list of uncounted traffic ids. If so, subtract the coordinate value of the detection frame at this time from the coordinate value corresponding to the id in the list of uncounted traffic ids. If the absolute value of the difference is greater than the set threshold, add 1 to the traffic flow F, remove the relevant data of the id in the list of uncounted traffic ids, and save the id value in the list of counted traffic ids. In the present invention, the basis for the vehicle id to be transferred to the list of counted traffic ids when it first belongs to the list of uncounted traffic ids is whether the absolute value of the change in the size of the vehicle detection frame exceeds the threshold, that is, a vehicle with a certain id is counted in the traffic only after traveling a certain distance.

[0038] Due to occlusion and other reasons, the tracking algorithm may have inconsistent front and back IDs for the same vehicle. To avoid a vehicle being counted multiple times, its ID and detection box coordinate information are only saved in the list of uncounted traffic IDs when the vehicle first enters the tracking area. In this way, even if the vehicle ID changes in the middle of the journey, its related information will not be saved in the list of uncounted traffic IDs, which will prevent the vehicle from being counted multiple times.

[0039] Step 6.2: Road time occupancy calculation

[0040] According to the virtual coil dynamic change strategy in step 3.2, the position of the virtual coil is updated. Let the number of times each virtual coil is occupied by a car be C. x , x represents the xth virtual coil, C x The initial value is 0. Assume that the module running time is T, the frame rate of the video stream is f FPS, the number of lanes is M, and for each x virtual coils, count how many vehicles occupy the coil in each frame image, C x Adding this statistical value, the road time occupancy rate Occ within the time T is calculated as:

[0041]

[0042] Step 7: Use the traffic flow F and road time occupancy rate Occ within the specified time period to comprehensively evaluate the road congestion. If only the traffic flow is used, when F is small, it is impossible to know whether the reason is traffic congestion or the reason is that there are few cars on the road section. Combining Occ can avoid the above situation. When F is small and Occ is also small, it means that there are few cars on the road section. When F is small but Occ is large, it means that there are many cars on the road section, but few vehicles pass through, and traffic congestion occurs. Set the traffic flow threshold within the specified time period during congestion to T F , road time occupancy threshold T Occ , when F is less than T F , Occ is greater than T Occ , it indicates that the road traffic congestion is serious and relevant measures should be taken in time.

Claims

1. A real-time traffic congestion status assessment method based on drone aerial video, Its characteristics are The following steps are involved: Step 1: Build a dataset of drone aerial videos, train the target detection network and target tracking network, and obtain weights; Step 2: The drone hovers in the air to take aerial photos of the road, and the drone RTMP address is read to obtain the video stream; Step 3: Set a tracking area for the aerial video frame, define the range for tracking the vehicle in the video stream read in step 2, and set N virtual coils in the tracking area, N>1, the virtual coils are used to count when the tracking vehicle passes through the coils, and the setting position of the virtual coils changes dynamically with the size of the vehicle detection frame to ensure that the distance between two adjacent virtual coils is less than the length of the vehicle detection frame; Step 4: Configure and initialize the uncounted traffic flow ID list and the counted traffic flow ID list. The uncounted traffic flow ID list is used to save the vehicle information that is not counted in the traffic flow, including the vehicle's detection box coordinates and ID value. The counted traffic flow ID list is used to save the vehicle ID value that is counted in the traffic flow. Step 5: Use the target detection network to detect all vehicles in the aerial video frame image in real time and track the vehicles entering the tracking area; Step 6: Traffic flow statistics and road time occupancy calculation: Step 6.1: Traffic flow statistics, set the initial value of traffic flow F to 0, count the traffic flow within a specified time period, use the target tracking network to track the vehicle when it enters the tracking area and assign an id value, when the vehicle first enters the tracking area, that is, passes through the boundary, the id related information cannot be found in the uncounted traffic id list and the counted traffic id list, save the id and detection box coordinate information at this time to the uncounted traffic id list, for each frame of the image detected vehicle, if it has entered the tracking area, check whether there is the same id value in the uncounted traffic id list, if so, subtract the detection box coordinate value at this time from the coordinate value corresponding to the id in the uncounted traffic id list, if the absolute value of the difference is greater than the set threshold, add 1 to the traffic flow F, remove the relevant data of the id in the uncounted traffic id list, and save the id value to the counted traffic id list; Step 6.2: Calculation of road time occupancy rate. There are N virtual coils in total. The coil position changes dynamically with the size of the vehicle detection frame. Let Cx be the number of times each virtual coil is occupied by a vehicle. x represents the xth virtual coil. The initial value of Cx is 0. The time for calculating the road time occupancy rate is designed to be T. The frame rate of the video stream is f FPS. The number of lanes is M. For each virtual coil, count how many vehicles occupy the coil in each frame image. Add the statistical value to Cx and calculate the road time occupancy rate Occ of the road within time T: Step 7: Use the average value of traffic flow and road time occupancy rate within the specified time period to comprehensively evaluate the traffic congestion status, and set the traffic flow threshold within the specified time period as T F , road occupancy threshold TOcc, when F is less than T F , Occ is greater than T O cc, it indicates that the road traffic is seriously congested and relevant measures should be taken in time.

2. According to the method for real-time traffic congestion status assessment based on drone aerial video in claim 1, it is characterized in that in step 1, historical video data collected by drone aerial photography is obtained, and a training data set is prepared to train a convolutional neural network for target detection and target tracking, the target detection network adopts YOLOv5, and the target tracking network adopts DeepSORT.

3. According to claim 1, a real-time traffic congestion status assessment method based on drone aerial video, Its characteristics are In step 2, the drone is controlled to fly to the road section where the traffic congestion status needs to be evaluated, the altitude and angle of the drone are adjusted to ensure that the clarity of the captured video image meets the image processing requirements, and the video stream is obtained by reading the drone RTMP address.

4. According to claim 1, a real-time traffic congestion status assessment method based on drone aerial video, Its characteristics are In step 3, a tracking area is set, and the vehicle is tracked only in the tracking area. N virtual coils are set. The virtual coils are detection lines that run through the lanes. The spacing between the virtual coils is dynamically adjusted with the size of the vehicle detection frame. First, the position of a virtual coil is fixed, and then the spacing between the virtual coils is set according to the average length of the vehicle detection frame, and is dynamically adjusted as the average length of the vehicle detection frame changes.

Images