A method, system and device for predicting traffic flow based on drone

Abstract

The present invention provides a method, system and device for predicting traffic flow based on a drone, the method comprising: obtaining a picture set collected by a drone, preprocessing the picture set to obtain a picture data set, wherein the picture data set includes a plurality of picture data; using a lane segmentation model to obtain a characteristic lane contained in each picture data; inputting the picture data set into a pre-trained traffic flow prediction model, performing traffic flow prediction on each characteristic lane based on each picture data, and obtaining the congestion type of the characteristic lane.

Description

Technical Field

[0001] The present application relates to the field of traffic detection, and in particular to a method, system and device for predicting traffic flow based on drones. Background Art

[0002] Traffic congestion is characterized by slow vehicle speeds and long queues. In large cities with a large number of vehicles, traffic congestion is a problem that management departments are concerned about and urgently need to solve. At present, the main methods for judging road traffic congestion include underground induction detection, microwave detection, GPS floating vehicle detection, video detection technology, etc. Most of them use various traffic sensor detection technologies to obtain the required traffic status parameters, and analyze and process these traffic status parameters to judge the road congestion.

[0003] However, in the related art, the buried induction detection requires the induction coil to be laid under the road surface of each section, which has the problems of difficult to determine the directionality and unclear detection area, easy damage and difficult to repair, and complex maintenance and construction. Microwave detection detects road congestion by emitting microwave signals and vehicles reflecting radar microwaves. It has complex technology, high price, and cannot detect stationary or slow-moving vehicles. GPS floating vehicle detection technology estimates the road traffic status through data information obtained by GPS devices. Its disadvantages are insufficient coverage, detection blind spots, and inability to record on site. Video detection technology obtains video images of the detection area through a camera and uses image processing technology to analyze road congestion. Compared with other detection technologies, video detection technology has the advantages of not destroying the road, not causing traffic interruptions, obtaining diverse information, and being able to record images. It has been rapidly developed and widely used in recent years. Therefore, it is of great significance to make full use of video detection technology to serve traffic congestion detection. However, most of the existing video detection technologies are based on ground perspectives, and the target objects are severely blocked. The video processing of low-resolution, complex scenes, and bad weather conditions has poor accuracy. Summary of the invention

[0004] In view of the above problems, the present invention proposes a traffic flow prediction method, system and device based on drones. The method combines drone technology, obtains image data from the drone's perspective, completes model input image filtering through preprocessing modules such as blurring and signal light detection, and then performs congestion event analysis. The method forms a complete end-to-end traffic congestion detection and analysis method, improves model accuracy and efficiency, and realizes real-time monitoring of road congestion events.

[0005] In a first aspect, an embodiment of the present application provides a traffic flow prediction method based on a drone, the method comprising: obtaining a set of pictures collected by a drone, preprocessing the set of pictures to obtain a picture data set, wherein the picture data set includes multiple picture data; using a lane segmentation model to obtain a characteristic lane contained in each picture data; inputting the picture data set into a pre-trained traffic flow prediction model, performing traffic flow prediction on each characteristic lane based on each picture data, and obtaining the congestion type of the characteristic lane.

[0006] In some embodiments, the preprocessing of the picture set to obtain the picture data set includes: inputting the picture set into a picture filtering model, filtering the picture set for abnormal pictures to obtain a first picture data set; inputting the first picture data set into a first feature detection model, the first feature detection model detects a first feature type in each first picture data, and filtering the first picture data set for the first feature type to obtain the picture data set.

[0007] In some embodiments, the training process of the first feature detection model includes: obtaining a set of pictures collected by a drone, marking each picture in the set with a first feature type, and obtaining a first training set consisting of each picture and its corresponding first feature type; inputting the first training set into a YOLOV7 model, and obtaining the first feature detection model through multiple convolutions.

[0008] In some embodiments, the training process of the traffic flow prediction model includes: obtaining a data set acquired by a drone, preprocessing the data set to obtain a data set to be trained; marking the traffic type of each picture included in the data set to be trained to obtain a traffic training set consisting of each picture and its corresponding traffic type; inputting the traffic training set into an improved EfficientNet backbone network, and obtaining the traffic flow prediction model after multiple convolutions.

[0009] In some embodiments, the method also includes: inputting the image data set into a vehicle detection model, obtaining the vehicle direction and the number of vehicles in each feature lane through the vehicle detection model; obtaining the speed of each vehicle in the feature lane based on the average pixel moving speed of each vehicle in the feature lane; and outputting the congestion result of the feature lane based on the congestion type, number of vehicles, vehicle direction and speed.

[0010] In some embodiments, after outputting the congestion result of the characteristic lane, the method further includes: calculating and outputting the traffic density in the image data according to the number of vehicles and directions of vehicles in each characteristic lane.

[0011] In some embodiments, the method further includes: when the congestion type of the characteristic lane is a first type, outputting warning information.

[0012] An embodiment of the present application provides a traffic flow prediction system based on a drone, the system comprising: a drone, used to collect a picture set; a preprocessing module, used to obtain the picture set collected by the drone, and preprocess the picture set to obtain a picture data set, wherein the picture data set includes multiple picture data; a lane segmentation model, used to obtain a characteristic lane contained in each picture data; a traffic flow prediction model, used to perform traffic flow prediction on each characteristic lane based on each picture data, and obtain the congestion type of the characteristic lane.

[0013] The present invention proposes a traffic density estimation method that combines vehicle flow classification with directional vehicle detection to detect and count vehicles and realize the analysis of traffic density and flow patterns. It provides an effective preprocessing method for traffic congestion, including fuzzy image filtering, traffic light detection and lane mask segmentation, which significantly improves the accuracy of the congestion prediction model. By integrating different functional modules together, it provides an effective and complete solution for short-term traffic congestion prediction, demonstrates the feasibility of traffic congestion prediction methods, and provides operability for traffic management and urban planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of the flow chart of the vehicle flow prediction method based on drone provided in Example 1 of the present application;

[0015] Figure 2 A schematic diagram of the training process of the lane segmentation model provided in Example 1 of the present application;

[0016] Figure 3 A comparison diagram of the original EfficientNet (left) and the improved EfficientNet backbone network (right) provided in Example 1 of the present application;

[0017] Figure 4 A schematic diagram of a CBAM provided in Example 1 of the present application;

[0018] Figure 5 A schematic diagram of a vehicle marking provided in Example 1 of the present application;

[0019] Figure 6 A schematic diagram of the judgment process provided in Example 1 of the present application;

[0020] Figure 7 A schematic diagram of the composition of the drone-based traffic prediction system provided in Example 2 of the present application. DETAILED DESCRIPTION

[0021] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0022] Embodiment 1 Embodiment 1 of the present application provides a method for predicting traffic flow based on a drone. Figure 1 The flowchart of the vehicle flow prediction method based on drone provided in Example 1 of the present application is shown in FIG. Figure 1 , the method comprising:

[0023] Step S101: obtaining a picture set collected by a drone, and preprocessing the picture set to obtain a picture data set, wherein the picture data set includes a plurality of picture data.

[0024] In an embodiment of the present application, multiple drones are used to perform aerial photography to collect traffic road data, such as collecting road pictures, or drones shoot road videos and extract them as frames to obtain a picture set. The picture set collected by the drones should include multiple scenes such as day and night, and pictures from multiple angles such as looking down, looking straight, and looking up.

[0025] Here, preprocessing the picture set to obtain a picture data set includes:

[0026] Step S110: inputting the picture set into a picture filtering model, and filtering abnormal pictures in the picture set to obtain a first picture data set.

[0027] The image filtering model here is a binary classifier based on YOLOV5, which judges the input image as qualified or unqualified. The unqualified judgment here includes but is not limited to blurred images or abnormal images such as fog, rain, strong light, etc. The image set is input into the image filtering model, and the blurred and abnormal images are removed, and the qualified images are retained to obtain the first image data set, thereby improving the accuracy of the subsequent algorithm.

[0028] Step S120: inputting the first picture data set into a first feature detection model, wherein the first feature detection model detects a first feature type in each first picture data and filters the first picture data set by the first feature type to obtain the picture data set.

[0029] Here, filtering the first image data set by the first feature type to obtain the image data set specifically includes: when the first feature type meets the preset conditions, retaining the first image, and when the first feature type does not meet the preset conditions, deleting the first image, thereby obtaining the image data set. The first feature can be a feature parameter that requires secondary filtering. For example, the first feature is a traffic light, a zebra crossing, a sidewalk or other traffic signs. Correspondingly, the first feature type can include the type of traffic light, whether there is a zebra crossing, etc. The preset condition can be specifically set according to the first feature. For example, the preset condition is that the traffic light type is green, or there is a zebra crossing, etc., so as to retain the first image corresponding to the required first feature type. Filter the vehicle accumulation caused by red lights or zebra crossings to reduce subsequent judgment errors.

[0030] In the embodiment of the present application, the training process of the first feature detection model includes:

[0031] Step S111: obtaining a picture set collected by a drone, marking each picture in the picture set with a first feature type, and obtaining a first training set consisting of each picture and its corresponding first feature type.

[0032] Step S112: input the first training set into the YOLOV7 model, and obtain the first feature detection model through multiple convolutions.

[0033] Step S201: using a lane segmentation model to obtain characteristic lanes contained in each image data.

[0034] In the embodiment of the present application, the lane segmentation model is obtained based on the TwinLiteNet segmentation network, and its backbone network architecture is the ESPNet network, which outputs the input single image as lane lines and drivable areas. Figure 2 A schematic diagram of the training process of the lane segmentation model provided in the embodiment of the present application is shown in FIG. Figure 2 The specific training process includes: firstly, building a TwinLiteNet lane segmentation network model 210, then inputting each image data 220 into the TwinLiteNet lane segmentation network model 210, outputting the lane line mask 240 and the drivable area mask 230 in the image data, combining the obtained lane line mask 240 and the drivable area mask 230 with the image data 220 to obtain the key feature area of ​​interest, i.e., the feature lane of interest. The lane segmentation model can extract multiple lanes contained in the input image data. According to the calculation requirements, all lanes can be regarded as feature lanes to perform traffic flow prediction for all lanes of the entire road, or only one or several lanes can be obtained as feature lanes for subsequent traffic flow prediction.

[0035] Step S301: input the image data set into a pre-trained traffic flow prediction model, perform traffic flow prediction on each characteristic lane based on each image data, and obtain the congestion type of the characteristic lane.

[0036] Here, the training process of the traffic flow prediction model includes:

[0037] Step S310: Obtain the data set acquired by the drone, and pre-process the data set to obtain the image data set to be trained; Step S310 is the same as step S101, and the data set here can be the image set in step S101, or a pre-collected data set, traffic road data obtained by aerial photography of multiple drones, such as collecting road pictures, or taking road videos by drones. The data set acquired by the drone should include multiple scenes such as day and night, and multiple angles such as looking down, looking straight, and looking up. The pre-processing process of the data set includes filtering abnormal images based on the binary classifier of YOLOV5, removing blurred and abnormal unqualified images, and then filtering the first feature type through the first feature detection model to obtain the image data set to be trained, and the training data set includes multiple images.

[0038] Step S320: Mark the traffic type for each picture included in the data set to be trained, and obtain a traffic training set consisting of each picture and its corresponding traffic type. Here, each picture in the training data set is marked with traffic by manual labeling or machine labeling. The labeling process here includes first obtaining training data by manual labeling, and then training an initial labeling model based on the training data; then inputting the picture into the initial labeling model for labeling; and again manually checking whether the data labeled by the initial labeling model is correct, and modifying the wrongly labeled pictures, thereby obtaining a labeling model. Using manual labeling or model labeling methods, the traffic types are marked as three traffic states: "high", "medium", and "low".

[0039] Step S330: Input the traffic flow training set into the improved EfficientNet backbone network, and obtain the traffic flow prediction model after multiple convolutions.

[0040] Figure 3 For a comparison of the original EfficientNet (left) and the improved EfficientNet backbone network (right), see Figure 3The original EfficientNet consists of 1 Stem (a 3×3 ordinary convolution layer with a stride of 2, including BN and activation function Swish), 16 MBConv modules (with two types of convolution kernels, k3×3 and k5×5), Con2D modules, a global average pooling layer GlobalAveragePooling2D, and Dense (i.e., the FC layer in the figure). The GlobalAveragePooling2D layer is a global average pooling layer. In this layer, after the features of each feature map are calculated, a scalar feature is obtained by taking the average of each feature map. In this way, no matter how the size of the input image changes, the final feature is of fixed length. In order to reduce model parameters and speed up the running speed, in the improved EfficientNet backbone network in the embodiment of the present application, the k5×5 convolution kernel in the MBConv module is replaced with a k3×3 convolution kernel, and the CBAM attention mechanism is added to enhance feature representation.

[0041] Figure 4 For a schematic diagram of a CBAM provided in an embodiment of the present application, see Figure 4 CBAM 400 includes a channel attention module 410 and a spatial attention module 420. By multiplying the input features with the output of the channel attention module 410 and the output of the spatial attention module 420 in sequence, the optimized features are obtained, that is, the final attention-enhanced features are obtained, which are inserted into the optimized EfficientNet backbone network of the embodiment of the present application to improve the model performance. CBAM can significantly improve the performance of computer vision tasks, such as target detection, image classification, and semantic segmentation.

[0042] The image data set is input into the above-mentioned traffic flow prediction model, and the traffic flow prediction is performed on each characteristic lane based on each image data to obtain the congestion type of the characteristic lane. Here, the traffic flow prediction model can detect the congestion type of each characteristic lane in the input image data, and the congestion types include high, medium, and low. In the embodiment of the present application, when the congestion type of the characteristic lane is the first type, an alarm message is output. For example, when the output congestion type is high, an alarm message is output.

[0043] In the embodiment of the present application, the method further includes: step S401: inputting the image data set into a vehicle detection model, and obtaining the vehicle direction and the number of vehicles in each characteristic lane through the vehicle detection model. In the embodiment of the present application, step S301 and step S401 have no order, and can be performed simultaneously or sequentially.

[0044] The vehicle detection model here is trained through the following process: the images collected by the drone are preprocessed to obtain an image dataset, and the position coordinates of the four vertices of each vehicle are marked in each image data through manual or machine annotation. Figure 5 For a schematic diagram of a vehicle marking provided in an embodiment of the present application, see Figure 5 , select the vehicle, mark the four vertices of the vehicle box in the image data as P1, P2, P3, and P4 respectively, and get the corresponding coordinate points. Then input the image data and its vehicle coordinate points into the YOLOV7 model and obtain the vehicle detection model after multiple convolutions. The vehicle detection model obtained through the above training process can obtain vehicle detection with direction detection annotations, that is, the detection box output by the model has directionality, so as to judge the direction of the vehicle, and combine the detected vehicle (with directionality) with the moving direction of the vehicle center point to judge the vehicle direction, and count the vehicles in the same direction to get the number of vehicles.

[0045] Step S501: obtaining the speed of each vehicle in the characteristic lane according to the average moving speed of the pixel points of each vehicle in the characteristic lane.

[0046] Step S601: Based on the congestion type, number of vehicles, vehicle direction and vehicle speed, the congestion result of the characteristic lane is output.

[0047] Figure 6 is a schematic diagram of the judgment process of step S601, see Figure 6 Step S601 includes obtaining the congestion type of the characteristic lane output by the traffic flow prediction model, outputting a first result when the congestion type meets the first preset condition, obtaining the vehicle direction output by the vehicle detection model when the congestion type does not meet the first preset condition, and respectively judging whether the number of forward vehicles and reverse vehicles meets the second preset condition, outputting the first result when the number of vehicles meets the second preset condition, and judging whether the vehicle speed meets the third preset condition when the vehicle speed meets the third preset condition, and outputting the second result when the vehicle speed does not meet the third preset condition.

[0048] Here, the first preset condition, the second preset condition and the third preset condition can be set according to traffic needs. For example, the first preset condition is that the congestion type is low, the second preset condition is that the number of vehicles is 0, and the third preset condition is that the vehicle speed is greater than 10km / h. The first result is no congestion, and the second result is congestion.

[0049] After step S601, the method further includes:

[0050] Step S701: Calculate and output the traffic density in the image data according to the number of vehicles and directions of vehicles in each characteristic lane.

[0051] The calculation process of traffic density includes: first, determine the direction of the vehicle based on vehicle movement and directional vehicle detection; if the vehicle is traveling downward (down), then the down counter count is increased by 1; if the vehicle is traveling upward (up), then the up counter count is increased by 1; if the vehicle is traveling to the left (left), then the left counter count is increased by 1; if the vehicle is traveling to the right (right), then the right counter count is increased by 1; 3. The traffic density can be obtained based on the size of the counter counts in different directions.

[0052] The drone-based traffic flow prediction method provided in the embodiment of the present application proposes a traffic density estimation method that combines traffic flow classification with directional vehicle detection, performs vehicle detection and counting, and realizes the analysis of traffic density and flow patterns, such as identifying congestion points, traffic density or asymmetric flow between lanes. A set of effective preprocessing methods is provided for traffic congestion, including fuzzy image filtering, traffic light detection and lane mask segmentation, which significantly improves the accuracy of the congestion prediction model. By integrating different functional modules together, an effective and complete solution is provided for short-term traffic congestion prediction, demonstrating the feasibility of traffic congestion prediction methods and providing operability for traffic management and urban planning.

[0053] Embodiment 2 Embodiment 2 of the present application provides a traffic flow prediction system based on drones. Figure 7 The schematic diagram of the composition of the drone-based traffic flow prediction system provided in Example 2 of the present application, the drone-based traffic flow prediction system 700 includes:

[0054] The drone 710 is used to collect a picture set; the preprocessing module 720 is used to obtain the picture set collected by the drone, and preprocess the picture set to obtain a picture data set, wherein the picture data set includes a plurality of picture data; the lane segmentation model 730 is used to obtain a characteristic lane contained in each picture data; the traffic flow prediction model 740 is used to perform traffic flow prediction on each characteristic lane based on each picture data to obtain the congestion type of the characteristic lane.

[0055] In some embodiments, the preprocessing module includes an image filtering model and a first feature detection model, wherein the image filtering model is used to filter abnormal images in the image set to obtain a first image data set; the first feature detection model is used to filter the first image data set by a first feature type to obtain a picture data set.

[0056] In some embodiments, the drone-based traffic flow prediction system 700 further includes: a vehicle detection model 750, which is used to obtain the direction and number of vehicles in each characteristic lane; a vehicle speed acquisition module, which is used to obtain the speed of each vehicle in the characteristic lane according to the average pixel moving speed of each vehicle in the characteristic lane; an output module, which is used to output the congestion result of the characteristic lane based on the congestion type, the number of vehicles, the direction of the vehicle and the speed; a traffic density calculation module, which is used to calculate and output the traffic density in the image data according to the number of vehicles and the direction of the vehicle in each characteristic lane; and an alarm module ( Figure 7 ), which is used to output warning information when the congestion type of the characteristic lane is the first type.

[0057] Embodiment 3 Embodiment 3 of the present application provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the memory may include, for example, a flash memory, a hard disk memory, a multimedia card micro memory, a card memory (for example, an SD card or an XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a PROM, a magnetic memory, a magnetic disk, and an optical disk. When the processor executes the computer program, the above-mentioned drone-based traffic prediction method is implemented.

[0058] Embodiment 4 Embodiment 4 of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the above-mentioned drone-based traffic flow prediction method can be implemented.

[0059] The above description is only the preferred embodiment of the present invention and is not intended to limit the present invention. Any technician familiar with the profession may make various changes, modifications, substitutions and variations to these embodiments without departing from the principle and purpose of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A traffic flow prediction method based on drones, characterized in that: The method comprises: Acquire a picture set collected by a drone, and preprocess the picture set to obtain a picture data set, wherein the picture data set includes a plurality of picture data; A lane segmentation model is used to obtain the characteristic lanes contained in each image data; Inputting the image data set into a pre-trained traffic flow prediction model, performing traffic flow prediction on each characteristic lane based on each image data, and obtaining the congestion type of the characteristic lane; The training process of the traffic flow prediction model includes: obtaining a data set obtained by a drone, preprocessing the data set to obtain a data set to be trained; marking the traffic type of each picture included in the data set to be trained to obtain a traffic flow training set consisting of each picture and its corresponding traffic type; inputting the traffic flow training set into an improved EfficientNet backbone network, and obtaining the traffic flow prediction model after multiple convolutions, wherein the improved EfficientNet backbone network includes multiple k3×3 convolution kernels and a CBAM attention mechanism.

2. The method according to claim 1, characterized in that The preprocessing of the picture set to obtain the picture data set comprises: Inputting the picture set into a picture filtering model, and filtering abnormal pictures in the picture set to obtain a first picture data set; The first image data set is input into a first feature detection model. The first feature detection model detects a first feature type in each first image data and filters the first image data set by the first feature type to obtain the image data set.

3. The method according to claim 2, characterized in that The training process of the first feature detection model includes: Acquire a picture set collected by a drone, and mark each picture in the picture set with a first feature type to obtain a first training set consisting of each picture and its corresponding first feature type; The first training set is input into the YOLOV7 model, and the first feature detection model is obtained through multiple convolutions.

4. The method according to claim 3, characterized in that The method further comprises: Inputting the image data set into a vehicle detection model, and obtaining the direction and number of vehicles in each characteristic lane through the vehicle detection model; Obtaining the speed of each vehicle in the characteristic lane according to the average moving speed of the pixel points of each vehicle in the characteristic lane; Based on the congestion type, number of vehicles, vehicle direction and vehicle speed, the congestion result of the characteristic lane is output.

5. The method according to claim 4, characterized in that After outputting the congestion result of the characteristic lane, the method further includes: According to the number and direction of vehicles in each characteristic lane, the traffic density in the image data is calculated and output.

6. The method according to claim 5, characterized in that The method further comprises: When the congestion type of the characteristic lane is the first type, an alarm message is output.

7. A traffic flow prediction system based on drones, characterized in that: The system comprises: drones, used to collect picture collections; A preprocessing module is used to obtain a picture set collected by a drone, and preprocess the picture set to obtain a picture data set, wherein the picture data set includes a plurality of picture data; Lane segmentation model, used to obtain the characteristic lanes contained in each image data; A traffic flow prediction model is used to predict the traffic flow for each characteristic lane based on each image data to obtain the congestion type of the characteristic lane. The training process of the traffic flow prediction model includes: obtaining a data set obtained by a drone, and preprocessing the data set to obtain a data set to be trained; marking the traffic type of each image included in the data set to be trained to obtain a traffic flow training set consisting of each image and its corresponding traffic type; inputting the traffic flow training set into an improved EfficientNet backbone network, and obtaining the traffic flow prediction model after multiple convolutions, wherein the improved EfficientNet backbone network includes multiple k3×3 convolution kernels and a CBAM attention mechanism.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the drone-based traffic flow prediction method as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the drone-based traffic flow prediction method according to any one of claims 1 to 6 can be implemented.

Images