A road surface three-dimensional reconstruction method and system for dynamic traffic scenes

Abstract

The present invention discloses a method and system for 3D reconstruction of a road surface for dynamic traffic scenes. The present invention realizes 3D reconstruction of a road surface under the influence of dynamic traffic based on unmanned aerial vehicle stereo photography and deep learning, adopts a lightweight deep learning framework to remove vehicle noise on aerial images, and automatically adjusts the image overlap rate in different traffic density areas through image sequence space optimization to improve the quality and speed of 3D modeling of the road surface.

Description

Technical Field

[0001] The present invention relates to the field of road health monitoring, and in particular to a road surface three-dimensional reconstruction method and system for dynamic traffic scenes. Background Art

[0002] The road condition has a huge impact on the safety, comfort and economy of transportation, and road health monitoring is the basis for ensuring the normal operation of roads. At present, most of the automated detection of road defects is based on two-dimensional images, while the three-dimensional model of the road surface can achieve more accurate disease identification and obtain three-dimensional size information of various diseases. With the development of three-dimensional measurement technology, the collection of three-dimensional data of road surface has become a hot topic in current research.

[0003] At present, road surface 3D measurement technology can be mainly divided into two types: laser imaging and stereo vision. At present, most of the 3D measurement devices equipped with multi-functional road inspection vehicles are based on laser imaging technology, and their imaging methods mainly use ToF (Time of Flight) principle or structured light principle. Although laser 3D imaging can quickly generate road surface models, it is easily affected by strong light and vibration, and the equipment cost is high.

[0004] Therefore, stereoscopic 3D imaging based on color images has become an alternative method for low-cost, high-precision 3D reconstruction of roads. According to the number of cameras, stereoscopic 3D reconstruction can be mainly divided into binocular stereo imaging and monocular stereo imaging. Binocular stereo imaging restores the depth of the scene through the parallax generated between two cameras with fixed positions. The advantage of binocular stereo imaging is its fast imaging speed, but it also has the disadvantage of poor modeling resolution, and the offset of the camera installation position will lead to inaccurate 3D reconstruction.

[0005] Monocular stereo imaging can generate a 3D model through images taken by a single camera while moving. In the field of 3D reconstruction of transportation infrastructure, monocular stereo imaging mainly uses drone photography to achieve 3D reconstruction of scenes. Based on multi-view road images, a 3D point cloud model of the road surface is generated through image feature matching and epipolar geometry. Although drone monocular stereo imaging can already achieve 3D reconstruction of large-scale road scenes, there are still two difficulties in its application: (1) Drone stereo photography can only be implemented on closed roads. Vehicles running on open roads will block the road surface and generate a lot of noise during the modeling process; (2) In order to observe the road area blocked by vehicles, the image acquisition density can only be greatly increased, resulting in a sharp increase in modeling processing time.

[0006] In order to realize automatic recognition of vehicles in aerial images, current research has tried to use image processing methods or deep learning to identify objects such as roads, bridges and other buildings. However, the current target recognition methods are computationally intensive, and the size changes of objects in drone images at different heights will have a huge impact on the recognition accuracy. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and to provide a method and system for three-dimensional reconstruction of road surfaces for dynamic traffic scenes, so as to solve the problems of the prior art in road operation environments, such as the lack of reconstructed models and a large amount of surface noise caused by occlusion by moving vehicles, as well as the low efficiency of three-dimensional reconstruction of road surfaces with a large field of view.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A method for three-dimensional reconstruction of a road surface for dynamic traffic scenes comprises the following steps:

[0010] Step 1, collect images taken by drone;

[0011] Step 2: Input the captured image as a prediction set image into the improved YOLO vehicle detector to obtain the coordinate file of the vehicle in the captured prediction set image; based on the coordinate file of the vehicle, the area within the vehicle boundary box in the prediction set image is singularized;

[0012] Step 3: obtain the spatial relationship between adjacent images in the prediction set image by feature extraction method, calculate the actual occlusion rate and ground overlap rate between adjacent images by the spatial relationship between adjacent images and the vehicle bounding box on each image, identify invalid images by combining image sequence space optimization, remove invalid images, and obtain preprocessed images;

[0013] Combine the preprocessed image and the image coordinate file to reconstruct the three-dimensional shape of the road surface through stereo vision.

[0014] A further improvement of the present invention is:

[0015] Preferably, before step 1, the boundary coordinates of the planned road surface shooting area are extracted from the electronic map, a flight area file is generated, the flight area file is imported into the drone, and the drone shoots the planned road surface shooting area.

[0016] Preferably, in step 2, the process of obtaining the coordinate file of the vehicle in the shooting prediction set image by improving the YOLO vehicle detector is as follows:

[0017] (1) Input the captured image with an initial size of (512,512,3) into the improved YOLO vehicle detector, preliminarily predict the vehicle bounding box, and calculate the average pixel width of the vehicle;

[0018] (2) Calculate the scaling factor k based on the ratio of the average pixel width of the vehicle on the training set image to the average pixel width of the vehicle on the prediction set image.

[0019] (3) The input size of the prediction set image is adjusted to (512*k, 512*k, 3) by the scaling factor and then input into the deep learning framework again to predict the final accurate vehicle bounding box.

[0020] Preferably, the process of obtaining the improved YOLO vehicle detector is:

[0021] The images extracted from the input images by the 6th, 12th and 14th layers of the backbone feature extraction network are sampled, spliced ​​and feature extracted respectively. Then, based on the 9 predefined anchor boxes, the type, position and confidence of the bounding box are predicted through standard convolution to obtain the vehicle bounding box corner point coordinate file dataset of the predicted image.

[0022] Compare the labeled image vehicle bounding box corner point coordinate file dataset and the predicted image vehicle bounding box corner point coordinate file dataset to obtain an error, train the YOLO vehicle detector until the error is less than a set value, and obtain an improved YOLO vehicle detector;

[0023] The backbone feature extraction network includes 1 CBL module and 13 DBR modules; the CBL module includes a standard convolution layer, a batch normalization layer and a Leaky ReLU activation layer; the DBR module includes a depth-separable convolution layer, a batch normalization layer and a ReLU activation layer.

[0024] Preferably, in step 3, the spatial relationship between adjacent images is calculated by the following formula:

[0025]

[0026] in, are the feature points on the image, R is the rotation matrix, and T is the translation matrix. Through multiple feature point pairs, the actual values ​​of the rotation matrix R and the translation matrix T are calculated to obtain the spatial relationship between adjacent images.

[0027] Preferably, in step 3, the process of identifying invalid images is:

[0028] (1) Assume the overlap ratio of the original image sequence is IOR max , then the lowest image overlap rate after optimization is IOR min =1-2*(1-IOR max );

[0029] (2) Starting from the first image, search at intervals of 2 steps to calculate the occlusion rate OCR1 obtained by combining the i-th image and the i+2-th image, that is, the ratio of the vehicle overlap area of ​​the two images to the image overlap area. At the same time, the actual ground overlap rate GOR1 of the area outside the vehicle occlusion position can be obtained;

[0030] (3) Calculate the occlusion rate OCR2 obtained by combining the i-th image, the i+1-th image, and the i+2-th image, that is, the ratio of the vehicle overlapping area of ​​the three images to the image overlapping area. At the same time, the actual ground overlap rate GOR2 of the area outside the vehicle occlusion position can be obtained.

[0031] (4) If OCR2>OCR1 and GOR2>GOR1, keep the i+1th image; otherwise, delete the i+1th image.

[0032] Preferably, the calculation formula of the occlusion rate OCR is:

[0033] Occlusion rate (OCR) = the area of ​​the part still blocked by the vehicle after the two images are superimposed / the area of ​​the original image;

[0034] The calculation formula of ground overlap rate GOR is:

[0035] Ground overlap ratio GOR = area of ​​the vehicle-blocked part after superimposing the two images / area of ​​the original image.

[0036] Preferably, in step 4, firstly match the coordinates of the preprocessed image and the image in step 1, and preliminarily estimate the relative spatial position of the camera between taking different photos; extract the feature points on each image, and match the feature points of adjacent images; inversely calculate the spatial coordinates of the camera based on the feature point pairs and spatial epipolar geometry; solve the spatial three-dimensional coordinates of all feature points on the image based on the camera spatial coordinates to obtain a point cloud model, and then reconstruct the three-dimensional morphology of the road surface through stereo vision.

[0037] A 3D road reconstruction for dynamic traffic scenes, including:

[0038] An image acquisition module, used to collect images taken by drones;

[0039] The coordinate acquisition module inputs the captured image as the prediction set image into the improved YOLO vehicle detector to obtain the coordinate file of the vehicle in the captured prediction set image; based on the vehicle coordinate file, the area within the vehicle boundary box in the prediction set image is singularized;

[0040] The image sequence optimization module obtains the spatial relationship between adjacent images in the prediction set through feature extraction method, calculates the actual occlusion rate and ground overlap rate between adjacent images through the spatial relationship between adjacent images and the vehicle boundary box on each image, and identifies invalid images by combining image sequence spatial optimization, removes invalid images, and obtains preprocessed images;

[0041] The reconstruction module is used to combine the preprocessed image and the image coordinate file to reconstruct the three-dimensional shape of the road surface through stereo vision.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention discloses a method and system for 3D reconstruction of road surfaces in dynamic traffic scenes. The present invention realizes 3D reconstruction of road surfaces under the influence of dynamic traffic based on drone stereo photography and deep learning, uses a lightweight deep learning framework to remove vehicle noise on aerial images, and automatically adjusts the image overlap rate in different traffic density areas through image sequence space optimization to improve the quality and speed of 3D modeling of road surfaces. The advantages of the present invention are:

[0044] (1) This image 3D reconstruction framework can be used for 3D reconstruction of road surfaces in open road environments without being affected by running vehicles; (2) Vehicle noise on aerial images can be quickly identified and eliminated, and changes in vehicle size in drone images at different altitudes will not affect recognition accuracy; (3) The number of images required under different traffic densities can be identified, the spatial distribution of aerial image sequences can be optimized, and invalid images can be reduced; (4) The integrity and spatial accuracy of the reconstructed road point cloud model can be guaranteed, while improving the 3D modeling speed of large-scale scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the overall framework diagram of the present invention;

[0046] Figure 2 Diagram of a deep learning framework for vehicle recognition in drone images;

[0047] Figure 3 This is the effect diagram of vehicle recognition and noise removal in drone images;

[0048] Figure 4 Spatial relationship estimation for UAV images;

[0049] Figure 5 This is a schematic diagram for calculating the vehicle occlusion rate and ground overlap rate;

[0050] Figure 6 A framework diagram for spatial optimization of UAV image sequences;

[0051] Figure 7 Spatial distribution map before and after optimization for UAV image sequences;

[0052] Among them, (a) is the original image sequence of section 1; (b) is the image sequence of section 1 after spatial optimization; (c) is the original image sequence of section 2; (d) is the image sequence of section 2 after spatial optimization;

[0053] Figure 8 This is a three-dimensional reconstruction effect diagram of the road surface under the influence of traffic in the present invention;

[0054] Among them, (a) is the 3D model reconstructed for section 1; (b) is the 3D model reconstructed for section 2;

[0055] Fig. 9 A comparison chart of the modeling speed of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0057] In the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention; the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; in addition, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0058] refer to Figure 1 The present invention provides a method for rapid three-dimensional reconstruction of a road surface based on drone stereo photography and deep learning, which specifically comprises the following steps:

[0059] Step 1: Automatically collect road surface images on open roads using a multi-rotor drone and record the spatial coordinates when the images are taken.

[0060] Step 1.1, extract the boundary coordinates of the road area planned to be photographed in the electronic map, and generate a kml format flight area file. Specifically, the boundary coordinates of the pre-selected aerial survey road area should be in a closed loop form, that is, {(x1, y1), (x2, y2), …, (x n ,y n ),(x1,y1)}.

[0061] Step 1.2, import the generated flight area file into the drone and set the photography parameters, namely the camera inclination angle, camera resolution, photography height, longitudinal overlap rate and lateral overlap rate. Specifically, the photography parameters should be set according to different traffic environments and modeling accuracy requirements. On roads open to traffic, the drone's flight altitude should not be less than 10 meters, the camera inclination angle should be between 60 and 90 degrees, the longitudinal overlap rate of adjacent images should not be less than 90%, and the lateral overlap rate of images should not be less than 60%.

[0062] Step 1.3, perform the aerial photography task, obtain aerial images of different road locations, record the location information when the image was taken, and generate an XML coordinate file that matches the image name.

[0063] Through step 1, the flight area file in kml format of the planned road surface area and the xml coordinate file of the captured image are obtained.

[0064] Step 2: Build an improved YOLO (You Only Look Once) deep learning framework to detect vehicle locations and remove areas containing vehicles.

[0065] This part is divided into the early training stage and the use stage during actual application.

[0066] 1. Training phase

[0067] Step 2.1, select some aerial images as training set images for annotation, mark the vehicle positions on the image with bounding boxes, and generate bounding box corner point coordinate files. During the image annotation process, there is no need to distinguish specific vehicle models, only the vehicle bounding boxes need to be classified into one category as the annotated image.

[0068] Step 2.2, reference Figure 2 , an improved YOLO deep learning framework integrating depthwise separable convolution and image resolution active adjustment unit is constructed, namely the YOLO vehicle detector, which is used to locate the positions of various vehicles on aerial images using bounding boxes. Among them, the depthwise separable convolution embedded in the improved YOLO vehicle detector includes two parts: channel-by-channel convolution and point-by-point convolution. The purpose of the image resolution active adjustment unit embedded in the improved YOLO vehicle detector is to enhance the generalization performance of the model. The standard convolution in the original YOLO deep learning framework is replaced by depthwise separable convolution to reduce the amount of calculation and improve the feature extraction performance.

[0069] At the same time, the image resolution active adjustment unit is used to adjust the resolution of the network input image according to the size of the vehicle in the aerial image at different heights. The overall framework is as follows: the input of the improved YOLO network is a three-channel color image. The input size of the image in the training stage is (512,512,3), because the original image needs to be converted into an input image of (512*k,512*k,3) size, where k is the scaling factor, and 512*k needs to be a multiple of 32. The image resolution active adjustment unit is used to preliminarily predict the pixel size of the vehicle on the image, and the appropriate scaling factor k is selected according to the size to generate an input image with the best resolution.

[0070] In step 2.3, use the image dataset with the labeled size of (512,512,3) and input it into the model built in step 2.2 to train the constructed YOLO vehicle detector.

[0071] The process of predicting the image bounding box corner coordinate file by the YOLO vehicle detector is as follows: for the input image with adjusted resolution, the backbone network is first used to extract features. The backbone network includes the basic components CBL module and DBR module. The CBL module includes 1 standard convolution layer (C), 1 batch normalization layer (B) and 1 Leaky ReLU activation layer (L). The DBR module consists of 1 depth-separable convolution layer (D), 1 batch normalization layer (B) and 1 ReLU activation layer (R). The backbone feature extraction network consists of 1 CBL module and 13 DBR modules, and the end performs feature fusion through spatial pyramid pooling. The image features extracted by the 6th, 12th and 14th layers (end) in the backbone feature extraction network are upsampled, spliced ​​and feature extracted respectively. Finally, through standard convolution, based on 9 predefined anchor boxes, the type, position and confidence of the bounding box are predicted to obtain the bounding box corner coordinate file of each image, forming a vehicle bounding box corner coordinate file dataset for each image.

[0072] Estimate the error between the predicted image and the annotated image, and define the model loss with CIoU until the training loss is stable and the prediction value error is less than the preset value. The network training process can adopt training techniques such as transfer learning, training freezing, dynamic learning rate, and early stopping.

[0073] (II) Use phase

[0074] Step 2.4, use the trained improved YOLO vehicle detector to identify and locate various types of vehicles on different aerial images, and then predict the positions of other bounding boxes with high accuracy, apply the image resolution active adjustment unit, and use the detector to predict other data sets, which mainly includes the following steps:

[0075] (1) The prediction set images are input into the improved YOLO vehicle detector with the initial size of (512,512,3), the vehicle bounding box is preliminarily predicted, and the average pixel width of the vehicle is calculated;

[0076] (2) Calculate the scaling factor k based on the ratio of the average pixel width of the vehicle on the training set image to the average pixel width of the vehicle on the prediction set image.

[0077] (3) Adjust the input size of the prediction set image to (512*k, 512*k, 3), input the improved YOLO vehicle detector again, and predict the final accurate vehicle bounding box, that is, the coordinate file of the prediction set image.

[0078] Reset the RGB values ​​of the pixels inside the predicted bounding box to (255, 255, 255) to remove the vehicle noise on the image.

[0079] Figure 3 The results of vehicle recognition using the improved YOLO deep learning framework on various aerial image datasets are shown. After the position of the bounding box is predicted with high precision, the image pixels inside the bounding box are reset to (255, 255, 255), that is, the area containing the vehicle is univalued.

[0080] Step 3: Re-match the image positions, calculate the ground overlap rate and occlusion rate between different images, dynamically optimize the spatial distribution of images, and reduce the number of aerial photos.

[0081] Step 3.1, extract the feature points on all images through the feature extraction algorithm, match the feature points on adjacent images according to the similarity of the feature points, and obtain the actual overlapping area and relative position relationship of adjacent images. The image feature extraction algorithm can use SIFT (Scale Invariant Feature Transform) algorithm, SURF (Speeded Up RobustFeature) algorithm, FAST (Features From Accelerated Segment Test) algorithm, etc. The position relationship formula of adjacent images can be solved through multiple known point pairs to obtain the rotation matrix and translation matrix.

[0082] refer to Figure 4 In order to estimate the actual spatial relationship between aerial images, feature extraction algorithms are used to extract feature points on all images. And match the feature points on adjacent images according to the similarity to generate feature point pairs

[0083] The actual overlapping area and relative position relationship of adjacent images can be described by the rotation matrix R and the translation matrix T:

[0084]

[0085] By solving the actual values ​​of the rotation matrix R and the translation matrix T through multiple feature point pairs, the spatial relationship between images can be obtained.

[0086] Step 3.2, see Figure 5 , according to the spatial position of adjacent images and the vehicle occlusion position on the image, calculate the actual occlusion rate OCR and ground overlap rate GOR between images. The vehicle occlusion rate is the ratio of the vehicle overlap area on multiple images to the image overlap area, and the ground overlap rate is the overlap ratio of the area outside the vehicle occlusion position. The overlap rate of the original image sequence should not be less than 90%, and the optimized image overlap rate should be between 80% and 90%. The ground overlap rate should be greater than 70% to ensure the alignment accuracy during the 3D reconstruction of the road surface.

[0087] Specifically,

[0088] Occlusion rate (OCR) = the area of ​​the part still blocked by the vehicle after the two images are superimposed / the area of ​​the original image.

[0089] Ground overlap ratio GOR = area of ​​the vehicle-blocked part after superimposing the two images / area of ​​the original image.

[0090] In step 3.3, according to the actual occlusion rate and ground overlap rate between images, invalid photos are identified through image sequence space optimization to reduce the number of images used for 3D reconstruction.

[0091] refer to Figure 6 ,According to the actual occlusion rate OCR and ground overlap rate GOR between images, the spatial distribution of the image sequence is optimized through the following steps:

[0092] (1) Assume the overlap ratio of the original image sequence is IOR max , then the lowest image overlap rate after optimization is IOR min =1-2*(1-IOR max );

[0093] (2) Starting from the first image, the search is performed at 2-step intervals to calculate the occlusion rate OCR1 obtained by combining the i-th image and the i+2-th image, that is, the ratio of the vehicle overlap area of ​​the two images to the image overlap area. At the same time, the actual ground overlap rate GOR1 of the area outside the vehicle occlusion position can be obtained.

[0094] (3) Calculate the occlusion rate OCR2 obtained by combining the i-th image, the i+1-th image, and the i+2-th image, that is, the ratio of the vehicle overlapping area of ​​the three images to the image overlapping area. At the same time, the actual ground overlap rate GOR2 of the area outside the vehicle occlusion position can be obtained.

[0095] (4) If OCR2>OCR1 and GOR2>GOR1, keep the i+1th image; otherwise, delete the i+1th image.

[0096] refer to Figure 7 , shows the image distribution of two road sections before and after the spatial optimization of aerial image sequences. After optimization, the number of images involved in modeling can be greatly reduced, the image density in areas with low traffic volume can be effectively reduced, and redundant images can be deleted.

[0097] Step 4: Reconstruct the three-dimensional morphology of the road surface through stereo vision based on the preprocessed aerial images.

[0098] Step 4.1, re-match the preprocessed image generated in step 3 with the image coordinates extracted in step 1 to preliminarily estimate the relative spatial position of the camera; extract the feature points on each photo and match the feature points of adjacent images; and inversely calculate the spatial coordinates of the camera based on the feature point pairs and spatial epipolar geometry.

[0099] Step 4.2, based on the obtained camera space coordinates, solve the spatial three-dimensional coordinates of all feature points on the image to obtain a point cloud model.

[0100] As one of the preferred solutions, the aerial image preprocessing operations in steps 2 and 3 can be performed simultaneously with the image three-dimensional reconstruction operation in step 4, and the processing can be performed in batches.

[0101] Based on the preprocessed image sequence, the precise position of the camera is calculated through stereo vision, and the spatial coordinates of the feature points on the image are estimated. The generated point cloud model is as follows: Figure 8 At the same time, the spatial accuracy of the road surface point cloud model generated by the present invention is verified.

[0102] The present invention also discloses a three-dimensional road reconstruction for dynamic traffic scenes, including:

[0103] An image acquisition module, used to collect images taken by drones;

[0104] The coordinate acquisition module inputs the captured image as the prediction set image into the improved YOLO vehicle detector to obtain the coordinate file of the vehicle in the captured prediction set image; based on the vehicle coordinate file, the area within the vehicle boundary box in the prediction set image is singularized;

[0105] The image sequence optimization module obtains the spatial relationship between adjacent images in the prediction set through feature extraction method, calculates the actual occlusion rate and ground overlap rate between adjacent images through the spatial relationship between adjacent images and the vehicle boundary box on each image, and identifies invalid images by combining image sequence spatial optimization, removes invalid images, and obtains preprocessed images;

[0106] The reconstruction module is used to combine the preprocessed image and the image coordinate file to reconstruct the three-dimensional shape of the road surface through stereo vision.

[0107] Table 1 below shows the comparison between the digital model measurement and the manual measurement results. The average relative errors of the 7-meter, 12-meter and 20-meter models reconstructed in this embodiment are 0.23%, 0.23% and 0.29% respectively, and it is found that the road point cloud models generated under different photography heights and traffic volumes have a high level of spatial accuracy.

[0108] Table 1 Measurement accuracy of digital model

[0109]

[0110]

[0111] refer to Fig. 9 , the processing time of the modeling method of the present invention and the traditional modeling method in road 3D reconstruction was compared. Compared with the traditional modeling method, the modeling time of the modeling method of the present invention can be reduced by 38.28% on average. According to the traffic flow conditions of different road sections, the optimization efficiency of the present invention is between 30% and 50%.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A road surface 3D reconstruction method for dynamic traffic scenes, characterized in that: The following steps are involved: Step 1, collect images taken by drone; Step 2: Input the captured image as a prediction set image into the improved YOLO vehicle detector to obtain the coordinate file of the vehicle in the captured prediction set image; based on the coordinate file of the vehicle, the area within the vehicle boundary box in the prediction set image is singularized; The process of obtaining the improved YOLO vehicle detector is as follows: The images extracted from the input images by the 6th, 12th and 14th layers of the backbone feature extraction network are sampled, spliced ​​and feature extracted respectively. Then, based on the 9 predefined anchor boxes, the type, position and confidence of the bounding box are predicted through standard convolution to obtain the vehicle bounding box corner point coordinate file dataset of the predicted image. Compare the labeled image vehicle bounding box corner point coordinate file dataset and the predicted image vehicle bounding box corner point coordinate file dataset to obtain an error, train the YOLO vehicle detector until the error is less than a set value, and obtain an improved YOLO vehicle detector; The backbone feature extraction network includes 1 CBL module and 13 DBR modules; the CBL module includes a standard convolution layer, a batch normalization layer and a Leaky ReLU activation layer; the DBR module includes a depth-separable convolution layer, a batch normalization layer and a ReLU activation layer; Step 3: obtain the spatial relationship between adjacent images in the prediction set image by feature extraction method, calculate the actual occlusion rate and ground overlap rate between adjacent images by the spatial relationship between adjacent images and the vehicle bounding box on each image, identify invalid images by combining image sequence space optimization, remove invalid images, and obtain preprocessed images; Combine the preprocessed image and the image coordinate file to reconstruct the three-dimensional shape of the road surface through stereo vision.

2. The method for three-dimensional road reconstruction for dynamic traffic scenes according to claim 1, characterized in that: Before step 1, the boundary coordinates of the planned road surface area are extracted from the electronic map, a flight area file is generated, the flight area file is imported into the drone, and the drone shoots the planned road surface area.

3. The method for three-dimensional road reconstruction for dynamic traffic scenes according to claim 1, characterized in that: In step 2, the process of obtaining the coordinate file of the vehicle in the shooting prediction set image by improving the YOLO vehicle detector is as follows: (1) Input the captured image into the improved YOLO vehicle detector with an initial size of (512, 512, 3), preliminarily predict the vehicle bounding box, and calculate the average pixel width of the vehicle; (2) Calculate the scaling factor k based on the ratio of the average pixel width of the vehicle in the training set image to the average pixel width of the vehicle in the prediction set image; (3) The input size of the prediction set image is adjusted to (512*k, 512*k, 3) by the scaling factor and then input into the deep learning framework again to predict the final accurate vehicle bounding box.

4. The method for three-dimensional road reconstruction for dynamic traffic scenes according to claim 1, characterized in that: In step 3, the spatial relationship between the adjacent images is calculated by the following formula: in, is the feature point on the image, R is the rotation matrix, and T is the translation matrix; through multiple feature point pairs, the rotation matrix is ​​calculated and translation matrix The actual value of is used to obtain the spatial relationship between adjacent images.

5. The method for three-dimensional road reconstruction for dynamic traffic scenes according to claim 1, characterized in that: In step 3, the process of identifying invalid images is: (1) Assume that the overlap rate of the original image sequence is , then the lowest image overlap rate after optimization is ; (2) Start searching from the first image at 2-step intervals and calculate the i image and i+ Occlusion rate obtained by combining two images , that is, the ratio of the vehicle overlap area of ​​the two images to the image overlap area; at the same time, the actual ground overlap rate of the area outside the vehicle occlusion position can be obtained ; (3) Calculate the i image, i+ 1 image and i+ Occlusion rate obtained by combining two images , that is, the ratio of the vehicle overlap area of ​​the three images to the image overlap area. At the same time, the actual ground overlap rate of the area outside the vehicle occlusion position can be obtained. ; (4) If and , then keep the i+ 1 image, otherwise delete the i+ 1 image.

6. The method for three-dimensional road reconstruction for dynamic traffic scenes according to claim 5, characterized in that: The calculation formula of occlusion rate OCR is: Occlusion Rate = the area of ​​the part still blocked by the vehicle after the two images are superimposed / the area of ​​the original image; The calculation formula of ground overlap rate GOR is: Ground overlap =The area of ​​the vehicle-occluded part after the two images are superimposed / the area of ​​the original image.

7. The method for three-dimensional road reconstruction for dynamic traffic scenes according to claim 1, characterized in that: In step 4, firstly, the coordinates of the preprocessed image and the image in step 1 are matched to preliminarily estimate the relative spatial position of the camera between different photos; feature points on each image are extracted and matched with feature points of adjacent images; The spatial coordinates of the camera are inversely calculated based on the feature point pairs and spatial epipolar geometry; the spatial three-dimensional coordinates of all feature points on the image are solved based on the camera spatial coordinates to obtain a point cloud model, and then the three-dimensional morphology of the road surface is reconstructed through stereo vision.

8. A road surface 3D reconstruction system for dynamic traffic scenes, characterized in that: include An image acquisition module, used to collect images taken by drones; The coordinate acquisition module inputs the captured image as the prediction set image into the improved YOLO vehicle detector to obtain the coordinate file of the vehicle in the captured prediction set image; based on the vehicle coordinate file, the area within the vehicle boundary box in the prediction set image is singularized; The process of obtaining the improved YOLO vehicle detector is as follows: The images extracted from the input images by the 6th, 12th and 14th layers of the backbone feature extraction network are sampled, spliced ​​and feature extracted respectively. Then, based on the 9 predefined anchor boxes, the type, position and confidence of the bounding box are predicted through standard convolution to obtain the vehicle bounding box corner point coordinate file dataset of the predicted image. Compare the labeled image vehicle bounding box corner point coordinate file dataset and the predicted image vehicle bounding box corner point coordinate file dataset to obtain an error, train the YOLO vehicle detector until the error is less than a set value, and obtain an improved YOLO vehicle detector; The backbone feature extraction network includes 1 CBL module and 13 DBR modules; the CBL module includes a standard convolution layer, a batch normalization layer and a Leaky ReLU activation layer; the DBR module includes a depth-separable convolution layer, a batch normalization layer and a ReLU activation layer; The image sequence optimization module obtains the spatial relationship between adjacent images in the prediction set through feature extraction method, calculates the actual occlusion rate and ground overlap rate between adjacent images through the spatial relationship between adjacent images and the vehicle boundary box on each image, and identifies invalid images by combining image sequence spatial optimization, removes invalid images, and obtains preprocessed images; The reconstruction module is used to combine the preprocessed image and the image coordinate file to reconstruct the three-dimensional shape of the road surface through stereo vision.

Images