A method for calculating the height difference of road kiln manhole covers based on fixed-point aerial photography of unmanned aerial vehicles

Abstract

The present invention relates to the field of transportation infrastructure detection technology, specifically, a method for calculating the height difference of road manhole covers based on fixed-point aerial photography by unmanned aerial vehicles, comprising the following steps: step 1, determining the location of the manhole cover whose height difference needs to be calculated; step 2, taking five photos at fixed points above the manhole cover using an unmanned aerial vehicle according to the manhole cover whose height difference is to be calculated in step 1; step 3, performing three-dimensional reconstruction on the photos taken by the unmanned aerial vehicle in step 2; step 4, calculating the height difference of the manhole cover according to the three-dimensional reconstruction results of the aerial photos in step 3. The method for calculating the height difference of road manhole covers based on fixed-point aerial photography by unmanned aerial vehicles provided by the present invention uses a camera mounted on an unmanned aerial vehicle to perform fixed-point aerial photography on the road manhole covers to obtain photos of the manhole covers for three-dimensional reconstruction, and calculates the height difference of the manhole covers according to the three-dimensional reconstruction results. The method has the characteristics of non-contact, non-interruption of traffic, real-time, high intelligence, simple operation, high efficiency, low cost, and measurability.

Description

Technical Field

[0001] The invention relates to the technical field of transportation infrastructure detection, and in particular to a method for calculating the height difference of road manhole covers based on fixed-point aerial photography by an unmanned aerial vehicle. Background Art

[0002] At present, there are tens of thousands of manhole covers on urban roads. Every year, there are endless car accidents and pedestrian injuries caused by missing, moving, or damaged manhole covers. At present, manual inspection and intelligent sensors are mainly used to detect and monitor manhole covers. Manual inspection uses a wooden ruler and a steel feeler gauge to measure the height difference of the manhole cover more accurately. However, the efficiency of manual measurement is extremely low. Due to the requirements of road traffic, it is not feasible to use traditional manual methods to detect a large number of manhole covers. With the use of intelligent sensors, due to the limitations of the manhole cover detection environment, only GPS positioning technology and inclination sensors can be used to determine whether the manhole cover is moved or missing. The height difference of the manhole cover is the most common disease of the road manhole cover, and it is also an important indicator that affects the stability and comfort of driving. At present, there is no method to quickly and effectively measure the height difference of the manhole cover. Summary of the invention

[0003] The purpose of the present invention is to solve the shortcomings of the prior art and provide a method for calculating the height difference of kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles, so as to realize the rapid identification and calculation of the height difference of kiln manhole covers on roads. This method is a non-contact fixed-point measurement, does not affect traffic, and achieves low cost, high precision and high measurement efficiency.

[0004] In order to achieve the above purpose, a method for calculating the height difference of kiln manhole cover based on fixed-point aerial photography by UAV is designed, which includes the following steps:

[0005] Step 1, determine the location of the kiln manhole cover whose height difference needs to be calculated;

[0006] Step 2, according to the kiln manhole cover whose height difference is to be calculated in step 1, use a drone to take five photos at a fixed point above the manhole cover;

[0007] Step 3, three-dimensionally reconstructing the photos taken by the drone in step 2;

[0008] Step 4, calculate the height difference of the kiln manhole cover based on the 3D reconstruction results of the aerial photos in step 3.

[0009] The present invention also has the following preferred technical solutions:

[0010] Furthermore, the method for determining the position of the kiln manhole cover in step 1 includes:

[0011] a. The longitude and latitude coordinates of the manhole cover to be tested obtained from the map information; or

[0012] b. Based on the road name combined with distance and lane position information; or

[0013] c.Specified on site.

[0014] Furthermore, in step 2, the drone's fixed-point aerial photography is completed in one shot at the manhole cover's automatic identification and automatic fixed position point above it.

[0015] Furthermore, the drone shooting method in step 2 is as follows:

[0016] 1. Determination of the shooting height of the drone. The conversion relationship between the photography resolution and the shooting height of the drone and the pixel size of the mounted camera is as follows:

[0017]

[0018] Where, D: normal photography resolution; ccd: camera sensor size; H: shooting height; d: camera focal length;

[0019] 2. Assuming the diameter of the manhole cover is l, the relationship between the drone shooting height and the camera gimbal angle is as follows:

[0020] In the formula, α is the aerial view angle of the drone’s gimbal;

[0021]

[0022] Because the sensor size CCD and focal length d are much smaller than the drone shooting height, formula (2) can be simplified to:

[0023] Furthermore, in step 2, the manual shooting method of the drone is as follows:

[0024] (1) Manually control the drone to take off to the top of the kiln manhole cover in step 1;

[0025] (2) Adjust the drone shooting height;

[0026] (3) Take a photo from a bird’s-eye view;

[0027] (4) Adjust the camera gimbal angle to a downward angle of α degrees and take a photo;

[0028] (5) Rotate the drone 90° in one direction for a total of three times, taking a photo each time.

[0029] Furthermore, in step 2, the automatic shooting method of the drone is as follows:

[0030] (1) Collect photos of road manhole covers of various shapes, sizes and materials;

[0031] (2) Use the LabelImg or LabelMe image standard tool to frame or click on the kiln manhole cover photo;

[0032] (3) Use YOLO series or Yolact series deep learning algorithms to train road manhole cover photo samples, obtain manhole cover target classification or semantic segmentation model, and deploy the trained model on the drone;

[0033] (4) The UAV automatically flies to the designated location based on the location determined in step 1, and locks the exact location of the kiln manhole cover based on the kiln manhole cover target classification or semantic segmentation recognition results;

[0034] (5) Based on the manhole cover position recognition results and the pre-set shooting height and angle requirements, the drone automatically adjusts its position and takes 5 photos;

[0035] (6) The shooting height range of the UAV can be H meters, and the shooting angle is α degrees. A bird's-eye view photo is taken at a determined shooting height, and the camera gimbal is adjusted to take a bird's-eye view photo of α degrees. Then, the UAV takes a photo every time it rotates 90°, and rotates three times 90° in total, thereby obtaining a bird's-eye view photo and a bird's-eye view photo of α degrees in four directions. A total of 5 photos are used for the three-dimensional reconstruction of the kiln manhole cover.

[0036] Furthermore, the specific method of step 3 is as follows:

[0037] (1) Based on aerial photos, image feature points are extracted, feature matching between images is calculated based on the feature points, sparse reconstruction is performed based on the matched features to obtain the camera pose and sparse feature point cloud of each image, dense reconstruction is performed based on the camera pose to obtain a dense point cloud, and a grid is reconstructed based on the point cloud;

[0038] (2) Based on the oblique photography images of unmanned aerial vehicles, the high-precision positioning of the POS system and the ground control points, the oblique images are subjected to aerial triangulation and joint adjustment of the regional network. After dense matching of point clouds, construction of irregular triangulated networks and texture mapping, three-dimensional modeling is completed.

[0039] Furthermore, the specific method of step 4 is as follows:

[0040] (1) The difference between the average value of the height coordinates of the three-dimensional point cloud of the manhole cover and the average value of the height coordinates of the point cloud of the road surface around the manhole cover is the manhole cover height difference;

[0041] (2) The height difference of the manhole cover is the difference between the average height coordinates of the three points on the manhole cover and the average height coordinates of the three points on the road surface around the manhole cover.

[0042] Advantageous Effects of the Invention

[0043] The advantages of the method for calculating the height difference of road manhole covers based on fixed-point aerial photography by unmanned aerial vehicles provided by the present invention include but are not limited to: the present invention uses a camera mounted on an unmanned aerial vehicle to perform fixed-point aerial photography of road manhole covers to obtain photos of the manhole covers for three-dimensional reconstruction, and calculates the height difference of the manhole covers based on the three-dimensional reconstruction results. This method has the characteristics of non-contact, non-interruption of traffic, real-time, high degree of intelligence, simple operation, high efficiency, low cost, and measurability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The schematic diagram of the process of calculating the height difference of a road kiln manhole cover based on fixed-point aerial photography of a UAV according to the present invention is exemplified;

[0045] Figure 2 An illustrative top view of the shooting angle of the automatic shooting method of a drone described in the present invention is shown;

[0046] Figure 3 The side view schematic diagram of the shooting angle of the automatic shooting method of the drone described in the present invention is exemplified. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that unless otherwise specifically stated, the descriptions of the components and steps, numerical expressions and numerical values ​​described in these embodiments do not limit the scope of the present invention.

[0048] See also Figure 1 The purpose of the present invention is to provide a method for calculating the height difference of kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles in view of the height difference generated during the use of kiln manhole covers. The present invention realizes the rapid identification and calculation of the height difference of kiln manhole covers on roads. The method is a non-contact fixed-point measurement, which does not affect traffic and has low cost, high accuracy and high measurement efficiency.

[0049] The technical solution adopted by the present invention to solve the technical problem is: construct a method for calculating the height difference of kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles, comprising the following steps: A method for calculating the height difference of kiln manhole covers on roads based on fixed-point aerial photography by unmanned aerial vehicles, characterized in that:

[0050] include

[0051] Step 1, determine the location of the kiln manhole cover whose height difference needs to be calculated.

[0052] Step 2: Based on the kiln manhole cover whose height difference is to be calculated in step 1, use a drone to take 5 photos at a fixed point above the manhole cover.

[0053] Step 3: Reconstruct the three-dimensional photos taken by the drone in step 2.

[0054] Step 4, calculate the height difference of the kiln manhole cover based on the 3D reconstruction results of the aerial photos in step 3.

[0055] The method for determining the position of the kiln manhole cover in step 1 comprises:

[0056] a) The longitude and latitude coordinates of the manhole cover to be tested obtained through Baidu Maps, Amap, Google Maps, GIS, etc.;

[0057] b) Combine the road name with the location information such as distance and lane;

[0058] c) On-site designation;

[0059] In step 2, the fixed-point aerial photography of the drone only needs to fix the position point above the manhole cover and take 5 photos of the manhole cover at a time according to the specified rules, such as Figure 2 and Figure 3 As shown, a high-precision three-dimensional model of the manhole cover can be obtained through three-dimensional reconstruction of the image, and the height difference of the manhole cover can be accurately calculated.

[0060] In step 3, the drone takes photos of the manhole cover at a fixed point, and the shooting rule parameters are determined as follows:

[0061] The determination of the drone shooting height, in addition to being affected by the surrounding environmental factors above the kiln manhole cover, should be as low as possible under the conditions of flight conditions, so as to obtain a higher 3D reconstruction resolution. Figure 2 As shown in the figure, the conversion relationship between the resolution of the front view photography and the shooting height of the drone and the pixel size of the mounted camera is as follows:

[0062]

[0063] Where: D: normal photography resolution;

[0064] ccd: camera sensor size;

[0065] H: shooting height;

[0066] d: Camera focal length.

[0067] according to Figure 2 , assuming the diameter of the manhole cover is l, the relationship between the drone shooting height and the camera gimbal angle is as follows:

[0068]

[0069] Where: α: The aerial view angle of the drone’s gimbal;

[0070]

[0071] Because the sensor size CCD and focal length d are much smaller than the drone shooting height, formula (2) can be simplified to:

[0072]

[0073] In step 2, the drone takes photos of the manhole cover at a fixed point. The drone can be manually controlled to take photos of the manhole cover for 3D reconstruction. The steps are as follows:

[0074] Manually control the drone to take off to just above the kiln manhole cover in step 1;

[0075] Adjust the drone to a suitable shooting height;

[0076] Take a photo from a bird's-eye view;

[0077] Adjust the camera gimbal angle to α degrees and take a photo;

[0078] Rotate the drone 90° in one direction for 3 times, taking a photo each time.

[0079] After obtaining 5 photos of the kiln manhole cover to be tested, the three-dimensional reconstruction of the kiln manhole cover can be performed.

[0080] In step 2, the drone takes photos of the manhole cover at a fixed point. The drone can automatically take photos of the manhole cover for 3D reconstruction. The steps are as follows:

[0081] Collect photos of road manhole covers of various shapes, sizes and materials;

[0082] Use standard image tools such as LabelImg or LabelMe to frame or click on the kiln manhole cover photos;

[0083] Use deep learning algorithms such as the YOLO series or Yolact series to train road manhole cover photo samples, obtain manhole cover target classification or semantic segmentation models, and deploy the trained models on drones;

[0084] The drone automatically flies to the designated location based on the location determined in step 1, and locks the exact location of the kiln manhole cover based on the kiln manhole cover target classification or semantic segmentation recognition results;

[0085] According to the manhole cover position recognition results and the pre-set shooting height and angle requirements, the drone automatically adjusts its position and takes 5 photos;

[0086] The shooting height range of the drone can be H meters, and the shooting angle is α degrees. Take a bird's-eye view photo at the determined shooting height, and adjust the camera gimbal to take a bird's-eye view photo at α degrees. Then, the drone takes a photo every 90° rotation, and rotates 90° three times in total. Thus, a bird's-eye view photo and a bird's-eye view photo of α degrees in four directions are obtained, and a total of 5 photos are used for the 3D reconstruction of the kiln manhole cover.

[0087] By mounting a five-lens tilt camera and taking a picture once at a fixed height, five photos can be obtained for the three-dimensional reconstruction of the kiln manhole cover.

[0088] The 3D reconstruction method of the fixed-point aerial photos taken by the UAV in step 3 includes:

[0089] Based on aerial photos, image feature points are extracted, and feature matching between images is calculated based on the feature points. Sparse reconstruction is performed based on the matched features to obtain the camera pose and sparse feature point cloud of each image. Dense reconstruction is performed based on the camera pose to obtain a dense point cloud. The grid is reconstructed based on the point cloud. The model accuracy can reach 1mm, which meets the requirements for manhole cover height difference measurement.

[0090] Based on the oblique photography images of drones, high-precision positioning of the POS system and ground control points, aerial triangulation and regional network joint adjustment are carried out on the oblique images. After dense matching of point clouds, construction of irregular triangulated networks and texture mapping, three-dimensional modeling is completed. The model accuracy can reach 0.5mm, which meets the requirements for manhole cover height difference measurement.

[0091] The calculation method of the height difference of the kiln manhole cover in step 4 includes:

[0092] The difference between the average height coordinate of the three-dimensional point cloud of the manhole cover and the average height coordinate of the point cloud of the road surface around the manhole cover is the manhole cover height difference.

[0093] The difference between the average height coordinates of the three points on the manhole cover and the average height coordinates of the three points on the road surface around the manhole cover is the manhole cover height difference.

[0094] As a preferred embodiment, the embodiment specifically includes the following steps:

[0095] Step 1: Determine the exact location of the manhole cover to be evaluated based on the latitude and longitude coordinates of the manhole cover to be tested obtained through Baidu Maps, Amap, Google Maps, GIS, etc., or describe the exact location of a given manhole cover by location such as road name and distance to the lane.

[0096] Step 2: Determine the flight altitude and bird's-eye view angle of the drone based on the characteristics of the surrounding environment of the manhole cover and the size and focal length of the mounted camera sensor.

[0097] Step three, according to the flight height H and the overlooking angle α calculated in step two, manually control the drone to fly to the top of the manhole cover, adjust the flight height to H to hover and take a bird's-eye view photo, and then adjust the gimbal to angle α to take a bird's-eye view photo. Adjust the flight direction of the drone, rotate 90° clockwise horizontally, take the second bird's-eye view photo, and then take a bird's-eye view photo every 90° rotation, and take a total of 4 bird's-eye view photos. After the shooting is completed, retract the drone. In addition, an automatic method of artificial intelligence can be used to take photos of manhole covers. The implementation method is to annotate the manhole cover photos collected online through LabelImage or Labelme, and then train the yolo series manhole cover target recognition or yolact series manhole cover semantic segmentation neural network model based on the annotated photos, and arrange the model on the drone. By designing the flight height and the overlooking angle parameters, the drone automatically takes off, locks the target after identifying the manhole cover, automatically adjusts the flight height, and takes photos according to the set parameters. After the shooting is completed, it automatically returns to the take-off location or flies to the next manhole cover monitoring point.

[0098] Step 4: Based on the aerial photos of the manhole cover, the feature point matching method can be used for 3D modeling. If the drone has a high-precision positioning function, the high-precision positioning information of its POS system can be imported, and the 3D model of the manhole cover can be established using the oblique photogrammetry method.

[0099] Step 5: Based on the three-dimensional model of the manhole cover, the difference between the average height coordinate of the three-dimensional point cloud of the manhole cover and the average height coordinate of the point cloud of the road surface around the manhole cover is extracted as the manhole cover height difference. Alternatively, the difference between the average height coordinate of three points on the manhole cover and the average height coordinate of three points on the road surface around the manhole cover is extracted as the manhole cover height difference.

[0100] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical solutions and novel concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for calculating the height difference of road kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles, characterized in that: The method comprises the following steps: Step 1, determine the location of the kiln manhole cover whose height difference needs to be calculated; Step 2, according to the kiln manhole cover whose height difference is to be calculated in step 1, use a drone to take five photos at a fixed point above the manhole cover; Step 3, three-dimensionally reconstructing the photos taken by the drone in step 2; Step 4, calculating the height difference of the kiln manhole cover according to the three-dimensional reconstruction result of the aerial photo in step 3; wherein, The drone shooting method in step 2 is as follows:

1. Determination of the shooting height of the drone. The conversion relationship between the photography resolution and the shooting height of the drone and the pixel size of the mounted camera is as follows: Where, D: normal photography resolution; ccd: camera sensor size; H: shooting height; d: camera focal length; 2. Assuming the diameter of the manhole cover is l, the relationship between the drone shooting height and the camera gimbal angle is as follows: In the formula, α is the aerial view angle of the drone’s gimbal; Because the sensor size CCD and focal length d are much smaller than the drone shooting height, formula (2) can be simplified to:

2. The method for calculating the height difference of road kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles as claimed in claim 1 is characterized in that: The method for determining the position of the kiln manhole cover in step 1 comprises: a. The longitude and latitude coordinates of the manhole cover to be tested obtained from the map information; or b. Based on the road name combined with distance and lane position information; or c.Specified on site.

3. The method for calculating the height difference of road kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles as claimed in claim 1 is characterized in that: In step 2, the drone performs fixed-point aerial photography at the manhole cover automatically identified and automatically fixed position points above it, and the shooting is completed in one go.

4. The method for calculating the height difference of road kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles as claimed in claim 1 is characterized in that: In step 2, the manual shooting method of the drone is as follows: (1) Manually control the drone to take off to the top of the kiln manhole cover in step 1; (2) Adjust the drone shooting height; (3) Take a photo from a bird’s-eye view; (4) Adjust the camera gimbal angle to a downward angle of α degrees and take a photo; (5) Rotate the drone 90° in one direction for a total of three times, taking a photo each time.

5. The method for calculating the height difference of road kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles as claimed in claim 1 is characterized in that: In step 2, the automatic shooting method of the drone is as follows: (1) Collect photos of road manhole covers of various shapes, sizes and materials; (2) Use the LabelImg or LabelMe image standard tool to frame or click on the kiln manhole cover photo; (3) Use YOLO series or Yolact series deep learning algorithms to train road manhole cover photo samples, obtain manhole cover target classification or semantic segmentation model, and deploy the trained model on the drone; (4) The UAV automatically flies to the designated location based on the location determined in step 1, and locks the exact location of the kiln manhole cover based on the kiln manhole cover target classification or semantic segmentation recognition results; (5) Based on the manhole cover position recognition results and the pre-set shooting height and angle requirements, the drone automatically adjusts its position and takes 5 photos; (6) The shooting height range of the UAV can be H meters, and the shooting angle is α degrees. A bird's-eye view photo is taken at a determined shooting height, and the camera gimbal is adjusted to take a bird's-eye view photo of α degrees. Then, the UAV takes a photo every time it rotates 90°, and rotates three times 90° in total, thereby obtaining a bird's-eye view photo and a bird's-eye view photo of α degrees in four directions. A total of 5 photos are used for the three-dimensional reconstruction of the kiln manhole cover.

6. The method for calculating the height difference of road kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles as claimed in claim 1 is characterized in that: The specific method of step 3 is as follows: (1) Based on aerial photos, image feature points are extracted, feature matching between images is calculated based on the feature points, sparse reconstruction is performed based on the matched features to obtain the camera pose and sparse feature point cloud of each image, dense reconstruction is performed based on the camera pose to obtain a dense point cloud, and a grid is reconstructed based on the point cloud; (2) Based on the oblique photography images of unmanned aerial vehicles, the high-precision positioning of the POS system and the ground control points, the oblique images are subjected to aerial triangulation and joint adjustment of the regional network. After dense matching of point clouds, construction of irregular triangulated networks and texture mapping, three-dimensional modeling is completed.

7. The method for calculating the height difference of road kiln manhole covers based on fixed-point aerial photography by unmanned aerial vehicles as claimed in claim 1 is characterized in that: The specific method of step 4 is as follows: (1) The difference between the average value of the height coordinates of the three-dimensional point cloud of the manhole cover and the average value of the height coordinates of the point cloud of the road surface around the manhole cover is the height difference of the manhole cover; or (2) The height difference of the manhole cover is the difference between the average height coordinates of the three points on the manhole cover and the average height coordinates of the three points on the road surface around the manhole cover.

Images