Diversion tunnel defect three-dimensional detection device and method based on unmanned aerial vehicle

Abstract

The invention provides a diversion tunnel defect three-dimensional detection device and method based on an unmanned aerial vehicle, and belongs to the technical field of diversion tunnel defect detection.The diversion tunnel defect three-dimensional detection device comprises the unmanned aerial vehicle, a multi-view high-definition camera, a lighting device, a laser radar device, a positioning module and a data processing module, the positioning module comprises an inertial measurement unit, a binocular positioning camera and a barometric altimeter, and the positioning module fuses and calibrates measurement data of the inertial measurement unit, the binocular positioning camera and the barometric altimeter by adopting a filtering algorithm to obtain accurate position data of the unmanned aerial vehicle; and the data processing module is used for constructing a tunnel high-definition live-action three-dimensional model according to the high-definition image data and the position data acquired by the multi-view high-definition camera and the point cloud data acquired by the laser radar device, performing multi-modal defect automatic detection and performing three-dimensional quantitative labeling and display of defects in the tunnel high-definition live-action three-dimensional model. According to the invention, the automation level of tunnel defect detection and the richness of results are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of diversion tunnel defect detection, more specifically, to a diversion tunnel defect three-dimensional detection device and method based on an unmanned aerial vehicle. BACKGROUND

[0002] As a key facility of water conservancy projects, diversion tunnels are prone to defects such as cracks, seepage, lining shedding, bulging and collapse due to long-term water erosion, geological changes and other factors. In the traditional detection method, manual detection relying on naked eye observation or simple tools has the limitations of low efficiency and strong subjectivity; and the detection period is long and the safety risk is high.

[0003] The tunnel defect detection based on an unmanned aerial vehicle (UAV) realizes non-contact and rapid inspection by carrying multiple sensors such as cameras, laser radars and infrared thermal imagers, but the existing technology still has the following challenges: (1) During data collection, there is no GNSS signal in the tunnel, the positioning accuracy is not accurate, the error of the traditional UAV positioning method (such as inertial navigation) accumulates significantly, which affects the accuracy of modeling and defect positioning, and there is a risk of crashing the UAV and damaging the inner wall of the tunnel; and the conventional UAV single lens is located below the fuselage, which cannot quickly obtain high-definition straight-on shooting images of the full cross section of the ring-shaped tunnel, especially the top area of the tunnel. The obtained images may have the problems of incomplete coverage, low resolution and large distortion caused by oblique shooting, which affects the quality of tunnel modeling.

[0004] (2) During data modeling, the geometric accuracy of pure image modeling is low, and the tunnel curved surface structure is prone to geometric distortion during three-dimensional reconstruction; and pure point cloud modeling (such as SLAM modeling) only uses low-resolution panoramic images to colorize the point cloud, although its geometric accuracy is higher, but the texture clarity is lower, which affects the visualization extraction and quantitative analysis of the apparent defects of the tunnel.

[0005] (3) During defect identification, the conventional two-dimensional identification and detection technology based on images or videos cannot obtain the accurate spatial position and three-dimensional shape of the defects, cannot quantify key parameters such as defect depth and volume, and is prone to misidentification of defects such as bulging and collapse which have obvious three-dimensional features but not obvious texture features. SUMMARY

[0006] The present application aims to provide a diversion tunnel defect three-dimensional detection device and method based on an unmanned aerial vehicle to solve at least one of the above problems in the prior art.

[0007] To achieve the above-mentioned purpose, the first aspect of the present application provides a diversion tunnel defect three-dimensional detection device based on an unmanned aerial vehicle, comprising: an unmanned aerial vehicle; A multi-view high-definition camera, a plurality of high-definition lenses of the multi-view high-definition camera are arranged in a ring array on the unmanned aerial vehicle; A lighting device, comprising a plurality of lighting lamps, a plurality of the lighting lamps are arranged in a ring array on the unmanned aerial vehicle, and a plurality of the lighting lamps are arranged in a uniform staggered manner with a plurality of the high-definition lenses; A laser radar device arranged on the unmanned aerial vehicle; A positioning module, comprising an inertial measurement unit, a binocular positioning camera and a barometric altimeter; the positioning module is arranged on the unmanned aerial vehicle; the positioning module adopts a filtering algorithm to fuse and calibrate the measurement data of the inertial measurement unit, the binocular positioning camera and the barometric altimeter, and obtains accurate position data of the unmanned aerial vehicle; and A data processing module for constructing a high-definition real scene three-dimensional model of a tunnel, performing multi-modal defect automatic detection, and performing three-dimensional quantitative labeling and display of defects in the high-definition real scene three-dimensional model of the tunnel according to high-definition image data collected by the multi-view high-definition camera, the position data and point cloud data collected by the laser radar device.

[0008] Further, it further comprises a protection module, the protection module comprises a plurality of protective mesh covers, and a plurality of the protective mesh covers are respectively arranged on the propellers of the unmanned aerial vehicle.

[0009] Further, the construction of the high-definition real scene three-dimensional model of the tunnel comprises the following steps: Pretreatment is performed on the point cloud data to eliminate abnormal data; Image exterior orientation elements and three-dimensional coordinates of encrypted points are obtained according to the image data; Image encrypted point cloud and the point cloud data are spatially registered to obtain fused point cloud; Triangular net is established for the fused point cloud; Texture mapping is performed on the triangular net constructed based on the image data and the spatial relationship of the aerial triangulation.

[0010] Further, the multi-modal defect automatic detection comprises the following steps: According to the image data, a defect image data set is made, and the defect classification of the defect image data set includes cracks, water seepage and erosion; According to the defect image data set, an image semantic segmentation network is trained; The trained image semantic segmentation network is used to detect defects in the high-definition image of the diversion tunnel to be detected.

[0011] Further, the multi-modal defect automatic detection comprises the following steps: According to the point cloud data, a defect point cloud data set is made, and the defect classification of the defect point cloud data set includes bulges and collapse; Based on the defective point cloud dataset, a point cloud semantic segmentation network is trained; A trained point cloud semantic segmentation network is used to classify and detect defects in the point cloud data of the water diversion tunnel to be inspected.

[0012] Furthermore, the three-dimensional quantitative annotation and display of the defect includes the following steps: Based on the defect classification results of the high-definition images, a binary mask image is created for each defect type. Based on the position coordinates of each defect point in the image in the binary mask image, the corresponding position in the high-definition real-scene 3D model of the tunnel is calculated and mapped, and automatically labeled in the form of patches. Different types of defects are labeled with different colors, and the size information of the defects is statistically analyzed based on the label size. Point cloud classification is performed based on the three-dimensional geometric features of the point cloud to separate the point cloud corresponding to the defect; using the defect point cloud obtained from the classification, three-dimensional equidistant line annotations of the defect are generated; the geometric information of bulges and collapses is calculated for quantitative annotation; and the spatial coordinates of the point cloud and the three-dimensional equidistant line annotations are displayed in the high-definition real-scene three-dimensional model of the tunnel.

[0013] Furthermore, the main body of the drone includes two sets of square metal frames and an outer shell connected between the two sets of square metal frames; a landing bracket is connected to each of the four corners of the two sets of square metal frames; and multiple lighting lamps and multiple high-definition lenses are evenly and staggeredly arranged on the outer shell.

[0014] Furthermore, the lidar device is fixed to the square metal frame by a pitch angle gimbal.

[0015] Furthermore, the binocular positioning camera is fixed to the square metal frame by a pitch angle gimbal.

[0016] A second aspect of the present invention provides a method for detecting defects in water diversion tunnels using the aforementioned UAV-based three-dimensional detection device, comprising the following steps: Drones were used to collect high-definition image data and laser point cloud data covering the water diversion tunnel. By constructing point clouds through aerial triangulation of the image data and registering the point cloud data, the image data and point cloud data are fused together to perform multi-source data fusion modeling of the water diversion tunnel, resulting in a high-definition real-scene 3D model of the tunnel. To identify the different characteristics of various typical defects in water diversion tunnels, a multimodal identification strategy is adopted, and a semantic segmentation algorithm is trained to identify defects. Based on the multimodal tunnel defect classification results, the defects are quantitatively labeled and displayed in three dimensions in the high-definition real-scene 3D model of the tunnel according to spatial information.

[0017] Compared with the prior art, the application has the following technical effects: The unmanned aerial vehicle-based diversion tunnel defect three-dimensional detection device provided by the application solves the navigation problem in a tunnel environment by providing high-precision positioning information for data collection in a tunnel GPS signal loss environment through the positioning module arranged on the unmanned aerial vehicle; the device realizes efficient, high-resolution and uniform light multi-source data synchronous collection in a tunnel by arranging a ring array of multi-lens high-definition cameras, lighting devices and a laser radar device on the unmanned aerial vehicle, and solves the problems of tunnel light loss, low efficiency of traditional unmanned aerial vehicle data collection methods, incomplete image coverage and large distortion; the device realizes multi-source data fusion registration and constructs a tunnel high-definition real scene three-dimensional model through the data processing module based on the high-definition image data collected by the multi-lens high-definition cameras and the laser point cloud data collected by the laser radar device, improves the geometric precision and texture definition of the tunnel three-dimensional real scene model, and performs multi-modal defect automatic detection through the data processing module and three-dimensional quantitative labeling and display of defects in the tunnel high-definition real scene three-dimensional model, thereby effectively improving the automation level and richness of diversion tunnel defect detection and providing precise data support for tunnel health monitoring and maintenance decision-making.

[0018] The unmanned aerial vehicle-based diversion tunnel defect three-dimensional detection device and method linkage provided by the application can form a complete diversion tunnel defect three-dimensional detection process, realize unmanned aerial vehicle automatic inspection of diversion tunnel defects, improve the efficiency and quality of tunnel defect detection, and reduce labor costs and safety risks. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A structure diagram of an unmanned aerial vehicle provided for Embodiment 1 of the application; Figure 2 A left view of Figure 1 ; Figure 3 A flowchart of a method for detecting by the unmanned aerial vehicle-based diversion tunnel defect three-dimensional detection device provided for Embodiment 2 of the application; Figure 4 A schematic diagram of three-dimensional labeling of diversion tunnel defects provided for Embodiment 2 of the application.

[0021] Among them, Figure 1 , Figure 2The reference signs in the drawings are: 101, square metal frame, 102, landing support, 103, propeller, 201, high-definition lens, 202, illuminating lamp, 203, shell, 301, laser radar device, 401, inertial measurement unit, 402, binocular positioning camera, 403, barometric altimeter, 501, protective mesh cover. DETAILED DESCRIPTION

[0022] In order to make the technical problems to be solved by the present application, the technical solutions and beneficial effects clearer, the present application will be further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.

[0023] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0024] Embodiment 1 Embodiment 1 of the present application provides a three-dimensional detection device for defects of a diversion tunnel based on a UAV, which comprises a UAV and a multi-view high-definition camera, an illuminating device, a laser radar device and a positioning module carried on the UAV, and further comprises a data processing module. The structure of the UAV is as shown in Figure 1 , Figure 2 .

[0025] In one embodiment, the UAV is a quadcopter, the main body of the UAV includes two square metal frames 101 and a shell 203 connected between the two square metal frames 101, one landing support 102 is connected to each corner of the bottom of the frame, and metal arms are respectively extended to the oblique front and oblique rear of the four corners of the frame, and four sets of propellers 103 are fixed. Further, a protective module is also provided on the UAV, the protective module comprises a plurality of protective mesh covers 501, and the plurality of protective mesh covers 501 are respectively arranged on the propellers 103 of the UAV. In one embodiment, the protective module includes four semicircular protective mesh covers 501, which respectively fix and surround the outer half circle of each propeller 103, for protecting the propellers 103 of the UAV, avoiding the wing breakage when the UAV falls and collides with the wall, and preventing the wing from scratching the inner wall of the tunnel. Since each semicircular protective mesh cover 501 only covers the outer half circle area, it effectively blocks the contact between the front, rear and outer side of the propeller and external objects, while reducing the weight of the UAV; the semicircular protective mesh cover 501 is composed of a thin tubular hard plastic or a thin metal strip, which is relatively light and has sufficient hardness and buffering property.

[0026] The multi-view high-definition camera includes a plurality of high-resolution high-definition lenses 201 arranged in a ring array on the unmanned aerial vehicle. In one embodiment, the multi-view high-definition camera includes 6 high-resolution high-definition lenses 201 arranged in a vertical ring array around the unmanned aerial vehicle body, with an interval angle of 60° between adjacent lenses, and the array plane is perpendicular to the direction of the unmanned aerial vehicle head. When collecting tunnel section image data, the camera lens axis is perpendicular to the tunnel axis (forward direction), and each lens is perpendicular to the tunnel wall for shooting, ensuring that the acquired image has a sub-millimeter resolution and is clear with less distortion.

[0027] The lighting device includes a plurality of lighting lamps 202 arranged in a ring array on the unmanned aerial vehicle, and the plurality of lighting lamps 202 are arranged with uniform intervals with the plurality of high-definition lenses 201. In one embodiment, the multi-view high-definition camera and the lighting device matrix are located in the middle of the unmanned aerial vehicle, the lighting device includes 6 independent light sources of the same specification, and each lighting lamp 202 is installed between adjacent high-definition lenses 201 of the multi-view high-definition camera. The 6 lighting lamps 202 and the 6 high-definition lenses 201 are arranged on the side of the shell 203 with uniform intervals, that is, the multi-view high-definition camera and the lighting device matrix are assembled between the two groups of square metal frames 101, the front and back of the shell 203 are connected and fixed with the two groups of metal frames, realizing the 360° ring load of the multi-view high-definition camera and the lighting device in the middle of the unmanned aerial vehicle, for tunnel section image collection in weak light or no light conditions, so as to ensure that each lens works under the same and relatively uniform light intensity, and improve the quality of collected data.

[0028] The laser radar device is arranged on the unmanned aerial vehicle, and in one embodiment, the laser radar device includes 1 laser radar 301 arranged at the tail end of the unmanned aerial vehicle, for synchronously collecting three-dimensional laser point cloud data of the tunnel when collecting the tunnel section image. Further, the laser radar device is fixed on the square metal frame 101 at the tail of the unmanned aerial vehicle through the pitch angle holder, and the fine adjustment of the laser radar sensing angle is realized through the motor control, realizing the accurate and stable reception of target echo information.

[0029] The positioning module comprises an inertial measurement unit (IMU) 401, a binocular positioning camera 402, and a barometric altimeter 403; the positioning module is arranged on the unmanned aerial vehicle; the positioning module acquires position data in real time through the inertial measurement unit 401 and the binocular positioning camera 402 asynchronously, and acquires air pressure data in real time through the barometric altimeter 403 to estimate the height; for the multiple heterogeneous data acquired, asynchronous fusion processing is performed through the built-in filtering algorithm, and the positioning cumulative error is reduced through multi-source calibration, so that high-precision real-time positioning information of the unmanned aerial vehicle inside the diversion tunnel is obtained. In an embodiment, the inertial measurement unit 401 and the binocular positioning camera 402 are fixed on the square metal frame 101 at the front end of the unmanned aerial vehicle through a support, and the fixed direction faces the flight direction; the fixed support of the inertial measurement unit 401 is a non-rotatable connecting frame; the fixed support of the binocular positioning camera 402 comprises a section of pan-tilt, which facilitates the adjustment of the camera view angle for positioning. The barometric altimeter 403 is fixed on the side rod of the front half square metal frame 101.

[0030] The inertial measurement unit (IMU) 401 integrates a three-axis accelerometer, a gyroscope, and a magnetometer to acquire real-time unmanned aerial vehicle attitude (pitch, roll, yaw) and angular velocity data; the binocular positioning camera 402 is installed at the front end of the unmanned aerial vehicle in the driving direction, and realizes visual range calculation through triangulation and feature point matching to obtain the relative position offset of the unmanned aerial vehicle; the barometric altimeter 403 calculates the height of the aircraft by measuring the change of atmospheric pressure. The positioning module fuses and calibrates the position information of the above-mentioned multiple sensors in real time using a filtering algorithm to obtain dynamic accurate position information of the unmanned aerial vehicle.

[0031] The data processing module is used to construct a tunnel high-definition real scene three-dimensional model, perform automatic detection of multi-modal defects, and perform three-dimensional quantitative labeling and display of defects in the tunnel high-definition real scene three-dimensional model according to high-definition image data collected by the multi-view high-definition camera, accurate position data obtained by the positioning module, and point cloud data collected by the laser radar device.

[0032] In an embodiment, constructing a tunnel high-definition real scene three-dimensional model comprises the following steps: Step 1: collecting tunnel data by using an unmanned aerial vehicle. When collecting tunnel data by using an unmanned aerial vehicle, a flight route in the tunnel is planned for data collection. Preferably, for a large-diameter tunnel, a spiral flight trajectory can be planned for the unmanned aerial vehicle to take pictures, so that the lens is further close to the inner wall during the shooting process, thereby improving the resolution and clarity of the shooting image.

[0033] Step 2: multi-source data fusion modeling. After the unmanned aerial vehicle completes the collection of tunnel data, the tunnel high-definition image and position data collected by the unmanned aerial vehicle and the laser point cloud data are used to construct a tunnel high-definition real scene three-dimensional model after calculation, preprocessing, and fusion, specifically including: Step 2.1: For high-definition image data, based on image position and attitude information, same-name point matching and aerial triangulation encryption are performed to generate an aerial triangulation dense point cloud; and image exterior orientation elements and three-dimensional coordinates of the encrypted points are obtained according to the image data collected by the multi-view high-definition camera; the laser point cloud data collected by the laser device is preprocessed to eliminate abnormal data; Step 2.2: The aerial triangulation encrypted point cloud of the high-definition image and the laser point cloud data are registered and aligned by spatial ICP to obtain a multi-source fusion dense point cloud. Step 2.3: The modeling software is used to establish a triangular network for the multi-source fusion dense point cloud; and based on the relationship between the image data and the aerial triangulation position coordinates, the triangular network constructed is texture mapped to obtain a high-definition real scene three-dimensional model of the tunnel.

[0034] There are typical defects such as cracks, seepage, erosion, bulges, and collapse in the diversion tunnel. According to the different recognition characteristics of different defects, a multi-modal recognition strategy is used for defect recognition. In one embodiment, the multi-modal defect automatic detection includes the following steps: Step 3.1: According to the image data, a defect image data set is made, and the defect classification of the defect image data set includes cracks, seepage, and erosion; an image semantic segmentation network is trained according to the defect image data set; and the trained image semantic segmentation network is used to detect the defects of the diversion tunnel high-definition image to be detected.

[0035] Specifically, for defects with low three-dimensional stereoscopic degree and strong texture contrast with the wall surface (such as cracks, seepage, and erosion), based on image texture features, computer vision algorithms such as semantic segmentation are used to classify defects in the image. Further, according to the pixel-level classification result, for each high-definition image, a binary mask image (Mask) with the same size as the image is made for each defect type, with the pixels of the defect being assigned a value of 1 and the pixels of the non-defect type being assigned a value of 0.

[0036] Step 3.2: According to the laser point cloud data, a defect point cloud data set is made, and the defect classification of the defect point cloud data set includes bulges and collapse; a point cloud semantic segmentation network is trained according to the defect point cloud data set; and the trained point cloud semantic segmentation network is used to detect the defects of the point cloud data of the diversion tunnel to be detected.

[0037] Specifically, for defects with significant three-dimensional features and weak texture contrast with normal wall surface in two-dimensional space (such as bulges and collapse), based on the three-dimensional geometric features of the laser point cloud, three-dimensional point cloud neural networks are used to extract point cloud features for point cloud classification to separate out the defect corresponding point cloud.

[0038] In one embodiment, after the defect detection is completed, the three-dimensional quantitative labeling and display of the defects include the following steps: Step 4.1: According to the defect classification result of the high-definition image, a binary mask is made for each defect type; according to the position coordinates of each point of the defect in the image in the binary mask, the corresponding position in the high-definition real scene three-dimensional model is inversely calculated and automatically labeled in the form of a patch, and different types of defects are labeled in different colors. The size information of the defects is counted according to the size of the label.

[0039] Specifically, for defects such as cracks, water seepage and denudation with low stereoscopic degree, according to the position coordinates of each pixel point of the defect in the image in the mask, the corresponding position in the three-dimensional model is inversely calculated through the image and the corresponding coordinate information in the aerial triangulation result, and automatic labeling is realized in the form of a patch in the three-dimensional model.

[0040] Step 4.2: Classify point clouds based on three-dimensional geometric features of point clouds to separate defect corresponding point clouds; generate three-dimensional equidistant line annotations of defects using classified defect point clouds; calculate geometric information of bulges and collapses for quantitative labeling; and display the three-dimensional equidistant line annotations of defects based on the spatial coordinates of point clouds and three-dimensional equidistant line annotations in the high-definition real scene three-dimensional model of the tunnel.

[0041] Specifically, for defects such as bulges and collapses with more obvious three-dimensional morphology, the three-dimensional equidistant line annotations of defects are generated by connecting the outermost circle of point clouds to form a reference line using the classified defect point clouds; and the geometric information of bulges and collapses is calculated for quantitative labeling, realizing three-dimensional detection and expression of defects. At the same time, based on the spatial coordinates of point clouds, the three-dimensional equidistant line annotations of defects can be directly fitted into the three-dimensional model of the tunnel for display.

[0042] After the detection device of the embodiment of the application is assembled, the method and principle of water diversion tunnel data acquisition are as follows: Before taking off, the tunnel flight path is planned, and the shape of the flight path should be selected according to the size of the tunnel radius. When the tunnel radius is small, the straight flight mode along the tunnel central axis is adopted; when the tunnel radius is large, in order to ensure clearer photographing effect, the spiral flight path around the tunnel central axis should be planned for close-range photogrammetry of the tunnel wall. In the embodiment of the application, the mathematical equation of the spiral flight trajectory is as follows:

[0043] In the formula, : spiral radius (related to the tunnel radius); : rotation angle (, , angular velocity); : axial flight speed; t: time; x, y, z: respectively represent the coordinate components in the x-axis, y-axis and z-axis directions of the space rectangular coordinate system.

[0044] When flying, the unmanned aerial vehicle and the tunnel wall should maintain a safe distance , spiral radius and tunnel radius The following relationship should be satisfied:

[0045] UAV axial flight speed It should not be too large, and the axial overlap of the image taken during flight should be no less than 60%.

[0046] When flying according to the set trajectory, the six illuminating lamps 202 remain in the lighted state to provide a stable light source; the laser radar 301 continuously and uninterruptedly scans the tunnel to collect point cloud data inside the tunnel; and the six high-definition lenses 201 continuously and simultaneously take pictures of the tunnel section to efficiently collect high-overlap tunnel section images.

[0047] During the flight of the UAV, the inertial measurement unit 401, the binocular positioning camera 402 and the barometric altimeter 403 in the positioning module collect real-time position information. Since the attitude of the inertial measurement unit 401 involves nonlinear updating, a nonlinear filtering algorithm should be used when fusing the positioning data; since the data acquisition frequency of the inertial measurement unit 401 is higher, while the data acquisition frequency of the binocular positioning camera 402 and the barometric altimeter 403 is lower, an asynchronous updating strategy should be used when fusing the data, and the positioning information is updated by the data of the inertial measurement unit 401 first, and then calibrated according to the positioning data of the binocular positioning camera 402 and the barometric altimeter 403.

[0048] Preferably, the extended Kalman filter algorithm (EKF) is used to perform real-time nonlinear asynchronous fusion and calibration of multiple positioning data to obtain the accurate position of the UAV inside the tunnel. Through the real-time accurate position, the UAV can fly according to the set flight path, accurately approach the tunnel wall to collect data, and the images and point clouds contain accurate spatial position and size information.

[0049] In view of the characteristics of weak signal in the tunnel environment and easy damage to the inner wall of the tunnel, the water diversion tunnel defect three-dimensional detection device based on the UAV of the embodiment of the application is provided with a positioning module including an inertial measurement unit 401, a binocular positioning camera 402 and a barometric altimeter 403 on the UAV, which provides high-precision positioning information for data collection in the environment where the tunnel GPS signal is missing, and solves the navigation problem in the tunnel environment; at the same time, the UAV device is equipped with a thin tube protective mesh cover 501, which can effectively protect the propeller 103 of the UAV in case of accident, and prevent the propeller 103 from causing damage to the inner wall of the tunnel.

[0050] The embodiment of the present application sets up a multi-view high-definition camera, a lighting device and a laser radar device on the unmanned aerial vehicle to realize efficient, high-resolution and uniform light multi-source data synchronous collection in the tunnel.

[0051] The embodiment of the present application realizes multi-source data fusion registration and constructs a tunnel high-definition real scene three-dimensional model through a data processing module based on high-definition image data collected by the multi-view high-definition camera and laser point cloud data collected by the laser radar device, improves the geometric precision and texture definition of the tunnel three-dimensional real scene model, and performs multi-modal defect automatic detection through the data processing module and three-dimensional quantitative labeling and display of defects in the tunnel high-definition real scene three-dimensional model, effectively improves the automation level and richness of the diversion tunnel defect detection, and provides accurate data support for tunnel health monitoring and maintenance decision-making.

[0052] Embodiment 2 The embodiment of the present application provides a method for detecting by using the diversion tunnel defect three-dimensional detection device based on the unmanned aerial vehicle of the above-mentioned embodiment 1, and the flow is as shown in Figure 3 The embodiment of the present application provides a method for detecting by using the diversion tunnel defect three-dimensional detection device based on the unmanned aerial vehicle of the above-mentioned embodiment 1, and the flow is as shown in Step 1: based on the detection device and operation mode described in the embodiment 1 of the present application, the unmanned aerial vehicle is used to collect full-coverage high-definition image data and laser point cloud data of the diversion tunnel; Step 2: the image data aerial triangulation encryption is used to construct point cloud and laser point cloud data registration, fuse high-definition image data and laser point cloud data (laser radar data), perform diversion tunnel multi-source data fusion modeling, and obtain a tunnel high-definition real scene three-dimensional model; specifically, the following steps are included: laser radar point cloud data preprocessing, image data aerial triangulation encryption point cloud construction, image encryption point cloud and laser point cloud space registration, triangular net construction and texture mapping. The step can be further described as: Step 2.1: the three-dimensional point cloud software is used to remove abnormal points and preprocess the laser radar point cloud.

[0053] Step 2.2: the internal orientation elements of the image are obtained by using the fixed sensor size and lens focal length of the high-definition camera, then the image aerial triangulation is performed, the image feature points are extracted, the feature points are matched, the feature points are connected, the image external orientation elements and the three-dimensional coordinates of the encryption points are calculated.

[0054] Step 2.3: Based on the ICP (Iterative Closest Point) algorithm, the image encrypted point cloud and the laser point cloud are spatially registered. In one feasible scheme, the registration between the point clouds is realized by setting the number of iterations, the corresponding point search radius and the convergence threshold in the point cloud processing software, and finally the fused point cloud meeting the threshold requirement is output.

[0055] Step 2.4: In one executable scheme, the Delaunay triangulation method can be used to establish a triangular net for the fused point cloud after registration.

[0056] Step 2.5: Based on the high-definition image data and the aerial triangulation position information, the constructed triangular net is texture-mapped.

[0057] In the embodiment, the above steps 2.1 and 2.3 require manual intervention for debugging; 2.2, 2.4 and 2.5 can be preferably directly processed by running the three-dimensional modeling software, which specifically includes: first, importing the image data into the three-dimensional modeling software (such as DJI ZhiTu, ContextCaptureCenter, Metashape, etc.) for aerial triangulation, then importing the laser point cloud file after the preprocessing of step 2.1 and the spatial registration of step 2.3, constructing a triangular net model for the fused point cloud, and finally automatically mapping the texture based on the image.

[0058] Step 3: For different recognition features of different typical defects in the diversion tunnel, a multi-modal recognition strategy is adopted to train a semantic segmentation algorithm for defect recognition and detection; specifically including the following steps: Step 3.1: For cracks, water seepage and denudation defects with low three-dimensional stereoscopic degree and strong texture features, train semantic segmentation parameters for feature recognition and classification. In this embodiment, the steps include: Step 3.1.1: Make a defect image dataset. Collect network public datasets, tunnel inspection images and divide them into uniform pixel sizes, vectorize, assign values and rasterize the crack, water seepage and denudation type defects in the images, make image pixel-by-pixel classification ground truth (GT), form a sample set of tunnel defects, and divide the training set and the validation set according to the ratio of 8:2. Preferably, in this embodiment, the image classification is set to 4 categories, and the value-type correspondence is: 0-background, 1-crack, 2-water seepage, 3-denudation.

[0059] Step 3.1.2: Train the image semantic segmentation network. Use the U-Net network as the image semantic segmentation backbone network, input the training set image and GT, and update the network parameters iteratively until the cross-entropy loss on the validation set converges, that is, the training is considered complete.

[0060] Step 3.1.3: High-definition image defect classification of water diversion tunnel. The high-definition image is batched and cut into the pixel size of the input layer of the semantic segmentation network, and the trained semantic segmentation network is used to classify the high-definition image. The classification results are mapped to the original image according to the original position, and the pixel-level defect classification of the high-definition image is completed.

[0061] Step 3.2: For the drum bump and collapse type defects with high three-dimensional stereoscopic degree and weak texture features, the semantic segmentation parameters are trained for feature recognition and classification. In this embodiment, the steps include: Step 3.2.1: Make a defect point cloud data set. Manually select point sets in different types of sample point clouds and assign corresponding type values to form a point cloud point-by-point classification label (Label). Preferably, in this embodiment, 3 types of point cloud classification are set, and the value-type correspondence is: 0-background, 1-bump, and 2-collapse.

[0062] Step 3.2.2: Train the point cloud semantic segmentation network. In this embodiment, RandLa-Net is used as the point cloud semantic segmentation backbone network, the input sample point cloud and the point-by-point corresponding label are iteratively trained until the loss function converges.

[0063] Step 3.2.3: Laser point cloud defect classification of water diversion tunnel. The collected and registered laser point cloud data of the water diversion tunnel is input into the trained segmentation network to complete point-by-point type prediction and extract the corresponding point cloud at the defect.

[0064] Step 4: After completing the multi-modal water diversion tunnel defect recognition, based on the multi-modal tunnel defect classification and recognition results, the three-dimensional quantitative labeling and display of defects in the tunnel high-definition real scene three-dimensional model are realized according to the spatial information, i.e. the three-dimensional automatic labeling, measurement and display of defects are realized, as shown in Figure 4 The specific steps include: Step 4.1: After pixel-level separation of the tunnel defects obtained by image classification, the classified color is mapped to the tunnel high-definition real scene three-dimensional model, and the defect size is counted. The steps include: Step 4.1.1: For each high-definition image, according to the pixel-level classification results, the pixels of the three defect types of cracks, water seepage and erosion are extracted and a defect mask (Mask) is made for each type. The Mask has the same size and pixel position as the original image; in the Mask corresponding to each defect type, the pixels at the defect are assigned a value of 1, and the pixels at the non-defect are assigned a value of 0.

[0065] Step 4.1.2: Labeling three types of defects respectively. When labeling one type of defect, replace the high-definition image used in texture mapping with all masks corresponding to this type of defect. At this time, the number, size, and figure name of the masks are consistent with the original high-definition image set. Through the space information, map the 1 value in the mask to the tunnel high-definition real scene three-dimensional model, and fill in the color as the face-shaped labeling of this type of defect, which can be superimposed on the tunnel high-definition real scene three-dimensional model for full display of the tunnel defects. In this embodiment, the labeling color-type correspondence is set as: red-crack, blue-water seepage, and yellow-erosion.

[0066] Step 4.1.3: Each patch in the labeling is regarded as a defect, and the area, length, and other data are calculated for each patch to obtain the corresponding size of each defect, which can be used for overall and local statistics of each type of defect.

[0067] Step 4.2: For the tunnel defects obtained by classifying the laser point cloud, generate three-dimensional linear labeling by extracting the point cloud at the defect, and calculate the volume and other sizes. The steps include: Step 4.2.1: First, based on the classification results of the laser point cloud, extract the bulge and collapse type points respectively and save them as point cloud files; further, in each type of point cloud file, set a spatial aggregation degree threshold, and regard the point cloud with high aggregation degree as the same defect, and divide it into a separate defect point cloud block.

[0068] Step 4.2.2: For each defect corresponding point cloud block, generate three-dimensional linear labeling. Connect the outermost points of the point cloud block to form a reference line, and the plane where the line is located is the reference plane; based on the reference plane, generate a line loop according to the vertical distance of the remaining points to the reference plane, forming a three-dimensional equidistant linear labeling of the defect; finally, through the three-dimensional coordinate information of the point cloud, align and superimpose the labeling to the tunnel high-definition real scene three-dimensional model for display. Preferably, the bulge type defect is marked as an orange equidistant line with a "+" mark beside the line; the collapse type defect is marked as a purple equidistant line with a "-" mark beside the line.

[0069] Step 4.2.3: Calculate and count the mathematical information of each defect. Specifically, calculate the occupied area of each defect according to the line connecting the outermost points of the point cloud block; calculate the maximum value of the bulge and collapse according to the distance of the points in the point cloud block to the reference plane (the plane where the line connecting the outermost points is located), and make quantitative labeling; generate a triangular mesh model according to the point cloud block to calculate the volume corresponding to the defect.

[0070] The detection method of the embodiment of the application is based on high-definition video and laser point cloud data, and on the one hand, realizes multi-source data fusion registration and modeling, and improves the geometric precision and texture definition of the tunnel three-dimensional real scene model; on the other hand, according to the texture and geometric feature saliency of different types of defects, combined with a multi-modal strategy, intelligent recognition is respectively performed based on two-dimensional and three-dimensional features of multi-source data, and finally, automatic three-dimensional labeling of defects on the tunnel three-dimensional real scene model is realized by using spatial coordinate mapping, so that the automation level and richness of the results of the diversion tunnel defect detection are effectively improved, and accurate data support is provided for the tunnel health monitoring and maintenance decision.

[0071] The diversion tunnel defect three-dimensional detection device and method based on the unmanned aerial vehicle can form a complete diversion tunnel defect three-dimensional detection process, realize automatic unmanned aerial vehicle inspection of the diversion tunnel defects, improve the efficiency and quality of the tunnel defect detection, and reduce the labor cost and safety risk.

[0072] The above embodiments only express several embodiments of the application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent of the application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which all belong to the protection scope of the application. Therefore, the protection scope of the patent of the application should be subject to the appended claims.

Claims

1. A three-dimensional detection device for defects in a pilot tunnel based on a drone, characterized by, The application relates to a tunnel inspection system, which comprises the following parts: a drone; a multi-view high-definition camera, wherein a plurality of high-definition lenses are arranged in a ring array on the drone; a lighting device, which comprises a plurality of lighting lamps, wherein the plurality of lighting lamps are arranged in a ring array on the drone, and the plurality of lighting lamps are arranged in a uniform staggered manner with the plurality of high-definition lenses; a laser radar device arranged on the drone; a positioning module, which comprises an inertial measurement unit, a binocular positioning camera and a barometric altimeter; the positioning module is arranged on the drone; the positioning module adopts a filtering algorithm to fuse and calibrate the measurement data of the inertial measurement unit, the binocular positioning camera and the barometric altimeter, so as to obtain accurate position data of the drone; and a data processing module, which is used for constructing a tunnel high-definition real scene three-dimensional model, performing multi-modal defect automatic detection, and performing three-dimensional quantitative labeling and display of defects in the tunnel high-definition real scene three-dimensional model according to high-definition image data collected by the multi-view high-definition camera, the position data and point cloud data collected by the laser radar device.

2. The unmanned aerial vehicle based three-dimensional detection device for defects in a pilot tunnel according to claim 1, characterized in that, The application further comprises a protection module, wherein the protection module comprises a plurality of protection net covers, and the plurality of protection net covers are respectively arranged on the propellers of the drone.

3. The unmanned aerial vehicle based three-dimensional detection device for defects in a pilot tunnel according to claim 1, characterized in that, The construction of the tunnel high-definition real scene three-dimensional model comprises the following steps: preprocessing the point cloud data to remove abnormal data; obtaining image exterior orientation elements and three-dimensional coordinates of encrypted points according to the image data; performing spatial registration on the image encrypted point cloud and the point cloud data to obtain fused point cloud; establishing a triangular net for the fused point cloud; performing texture mapping on the triangular net according to the image data and the spatial relationship of the aerial triangulation.

4. The unmanned aerial vehicle based three-dimensional detection device for defects in a pilot tunnel according to claim 1, characterized in that, The multi-modal defect automatic detection comprises the following steps: according to the image data, preparing a defect image data set, wherein the defect classification of the defect image data set comprises cracks, water seepage and erosion; training an image semantic segmentation network according to the defect image data set; using the trained image semantic segmentation network to perform defect classification detection on the high-definition image of the diversion tunnel to be detected.

5. The UAV-based 3D inspection apparatus for a pilot tunnel defect according to claim 4, wherein, The multi-modal defect automatic detection comprises the following steps: according to the point cloud data, preparing a defect point cloud data set, wherein the defect classification of the defect point cloud data set comprises bulges and collapse; training a point cloud semantic segmentation network according to the defect point cloud data set; using the trained point cloud semantic segmentation network to perform defect classification detection on the point cloud data of the diversion tunnel to be detected.

6. The UAV-based 3D inspection apparatus for a pilot tunnel defect according to claim 5, wherein, The three-dimensional quantitative labeling and display of defects comprises the following steps: according to the defect classification result of the high-definition image, preparing a binary mask image for each defect type; according to the position coordinates of each point of the defect in the binary mask image, the corresponding position in the tunnel high-definition real scene three-dimensional model is inversely calculated and mapped, and the defect is automatically labeled in the form of a surface patch; different types of defects are labeled in different colors, and the size information of the defects is calculated according to the labeling size. Point cloud classification is performed based on point cloud three-dimensional geometric features to separate defect corresponding point clouds; three-dimensional equidistant line annotations of defects are generated using the classified defect point clouds; geometric information of bulges and collapses is calculated for quantitative annotation; and the spatial coordinates of the point clouds and the three-dimensional equidistant line annotations are displayed in the high-definition real scene three-dimensional model of the tunnel.

7. The unmanned aerial vehicle based three-dimensional detection device for defects in a pilot tunnel according to any one of claims 1 to 6, characterized in that, The body of the unmanned aerial vehicle comprises two groups of square metal frames and a shell connected between the two groups of square metal frames; each corner of the two groups of square metal frames is connected with a landing support; and the plurality of lighting lamps and the plurality of high-definition lenses are arranged on the shell in a uniform staggered manner.

8. The UAV-based 3D inspection apparatus for a pilot tunnel defect according to claim 7, wherein, The laser radar device is fixed on the square metal frame through a pitch angle holder.

9. The UAV-based 3D inspection apparatus for a pilot tunnel defect according to claim 7, wherein, The binocular positioning camera is fixed on the square metal frame through a pitch angle holder.

10. A method for detecting defects in a pilot tunnel using the unmanned aerial vehicle-based three-dimensional detection device according to any one of claims 1-9, characterized in that, The method comprises the following steps: The diversion tunnel is fully covered with high-definition image data and laser point cloud data collected by the unmanned aerial vehicle; The image data is triangulated in the air to encrypt the point cloud, and the point cloud data is registered to fuse the image data and the point cloud data, thereby performing multi-source data fusion modeling of the diversion tunnel to obtain a high-definition real scene three-dimensional model of the tunnel; A multi-modal recognition strategy is adopted to train a semantic segmentation algorithm for defect recognition according to different recognition features of different typical defects in the diversion tunnel; Based on the multi-modal tunnel defect classification results, three-dimensional quantitative annotation and display of defects are realized in the high-definition real scene three-dimensional model of the tunnel according to spatial information.

Images