Dam piping checking method, device, system and equipment

Abstract

The invention discloses a dam piping checking method, device, system and equipment, and relates to the field of dam piping detection. The method comprises the following steps: determining a first piping suspected point from a thermal infrared image; determining a second piping suspected point in the visible light image; determining the same piping suspected point in the first piping suspected point and the second piping suspected point as a first piping point; reconstructing a three-dimensional terrain digital elevation model according to the laser point cloud data; determining the material type of an object in the target dam investigation area according to the laser radar reflection signal; matching objects in the first piping suspected point, the three-dimensional terrain digital elevation model and the target dam troubleshooting area; according to the matched three-dimensional terrain digital elevation model and the material type of the object, misjudged piping suspected points in the first piping suspected points are deleted, and second piping points are obtained; and determining the first piping point and the second piping point as the piping points of the target dam investigation area. According to the invention, the accuracy of dam piping point troubleshooting can be greatly improved.

Description

Technical Field

[0001] The present application relates to the field of levee piping detection, and particularly to a method, device, system and equipment for levee piping investigation. Background Art

[0002] Piping refers to a concentrated water leakage phenomenon in which river water penetrates through a pervious layer under the action of the water level pressure difference between the upstream and downstream sides, and breaks through the relatively impervious surface layer near the toe of the downstream levee (dam) and in pits, depressions, paddy fields, etc. outside the toe of the levee (dam), and takes away or flushes out the fine particles in the pervious layer from the pores of the coarse particles. It usually occurs in sandy gravel geology. Generally, the degree of danger of the piping danger is judged according to the distance of the piping orifice from the toe of the levee, the diameter of the piping orifice, the water discharge, the turbidity of the gushing water, the water head of the gushing water, the expansion of the orifice, etc. The closer to the toe of the levee, the larger the piping orifice, the more the water discharge, the more sediment carried, and the faster the water flow, the greater the harm. Piping may be single, multiple, or multiple in clusters. Piping is a common danger in levee projects. Under the action of continuous high water levels, if not disposed of in time, a large amount of gushing water and sand turning will damage and scouring out the soil skeleton of the levee foundation, resulting in major dangers such as levee collapse and breach.

[0003] At present, the main means of piping investigation is still the traditional manual investigation, usually adopting the "dragnet" levee inspection method. Although it has intuitiveness and flexibility, it also has obvious limitations such as low efficiency, strong subjectivity, high risk of personal safety in the investigation, and difficulty in detecting hidden piping hazards. With the development of unmanned aerial vehicles, there has emerged a method of using a single sensor (thermal infrared camera or visible light camera) or a combination of thermal infrared cameras and visible light cameras for piping investigation, but it basically remains in the laboratory stage. Its multi-source data fusion is mostly limited to static analysis, lacking the ability to capture dynamic features and real-time processing. It is severely interfered by the environment in actual piping investigation applications, and the misjudgment rate is relatively high, resulting in a still very heavy workload for manual review.

[0004] Therefore, how to efficiently and accurately investigate levee piping is an urgent problem to be solved at present. Summary of the Invention

[0005] The purpose of the present application is to provide a method, device, system and equipment for levee piping investigation, which can greatly improve the accuracy of levee piping point investigation.

[0006] To achieve the above purpose, the present application provides the following solutions.

[0007] In a first aspect, the present application provides a method for detecting and investigating piping in a dike, including: obtaining multiple thermal infrared images, multiple visible light images, and all lidar point cloud data of a target dike investigation area collected simultaneously by a thermal infrared camera, a visible light camera, and a lidar; extracting multi-layer temperature field areas with gradually increasing temperatures from the center to the edge in each thermal infrared image, and determining them as first suspected piping points; determining second suspected piping points in each visible light image; determining the same suspected piping points among all the first suspected piping points and all the second suspected piping points as the first piping points; reconstructing a three-dimensional terrain digital elevation model of the target dike investigation area according to all the lidar point cloud data; extracting material features based on the reflection signals of objects in the target dike investigation area received by the lidar, and determining the material types of the objects in the target dike investigation area according to the extracted material features; registering and fusing the thermal infrared images, visible light images, and lidar point cloud data to obtain thermal infrared images, visible light images, and lidar point cloud data aligned to a unified coordinate reference; matching the first suspected piping points, the three-dimensional terrain digital elevation model, and the objects in the target dike investigation area according to the thermal infrared images, visible light images, and lidar point cloud data under the unified coordinate reference; deleting misjudged first suspected piping points based on the matched three-dimensional terrain digital elevation model and the material types of the objects in the target dike investigation area to obtain second piping points; and determining the first piping points and the second piping points together as the piping points of the target dike investigation area.

[0008] In a second aspect, the present application provides a device for detecting and investigating piping in a dike, including: a drone, a thermal infrared camera, a visible light camera, a lidar, a multi-sensor synchronous controller based on FPGA (Field-Programmable Gate Array), and a host computer. The drone is equipped with a thermal infrared camera, a visible light camera, a lidar, and a multi-sensor synchronous controller based on FPGA; the drone is used to fly above the target dike investigation area according to a preset inspection route; during the flight of the drone, the multi-sensor synchronous controller based on FPGA controls the thermal infrared camera, the visible light camera, and the lidar to collect multiple thermal infrared images, multiple visible light images, and all lidar point cloud data of the target dike investigation area simultaneously; the multi-sensor synchronous controller based on FPGA ensures millisecond-level time synchronization of the thermal infrared camera, the visible light camera, and the lidar through a hardware trigger signal, reducing the registration error; the host computer is used to determine the piping points by using the above-mentioned method for detecting and investigating piping in a dike according to the multiple thermal infrared images, multiple visible light images, and all lidar point cloud data.

[0009] In a third aspect, the present application provides a levee piping inspection system, including: a collection module, a first piping suspected point determination module, a second piping suspected point determination module, a screening module, a reconstruction module, a material feature extraction module, a fusion module, a matching module, a deletion module, and a determination module.

[0010] The collection module is used to obtain multiple thermal infrared images, multiple visible light images, and all lidar point cloud data of the target levee inspection area collected by a thermal infrared camera, a visible light camera, and a lidar at the same time. The first piping suspected point determination module is used to extract multi-layer temperature field areas with gradually increasing temperatures from the center to the edge in each thermal infrared image and determine them as the first piping suspected points. The second piping suspected point determination module is used to determine the second piping suspected points in each visible light image. The screening module is used to determine the same piping suspected points among all the first piping suspected points and all the second piping suspected points as the first piping points. The reconstruction module is used to reconstruct a three-dimensional terrain digital elevation model of the target levee inspection area according to all the lidar point cloud data. The material feature extraction module is used to extract material features according to the reflection signals of the objects in the target levee inspection area received by the lidar and determine the material types of the objects in the target levee inspection area according to the extracted material features. The fusion module is used to register and fuse the thermal infrared images, visible light images, and lidar point cloud data to obtain thermal infrared images, visible light images, and lidar point cloud data aligned to a unified coordinate reference. The matching module is used to match the first piping suspected points, the three-dimensional terrain digital elevation model, and the objects in the target levee inspection area according to the thermal infrared images, visible light images, and lidar point cloud data under the unified coordinate reference. The deletion module is used to delete the misjudged piping suspected points in the first piping suspected points according to the matched three-dimensional terrain digital elevation model and the material types of the objects in the target levee inspection area to obtain the second piping points. The determination module is used to jointly determine the first piping points and the second piping points as the piping points of the target levee inspection area.

[0011] In a fourth aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the above-mentioned levee piping inspection method.

[0012] According to the specific embodiments provided by the present application, the present application has the following technical effects.

[0013] The present application provides a method, device, system and equipment for detecting piping in dikes. Local temperature differences caused by seepage on the dike surface or by piping flow in the nearby ground (or pond) can be captured from the thermal infrared image, thereby helping to identify potential risk points. The visible light image can display the clear surface of the target dike inspection area, and can exclude some misjudged piping suspected points generated by the thermal infrared image, improving the accuracy of hidden danger identification to a certain extent. Since piping usually occurs near the backwater side dike (dam) foot and in pits, depressions, paddy fields, etc. outside the dike (dam) foot, and the center of the piping point usually has a lower terrain with relatively significant terrain features, the three-dimensional terrain digital elevation model can be used to eliminate some piping suspected points in the thermal infrared recognition results. The lidar can also invert the material of the object, and the material of the object can be used to quickly eliminate a large number of piping suspected points in the thermal infrared recognition results. Based on the multi-source heterogeneous data composed of thermal infrared images, visible light images and lidar point cloud data, the present application greatly improves the accuracy of detecting piping points in dikes. Description of the Drawings

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0015] Figure 1 It is a schematic flow chart of a method for detecting piping in dikes provided by an embodiment of the present application.

[0016] Figure 2 It is a schematic brief flow chart of a method for detecting piping in dikes provided by an embodiment of the present application.

[0017] Figure 3 It is a schematic flow chart of determining the first piping suspected point provided by another embodiment of the present application.

[0018] Figure 4 It is a schematic flow chart of determining the second piping suspected point provided by another embodiment of the present application.

[0019] Figure 5 It is a schematic flow chart of reconstructing the three-dimensional terrain digital elevation model provided by another embodiment of the present application.

[0020] Figure 6 It is a schematic flow chart of determining the material type provided by another embodiment of the present application.

[0021] Figure 7 It is a schematic diagram of the multi-source heterogeneous data fusion process provided by another embodiment of the present application.

[0022] Figure 8 Schematic diagram of the structure of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0023] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0024] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0025] Infrared thermal imaging technology is a method of judging temperature differences by measuring the infrared radiation of a target object. Since all objects with a temperature higher than absolute zero are constantly emitting infrared radiation energy into the surrounding space, the magnitude of the infrared radiation energy of an object and the distribution of its wavelength are closely related to its surface temperature. Therefore, by measuring the infrared energy radiated by an object itself, its surface temperature can be accurately measured. By mounting a thermal infrared camera on a drone and scanning the object to be measured, an infrared thermal image of its surface can be obtained. Different colors in the infrared thermal image represent different temperatures. By comparing the color differences in different regions, the temperature differences on the surface of the object can be judged.

[0026] In the detection of dam leakage or piping hazards, the thermal infrared camera can capture the local temperature differences on the dam surface due to leakage or on the nearby ground (or pond) due to piping water flow, thereby helping to identify potential risk points. At the same time, the high-resolution visible light camera mounted on the drone can provide clear surface images of the target dam inspection area, including relevant water flow and environmental data, providing intuitive visual data support for hazard inspection. It can not only help identify leaks, cracks, collapses, and other visible hazards on the dam surface, as well as piping points with obvious turbidity and tumbling characteristics in the target dam inspection area, but also exclude some misjudgment risk points generated by the thermal infrared camera, improving the accuracy of hazard identification to a certain extent.

[0027] However, in practical engineering applications, climate change and surface vegetation cover can cause significant interference to the monitoring results of thermal infrared cameras. For example, changes in temperature may affect the temperature distribution on the dam surface, while vegetation occlusion may affect the line of sight of thermal infrared cameras. At the same time, the temperature difference between the vegetation itself and the surrounding environment will have a greater impact on the judgment results of thermal infrared data. Especially at night, visible light cameras cannot obtain relevant images, and relying solely on thermal infrared cameras will result in more risk misjudgments, increasing the workload of manual re-inspection during flood control or emergency management in the flood season.

[0028] To solve the above problems, survey-grade lidar is introduced into the investigation of dam hidden dangers. Currently, lidar has been widely used in many fields such as three-dimensional topographic mapping, construction, agriculture, forestry, and unmanned driving due to its advantages of high precision, high efficiency, and all-weather operation. First of all, as a new type of active remote sensing technology, lidar has the ability to penetrate dense vegetation, which makes it play an important role in many fields. Lidar obtains information about target objects by emitting laser pulses and receiving their echoes. These laser pulses can penetrate vegetation, reach the ground or objects below the vegetation and reflect back. By analyzing the reflected laser pulses, lidar can identify vegetation and obtain parameters such as the height, density, and biomass of vegetation as well as the topographic information of the ground.

[0029] Secondly, by scanning the target dam investigation area and continuously emitting and receiving laser pulses, lidar can obtain a large amount of laser point cloud data, which contains the three-dimensional coordinate information of the target object. By processing and fusing the collected laser point cloud data, such as data filtering, denoising, segmentation, and classification, a three-dimensional point cloud terrain model can be generated to visually display features such as the undulation and slope of the terrain. Since piping usually occurs near the backwater side dam (embankment) foot and in pits, depressions, paddy fields, etc. outside the dam (embankment) foot, and the center of the piping point usually has a lower terrain, with relatively significant topographic features, some misjudged points in the thermal infrared recognition results can be eliminated.

[0030] At the same time, lidar can also invert the material of the measured target object. Using this characteristic, a large number of misjudged points in the thermal infrared recognition results can be quickly eliminated.

[0031] In addition, the positioning accuracy of survey-grade lidar can reach 3 cm, effectively reducing the difficulty of on-site manual verification.

[0032] Based on the above-introduced thermal infrared camera, visible light camera, and lidar, in an exemplary embodiment, as Figure 1 shown, the present application provides a method for investigating dam piping, including the following steps 101 to step 110.

[0033] Step 101: Obtain multiple thermal infrared images, multiple visible light images, and all lidar point cloud data of the target dam inspection area collected simultaneously by the thermal infrared camera, visible light camera, and lidar.

[0034] Step 102: Extract multiple temperature field regions with gradually increasing temperature from the center to the edge in each thermal infrared image, and determine them as the first suspected piping points.

[0035] Step 103: Determine the second suspected piping points in each visible light image.

[0036] Step 104: Determine the same suspected piping points among all the first suspected piping points and all the second suspected piping points as the first piping points.

[0037] Step 105: Reconstruct the three-dimensional terrain digital elevation model of the target dam inspection area based on all the lidar point cloud data.

[0038] Step 106: Extract the material characteristics according to the reflection signals of the objects in the target dam inspection area received by the lidar, and determine the material types of the objects in the target dam inspection area based on the extracted material characteristics.

[0039] Step 107: Register and fuse the thermal infrared images, visible light images, and lidar point cloud data to obtain the thermal infrared images, visible light images, and lidar point cloud data aligned to the unified coordinate reference.

[0040] Step 108: Match the first suspected piping points, the three-dimensional terrain digital elevation model, and the objects in the target dam inspection area according to the thermal infrared images, visible light images, and lidar point cloud data under the unified coordinate reference.

[0041] Step 109: Delete the misjudged suspected piping points in the first suspected piping points according to the matched three-dimensional terrain digital elevation model and the material types of the objects in the target dam inspection area to obtain the second piping points.

[0042] Step 110: Determine both the first piping points and the second piping points as the piping points in the target dam inspection area.

[0043] Figure 2 Illustrate the brief process of a dam piping inspection method of the present application.

[0044] In another exemplary embodiment of the present application, generally, the location where piping occurs will show low-temperature anomaly characteristics during the day, and the piping influence area will show a temperature gradient change texture feature centered on the piping point in the thermal image, and at the same time has a trailing contour feature along the water flow direction. That is, after the formation of the piping area, a relatively obvious temperature diffusion effect appears in the piping area, and gradually presents multiple temperature field regions with gradually increasing temperature from the center to the edge. Therefore, as Figure 2As shown, the first suspected piping point ① can be identified from the thermal infrared image according to the temperature anomaly information. Figure 3 As shown, the above step 102 can be replaced by the following steps 201 to 206.

[0045] Step 201: Preprocess the thermal infrared image to obtain the preprocessed thermal infrared image; the preprocessing includes noise removal, temperature calibration, data correction and enhancement.

[0046] (1) Noise removal: The original thermal infrared image data collected may contain various noises, such as equipment noise, environmental noise, etc. These noises will reduce the data quality and affect subsequent analysis. Therefore, it is necessary to remove these noises through filtering algorithms or image processing techniques. Common filtering methods include median filtering, Gaussian filtering, etc. These filtering methods can effectively improve the signal-to-noise ratio of the data and provide a clearer data basis for subsequent analysis.

[0047] (2) Temperature calibration: Due to the influence of equipment or environmental factors, the temperature values in the original data may be deviated. Therefore, it is necessary to perform temperature calibration on the data to ensure the accuracy of the data. Temperature calibration can be achieved through methods such as blackbody radiation correction. According to the equipment calibration parameters and environmental conditions, the collected data is corrected to eliminate systematic errors and environmental interference. Among them, the equipment calibration parameters mainly include temperature deviation correction (correcting the possible deviation of the equipment when measuring temperature to ensure the accuracy of the measurement result), gain and offset correction (adjusting the gain and offset parameters of the equipment to optimize the response characteristics and measurement accuracy of the equipment), detector stability correction (correcting the possible stability problems of the detector due to long-term use or environmental changes to ensure the long-term reliability of the equipment), and optical system correction (correcting the aberrations, distortions, etc. of the optical system to improve the imaging quality and measurement accuracy of the equipment). The environmental conditions mainly include environmental temperature, environmental humidity and atmospheric attenuation.

[0048] (3) Data correction and enhancement: In addition to temperature calibration, other types of data correction may also be required to further eliminate errors and improve data quality. In addition, data enhancement techniques such as image enhancement and contrast adjustment can be used to improve the visualization effect of the data.

[0049] The preprocessed thermal infrared image is the Figure 3 result of the thermal infrared image preprocessing in

[0050] Step 202: Extract the temperature values of all pixels in the preprocessed thermal infrared image, obtain adjacent pixel points with a temperature difference greater than the preset temperature threshold, and use the set of pixel points with lower temperature among the obtained adjacent pixel points as the center.

[0051] Exemplarily, the preset temperature threshold is 2°C to 3°C. Among adjacent pixel points, the pixel points with lower temperature are abnormally low. There are multiple pixel points with abnormally low temperature, and the set of pixel points with abnormally low temperature forms an area, taking this area as the center.

[0052] Step 203: According to the preprocessed thermal infrared image, use one of the spatial autocorrelation method, co-occurrence matrix method, and Tamura method to extract the gradient texture feature with gradually increasing temperature from the center to the edge.

[0053] Step 204: According to the preprocessed thermal infrared image, use an edge detection method to extract the trailing contour edge feature along the water flow direction; the edge detection method is one of the first-order differential edge detection operator, multi-scale edge detection method, and fuzzy enhancement edge detection method.

[0054] Step 205: According to the preprocessed thermal infrared image, use a region detection algorithm to detect the temperature anomaly region.

[0055] Step 206: Determine the temperature anomaly region with both gradient texture feature and trailing contour edge feature as the first suspected piping point.

[0056] The method for extracting thermal infrared image features adopts a combination of texture, edge feature extraction methods based on traditional image processing and region detection feature extraction algorithms. On the one hand, texture features reflect the spatial variation of image brightness, and common extraction methods include the spatial autocorrelation method, co-occurrence matrix method, Tamura method, etc.; while edge features refer to the set of pixels where the image gray level undergoes a spatial mutation or a mutation in the gradient direction, and common extraction methods include the first-order differential edge detection operator (such as Roberts operator, Sobel operator, Canny operator, etc.), multi-scale edge detection method, fuzzy enhancement edge detection method, etc. On the other hand, the piping features can be extracted through a region detection algorithm, using algorithms such as Salient Region, Edge-Based Region (EBR), Intensity-Based Region (IBR), and Maximum Stable Extremal Regions (MSER) to detect the significant regions (temperature anomaly regions) or stable regions in the thermal infrared image. These regions usually contain important information or targets such as dam leakage and piping. Combining the above two methods, the suspected piping points are extracted.

[0057] In another exemplary embodiment of the present application, for dam cracks and leakage, their characteristics are manifested as cracks, wet spots, color changes, etc.; for piping, its characteristics are manifested as the water surface showing obvious turbidity and continuous churning state. Such asFigure 4 As shown, the above step 103 can be replaced by the following steps 301 to 306.

[0058] Step 301: Denoise, enhance, and correct each visible light image in sequence to obtain each preprocessed visible light image; denoising includes median filtering and Gaussian filtering; enhancement includes histogram equalization, contrast enhancement, and sharpening; correction includes geometric correction and lens distortion correction.

[0059] (1) Image denoising: Due to the complex and variable shooting environment, the image may contain various noises, such as Gaussian noise, salt-and-pepper noise, etc. These noises will reduce the image quality and affect subsequent analysis. Therefore, image denoising processing is required. The denoising methods adopted include: ① Median filtering, that is, removing salt-and-pepper noise in the image by replacing the pixel value with the median of its neighboring pixel values. ② Gaussian filtering, that is, by performing weighted average processing on the image and replacing the original pixel value with the weighted average gray value of the pixels in the neighborhood to remove Gaussian noise, etc.

[0060] (2) Image enhancement: Image enhancement aims to improve the visual effect of the image and make it easier to analyze and recognize. The image enhancement methods adopted include: ① Histogram equalization, that is, by adjusting the gray histogram of the image to make its distribution more uniform, thereby enhancing the contrast of the image. ② Contrast enhancement, that is, by stretching the gray level range or applying nonlinear transformation and other methods to adjust the contrast of the image to make it clearer. ③ Sharpening, that is, by applying methods such as the Laplace operator or gradient operator to enhance the edges and details of the image to make it clearer.

[0061] (3) Image correction: Image correction aims to eliminate geometric distortion and aberration in the image and ensure the correct geometric shape of the image. The image correction methods adopted include: ① Geometric correction, that is, using a geometric transformation matrix to perform operations such as rotation, scaling, and translation on the image to eliminate geometric distortion in the image. ② Lens distortion correction, that is, using a lens distortion correction algorithm to eliminate barrel distortion or pincushion distortion, etc. caused by the lens.

[0062] (4) Image registration and stitching: During the UAV inspection process, multiple images need to be taken to cover the entire target dam inspection area. Through image registration and stitching, multiple images can be combined into one image. First, match the same feature points in multiple images to determine their relative position relationship, that is, image registration. Then seamlessly stitch the registered images to form a complete image.

[0063] The preprocessed visible light image is the Figure 4 result of the visible light image preprocessing in

[0064] Step 302: Register and splice all visible light images to form a complete visible light image of the target dike inspection area.

[0065] Step 303: Convert the complete visible light image from the RGB (Red, Green, Blue) color space to the HSV (Hue, Saturation, Value) color space.

[0066] Step 304: Extract the turbidity texture information using a Gabor filter based on the complete visible light image in the HSV color space, and determine the area with turbidity texture information.

[0067] Step 305: Extract the surging texture features using local binary pattern based on the complete visible light image in the HSV color space, and determine the area with surging texture features.

[0068] Step 306: Determine the areas with both turbidity texture information and surging texture features as the second suspected piping points.

[0069] Under different lighting conditions, some features reflecting the essential attributes of objects in the image, such as shape, texture, and reflection characteristics, can remain unchanged or relatively stable. Extracting illumination-invariant features is of great significance for improving the accuracy and robustness of tasks such as image recognition, classification, and retrieval. A method combining color space conversion and texture feature extraction is adopted for the dangerous situation features such as dike leakage and piping. Since the brightness component of the HSV color space is not affected by illumination and is very suitable for extracting illumination-invariant features, the image is converted from the RGB color space to the HSV color space to weaken the influence of illumination. At the same time, as the minute details of objects in the image, texture can provide information about the surface characteristics of objects and also has a certain degree of illumination invariance. The adopted texture feature extraction methods include Gabor filters and local binary pattern (LBP). Among them, Gabor filters can extract the texture information of objects and have a certain degree of illumination invariance to a certain extent; while LBP extracts texture features by comparing the relationship between a pixel point and its neighboring pixel points.

[0070] In another exemplary embodiment of the present application, the above step 103 may also be: Based on each visible light image, use a deep learning-based object detection algorithm to identify the second suspected piping points.

[0071] Principle of object detection algorithms based on deep learning: If it is not only necessary to determine whether there is a piping in the image, but also to locate the position of the piping, then object detection algorithms such as YOLO (You Only Look Once) and SSD (Single Shot MultiBoxDetector) are very useful. They predict the class probability and position coordinates of the object in a single forward pass in an end-to-end manner, and can quickly frame the suspected piping area in the image. Advantages: Fast speed and relatively high accuracy. For monitoring scenarios with certain requirements for real-time performance, it can quickly give results; moreover, it can give the specific position of the object, which is convenient for subsequent precise marking and analysis, meeting the needs of dike inspection.

[0072] Taking the YOLO algorithm as an example, based on the temperature anomaly and thermal infrared imaging characteristics of the piping occurrence point, as well as the turbidity and tumbling characteristics in the visible light image, a deep learning algorithm is used to train a piping recognition model. Based on the powerful capabilities of convolutional neural networks in feature extraction and classification, the YOLO algorithm is used to achieve object detection and recognition of thermal infrared and visible light images.

[0073] YOLO (You Only Look Once) is a real-time object detection algorithm. Its core idea is to transform the object detection task into a regression problem and simultaneously predict the class and position of the object through a single neural network. The YOLO algorithm is an excellent object detection algorithm with advantages such as fast speed, high accuracy, good real-time performance, and strong generalization ability. With the iteration of versions and the continuous optimization of the network structure, the detection accuracy and speed of the YOLO algorithm are also constantly improving.

[0074] The YOLO algorithm divides the input image into a grid of a fixed size and predicts the position and class of the object in each grid cell. Specifically, it divides the input image into S×S grid cells, and each grid cell is responsible for detecting the object within that grid. S represents the number of grids in one dimension of the input image. In each grid cell, the algorithm predicts the position (x, y, w, h) of the bounding box containing the object and the class probability of the object. Among them, x and y represent the offsets of the center of the bounding box relative to the upper left corner of the grid cell, and w and h represent the ratios of the bounding box relative to the width of the entire image.

[0075] Exemplarily, the YOLOv5 object detection model can be used to identify the second suspected piping point.

[0076] When extracting the surging texture features in step 305, a recurrent neural network, or a long short-term memory network (LSTM), or a gated recurrent unit (GRU) can also be used. Considering that the "surging" of piping is a feature that changes dynamically over time, visible light data in video form can better reflect this characteristic. Recurrent neural networks are good at processing sequential data and can capture the temporal correlation between image frames. Their improved versions, LSTM and GRU, can better handle the problem of gradient disappearance in long sequences and accurately learn the dynamic evolution pattern of piping over time. The advantage of this method is its strong ability to capture dynamic features. Combining with continuously captured visible light image data, it can more accurately determine whether it is a real piping and reduce misjudgments caused by instantaneous light and shadow changes and water flow fluctuations.

[0077] In another exemplary embodiment of the present application, referring to Figure 2 , in the above step 104, the second suspected piping points that do not belong to the first suspected piping points are determined as non-piping points and excluded. The same suspected piping points among all the first suspected piping points and all the second suspected piping points are determined as the first piping points, that is Figure 2 ② in ①, ② are the second suspected piping points, and ① is the first suspected piping point.

[0078] In another exemplary embodiment of the present application, as shown in Figure 2 , by performing point cloud data processing on the laser point cloud position information, the three-dimensional terrain information DEM (Digital Elevation Model) can be obtained. The process of reconstructing the three-dimensional terrain digital elevation model in the above step 105 is as shown in Figure 5 . First, perform quality inspections on the collected original point cloud data, such as point cloud accuracy and point cloud density. After passing the inspection, perform GPS (Global Positioning System) dynamic post-processing kinematic (PPK) solution, reconstruction, spatial coordinate transformation, resampling, denoising, and smoothing processing on it. Then, perform classification processing on the point cloud data to separate the ground points from the non-ground points, so as to extract the original terrain surface points and make the DEM.

[0079] 1) Point cloud calculation: The original data is positioned through the calculation of the Differential Global Positioning System (DGPS), and then the combined calculation results of the Inertial Measurement Unit (IMU) / Global Navigation Satellite System (GNSS) data and system parameters are imported to achieve the geodetic orientation of the point cloud. After calibration, it is converted to the required coordinate system, and then the information, coordinates, and attributes of the points are arranged and output in an editable point cloud format according to certain rules.

[0080] 2) Point cloud resampling: To improve the data reading and processing speed, for point cloud data with a large amount of data, resampling can be used to thin out the point cloud. By controlling the sampling interval or sampling rate of the data, the point cloud data can retain fewer points, thereby improving the running speed of the point cloud data during reading and processing.

[0081] 3) Point cloud denoising: Noise points will appear in the data collected by lidar. These noise points have nothing to do with the actual ground object information but will affect the accuracy and quality of the data. Common noises include high-level gross errors and low-level gross errors, which can also be simply understood as noise points suspended at high altitudes and extremely low points. To prevent these noise points from affecting the data classification effect and accuracy, noise points are usually removed by selecting appropriate parameters before classification to improve the data quality.

[0082] 4) Point cloud smoothing: Point cloud smoothing is to fit the optimal fitting plane equation according to adjacent points, obtain the elevation of the center point, and adjust the original value. This operation can make the appearance of the point cloud more consistent. For the surface area, the surface thickness can be thinned, thereby improving the accuracy of the DEM.

[0083] In another exemplary embodiment of the present application, lidar can invert the material of the measured target object. When a laser pulse irradiates the surface of an object, reflection and scattering will occur. This reflection and scattering effect depends on parameters such as the material, shape, surface roughness of the irradiated object, and the wavelength and power of the emitted laser.

[0084] After the lidar receives the signals reflected and scattered back by the target object, it performs processing such as filtering, denoising, and enhancement on them, and can extract feature information related to the material of the object, such as reflectivity, intensity, directionality, and polarization of the scattered light (see Figure 2 for obtaining the signal-to-noise ratio of the laser point cloud based on the reflected signal), and then compares the extracted feature information with a known material database or uses machine learning algorithms for classification and recognition to determine the material type of the target object (see Figure 2 for inverting the material information from the signal-to-noise ratio of the laser point cloud to obtain the material information).

[0085] The reflected signal is one of the main signals received by lidar. Objects of different materials have different reflection characteristics for laser light, and these characteristics mainly include reflectivity, directionality, and polarization of the reflected light, which are specifically manifested as follows: (1) Objects of different materials have different reflectivities. Reflectivity refers to the ratio of the laser energy reflected from the object surface to the incident laser energy. For example, the reflectivity of a metal surface is usually high, the reflectivities of non-metal materials such as plastics and wood are low, and the reflectivity of water surface is relatively even lower.

[0086] (2) The roughness of the object surface and the shape of the object itself determine the directionality of the reflected light. For a smooth surface, the reflected light is mainly reflected back along the direction of the incident light, while for a rough surface, the reflected light will scatter in all directions.

[0087] (3) Since the laser beam usually has a specific polarization state, when the laser beam irradiates objects with different materials and different surface characteristics, the polarization state of the reflected light will change differently. Based on the change of the polarization state, the material type of the target object can be identified and determined.

[0088] In addition, lidar can also receive scattered signals, including diffuse reflection and directional scattering. The laser beam will undergo random scattering on a rough surface or an irregular object, which is called diffuse reflection and usually has a low intensity and a wide direction distribution; while the directional scattering of the laser beam on the object surface is related to the material characteristics and microstructure of the object. Based on the characteristics of these scattered signals, it can help identify and determine the material type of the target object.

[0089] As Figure 6 shown, the above step 106 can be replaced by the following steps 401 to 402.

[0090] Step 401: After filtering, denoising, and enhancing the reflected signals of the objects in the target dam inspection area received by lidar in sequence, extract the material characteristics from them; the material characteristics include reflectivity, intensity, directionality, and polarization of the scattered light.

[0091] Step 402: Compare the extracted material characteristics with the material database to determine the material type of the objects in the target dam inspection area.

[0092] In summary, in the inspection of dam leakage or piping, the lidar carried by an unmanned aerial vehicle can quickly scan a large area, obtain high-precision three-dimensional coordinates, terrain, and material information in real time, effectively eliminate misjudgment points in the recognition results of thermal infrared cameras and visible light cameras, greatly improve the operation efficiency, and reduce the workload of manual on-site review. At the same time, lidar is not affected by weather and lighting conditions and can operate at night and in various complex environments.

[0093] In another exemplary embodiment of the present application, the above step 107 can be replaced by the following steps 501 to 507.

[0094] Step 501: Extract the same feature points of the preset target in the thermal infrared image, visible light image, and lidar point cloud data, and name them homologous points.

[0095] Step 502: Use the feature matching algorithm to perform feature matching on the homologous points of the thermal infrared image, visible light image, and lidar point cloud data to obtain the homologous points after feature matching.

[0096] Step 503: Perform geometric correction on the thermal infrared image, visible light image, and lidar point cloud data.

[0097] Step 504: Determine the imaging geometric matrix between the geometrically corrected thermal infrared image, visible light image, and lidar point cloud data according to the homologous points after feature matching.

[0098] Step 505: Use the imaging geometric matrix to align the geometrically corrected thermal infrared image, visible light image, and lidar point cloud data to the same coordinate system.

[0099] Step 506: Fuse the aligned thermal infrared image, visible light image, and lidar point cloud data to obtain the initially fused thermal infrared image, visible light image, and lidar point cloud data.

[0100] Step 507: Convert the initially fused thermal infrared image, visible light image, and lidar point cloud data to a unified coordinate reference to obtain the thermal infrared image, visible light image, and lidar point cloud data aligned to the unified coordinate reference; the unified coordinate reference is the position reference provided by differential GPS high-precision positioning.

[0101] The above steps 501 to 507 achieve multi-source heterogeneous data fusion. As Figure 7 shown, the more detailed process of multi-source heterogeneous data fusion is as follows.

[0102] 1) Feature extraction: For thermal infrared, visible light, and lidar data, different feature extraction algorithms can be used to extract the feature information of the target. In the visible light image, algorithms such as edge detection and corner detection can be used to extract the shape and contour features of the target; in the thermal infrared image, the temperature difference can be used to extract the thermal features of the target; in the lidar data, the point cloud data can be used to extract the geometric features of the target. Through feature extraction, several homologous points of the target dam inspection area are obtained.

[0103] 2) Feature matching: After the feature information (homologous points) of the target is extracted, a feature matching algorithm can be used to match the target features obtained by different sensors. Common feature matching algorithms include distance-based matching, shape-based matching, and feature-based matching, etc. Through feature matching, the target information obtained by different sensors can be associated and fused to obtain more accurate target information.

[0104] 3) Imaging geometric matrix transformation: In multi-source heterogeneous data fusion, imaging geometric matrix transformation is an important technology, which is used to process the geometric relationship between the image data obtained by different sensors and realize the precise registration and fusion of images. Imaging geometric matrix transformation is based on matrix operations in mathematics, and describes the geometric relationship between images by constructing a transformation matrix. The transformation matrix includes translation matrix, rotation matrix, scaling matrix, and more complex affine transformation matrix or projection transformation matrix. Through matrix operations, the points in one image can be mapped to the corresponding positions in another image, thus realizing image registration and fusion. Imaging geometric matrix transformation includes (1) image geometric correction, (2) image registration, and (3) image fusion.

[0105] (1) Image geometric correction: In multi-source heterogeneous data fusion, due to the influence of internal factors, azimuth changes, different media, etc. when different sensors acquire images, the acquired images have geometric distortions. Image geometric correction is the process of performing geometric transformation on the image to eliminate geometric distortions, so that the images of different sensors can be precisely registered and fused.

[0106] (2) Image registration: In multi-source heterogeneous data fusion, the images obtained by different sensors often have geometric differences, such as rotation, scaling, translation, etc. Through imaging geometric matrix transformation, the transformation matrix between these images can be calculated, and one image can be transformed to the same coordinate system as another image to achieve precise image registration. This is crucial for subsequent image fusion and analysis, because only on the basis of registration can the image information of different sensors be effectively fused.

[0107] (3) Image fusion: In the process of image fusion, imaging geometric matrix transformation can be used to align the image data obtained by different sensors to the same coordinate system. Through weighted averaging, maximum value selection or more complex fusion algorithms on the aligned images, the fused image can be obtained. The fused image combines the advantages of different sensors, provides richer information, and helps with subsequent image analysis and processing.

[0108] 4) Differential GPS High-Precision Positioning: The differential GPS technology uses a differential GPS reference station with known precise three-dimensional coordinates to obtain pseudorange correction amounts or position correction amounts, and then sends this correction amount to the user (GPS navigator) in real time or afterwards to correct the user's measurement data, so as to improve the GPS positioning accuracy. In the fusion of multi-source heterogeneous data, the data obtained by various sensors may have different coordinate systems and accuracies. Differential GPS high-precision positioning provides a unified and high-precision position reference, enabling the data obtained by different sensors to be aligned and fused in the same coordinate system, ensuring the consistency and accuracy of the fused data, and providing a reliable basis for subsequent data analysis and processing.

[0109] At the same time, the differential GPS technology can significantly reduce the errors in traditional GPS positioning, such as the influence of satellite orbit errors, clock errors, ionospheric and tropospheric delays, etc. When differential GPS is fused with other sensor data, these corrected high-precision position information can significantly improve the accuracy and reliability of the fusion results.

[0110] 5) Coordinate Projection Matching: Through coordinate projection matching, the thermal infrared image, visible light image, and lidar three-dimensional point cloud data are converted to a unified coordinate system and projection method, that is, the position reference provided by differential GPS high-precision positioning (usually the plane is in the CGCS2000 coordinate system, and the elevation is the 1985 National Elevation Datum), so as to realize the data fusion of complex scenes and provide basic support for data analysis and applications. The multi-source heterogeneous data after coordinate projection matching can more accurately reflect the actual spatial relationship and information.

[0111] In another exemplary embodiment of the present application, in step 109 above, according to the matched three-dimensional terrain digital elevation model and the material type of the object in the target dam inspection area, the misjudged pipe burst suspected points in the first pipe burst suspected points are deleted. Specifically, if the elevation of the three-dimensional terrain digital elevation model corresponding to the first pipe burst suspected point is greater than the preset elevation threshold, it is determined that the first pipe burst suspected point is a misjudged pipe burst suspected point and is deleted. If the material type of the object in the target dam inspection area corresponding to the first pipe burst suspected point is not water, it is determined that the first pipe burst suspected point is a misjudged pipe burst suspected point and is deleted.

[0112] As Figure 2 shown, through the multi-source information fusion discrimination of the three-dimensional terrain information DEM, material information, and the first pipe burst suspected point ①, the second pipe burst point is obtained.

[0113] In another exemplary embodiment of the present application, when extracting features from thermal infrared images and visible light images, a Convolutional Neural Network (CNN) or a combination of a convolutional neural network and LSTM can also be used. A convolutional neural network is good at processing image data. It automatically extracts features in images through convolutional layers, such as visual features like the unique texture and shape of the piping area. Different convolutional kernels can capture multi-level features from low-level edges and lines to high-level object contours and complex patterns. Then, through the pooling layer for dimensionality reduction to reduce the amount of computation, and the fully connected layer for classification decision-making. Advantage: It performs excellently in image classification tasks. There are a large number of mature network architectures (such as AlexNet, VGG (Visual Geometry Group), ResNet (Residual Network), etc.) that can be directly used for transfer learning. By using the general features learned from a large amount of natural image data by a pre-trained model and then fine-tuning the parameters to adapt to the piping image data, a good recognition accuracy can be quickly achieved, saving training time and data volume.

[0114] Compared with the method of only using thermal infrared and visible light to detect piping, the present application can control the misjudgment rate at about 10% in actual engineering applications, and the misjudgment points are reduced by an average of 70%-80%.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a dam piping detection device for implementing the dam piping detection method involved above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the dam piping detection device provided below can refer to the limitations on the dam piping detection method in the above text, and will not be repeated here.

[0116] In an exemplary embodiment, a dam piping detection device is provided, including: a drone, a thermal infrared camera, a visible light camera, a lidar, an FPGA-based multi-sensor synchronization controller, and a host computer. The drone is equipped with a thermal infrared camera, a visible light camera, and a lidar. The drone is used to fly above the target dam inspection area according to a preset inspection route. During the flight of the drone, the FPGA-based multi-sensor synchronization controller controls the thermal infrared camera, the visible light camera, and the lidar to simultaneously collect multiple thermal infrared images, multiple visible light images, and all lidar point cloud data of the target dam inspection area. The host computer is used to determine the piping points according to the multiple thermal infrared images, multiple visible light images, and multiple lidar point cloud data by using the above dam piping detection method.

[0117] The FPGA-based multi-sensor synchronization controller ensures the millisecond-level time synchronization of the thermal infrared camera, the visible light camera, and the lidar through a hardware trigger signal, reducing the registration error.

[0118] In an exemplary embodiment, the present application further provides a levee piping detection system, including: a collection module, a first piping suspected point determination module, a second piping suspected point determination module, a screening module, a reconstruction module, a material feature extraction module, a fusion module, a matching module, a deletion module, and a determination module.

[0119] The collection module is used to obtain multiple thermal infrared images, multiple visible light images, and multiple lidar point cloud data of the target levee inspection area collected by a thermal infrared camera, a visible light camera, and a lidar at the same time. The first piping suspected point determination module is used to extract multi-layer temperature field areas with gradually increasing temperature from the center to the edge in each thermal infrared image and determine them as the first piping suspected points. The second piping suspected point determination module is used to determine the second piping suspected points in each visible light image. The screening module is used to determine the same piping suspected points among all the first piping suspected points and all the second piping suspected points as the first piping points. The reconstruction module is used to reconstruct a three-dimensional terrain digital elevation model based on multiple lidar point cloud data. The material feature extraction module is used to extract material features according to the reflection signals of objects in the target levee inspection area received by the lidar, and determine the material types of the objects in the target levee inspection area according to the extracted material features. The fusion module is used to register and fuse the thermal infrared images, visible light images, and lidar point cloud data to obtain thermal infrared images, visible light images, and lidar point cloud data aligned to a unified coordinate reference. The matching module is used to match the first piping suspected points, the three-dimensional terrain digital elevation model, and the objects in the target levee inspection area according to the thermal infrared images, visible light images, and lidar point cloud data under the unified coordinate reference. The deletion module is used to delete the misjudged piping suspected points in the first piping suspected points according to the matched three-dimensional terrain digital elevation model and the material types of the objects in the target levee inspection area to obtain the second piping points. The determination module is used to jointly determine the first piping points and the second piping points as the piping points of the target levee inspection area.

[0120] In an exemplary embodiment, a computer device is provided. The computer device can be a server or a terminal, and its internal structure diagram can be as Figure 8As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the piping points in the target dam inspection area. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method for inspecting dam piping.

[0121] Those skilled in the art can understand that Figure 8 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0122] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0123] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties.

[0125] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0126] The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logics, data processing logics based on quantum computing, etc., without limitation.

[0127] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0128] In this article, specific examples are used to elaborate on the principles and implementation manners of the present application. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for checking dam piping, characterized in that: include: Acquire multiple thermal infrared images, multiple visible light images and all laser point cloud data of the target dam inspection area collected simultaneously by thermal infrared cameras, visible light cameras and lidar; Extract the multi-layer temperature field area with gradually higher temperature from the center to the edge of each thermal infrared image and determine it as the first suspected piping point; Determine the suspected point of the second piping in each visible light image; Determine the same suspected piping point among all the first suspected piping points and all the second suspected piping points as the first piping point; Reconstructing a three-dimensional terrain digital elevation model of the target dam inspection area based on all the laser point cloud data; Extract material features based on the reflected signals of objects in the target dam inspection area received by the laser radar, and determine the material type of the objects in the target dam inspection area based on the extracted material features; Register and fuse thermal infrared images, visible light images and laser point cloud data to obtain thermal infrared images, visible light images and laser point cloud data aligned to a unified coordinate reference; Match the first suspected piping point, the three-dimensional terrain digital elevation model and the objects in the target dam inspection area based on the thermal infrared image, visible light image and laser point cloud data under the unified coordinate reference; According to the matched three-dimensional terrain digital elevation model and the material type of the object in the target dam inspection area, the misjudged piping suspected points in the first piping suspected points are deleted to obtain the second piping points; The first piping point and the second piping point are determined together as the piping points in the target dam inspection area.

2. The method for checking dam piping according to claim 1, characterized in that: Extract the multi-layer temperature field area with gradually higher temperature from the center to the edge of each thermal infrared image and determine it as the first suspected piping point, including: Preprocessing the thermal infrared image to obtain a preprocessed thermal infrared image; the preprocessing includes noise removal, temperature calibration, data correction and enhancement; Extract the temperature values ​​of all pixels in the preprocessed thermal infrared image, obtain adjacent pixels whose temperature difference is greater than a preset temperature threshold, and take the set of pixels with lower temperature among the obtained adjacent pixels as the center; According to the preprocessed thermal infrared image, a spatial autocorrelation method, a co-occurrence matrix method and a Tamura method are used to extract a gradient texture feature with gradually increasing temperature from the center to the edge; According to the preprocessed thermal infrared image, an edge detection method is used to extract edge features with a trailing contour along the water flow direction; the edge detection method is one of a first-order differential edge detection operator, a multi-scale edge detection method, and a fuzzy enhancement edge detection method; According to the preprocessed thermal infrared image, a region detection algorithm is used to detect the temperature abnormality area; the region detection algorithm includes a significant region algorithm, an edge-based region detection algorithm, an intensity-based region detection algorithm and a maximum stable extreme value region algorithm; The temperature anomaly area having both the gradient texture feature and the trailing contour edge feature is determined as the first piping suspected point.

3. The method for checking dam piping according to claim 1, characterized in that: Determine the suspected point of the second piping in each visible light image, specifically including: Denoising, enhancing and correcting each visible light image in turn to obtain each preprocessed visible light image; the denoising includes median filtering and Gaussian filtering; the enhancement includes histogram equalization, contrast enhancement and sharpening; the correction includes geometric correction and lens distortion correction; All visible light images are registered and stitched together to form a complete visible light image of the target dam inspection area; Converting the complete visible light image from the RGB color space to the HSV color space; According to the complete visible light image in HSV color space, Gabor filter is used to extract turbid texture information and determine the area with turbid texture information; According to the complete visible light image in HSV color space, the local binary pattern is used to extract the surging texture features and determine the area with surging texture features; The area with both turbid texture information and surging texture characteristics is determined as the second pipe burst suspected point.

4. The method for checking dam piping according to claim 1, characterized in that: Determine the suspected point of the second piping in each visible light image, specifically including: According to each visible light image, a deep learning-based target detection algorithm is used to identify the suspected point of the second piping.

5. The method for checking dam piping according to claim 1, characterized in that: According to the reflected signal of the object in the target dam inspection area received by the laser radar, the material features are extracted, and the material type of the object in the target dam inspection area is determined according to the extracted material features, including: After filtering, denoising and enhancing the reflected signals of objects in the target dam inspection area received by the laser radar, material features are extracted from them; the material features include reflectivity, intensity, directionality and polarization of scattered light; The extracted material features are compared with the material database to determine the material type of objects in the target dam inspection area.

6. The method for checking dam piping according to claim 1, characterized in that: The thermal infrared image, visible light image and laser point cloud data are registered and fused to obtain the thermal infrared image, visible light image and laser point cloud data aligned to the same coordinate reference, including: Extract the same feature points of the preset targets in the thermal infrared image, visible light image and laser point cloud data and name them as the same-name points; Use feature matching algorithm to perform feature matching on the same-name points in thermal infrared image, visible light image and laser point cloud data to obtain the same-name points after feature matching; Perform geometric correction on thermal infrared images, visible light images and laser point cloud data; According to the points of the same name after feature matching, the imaging geometric matrix between the thermal infrared image, the visible light image and the laser point cloud data after geometric correction is determined; Using the imaging geometry matrix, the geometrically corrected thermal infrared image, visible light image and laser point cloud data are aligned to the same coordinate system; The aligned thermal infrared image, visible light image and laser point cloud data are fused to obtain the initially fused thermal infrared image, visible light image and laser point cloud data; The initially fused thermal infrared image, visible light image and laser point cloud data are converted to a unified coordinate reference to obtain thermal infrared image, visible light image and laser point cloud data aligned to the unified coordinate reference; the unified coordinate reference is a position reference provided by differential GPS high-precision positioning.

7. The method for checking dam piping according to claim 1, characterized in that: According to the matched three-dimensional terrain digital elevation model and the material type of the objects in the target dam inspection area, the misjudged piping suspected points in the first piping suspected points are deleted, including: If the elevation of the three-dimensional terrain digital elevation model corresponding to the first suspected piping point is greater than a preset elevation threshold, the first suspected piping point is determined to be a misjudged suspected piping point and is deleted; If the material type of the object in the target dam inspection area corresponding to the first suspected piping point is not water, the first suspected piping point is determined to be a misjudged suspected piping point and is deleted.

8. A device for checking dam piping, characterized in that: include: UAV, thermal infrared camera, visible light camera, lidar, FPGA-based multi-sensor synchronization controller and host computer; The drone is equipped with a thermal infrared camera, a visible light camera, and a lidar; The drone is used to fly over the target dam inspection area according to a preset inspection route; During the flight of the UAV, the FPGA-based multi-sensor synchronous controller controls the thermal infrared camera, the visible light camera and the laser radar to simultaneously collect multiple thermal infrared images, multiple visible light images and all laser point cloud data of the target dam inspection area; The host computer is used to determine piping points according to a plurality of thermal infrared images, a plurality of visible light images and all laser point cloud data by using the dam piping investigation method according to any one of claims 1 to 7.

9. A dam piping inspection system, characterized in that: The dam piping inspection system comprises: An acquisition module is used to obtain multiple thermal infrared images, multiple visible light images and all laser point cloud data of the target dam inspection area collected simultaneously by a thermal infrared camera, a visible light camera and a laser radar; The first piping suspected point determination module is used to extract the multi-layer temperature field area with gradually higher temperature from the center to the edge of each thermal infrared image, and determine it as the first piping suspected point; A second piping suspected point determination module, used to determine the second piping suspected point in each visible light image; A screening module, used for determining the same suspected piping point among all the first suspected piping points and all the second suspected piping points as the first piping point; A reconstruction module, used to reconstruct a three-dimensional terrain digital elevation model based on all the laser point cloud data; A material feature extraction module is used to extract material features based on the reflection signal of the object in the target dam inspection area received by the laser radar, and determine the material type of the object in the target dam inspection area based on the extracted material features; A fusion module is used to register and fuse thermal infrared images, visible light images and laser point cloud data to obtain thermal infrared images, visible light images and laser point cloud data aligned to a unified coordinate reference; A matching module is used to match the first suspected piping point, the three-dimensional terrain digital elevation model and the objects in the target dam inspection area according to the thermal infrared image, visible light image and laser point cloud data under a unified coordinate reference; A deletion module is used to delete the misjudged piping suspected points in the first piping suspected points according to the matched three-dimensional terrain digital elevation model and the material type of the objects in the target dam inspection area, and obtain the second piping point; The determination module is used to determine the first piping point and the second piping point together as piping points in the target dam inspection area.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for detecting dam piping according to any one of claims 1 to 7.

Images