Unmanned aerial vehicle facility crack identification method and system

Abstract

The invention discloses an unmanned aerial vehicle facility crack identification method and system, and relates to the technical field of unmanned aerial vehicle identification, and the method comprises the steps: obtaining the image data and point cloud data of a target region collected by an unmanned aerial vehicle; respectively preprocessing the image data and the point cloud data; according to the preprocessed image data, utilizing an improved Mask R-CNN model to carry out crack identification, after the crack is identified, calculating the length and the width of the crack, and according to the preprocessed point cloud data, calculating the depth of the crack; according to the length, the width and the depth of the crack, the damage degree of the crack is graded, and the damage degree grade of the crack is determined; and formulating a corresponding crack repairing scheme according to the damage degree grade of the crack. According to the method, the length, the width and the depth of the crack can be identified more accurately, and the damage degree grade of the crack can be accurately divided.

Description

Technical Field

[0001] The present application relates to the field of drone identification technology, and in particular to a method and system for identifying cracks in drone facilities. Background Art

[0002] With the continuous development of society, the maintenance demand for various infrastructures is increasing, including maintenance work in key areas such as roads, tunnels, buildings and solar panels. Among them, roads, tunnels, buildings and solar panels are inevitably prone to damage such as cracks, and the identification and repair of cracks are particularly important.

[0003] Currently, drone technology can be used to remotely collect the surface conditions of these facilities and obtain the size of cracks through image recognition technology. However, the existing drone methods for identifying and measuring cracks generally have low accuracy, and the accuracy of crack identification is not enough, so it is impossible to accurately classify the degree of damage of cracks. Summary of the invention

[0004] The purpose of this application is to provide a method and system for identifying cracks in UAV facilities, which can more accurately identify the length, width and depth of the cracks and accurately classify the degree of hazard of the cracks.

[0005] To achieve the above objectives, this application provides the following solutions.

[0006] In a first aspect, the present application provides a method for identifying cracks in UAV facilities, and the method for identifying cracks in UAV facilities includes the following steps.

[0007] Acquire image data and point cloud data of the target area collected by the drone.

[0008] The image data and the point cloud data are preprocessed respectively to obtain preprocessed image data and preprocessed point cloud data.

[0009] According to the preprocessed image data, crack recognition is performed using an improved Mask R-CNN model, and after the crack is recognized, the length and width of the crack are calculated, and according to the preprocessed point cloud data, the depth of the crack is calculated.

[0010] The degree of damage of the crack is graded according to the length, width and depth of the crack to determine the degree of damage of the crack.

[0011] According to the degree of hazard of the cracks, a corresponding crack repair plan is formulated.

[0012] Optionally, the drone is equipped with a visual sensor, a lidar, a GPS and an IMU, the visual sensor is used to collect image data of the target area, the lidar is used to collect point cloud data of the target area, and the GPS and IMU are used to record the flight altitude, time, position and posture of the drone.

[0013] Acquiring image data and point cloud data of a target area collected by a drone specifically includes the following steps.

[0014] The image data, the point cloud data, and the flight altitude, time, position, and posture of the unmanned aerial vehicle when collecting the image data and the point cloud data are obtained.

[0015] Optionally, the image data and the point cloud data are preprocessed respectively to obtain preprocessed image data and preprocessed point cloud data, which specifically includes the following steps.

[0016] The crack area in the image data is marked using VIA, and a json file is exported as a label of the image data.

[0017] De-noising and outlier elimination are performed on the point cloud data to obtain the point cloud data after de-noising and outlier elimination.

[0018] The surface point cloud and non-surface point cloud contained in the point cloud data after denoising and outlier removal are segmented to extract the surface point cloud and obtain segmented point cloud data.

[0019] According to the flight altitude, time, position and posture of the UAV, the image data and the segmented point cloud data are aligned to obtain preprocessed image data and preprocessed point cloud data.

[0020] Optionally, before the step of identifying cracks using an improved Mask R-CNN model based on the preprocessed image data, and calculating the length and width of the cracks after identifying the cracks, and calculating the depth of the cracks based on the preprocessed point cloud data, the UAV facility crack identification method also includes the following steps.

[0021] Based on the image data and the corresponding labels, a training set, a test set and a validation set are divided according to a preset ratio.

[0022] The improved Mask R-CNN model is trained using the training set, and the improved Mask R-CNN model is tested and verified using the test set and the validation set to obtain an improved Mask R-CNN optimized model; the improved Mask R-CNN optimized model is used as the improved Mask R-CNN model to identify cracks and calculate the length and width of the cracks.

[0023] Optionally, based on the preprocessed image data, an improved Mask R-CNN model is used to identify cracks, and after identifying the cracks, the length and width of the cracks are calculated, and based on the preprocessed point cloud data, the depth of the cracks is calculated, which specifically includes the following steps.

[0024] The preprocessed image data is input into the improved Mask R-CNN model to obtain crack recognition results.

[0025] When the crack identification result is that there is a crack, the length, width and depth of the crack are calculated respectively, including the following steps.

[0026] According to the preprocessed image data, the crack mask is acquired using the improved Mask R-CNN model.

[0027] Based on the crack mask, a crack skeleton is extracted.

[0028] The number of all pixels in the pixel point set of the crack skeleton is calculated to obtain the length of the crack.

[0029] The least square method is used to fit the crack skeleton with a quadratic function, and the length of the intersection of the normal line of each point of the fitting function and the crack mask is calculated as the width of the crack.

[0030] According to the preprocessed point cloud data, a RANSAC algorithm or a least squares method is used to perform plane fitting on the surrounding surface point clouds in the preprocessed point cloud data to obtain a fitting plane.

[0031] Based on the fitting plane, the vertical distance from each point in the crack area to the fitting plane is calculated.

[0032] The depth of the crack is determined according to the vertical distance from each point in the crack area to the fitting plane.

[0033] Optionally, determining the depth of the crack according to the vertical distance from each point in the crack area to the fitting plane specifically includes the following steps.

[0034] According to the vertical distances from each point in the crack region to the fitting plane, the maximum value and the average value of all vertical distance values ​​are counted to obtain statistical results.

[0035] According to the statistical results, the depth of the crack as a whole is evaluated to determine the depth of the crack.

[0036] Optionally, determining the depth of the crack according to the vertical distance from each point in the crack area to the fitting plane specifically includes the following steps.

[0037] A depth map is generated according to the vertical distance from each point in the crack area to the fitting plane.

[0038] Based on the depth map, the depth of the crack as a whole is evaluated to determine the depth of the crack.

[0039] Optionally, the hazard level of the cracks includes a mild crack hazard level, a moderate crack hazard level and a severe crack hazard level.

[0040] The crack width corresponding to the mild crack hazard level is less than 3 mm, the length is less than or equal to 30 cm, and the depth is less than 2 to 3 cm.

[0041] The cracks corresponding to the moderate crack hazard level are 3 to 5 mm in width, 30 to 100 cm in length, and more than half of the depth of the surface layer.

[0042] The severe crack hazard level corresponds to cracks with a width greater than 5 mm, a length greater than 1 m, and a depth that penetrates the surface and even reaches the base layer or core structure.

[0043] The crack repair scheme includes a crack repair scheme corresponding to the mild crack hazard level, a crack repair scheme corresponding to the moderate crack hazard level, and a crack repair scheme corresponding to the severe crack hazard level.

[0044] The crack repair solution corresponding to the mild crack hazard level is to repair the cracks by sealing or grouting, regularly observe the cracks, and evaluate whether the cracks are expanding.

[0045] The crack repair solution corresponding to the moderate crack hazard level is to use grooving and milling methods to repair the cracks, supplemented by local reinforcement or thin layer covering.

[0046] The crack repair solution corresponding to the severe crack hazard level is to perform local cutting, overall milling and renovation or local reconstruction of the cracks, and determine whether to replace or redo the foundation layer based on the size of the cracks.

[0047] Optionally, after the step of formulating a corresponding crack repair plan according to the hazard level of the crack, the UAV facility crack identification method further includes the following steps.

[0048] A report is generated based on the image data, the point cloud data, the hazard level of the cracks and the crack repair plan, wherein the content of the report includes the location, length, width, depth and hazard level of the cracks and the corresponding crack repair plan, and the report is stored in a standard format.

[0049] In a second aspect, the present application provides a UAV facility crack identification system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the UAV facility crack identification method described in the first aspect.

[0050] According to the specific embodiments provided in this application, this application has the following technical effects.

[0051] The present application provides a method and system for identifying cracks in UAV facilities. By acquiring image data and point cloud data collected by UAVs and preprocessing them, based on the preprocessed image data, the improved Mask R-CNN model can be used to identify whether cracks exist in the target area, and calculate the length and width of the cracks. At the same time, the depth of the cracks can be further calculated based on the preprocessed point cloud data. Therefore, the present application can quantitatively analyze more comprehensive size information of the cracks at the same time, taking into account the length, width and depth of the cracks, so that the degree of damage of the cracks can be more comprehensively, accurately and objectively graded, the degree of damage of the cracks can be accurately determined, and corresponding crack repair plans can be formulated, which is conducive to effective and targeted repair of cracks. It is suitable for crack identification and repair of infrastructure such as roads, tunnels, buildings and solar panels, and ensures the stability of infrastructure structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flowchart of a method for identifying cracks in UAV facilities provided in one embodiment of the present application.

[0054] Figure 2 A flowchart of the specific implementation process of a method for identifying cracks in UAV facilities provided in one embodiment of the present application.

[0055] Figure 3 A flowchart of crack identification using an improved Mask R-CNN model provided in one embodiment of the present application.

[0056] Figure 4 A flowchart of using an improved Mask R-CNN model to extract crack length and width and calculate depth using point cloud data is provided in one embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0059] like Figure 1 As shown, this embodiment proposes a method for identifying cracks in drone facilities, which mainly uses drone technology to identify cracks on the surface of infrastructure such as roads, tunnels, buildings, and solar panels, and accurately classifies the degree of damage of the cracks, and formulates corresponding crack repair plans. The method for identifying cracks in drone facilities specifically includes the following steps.

[0060] Step S1: Acquire image data and point cloud data of a target area collected by a drone.

[0061] In this embodiment, the drone is equipped with a visual sensor, a lidar, a GPS (Global Positioning System) and an IMU (Inertial Measurement Unit). The visual sensor is used to collect image data of the target area, the lidar is used to collect point cloud data of the target area, and the GPS and IMU are used to record the flight altitude, time, position and posture of the drone.

[0062] In this embodiment, step S1 acquires image data and point cloud data of the target area collected by the drone, which specifically includes the following steps.

[0063] The image data, the point cloud data, and the flight altitude, time, position, and posture of the unmanned aerial vehicle when collecting the image data and the point cloud data are obtained.

[0064] Step S2: preprocess the image data and the point cloud data respectively to obtain preprocessed image data and preprocessed point cloud data.

[0065] In this embodiment, step S2 preprocesses the image data and the point cloud data respectively to obtain preprocessed image data and preprocessed point cloud data, which specifically includes the following steps.

[0066] Step S21: using VIA (VGG Image Annotator) to mark the crack area in the image data, and exporting a json file as a label of the image data.

[0067] Step S22: denoising and removing outliers from the point cloud data to obtain denoised and outlier-removed point cloud data.

[0068] Step S23, segmenting the surface point cloud and non-surface point cloud contained in the point cloud data after denoising and removing outliers, so as to extract the surface point cloud and obtain segmented point cloud data.

[0069] Step S24: align the image data and the segmented point cloud data according to the flight altitude, time, position and posture of the UAV to obtain preprocessed image data and preprocessed point cloud data.

[0070] Step S3: According to the preprocessed image data, an improved Mask R-CNN (Mask Region-based Convolutional Neural Network, a deep learning network based on object detection and instance segmentation) model is used to identify cracks, and after identifying the cracks, the length and width of the cracks are calculated, and the depth of the cracks is calculated according to the preprocessed point cloud data.

[0071] In this embodiment, before using the improved Mask R-CNN model, the improved Mask R-CNN model needs to be trained. Therefore, before step S3, the UAV facility crack identification method may further include the following steps.

[0072] Based on the image data and the corresponding labels, a training set, a test set and a validation set are divided according to a preset ratio.

[0073] The improved Mask R-CNN model is trained using the training set, and the improved Mask R-CNN model is tested and verified using the test set and the validation set to obtain an improved Mask R-CNN optimized model; the improved Mask R-CNN optimized model is used as the improved Mask R-CNN model to identify cracks and calculate the length and width of the cracks.

[0074] In this embodiment, step S3 uses an improved Mask R-CNN model to identify cracks based on the preprocessed image data, and after identifying the cracks, calculates the length and width of the cracks, and calculates the depth of the cracks based on the preprocessed point cloud data, specifically including the following steps.

[0075] Step S31: input the preprocessed image data into the improved Mask R-CNN model to obtain a crack recognition result, wherein the crack recognition result includes whether there is a crack or not.

[0076] Step S32, when the crack identification result is that there is a crack, the length, width and depth of the crack are calculated respectively, including the following steps.

[0077] Step S321: Acquire a crack mask using the improved Mask R-CNN model according to the preprocessed image data.

[0078] Step S322: extract the crack skeleton according to the crack mask.

[0079] Step S323, calculating the number of all pixels in the pixel point set of the crack skeleton to obtain the length of the crack.

[0080] Step S324: Use the least squares method to fit a quadratic function to the crack skeleton, and calculate the length of the intersection of the normal line of each point of the fitting function and the crack mask as the width of the crack.

[0081] Step S325 : According to the preprocessed point cloud data, a RANSAC (Random Sample Consensus) algorithm or a least squares method is used to perform plane fitting on the surrounding surface point clouds in the preprocessed point cloud data to obtain a fitting plane.

[0082] Step S326: Based on the fitting plane, calculate the vertical distance from each point in the crack area to the fitting plane.

[0083] Step S327: Determine the depth of the crack according to the vertical distance from each point in the crack area to the fitting plane.

[0084] In this embodiment, step S327 determines the depth of the crack according to the vertical distance from each point in the crack area to the fitting plane. The depth of the crack can be determined using one of the following two methods.

[0085] (1) First, according to the vertical distance from each point in the crack area to the fitting plane, a depth statistics method is used, that is, the maximum value and average value of all vertical distance values ​​are counted to obtain statistical results; then, based on the statistical results, the depth of the crack as a whole is evaluated to determine the depth of the crack.

[0086] (2) First, a depth map is generated according to the vertical distance from each point in the crack area to the fitting plane using a depth map method; then, based on the depth map, a depth evaluation is performed on the entire crack to determine the depth of the crack.

[0087] Step S33: When the crack identification result is no crack, the above calculation operation is not required.

[0088] Step S4: classify the degree of damage of the crack according to the length, width and depth of the crack to determine the degree of damage of the crack.

[0089] In this embodiment, the hazard level of cracks includes a mild crack hazard level, a moderate crack hazard level, and a severe crack hazard level. Among them, the width of the crack corresponding to the mild crack hazard level is less than 3mm, the length is less than or equal to 30cm, and the depth is less than 2-3cm. The width of the crack corresponding to the moderate crack hazard level is 3-5mm, the length is 30-100cm, and the depth is greater than half of the surface layer. The width of the crack corresponding to the severe crack hazard level is greater than 5mm, the length is greater than 1m, and the depth penetrates the surface layer and even reaches the base layer or core structure. Among them, the surface layer, base layer, and core structure are frameworks for describing the characteristics of different levels of cracks. The surface layer refers to the visible part of the crack surface, which is in direct contact with the environment; the base layer refers to the middle layer of the crack, connecting the surface layer and the core, reflecting the expansion process of the crack; and the core structure is the innermost layer of the crack, which determines its formation mechanism and expansion direction.

[0090] Step S5: formulate a corresponding crack repair plan according to the degree of damage of the crack.

[0091] In this embodiment, the crack repair scheme includes a crack repair scheme corresponding to a mild crack hazard level, a crack repair scheme corresponding to a moderate crack hazard level, and a crack repair scheme corresponding to a severe crack hazard level. Among them, the crack repair scheme corresponding to the mild crack hazard level is: repair the cracks by sealing or grouting, regularly observing the cracks, and evaluating whether the cracks are expanding. The crack repair scheme corresponding to the moderate crack hazard level is: repair the cracks by grooving and milling, supplemented by local reinforcement or thin layer covering. The crack repair scheme corresponding to the severe crack hazard level is: local cutting of the cracks, overall milling renovation or local reconstruction, and determining whether to replace or redo the base layer according to the size of the cracks.

[0092] In this embodiment, after formulating a corresponding crack repair plan according to the hazard level of the crack in step S5, the UAV facility crack identification method may further include the following steps.

[0093] Step S6: Generate a report based on the image data, the point cloud data, the severity level of the crack and the crack repair plan, wherein the content of the report includes the location, length, width, depth and severity level of the crack and the corresponding crack repair plan, and the report is stored in a standard format.

[0094] In order to make the above technical solution of this embodiment clearer, the specific implementation process of the above technical solution of this embodiment is described in detail below in the form of examples. Figure 2 As shown, the specific implementation steps include the following.

[0095] S1: Check whether the drone hardware is normal, load the pre-trained weights of the deep learning model and the lidar data processing algorithm, and initialize the flight parameters, including the flight path, altitude, and speed.

[0096] In this embodiment, the deep learning model uses an improved Mask R-CNN model. The traditional Mask R-CNN model is a model for target detection and instance segmentation. The improved Mask R-CNN model in this embodiment further calculates the width and length of the crack based on the crack detection and instance segmentation functions of the traditional Mask R-CNN model. The laser radar data processing algorithm is mainly a method for processing point cloud data obtained by the laser radar, including preprocessing of point cloud data and analyzing the depth distribution of cracks based on point cloud data.

[0097] The image data and point cloud data obtained in this embodiment are obtained by the visual sensor and laser radar carried by the drone, respectively. It is necessary to check the hardware of the drone, including the inspection of the drone equipment, visual sensor, GPS, IMU and laser radar, etc., to ensure the safe operation of the drone hardware equipment, so as to obtain high-quality image data and point cloud data and ensure the accuracy of subsequent operations. In this embodiment, the visual sensor can use a high-definition camera. In order to obtain high-quality image data and point cloud data, the setting of the flight path, altitude, speed and interval time of the timed shooting of the drone should be changed according to the specific situation. For example, the flight altitude of the drone is set to 100 meters, the flight speed is 5m / s, and the shooting time interval is 7.5s, ensuring that the captured images have a certain overlap to facilitate subsequent image stitching.

[0098] S2: The drone uses a high-definition camera to capture high-resolution images of the target area, uses a lidar to generate three-dimensional point cloud data, and records the drone's flight altitude, position, time, and attitude.

[0099] In this embodiment, in order to ensure the accuracy of subsequent crack identification, when using a drone to obtain image data and point cloud data, the flight altitude, position, time and attitude of the drone when collecting the above data should be recorded to ensure data alignment. Among them, data alignment refers to unifying the information collected by different sensors (such as laser radar, IMU, GPS, etc.) into the same coordinate system or time reference, so that each laser point, image captured by a high-definition camera and the spatial position (position and attitude) of the drone can accurately correspond in the same coordinate frame. For example, a point scanned by the laser radar is (x, y, z) in the sensor coordinate system, and it is necessary to combine the actual position and attitude of the drone (the position and attitude are provided by GPS and IMU) to determine the true position of the point in the geographic coordinate system, thereby generating an accurate three-dimensional point cloud. Through data alignment, data from different sensors can correspond to each other in the same space and time coordinates to avoid misalignment or inaccurate overlap.

[0100] S3: Perform data preprocessing, including image preprocessing, point cloud data preprocessing, and data alignment.

[0101] When training the Mask R-CNN model, this embodiment needs to input the image and the label of the target area corresponding to the image, so the collected images need to be preprocessed, including using VIA to mark the crack area of ​​the image data. After all the images in each folder are marked, a json file is exported as the label of the image, and the acquired images are divided into a training set, a test set and a validation set in a ratio of 8:1:1. For point cloud data, in order to improve the accuracy of calculating the crack depth, it is necessary to denoise the point cloud data and remove outliers. This embodiment uses statistical filtering or radius filtering to remove outliers in the point cloud data, so as to achieve the purpose of denoising and removing outliers. Then the surface point cloud and non-surface point cloud contained are segmented to extract the surface point cloud data therein, so as to reduce interference and focus on the crack area. Among them, segmentation can be performed by RANSAC or a normal-based plane detection algorithm to extract the surface point cloud data. If the subsequent processing needs to reduce the amount of data, the point cloud data can be further downsampled, such as voxel grid downsampling, so that the amount of calculation can be significantly reduced and the calculation efficiency can be improved under the premise of ensuring geometric information. After completing the above operations, it is also necessary to perform coordinate correction based on external sensor data such as IMU and GPS to ensure that all point cloud fragments in the point cloud data are accurately aligned in a unified coordinate system, so that clean, structured point cloud data with reference coordinates is obtained, laying the foundation for subsequent feature extraction or target detection. In addition, this embodiment also performs data alignment between image data and point cloud data, so as to ensure that the image data and point cloud data correspond to the same crack, so that the length, width and depth of the crack obtained will be accurate and reliable.

[0102] S4: Use the improved Mask R-CNN model to segment the cracks in the image, calculate the width and length of the cracks, analyze the point cloud data, calculate the depth distribution of the cracks through the height difference, and use depth statistics or depth map methods to evaluate the overall depth of the cracks.

[0103] In step S4 of this embodiment, the image data of the training set and its labels are input into the established Mask R-CNN model for training, and the test set and the validation set are used for testing and verification. After the Mask R-CNN model identifies the cracks, the crack skeleton is extracted, and the length and width of the cracks are calculated using the crack skeleton. The depth of the cracks is calculated using the preprocessed point cloud data, and the extracted depth is used to generate a depth map to determine the depth distribution of the cracks.

[0104] This embodiment is based on the Mask R-CNN algorithm, with ResNet-FPN (Residual Network-Feature Pyramid Network) as the core network for feature extraction, which has high precision and fast running speed. ResNet (Residual Network) solves the degradation problem that deep networks are prone to by adding cross-layer connections to conventional neural networks. In order to further extract multi-scale image features, FPN (Feature Pyramid Network) is introduced on ResNet. FPN uses convolution kernels of different sizes for downsampling, obtains feature maps of multiple scales, and then fuses these feature maps, which significantly improves the accuracy of subsequent classification and prediction.

[0105] like Figure 3 As shown, the crack recognition process of the improved Mask R-CNN model in this embodiment includes the following steps: first, use a drone to acquire image data, use VIA to mark the crack area of ​​the acquired image data as a label, then build a data set and divide it into a training set, a test set and a validation set, then input the image and label to the improved Mask R-CNN model, use ResNet-FPN to extract features, use the backbone network to generate a feature map, and then hand over the feature map to RPN (Region Proposal Network) to generate a target proposal box, which is actually a candidate box for detecting and delineating cracks. At the same time, the feature map will also be handed over to the ROI Align (Region Of Interest Align) layer for pooling to perform classification, regression and segmentation on the candidate box. The classification and regression part is consistent with Faster R-CNN (Faster R-Convolutional Neural Network), and the segmentation part uses FCN (Fully Convolutional Networks) to generate the target mask. Finally, the output results of the improved Mask R-CNN model include candidate boxes for detecting cracks and corresponding mask results, which are referred to as crack masks.

[0106] like Figure 4As shown, this embodiment first performs crack detection through an improved Mask R-CNN model, obtains a crack mask, extracts a crack skeleton based on the crack mask, and calculates the length of the crack based on the crack skeleton. Then, a skeleton function fitting process is performed based on the crack skeleton to complete the calculation of the width of the crack. When calculating the depth of the crack, a nearby area is selected from the point cloud, and a fitting plane is fitted using an algorithm such as RANSAC or the least squares method, and the depth of the crack is calculated based on the fitting plane. Since the image data and the point cloud data have been aligned during preprocessing, after obtaining the length, width, and depth of the crack, the data alignment operation can effectively ensure that the calculated length, width, and depth are the length, width, and depth of the same crack.

[0107] In this embodiment, the length of the crack is extracted using the Mask R-CNN algorithm, that is, the crack mask is obtained through the improved Mask R-CNN model, the image is thinned by the connected domain, the crack skeleton is obtained, and the length of the crack is obtained by calculating the number of all pixels in the pixel point set of the crack skeleton. Specifically, the following steps are included: a skeleton extraction algorithm based on the morphological connectivity principle is used to calculate the pixel points of the crack area in the input crack mask point by point. Here, the crack mask is a binary crack mask image. The principle is mainly based on the connectivity between the pixels. Through iteration, it is determined whether it is a boundary point one by one, and the boundary points of the crack mask are removed, and finally the central axis of the crack mask is retained as the crack skeleton. The skeleton extraction algorithm is an important implementation of the image thinning algorithm. An excellent skeleton extraction algorithm must meet the following conditions: first, the skeleton extraction algorithm has convergence; for the thinned image, the regional connectivity property can be maintained; the original image shape is retained; the image finally extracts the central axis of the original connected area; the skeleton extraction algorithm runs faster and the thinning result is more accurate. The skeleton extraction algorithm meets the above requirements. The extracted crack skeleton is continuous and smooth and fits the bending shape of the crack, so it can be used to calculate the length of the crack.

[0108] For the calculation of the crack width, a quadratic function is fitted to the extracted crack skeleton, and the length of the intersection of the normal line of each point of the fitting function and the crack mask is calculated to obtain the crack width. Specifically, the following steps are included: Fitting the quadratic curve function using the least squares method. First, the crack mask is scattered, and only the coordinates of the points with non-zero pixel values ​​are stored. The horizontal and vertical coordinates of these points are stored in a list, and the polyfit function in the third-party library numpy (numerical python) is used to perform curve fitting, and the coefficients and expressions of the fitting function are obtained. When calculating the width of the crack, the normal equation of each point of the fitting function is required. According to the property that the normal and tangent slopes are reciprocals of each other, the normal equation of each point on the fitting curve can be calculated. The quadratic curve equation used is as follows.

[0109] .

[0110] in, x and y are the horizontal and vertical coordinates of the points on the fitting curve, , , are all fitting curve coefficients, then the tangent slope of each point on the curve Is the following formula.

[0111] .

[0112] The slope of the normal Is the following formula.

[0113] .

[0114] Since the coordinates of the points on the fitting curve are known, the constant term of the normal equation corresponding to each point can be calculated , which is expressed as the following formula.

[0115] .

[0116] Therefore, the normal equation corresponding to each point on the fitting curve can be calculated. are the coordinates of the points on the fitting curve, is the slope of the tangent line at each point on the curve, is the slope of the normal line, and the number of pixels where the normal line intersects the crack mask is taken as the pixel width of the crack at that point.

[0117] To calculate the depth of the crack, first perform plane fitting on the surrounding surface point cloud, and then calculate the vertical distance from each point in the crack area to the fitting plane. Specifically, the following steps are included: select a nearby surface area from the point cloud data, and use algorithms such as RANSAC or least squares method to fit a fitting plane. If the equation of the fitting plane is as follows.

[0118] .

[0119] Among them, (x, y, z) refers to the coordinates of any point in the crack area, indicating the position of the point in three-dimensional space. d is a constant term, which together with a, b, and c determines the specific position of the fitting plane in space. is the normal vector of the fitted plane.

[0120] For any point in the crack region , whose coordinates are ( , , ), the following crack depth calculation formula can be used to calculate the point The perpendicular distance to the fitting plane (i.e., local depth).

[0121] .

[0122] in, Indicates the midpoint of the crack area The vertical distance to the plane, ( , , ) is the midpoint of the crack area The coordinates of .

[0123] By statistically analyzing the vertical distance values ​​of all points in the crack area (such as maximum value, average value) or directly visualizing them into a depth map, the overall depth evaluation of the crack can be obtained. In this way, the depth of the crack can be quantitatively calculated and analyzed based on the point cloud data, and the depth of the crack can be accurately calculated.

[0124] S5: Classify the degree of damage of cracks according to their width, length and depth, and formulate a practical crack repair plan based on the classification results.

[0125] This embodiment uses a comprehensive evaluation model to classify the degree of damage of cracks. The comprehensive evaluation model refers to the following comprehensive evaluation method and comprehensive evaluation standard for cracks.

[0126] For various types of cracks (such as roads, tunnels, solar panels, etc.), crack repair should be divided into hazard levels based on their length, width, depth (or height). The hazard levels include mild crack hazard level, moderate crack hazard level and severe crack hazard level, and corresponding crack repair plans should be formulated according to different hazard levels, specifically including the following three situations.

[0127] (1) The characteristics of the mild crack hazard level are: the width of the crack is less than 3 mm, the length is less than or equal to 30 cm, and the depth is less than 2 to 3 cm.

[0128] Crack repair solutions for mild crack hazard levels: Simple sealing or grouting methods can be used for repair, and large-scale structural repairs are usually not required. For such cracks, regular observation is required to assess whether the cracks are expanding.

[0129] (2) The characteristics of moderate crack hazard level are: the width of the crack is 3 to 5 mm, the length is 30 to 100 cm, and the depth is greater than half of the surface layer.

[0130] The crack repair plan corresponding to the moderate crack hazard level: use methods such as grooving and milling, supplemented by local reinforcement or thin layer covering to enhance the stability and safety of the overall structure.

[0131] (3) The characteristics of severe crack hazard level are: the width of the crack is greater than 5 mm, the length is relatively long (such as greater than 1 m), and the depth penetrates the surface and even reaches the base layer or core structure.

[0132] The crack repair plan for severe crack hazard level: partial cutting, overall milling renovation or partial reconstruction is required. In severe cases, the foundation layer may need to be replaced or redone. Such cracks are usually accompanied by greater safety hazards and need to be handled as a priority.

[0133] Repair considerations: When repairing cracks, the specific environment and material properties of the cracks need to be considered. For example, crack repair in a tunnel may involve high waterproofing requirements, while crack repair in a solar panel needs to avoid affecting its power generation efficiency. Therefore, crack repair solutions should be optimized according to different usage environments. The goal of crack repair is not only to restore surface integrity, but also to ensure the long-term safety and performance of the structure. Therefore, in addition to a single repair measure, enhanced monitoring may be required to ensure that the cracks do not deteriorate further.

[0134] S6: Generate a report based on the image, point cloud data, hazard level and crack repair plan, including the location, size, depth and hazard level of the crack and the corresponding crack repair plan, and store it in a standard format.

[0135] In this embodiment, the image data and point cloud data obtained by each inspection of the drone can be processed and integrated, and the specific location of the detected cracks can be marked in the report (geographic coordinates or coordinates relative to the reference point can be used), and the width, length, depth, and assessed hazard level and other information are attached, and then stored in a standard format (such as GeoTIFF, LAS / LAZ, Shapefile or PDF, etc.). The above standard formats are selected because they have good compatibility and versatility, which facilitates visualization, analysis and cross-platform exchange in GIS, CAD or point cloud processing software in the later stage. At the same time, different professional departments (such as maintenance, mapping, construction units, etc.) can also read and use these data more conveniently. For example, GeoTIFF is suitable for storing images and their geographic reference information, LAS / LAZ can record three-dimensional point clouds and have compression capabilities, Shapefile can be used to describe vector information (such as polygonal boundaries or center lines of cracks), and PDF reports are easy to archive and quickly access. By storing the data in a standard format, it can be ensured that the data will not lose accuracy during subsequent maintenance, reassessment or traceability. It also makes it convenient for all parties to check and compare the location, size, degree of hazard and crack repair plan of the cracks at any time, providing a reliable basis for the subsequent implementation of the crack repair plan.

[0136] In an exemplary embodiment, a UAV facility crack identification system is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the UAV facility crack identification method.

[0137] In this embodiment, a UAV is used in the air to obtain image data and point cloud data using visual sensors and laser radars. For image data, VIA is used to mark the crack area of ​​the image data. After all images in each folder are marked, a json file is exported as the label of the image. An improved Mask R-CNN model is built for crack recognition in UAV images, and the training set and its labels are input into the improved Mask R-CNN model for training. The test set and the validation set and the labels are input into the trained improved Mask R-CNN model to verify the crack recognition effect of the model. When the improved Mask R-CNN model identifies a crack, image thinning is performed on the connected domain to obtain the crack skeleton. The length of the crack is obtained by calculating the number of all pixels in the pixel point set of the crack skeleton. For the calculation of the crack width, a quadratic function is fitted to the extracted crack skeleton, and the length of the intersection of the normal of each point of the fitting function with the crack mask is calculated, thereby obtaining the width of the crack. For the depth of the crack, the point cloud data obtained by the laser radar carried by the drone can be used to calculate the depth of the crack. Since the depth is uneven, it can be deepened into a depth map to analyze the depth distribution of the crack to determine the depth of the crack. Finally, according to the length, width and depth of the crack, the degree of damage is divided, and then the corresponding crack repair plan is formulated according to its degree of damage.

[0138] When identifying cracks, this embodiment not only simply identifies whether there are cracks, but also identifies the characteristic information of the cracks, including the length, width and depth of the cracks, thereby avoiding the deficiency that the current crack identification cannot extract information such as the size and depth of the cracks, and combines the depth information to ensure the accuracy of crack identification, thereby improving the reliability of subsequent analysis of identification results. In addition, this embodiment establishes an evaluation method for the degree of damage of cracks by obtaining information such as the depth and size of the cracks, and formulates targeted crack repair plans based on the degree of damage of the cracks, thereby avoiding the limitation of using the same repair plan for all crack situations, ensuring the reliability and effectiveness of the crack repair plan, and improving the efficiency and effect of crack repair.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for identifying cracks in unmanned aerial vehicle facilities, characterized in that: The UAV facility crack identification method comprises: Obtain image data and point cloud data of the target area collected by the drone; Preprocessing the image data and the point cloud data respectively to obtain preprocessed image data and preprocessed point cloud data; According to the preprocessed image data, using the improved Mask R-CNN model to identify cracks, and after identifying the cracks, calculating the length and width of the cracks, and according to the preprocessed point cloud data, calculating the depth of the cracks; Classifying the degree of damage of the crack according to the length, width and depth of the crack to determine the degree of damage of the crack; According to the degree of hazard of the cracks, a corresponding crack repair plan is formulated.

2. The method for identifying cracks in unmanned aerial vehicle facilities according to claim 1, characterized in that: The drone is equipped with a visual sensor, a laser radar, a GPS and an IMU. The visual sensor is used to collect image data of the target area, the laser radar is used to collect point cloud data of the target area, and the GPS and IMU are used to record the flight altitude, time, position and posture of the drone; Obtain image data and point cloud data of the target area collected by the drone, including: The image data, the point cloud data, and the flight altitude, time, position, and posture of the unmanned aerial vehicle when collecting the image data and the point cloud data are obtained.

3. The method for identifying cracks in unmanned aerial vehicle facilities according to claim 2, characterized in that: Preprocessing the image data and the point cloud data respectively to obtain preprocessed image data and preprocessed point cloud data specifically includes: Using VIA to mark the crack area in the image data, and exporting a json file as a label of the image data; De-noising and outlier elimination are performed on the point cloud data to obtain the point cloud data after de-noising and outlier elimination; Segmenting the surface point cloud and the non-surface point cloud contained in the point cloud data after denoising and removing outliers, so as to extract the surface point cloud and obtain segmented point cloud data; According to the flight altitude, time, position and posture of the UAV, the image data and the segmented point cloud data are aligned to obtain preprocessed image data and preprocessed point cloud data.

4. The method for identifying cracks in UAV facilities according to claim 3, characterized in that: Before the steps of identifying cracks using the improved Mask R-CNN model according to the preprocessed image data, and calculating the length and width of the cracks after identifying the cracks, and calculating the depth of the cracks according to the preprocessed point cloud data, the UAV facility crack identification method further includes: Based on the image data and the corresponding labels, a training set, a test set and a validation set are divided according to a preset ratio; The improved Mask R-CNN model is trained using the training set, and the improved Mask R-CNN model is tested and verified using the test set and the validation set to obtain an improved Mask R-CNN optimized model; the improved Mask R-CNN optimized model is used as the improved Mask R-CNN model to identify cracks and calculate the length and width of the cracks.

5. The method for identifying cracks in UAV facilities according to claim 3, characterized in that: According to the preprocessed image data, cracks are identified using an improved Mask R-CNN model, and after the cracks are identified, the length and width of the cracks are calculated, and according to the preprocessed point cloud data, the depth of the cracks is calculated, specifically including: Inputting the preprocessed image data into the improved Mask R-CNN model to obtain a crack recognition result, wherein the crack recognition result includes whether there is a crack or not; When the crack identification result is that there is a crack, the length, width and depth of the crack are calculated respectively, including the following steps: According to the preprocessed image data, using the improved Mask R-CNN model to obtain a crack mask; Extracting a crack skeleton according to the crack mask; Calculate the number of all pixels in the pixel point set of the crack skeleton to obtain the length of the crack; Using the least square method, a quadratic function is fitted to the crack skeleton, and the length of the intersection of the normal line of each point of the fitting function and the crack mask is calculated as the width of the crack; According to the preprocessed point cloud data, a RANSAC algorithm or a least squares method is used to perform plane fitting on the surrounding surface point clouds in the preprocessed point cloud data to obtain a fitting plane; Based on the fitting plane, calculating the vertical distance from each point in the crack area to the fitting plane; Determine the depth of the crack according to the vertical distance from each point in the crack area to the fitting plane; When the crack identification result is no crack, no calculation operation is performed.

6. The method for identifying cracks in UAV facilities according to claim 5, characterized in that: Determining the depth of the crack according to the vertical distance from each point in the crack area to the fitting plane specifically includes: According to the vertical distances from each point in the crack area to the fitting plane, the maximum value and the average value of all vertical distance values ​​are counted to obtain statistical results; According to the statistical results, the depth of the crack as a whole is evaluated to determine the depth of the crack.

7. The method for identifying cracks in UAV facilities according to claim 5, characterized in that: Determining the depth of the crack according to the vertical distance from each point in the crack area to the fitting plane specifically includes: Generate a depth map according to the vertical distance from each point in the crack area to the fitting plane; According to the depth map, the depth of the crack as a whole is evaluated to determine the depth of the crack.

8. The method for identifying cracks in UAV facilities according to claim 1, characterized in that: The hazard level of the cracks includes a mild crack hazard level, a moderate crack hazard level and a severe crack hazard level; The cracks corresponding to the mild crack hazard level are less than 3 mm in width, less than or equal to 30 cm in length, and less than 2 to 3 cm in depth; The cracks corresponding to the moderate crack hazard level are 3 to 5 mm wide, 30 to 100 cm long, and more than half the depth of the surface layer; The severe crack hazard level corresponds to cracks with a width greater than 5 mm, a length greater than 1 m, and a depth that penetrates the surface layer and even reaches the base layer or core structure; The crack repair scheme includes a crack repair scheme corresponding to the mild crack hazard level, a crack repair scheme corresponding to the moderate crack hazard level, and a crack repair scheme corresponding to the severe crack hazard level; The crack repair solution corresponding to the mild crack hazard level is to repair the cracks by sealing or grouting, regularly observe the cracks, and evaluate whether the cracks are expanding; The crack repair solution corresponding to the moderate crack hazard level is to repair the cracks by grooving and milling, supplemented by local reinforcement or thin layer covering; The crack repair solution corresponding to the severe crack hazard level is to perform local cutting, overall milling and renovation or local reconstruction of the cracks, and determine whether to replace or redo the foundation layer based on the size of the cracks.

9. The method for identifying cracks in unmanned aerial vehicle facilities according to claim 1, characterized in that: After the step of formulating a corresponding crack repair plan according to the hazard level of the crack, the UAV facility crack identification method further includes: A report is generated based on the image data, the point cloud data, the hazard level of the cracks and the crack repair plan, wherein the content of the report includes the location, length, width, depth and hazard level of the cracks and the corresponding crack repair plan, and the report is stored in a standard format.

10. A UAV facility crack identification system, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for identifying cracks in unmanned aerial vehicle facilities as described in any one of claims 1 to 9.

Images