A dam crack identification and measurement method based on drone vision

Abstract

A method for identifying and measuring cracks in a dam body based on drone vision includes the following steps: Step 1: collecting images, collecting RGB images of the dam body to be detected, and the collected images can completely cover the detection area; Step 2: performing rough detection of cracks; Step 3: performing precise crack identification based on visual saliency; Step 4: performing edge extraction; Step 5: performing crack feature information calculation; Step 6: performing image stitching, restoring the panoramic view of the dam body through image stitching, thereby determining the position information of the local crack relative to the dam body; Step 7: performing crack positioning and evaluation. The purpose of the present invention is to solve the technical problems of the existing technology of large errors in dam body crack detection under complex environments and low precision in crack geometric information measurement, and to provide a method for identifying and measuring cracks in a dam body based on drone vision.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision detection and measurement, and in particular to a dam body crack recognition and measurement method based on unmanned aerial vehicle vision. Background Art

[0002] Cracks are a major hazard to the structural safety of hydropower station dams. The detection of crack positions and geometric shapes, especially the measurement of width, is an important part of dam inspection. Traditional inspection methods are mainly based on manual visual inspection, which requires the construction of scaffolding, hanging baskets or the use of telescopes. It consumes a lot of manpower and material resources, has great safety risks, and has low inspection efficiency. The inspection effect often depends on the experience of the inspectors.

[0003] With the development of artificial intelligence technology and the drone industry, more and more operation and maintenance personnel are introducing drones combined with AI inspections into the operation and maintenance of industries such as dams or roads and bridges. They use remote-controlled drones to collect images of the area to be inspected, and then use computer vision technology to identify cracks.

[0004] For example, the patent document with application number 2017103826144 proposes a surface crack recognition method based on saliency detection. This method first transforms the image from RGB to Lab color space, then pre-segments the image using a superpixel segmentation algorithm, clusters the image using a mean shift algorithm, then extracts the color saliency map and texture saliency map of the image, and finally adaptively segments the crack area.

[0005] Patent document No. 201811332035X proposes a method for detecting cracks on the bottom surface of a bridge based on drone vision. The method uses a visible light camera mounted on a drone to take a series of RGB images of the bridge in blocks according to a predetermined trajectory, and then processes the images by numbering them. The image is then enhanced and denoised to extract crack features, and finally the crack area is binarized and segmented. A three-layer BP neural network is also established in the method to classify the types of cracks.

[0006] When faced with complex background environments, the above-mentioned existing technologies have technical defects such as low algorithm recognition accuracy and slow speed, which make it difficult to meet the needs of real-time and efficient crack detection. At the same time, the detection lacks measurement of information such as crack length or width, resulting in inaccurate crack risk assessment. Summary of the invention

[0007] The purpose of the present invention is to solve the technical problems of large errors in dam crack detection in complex environments and low accuracy in crack geometry information measurement in the prior art, and to provide a dam crack identification and measurement method based on drone vision.

[0008] A method for identifying and measuring dam cracks based on UAV vision includes the following steps:

[0009] Step 1: Collect images, collect RGB images of the dam to be inspected, and the collected images can completely cover the inspection area;

[0010] Step 2: Perform a rough crack detection;

[0011] Step 3: Perform precise crack identification based on visual saliency;

[0012] Step 4: perform edge extraction;

[0013] Step 5: Calculate crack characteristic information;

[0014] Step 6: Perform image stitching; restore the panoramic view of the dam body through image stitching, so as to determine the position information of the local crack relative to the dam body;

[0015] Step 7: Perform crack location and assessment.

[0016] In step 2, a pre-trained convolutional neural network model is used to detect whether there are cracks in the image. If no cracks are detected, there is no need to perform the next step of refined detection, thereby improving the efficiency of the algorithm; if cracks are detected, the model outputs the crack boundary box in the image, thereby filtering out complex background interference information and improving the accuracy of subsequent crack edge extraction and crack measurement.

[0017] In step 3, based on the difference in color and brightness between the cracks in the image and the image background, the FT algorithm is used to obtain the saliency map of the image, and then the mean shift algorithm is used to adaptively calculate the image threshold, and then the cracks are segmented by binarization.

[0018] In step 4, the Canny edge detection algorithm is used to extract crack edge pixels, the image is smoothed with a Gaussian filter, the finite difference of the first-order partial derivative is used to calculate the amplitude and direction of the gradient, the gradient amplitude is non-maximum suppressed, and the double threshold algorithm is used to detect and connect the edges.

[0019] In step 5, the camera is calibrated first to calculate the internal and external parameters of the acquisition camera, which are used to calculate the conversion between the image coordinate system and the world coordinate system. Then, the width is calculated using the shortest distance method based on the crack edge pixels. The Euclidean distance between the starting point and the end point of the crack is calculated as the crack length. The angle of the crack fitting line is taken as the angle of the crack.

[0020] It also includes step 7: locating and evaluating the cracks; locating the position of the cracks relative to the dam body to facilitate the operation and maintenance personnel to accurately determine the location information of the cracks; determining the hazard level of the cracks through the length, width, angle and area information of the cracks in step 5; and predicting the development trend of the cracks, and promptly doing a good job in dam body hazard work to prevent them from happening.

[0021] In step 2, when performing a rough crack detection, the following steps are specifically adopted:

[0022] 2.1: Collect data sets; collect dam crack data in cloudy, foggy, sunny, and backlit environments, ensuring that the collected data sets cover various conditions under general testing environments;

[0023] 2.2. Expand the data set by rotating the image, changing the brightness, cutting, and adding Gaussian noise.

[0024] 2.3. Label the data set; use image annotation tools in semi-supervised learning to label the data set, improve the speed and accuracy of feature extraction, make the data into a specified format, and generate a specified file for the corresponding image;

[0025] 2.4. Optimize the network; (1) To address the problem of different crack sizes, use the K-means++ algorithm to redesign the prior box size and match it to the corresponding feature layer; (2) Introduce a multi-spectral channel attention module in the feature extraction network so that the network can autonomously learn the weight of each channel and enhance the information propagation between features, thereby strengthening the network's ability to distinguish foreground and background, and randomly input images of different sizes during the training iteration process to enhance the generalization ability of the model; (3) Use CIoU_loss as the regression loss function for the target detection task. CIoU_loss considers the overlapping area and center point distance between the prediction box and the target box. When the target box wraps the prediction box, the distance between the two boxes is directly measured, thereby considering the information of the distance between the center points of the bounding box and the scale information of the aspect ratio of the bounding box. At the same time, the aspect ratio of the prediction box and the target box is also considered to make the boundary regression result better. The loss function calculation formula is as follows

[0026]

[0027] Among them, IoU is the ratio of the intersection and union of the prediction box and the target box, the center point of the prediction box is represented by b, and the center point of the target box is represented by b gt represents, ρ(·) represents the Euclidean distance, c represents the diagonal distance of the minimum circumscribed rectangle between the intersecting prediction box and the target box, and α is a weight coefficient. ν represents the parameter for aspect ratio consistency, w gt and h gt Represent the width and height of the target box respectively, w and h represent the width and height of the prediction box;

[0028] 2.5. Train the model; bring the prepared data set into the optimized network for training, continuously iterate the model through parameter adjustment to minimize the loss function of the model; finally generate a crack detection model;

[0029] 2.6. Detect the crack area; call the crack detection model, input the image to be detected into the model, and output the crack category and bounding box through the end-to-end model;

[0030] In step 3, when performing precise crack identification based on visual saliency, the following steps are specifically adopted:

[0031] 3.1. Calculate the image saliency map; intercept the crack area image through the crack boundary identified by the crack detection model, perform Gaussian filtering and denoising on the image, and then convert the image color space from RGB to Lab; calculate the average value of L, a, and b of the entire image; according to the formula in the algorithm, calculate the Euclidean distance between the L, a, and b values ​​of each pixel and the average value of the three L, a, and b values ​​of the image to obtain the saliency map; normalize, divide the saliency value of each pixel in the image by the largest saliency value to obtain the final saliency map; the saliency of the pixel can be calculated using the following formula

[0032] S(p)=||I μ -I ωhc (p)||

[0033] Among them, I μ is the feature vector of the average image, I ωhc (p) is the Lab color feature of pixel p after Gaussian smoothing, ||.|| is the L2 norm, that is, the Euclidean distance between the previous and the next items in the Lab color space is calculated;

[0034] 3.2. Use the Meanshift method to divide the original image into K large blocks, and use the saliency calculation method in 3.1 to calculate the average Sk of the saliency map corresponding to each segmented block;

[0035] 3.3. Calculate the average value Sμ of the entire saliency map;

[0036] 3.4. Calculate the adaptive threshold Ta = 2*Sμ;

[0037] 3.5. When Sk>Ta, the region is considered to be a saliency map;

[0038] 3.6. Perform binary segmentation on the salient area to obtain a refined segmentation image of the crack.

[0039] In step 4, the Canny algorithm is used to detect the crack edge, which specifically includes the following steps:

[0040] 4.1. Eliminate noise; use Gaussian smoothing filter convolution to reduce noise and eliminate the noise around the cracks; the Gaussian kernel used in Gaussian filtering is the product of two one-dimensional Gaussians x and y. The standard deviation σ in the two dimensions is usually the same, in the following form: Use Gaussian smoothing filter convolution to reduce noise;

[0041] 4.2. Calculate the gradient magnitude and direction. Follow the steps of Sobel filtering. First, apply a pair of convolution arrays in the x and y directions respectively, that is, Then calculate the gradient magnitude and direction

[0042] 4.3, non-maximum suppression;

[0043] 4.4. Hysteresis threshold, i.e., hysteresis high threshold and low threshold; specifically, if the amplitude of the pixel position exceeds the high threshold, the pixel is retained as an edge pixel; if the amplitude of the pixel position is less than the low threshold, the pixel is excluded; if the amplitude of the pixel position is between the two thresholds, the pixel is retained only when it is connected to a pixel above the high threshold.

[0044] In step 5, when calculating the crack characteristic information, including measuring the length and width of the crack, the following steps are specifically included:

[0045] 5.1. Camera calibration to obtain internal and external parameters. After completing the crack image recognition, for the image containing the crack, calculate the edge coordinates of the crack at each point in the image, and calculate the actual size of the crack based on the coordinate transformation relationship. The Zhang Zhengyou calibration method can directly obtain the internal and external parameters of the camera; as shown in the following formula

[0046] T=[t 1 t 2 t 3 ] T ;

[0047] Among them, A is the camera's intrinsic matrix, R is the extrinsic rotation matrix, T is the translation vector, α = f / dx, β = f / dy, f is the focal length, dx and dy are the width and height of the pixel respectively; γ represents the deviation of the pixel point in the x and y directions, (u 0 ,v 0 ) as the reference point;

[0048] 5.2. Conversion from pixel coordinates to world coordinates; the relationship between the pixel coordinate system and the world coordinate system is shown in the following matrix:

[0049]

[0050] Where [XYZ] T Represents the world coordinate system, [uv] T Represents the pixel coordinate system, the coordinate is located in the upper left corner of the image, Z C is the corresponding Z-axis coordinate value in the camera coordinate system;

[0051] 5.3. Calculation of crack width: Using the shortest distance method, take the vertical crack as an example, after the crack image is marginalized, take the left edge point of each row as the center; find the minimum value of the distance between the right edge points in several adjacent rows as the crack width of the row; the Euclidean distance calculation method of the crack width is as follows:

[0052]

[0053] Where w is the crack width, Z is the width of the crack under the monocular camera. l =Z r =0; the maximum value of the crack w max =max(w i ), the average value of the crack is w i is the width of the i-th crack row, and N is the total number of rows;

[0054] 5.4. Length calculation: The skeleton information obtained after the thinning operation is a thin line graphic with a single pixel arrangement. The principle of calculating the length information of the dam body crack can be to calculate the Euclidean distance between the starting point and the end point of the thinned crack as the length information of the dam body crack;

[0055] 5.5. Angle calculation: Since the crack pattern of the dam body has an irregular shape, the crack image needs to be fitted into a straight line that can be calculated during the processing;

[0056] 5.6. Area calculation: The area of ​​the crack is the smallest circumscribed rectangular area of ​​the crack.

[0057] Compared with the prior art, the present invention has the following technical effects:

[0058] 1. The present invention proposes a crack recognition and measurement method based on drone vision. The dam body is automatically inspected by a drone equipped with a high-definition camera. The collected images are automatically processed by the crack image recognition algorithm proposed in the present invention, so that the cracks in the dam body can be quickly and accurately identified, and their length and width can be measured, thereby improving the efficiency of dam body maintenance and hazard prediction.

[0059] 2. The present invention proposes a crack detection method based on improved YOLOv5, which improves the speed and accuracy of crack detection by improving the YOLOv5 network model. By detecting the crack area and filtering the background interference information, the accuracy of subsequent crack segmentation and measurement is improved.

[0060] 3. The present invention proposes a Meanshift crack segmentation method based on saliency image detection. By introducing a saliency detection step into the traditional Meanshift segmentation algorithm, the accuracy and robustness of crack segmentation are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0062] Figure 1 It is a flow chart of the crack identification and measurement method based on UAV vision in the present invention;

[0063] Figure 2 It is a flow chart of the crack detection method based on improved YOLOv5 in the present invention;

[0064] Figure 3 is a flow chart of crack precise identification based on visual saliency in the present invention;

[0065] Figure 4 It is a schematic diagram of the crack area segmented by the method of the present invention. DETAILED DESCRIPTION

[0066] like Figure 1 As shown, a dam crack identification and measurement method based on UAV vision includes the following steps:

[0067] 1. Image acquisition: Use a DJI M300RTK drone equipped with a Zenmuse H20T camera to fly along a predetermined trajectory. By adjusting the drone’s hovering posture and the camera’s angle, focal length, and other information, the RGB images of the dam are collected sequentially.

[0068] 2. Rough crack detection based on YOLOv5: Use the pre-trained YOLOv5 convolutional neural network model to detect whether there are cracks in the image. If no cracks are detected, there is no need to do the next step of refined detection; if the image is detected to contain cracks, the model outputs the boundary box of the crack, thereby filtering out complex background interference information and improving the accuracy of subsequent crack edge extraction and crack measurement. The steps of rough crack detection based on YOLOv5 are as follows: Figure 2 shown.

[0069] 3. Precise crack recognition based on visual saliency: Based on the difference in color and brightness between the cracks in the image and the image background, the FT algorithm is used to obtain the saliency map of the image, and then the mean shift algorithm (MeanShift) is used to adaptively calculate the image threshold, and finally the cracks are segmented by binarization. The flowchart of precise crack recognition based on visual saliency is as follows: Figure 3 As shown in the figure, the binary image of the crack region after segmentation is as follows: Figure 4 shown.

[0070] 4. Edge extraction; use the Canny edge detection algorithm to extract crack edge pixels and use a Gaussian filter to smooth the image; use the finite difference of the first-order partial derivative to calculate the amplitude and direction of the gradient; perform non-maximum suppression on the gradient amplitude; use the dual threshold algorithm to detect and connect edges.

[0071] 5. Calculate the crack feature information; calibrate the camera to obtain the internal and external parameters of the camera. Then calculate the width using the shortest distance method based on the crack edge pixels; calculate the Euclidean distance between the crack start and end points as the crack length; and take the angle of the crack fitting line as the crack angle.

[0072] 6. Image stitching: restore the panoramic view of the dam body through image stitching, so as to determine the position information of the local crack relative to the dam body. The optional image stitching algorithms include SIFT, SURF, and ORB matching methods.

[0073] 7. Crack location and assessment: Locate the position of the crack relative to the dam body, so that the operation and maintenance personnel can accurately determine the location information of the crack. Determine the hazard level of the crack by comparing the length, width, angle and area of ​​the crack in step 5. And predict the development trend of the crack, and do a good job in dam body hazard work in time to prevent it.

[0074] like Figure 2 As shown, the present invention also provides a crack detection method, which comprises the following steps:

[0075] 1) Collect data sets; use the Zenmuse H20T camera to collect dam crack data in cloudy, foggy, sunny, and backlit environments to ensure that the collected data sets cover various conditions under general testing environments.

[0076] 2) Expand the data set by rotating the image, changing the brightness, cutting it, adding Gaussian noise, etc.

[0077] 3) Label the dataset; use the LabelImg tool to label the dataset, convert the data into VOC format, and generate an .xml file for the corresponding image.

[0078] 4) Optimize the network; (1) To address the problem of different crack sizes, use the K-means++ algorithm to redesign the prior box size and match it to the corresponding feature layer; (2) Introduce a multi-spectral channel attention module in the feature extraction network so that the network can autonomously learn the weight of each channel and enhance the information propagation between features, thereby strengthening the network's ability to distinguish foreground and background, and randomly input images of different sizes during the training iteration process to enhance the generalization ability of the model. (3) Use CIoU_loss as the regression loss function for the target detection task. CIoU_loss takes into account the overlapping area and center point distance between the prediction box and the target box. When the target box wraps the prediction box, the distance between the two boxes is directly measured, thereby taking into account the information of the distance between the center points of the bounding box and the scale information of the aspect ratio of the bounding box. At the same time, the aspect ratio of the prediction box and the target box is also considered to make the boundary regression result better. The loss function calculation formula is as follows

[0079]

[0080] Among them, IoU is the ratio of the intersection and union of the prediction box and the target box, the center point of the prediction box is represented by b, and the center point of the target box is represented by b gt represents, ρ(·) represents the Euclidean distance, c represents the diagonal distance of the minimum circumscribed rectangle between the intersecting prediction box and the target box, and α is a weight coefficient. ν represents the parameter for aspect ratio consistency, w gt and h gt Represent the width and height of the target box, respectively, and w and h represent the width and height of the prediction box.

[0081] 5) Train the model; replace the anchor values ​​in the network with the anchor values ​​obtained by clustering the crack data set, set epochs to 10000, batch_size to 64, and the initial value of the learning rate to 0.01. Train the model to minimize the loss function and iterate to generate the YOLOv5 crack detection model.

[0082] 6) Detection of crack areas: Call the YOLOv5 crack detection model, input the image to be detected into the model, and output the crack category and bounding box through the end-to-end model.

[0083] In step 4), the network is a YOLOv5 network; in step 5), the model is a YOLOv5 model;

[0084] like Figure 3 As shown, the present invention also includes a Meanshift crack segmentation method based on saliency image detection, which includes the following steps:

[0085] 1) Calculate the image saliency map; identify the crack boundary through the YOLOv5 model, intercept the crack area image, perform Gaussian filtering and denoising on the image, and then convert the image color space from RGB to Lab. Calculate the average value of L, a, and b for the entire image. According to the formula in the algorithm, calculate the Euclidean distance between the L, a, and b values ​​of each pixel and the average value of the three L, a, and b values ​​of the image to obtain the saliency map. Normalize, divide the saliency value of each pixel in the image by the largest saliency value to obtain the final saliency map. The saliency of a pixel can be calculated using the following formula

[0086] S(p)=||I μ -I ωhc (p)||

[0087] Among them, I μ is the feature vector of the average image, I ωhc(p) is the Lab color feature of pixel p after Gaussian smoothing, ||.|| is the L2 norm, that is, the Euclidean distance between the previous and the next items in the Lab color space is calculated;

[0088] 2) Use the Meanshift method to divide the original image into K large blocks and calculate the average Sk of the saliency map corresponding to each block;

[0089] 3) Calculate the average value Sμ of the entire saliency map;

[0090] 4) Calculate the adaptive threshold Ta = 2*Sμ;

[0091] 5) When Sk>Ta, the region is considered to be a saliency map;

[0092] 6) Perform binary segmentation on the salient area to obtain a refined segmentation image of the crack;

[0093] In step 4, the crack edge detection using the Canny algorithm specifically includes the following steps:

[0094] 4.1. Eliminate noise: Use Gaussian smoothing filter convolution to reduce noise and eliminate the noise around the cracks. The Gaussian kernel used in Gaussian filtering is the product of two one-dimensional Gaussians x and y. The standard deviation σ in the two dimensions is usually the same, in the following form: Here, a Gaussian smoothing filter is used for convolution noise reduction. The Gaussian kernel of size = 5 is selected as follows:

[0095]

[0096] 4.2. Calculate the gradient magnitude and direction. Here we follow the steps of Sobel filtering. First, use a pair of convolution arrays to act on the x and y directions respectively, that is,

[0097] Then calculate the gradient magnitude and direction

[0098] 4.3. Non-maximum suppression: used to exclude non-edge pixels and retain only some fine lines;

[0099] 4.4. Hysteresis threshold, i.e., hysteresis high threshold and low threshold. If the amplitude of the pixel position exceeds the high threshold, the pixel is retained as an edge pixel; if the amplitude of the pixel position is less than the low threshold, the pixel is excluded; if the amplitude of the pixel position is between the two thresholds, the pixel is retained only when it is connected to a pixel above the high threshold;

[0100] The present invention also includes a method for measuring the length and width of a dam body crack, which specifically comprises the following steps:

[0101] 5.1. Camera calibration to obtain internal and external parameters; After completing the crack image recognition, for images containing cracks, in order to measure the actual size of the cracks later, it is necessary to calculate the edge coordinates of the cracks at each point in the image, and calculate the actual size of the cracks based on the coordinate transformation relationship; the Zhang Zhengyou calibration method can directly obtain the internal and external parameters of the camera. As shown in the following formula:

[0102] T=[t 1 t 2 t 3 ] T ;

[0103] Among them, A is the camera's intrinsic matrix, R is the extrinsic rotation matrix, T is the translation vector, α = f / dx, β = f / dy, f is the focal length, dx and dy are the width and height of the pixel respectively; γ represents the deviation of the pixel point in the x and y directions, (u 0 ,v 0 ) as the reference point;

[0104] 5.2. Conversion from pixel coordinates to world coordinates; the relationship between the pixel coordinate system and the world coordinate system is shown in the following matrix:

[0105]

[0106] Where [XYZ] T Represents the world coordinate system, [uv] T Represents the pixel coordinate system, the coordinate is located in the upper left corner of the image, Z C is the corresponding Z-axis coordinate value in the camera coordinate system;

[0107] 5.3. Calculation of crack width: Using the shortest distance method, take the vertical crack as an example. After the crack image is marginalized, the left edge point of each row is taken as the center, and the minimum distance to the right edge point within 5 adjacent rows is calculated as the crack width of the row. The Euclidean distance calculation method of the crack width is as follows:

[0108]

[0109] Where w is the crack width, Z is the width of the crack under the monocular camera. l =Z r = 0. The maximum value of the crack w max =max(w i ), the average value of the crack is w i is the width of the i-th crack row, and N is the total number of rows;

[0110] 5.4. Length calculation: The skeleton information obtained after the thinning operation is a thin line graphic with a single pixel arrangement. The principle of calculating the length information of the dam body crack can be to calculate the Euclidean distance between the starting point and the end point of the thinned crack as the length information of the dam body crack;

[0111] 5.5. Angle calculation: Since the crack pattern of the dam body has an irregular shape, the crack image needs to be fitted into a straight line that can be calculated during the processing;

[0112] 5.6. Area calculation: The area of ​​the crack is the smallest circumscribed rectangular area of ​​the crack.

[0113] The present invention proposes a dam crack recognition and measurement method based on drone vision. The method collects crack images of the dam through a high-definition camera carried by a drone, roughly recognizes the cracks through an improved YOLOv5, and then segments the crack area through an improved saliency detection algorithm. The edge information of the crack is extracted through edge detection, thereby calculating the length, width and other information of the crack, and finally locating the position information of the crack relative to the dam through splicing images. The present invention can quickly and accurately identify dam cracks and measure their length and width, thereby improving the efficiency of dam maintenance and hazard prediction.

Claims

1. A dam crack identification and measurement method based on drone vision, characterized in that: It includes the following steps: Step 1: Collect images, collect RGB images of the dam to be inspected, and the collected images can completely cover the inspection area; Step 2: Perform a rough crack detection; Step 3: Perform precise crack identification based on visual saliency; Step 4: perform edge extraction; Step 5: Calculate crack characteristic information; Step 6: Perform image stitching to restore the panoramic view of the dam body, thereby determining the position information of the local crack relative to the dam body; Step 7: Crack location and evaluation; In step 2, when performing a rough crack detection, the following steps are specifically adopted: Step 2.1: Collect data sets; collect dam crack data in cloudy, foggy, sunny and backlit environments to ensure that the collected data sets cover various conditions under general testing environments; Step 2.2: Expand the data set by rotating the image, changing the brightness, cutting, and adding Gaussian noise. Step 2.3: Label the data set; use image annotation tools in semi-supervised learning to label the data set, improve the speed and accuracy of feature extraction, make the data into a specified format, and generate a specified file for the corresponding image; Step 2.4: Optimize the YOLOv5 network; (1) To address the problem of different crack sizes, the K-means++ algorithm is used to redesign the prior box size and match it to the corresponding feature layer; (2) A multi-spectral channel attention module is introduced into the feature extraction network, so that the network can autonomously learn the weight of each channel and enhance the information propagation between features, thereby strengthening the network's ability to distinguish foreground and background. In addition, images of different sizes are randomly input during the training iteration process to enhance the generalization ability of the model; (3) CIoU_loss is used as the regression loss function for the target detection task. CIoU_loss considers the overlapping area and center point distance between the prediction box and the target box. When the target box wraps the prediction box, the distance between the two boxes is directly measured, thereby considering the information of the distance between the center points of the bounding box and the scale information of the aspect ratio of the bounding box. At the same time, the aspect ratio of the prediction box and the target box is also considered to make the boundary regression result better. The loss function calculation formula is as follows Among them, IoU is the ratio of the intersection and union of the prediction box and the target box, the center point of the prediction box is represented by b, and the center point of the target box is represented by b gt represents, ρ(·) represents the Euclidean distance, c represents the diagonal distance of the minimum circumscribed rectangle between the intersecting prediction box and the target box, and α is a weight coefficient. A parameter representing aspect ratio consistency, w gt and h gt Represent the width and height of the target box respectively, w and h represent the width and height of the prediction box; Step 2.5: Train the model; bring the prepared data set into the optimized YOLOv5 network for training, continuously iterate the model through parameter adjustment to minimize the loss function of the model; finally generate a crack detection model; Step 2.6: Detect the crack area; call the crack detection model, input the image to be detected into the model, and output the crack category and bounding box through the end-to-end model; In step 3, when performing precise crack identification based on visual saliency, the following steps are specifically adopted: Step 3.1: Calculate the image saliency map; intercept the crack area image through the crack boundary identified by the crack detection model, perform Gaussian filtering and denoising on the image, and then convert the image color space from RGB to Lab; calculate the average value of L, a, and b of the entire image; according to the formula in the algorithm, calculate the Euclidean distance between the L, a, and b values ​​of each pixel and the mean value of the three L, a, and b values ​​of the image to obtain the saliency map; normalize, divide the saliency value of each pixel in the image by the largest saliency value to obtain the final saliency map; the saliency of the pixel can be calculated using the following formula S(p)=||I μ -I ωhc (p)|| Among them, I μ is the feature vector of the average image, I ωhc (p) is the Lab color feature of pixel p after Gaussian smoothing, ||.|| is the L2 norm, that is, the Euclidean distance between the previous and the next items in the Lab color space is calculated; Step 3.2: Use the Meanshift method to divide the original image into K large blocks, and use the saliency calculation method in 3.1 to calculate the average Sk of the saliency map corresponding to each segmented block; Step 3.3: Calculate the average value Sμ of the entire saliency map; Step 3.4: Calculate the adaptive threshold Ta = 2*Sμ; Step 3.5: When Sk>Ta, the region is considered to be a saliency map; Step 3.6: Binarize and segment the salient area to obtain a refined crack segmentation image.

2. The method according to claim 1, characterized in that In step 2, a pre-trained convolutional neural network model is used to detect whether there are cracks in the image. If no cracks are detected, there is no need to perform the next step of refined detection, thereby improving the efficiency of the algorithm; if cracks are detected, the model outputs the crack boundary box in the image, thereby filtering out complex background interference information and improving the accuracy of subsequent crack edge extraction and crack measurement.

3. The method according to claim 1, characterized in that In step 3, based on the difference in color and brightness between the cracks in the image and the image background, the FT saliency detection algorithm is used to obtain the saliency map of the image, and then the mean shift algorithm is used to adaptively calculate the image threshold, and then the cracks are segmented by binarization.

4. The method according to claim 1, characterized in that In step 4, the Canny edge detection algorithm is used to extract crack edge pixels, the image is smoothed with a Gaussian filter, the finite difference of the first-order partial derivative is used to calculate the amplitude and direction of the gradient, the gradient amplitude is non-maximum suppressed, and the double threshold algorithm is used to detect and connect the edges.

5. The method according to claim 1, characterized in that In step 5, the camera is calibrated first to calculate the internal and external parameters of the acquisition camera, which are used to calculate the conversion between the image coordinate system and the world coordinate system. Then, the width is calculated using the shortest distance method based on the crack edge pixels. The Euclidean distance between the starting point and the end point of the crack is calculated as the crack length. The angle of the crack fitting line is taken as the angle of the crack.

6. The method according to claim 1, characterized in that In step 7: locate and evaluate the cracks; locate the position of the cracks relative to the dam body, so that the operation and maintenance personnel can accurately determine the location information of the cracks; determine the hazard level of the cracks through the length, width, angle and area information of the cracks in step 5; and predict the development trend of the cracks, and do a good job in dam body hazard work in time to prevent them from happening.

7. The method according to claim 1 or 4, characterized in that: In step 4, the Canny algorithm is used to detect the crack edge, which specifically includes the following steps: 4.

1. Eliminate noise; use Gaussian smoothing filter convolution to reduce noise and eliminate noise points around the cracks; the Gaussian kernel used in Gaussian filtering is the product of two one-dimensional Gaussians x and y. The standard deviation σ in the two dimensions is usually the same, in the following form: Use Gaussian smoothing filter convolution to reduce noise; 4.

2. Calculate the gradient magnitude and direction. Follow the steps of Sobel filtering. First, apply a pair of convolution arrays in the x and y directions respectively, that is, Then calculate the gradient magnitude and direction 4.3, non-maximum suppression; 4.

4. Hysteresis threshold, i.e., hysteresis high threshold and low threshold; specifically, if the amplitude of the pixel position exceeds the high threshold, the pixel is retained as an edge pixel; if the amplitude of the pixel position is less than the low threshold, the pixel is excluded; if the amplitude of the pixel position is between the two thresholds, the pixel is retained only when it is connected to a pixel above the high threshold.

8. The method according to claim 1 or 5, characterized in that: In step 5, when calculating the crack characteristic information, including measuring the length and width of the crack, the following steps are specifically included: 5.

1. Camera calibration to obtain internal and external parameters. After completing the crack image recognition, for the image containing the crack, calculate the edge coordinates of the crack at each point in the image, and calculate the actual size of the crack based on the coordinate transformation relationship. The Zhang Zhengyou calibration method can directly obtain the internal and external parameters of the camera; as shown in the following formula Among them, A is the camera's intrinsic matrix, R is the extrinsic rotation matrix, T is the translation vector, α = f / dx, β = f / dy, f is the focal length, dx and dy are the width and height of the pixel respectively; γ represents the deviation of the pixel point in the x and y directions, and (u0, v0) is the reference point; 5.

2. Conversion from pixel coordinates to world coordinates; the relationship between the pixel coordinate system and the world coordinate system is shown in the following matrix: Where [XYZ] T Represents the world coordinate system, [uv] T Represents the pixel coordinate system, the coordinate is located in the upper left corner of the image, Z C is the corresponding Z-axis coordinate value in the camera coordinate system; 5.

3. Calculation of crack width: Using the shortest distance method, take the vertical crack as an example, after the crack image is marginalized, take the left edge point of each row as the center; find the minimum value of the distance between the right edge points in several adjacent rows as the crack width of the row; the Euclidean distance calculation method of the crack width is as follows: Where w is the crack width, Z is the width of the crack under the monocular camera. l =Z r =0; the maximum value of the crack w max =max(w i ), the average value of the crack is w i is the width of the i-th crack row, and N is the total number of rows; 5.

4. Length calculation: The skeleton information obtained after the thinning operation is a thin line graphic with a single pixel arrangement. The principle of calculating the length information of the dam body crack can be to calculate the Euclidean distance between the starting point and the end point of the thinned crack as the length information of the dam body crack; 5.

5. Angle calculation: Since the crack pattern of the dam body has an irregular shape, the crack image needs to be fitted into a straight line that can be calculated during the processing; 5.

6. Area calculation: The area of ​​the crack is the smallest circumscribed rectangular area of ​​the crack.

Images