A method and system for detecting cracks on building facades based on drones

Abstract

The present invention provides a method and system for detecting cracks on the facade of a building based on an unmanned aerial vehicle, which construct a crack image training data set and a verification data set, use an unmanned aerial vehicle to obtain crack images of a building to be detected and distance information between the unmanned aerial vehicle and the building to be detected; construct a convolutional neural network based on multi-feature regional attention, specifically including a regional channel attention network, a feature extraction network and a feature recovery network; perform loss calculation, use the training data set to train the convolutional neural network based on multi-feature regional attention, and obtain a semantic segmentation model based on multi-feature regional attention; input the verification data set or the crack image obtained by using the unmanned aerial vehicle into the semantic segmentation model based on multi-feature regional attention, and obtain a crack binary map; process the crack binary map and calculate the actual length of the crack. The present invention integrates image acquisition, crack detection and crack measurement, and has the advantages of high efficiency and automatic detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of building health detection, and specifically belongs to a method and system for detecting cracks on a building facade based on an unmanned aerial vehicle. Background Art

[0002] The service life of civil infrastructure, such as bridges, roads, and buildings, usually exceeds their design life. In addition, due to the coupling of multiple factors such as load, environmental erosion, and sudden disasters during the construction and use of concrete structures, the exterior walls of some buildings have experienced varying degrees of cracks, hollows, and insufficient adhesion. Especially under extreme wind and rain conditions, problems such as falling off and falling have occurred. There have been many cases of exterior walls and tiles falling off, injuring pedestrians and damaging property. Therefore, efficient testing methods are needed to inspect exterior walls, especially non-destructive testing, to provide an evaluation basis for prevention and maintenance.

[0003] The existing building health detection methods mainly include: manual detection method, sensor detection method, image processing detection method and traditional machine learning detection method. Manual detection method requires a lot of manpower and material resources. Sensor detection method is costly and requires professional detection personnel. Image processing detection method has the problem of insufficient detection robustness. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a method and system for detecting cracks on the facade of a building based on a drone, which integrates image acquisition, surface crack detection on the facade of a building and measurement of crack geometric parameters, and has the advantages of high efficiency and automated detection.

[0005] To achieve the above purpose, the present invention provides the following technical solution: a method for detecting cracks on the facade of a building based on a drone, the specific steps are as follows:

[0006] S1 builds a crack image training dataset and a validation dataset, uses drones to detect the structure of the building to be detected and builds a model, determines the location of the cracks in the building and obtains crack images, and obtains the distance information between the drone and the building to be detected;

[0007] S2 builds a convolutional neural network based on multi-feature regional attention, which includes a regional channel attention network, a feature extraction network, and a feature recovery network;

[0008] S3 establishes a loss function, performs loss calculation, and uses the training data set to train the convolutional neural network based on multi-feature region attention to obtain a semantic segmentation model based on multi-feature region attention;

[0009] S4 inputs the verification data set or the crack image of the building to be inspected obtained by using a drone into the semantic segmentation model based on multi-feature regional attention to obtain a crack binary map;

[0010] S5 processes the crack binary image to obtain a crack skeleton image, obtains the number of pixel points of the crack skeleton image by length calculation, obtains the calibration pixel ratio relationship according to the distance information between the UAV and the building to be inspected, and obtains the actual length of the crack.

[0011] Furthermore, in step S2, the feature extraction network is used to extract crack feature information of different scales in the crack image to obtain original feature maps of cracks of different scales, specifically:

[0012] The feature extraction network has five layers in total. Each of the first four layers contains two 3×3 convolutions. Each 3×3 convolution is connected to a ReLu nonlinear activation. The layers are connected through a 2×2 maximum pooling. The first four layers in the feature extraction network output two low-level original feature maps L1 and L2, and two high-level original feature maps H3 and H4; the last layer outputs a crack high-level feature map H5.

[0013] Furthermore, in step S2, the feature recovery network is used to recover the edge information of the original feature map obtained by each layer of the feature extraction network layer by layer, specifically:

[0014] The feature recovery network has four layers, each of which contains two 3×3 convolutions, each of which is connected to a ReLu nonlinear activation and two 2×2 upsampling layers, one of which is connected to a region-level channel attention network.

[0015] In the first layer of the feature recovery network, the crack high-level feature map H5 is upsampled to obtain two crack recovery feature maps of the same scale, namely, the crack recovery feature map RH41 and the crack recovery feature map RH42;

[0016] The crack recovery feature map RH41 is input into the region-level channel attention network to obtain a crack feature map with enhanced region-level information;

[0017] The crack feature map with enhanced regional level information is spliced ​​with the crack recovery feature map RH42 to provide more semantic information for the feature recovery process, thereby obtaining the crack recovery feature map RH43;

[0018] The above operation is repeated three times to obtain the crack recovery feature map RH1. The crack recovery feature map RH1 is passed through two 3×3 convolutions, one 1×1 convolution and a softmax classifier to output a crack binary map.

[0019] Furthermore, in step S2, the region-level channel attention network is divided into two parts: a squeezing operation and an excitation operation, wherein:

[0020] The squeezing operation is used to generate a region-level channel descriptor by focusing on the feature map of the crack restoration feature map region spatial dimension. The descriptor represents the crack region-level channel feature information. Specifically, the crack restoration feature map X is obtained by formula (1) to obtain the region-level channel descriptor Y:

[0021] Y = Avgpooling(X)(1)

[0022] In the formula, It is a feature map of size C×W×H; Avgpooling represents an average pooling operation with a stride of 4 and a convolution kernel of 4;

[0023] The excitation operation takes the regional channel descriptor as input and obtains the crack feature map weight s through formula (2), which is as follows:

[0024] s=F U {σ[W 2 δ(W 1 Y)]}(2)

[0025] In the formula, W 1 , W 2 is an intermediate operation for channel number conversion. in middle is the number of channels after transformation, and C is the number of channels before transformation; In the equation, C is the number of channels after transformation, is the number of channels before transformation; γ is the dimensionality reduction rate, is the feature map size. In this paper, γ is set to 2, δ is the ReLu nonlinear activation, σ is the sigmoid nonlinear activation, and F U For upsampling;

[0026] The crack feature map weights are multiplied by the corresponding pixels of the input crack recovery feature map to obtain the crack feature map for enhanced learning. Then, the input crack recovery feature map and the crack feature map for enhanced learning are added by corresponding pixels to obtain the crack feature map P for enhanced regional level information, as follows:

[0027]

[0028] In the formula, The corresponding pixels of the feature map are multiplied, and ⊕ is the corresponding pixels of the feature map are added.

[0029] Furthermore, in step S3, the focal loss function is used to calculate the crack detection loss, specifically:

[0030]

[0031] Where W represents the image height, H represents the image width, is the label prediction probability corresponding to the position (w,h) in the image, y (w,h) is the label corresponding to the position (w, h) in the image, and a and λ are hyperparameters.

[0032] Furthermore, in step S5, the crack binary image is denoised by threshold segmentation, and then the crack binary image is processed by the K3M skeleton extraction algorithm to obtain a crack skeleton image with a width of one pixel. The number of pixels in the crack skeleton image is obtained by length calculation, and the total number of pixels is the crack skeleton length.

[0033] Furthermore, in step S5, the distance information between the drone and the building to be inspected is input into the calibration pixel ratio relationship to obtain the scale factor of the crack, and then the scale factor is multiplied by the crack skeleton length to obtain the actual length of the crack.

[0034] Furthermore, the actual length corresponding to the pixel point is fitted with the distance between the drone and the building to be inspected to obtain the scale factor of the crack. The specific formula is:

[0035] y=p 1 x+P 2 (5)

[0036] Where y is the scale factor, that is, the actual length corresponding to a single pixel in the image, and x is the distance between the drone and the building to be detected, x∈{3000,7000}.

[0037] The present invention also provides a building facade crack detection system based on a drone, which specifically includes:

[0038] An image acquisition module is used to construct a crack image training dataset and a verification dataset, and use a drone to obtain images of the building to be inspected and the distance information between the drone and the building to be inspected;

[0039] Network building module for building convolutional neural networks based on multi-feature region attention;

[0040] The model training module is used to establish a loss function, perform loss calculation, and use the crack image training data set to train the convolutional neural network based on multi-feature region attention to obtain a semantic segmentation model based on multi-feature region attention;

[0041] The retrieval module is used to input the verification data set or the image of the building to be detected obtained by using a drone into the semantic segmentation model based on multi-feature region attention to obtain a crack binary map;

[0042] The image processing module is used to perform noise reduction and skeleton extraction on the crack binary image to obtain the crack skeleton image, and obtain the number of pixels in the crack skeleton image by length calculation, and the actual length of the crack is obtained by combining the calibrated pixel ratio relationship.

[0043] Furthermore, the UAV is an unmanned aircraft controlled by a radio remote control device and a self-contained program control device. The UAV is provided with a radar scanner, a laser rangefinder and a GPS locator. The radar scanner is used to detect the structure of the building to be tested, model the building, and display the location of the cracks in the building; the laser rangefinder is used to detect the distance between the UAV and the building to be tested; and the GPS locator is used to record the location information when the UAV takes pictures of the cracks.

[0044] Compared with the prior art, the present invention has at least the following beneficial effects:

[0045] The present invention provides a method for detecting cracks on the exterior facade of a building based on an unmanned aerial vehicle. Aiming at the problems existing in the existing building health detection technology of time-consuming, labor-intensive, high-cost and certain danger in detection, the present invention realizes the acquisition of images to be detected, the acquisition of position information when recording images, and the detection of cracks on the exterior facade of a building, and the measurement of crack sizes, thereby improving the detection efficiency and reducing the detection cost, and providing a non-contact solution integrating the acquisition of images to be detected, surface crack detection and crack size measurement in the field of building health detection technology.

[0046] The implementation of the present invention can reduce the workload of inspection personnel, lower the operating threshold of inspection personnel, and achieve reduction in staff and increase in efficiency in inspection work; compared with traditional manual inspection methods, the present invention combines drone technology and computer vision technology to feed back the image information to be tested and the location information of the recorded image to the staff, thereby reducing the safety risks of high-altitude building health inspections and increasing inspection efficiency; compared with traditional drone inspection methods, the present invention evaluates the degree of building damage through building health inspections and provides recommendations for whether to take subsequent repair measures.

[0047] The invention discloses a building facade crack detection system based on drone, which detects the building to be detected by drone monitoring. In the detection process, non-destructive detection of the building facade can be achieved, and defect detection of cracks in the building facade is performed. The detection method is reliable and accurate. The detection process is performed by machines of various technical types, with computer vision technology as a scientific basis. The detection speed is fast and the detection time can be effectively reduced. At the same time, the drone has high flexibility and can adapt to the detection of various walls. The operation difficulty is low during use, which is convenient for staff to operate. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of a method for detecting cracks on a building facade based on a drone in the present invention;

[0049] Figure 2 It is a structural diagram of the semantic segmentation model based on multi-feature region attention in the present invention;

[0050] Figure 3 It is a structural diagram of the feature extraction network in the present invention;

[0051] Figure 4 It is a structural diagram of the feature recovery network in the present invention;

[0052] Figure 5 It is a structural diagram of the regional channel attention network in the present invention;

[0053] Figure 6 These are some experimental results of the present invention on the Bridge_Crack_Image_Data dataset. DETAILED DESCRIPTION

[0054] To make the technical solution of the present invention clearer, the present invention will be further described below in conjunction with the accompanying drawings in the embodiments of the present invention. It should be noted that the embodiments described below are only part of the embodiments of the present invention, rather than all of the embodiments.

[0055] The present invention provides a method for detecting cracks on a building facade based on an unmanned aerial vehicle, and the specific steps are as follows:

[0056] 1) Download the Bridge_Crack_Image_Data dataset, preprocess the images to obtain the dataset, and divide the dataset into a crack image training dataset and a verification dataset in a ratio of 9:1.

[0057] Preferably, the preprocessing weakens the color and brightness of the image to reduce the influence of external factors on the final analysis results; the data set is expanded through operations such as cropping, horizontal flipping, vertical flipping, and rotation to enhance the generalization ability of the semantic segmentation model.

[0058] (2) Construct a convolutional neural network based on multi-feature regional attention, which specifically includes a regional channel attention network, a feature extraction network, and a feature recovery network. The feature extraction network is used to extract multi-scale features of the input crack image. The crack feature maps of different scales are obtained after two 3×3 convolutions of the feature extraction network to obtain the original feature map, and then the original feature map is output to the feature recovery network for feature fusion; the feature recovery network is used to use the regional channel attention network to enhance the crack features of the original feature map in the upsampling stage and suppress non-crack features to obtain a crack feature map with enhanced regional level information. The crack feature map with enhanced regional level information is fused with the original feature map extracted by the feature extraction network to achieve multi-scale feature recovery of the original feature map, and output a crack binary map.

[0059] 1) Construct feature extraction network

[0060] The feature extraction network is used to extract the crack feature information of different scales in the crack image and obtain the original feature maps of cracks of different scales.

[0061] The feature extraction network has five layers, among which each of the first four layers contains two 3×3 convolutions, each of which is connected to a ReLu nonlinear activation, and the layers are connected by a 2×2 maximum pooling. The first four layers in the feature extraction network output two low-level original feature maps L1 and L2, and two high-level original feature maps H3 and H4. The low-level original feature maps pay more attention to the semantic information of the cracks, and the high-level original feature maps pay more attention to the edge information of the cracks. The low-level original feature maps L1 and L2 and the high-level original feature maps H3 and H4 are transmitted to the regional channel attention network of the feature recovery network through jump connections; the last layer of the feature extraction network outputs the high-level feature map H5 of the cracks;

[0062] 2) Constructing feature recovery network

[0063] The feature recovery network is used to recover the edge information of the original feature map obtained by each layer of the feature extraction network layer by layer. Specifically, the feature recovery network has four layers, each of which contains two 3×3 convolutions. Each 3×3 convolution is connected to a ReLu nonlinear activation and two 2×2 upsampling layers, one of which is connected to a regional channel attention network.

[0064] The feature recovery network upsamples the crack high-level feature map H5 to obtain two crack recovery feature maps of the same scale, namely, crack recovery feature map RH41 and crack recovery feature map RH42. The crack recovery feature map RH41 is input into the regional channel attention network to obtain a crack feature map with enhanced regional information. The crack feature map with enhanced regional information is then concatenated with the crack recovery feature map RH42 to provide more semantic information for the feature recovery process, and the crack recovery feature map RH43 is obtained. The above operation is repeated three times to obtain the recovery feature map RH1. Finally, the recovery feature map RH1 is subjected to two 3×3 convolutions, one 1×1 convolution and a softmax classifier, and a crack binary map with white as the target and black as the background is output (the white area represents the crack part, and the black area represents the complete part).

[0065] 3) The regional channel attention network is divided into two parts: squeezing operation and excitation operation

[0066] The squeezing operation generates a region-level channel descriptor by focusing on the feature map of the crack restoration feature map region spatial dimension. The descriptor represents the crack region-level channel feature information. Specifically, the input crack restoration feature map is subjected to average pooling with a step size of 4 and a convolution kernel of 4. Specifically, the 4×4 pixel area in the crack feature map is summed and then the average value is taken. Finally, the region-level channel descriptor is output. Assuming that the input crack restoration feature map is X, the region-level channel descriptor Y is obtained by formula (1). Specifically:

[0067] Y = Avgpooling(X)(1)

[0068] In the formula, It is a feature map of size C×W×H; Avgpooling represents an average pooling operation with a stride of 4 and a convolution kernel of 4.

[0069] The specific incentive operations are:

[0070] Firstly, the regional channel descriptor is used as input, and two 1×1 convolutions are performed to increase the interaction between different feature channels, improve the connectivity of crack semantic information and spatial information, enhance the sensitivity of crack features to regional information, and obtain the stimulated crack feature map.

[0071] Then, the stimulated crack recovery feature map is upsampled to restore the size of the feature map, and the crack feature map weight s is obtained. The crack feature map weight reflects the relationship between the crack feature and the background feature, as shown in formula (2):

[0072] s=F U {σ[W 2 δ(W 1 Y)]}(2)

[0073] In the formula, W 1 , W 2 is an intermediate operation for channel number conversion. in middle is the number of channels after transformation, and C is the number of channels before transformation; In the equation, C is the number of channels after transformation, is the number of channels before transformation; γ is the dimensionality reduction rate, is the feature map size. In this paper, γ is set to 2, δ is the ReLu nonlinear activation, σ is the sigmoid nonlinear activation, and F U For upsampling.

[0074] The crack feature map weight is multiplied by the corresponding pixel of the input crack recovery feature map to obtain the crack feature map of enhanced learning. Then, the input crack recovery feature map and the crack feature map of enhanced learning are added by corresponding pixels to obtain the crack feature map with enhanced regional level information: The details are as follows:

[0075]

[0076] In the formula, The corresponding pixels of the feature map are multiplied, and ⊕ is the corresponding pixels of the feature map are added.

[0077] (4) Loss calculation: The crack detection loss is calculated based on the crack binary image and the corresponding true value image. Since the distribution of cracks and background in the crack image is extremely unbalanced, that is, the proportion of cracks is much smaller than that of the background, the Focal loss function is used to calculate the loss to solve this problem. The formula is as follows:

[0078]

[0079] Where W represents the image height, H represents the image width, is the predicted probability of the label corresponding to (w,h) in the image, The closer it is to 1, the higher the prediction accuracy and the smaller the loss. (w,h) is the label corresponding to the position (w, h) in the image, a and λ are hyperparameters. The experiment found that the best effect was achieved when λ=2 and a=0.25.

[0080] (5) Use the training data set to train the convolutional neural network based on multi-feature region attention, set the epoch to 60, use the stochastic gradient descent method with restart, the initial learning rate is 0.001258, and the minimum learning rate is set to 10 -7After 15 epochs, the learning rate decays rapidly to 0, and then rebounds rapidly to the initial value in the next epoch. The semantic segmentation model based on multi-feature regional attention is obtained by repeatedly adjusting parameters.

[0081] (6) Using drones to detect the structure of the building to be inspected, modeling the building, determining the location of the cracks in the building and obtaining crack images, and obtaining the distance information between the drone and the building to be inspected;

[0082] The crack image is input into the semantic segmentation model based on multi-feature region attention to obtain a crack binary map.

[0083] (7) The crack binary image is denoised through threshold segmentation to remove noise, enhance the crack feature expression, and output the denoised crack binary image.

[0084] (8) The crack binary image after denoising is subjected to skeleton extraction to obtain a crack skeleton image. Specifically:

[0085] Using the K3M skeleton extraction algorithm, the crack binary image is continuously refined to extract the pseudo skeleton, but some areas of the pseudo skeleton have multiple pixel widths. The pseudo skeleton is further optimized and refined into a real skeleton with a pixel width to obtain the crack skeleton image. The specific steps are as follows:

[0086] 1) Extract target contour points and mark them;

[0087] 2) Check the 8-pixel neighborhood of the target contour point in turn. If it contains only 3 adjacent foreground pixels, delete this point from the contour point;

[0088] 3) Check the 8-pixel neighborhood of the remaining contour points in 2) one by one. If there are only 3 or 4 adjacent foreground pixels, delete this point from the contour points.

[0089] 4) Check the 8-pixel neighborhood of the remaining contour points in 3) in turn. If it contains only 3, 4 or 5 adjacent foreground pixels, delete this point from the contour points;

[0090] 5) Check the 8-pixel neighborhood of the remaining contour points in 4) in turn. If it contains only 3, 4, 5 or 6 adjacent foreground pixels, delete this point from the contour points;

[0091] 6) Check the 8-pixel neighborhood of the remaining contour points in 5) in turn. If it contains only 3, 4, 5, 6 or 7 adjacent foreground pixels, delete this point from the contour points;

[0092] 7) Cancel the remaining marked contour points. If no pixel is deleted in step 6), stop the iteration, otherwise return to 1);

[0093] 8) After iterative refinement, some areas of the skeleton are still multiple pixels wide, while the target skeleton is a single pixel wide. Therefore, the 8-pixel neighborhood of the pseudo-skeleton is checked in turn. If it contains only 2, 3, 4, 5, 6 or 7 adjacent foreground pixels, this point is deleted from the pseudo-skeleton, and finally a real skeleton with a width of one pixel is obtained.

[0094] (9) The number of pixels in the crack skeleton image is obtained by length calculation, and the length of the crack skeleton is obtained according to the number of pixels.

[0095] Furthermore, the crack skeleton graph is traversed to obtain the total number of pixels, which is the crack skeleton length.

[0096] (10) The distance information between the drone and the building to be inspected is input into the calibration pixel ratio relationship to obtain the scale factor of the crack, and then the scale factor is multiplied by the crack skeleton length to obtain the actual length of the crack. At the same time, GPS positioning sends the position relationship during shooting back to the control system.

[0097] The present invention also provides a building facade crack detection system based on a drone, which specifically includes:

[0098] An image acquisition module is used to construct a crack image training dataset and a verification dataset, and use a drone to obtain images of the building to be inspected and the distance information between the drone and the building to be inspected;

[0099] Network building module for building convolutional neural networks based on multi-feature region attention;

[0100] The model training module is used to establish a loss function, perform loss calculation, and use the crack image training data set to train the convolutional neural network based on multi-feature region attention to obtain a semantic segmentation model based on multi-feature region attention;

[0101] The retrieval module is used to input the verification data set or the image of the building to be detected obtained by using a drone into the semantic segmentation model based on multi-feature region attention to obtain a crack binary map;

[0102] The image processing module is used to perform noise reduction and skeleton extraction on the crack binary image to obtain the crack skeleton image, and obtain the number of pixels in the crack skeleton image by length calculation, and the actual length of the crack is obtained by combining the calibrated pixel ratio relationship.

[0103] Preferably, the UAV is an unmanned aircraft controlled by a radio remote control device and a self-contained program control device. It relies on the equipment installed on the aircraft to complete various inspection and monitoring tasks, works according to a predetermined program or ground instructions, and finally transmits the obtained information and images back to the ground in real time. It can also record all the information obtained and retrieve it once when the UAV is recovered.

[0104] Preferably, the drone is provided with a radar scanner, a laser rangefinder and a GPS locator. The radar scanner is used to detect the structure of the building to be tested, model the building, and display the location of the cracks in the building; the laser rangefinder is used to detect the distance between the drone and the building to be tested and obtain the scale factor; the GPS locator is used to record the location information when the drone takes pictures of the cracks.

[0105] Furthermore, the signal processing can input the distance measured by the rangefinder into the calibration ratio relationship to obtain the scale factor, which is used as the actual length of a single pixel in the image to provide a basis for subsequent crack length measurement.

[0106] Furthermore, the actual length (mm) corresponding to each pixel point is fitted with the distance (mm) between the drone and the building to be inspected to obtain the scale factor of the crack, as follows:

[0107] Determine the building to be detected, control the drone to take pictures of the building to be detected, and the building to be detected is always in the center of the camera image on the drone. The drone starts from the initial position 3000mm away from the building to be detected and moves in steps of 500mm each time to the maximum position of 7000mm away from the building to be detected. Then the drone returns to the initial position to complete a cycle. At the same time, the distance between the laser rangefinder and the building to be detected is recorded during the movement. Repeat the experiment three times and take the average value; finally, perform linear fitting on the experimental results to obtain the scale factor, which is expressed as:

[0108] y=p 1 x+p 2 (6)

[0109] Where y is the scale factor, i.e., the actual length of a single pixel in the image (mm), x is the distance between the drone and the building to be detected (mm), x∈{3000,7000}, p 2 =0.000428, p 2 =-0.8567.

[0110] Example 1

[0111] like Figure 1 As shown, the present invention is a method for detecting cracks on the facade of a building based on a drone. First, a semantic segmentation model based on multi-feature regional attention is constructed and trained, and then the crack image is input into the trained semantic segmentation model based on multi-feature regional attention to output the crack detection result. Then, the crack detection result is subjected to threshold segmentation, skeleton extraction, and length calculation to obtain the crack pixel length. Finally, the real size of the crack is obtained by combining the scale factor. The specific steps are as follows:

[0112] 1. First, download the Bridge_Crack_Image_Data dataset and perform data screening on its images to ensure the integrity of the data images. Then, preprocess the dataset by weakening the color and brightness of the 2,000 crack images to reduce the influence of external factors on the final analysis results. Then, use operations such as cropping, horizontal flipping, vertical flipping, and rotation to expand it to 28,818 256×256 resolution datasets to enhance the robustness of the CNN model. Finally, divide the dataset into a training dataset and a validation dataset in a ratio of 9:1.

[0113] 2. If Figure 2 As shown in the figure, a convolutional neural network based on multi-feature regional attention is constructed and trained using a training data set to obtain a semantic segmentation model based on multi-feature regional attention, including: regional channel attention network, feature extraction network and feature recovery network. The specific steps are as follows:

[0114] (1) Figure 3 As shown in the figure, a feature extraction network is constructed. The feature extraction network extracts image features layer by layer. Its structure is divided into five layers. The feature extraction network has five layers in total. Each of the first four layers contains two 3×3 convolutions, each of which is connected to a ReLu nonlinear activation. The layers are connected by a 2×2 maximum pooling. The first four layers in the feature extraction network output two low-level original feature maps L1 and L2 and two high-level original feature maps H3 and H4. The low-level original feature maps pay more attention to the semantic information of the cracks, and the high-level original feature maps pay more attention to the edge information of the cracks. The low-level original feature maps L1 and L2 and the high-level original feature maps H3 and H4 are transmitted to the regional channel attention network of the feature recovery network through jump connections; the last layer of the feature extraction network outputs the high-level feature map H5 of the crack.

[0115] (2) Figure 4 As shown in the figure, a feature recovery network is constructed. The feature recovery network is used to recover the edge information of the original feature map obtained by each layer of the feature extraction network layer by layer. Specifically, the feature recovery network has four layers, each of which contains two 3×3 convolutions, each of which is connected to a ReLu nonlinear activation, a region-level channel attention network, and two 2×2 upsampling layers.

[0116] First, the feature recovery network upsamples the crack high-level feature map H5 to obtain two crack recovery feature maps of the same scale, namely, crack recovery feature map RH41 and crack recovery feature map RH42. The crack recovery feature map RH41 is input into the regional-level channel attention network to obtain a crack feature map with enhanced regional-level information. Then, the crack feature map with enhanced regional-level information is concatenated with the crack recovery feature map RH42 to provide more semantic information for the feature recovery process, and a crack recovery feature map RH43 is obtained.

[0117] Secondly, the crack restoration feature map RH43 is upsampled to obtain crack restoration feature maps RH31 and RH32, where the crack restoration feature map RH31 is input into the regional-level channel attention network to obtain a crack feature map with enhanced regional-level information, and then the crack feature map with enhanced regional-level information is concatenated with the crack restoration feature map RH32 to obtain the crack restoration feature map RH33.

[0118] Then, the crack restoration feature map RH33 is upsampled to obtain crack restoration feature maps RH21 and RH22, wherein the crack restoration feature map RH21 is input into the regional-level channel attention network to obtain a crack feature map with enhanced regional-level information, and then the crack feature map with enhanced regional-level information is concatenated with the crack restoration feature map RH22 to obtain the crack restoration feature map RH23.

[0119] Next, the crack recovery feature map RH23 is upsampled to obtain crack recovery feature maps RH11 and RH12, wherein the crack recovery feature map RH11 is input into the regional-level channel attention network to obtain a crack feature map with enhanced regional-level information, and then the crack feature map with enhanced regional-level information is concatenated with the crack recovery feature map RH12 to obtain the crack recovery feature map RH1.

[0120] Finally, the input crack recovery feature map RH1 is input into the softmax classifier, and a crack binary map with white as the target and black as the background is output.

[0121] (3) Figure 5 As shown in FIG. 1 , a regional channel attention network is constructed. The regional channel attention network provided by the present invention is used to enhance the weight of crack features and reduce the weight of background features. Its structure is divided into pooling, convolution, activation function, upsampling, and residual connection, as shown in FIG. Figure 5 The structural diagram of the regional channel attention network is shown in Figure 2. The regional channel attention network is divided into two parts: the squeeze operation and the excitation operation:

[0122] The squeezing operation generates a region-level channel descriptor by focusing on the feature map in the spatial dimension of the crack restoration feature map region. The descriptor represents the crack region-level channel feature information. Specifically, the input crack restoration feature map is averaged with a step size of 4 and a convolution kernel of 4. Specifically, the 4×4 pixel area in the crack feature map is summed and then the average is taken. Finally, the region-level channel descriptor is output. Assume that the input crack restoration feature map is The region-level channel descriptor is obtained by formula (1):

[0123] Y = Avgpooling(X)(1)

[0124] In the formula, Avgpooling represents the average pooling operation with a step size of 4 and a convolution kernel of 4. The excitation operation extracts the degree of learning of the crack feature by the regional-level channel of the crack feature map, and obtains the crack feature map with enhanced regional-level information:

[0125] The excitation operation takes the regional channel descriptor as input, and the crack feature map weight is obtained by formula (2): The details are as follows:

[0126] s=F U {σ[W 2 δ(W 1 Y)]}(2)

[0127] In the formula, C is the number of input channels, γ is the dimensionality reduction rate, in this paper, γ is set to 2, δ is the ReLu nonlinear activation, σ is the sigmoid nonlinear activation, F U The operation of reducing the dimension of the feature map first and then increasing the dimension helps to improve the continuity of crack semantic information and spatial information and enhance the sensitivity of crack features to regional information.

[0128] The crack feature map weight is multiplied by the corresponding pixels of the input crack recovery feature map to obtain the crack feature map of enhanced learning. Then, the input crack recovery feature map and the crack feature map of enhanced learning are added by corresponding pixels to obtain the crack feature map with enhanced regional level information: The details are as follows:

[0129]

[0130] In the formula, The corresponding pixels of the feature map are multiplied, and ⊕ is the corresponding pixels of the feature map are added.

[0131] (4) Loss calculation: The crack detection loss is calculated based on the crack prediction results and the corresponding true value map. Since the distribution of cracks and background in the crack image is extremely unbalanced, that is, the proportion of cracks is much smaller than that of the background, the focal loss function is used to calculate the loss to solve this problem. The formula is as follows:

[0132]

[0133] Where W represents the image height, H represents the image width, is the predicted probability of the label corresponding to (w,h) in the image, The closer it is to 1, the higher the prediction accuracy and the smaller the loss. (w,h) is the label corresponding to the position (w, h) in the image, a and λ are hyperparameters. The experiment found that the best effect was achieved when λ=2 and a=0.25.

[0134] (5) The convolutional neural network based on multi-feature region attention is trained using the training data set. The epoch is set to 60, and the stochastic gradient descent method with restart is used for training. The initial learning rate is 0.001258, and the minimum learning rate is set to 10 -7 After 15 epochs, the learning rate decays rapidly to 0, and then rebounds rapidly to the initial value in the next epoch. The network accuracy is adjusted repeatedly to the optimal value, and a semantic segmentation model based on multi-feature region attention is obtained.

[0135] (6) Using drones to detect the structure of the building to be inspected, modeling the building, determining the location of the cracks in the building and obtaining crack images, and obtaining the distance information between the drone and the building to be inspected;

[0136] The crack image is input into the semantic segmentation model based on multi-feature region attention to obtain a crack binary map.

[0137] (7) The crack binary image is denoised by threshold segmentation to remove noise and enhance the crack feature expression, thus obtaining a denoised crack binary image.

[0138] (8) Perform skeleton extraction on the crack binary image after denoising to obtain a crack skeleton image.

[0139] (9) The crack skeleton length is obtained by length calculation.

[0140] (10) The distance measured by the laser rangefinder is combined with the calibrated pixel ratio to obtain the scale factor.

[0141] (11) The crack skeleton length is multiplied by the scale factor to obtain the actual length of the crack.

[0142] The length of the crack is a key indicator for assessing the degree of damage to a building. The present invention proposes a facade crack detection method based on a drone, which detects and measures cracks on the surface of the building in a non-contact manner, reducing the dependence on inspectors and inspection equipment during manual inspection, realizing the automation and intelligence of building health inspection, ensuring the stability and accuracy of the inspection results, and providing a promising solution for the field of building health inspection.

[0143] Figure 6 This is a demonstration of some experimental results of the present invention on the Bridge_Crack_Image_Data dataset. Figure 6The first column is the crack image, the second column is the truth image, and the third to seventh columns are FCN, Deeplabv3, SegNet, PSPNet, Unet and semantic segmentation models of multi-feature regional attention. By comparing the results and label images, it can be seen that all networks can accurately identify some regular and clear-featured crack images, while for some complex cracks, the segmentation results of FCN, DeeplabV3 and SegNet are poor, and the prediction results of crack pixels in the test image are discontinuous, and the crack features cannot be accurately identified, and there are omissions. The segmentation performance of PSPNet and Unet is good, but the detection effect in small crack images is not good, and there are false detections and omissions. Compared with other models in the comparative experiment, the semantic segmentation model based on multi-feature regional attention of the present invention effectively solves the interference of crack detection results caused by complex background of cracks, small difference between background and foreground, linear and slender cracks and easy fracture, and can accurately segment the concrete crack area.

[0144] Table 1 Quantitative analysis comparison of Bridge_Crack_Image_Data dataset

[0145]

[0146] Table 1 is a quantitative analysis of the detection results of the present invention on the Bridge_Crack_Image_Data dataset. The semantic segmentation model based on multi-feature regional attention proposed in the present invention is 12%, 6.72%, 6.4%, 2.92%, and 3.51% higher in accuracy than FCN, DeeplabV3, SegNet, PSPNet, and Unet, respectively; and 10.77%, 4.21%, 3.99%, 2.72%, and 1.51% higher in recall than FCN, DeeplabV3, SegNet, PSPNet, and Unet, respectively; and 11.38%, 5.45%, 5.48%, 2.82%, and 2.5% higher in F1 score than FCN, DeeplabV3, SegNet, PSPNet, and Unet, respectively, and has better recognition ability.

[0147] In order to solve the problem of discontinuity in detecting small cracks, the semantic segmentation model based on deep learning is extracted to improve the network's ability to express regional-level semantic information of cracks. The focal loss function is used to suppress the impact of uneven distribution of cracks and background.

[0148] The above embodiments are only used to illustrate the present invention, and are not used to limit the present invention. As long as they are based on the technical essence of the present invention, changes and modifications to the above embodiments will fall within the scope of the claims of the present invention.

Claims

1. A method for detecting cracks on building facades based on drones, characterized in that: The specific steps are as follows: S1 builds a crack image training dataset and a validation dataset, uses drones to detect the structure of the building to be detected and builds a model, determines the location of the cracks in the building and obtains crack images, and obtains the distance information between the drone and the building to be detected; S2 builds a convolutional neural network based on multi-feature regional attention, which includes a regional channel attention network, a feature extraction network, and a feature recovery network; S3 establishes a loss function, performs loss calculation, and uses the training data set to train the convolutional neural network based on multi-feature region attention to obtain a semantic segmentation model based on multi-feature region attention; S4 inputs the verification data set or the crack image of the building to be inspected obtained by using a drone into the semantic segmentation model based on multi-feature regional attention to obtain a crack binary map; S5 processes the crack binary image to obtain a crack skeleton image, obtains the number of pixel points of the crack skeleton image by length calculation, obtains the calibration pixel ratio relationship according to the distance information between the UAV and the building to be inspected, and obtains the actual length of the crack; In step S2, the region-level channel attention network is divided into two parts: a squeeze operation and an excitation operation, where: The squeezing operation is used to generate a region-level channel descriptor by focusing on the feature map of the crack restoration feature map region spatial dimension. The descriptor represents the crack region-level channel feature information. Specifically, the crack restoration feature map X is obtained by formula (1) to obtain the region-level channel descriptor Y: Y = Avgpooling(X) (1) In the formula, It is a feature map of size C×W×H; Avgpooling represents an average pooling operation with a stride of 4 and a convolution kernel of 4; The excitation operation takes the regional channel descriptor as input and obtains the crack feature map weight s through formula (2), which is as follows: s=F U {σ[W2δ(W1Y)]} (2) In the formula, W1 and W2 are an intermediate operation for channel number conversion. in middle is the number of channels after transformation, and C is the number of channels before transformation; In the equation, C is the number of channels after transformation, is the number of channels before transformation; γ is the dimensionality reduction rate, is the feature map size. In this paper, γ is set to 2, δ is the ReLu nonlinear activation, σ is the sigmoid nonlinear activation, and F U For upsampling; The crack feature map weights are multiplied by the corresponding pixels of the input crack recovery feature map to obtain the crack feature map for enhanced learning. Then, the input crack recovery feature map and the crack feature map for enhanced learning are added by corresponding pixels to obtain the crack feature map P for enhanced regional level information, as follows: In the formula, Multiply the corresponding pixels of the feature map, Add the corresponding pixels of the feature map.

2. A method for detecting cracks on a building facade based on an unmanned aerial vehicle according to claim 1, characterized in that: In step S2, the feature extraction network is used to extract crack feature information of different scales in the crack image to obtain original crack feature maps of different scales. Specifically: The feature extraction network has five layers in total. Each of the first four layers contains two 3×3 convolutions. Each 3×3 convolution is connected to a ReLu nonlinear activation. The layers are connected through a 2×2 maximum pooling. The first four layers in the feature extraction network output two low-level original feature maps L1 and L2, and two high-level original feature maps H3 and H4; the last layer outputs a crack high-level feature map H5.

3. A method for detecting cracks on a building facade based on an unmanned aerial vehicle according to claim 2, characterized in that: In step S2, the feature recovery network has four layers, each of which contains two 3×3 convolutions, each of which is connected to a ReLu nonlinear activation and two 2×2 upsampling layers, one of which is connected to a region-level channel attention network. The feature recovery network is used to restore the edge information of the original feature map obtained by each layer of the feature extraction network layer by layer. The specific steps are as follows: Step 1: In the first layer of the feature recovery network, the crack high-level feature map H5 is upsampled to obtain two crack recovery feature maps of the same scale, namely, the crack recovery feature map RH41 and the crack recovery feature map RH42; Step 2: Input the crack recovery feature map RH41 into the regional channel attention network to obtain a crack feature map with enhanced regional level information; Step 3: splice the crack feature map with enhanced regional level information with the crack recovery feature map RH42 to provide more semantic information for the feature recovery process, and obtain the crack recovery feature map RH43; Repeat steps 1 to 3 three times to obtain the crack recovery feature map RH1. The crack recovery feature map RH1 is passed through two 3×3 convolutions, one 1×1 convolution and a softmax classifier to output the crack binary value.

4. The method for detecting cracks on a building facade based on an unmanned aerial vehicle according to claim 1, characterized in that: In step S3, the focal loss function is used to calculate the crack detection loss, specifically: Where W represents the image height, H represents the image width, is the label prediction probability corresponding to the position (w,h) in the image, y (w,h) is the label corresponding to the position (w, h) in the image, and a and λ are hyperparameters.

5. The method for detecting cracks on a building facade based on an unmanned aerial vehicle according to claim 1, characterized in that: In step S5, the crack binary image is denoised by threshold segmentation, and then the crack binary image is processed by the K3M skeleton extraction algorithm to obtain a crack skeleton image with a pixel width. The number of pixels in the crack skeleton image is obtained by length calculation, and the total number of pixels is the crack skeleton length.

6. The method for detecting cracks on a building facade based on an unmanned aerial vehicle according to claim 1, characterized in that: In step S5, the distance information between the drone and the building to be inspected is input into the calibration pixel ratio relationship to obtain the scale factor of the crack, and then the scale factor is multiplied by the crack skeleton length to obtain the actual length of the crack.

7. A method for detecting cracks on a building facade based on an unmanned aerial vehicle according to claim 6, characterized in that: The actual length corresponding to the pixel point is fitted with the distance between the drone and the building to be inspected to obtain the scale factor of the crack. The specific formula is: y=p1x+p2 (5) Where y is the scale factor, that is, the actual length corresponding to a single pixel in the image, and x is the distance between the drone and the building to be detected, x∈{3000,7000}.

8. A building facade crack detection system based on drone, characterized in that: The steps of implementing the method for detecting cracks on the exterior facade of a building based on a drone as described in claim 1, the system specifically comprises: An image acquisition module is used to construct a crack image training dataset and a verification dataset, and use a drone to obtain images of the building to be inspected and the distance information between the drone and the building to be inspected; Network building module for building convolutional neural networks based on multi-feature region attention; The model training module is used to establish a loss function, perform loss calculation, and use the crack image training data set to train the convolutional neural network based on multi-feature region attention to obtain a semantic segmentation model based on multi-feature region attention; The retrieval module is used to input the verification data set or the image of the building to be detected obtained by using a drone into the semantic segmentation model based on multi-feature region attention to obtain a crack binary map; The image processing module is used to perform noise reduction and skeleton extraction on the crack binary image to obtain the crack skeleton image, and obtain the number of pixels in the crack skeleton image by length calculation, and the actual length of the crack is obtained by combining the calibrated pixel ratio relationship.

9. The building facade crack detection system based on drone according to claim 8, characterized in that: The UAV is an unmanned aircraft controlled by a radio remote control device and a self-contained program control device. The UAV is provided with a radar scanner, a laser rangefinder and a GPS locator. The radar scanner is used to detect the structure of the building to be tested, model the building, and display the location of the cracks in the building; the laser rangefinder is used to detect the distance between the UAV and the building to be tested; and the GPS locator is used to record the location information when the UAV takes pictures of the cracks.

Images