Unmanned aerial vehicle bridge crack beam disease identification method based on progressive domain adaptation strategy

Abstract

The invention discloses an unmanned aerial vehicle bridge crack beam disease identification method based on a progressive domain adaptation strategy. A bridge image is acquired through an unmanned aerial vehicle, the bridge image acquired by the unmanned aerial vehicle is subjected to overlapping type blocking, the segmented bridge image is marked, and bridge body image blocks are classified by adopting a CNN model; performing small-scale marking on cracks in the obtained bridge body image blocks, mixing the cracks with a public data set to obtain a mixed data set, and training a crack segmentation model based on the mixed data set; and carrying out type classification on the binary image after crack segmentation, and measuring a crack index at the same time. The bridge crack disease detection problem is decomposed into three stages of bridge body image block classification, bridge body crack segmentation, crack classification and geometric morphology estimation. The domain adaptation problem is dispersed to three different stages, so that the labeling cost is reduced, and the pressure of data acquisition is reduced. And an effective solution is provided for the bridge crack disease problem.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge health assessment, and in particular to a method for identifying cracked beam diseases of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy. Background Art

[0002] As the number of bridges increases and their service life increases, cracks and other defects are becoming more and more apparent. Cracks often indicate structural stress concentration or material aging. If not treated in time, they may continue to expand, causing damage to the structure itself and exposure and corrosion of internal steel bars, thereby reducing bridge stability and shortening the service life of the bridge. Therefore, bridge health monitoring is becoming increasingly important.

[0003] Traditional detection methods rely on manual visual inspection and simple tool measurement. The accuracy and reliability are affected by experience, and it is easy to miss or misjudge. The efficiency is low, and large-scale detection equipment is required for hard-to-reach places. This not only increases the load-bearing capacity of the bridge and blocks traffic, but also has high equipment prices and maintenance costs. In order to overcome the limitations of traditional detection, modern bridge detection has gradually introduced advanced technologies such as high-precision sensors, drones, and laser scanning. Among them, drones use high-precision sensors to efficiently inspect bridges, especially at high altitudes and dangerous areas. Real-time image data collected improves detection efficiency and safety. Combined with automated detection, it can automatically identify and analyze cracks, reduce errors, and improve the accuracy of bridge health assessment.

[0004] Crack detection technology is divided into traditional methods and deep learning-based methods. Traditional digital image processing methods are widely used in bridge inspection and mainly include two categories:

[0005] 1) Crack recognition based on edge algorithms: Edges are detected by calculating the grayscale gradient or derivative of the image. Common algorithms include Roberts, Prewitt, Sobel, Canny and Laplacian operators. Reference [1]: Dorafshan, S., Thomas, RJ and Maguire, M., 2018. Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete. Construction and Building Materials, 186, pp. 1031-1045. The deep convolutional neural network (DCNN) is compared with the common edge detection algorithm in the field of crack detection. The experiment shows that the edge detection algorithm is not as accurate as DCNN and retains a large amount of background noise. The disadvantage of the method based on edge features is that it is sensitive to image noise and lacks autonomous learning ability, resulting in poor generalization.

[0006] 2) The threshold-based method removes noise by preprocessing the image, sets the threshold by using the gray value or color intensity difference, and distinguishes the area that may contain cracks from the background. The threshold acquisition methods include: global threshold, local threshold, iterative threshold, maximum entropy threshold algorithm and Otsu method. Reference [2]: Vivekananthan, V., Vignesh, R., Vasanthaseelan, S., Joel, E. and Kumar, KS, 2023. Concrete bridge crack detection by image processing technique by using the improved OTSU method. Materials Today: Proceedings, 74, pp. 1002-1007. The image is preprocessed using the grayscale discrimination method, the threshold is set using the Otsu algorithm, and the Sobel filter is used to detect cracks at the edge of the image pixel. The accuracy of this method is as high as 95%. However, this method is limited by the aspect ratio RA and margin parameters, and its generalization ability is weak. The threshold-based method often faces the following problems: the threshold selection is difficult, and different images require different thresholds, which seriously limits the generalization ability of the algorithm. At the same time, this method is also sensitive to image noise.

[0007] Due to the above problems of traditional image processing methods, methods based on deep learning have gradually become mainstream. Deep learning can autonomously learn image features and link classification or recognition results with feature extraction to achieve better generalization capabilities. Reference [3]: Tran, TS, Nguyen, SD, Lee, HJ and Tran, VP, 2023. Advanced crack detection and segmentation on bridge decks using deep learning. Construction and Building Materials, 400, p. 132839. A deep learning method based on YOLOv7 and U-Net is proposed, and five deep learning algorithms are evaluated for crack detection. It can handle complex cracks and maintain high detection accuracy. Experiments show that the proposed method has a high accuracy rate for crack length, width and classification. However, this method requires a large amount of labeled data and computing resources.

[0008] Reference [4]: ​​Laxman, KC, Tabassum, N., Ai, L., Cole, C. and Ziehl, P., 2023. Automated crack detection and crack depth prediction for reinforced concrete structures using deep learning. Construction and Building Materials, 370, p. 130709. A comprehensive automatic crack detection and crack depth assessment framework was developed, which can detect crack location and predict depth, but its robustness is insufficient in complex environments.

[0009] Reference [5]: Gupta, P. and Dixit, M., 2022. Image-based crack detection approaches: a comprehensive survey. Multimedia Tools and Applications, 81(28), pp. 40181-40229. A bridge crack detection method based on deep learning combining "double detection + single segmentation" coarse and fine combination is proposed. On the basis of initial detection, YOLOV5(x) is used for fine crack detection, and finally the accurately detected cracks are segmented by the U-Net model. Although the accuracy and efficiency are improved, the network structure is complex, the computing resources are high, and it depends on high-quality training data.

[0010] Although deep learning-based methods can improve crack detection performance, they require large-scale data sets, high annotation costs, complex networks, and high consumption of computing resources. To achieve strong generalization capabilities, it is necessary to collect large-scale data sets of various bridges and cracks, which is not only difficult to collect but also has high annotation costs and is obviously unrealistic. Currently available data sets, especially bridge images taken by drones, are small in number and small in scale, making it difficult to support complex model training. They also focus on specific crack types and scenarios, which can easily lead to model overfitting or poor generalization capabilities.

[0011] Figure 1The comparison between the public dataset and the image blocks of the bridge crack area taken by the actual drone is shown, including: sub-image (a) and sub-image (b). It can be seen that there is a big difference between the public dataset and the bridge crack images taken by the actual drone, and they cannot cover the real bridge data distribution involved in the experiment. Therefore, although the model trained on the public dataset performs well on the corresponding test set, it fails to detect when used on other bridge datasets. This is a typical domain adaptation problem. The training data and test datasets used are only a subset of the data distribution in the real problem world. The model trained based on them cannot be applied to other subsets of the data distribution in the real problem world. However, even for a specific bridge, images cannot be collected in various weather conditions and at various angles, and the data distribution taken at different detection time periods will also be different. In this case, if only the limited dataset collected is used to train the crack disease recognition model, the generalization ability of the model cannot be guaranteed.

[0012] Since drones have great advantages over traditional detection methods, many scholars have begun to explore the integration of drones and detection algorithm modules. Reference [6]: Song, F., Liu, B. and Yuan, G., 2024. Pixel-Level Crack Identification for Bridge Concrete Structures Using Unmanned Aerial Vehicle Photography and Deep Learning. Structural Control and Health. A real-time crack recognition algorithm combining machine vision (MV) and deep learning (ResNet-18) algorithms is proposed. The results show that it still has high accuracy in complex environments. However, it is still limited by drone flight stability, image quality, lighting and algorithm generalization ability.

[0013] The patent "A UAV bridge crack detection method and system based on deep learning" (CN202310033923.6) uses UAV A to collect three-dimensional data, plans the flight path of UAV B, and detects images with an improved SSD algorithm. However, it is affected by the natural environment and model accuracy, which may lead to unstable flight process and higher cost risks.

[0014] The patent "A bridge crack detection method based on drone and Yolov5 model" (CN202310039881.7) collects bridge deck video through drone, saves and annotates crack information in frames, and uses YOLOv5 algorithm to identify cracks, but it requires massive labeled data and has insufficient generalization ability, making it difficult to adapt to different bridges and environmental conditions.

[0015] Therefore, the automatic analysis of UAV bridge crack disease images faces the problem of deep learning technology and the limitations of UAV photography itself. UAV shooting has a wide range of viewing angles, and different bridge environments vary greatly. It is challenging to directly apply deep learning methods to detect cracks. On the one hand, safety issues limit UAV close-range shooting, resulting in images containing too much irrelevant content, such as the surrounding environment of the bridge. Figure 2 On the other hand, the crack targets are relatively small and limited in number, and it is difficult to capture these small crack areas clearly in the image.

[0016] In addition, due to cost and safety issues, drones cannot take close-up photos of bridges in real scenarios. The images taken contain a lot of redundant background information, which can easily interfere with the crack detection results, thereby affecting the final crack detection accuracy. On the other hand, the results of training a data set for a single bridge in reality are often only applicable to this bridge, that is, the generalization ability is weak. If you want to better model real-world problems, you need to face high annotation costs, and there is currently no data set of this scale.

[0017] Automatic identification of bridge cracks based on drone images needs to overcome the following difficulties:

[0018] 1) Crack targets are small and limited in number. The training of a general target detection model not only requires a large-scale dataset covering various situations and high annotation costs, but also cannot guarantee high accuracy for cracks in a single bridge. Currently, there is no such public dataset.

[0019] 2) The crack target detection model trained on the small-scale bridge crack dataset collected by drones has poor domain adaptability.

[0020] 3) The estimation of crack geometry based on deep learning, such as width and direction, has poor interpretability. Traditional image processing methods cannot effectively deal with various noise interferences in crack images.

[0021] 4) Crack targets are small and limited in number. The training of a general target detection model not only requires a large-scale dataset covering various situations and high annotation costs, but also cannot guarantee high accuracy for cracks in a single bridge. Currently, there is no such public dataset.

[0022] 5) The crack target detection model trained on the small-scale bridge crack dataset collected by drones has poor domain adaptability.

[0023] 6) The estimation of crack geometry based on deep learning, such as width and direction, has poor interpretability. Traditional image processing methods cannot effectively deal with various noise interferences in crack images. Summary of the invention

[0024] In view of the above problems, the present invention adopts a progressive domain adaptation strategy, combines deep learning methods with traditional computer vision methods, and proposes a method for identifying bridge crack beam defects using drones based on a progressive domain adaptation strategy. The bridge crack disease detection problem is decomposed into three stages: bridge body image block classification, bridge body crack segmentation, crack classification, and geometric morphology estimation. By dispersing the domain adaptation problem into three different stages, it not only reduces the annotation cost and the pressure of data collection, but also ensures the overall versatility and specificity of the algorithm. It provides an effective solution to the problem of bridge crack diseases.

[0025] The technical solution adopted by the present invention is:

[0026] The method for identifying cracked beam defects of UAV bridge based on progressive domain adaptation strategy includes the following steps:

[0027] Step 1: Use drones to collect bridge images, divide the bridge images collected by drones into overlapping blocks, annotate the segmented bridge images, and use the CNN model to classify the bridge image blocks;

[0028] Step 2: After small-scale annotation of the cracks in the bridge image blocks obtained in step 1, the cracks are mixed with the public dataset to obtain a mixed dataset, and the crack segmentation model is trained based on the mixed dataset;

[0029] Step 3: Classify the binary images after crack segmentation and measure the crack indicators.

[0030] The step 1 comprises the following steps:

[0031] First, the collected bridge image dataset is divided into overlapping blocks:

[0032] The original bridge image taken by the drone is cut into 448*448 size images with 50% overlap.

[0033] Afterwards, the cut images are labeled, and the image blocks containing 90% or more of the bridge body in the image are labeled as positive samples, and their labels are 1; the image blocks containing less than 90% of the bridge body are labeled as negative samples, and their labels are 0.

[0034] Finally, VGG16 is used to classify the bridge image blocks of the obtained dataset.

[0035] The binary classification threshold used is 90%. When the binary classification probability is higher than 90%, VGG16 determines that the image result is a bridge, otherwise it is background.

[0036] In the step 2, the relevant publicly available crack image dataset is mixed with the self-collected and classified bridge body image blocks containing cracks to obtain a data set that is more conducive to the training of the crack segmentation model.

[0037] In the step 2, a method of first labeling and then segmenting is adopted: first, the cracks in the bridge image taken by the drone are labeled for semantic segmentation, and then the overlapping segmentation of the bridge image in step 1 is synchronized to segment the labeled results of the semantic segmentation.

[0038] In the step 2, the crack segmentation model adopts the Unet network.

[0039] In the step 2, the Unet network is tested using a mixed data set and a private data set, respectively, wherein the private data set is 11,812 images, and the mixed data set is a mixed data set synthesized from the private data set 11,812 images and the public crack image data set 11,298 images. The training set, validation set, and test set are divided into a ratio of 6:3:1. After the model is trained using the mixed data set and the private data set through the Unet network, the private data set is used to test it.

[0040] The private dataset refers to a dataset obtained by performing small-scale annotation on the cracks in the bridge image blocks obtained in step one.

[0041] In the step 2, the mixed data set includes a binary image obtained by bridge crack segmentation and a binary image obtained by bridge crack segmentation of a public data set.

[0042] In the step three, during classification, only two types of cracks, ring-forming and non-ring-forming, are marked.

[0043] In the step three, a lightweight MobileNetv4 classification model is used. When training the MobileNetv4 classification model, the mixed data set needs to be divided into a training set and a validation set, and the ratio of the number of training sets to the number of validation sets is 8:2. In the training set and the validation set, the ratio of the number of loops to the number of acyclic sets is 1:1.

[0044] In the step 3, when measuring the crack index, the width of the crack in a single region is divided by using the morphological operations of corrosion and expansion on the binary image after the bridge crack is segmented by the Unet network; specifically:

[0045] Through the operations of corrosion and expansion of morphogens of different sizes, the binary image after the bridge crack segmentation is processed to find the corresponding morphogen size when each area of ​​the crack is completely corroded, and the width of the crack is defined by the operator size.

[0046] After reading the binary image after the bridge crack segmentation, the original binary image a after the crack segmentation is first copied to obtain image b. Image b is used as the "switch" for algorithm iteration, that is, only when all elements in image b are eroded, all pixels in image b are black and the iteration ends. Otherwise, the size of the morphological operator is incremented and the next iteration is performed. During the iteration, the width characteristics of the crack are evaluated by applying the erosion and dilation algorithms to image b.

[0047] The erosion algorithm shrinks the edge of the image inward, while the subsequent dilation operation expands the image outward. Relative to the current morphological sub-size, for small cracks, after the operation of first erosion and then dilation, all pixels in the area will be eroded and cannot be restored by dilation. Relative to the current morphological sub-size, for wider cracks, the morphological sub-size cannot completely erode them during the erosion process, so the shape of the crack will be restored after dilation.

[0048] The morphological operator is set to a square with an initial size of 3*3. Each row and column is expanded by 1 pixel in subsequent iterations. In each iteration, the morphological operator of the current size is used to perform corrosion first, followed by dilation. If the crack image changes, that is, by comparing the current image b with the image c, if there is a significant difference between the two, the eroded crack area is obtained through image b to image c, and this part of the area is marked as a crack type that matches the current morphological sub-size. Then the morphological operator size is updated, and the image b is replaced with the current image c, and the above process is repeated until the image c is completely black, that is, all crack pixels are eroded by the corresponding morphological operator and cannot be restored by the subsequent dilation operation.

[0049] After obtaining the crack area information matching different size morphological operators, the pixel size of the crack image can be calculated according to the distance from the bridge body when the UAV is photographed.

[0050] The present invention provides a method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy, and the technical effects are as follows:

[0051] On the one hand, the invention reduces the cost of labeling large-scale crack data sets; on the other hand, it combines deep learning methods with traditional image processing methods, that is, decomposing the problem of UAV bridge crack disease identification into three components: bridge body image block classification; bridge crack disease image segmentation based on mixed data sets; and bridge crack segmentation result classification and measurement combined with deep learning and traditional image processing technology. The labeling cost is decomposed into the first, second and third stages in different proportions.

[0052] In the first stage, drones are used to collect data on the bridges to be monitored and detected, and the images taken by drones are divided into overlapping blocks. The segmented images are annotated, including 90% of the bridge body image blocks as positive samples and the others as negative samples. Such annotation costs are sharply reduced for rectangular box annotations of target detection models such as YOLO. The present invention does not limit the classification model, and any classification model that can meet the needs is applicable. The CNN model used in the present invention shows higher efficiency in classification tasks, with a simple network structure, a mature training process, and very complete optimization strategies (such as gradient descent and regularization), which can significantly reduce the training cost of the model and reduce the interference of background information on detection accuracy. Therefore, in the first stage, a CNN model is used to classify the bridge body image blocks.

[0053] In the second stage, we make full use of the public crack datasets (various scenarios), and after small-scale annotation of the cracks in the bridge image blocks obtained in the first stage, we mix them with the public datasets and train the crack segmentation model based on them.

[0054] In the third stage, all data are binary images after crack segmentation. Such images have greatly reduced the information of the image shooting environment. Therefore, the classification or regression model trained based on such images is not easily affected by environmental factors and has good generalization ability. In addition, in order to reduce the annotation cost, only the type of crack is annotated in this stage: with ring, without ring, and the size and direction of the crack are evaluated by traditional image processing methods. Because the input image is a binary image, it also has good generalization ability for traditional image processing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0056] Figure 1 A comparison chart between public datasets and our own datasets;

[0057] Figure 1 In the figure, part a shows examples of cracks in various public datasets; part b shows examples of cracks in self-built datasets.

[0058] Figure 2 Take real-life pictures for drones.

[0059] Figure 3 This is a system scheme diagram of the present invention.

[0060] Figure 4 This is a flowchart for overlapping block segmentation and bridge image block classification.

[0061] Figure 5 This is an example of drone image segmentation.

[0062] Figure 6 Schematic diagram of the VGG16 network.

[0063] Figure 7 This is the training result diagram.

[0064] Figure 8 This is the Unet network structure diagram.

[0065] Fig. 9 The following is a comparison chart of the segmentation performance of the models under different data sets.

[0066] Fig.10 Comparison of segmentation results output by models trained on mixed datasets and private datasets;

[0067] Fig.10 In the figure, part a is the result after mixed data training; part b is the result after private data training.

[0068] Fig.11 There are cyclic and acyclic graphs.

[0069] Fig.12 This is a network structure diagram.

[0070] Fig.13 This is a curve chart showing the change in model accuracy during the training process.

[0071] Fig.14 This is a curve chart of the loss rate change of the model during the training process.

[0072] Fig.15 This is the flow chart of the crack determination algorithm.

[0073] Fig.16 Examples of crack width visualization images and pie charts showing crack width ratios. DETAILED DESCRIPTION

[0074] The present invention proposes an effective solution to the problem of automatic identification of cracks in bridge images taken by drones. The key points include the following two points:

[0075] 1. The bridge crack disease detection problem is decomposed into three stages: bridge image block classification, bridge crack segmentation, crack classification and geometric morphology estimation. By dispersing the domain adaptation problem into three different stages, it not only reduces the annotation cost and the pressure of data collection, but also ensures the overall versatility and specificity of the algorithm. The specificity here means that the entire algorithm needs to train a corresponding bridge image block classification model for each bridge; the generality refers to crack classification and its geometric morphology estimation. By combining this special model with a general model, an effective solution to the bridge crack disease problem has been found.

[0076] 2. A crack geometry recognition scheme combining deep learning with a multi-scale morphological image processing algorithm with a certain degree of interpretability is proposed. In order to improve the domain adaptability of the algorithm, the results after crack segmentation are used for judgment, which greatly reduces the impact of various backgrounds, so that ring and non-ring cracks can be quickly classified through a lightweight classification network. For the width and direction characteristics of the cracks, the present invention adopts a multi-scale morphogen and uses a combination of corrosion and expansion operations to design an indirect method for estimating the width of the cracks. This method does not require data annotation and is efficient. The results are also well interpretable and meet the requirements of bridge disease detection.

[0077] A method for identifying cracks using drones based on a progressive domain adaptation strategy. The method mainly includes three key steps: overlapping block segmentation of drone bridge images and classification of bridge image blocks, segmentation of bridge crack disease images based on a mixed data set, classification of crack types, and interpretation estimation of crack size. Figure 3 As shown in the figure, during the model training process, the data sets used mainly come from the Internet and independently collected bridge disease images. This method aims to improve the detection accuracy and adaptability of UAVs to bridge cracks through a progressive domain adaptation strategy. The specific contents are as follows:

[0078] Step 1: Overlapping blocks of UAV bridge images and classification of bridge image blocks:

[0079] In response to the data collection problem, the present invention proposes a solution for separate data collection for different bridges. First, data needs to be collected during regular inspections of each bridge, so this is a routine operation. Secondly, at this stage, data annotation only involves selecting images that contain the bridge body in the image blocks. The annotation cost is very low, and the amount of data is relatively limited, so it can be completed in a short time. Finally, the deep learning model trained for each bridge can be reused in the crack disease of the bridge. For multiple inspections, as the amount of data continues to increase, the performance of the model can be continuously improved.

[0080] The overall process of this module is as follows Figure 4 As shown in Figure 1, first, the collected data set is divided into overlapping blocks. Specifically, the original bridge image taken by the drone is cut into 448*448 size by sliding cutting, so as to facilitate the mixed training of the public data set and keep 50% of the partial overlapping area. The purpose is to ensure that the bridge area that may be contained in the image will not be ignored due to the cutting process. The segmented data set is shown in Figure 1. Figure 5As shown. After that, the segmented images are labeled, and the images containing 90% or more of the bridge area are labeled as positive samples, and their labels are 1; the images containing less than 90% of the bridge area are labeled as negative samples, and their labels are 0. Since this type of labeling can be quickly distinguished by the human eye, the labeling cost is very low. For the segmented data set, the human eye can distinguish positive and negative samples very quickly. It only takes 60-80 seconds for a normal person to classify about 200 images at a time.

[0081] Finally, the obtained data set is subjected to bridge image block classification. The present invention does not limit the network model used for bridge image block classification. The present invention uses VGG16 here. The network diagram is as follows Figure 6 As shown, other deep learning models that can achieve classification accuracy are applicable. Before training the model, the present invention uses data image enhancement technology, such as adjusting hue, image saturation, image exposure, etc., so that the model can have strong generalization ability under different conditions.

[0082] The binary classification threshold used in the present invention is 90%. When the binary classification probability is higher than 90%, the model determines that the image result is a bridge, otherwise it is background. The ratio of the number of training sets to the number of validation sets is 7:3. After training with the VGG16 model, the final result of the validation set verification shows that the accuracy rate is 99.3%. Figure 7 As shown, it is proved that the algorithm adopted by the present invention has a good application space.

[0083] Step 2: Bridge crack disease image segmentation based on mixed data set:

[0084] The above-mentioned block processing decomposes the target detection and example segmentation problem of bridge crack disease in complex situations into the classification problem of bridge area blocks and the semantic segmentation problem of cracks in bridge area blocks. As for the crack segmentation problem, since the problem it focuses on is various types of cracks, the relevant public datasets can be mixed with the self-collected and classified bridge area image blocks to obtain a set that is more conducive to the training of the segmentation model, such as Figure 1 As shown, Figure 1 The neutron image (a) is a public crack image dataset. Figure 2 The bridge image (including cracks) collected by ourselves. It can be seen that due to the introduction of different public datasets in the crack image semantic segmentation process, the trained segmentation model has better domain adaptability, that is, generalization ability. This mixture effectively expands the diversity of crack images. The entire algorithm is effectively gradually transitioned from the first stage classification model that is only applicable to a certain bridge to the second stage crack segmentation model with better generalization ability.

[0085] Because the public data set of bridge crack disease has been labeled, in the semantic segmentation, only the block images with cracks in the self-collected data set need to be labeled, which greatly saves the labeling cost. In order to further improve efficiency, the present invention adopts the method of labeling first and then segmenting. Specifically, the cracks in the bridge image taken by the drone are first labeled by semantic segmentation, and then synchronized with the overlapping segmentation of the bridge image in the first stage, and the segmented labeling results are segmented, which effectively avoids repeated labeling of each block with overlapping areas.

[0086] For semantic segmentation, like the bridge block classification model, the present invention does not limit the segmentation model, and any segmentation model that can meet the needs is applicable. The present invention uses the Unet network, which is a widely used semantic segmentation model with excellent performance. Figure 8 Shown is the Unet network structure diagram.

[0087] In order to reflect the impact of the data set on the results, the present invention uses a mixed data set and a private data set to test the Unet model respectively. Among them, the private data set is 11812 images, and the mixed data set is a mixed data set synthesized by the private data set 11812 images and the public data set 11298 images. The training set, validation set, and test set are divided into training set, validation set, and test set according to the ratio of 6:3:1. After the Unet model is trained using the mixed data set and the private data set, the private data set is used to test it. The test results are shown in the figure. Fig. 9 shown.

[0088] from Fig. 9 It can be seen that after the model is trained using the mixed data set, the mIoU (Mean Intersection over Union: the average value of the intersection over union (IoU) of all categories, used to evaluate the segmentation performance of the model in each category), mPA (Mean Pixel Accurate: the average value of the pixel accuracy of all categories, indicating the accuracy of the model in each category), Precision (the proportion of samples correctly classified as positive to all samples classified as positive, used to measure the accuracy of the model), and Recall (the proportion of samples correctly classified as positive to all samples that are actually positive, used to measure the model's detection ability for positive samples) are significantly better than the parameters trained using only the private data set. Therefore, the present invention uses a mixed data set of a public data set plus a private data set to train the model. On the one hand, it can avoid overfitting caused by the private data set, and on the other hand, it can also effectively transplant detection related to any bridge. Fig.10These are the results after testing using a mixed data set and the results after testing using private data. It can be seen intuitively that after training the Unet model using a mixed data set, the model crack segmentation is more precise and can effectively improve the anti-interference ability of the model during segmentation.

[0089] Step 3: Crack type classification and crack size interpretability estimation:

[0090] The third stage requires the binary image after crack segmentation to be classified into types: with rings or without rings; at the same time, the crack index (crack width) is measured. Similarly, when classifying, only the following Fig.11 The two types of cracks, rings and no rings, are labeled, and the cost of such labeling is not high. For a single bridge, the number of cracks of the two types is not balanced and limited, so a mixed data set (second-stage crack segmentation) is needed to effectively solve this problem. The mixed data set is mainly composed of binary images obtained by existing bridge crack segmentation and binary images obtained by bridge crack segmentation of public data sets, which can improve the generalization ability of the classification model.

[0091] In the present invention, a lightweight MobileNetv4 classification model is used. MobileNetv4 is a deep convolutional neural network model in the MobileNet series, which is mainly used for tasks such as image classification on mobile devices and embedded systems. As in the first and second stages, the present invention does not impose any restrictions on the classification model here, and any classification model that meets the requirements can be used. The network structure diagram used in the present invention is as follows: Fig.12 When training a classification model, the data set needs to be divided into a training set and a validation set, and the ratio of the number of training sets to the validation set is 8:2. In the training set and the validation set, the ratio of the number of loops to the number of loops is 1:1. Fig.13 and Fig.14 They are the accuracy and loss change curves obtained by classification training of crack binary images. It can be seen from the experimental results that the model has an accuracy of 99% and 97% on the training set and validation set respectively. Therefore, the classification model trained with a mixed data set in the training set has a good ability to distinguish the two types of cracks.

[0092] When measuring crack indicators (crack width), this paper uses morphological operations of corrosion and expansion on the binary image after the bridge crack segmentation by the Unet network to divide the width of the crack in a single area. Specifically, the present invention proposes a crack width estimation method based on multi-scale morphological corrosion and expansion operations. The core idea is to process the crack binary image through corrosion and expansion operations of morphons of different sizes, find the corresponding morphon size when each crack area is completely corroded, and define the width of the crack by the operator size. The specific flow chart is as follows Fig.15 shown.

[0093] After reading the crack binary image, the present invention first copies the data of the original binary image a after crack segmentation to obtain image b. Image b serves as the "switch" for algorithm iteration, that is, only when all elements in image b are eroded, all pixels in image b are set to black and the iteration ends, otherwise the size of the morphological operator is incremented and the next iteration is performed. During the iteration, the width characteristics of the crack are evaluated by applying the erosion and expansion algorithms to image b. The erosion algorithm shrinks the edge of the image inward, while the subsequent expansion operation expands the image outward. For small cracks (relative to the current morphological sub-size), after the operation of first erosion and then expansion, all pixels in the area will be eroded and cannot be restored by expansion; for wider cracks (relative to the current morphological sub-size), the morphological sub-cannot completely erode it during the erosion process, so the shape of the crack will be restored after expansion.

[0094] Since the crack direction is random, the present invention sets the morphological operator to a square with an initial size of 3*3. Each row and column is expanded by 1 pixel in subsequent iterations. In each iteration, the morphological operator of the current size is used to perform corrosion first, followed by an expansion operation. If the crack image changes, that is, by comparing the current image b with the image c, if there is a significant difference between the two, the eroded crack area is obtained through images b to c, and this part of the area is identified as a crack type that matches the current morphological sub-size. Then the morphological operator size is updated, and the current image c is used to replace image b, and the above process is repeated until image c is completely black, that is, all crack pixels are eroded by the corresponding morphological operator and cannot be restored by the subsequent expansion operation.

[0095] The algorithm for determining the width of cracks by corrosion and expansion algorithms is as follows:

[0096]

[0097]

[0098] After obtaining the crack area information matching different size morphological operators, the ratio of the crack image pixel size to the actual crack size can be calculated based on the distance from the bridge body when the drone is photographed, thereby realizing the estimation of the physical size of the crack. However, for the present invention, only the image level is concerned, that is, the pixel size corresponding to the crack image is defined, and the conversion part with the physical size is not involved.

[0099] go through Fig.15After the processing shown, the size of the morphological operator corresponding to each crack area can be obtained. The crack width can be classified according to this size, and the specific number of levels can be determined according to the actual situation. The present invention is divided into two levels, namely wide cracks and narrow cracks. Specifically, cracks that match the morphological operator with a size less than 5*5 are classified as narrow cracks, otherwise they are wide cracks. Fig.16 As shown in the figure, red is a narrow crack and green is a wide crack. Since cracks have a transition from wide to narrow, we can make a more reasonable judgment on the crack size by counting the ratio of wide and narrow cracks, such as Fig.16 The pie chart statistics are shown on the right.

[0100] Because the third stage is aimed at binary images without background, the difficulty of the problems faced is relatively lower than that of the first and second stages. Therefore, the size of the data set for training crack type classification does not need to be particularly large, which also reduces the workload of labeling. For the estimation of crack morphology indicators, due to traditional image processing methods, no labeling is required.

[0101] This solution breaks down the problem into three stages. The first stage is the acquisition and initial classification of bridge image data. Regarding the data acquisition problem, the present invention proposes to conduct separate data collection for different bridges, which is feasible in practical applications. First, data needs to be collected during regular inspections of each bridge, so this is a routine operation. Secondly, at this stage, data annotation only involves selecting images that contain the bridge body in the image blocks. The annotation cost is very low, and the amount of data is relatively limited, so it can be completed in a short time. Finally, the deep learning model trained for each bridge can be reused in the crack disease of the bridge. For multiple inspections, as the amount of data continues to increase, the performance of the model can be continuously improved.

[0102] In the first stage, the present invention divides the collected images into overlapping blocks, and trains the model to classify the image blocks to find those image blocks that only contain the bridge area. Specifically, the image blocks in which the bridge area accounts for more than 90% of the image are the bridge area image blocks. Such classification can effectively reduce the difficulty of the subsequent crack segmentation problem, thereby reducing the requirements for the model parameter scale and the training data set scale in the subsequent segmentation stage. This stage does not limit the network model used for image block classification. The present invention uses the Vgg16 model here, and other deep learning models that can achieve classification accuracy are applicable.

[0103] The second stage is the crack segmentation stage. This stage segments the crack area in the bridge image block obtained in the first stage. In order to make the model more generalizable, the present invention combines the collected images with the public crack image set to expand the scale of crack images. Unlike the first stage, the models in the second and subsequent third stages need to be more generalizable, that is, have better domain adaptability. The domain adaptability of the first stage is limited to the image data set space collected by a single bridge, as long as it can cope with the differences between images collected by different weather and different drone equipment. Since the second and third stages also involve annotation (segmentation annotation, crack type classification annotation), especially the high cost of segmentation annotation, it is necessary to use public data sets containing labeled data. Another reason is that the number of image blocks containing cracks in the image data of a bridge collected in the first stage is very limited, and the data provided is far from enough for the training of the segmentation model. One reason why this approach is effective is that for image blocks containing only the bridge area, the background is single, and there is little difference from the crack images in the public data set: bridge, ground, wall, etc.

[0104] The third stage is the further analysis of the crack segmentation results, including the classification of crack types, where cracks are divided into two categories: ringed and acyclic. In addition to classification, the width and direction of the cracks need to be estimated. To this end, the present invention adopts a solution combining deep learning models with traditional image processing technology in this stage. The crack classification adopts the lightweight classification model mobileNet v3; the crack geometry is indirectly estimated using the corrosion and expansion operations of multi-scale morphogens.

[0105] Since this stage is faced with crack segmentation results, the image space is already very different from the original crack image and the image space corresponding to the bridge body image block, so the difference in the input image at this stage is relatively small. In this case, the trained crack type classification model has relatively good universality. The parameters set by the crack geometry estimation algorithm based on traditional image processing technology are also relatively universal, thus solving the domain adaptability problem of the overall bridge crack disease detection algorithm.

Claims

1. A method for identifying cracked beam defects in unmanned aerial vehicle bridges based on a progressive domain adaptation strategy, characterized by The following steps are involved: Step 1: Use drones to collect bridge images, divide the bridge images collected by drones into overlapping blocks, annotate the segmented bridge images, and use the CNN model to classify the bridge image blocks; Step 2: After small-scale annotation of the cracks in the bridge image blocks obtained in step 1, the cracks are mixed with the public dataset to obtain a mixed dataset, and the crack segmentation model is trained based on the mixed dataset; Step 3: Classify the binary images after crack segmentation and measure the crack indicators.

2. According to claim 1, the method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy is characterized by: The step 1 comprises the following steps: First, the collected bridge image dataset is divided into overlapping blocks: The original bridge image taken by the drone is cut into 448*448 size images with 50% overlap. After that, the cut images are labeled, and the image blocks containing 90% or more of the bridge body are labeled as positive samples, and their labels are 1; the image blocks containing less than 90% of the bridge body are labeled as negative samples, and their labels are 0; Finally, VGG16 is used to classify the bridge image blocks of the obtained dataset.

3. The method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy according to claim 2 is characterized by: The binary classification threshold used is 90%. When the binary classification probability is higher than 90%, VGG16 determines that the image result is a bridge, otherwise it is background.

4. According to claim 1, the method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy is characterized by: In the step 2, the relevant public data set is mixed with the self-collected and classified bridge image blocks containing cracks to obtain a data set that is more conducive to the training of the crack segmentation model.

5. The method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy according to claim 1 is characterized by: In the step 2, a method of first labeling and then segmenting is adopted: first, the cracks in the bridge image taken by the drone are labeled for semantic segmentation, and then the overlapping segmentation of the bridge image in step 1 is synchronized to segment the labeled results of the semantic segmentation.

6. The method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy according to claim 1 is characterized by: In the step 2, the crack segmentation model adopts the Unet network.

7. The method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy according to claim 6 is characterized by: In the step 2, the Unet network is tested using a mixed data set and a private data set, respectively. The private data set is 11,812 images, and the mixed data set is a mixed data set synthesized from the private data set of 11,812 images and the public data set of 11,298 images. The training set, validation set, and test set are divided into a ratio of 6:3:

1. After the model is trained using the mixed data set and the private data set through the Unet network, the private data set is used to test it. The private dataset refers to a dataset obtained by performing small-scale annotation on the cracks in the bridge image blocks obtained in step one.

8. The method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy according to claim 7 is characterized by: In the step 2, the mixed data set includes the binary image obtained by bridge crack segmentation and the binary image obtained by bridge crack segmentation of the public data set.

9. The method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy according to claim 1 is characterized by: In the step three, a lightweight MobileNetv4 classification model is used. When training the MobileNetv4 classification model, the mixed data set is divided into a training set and a validation set, and the ratio of the number of training sets to the validation set is 8:

2. In the training set and the validation set, the ratio of the number of loops to the number of acyclic sets is 1:

1.

10. The method for identifying cracked beam defects of unmanned aerial vehicle bridges based on a progressive domain adaptation strategy according to claim 1, characterized in that: In the step 3, when measuring the crack index, the width of the crack in a single region is divided by using the morphological operations of corrosion and expansion on the binary image after the bridge crack is segmented by the Unet network; specifically: Through the operation of corrosion and expansion of morphogens of different sizes, the binary image after the bridge crack segmentation is processed, and the corresponding morphogen size when each area of ​​the crack is completely corroded is found, and the width of the crack is defined by the operator size; After reading the binary image after the crack segmentation of the bridge body, firstly copy the data of the original binary image a after the crack segmentation to obtain image b. Image b is used as the "switch" of the algorithm iteration, that is, only when all elements in image b are eroded, all pixels in image b are set to black and the iteration ends. Otherwise, the size of the morphological operator is incremented and the next iteration is performed. During the iteration, the width characteristics of the crack are evaluated by applying the erosion and dilation algorithms to image b. The erosion algorithm shrinks the edge of the image inward, and the subsequent dilation operation expands the image outward; relative to the current morphological sub-size, for small cracks, after the operation of first erosion and then dilation, all pixels in the area will be eroded and cannot be restored by dilation; relative to the current morphological sub-size, for wider cracks, the morphological sub-size cannot completely erode them during the erosion process, so the shape of the crack will be restored after dilation; The morphological operator is set to a square with an initial size of 3*3. Each row and column is expanded by 1 pixel in subsequent iterations. In each iteration, the morphological operator of the current size is used to perform corrosion first, followed by dilation. If the crack image changes, that is, by comparing the current image b with the image c, if there is a significant difference between the two, the eroded crack area is obtained through the images b to c, and this part of the area is marked as a crack type that matches the current morphological sub-size. Then the morphological operator size is updated, and after replacing image b with the current image c, the above process is repeated until image c is completely black, that is, all crack pixels are eroded by the corresponding morphological operator and cannot be restored by the subsequent dilation operation. After obtaining the crack area information matched with different size morphological operators, the pixel size of the crack image can be calculated according to the distance from the bridge body when the drone is photographed.

Images