Unmanned aerial vehicle bridge crack beam disease identification method based on progressive domain adaptation strategy

Abstract

The application discloses a UAV bridge crack beam disease identification method based on a progressive domain adaptation strategy, and the method comprises the following steps: collecting bridge images by using a UAV, performing overlapped block division on the bridge images collected by the UAV, labeling the divided bridge images, and classifying the bridge image blocks by using a CNN model; after small-scale labeling of cracks in the obtained bridge image blocks, mixed data sets are obtained by mixing the labeled bridge image blocks with public data sets, and a crack segmentation model is trained based on the mixed data sets; binary images after crack segmentation are classified by types, and crack indexes are measured.The application divides the bridge crack disease detection problem into three stages, i.e., bridge image block classification, bridge crack segmentation, crack classification and geometric shape estimation. By dispersing the domain adaptation problem to the three different stages, the labeling cost is reduced, and the data collection pressure is reduced. The application provides an effective solution for the bridge crack disease problem.

Description

Technical Field

[0001] This invention relates to the field of bridge health assessment technology, specifically to a method for identifying cracked beam defects in bridges using unmanned aerial vehicles (UAVs) based on a progressive domain adaptation strategy. Background Technology

[0002] With the increasing number of bridges and their extended service life, defects such as cracks are becoming more and more apparent. Cracks often indicate structural stress concentration or material aging. If not addressed promptly, they may continue to expand, leading to structural damage, exposure and corrosion of internal steel reinforcement, thereby reducing bridge stability and shortening its service life. Therefore, bridge health monitoring is becoming increasingly important.

[0003] Traditional bridge inspection methods rely on manual visual inspection and simple tool measurements. Their accuracy and reliability are influenced by experience, leading to missed detections or misjudgments, low efficiency, and the need for large inspection equipment in inaccessible areas. This not only increases the bridge's load and congests traffic but also incurs high equipment costs and maintenance expenses. To overcome the limitations of traditional inspection methods, modern bridge inspection has gradually introduced advanced technologies such as high-precision sensors, drones, and laser scanning. Drones, in particular, efficiently inspect bridges using high-precision sensors, especially in high-altitude and hazardous areas, with real-time image data acquisition improving inspection efficiency and safety. Combined with automated inspection, cracks can be automatically identified and analyzed, reducing errors and improving the accuracy of bridge health assessments.

[0004] Crack detection techniques can be categorized into traditional methods and deep learning-based methods. Traditional digital image processing methods are widely used in bridge inspection and mainly include two types:

[0005] 1) Crack detection based on edge algorithms. Edges are detected by calculating the gray-level gradient or derivative of the image. Commonly used algorithms include Roberts, Prewitt, Sobel, Canny and Laplacian operators. Reference [1]: Dorafshan, S., Thomas, RJ and Maguire, M., 2018. Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete. Construction and Building Materials, 186, pp. 1031-1045. The deep convolutional neural network (DCNN) and common edge detection algorithms are compared in the field of crack detection. The experiment shows that the accuracy of edge detection algorithms is not as good as DCNN and retains a lot of background noise. The shortcomings of edge feature-based methods are that they are sensitive to image noise and lack self-learning ability, resulting in poor generalization.

[0006] 2) Threshold-based methods remove noise by preprocessing the image and set a threshold using grayscale value or color intensity difference to distinguish areas that may contain cracks from the background. Threshold acquisition methods include: global threshold, local threshold, iterative threshold, maximum entropy threshold algorithm and Otsu method. Reference [2]: Vivekananthan, V., Vignesh, R., Vasanthaseelan, S., Joel, E. and Kumar, KS, 2023. Concrete bridge crack detection by imageprocessing technique by using the improved OTSU method. Materials Today: Proceedings, 74, pp. 1002-1007. The image is preprocessed using grayscale discrimination method, the threshold is set using Otsu algorithm, and the cracks at the edge of the image pixel are detected using Sobel filter. The accuracy of this method is as high as 95%. However, this method is limited by aspect ratio RA and margin parameter, and the generalization ability is weak. Threshold-based methods often face the following problems: the threshold selection is difficult, and different images require different thresholds, which seriously limits the generalization ability of the algorithm. At the same time, this method is also sensitive to image noise.

[0007] Due to the above problems of traditional image processing methods, deep learning-based methods have gradually become the mainstream. Deep learning can learn image features autonomously and link the classification or recognition results with the feature extraction process, thereby achieving better generalization ability. Reference [3]: Tran, TS, Nguyen, SD, Lee, HJ and Tran, VP, 2023. Advanced crack detection and segmentation on bridge decks using deep learning. Construction and Building Materials, 400, p. 132839. A deep learning method based on YOLOv7 and U-Net was proposed, and five deep learning algorithms were evaluated for crack detection. It can handle complex cracks and maintain high detection accuracy. Experiments show that the proposed method has high accuracy for crack length and width and classification. However, this method requires a large amount of labeled data and computing resources.

[0008] Reference [4]: ​​Laxman, KC, Tabassum, N., Ai, L., Cole, C. and Ziehl, P., 2023. Automated crack detection and crack depth prediction for reinforced concrete structures using deep learning. Construction and Building Materials, 370, p. 130709. A comprehensive automatic crack detection and crack depth prediction framework was developed, which can detect crack location and predict depth, but its robustness is insufficient in complex environments.

[0009] Reference [5]: Gupta, P. and Dixit, M., 2022. Image-based crack detection approaches: a comprehensive survey. Multimedia Tools and Applications, 81(28), pp. 40181-40229. A bridge crack detection method based on deep learning, combining "dual detection + single segmentation" coarse and fine detection, is proposed. Based on the initial detection, YOLOV5(x) is used for fine crack detection, and finally, the U-Net model is used to segment the accurately detected cracks. Although the accuracy and efficiency are improved, the network structure is complex, the computational resources are high, and it depends on high-quality training data.

[0010] While deep learning-based methods can improve crack detection performance, they require large-scale datasets, incurring high annotation costs and complex networks that consume significant computational resources. Achieving strong generalization capabilities necessitates collecting large-scale datasets of various bridge types and cracks, which is not only difficult to collect but also prohibitively expensive to annotate, making it impractical. Currently available datasets, especially bridge images captured by drones, are limited in number and scale, making it difficult to support complex model training. Furthermore, they are concentrated on specific crack types and scenarios, easily leading to model overfitting or poor generalization.

[0011] Figure 1The comparison between publicly available datasets and actual drone-captured images of bridge crack areas is shown in sub-images (a) and (b). It can be seen that there are significant differences between the publicly available datasets and the actual drone-captured bridge crack images, failing to cover the real-world bridge data distribution involved in the experiment. Therefore, although the model trained on the publicly available dataset performs well on the corresponding test set, it fails to detect cracks on other bridge datasets. This is a typical domain adaptation problem; the training and test datasets used represent only a subset of the data distribution in the real-world problem, and the model trained on them is not applicable to other subsets of the real-world data distribution. Even for a specific bridge, images cannot be captured under all weather conditions and from all angles, and the data distribution will vary across different detection time periods. In such cases, if only the limited dataset is used to train the crack detection model, the model's generalization ability cannot be guaranteed.

[0012] Because UAVs have significant advantages over traditional detection methods, many scholars have begun to explore the integration of UAVs with detection algorithm modules. Reference [6]: Song, F., Liu, B. and Yuan, G., 2024. Pixel-Level Crack Identification for Bridge Concrete Structures Using Unmanned Aerial Vehicle Photography and Deep Learning. Structural Control and Health. A real-time crack identification algorithm combining machine vision (MV) and deep learning (ResNet-18) was proposed, and the results showed that it still has high accuracy in complex environments. However, it is still limited by the flight stability of UAVs, image quality, lighting and algorithm generalization ability.

[0013] The patent "A method and system for detecting bridge cracks by UAV based on deep learning" (CN202310033923.6) uses UAV A to collect three-dimensional data, plans the flight path of UAV B, and uses an improved SSD algorithm to detect images. However, due to the influence of the natural environment and model accuracy, it may lead to unstable flight process and high cost risk.

[0014] The patent "A method for detecting bridge cracks based on UAV and YOLOv5 model" (CN202310039881.7) collects bridge deck video by UAV, saves and annotates crack information frame by frame, and uses YOLOv5 algorithm to identify cracks. However, it requires a large amount of labeled data and has insufficient generalization ability, making it difficult to adapt to different bridges and environmental conditions.

[0015] Therefore, the automatic analysis of bridge crack damage images using drones faces both challenges related to deep learning technology and limitations inherent in drone photography itself. Drones offer a wide field of view, and the environments of different bridges vary greatly, making the direct application of deep learning methods for crack detection challenging. On one hand, safety concerns limit close-range drone photography, resulting in images containing excessive irrelevant content, such as the surrounding environment of the bridge. Figure 2 As shown. On the other hand, the crack targets are relatively small and limited in number, making it difficult to clearly capture these small crack areas in the images.

[0016] Furthermore, due to cost and safety concerns, drones cannot take close-up photos of bridges in real-world scenarios. The resulting images contain a large amount of redundant background information, which can easily interfere with crack detection results, thus affecting the final crack detection accuracy. On the other hand, training data on a single bridge in reality often only applies to that specific bridge, meaning it has weak generalization ability. To effectively model real-world problems, high annotation costs are necessary, and currently, datasets of this scale do not exist.

[0017] Automatic identification of bridge cracks based on drone images needs to overcome the following difficulties:

[0018] 1) Cracks are small and limited in number. Training a general-purpose target detection model requires not only a large-scale dataset covering various situations and high annotation costs, but also the inability to guarantee high accuracy for cracks on a single bridge. Currently, there is no such publicly available dataset.

[0019] 2) Crack target detection models trained on small-scale bridge crack datasets collected by drones have poor domain adaptability.

[0020] 3) Deep learning-based estimation of crack geometry, such as width and direction, has poor interpretability. Furthermore, traditional image processing methods cannot effectively handle various noise interferences present in crack images.

[0021] 4) Cracks are small and limited in number. Training a general-purpose target detection model requires not only a large-scale dataset covering various scenarios and high annotation costs, but also the inability to guarantee high accuracy for cracks on a single bridge. Currently, there are no publicly available datasets of this kind.

[0022] 5) Crack target detection models trained on small-scale bridge crack datasets collected by drones have poor domain adaptability.

[0023] 6) Deep learning-based estimation of crack geometry, such as width and direction, has poor interpretability. Furthermore, traditional image processing methods cannot effectively handle various noise interferences present in crack images. Summary of the Invention

[0024] To address the aforementioned problems, this invention employs a progressive domain adaptation strategy and combines deep learning methods with traditional computer vision methods, proposing a UAV-based method for identifying bridge crack defects. The bridge crack defect detection problem is decomposed into three stages: bridge image block classification, bridge crack segmentation, and crack classification and geometric morphology estimation. By distributing the domain adaptation problem across these three different stages, not only are annotation costs reduced and data acquisition pressure alleviated, but the overall versatility and specialization of the algorithm are also ensured. This provides an effective solution to the problem of bridge crack defects.

[0025] The technical solution adopted in this invention is as follows:

[0026] A method for identifying cracked beam defects in bridges using unmanned aerial vehicles (UAVs) based on a progressive domain adaptation strategy includes the following steps:

[0027] Step 1: Collect bridge images using drones, divide the collected bridge images into overlapping blocks, label the segmented bridge images, and use a CNN model to classify the bridge image blocks.

[0028] Step 2: After small-scale annotation of the cracks in the bridge image blocks obtained in Step 1, the cracks are mixed with the public dataset to obtain a mixed dataset, and the crack segmentation model is trained based on the mixed dataset.

[0029] Step 3: Classify the type of the binary image after crack segmentation, and measure the crack index at the same time.

[0030] Step one includes the following steps:

[0031] First, the collected bridge image dataset is divided into overlapping blocks:

[0032] The original bridge images captured by the drone were sliced ​​into 448*448 images, while maintaining a 50% overlap.

[0033] Next, the cut images were labeled. Image blocks containing 90% or more of the bridge structure were labeled as positive samples with a label of 1; image blocks containing less than 90% of the bridge structure were labeled as negative samples with a label of 0.

[0034] Finally, VGG16 was used to classify bridge image patches in the obtained dataset.

[0035] The binary classification threshold used is 90%. When the binary classification probability is higher than 90%, VGG16 determines the image result as a bridge, otherwise it is the background.

[0036] In step two, the publicly available crack image dataset is mixed with bridge image patches containing cracks that are collected and classified by ourselves to obtain a dataset that is more conducive to the training of crack segmentation models.

[0037] In step two, a labeling-then-blocking approach is adopted: First, the cracks in the bridge image captured by the UAV are semantically segmented and labeled. Then, the semantic segmentation labeling results are blocked synchronously with the bridge image in step one in an overlapping manner.

[0038] In step two, the crack segmentation model uses the Unet network.

[0039] In step two, the Unet network is tested using a hybrid dataset and a private dataset. The private dataset consists of 11,812 images, while the hybrid dataset is a composite dataset of 11,812 images from the private dataset and 11,298 images from a publicly available crack image dataset. The training set, validation set, and test set are divided in a 6:3:1 ratio. After training the model using the hybrid dataset and the private dataset, the Unet network is then used to detect the crack images.

[0040] A private dataset refers to a dataset obtained by annotating cracks in the bridge image patch obtained in step one on a small scale.

[0041] In step two, the hybrid dataset includes binary images obtained from bridge crack segmentation and binary images obtained from bridge crack segmentation in a public dataset.

[0042] In step three, during the classification process, only the two types of cracks, namely those with rings and those without rings, are labeled.

[0043] In step three, a lightweight MobileNetv4 classification model is used. When training the MobileNetv4 classification model, the mixed dataset needs to be divided into a training set and a validation set, with a ratio of 8:2 between the training set and the validation set. In the training set and the validation set, the ratio of the number of sets with and without loops is 1:1.

[0044] In step three, when measuring crack indices, the width of individual crack regions is determined by applying morphological operations of erosion and dilation to the binary image of the bridge crack segmentation using the Unet network; specifically:

[0045] By performing erosion and dilation operations on morphites of different sizes, the binary image of the bridge crack segmentation process is processed to find the morphite size corresponding to the complete erosion of each region of the crack, and the width of the crack is defined by the operator size.

[0046] After reading the binary image of the bridge crack segmentation, the original binary image 'a' is first copied to obtain image 'b'. Image 'b' serves as a "switch" for algorithm iteration; that is, the iteration ends only when all elements in image 'b' are completely eroded, at which point all pixels in image 'b' are set to black. Otherwise, the size of the morphological operator is incremented, and the next iteration begins. During iteration, erosion and dilation algorithms are applied to image 'b' to evaluate the crack width characteristics.

[0047] The erosion algorithm shrinks the image edges inward, while the subsequent dilation operation expands the image outward. For small cracks, the pixels in the area will be completely eroded and cannot be recovered by dilation after the erosion and dilation operation, relative to the current morphotype size. For wider cracks, the morphotype cannot completely erode them during the erosion process, so the shape of the crack will be restored after dilation.

[0048] The morphological operator is set to a square with an initial size of 3*3. In subsequent iterations, the size of each row and column is increased by one pixel. In each iteration, erosion is performed first using the morphological operator of the current size, followed by dilation. If the crack image changes (i.e., by comparing the current image b with image c), and a significant difference exists, the eroded crack region is obtained from images b to c, and this region is identified as a crack type matching the current morphological operator size. The morphological operator size is then updated, and image b is replaced with the current image c. This process is repeated until image c is completely black, meaning all crack pixels have been eroded by the corresponding morphological operator and cannot be recovered by the subsequent dilation operation.

[0049] After obtaining crack region information that matches operators of different sizes and shapes, the crack image pixel size can be calculated based on the distance from the bridge body when the drone takes the picture.

[0050] This invention discloses a method for identifying cracked beam defects in bridges using unmanned aerial vehicles (UAVs) based on a progressive domain adaptation strategy. The technical advantages are as follows:

[0051] This invention reduces the cost of labeling large-scale crack datasets on one hand; on the other hand, it combines deep learning methods with traditional image processing methods, decomposing the problem of identifying bridge crack defects using UAVs into three components: bridge image block classification; bridge crack defect image segmentation based on hybrid datasets; and classification and measurement of bridge crack segmentation results using a combination of deep learning and traditional image processing techniques. The labeling cost is proportionally distributed across the first, second, and third stages.

[0052] In the first stage, drones are used to collect data on the bridges to be monitored and detected, and the drone-captured images are divided into overlapping blocks. The segmented images are then labeled, with 90% of the bridge image blocks designated as positive samples and the rest as negative samples. This labeling method drastically reduces the cost compared to bounding box labeling in object detection models such as YOLO. This invention does not limit the classification model; any classification model that meets the requirements is applicable. The CNN model used in this invention exhibits higher efficiency in classification tasks, with a simple network structure, mature training process, and well-developed optimization strategies (such as gradient descent and regularization), which can significantly reduce the model's training cost and minimize the interference of background information on detection accuracy. Therefore, in the first stage, a CNN model is used to classify the bridge image blocks.

[0053] In the second stage, we make full use of publicly available crack datasets (various scenarios), and after small-scale annotation of cracks in the bridge image blocks obtained in the first stage, we mix them with the publicly available datasets and train the crack segmentation model based on them.

[0054] In the third stage, all data consists of binary images of the cracks after segmentation. These images significantly reduce information about the image capture environment, making classification or regression models trained on them less susceptible to environmental factors and exhibiting better generalization ability. Furthermore, to reduce annotation costs, this stage only labels the crack type: with loops or without loops, while the size and orientation of the cracks are evaluated using traditional image processing methods. Because the input images are binary, they also demonstrate good generalization ability compared to traditional image processing techniques. Attached Figure Description

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0056] Figure 1 A comparison chart of publicly available datasets and proprietary datasets;

[0057] Figure 1 In the text, part a presents examples of cracks in various public datasets; part b presents examples of cracks in self-built datasets.

[0058] Figure 2 Real-world images taken by drones.

[0059] Figure 3 This is a system schematic diagram of the present invention.

[0060] Figure 4 A flowchart for overlapping block segmentation and bridge image block classification.

[0061] Figure 5 Example image segmentation for drone images.

[0062] Figure 6 This is a schematic diagram of a VGG16 network.

[0063] Figure 7 This is a graph showing the training results.

[0064] Figure 8 This is a diagram of the Unet network structure.

[0065] Figure 9 This is a comparison chart of the segmentation performance of the model on different datasets.

[0066] Figure 10 This is a comparison chart of the segmentation results output by models trained on a hybrid dataset and a private dataset.

[0067] Figure 10 In the diagram, part a represents the results after training on mixed-data sets; part b represents the results after training on private-data sets.

[0068] Figure 11 This is a type diagram showing both cyclic and acyclic elements.

[0069] Figure 12 This is a network structure diagram.

[0070] Figure 13 This is a graph showing the change in the model's accuracy during the training process.

[0071] Figure 14 This is a graph showing the change in the model's loss rate during training.

[0072] Figure 15 This is a flowchart of the crack detection algorithm.

[0073] Figure 16 Examples of visual images showing the width of cracks and pie charts showing the proportion of crack width. Detailed Implementation

[0074] This invention addresses the problem of automatic identification of cracks and defects in bridge images captured by drones, proposing an effective solution. Its core key points include the following two aspects:

[0075] 1. The problem of bridge crack detection is decomposed into three stages: bridge image patch classification, bridge crack segmentation, and crack classification and geometric morphology estimation. By distributing the domain adaptation problem across these three different stages, not only are annotation costs reduced and data acquisition pressure alleviated, but the overall versatility and specificity of the algorithm are also ensured. Specificity refers to the need to train a corresponding bridge image patch classification model for each bridge; versatility refers to the ability to classify cracks and estimate their geometric morphology. This combination of specific and general models provides an effective solution to the bridge crack detection problem.

[0076] 2. This paper proposes a crack geometry identification scheme with interpretability that combines deep learning with multi-scale morphological image processing algorithms. To improve the algorithm's domain adaptability, the results of crack segmentation are used for judgment, greatly reducing the influence of various backgrounds. This allows for rapid classification of cracks with and without loops using a lightweight classification network. For the crack width and orientation features, this invention employs multi-scale morphology and utilizes a combination of erosion and dilation operations to design an indirect method for estimating crack width. This method requires no data annotation, is highly efficient, and the results have good interpretability, meeting the requirements for bridge defect detection.

[0077] A UAV crack identification method based on a progressive domain adaptation strategy. This method mainly includes three key steps: overlapping block segmentation of UAV bridge images and bridge image block classification; bridge crack defect image segmentation based on a hybrid dataset; and crack type classification and crack size interpretability estimation. The system scheme diagram of this invention is shown below. Figure 3 As shown, the dataset used in the model training process mainly comes from the internet and self-collected bridge defect images. This method aims to improve the detection accuracy and adaptability of UAVs for bridge cracks through a progressive domain adaptation strategy. The specific details are as follows:

[0078] Step 1: Overlapping and segmenting of UAV bridge images and classification of bridge image blocks:

[0079] To address the data acquisition problem, this invention proposes a solution for separate data acquisition for different bridges. First, data collection is required during regular inspections of each bridge, making it a routine operation. Second, data annotation at this stage only involves selecting images containing bridge sections within image blocks, resulting in low annotation costs and a limited data volume, allowing for quick completion. Finally, the deep learning model trained for each bridge can be repeatedly used to detect bridge cracks and defects. With repeated inspections and increasing data volume, the model's performance can be continuously improved.

[0080] The overall process of this module is as follows: Figure 4 As shown, firstly, the collected dataset is divided into overlapping blocks. Specifically, the original bridge images captured by the drone are sliced ​​into 448*448 pixels to facilitate mixed training with public datasets, while maintaining a 50% overlap. The purpose is to ensure that any bridge structures that may be present in the images are not ignored during the slicing process. The segmented dataset is shown below. Figure 5As shown in the diagram. Next, the segmented images are labeled. Images containing 90% or more of the bridge's surface area are labeled as positive samples with a label of 1; images containing less than 90% of the bridge's surface area are labeled as negative samples with a label of 0. Since this type of labeling is easily distinguishable by the human eye, the labeling cost is very low. For the segmented dataset, the human eye can distinguish between positive and negative samples very quickly; a normal person can classify approximately 200 images in just 60-80 seconds.

[0081] Finally, the obtained dataset is used for bridge image patch classification. This invention does not limit the network model used for bridge image patch classification; here, VGG16 is used, and the network diagram is shown below. Figure 6 As shown, this invention applies to other deep learning models that can achieve classification accuracy. Before training the model, this invention uses data image enhancement techniques, such as adjusting hue, image saturation, and image exposure, to enable the model to have strong generalization ability under different conditions.

[0082] This invention uses a binary classification threshold of 90%. When the binary classification probability is higher than 90%, the model classifies the image as a bridge; otherwise, it classifies it as background. The ratio of training set to validation set is 7:3. After training with the VGG16 model, the final validation set shows an accuracy of 99.3%. Figure 7 As shown, the algorithm used in this invention has good application potential.

[0083] Step 2: Image segmentation of bridge cracks based on hybrid datasets:

[0084] The above-described block processing decomposes the problem of target detection and example segmentation of bridge crack defects under complex conditions into a classification problem of bridge region image patches and a semantic segmentation problem of cracks within those patches. For the crack segmentation problem, since the focus is on various types of cracks, relevant public datasets can be mixed with self-collected and classified bridge region image patches to obtain a set more conducive to training the segmentation model, such as... Figure 1 As shown, Figure 1 Neutron map (a) is a publicly available dataset of crack images. Figure 2 The images shown are self-collected bridge tiles (containing cracks). It can be seen that, due to the introduction of different public datasets in the crack image semantic segmentation process, the trained segmentation model exhibits better domain adaptability, i.e., generalization ability. This hybrid approach effectively expands the diversity of crack images. It effectively transitions the entire algorithm from a first-stage classification model applicable only to a specific bridge to a second-stage crack segmentation model with better generalization ability.

[0085] Because the publicly available dataset of bridge crack defects already provides annotations, the semantic segmentation only requires annotation of the crack-containing image blocks in the self-collected dataset, significantly reducing annotation costs. To further improve efficiency, this invention adopts a method of annotation before segmentation. Specifically, the cracks in the bridge images captured by the UAV are first semantically segmented and annotated. Then, in sync with the overlapping segmentation of the bridge images in the first stage, the segmented annotation results are further segmented, effectively avoiding repeated annotation of images with overlapping regions.

[0086] Similar to the bridge tile classification model, this invention does not restrict the segmentation model for semantic segmentation; any segmentation model that meets the requirements is applicable. This invention employs the Unet network, a widely used semantic segmentation model with excellent performance. Figure 8 The diagram shows the Unet network structure.

[0087] To demonstrate the impact of the dataset on the results, this invention uses a hybrid dataset and a private dataset to test the Unet model. The private dataset contains 11,812 images, while the hybrid dataset is a composite of the 11,812 images from the private dataset and 11,298 images from the public dataset. The training, validation, and test sets are divided in a 6:3:1 ratio. After training the Unet model using both the hybrid and private datasets, the private dataset is then used for detection. The detection results are as follows: Figure 9 As shown.

[0088] from Figure 9 As can be seen, training the model using the hybrid dataset significantly improves various parameters compared to training with only the private dataset. Specifically, it enhances mIoU (Mean Intersection over Union: the average IoU across all classes, used to evaluate the model's segmentation performance across categories), mPA (Mean Pixel Accurate: the average pixel accuracy across all categories, representing the model's accuracy in each category), Precision (the proportion of correctly classified positive samples out of all positive samples, used to measure the model's accuracy), and Recall (the proportion of correctly classified positive samples out of all actual positive samples, used to measure the model's ability to detect positive samples). Therefore, this invention uses a hybrid dataset (public and private datasets) to train the model. This avoids overfitting caused by the private dataset and effectively supports detection of any bridge-related parameters. Figure 10The results are shown below, with the mixed dataset and the private data used for testing. It can be clearly seen that training the Unet model with the mixed dataset results in higher precision in crack segmentation and effectively improves the model's anti-interference ability during segmentation.

[0089] Step 3: Crack type classification and crack size interpretability estimation:

[0090] The third stage requires classifying the binary images after crack segmentation into categories: those with loops and those without; and simultaneously measuring crack indices (crack width). Similarly, during classification, only categories such as... Figure 11 The two types of cracks, with and without loops, are labeled, and the labeling cost is not high. For a single bridge, the number of cracks of the two types is not balanced and is limited. Therefore, a hybrid dataset (second-stage crack segmentation) is needed to effectively solve this problem. The hybrid dataset mainly consists of binary images obtained from existing bridge crack segmentation and binary images obtained from bridge crack segmentation in public datasets. This can improve the generalization ability of the classification model.

[0091] In this invention, the lightweight MobileNetv4 classification model is employed. MobileNetv4 is a deep convolutional neural network model in the MobileNet series, primarily used for tasks such as image classification on mobile devices and embedded systems. Similar to the first and second stages, this invention does not impose any restrictions on the classification model used; any suitable classification model is acceptable. The network structure diagram used in this invention is shown below. Figure 12 As shown, when training a classification model, the dataset needs to be divided into a training set and a validation set, with a ratio of 8:2. Within the training and validation sets, the ratio of units with cycles to those without cycles is 1:1. Figure 13 and Figure 14 The figures show the accuracy and loss curves obtained by classifying binary images of cracks. The experimental results show that the model has an accuracy of 99% and 97% on the training set and validation set, respectively. Therefore, the classification model trained on the mixed dataset has a good ability to distinguish between the two types of cracks.

[0092] When measuring crack indices (crack width), this invention uses the Unet network to segment binary images of bridge cracks and applies morphological operations of erosion and dilation to determine the width of individual crack regions. Specifically, this invention proposes a crack width estimation method based on multi-scale morphological erosion and dilation operations. The core idea is to process the binary crack image by performing erosion and dilation operations on morphogenetic units of different sizes, finding the morphogenetic unit size corresponding to when each region of the crack is completely eroded, and defining the crack width by the operator size. A detailed flowchart is shown below. Figure 15 As shown.

[0093] After reading the binary image of the crack, this invention first copies the original binary image 'a' after crack segmentation to obtain image 'b'. Image 'b' serves as a "switch" for algorithm iteration; that is, the iteration ends only when all elements in image 'b' are completely eroded, at which point all pixels in image 'b' are set to black. Otherwise, the size of the morphological operator is incremented, and the next iteration begins. During iteration, the width characteristics of the crack are evaluated by applying erosion and dilation algorithms to image 'b'. The erosion algorithm shrinks the image edges inward, while the subsequent dilation operation expands the image outward. For small cracks (relative to the current morphological size), after the erosion followed by dilation operation, all pixels in the region will be eroded and cannot be recovered by dilation. For wider cracks (relative to the current morphological size), the morphological operator cannot completely erode them during the erosion process, so the shape of the crack is restored after dilation.

[0094] Since the crack direction is random, this invention sets the morphological operator to a square with an initial size of 3*3. In subsequent iterations, the row and column size are increased by one pixel each time. In each iteration, erosion is performed first using the morphological operator of the current size, followed by dilation. If the crack image changes, i.e., by comparing the current image b with image c, if there is a significant difference, the eroded crack region is obtained from images b to c, and this region is identified as a crack type matching the current morphological operator size. Then, the morphological operator size is updated, and image b is replaced with the current image c. This process is repeated until image c is completely black, meaning all crack pixels have been eroded by the corresponding morphological operator and cannot be recovered by the subsequent dilation operation.

[0095] The algorithm for determining crack width using corrosion and expansion algorithms is as follows:

[0096]

[0097]

[0098] After obtaining crack region information matching operators of different sizes and shapes, the ratio of crack image pixel size to actual crack size can be calculated based on the distance from the bridge structure during drone photography, thereby estimating the physical size of the crack. However, for the purposes of this invention, only the image level is considered, i.e., defining the pixel size corresponding to the crack image, without involving the conversion to physical size.

[0099] go through Figure 15After the processing shown, the morphological operator dimensions corresponding to each crack region can be obtained. Based on these dimensions, the crack width can be classified, with the specific number of levels depending on the actual situation. This invention divides it into two levels: wide cracks and narrow cracks. Specifically, cracks matching morphological operators with dimensions smaller than 5*5 are classified as narrow cracks; otherwise, they are classified as wide cracks. For example... Figure 16 As shown, red represents narrow cracks, and green represents wide cracks. Because cracks can transition between narrow and wide, the proportion of wide and narrow cracks can be statistically analyzed to make a more reasonable judgment about crack size. Figure 16 The statistics are shown in the pie chart on the right.

[0100] Because the third stage deals with binary images without background, the problem is relatively less difficult compared to the first and second stages. Therefore, the dataset for training crack type classification does not need to be particularly large, which also reduces the workload of annotation. As for crack morphology index estimation, no annotation is required due to the use of traditional image processing methods.

[0101] This solution breaks down the problem into three stages. The first stage is bridge image data acquisition and initial classification. Regarding data acquisition, this invention proposes separate data acquisition for different bridges, a practice feasible in real-world applications. First, data collection is required for each bridge during regular inspections, making this a routine operation. Second, data annotation in this stage only involves selecting images containing bridge sections within image blocks, resulting in low annotation costs and a relatively limited data volume, allowing for quick completion. Finally, the deep learning model trained for each bridge can be repeatedly used to detect bridge cracks and defects, and its performance can be continuously improved with increasing data volume over multiple inspections.

[0102] In the first stage, the present invention divides the acquired images into overlapping blocks and trains a model to classify these blocks, identifying those that contain only the bridge region. Specifically, image blocks where the bridge region occupies more than 90% of the image are considered bridge region image blocks. This classification effectively reduces the difficulty of subsequent crack segmentation, thereby reducing the requirements for model parameter size and training dataset size in subsequent segmentation stages. There are no restrictions on the network model used for image block classification in this stage; the present invention uses the Vgg16 model, but other deep learning models that can achieve classification accuracy are also applicable.

[0103] The second stage is the crack segmentation stage. This stage segments the crack regions in the bridge image patches obtained in the first stage. To enhance the model's generalization ability, this invention combines the acquired images with publicly available crack image sets to expand the scale of the crack images. Unlike the first stage, the models in the second and subsequent third stages need to have greater generalization ability, i.e., better domain adaptability. The domain adaptability of the first stage is limited to the image dataset space acquired from a single bridge, and it only needs to be able to cope with the differences between images acquired under different weather conditions and by different drone equipment. The second and third stages also involve annotation (segmentation annotation, crack type classification annotation), especially since segmentation annotation is costly, so it is necessary to use publicly available datasets containing labeled data. Another reason is that the number of image patches containing cracks in the bridge image data acquired in the first stage is very limited, and the data provided is far from sufficient for training the segmentation model. One reason why this approach is effective is that for image patches containing only bridge areas, the background is uniform and not much different from crack images in publicly available datasets: bridge, ground, walls, etc.

[0104] The third stage involves further analysis of the crack segmentation results, including crack type classification, which categorizes cracks into those with and without loops. In addition to classification, the crack width and direction also need to be estimated. Therefore, this invention employs a combination of deep learning models and traditional image processing techniques in this stage. The lightweight classification model MobileNet v3 is used for crack classification; the crack geometry is indirectly estimated using erosion and dilation operations on multi-scale morphologists.

[0105] Since this stage deals with crack segmentation results, the image space is quite different from the original crack image and the image space corresponding to the bridge body image blocks. Therefore, the difference in the input images at this stage is relatively small, and the crack type classification model trained under these circumstances has relatively good generality. The parameters set by the crack geometry estimation algorithm based on traditional image processing techniques also have relatively good generality, thus solving the domain adaptability problem of the overall bridge crack defect detection algorithm.

Claims

1. A method for identifying cracked beam defects in bridges using unmanned aerial vehicles (UAVs) based on a progressive domain adaptation strategy, characterized in that... Includes the following steps: Step 1: Collect bridge images using drones, divide the collected bridge images into overlapping blocks, label the segmented bridge images, and use a CNN model to classify the bridge image blocks. Step 2: After small-scale annotation of the cracks in the bridge image blocks obtained in Step 1, the cracks are mixed with the public dataset to obtain a mixed dataset, and the crack segmentation model is trained based on the mixed dataset. Step 3: Classify the type of the binary image after crack segmentation, and measure the crack index at the same time. Step one includes the following steps: First, the collected bridge image dataset is divided into overlapping blocks: The original bridge images captured by the drone were sliced ​​into 448*448 images while maintaining a 50% overlap. Next, the cut images were labeled. Images containing 90% or more of the bridge structure were labeled as positive samples with a label of 1; images containing less than 90% of the bridge structure were labeled as negative samples with a label of 0. Finally, VGG16 was used to classify bridge image patches in the obtained dataset; In step two, a labeling-then-blocking approach is adopted: First, the cracks in the bridge image captured by the UAV are semantically segmented and labeled. Then, the semantic segmentation labeling results are blocked synchronously with the bridge image in step one in an overlapping manner.

2. The method for identifying cracked beam defects in UAV bridges based on a progressive domain adaptation strategy according to claim 1, characterized in that: The binary classification threshold used is 90%. When the binary classification probability is higher than 90%, VGG16 determines the image result as a bridge, otherwise it is the background.

3. The method for identifying cracked beam defects in UAV bridges based on a progressive domain adaptation strategy according to claim 1, characterized in that: In step two, the relevant public dataset is mixed with bridge image patches containing cracks that are collected and classified by ourselves to obtain a dataset that is more conducive to the training of crack segmentation models.

4. The method for identifying cracked beam defects in UAV bridges based on a progressive domain adaptation strategy according to claim 1, characterized in that: In step two, the crack segmentation model uses the Unet network.

5. The method for identifying cracked beam defects in UAV bridges based on a progressive domain adaptation strategy according to claim 4, characterized in that: In step two, the Unet network is tested using a hybrid dataset and a private dataset. The private dataset consists of 11,812 images, while the hybrid dataset is a composite dataset of 11,812 images from the private dataset and 11,298 images from the public dataset. The training set, validation set, and test set are divided in a 6:3:1 ratio. After training the model using the hybrid dataset and the private dataset, the model is then tested using the private dataset. A private dataset refers to a dataset obtained by annotating cracks in the bridge image patch obtained in step one on a small scale.

6. The method for identifying cracked beam defects in UAV bridges based on a progressive domain adaptation strategy according to claim 5, characterized in that: In step two, the hybrid dataset includes binary images obtained from bridge crack segmentation and binary images obtained from bridge crack segmentation in a public dataset.

7. The method for identifying cracked beam defects in UAV bridges based on a progressive domain adaptation strategy according to claim 1, characterized in that: In step three, a lightweight MobileNetv4 classification model is used. When training the MobileNetv4 classification model, the mixed dataset is divided into a training set and a validation set, with a ratio of 8:2 between the training set and the validation set. In the training set and the validation set, the ratio of the number of sets with and without loops is 1:

1.

8. The method for identifying cracked beam defects in UAV bridges based on a progressive domain adaptation strategy according to claim 1, characterized in that: In step three, when measuring crack indices, the width of individual crack regions is determined by applying morphological operations of erosion and dilation to the binary image of the bridge crack segmentation using the Unet network; specifically: By performing erosion and dilation operations on morphites of different sizes, the binary image of the bridge crack segmentation process is processed to find the morphite size corresponding to the complete erosion of each region of the crack, and the width of the crack is defined by the operator size. After reading the binary image of the bridge crack segmentation, the original binary image a is copied to obtain image b. Image b serves as the "switch" for algorithm iteration. That is, only when all elements in image b are eroded will all pixels in image b be blacked out and the iteration end. Otherwise, the size of the morphological operator is incremented and the next iteration is performed. During the iteration, the width characteristics of the crack are evaluated by applying erosion and dilation algorithms to image b. The erosion algorithm shrinks the image edges inward, while the subsequent dilation operation expands the image outward. For small cracks, the pixels in the area will be completely eroded and cannot be recovered by dilation after the erosion and dilation operation is performed relative to the current morphological size. For wider cracks, the morphological part cannot completely erode them during the erosion process, so the shape of the crack will be restored after dilation. The morphological operator is set to a square with an initial size of 3*3. In subsequent iterations, the size of each row and column is increased by 1 pixel. In each iteration, the morphological operator of the current size is used to perform erosion first, followed by dilation. If the crack image changes, i.e., by comparing the current image b with image c, if there is a significant difference between the two, the eroded crack area is obtained by comparing image b with image c. At the same time, this part of the area is marked as the crack type that matches the current morphological operator size. Then, the size of the morphological operator is updated, and image b is replaced with the current image c. The above process is repeated until image c is completely black, that is, all crack pixels have been eroded by the corresponding morphological operator and cannot be recovered by the subsequent dilation operation. After obtaining crack region information that matches operators of different sizes and shapes, the pixel size of the crack image can be calculated based on the distance from the bridge body when the drone takes the picture.

Images