Concrete crack recognition method based on synthetic data set and semi-supervised learning

Abstract

The application provides a concrete crack identification method based on a synthetic data set and semi-supervised learning, 1) an original data set is collected, and a certain mainstream neural network model is selected for supervised learning; 2) a benchmark identification model is updated based on a semi-supervised learning method; 3) unlabelled data in a personal data set is classified based on the updated benchmark identification model; 4) an image with higher confidence is selected, pixel-level mask annotation is performed on the image, and a crack segmentation model is determined; 5) the crack segmentation model is used for pixel-level annotation of the image with higher confidence; 6) based on the mask annotation result of the crack, a skeleton line of the crack is obtained, the maximum width and the average width are calculated, and error analysis of the crack width is performed; 7) the concrete crack identification model is verified; 8) the shooting result is imported into the concrete crack identification model, and the skeleton line, the maximum width and the average width of each crack are obtained.

Description

Technical Field

[0001] This invention relates to the field of structural health monitoring, specifically to the identification and detection of concrete cracks based on computer vision. Background Technology

[0002] Currently, bridge inspection still relies primarily on manual inspection, typically using instruments such as telescopes and crack observation devices to pinpoint the location of cracks at close range, followed by measurement of crack width. While this method can complete the inspection of most medium and small bridges, it is extremely labor-intensive, resource-intensive, and time-consuming, representing a pressing problem that needs to be solved in bridge inspection work.

[0003] Given the current surge in traffic management workload, traditional methods for crack detection are increasingly unable to meet engineering needs. In particular, manual management of drawings and documents can easily lead to the loss of project information and make continuous tracking difficult. Therefore, a new intelligent crack identification method is needed in engineering projects to automate and intelligently manage the detection process.

[0004] With the development of computer science, image digitization can transform real images into digital matrices that can be processed by computers, laying the foundation for crack image recognition. In the past decade or so, rapid advancements in both hardware and software have improved the speed and accuracy of digital image processing, making it possible to apply digital images to the field of detection. Due to increased computing power, digital image processing methods are constantly evolving towards greater complexity and refinement, resulting in increasingly diverse effects.

[0005] Crack detection based on image recognition currently mostly uses deep convolutional neural network models, which can complete recognition, localization and pixel-level segmentation tasks in engineering projects, but there are still some problems: (1) The final result of a deep convolutional neural network trained from scratch is related to many factors such as the neural network structure, the initialization of network layer parameters, the value of custom hyperparameters and optimization algorithms, so the test results after each training are subject to certain fluctuations. (2) When training the model, hundreds or thousands of images often need to be labeled to meet the accuracy requirements, but the labeled images can only be labeled manually, which will consume a lot of time and effort. At the same time, if all images need to be labeled manually, it is equivalent to the project having already completed manual detection, and it is not very meaningful to train the neural network for recognition. (3) The trained neural network model can only reflect the features of the training set. When the trained model is applied to a new project, the model performance will also be reduced because the feature distribution of the new project is not exactly the same as that of the original training set. Summary of the Invention

[0006] This invention aims to solve the above problems, thereby providing a method for identifying concrete cracks that is fast in detection and can produce stable and accurate results.

[0007] The technical solution adopted by the present invention to solve the aforementioned problem is as follows:

[0008] A concrete crack identification method based on synthetic datasets and semi-supervised learning.

[0009] Step 1: Collect the original dataset, perform supervised learning using multiple mainstream neural network models, and select a benchmark recognition model from them;

[0010] Step 2: Update the benchmark recognition model based on a semi-supervised learning method;

[0011] Step 3: Based on the updated baseline recognition model, classify the unlabeled data in the personal dataset;

[0012] Step 4: Select images with high confidence in the personal dataset, perform pixel-level mask annotation on them, and compare multiple semantic segmentation models based on the image and mask annotation results. Select the model with the highest MaskIoU index as the crack segmentation model.

[0013] Step 5: Use the crack segmentation model to perform pixel-level annotation on images with high confidence.

[0014] Step 6: Based on the mask dataset segmented in Step 5 and the edge pixels of the crack, obtain the skeleton line of the crack, calculate the maximum width and average width of the crack, and perform error analysis on the crack width to obtain the concrete crack recognition model.

[0015] Step 7: Validate the concrete crack identification model;

[0016] Step 8: Based on the camera's image of the structural surface, import the image into the concrete crack recognition model, divide each crack, and obtain the skeleton line, maximum width, and average width of each crack.

[0017] Compared with the prior art, the present invention employing the above structure has the following advantages:

[0018] The method provided by this invention has the characteristics of high detection accuracy and high speed, and can realize the real-time detection and monitoring of cracks on the surface of concrete structures. It can effectively solve the problems of high labor intensity, slow speed, low frequency and low accuracy in current concrete crack detection.

[0019] As a preferred embodiment, a further technical solution to the above structure is:

[0020] Step 1 specifically involves using six public datasets, from which 500 images (with and without cracks) are selected to form a comprehensive dataset of 3000 images. Supervised learning is then performed using VGGNet, ResNet, Inception-ResNet, and SE-ResNet-18 neural network models. A confusion matrix is ​​introduced in conjunction with the model's parameter count, accuracy, precision, recall, and F-axis. β The score index was used to test each neural network model, and SE-ResNet-18 was selected as the benchmark recognition model based on the test results.

[0021] In step 2, the benchmark recognition model is updated based on the unlabeled data in the personal dataset using an active learning method with stacked convolutional autoencoders.

[0022] In step 4, the crack segmentation model is the Mask R-CNN model with the highest MaskIoU index. The formula for the MaskIoU comparison index is:

[0023] inter = ∑ (x，y)∈P (M c (x, y) + M G (x, y) = 2)

[0024] union = ∑ (x，y)∈P (M c (x, y)==1)+∑ (x，y)∈P (M G (x, y)==1)-inter

[0025] Mask_IoU = (TP) / (TP+FP+FN).

[0026] In step 6, crack width identification involves: taking an image and converting the oblique shot to a frontal shot; inputting the converted frontal crack image into the trained Mask RCNN model to obtain a binary mask with crack pixels as the foreground and concrete surface as the background; defining the two crack edges as L1 and L2, the width, maximum width, and average width are defined as follows:

[0027] d i =min[(x i y i (x) j y j )],(x i y i )EL1, (x j y j )∈L2

[0028] d max =max(d i )

[0029] d Ave =sum(Mask) / L Ave

[0030] In Formula 7, L Ave For the edge, L1 and L2 determine the length of the crack skeleton line;

[0031] The maximum and average crack widths are calculated as follows: Based on the above definition of crack width, and using the extracted mask edges, any pixel on edge L1 is replaced by its outer edge corner point. The Euclidean distance from the corner point to the outer edge corner point of edge L2 is calculated. After traversing all corner points on L2 for any pixel on L1, the minimum value of the calculated result is the crack width originating from that point.

[0032] w max =min[(x i y i ), (x j y j )],(x i y i )∈L1, (x j y j )∈L2

[0033] The crack length is calculated as follows: A line is drawn connecting the midpoints of the widths of each point on the outer edge envelope, extending to the bounding rectangle of all crack pixels. The midpoints of the crack width lines are calculated based on L1 and L2 respectively, yielding the midpoints of the corresponding widths for each point in L1 and L2. All calculated midpoints are then connected to their nearest neighbors to obtain the crack skeleton lines. The total crack length is obtained by summing the lengths of the crack skeleton lines. The average crack width can be calculated using the following formula:

[0034] w Ave =γ×sum(Mask) / L Ave ;

[0035] The maximum crack width error analysis is as follows: The error in the maximum width mainly stems from the approximation of pixels in the crack width direction. Let the crack width direction be θ, and the actual crack width be D. Then, the theoretical difference in horizontal coordinates Δx and vertical coordinates Δy, and the actual difference in horizontal coordinates x and vertical coordinates y calculated are as follows:

[0036] Δx=Dcosθ, Δy=Dsinθ, 0°≤θ≤90°

[0037] Δx-0.5λ<x<Δx+0.5λ, Δy-0.5λ <y<Δy+0.5λ

[0038] The theoretical error is calculated as follows:

[0039]

[0040] Further simplification yields:

[0041] .

[0042] In step 7, the camera is calibrated using the Camera Calibrator toolbox in MATLAB, and the concrete recognition model results are verified based on simple scenes, complex scenes, and actual engineering scenes. Attached Figure Description

[0043] Figure 1 This is a technical roadmap of the concrete crack identification method provided by the present invention;

[0044] Figure 2 This is a schematic diagram of the confusion matrix provided by the present invention;

[0045] Figure 3 This is a flowchart of the self-training algorithm provided by the present invention;

[0046] Figure 4 This is a flowchart of the active learning algorithm provided by the present invention;

[0047] Figure 5 This is a flowchart of the active learning process combining a stacked convolutional encoder provided by the present invention;

[0048] Figure 6 (a) is a diagram showing the actual result of manual segmentation by the inventor;

[0049] Figure 6 (b) is the image segmentation result of the Mask RCNN of the present invention;

[0050] Figure 7 This is a comparison chart of the characteristics and results of the various models provided by this invention;

[0051] Figure 8 This is a flowchart of crack width measurement provided by the present invention;

[0052] Figure 9 This is a schematic diagram of crack width calculation provided by the present invention;

[0053] Figure 10 This is a schematic diagram of crack skeleton line calculation provided by the present invention;

[0054] Figure 11 This is a schematic diagram of the crack width calculation parameters provided by the present invention;

[0055] Figure 12 (a) is a mask image of the crack in this invention;

[0056] Figure 12 (b) is the edge pixel extraction image of the crack in this invention;

[0057] Figure 13 This invention provides a complex scene crack extraction and measurement diagram;

[0058] Figure 14 This is a crack correction mask recognition image for a real engineering scenario provided by the present invention. Detailed Implementation

[0059] The present invention will be further described below with reference to embodiments, which are intended only to provide a better understanding of the invention. Therefore, the examples given do not limit the scope of protection of the present invention.

[0060] This embodiment uses the detection of cracks on the concrete surface as an example for illustration, as follows: Figure 1 The process shown enables real-time detection of cracks on concrete surfaces by taking pictures of the concrete surface, which significantly reduces the labor intensity of concrete durability testing and improves testing efficiency and accuracy.

[0061] The concrete crack identification method based on synthetic datasets and semi-supervised learning of the present invention, in the concrete surface crack detection implementation case, includes the following steps:

[0062] Step 1: Collect the original dataset, perform supervised learning using multiple mainstream neural network models, and select a benchmark recognition model from them.

[0063] Specifically, based on six public datasets, 500 images with and without cracks were selected from each dataset, forming a comprehensive dataset of 3000 images. Supervised learning was performed using mainstream neural network models such as VGGNet, ResNet, Inception-ResNet, and SE-ResNet-18. Figure 2 The confusion matrix shown is related to the model's parameter count, accuracy, precision, recall, and F-value. β The scores and other metrics were used to test various mainstream neural network models, and SE-ResNet-18 was selected as the benchmark recognition model based on the test results.

[0064] The evaluation index formula is:

[0065] Precision = TP / (TP + FP)

[0066] Recall = TP / (TP + FN)

[0067] F β =(1+β) 2 )×(Precision×Recall) / (β 2×Precision + Recall).

[0068] Step 2: Due to the poor performance of the recognition model on the personal dataset, unlabeled data from the personal dataset was used. Based on semi-supervised learning, self-training, active learning, and stacked convolutional autoencoder active learning methods were designed to update the benchmark recognition model. Based on the recognition performance of the self-trained model, the stacked convolutional autoencoder active learning method was selected. (See [link to relevant documentation]). Figures 3-5 .

[0069] Step 3: Based on the updated baseline recognition model, classify the unlabeled data in the personal dataset.

[0070] The specific steps used in this chapter to train this network are as follows:

[0071] Dataset augmentation process: Select the top 1% of images that are difficult to label and accurately label these images. Augment the labeled images to increase both cracked and crack-free images to 1500 images, thus constructing a personal augmentation dataset.

[0072] Feature map acquisition process: A stacked convolutional autoencoder is trained on the personal augmentation dataset to obtain an 8×8×512 feature map.

[0073] The binary classification model training process is as follows: the features extracted by the autoencoder and the features extracted by the benchmark recognition model are concatenated in parallel, and the feature map size is reduced by global pooling to obtain a 1×1×1024 feature vector; this vector is then used as the input layer to train a single-layer binary classification model on the training set and the augmented personal reinforcement dataset.

[0074] Model update phase: The new model is resubmitted for annotation, and the model is updated after manual annotation.

[0075] Step 4: Select images with high confidence from the personal reinforcement dataset and perform pixel-level mask annotation on them. Based on the image and mask annotation results, use the semantic segmentation model evaluation metric MaskIoU to compare four semantic segmentation models: Big Law, FCN network, Pyramid Scene Parsing Network, and Mask RCNN. Figure 6 Image segmentation results from Mask RCNN. Figure 7 To compare the characteristics and results of each model, Mask R-CNN, which has the highest MaskIoU index, was selected as the crack segmentation model. The formula for the comparison index is as follows:

[0076] inter = ∑ (x，y)∈P (M C (x, y) + M G (x, y) = 2)

[0077] union = ∑ (x，y)∈P (M C (x, y)==1)+∑ (x，y)∈P (M G (x, y)==1)-inter

[0078] Mask_IoU = inter / union

[0079] In the specific context of this step, the label only represents cracks, and the mask is the binary classification result of crack pixels and non-crack pixels. A pixel identified as a crack is assigned a value of 1, and a pixel identified as a non-crack is assigned a value of 0; where M c To define the mask obtained from the recognized image, M G The mask obtained from the ground truth image.

[0080] The Loss values ​​for the Mask R-CNN loss function are as follows:

[0081] Loss = Loss Mask +Loss class +Loss Box .

[0082] Step 5: Perform pixel-level annotation on images with high confidence levels based on the Mask RCNN model.

[0083] Step 6: Based on the mask dataset segmented in Step 5 and the edge pixels of the crack, obtain the skeleton line of the crack, calculate the maximum width and average width of the crack, and perform error analysis on the crack width to obtain the concrete crack recognition model. (See [link to relevant documentation]). Figures 8-11 As shown.

[0084] Crack width recognition: Take an image and process the oblique photo into a frontal photo; input the converted frontal crack image into a trained Mask RCNN model to obtain a binary mask with crack pixels in the foreground and concrete surface in the background; define the two crack edges L1 and L2 as crack edges, and then define the width, maximum width, and average width as follows:

[0085] d i =min[(x i y i ), (x j y j )],(x i y i )∈L1, (x j y j )∈L2

[0086] d max =max(d i )

[0087] d Ave =sum(Mask) / L Ave

[0088] In Formula 7, L Ave The length of the crack skeleton line is determined by L1 and L2 as the edge.

[0089] Calculation of maximum and average crack width: Based on the definition of crack width above, and based on the extracted mask edge, replace any pixel on edge L1 with its outer edge corner point, and calculate the Euclidean distance from the corner point to the outer edge corner point of edge L2; after traversing all corner points on L2 for any pixel on L1, the minimum value of the calculated result is the crack width originating from that point.

[0090] w max =min[(x i y i ), (x j ·y j )],(x i y i )∈L1, (x j y j )∈L2.

[0091] Crack length calculation: A line is drawn connecting the midpoints of the widths of each point on the outer edge envelope, extending to the bounding rectangle of all crack pixels. The midpoints of the crack width lines are calculated based on L1 and L2 respectively, yielding the midpoints of the corresponding widths for each point in L1 and L2. All calculated midpoints are then connected to their nearest neighbors to obtain the crack skeleton lines. The total crack length is obtained by summing the lengths of the crack skeleton lines. The average crack width can be calculated using the following formula:

[0092] W Ave =γ×sum(Mask) / L Ave .

[0093] Maximum crack width error analysis: The error in maximum width mainly stems from the approximation of pixels in the crack width direction. Let the crack width direction be θ, and the actual crack width be D. Then, the theoretical difference in horizontal coordinates Δx and vertical coordinates Δy, and the actual difference in horizontal coordinates x and vertical coordinates y calculated are as follows:

[0094] Δx=Dcosθ, Δy=Dsinθ, 0°≤θ≤90°

[0095] Δx-0, 5λ<x<Δx+0.5λ, Δy-0.5λ<y<Δy+0.5λ

[0096] The theoretical error is calculated as follows:

[0097]

[0098] Further simplification yields:

[0099] .

[0100] Step 7, see Figures 12-14 The camera was calibrated using the Camera Calibrator toolbox in MATLAB. Then, the concrete crack recognition model was used to analyze cracks in simple, complex, and real-world engineering scenarios, thereby validating the concrete crack recognition model.

[0101] Simple scenario: Using a T-square, the actual size of each pixel is determined to be 0.016632mm × 0.016632mm. Single crack identification, bifurcation crack identification, and complex crack shape identification are then performed on cracks on a white wall. Based on the results read by the crack observation instrument and the results obtained in this invention, an error analysis of the maximum crack width is conducted.

[0102] Complex scenario: Images are taken of sub-blocks of the component. The actual area size of a single sub-block is 10cm × 10cm, and the actual size of a single pixel is 0.025mm × 0.025mm. After identifying the sub-blocks, they are stitched together to obtain a complete single crack. Based on the results read by the crack observation instrument and the results obtained in this invention, an error analysis of the maximum crack width is performed.

[0103] Real-world engineering scenario: Manual photography was conducted on the underside of a bridge in the suburbs of Shanghai. The bridge surface was divided into regions using 10cm x 10cm sub-blocks. After identification, the sub-blocks were stitched together to obtain a complete single-crack pattern. Based on the measured data, an error analysis was performed on the maximum crack width of this invention.

[0104] Step 8: Based on the camera's image of the structural surface, import the results into the concrete crack recognition model, divide each crack, and obtain the skeleton line, maximum width, and average width of each crack.

[0105] The features and advantages of the method provided by this invention are as follows: This invention considers using unlabeled data to perform semi-supervised learning updates on the pre-trained benchmark model, and then establishes an overall process for crack width detection based on the recognition results, thereby reducing manual inspection work in actual engineering.

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes made based on the description and drawings of the present invention are included within the scope of the present invention.

Claims

1. A method for identifying concrete cracks based on synthetic datasets and semi-supervised learning, characterized in that: Step 1: Collect the original dataset, perform supervised learning using multiple mainstream neural network models, and select a benchmark recognition model from them; Step 2: Update the benchmark recognition model based on a semi-supervised learning method; Step 3: Based on the updated baseline recognition model, classify the unlabeled data in the personal dataset; Step 4: Select images with high confidence and perform pixel-level mask annotations on them. Based on the image and mask annotation results, compare multiple semantic segmentation models and select the one with the highest MaskIoU index as the crack segmentation model. Step 5: Use the crack segmentation model to perform pixel-level annotation on images with high confidence. Step 6: Based on the mask dataset segmented in Step 5 and the edge pixels of the crack, obtain the skeleton line of the crack, calculate the maximum width and average width of the crack, and perform error analysis on the crack width to obtain the concrete crack recognition model. Step 7: Validate the concrete crack identification model; Step 8: Based on the camera's image of the structural surface, import the image into the concrete crack recognition model, divide each crack, and obtain the skeleton line, maximum width, and average width of each crack.

2. The concrete crack identification method based on synthetic datasets and semi-supervised learning according to claim 1, characterized in that: Step 1 specifically involves using six public datasets, from which 500 images (with and without cracks) are selected to form a comprehensive dataset of 3000 images. Supervised learning is then performed using VGGNet, ResNet, Inception-ResNet, and SE-ResNet-18 neural network models. A confusion matrix is ​​introduced in conjunction with the model's parameter count, accuracy, precision, recall, and F-axis. β The score index was used to test each neural network model, and SE-ResNet-18 was selected as the benchmark recognition model based on the test results.

3. The concrete crack identification method based on synthetic datasets and semi-supervised learning according to claim 1, characterized in that: In step 2, the benchmark recognition model is updated based on the unlabeled data in the personal dataset using an active learning method with stacked convolutional autoencoders.

4. The concrete crack identification method based on synthetic datasets and semi-supervised learning according to claim 1, characterized in that: In step 4, the crack segmentation model is the Mask R-CNN model with the highest MaskIoU index. The formula for the MaskIoU comparison index is: inter=∑ (x，y)∈P (M c (x,y)+M G (x,y)==2) union=∑ (x，y)∈P (M C (x,y)==1)+∑ (x，y)∈P (M G (x,y)==1)-inter Mask_IoU = (TP) / (TP+FP+FN).

5. The concrete crack identification method based on synthetic datasets and semi-supervised learning according to claim 1, characterized in that: In step 6, Crack width identification involves: capturing an image and converting an obliquely taken image to a frontal image; inputting the converted frontal crack image into a trained Mask RCNN model to obtain a binary mask with the crack pixels as the foreground and the concrete surface as the background; defining the two crack edges as L1 and L2, the width, maximum width, and average width are defined as follows: d i =min[(x i ,y i ),(x j ,y j )],(x i ,y i )∈L1,(x j ,y j )∈L2 d max =max(d i ) d Ave =sum(Mask) / L Ave In Formula 7, L Ave For the edge, L1 and L2 determine the length of the crack skeleton line; The maximum and average crack widths are calculated as follows: Based on the above definition of crack width, and using the extracted mask edges, any pixel on edge L1 is replaced by its outer edge corner point. The Euclidean distance from the corner point to the outer edge corner point of edge L2 is calculated. After traversing all corner points on L2 for any pixel on L1, the minimum value of the calculated result is the crack width originating from that point. w max =min[(x i ,y i ),(x j ,y j )],(x i ,y i )∈L1,(x j ,y j )∈L2 The crack length is calculated as follows: A line is drawn connecting the midpoints of the widths of each point on the outer edge envelope, extending to the bounding rectangle of all crack pixels. The midpoints of the crack width lines are calculated based on L1 and L2 respectively, yielding the midpoints of the corresponding widths for each point in L1 and L2. All calculated midpoints are then connected to their nearest neighbors to obtain the crack skeleton lines. The total crack length is obtained by summing the lengths of the crack skeleton lines. The average crack width can be calculated using the following formula: W Ave =γ×sum(Mask) / L Ave ; The maximum crack width error analysis is as follows: The error in the maximum width mainly stems from the approximation of pixels in the crack width direction. Let the crack width direction be θ, and the actual crack width be D. Then, the theoretical difference in horizontal coordinates Δx and vertical coordinates Δy, and the actual difference in horizontal coordinates x and vertical coordinates y calculated are as follows: Δx=Dcosθ, Δy=Dsinθ, 0°≤θ≤90° Δx-0.5λ<x<Δx+0.5λ, Δy-0.5λ<y<Δy+0.5λ The theoretical error is calculated as follows: Further simplification yields: 。 6. The concrete crack identification method based on synthetic datasets and semi-supervised learning according to claim 1, characterized in that: In step 7, the camera is calibrated using the Camera Calibrator toolbox in MATLAB, and the concrete recognition model results are verified based on simple scenes, complex scenes, and actual engineering scenes.

Images