A crack detection method based on automatic filling pseudo-labels
By automatically filling pseudo-labels and combining lightweight convolutional networks and Canny edge detection, the crack detection results are optimized, solving the problem of low detection accuracy under small data samples and achieving high-precision and robust crack detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUNAN BEIDOU MICROCHIP IND DEV CO LTD
- Filing Date
- 2022-11-10
- Publication Date
- 2026-05-29
AI Technical Summary
Existing deep learning-based crack detection methods are not good for small data samples, which can easily lead to false detections and false negatives, and the detection accuracy is not high.
We employ an automatic pseudo-label filling method to construct an object detector using a lightweight convolutional network, a feature fusion model, and a classification detector. We combine morphological processing and Canny edge detection to optimize the detection results and expand the data sample library using pseudo-labels to retrain the network model.
It improves the accuracy and robustness of crack detection, is applicable to different scenarios, can adaptively respond to object deformation, has strong generalization performance, and reduces the need for manually labeled data.
Smart Images
Figure CN115880224B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of crack detection technology, and in particular to a crack detection method based on automatic pseudo-label filling. Background Technology
[0002] Cracks appear in bridges, tunnels, buildings, and other structures due to prolonged use, ground subsidence, or substandard quality. Once these cracks exceed the safety threshold, they may cause structural instability or even collapse, posing serious safety hazards. Therefore, it is extremely important to design an intelligent crack detection method to avoid unnecessary harm.
[0003] Early intelligent crack detection methods primarily relied on background frame difference, correlation filtering, incremental learning, and sparse representation to detect cracks in the target region. While these methods were fast, their accuracy was low. With the improvement of computing power, especially the development of Graphics Processing Units (GPUs), deep learning has been widely applied in object detection. The way objects are represented has also changed. Compared to the object detection methods mentioned above that rely on manual feature extraction or sliding windows to find potential candidate regions, deep learning-based methods, utilizing deep convolutional networks, can more effectively extract complete object features, making them more suitable for object detection tasks and improving detection accuracy. However, deep learning-based detection methods tend to be large-scale models, requiring a large amount of labeled data samples for training. This is not conducive to detection methods based on small data samples, easily leading to false positives and false negatives. Summary of the Invention
[0004] Therefore, the purpose of this invention is to provide a crack detection method based on automatic pseudo-label filling, which can reliably identify the target to be detected, automatically expand the label data sample, and improve the target detection accuracy.
[0005] The technical solution adopted in this invention is as follows:
[0006] A crack detection method based on automatically filled pseudo-labels, the method comprising the following steps:
[0007] Step 1: Obtain multiple crack images from different scenes, annotate the cracks in the images using the labelimg annotation tool, and build a dataset sample library;
[0008] Step 2: Obtain network model weights based on training data samples from a lightweight object detector;
[0009] Step 3: Based on the weights of the trained network model, perform a one-stage crack detection on the newly acquired image database and select high-confidence samples and suboptimal samples.
[0010] Step 4: Set a mask to obtain images within the suboptimal sample area, and use morphological processing and correlation filtering methods to enhance image recognition.
[0011] Step 5: Use the Canny edge detection operator to perform two-stage crack detection on the suboptimal sample and optimize the detection results;
[0012] Step 6: Obtain the final detection results. By using the high-confidence samples from the first stage and the optimized samples from the second stage as pseudo-labels, construct a new data sample library from the data samples with the same labels, retrain the network model, and use it as the weight file for the new first-stage detection.
[0013] Furthermore, in step 2, the lightweight target detector is constructed from three parts: a lightweight convolutional network, a feature fusion model, and a classification detector.
[0014] Lightweight convolutional networks are used to learn image features. They are implemented by stacking 12 layers of convolution with a kernel of 3×3 and a stride of 1, and max pooling with a kernel of 3×3 and a stride of 2. The output of each convolution layer is subjected to non-linear activation and normalization.
[0015] The feature fusion model is used to fuse the outputs of the target appearance feature layer and the deep semantic feature layer. The target appearance feature layer is a shallow convolutional layer of a lightweight convolutional network, which can obtain the target's position information and contour edge information. The deep semantic feature layer is a deep convolutional layer of a lightweight convolutional network, which can obtain the target's texture, color, and category attribute information. The fusion process is implemented by bicubic interpolation and matrix operations. Bicubic interpolation is used to unify the resolution of the appearance feature layer and the deep semantic feature layer. The implementation process is as follows:
[0016]
[0017] Where (i,j) are the coordinates of the pixel in the original image, v and u are the row offset and column offset, respectively, x and y are used to obtain the coordinates of the 16 nearest neighbors of (i,j), which range from {-1,0,1,2}, and K(·) is the interpolation formula, expressed as follows:
[0018]
[0019] Where 'a' is a weighting coefficient with a value of -0.75.
[0020] The features output from the appearance feature layer after unifying the resolution are normalized and used as correction units to supplement the target location information and edge contour information missing in the deep semantic feature layer. The implementation process is as follows:
[0021] F(u ij )=F1(u ij )·[σ(F2(u ij))+1]
[0022] Among them, u ij F1 and F2 are the image feature response values, respectively, and the depths at a uniform resolution.
[0023] The features output by the semantic feature layer and the features output by the appearance feature layer, where σ is the sigmoid activation function;
[0024] The classification detector is used for final result prediction. By setting anchor boxes of three different sizes (large, medium, and small), and using a Logistic classifier and non-maximum suppression to predict the class, the optimal prediction box and class confidence are obtained.
[0025] Furthermore, in step 3, high-confidence samples and suboptimal samples are screened, and the prediction results of the lightweight target detector are processed as follows:
[0026] High-confidence samples: The confidence level of the predicted object class within the bounding box ranges from 0.95 to 1.00;
[0027] Second-best samples: The confidence level of the predicted object class within the bounding box ranges from 0.85 to 0.95.
[0028] Furthermore, in step 5, the processing steps to enhance the image recognition of suboptimal samples and improve the confidence of predicted samples include:
[0029] a) Set a mask and obtain the image within the suboptimal sample area based on the position of the suboptimal sample border;
[0030] b) Image grayscale conversion: The image is converted to grayscale using a weighted average value.
[0031] c) Image enhancement: Image enhancement is achieved by using grayscale histogram equalization.
[0032] d) Image denoising: Wiener filtering is used to denoise the image;
[0033] e) Crack detection: The Canny operator is used to perform edge detection on the denoised image to obtain the crack target.
[0034] f) Result regression: Based on the crack target, the crack region is identified through expansion and corrosion morphological features. Finally, the bounding box is predicted by calculating the outer rectangle of the crack target through regression.
[0035] Furthermore, in step 6, the method for obtaining the final crack prediction result is as follows:
[0036] The bounding box positions predicted for the suboptimal samples in the first-stage detection results are corrected based on the second-stage detection results, calculated as follows:
[0037] x1=x1+x′1
[0038] y1=y1+y′1
[0039] x² = x² - x′²
[0040] y2=y2-y′2
[0041] Where (x1,y1) and (x2,y2) are the top left and bottom right corners of the suboptimal sample region obtained in the mask, respectively, and (x'1,y'1) and (x'2,y'2) are the top left and bottom right corners of the two-stage predicted bounding box;
[0042] The bounding boxes of the high-confidence predicted samples in the first-stage detection and the bounding boxes of the suboptimal predicted samples in the second-stage correction are used as the final prediction results.
[0043] Furthermore, in step 6, the method for training the new network model weight file is as follows:
[0044] a) Use the high-confidence prediction samples from the first-stage prediction results and the suboptimal samples from the second-stage optimization as crack labels to expand the original data sample library.
[0045] b) Randomize the expanded data sample library and repeat step 2 to randomly select images from the sample library for training until all images in the sample library have been trained.
[0046] Compared with the prior art, the beneficial effects of the present invention are:
[0047] 1. Robustness: The network model comprehensively considers both deep and shallow surface features of the data samples, enabling a more complete representation of the features of the target object. A multi-scale classification detector is constructed, capable of adaptively responding to deformation changes of the object being detected. A two-stage detection optimizer is built, using traditional image enhancement and edge detection methods to optimize the results of the first-stage prediction, thereby improving the class confidence and detection accuracy of the prediction results. Ultimately, through the above detection scheme, this invention achieves superior detection performance.
[0048] 2. Generalization: This invention automatically fills in pseudo-labels as training data samples, making it suitable for crack detection in different scenarios. The weights of the trained network model can be attached to other data samples for further training, allowing for continuous iterative optimization of the network model. Attached Figure Description
[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0050] Figure 1 This is a schematic diagram of the overall process of a crack detection method based on automatic pseudo-label filling according to an embodiment of the present invention.
[0051] Figure 2 This is a schematic diagram of the first-stage processing result of a crack detection method based on automatic pseudo-label filling according to an embodiment of the present invention.
[0052] Figure 3 This is an example diagram of the two-stage processing results of a crack detection method based on automatic pseudo-label filling according to an embodiment of the present invention. Detailed Implementation
[0053] The principles and features of the present invention are described below with reference to the accompanying drawings. The listed embodiments are only used to explain the present invention and are not intended to limit the scope of the present invention.
[0054] Reference Figure 1 , Figure 2 and Figure 3 This invention provides a crack detection method based on automatic pseudo-label filling, the method comprising the following steps:
[0055] Step 1: Obtain multiple crack images from different scenes, annotate the cracks in the images using the labelimg annotation tool, and build a dataset sample library;
[0056] Step 2: Obtain network model weights based on training data samples from a lightweight object detector;
[0057] Step 3: Based on the weights of the trained network model, perform a one-stage crack detection on the newly acquired image database and select high-confidence samples and suboptimal samples.
[0058] Step 4: Set a mask to obtain images within the suboptimal sample area, and use morphological processing and correlation filtering methods to enhance image recognition.
[0059] Step 5: Use the Canny edge detection operator to perform two-stage crack detection on the suboptimal sample and optimize the detection results;
[0060] Step 6: Obtain the final detection results. By using the high-confidence samples from the first stage and the optimized samples from the second stage as pseudo-labels, construct a new data sample library from the data samples with the same labels, retrain the network model, and use it as the weight file for the new first-stage detection.
[0061] For example, in step 1, the source of the image is obtained, mainly based on camera shooting or web crawling, and the obtained image is expanded by methods such as cropping, rotating, and adding noise, and the cracks in the image are labeled using the labelimg annotation tool.
[0062] In step 2, the lightweight object detector is constructed from three parts: a lightweight convolutional network, a feature fusion model, and a classification detector.
[0063] Lightweight convolutional networks are used to learn image features. They are implemented by stacking 12 layers of convolution with a kernel of 3×3 and a stride of 1, and max pooling with a kernel of 3×3 and a stride of 2. The output of each convolution layer is subjected to non-linear activation and normalization.
[0064] The feature fusion model is used to fuse the outputs of the target appearance feature layer and the deep semantic feature layer. The target appearance feature layer is a shallow convolutional layer of a lightweight convolutional network, which can obtain the target's position information and contour edge information. The deep semantic feature layer is a deep convolutional layer of a lightweight convolutional network, which can obtain the target's texture, color, and category attribute information. The fusion process is implemented by bicubic interpolation and matrix operations. Bicubic interpolation is used to unify the resolution of the appearance feature layer and the deep semantic feature layer. The implementation process is as follows:
[0065]
[0066] Where (i,j) are the coordinates of the pixel in the original image, v and u are the row offset and column offset, respectively, x and y are used to obtain the coordinates of the 16 nearest neighbors of (i,j), which range from {-1,0,1,2}, and K(·) is the interpolation formula, expressed as follows:
[0067]
[0068] Where 'a' is a weighting coefficient with a value of -0.75.
[0069] The features output from the appearance feature layer after unifying the resolution are normalized and used as correction units to supplement the target location information and edge contour information missing in the deep semantic feature layer. The implementation process is as follows:
[0070] F(u ij )=F1(u ij )·[σ(F2(u ij ))+1]
[0071] Among them, u ij F1 and F2 are the image feature response values, respectively, and the depths at a uniform resolution.
[0072] The features output by the semantic feature layer and the features output by the appearance feature layer, where σ is the sigmoid activation function;
[0073] The classification detector is used for final result prediction. By setting anchor boxes of three different sizes (large, medium, and small), and using a Logistic classifier and non-maximum suppression to predict the class, the optimal prediction box and class confidence are obtained.
[0074] In step 3, high-confidence samples and suboptimal samples are screened, and the prediction results of the lightweight target detector are processed as follows:
[0075] High-confidence samples: The confidence level of the predicted object class within the bounding box ranges from 0.95 to 1.00;
[0076] Second-best samples: The confidence level of the predicted object class within the bounding box ranges from 0.85 to 0.95.
[0077] In step 5, the processing steps to enhance the image recognition of suboptimal samples and improve the confidence of predicted samples include:
[0078] g) Set a mask and obtain the image within the suboptimal sample area based on the position of the suboptimal sample border;
[0079] h) Image grayscale conversion: The image is converted to grayscale using a weighted average value;
[0080] i) Image enhancement: Image enhancement is achieved by using grayscale histogram equalization.
[0081] j) Image denoising: Wiener filtering is used to denoise the image;
[0082] k) Crack detection: The Canny operator is used to detect the edges of the denoised image to obtain the crack target;
[0083] l) The results are regressed. Based on the crack target, the crack area is identified by expansion and corrosion morphological features. Finally, the bounding rectangle of the crack target is calculated and the bounding box is predicted by regression.
[0084] In step 6, the method for obtaining the final crack prediction result is as follows:
[0085] The bounding box positions predicted for the suboptimal samples in the first-stage detection results are corrected based on the second-stage detection results, calculated as follows:
[0086] x1=x1+x′1
[0087] y1=y1+y′1
[0088] x² = x² - x′²
[0089] y2=y2-y′2
[0090] Where (x1,y1) and (x2,y2) are the top left and bottom right corners of the suboptimal sample region obtained in the mask, respectively, and (x'1,y'1) and (x'2,y'2) are the top left and bottom right corners of the two-stage predicted bounding box;
[0091] The bounding boxes of the high-confidence predicted samples in the first-stage detection and the bounding boxes of the suboptimal predicted samples in the second-stage correction are used as the final prediction results.
[0092] In step 6, the method for training the new network model weight file is as follows:
[0093] c) Use the high-confidence prediction samples from the first-stage prediction results and the suboptimal samples from the second-stage optimization as crack labels to expand the original data sample library.
[0094] d) Randomize the expanded data sample library and repeat step 2 to randomly select images from the sample library for training until all images in the sample library have been trained.
[0095] This invention proposes a crack detection method based on automatic pseudo-label filling. By integrating the detection results from two stages, it can quickly and accurately detect cracks in images, exhibiting strong robustness. Furthermore, this invention can provide early warning before cracks reach a safe threshold, avoiding unnecessary losses. In addition, this invention uses accurate detection results from different scenarios as new crack labels to fill the training data, making it applicable to crack detection in multiple scenarios. It eliminates the need for manual crack labeling to expand the data sample, demonstrating strong generalization performance.
[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A crack detection method based on automatically filled pseudo-labels, the method comprising the following steps: Step 1: Obtain multiple crack images from different scenes, annotate the cracks in the images using the labelimg annotation tool, and build a dataset sample library; Step 2: Based on the data samples in the training dataset of the lightweight object detector, obtain the network model weights. Specifically, the lightweight object detector is constructed from three parts: a lightweight convolutional network, a feature fusion model, and a classification detector. Lightweight convolutional networks are used to learn image features. They are implemented by stacking 12 layers of convolution with a kernel of 3×3 and a stride of 1, and max pooling with a kernel of 3×3 and a stride of 2. The output of each convolution layer is subjected to non-linear activation and normalization. The feature fusion model is used to fuse the outputs of the target appearance feature layer and the deep semantic feature layer. The target appearance feature layer is a shallow convolutional layer of a lightweight convolutional network, which can obtain the target's position information and contour edge information. The deep semantic feature layer is a deep convolutional layer of a lightweight convolutional network, which can obtain the target's texture, color, and category attribute information. The fusion process is implemented by bicubic interpolation and matrix operations. Bicubic interpolation is used to unify the resolution of the appearance feature layer and the deep semantic feature layer. The implementation process is as follows: in, These are the coordinates of the pixels in the original image. , These are the row offset and column offset, respectively. , Used to obtain The coordinates of the next 16 pixels, ranging from... , The interpolation formula is expressed as follows: in, This is the weighting factor, with a value of -0.
75. The features output from the appearance feature layer after unifying the resolution are normalized and used as correction units to supplement the target location information and edge contour information missing in the deep semantic feature layer. The implementation process is as follows: in, Image feature response value, , Each represents a depth of a uniform resolution. The features output by the semantic feature layer and the features output by the appearance feature layer It is the sigmoid activation function; The classification detector is used to predict the final result. By setting anchor boxes of three different sizes (large, medium, and small), and using a Logistic classifier and non-maximum suppression to predict the class, the optimal prediction box and class confidence are obtained. Step 3: Based on the weights of the trained network model, perform a one-stage crack detection on the newly acquired image database and select high-confidence samples and suboptimal samples. Step 4: Set a mask to obtain images within the suboptimal sample region, and enhance image recognition using morphological processing and correlation filtering methods. This includes the following steps: a) Set a mask and obtain the image within the suboptimal sample area based on the position of the suboptimal sample border; b) Image grayscale conversion: The image is converted to grayscale using a weighted average value. c) Image enhancement: Image enhancement is achieved by using grayscale histogram equalization. d) Image denoising: Wiener filtering is used to denoise the image; Step 5: Use the Canny edge detection operator to perform two-stage crack detection on the suboptimal sample processed in Step 4 to optimize the detection results; Step 6: Obtain the final detection results. Take the high-confidence samples from the first-stage detection results and the suboptimal samples after the second-stage optimization as pseudo-labels. Merge these pseudo-labels with the original labeled data samples to build a new data sample library. Retrain the network model and use it as the weight file for the new first-stage detection.
2. The crack detection method based on automatic pseudo-label filling according to claim 1, characterized in that, In step 3, high-confidence samples and suboptimal samples are screened, and the prediction results of the lightweight target detector are processed as follows: High-confidence samples: The confidence level of the predicted object class within the bounding box ranges from 0.95 to 1.00; Second-best samples: The confidence level of the predicted object class within the bounding box ranges from 0.85 to 0.
95.
3. The crack detection method based on automatic pseudo-label filling according to claim 2, characterized in that, In step 5, the processing steps to enhance the image recognition of suboptimal samples and improve the confidence of predicted samples include: e) Crack detection: The Canny operator is used to perform edge detection on the denoised image to obtain the crack target. f) Result regression: Based on the crack target, the crack region is identified through expansion and corrosion morphological features. Finally, the bounding box is predicted by calculating the outer rectangle of the crack target through regression.
4. The crack detection method based on automatic pseudo-label filling according to claim 3, characterized in that, In step 6, the method for obtaining the final crack prediction result is as follows: The bounding box positions predicted for the suboptimal samples in the first-stage detection results are corrected based on the second-stage detection results, calculated as follows: in, , These are the top left and bottom right corners of the suboptimal sample region obtained from the mask, respectively. , The top left and bottom right corners of the two-stage prediction bounding box; The bounding boxes of the high-confidence predicted samples in the first-stage detection and the bounding boxes of the suboptimal predicted samples in the second-stage correction are used as the final prediction results.
5. The crack detection method based on automatic pseudo-label filling according to claim 4, characterized in that, In step 6, the method for training the new network model weight file is as follows: a) Use the high-confidence prediction samples from the first-stage prediction results and the suboptimal samples from the second-stage optimization as crack labels to expand the original data sample library. b) Randomize the expanded data sample library and repeat step 2 to randomly select images from the sample library for training until all images in the sample library have been trained.