Concrete defect detection method and system based on deep learning
Through the improved YOLOv9 and SAM2 modules combined with data preprocessing and loss optimization, the problems of low efficiency, insufficient accuracy and poor adaptability of concrete defect detection in the prior art are solved, and high-precision and real-time defect detection and tracking are achieved, which is suitable for automated monitoring of infrastructure such as bridges and tunnels.
Patent Information
- Application Number
- CN202510545778.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-28
- Publication Date
- 2025-08-12
AI Technical Summary
The existing deep learning networks have problems such as low detection efficiency, insufficient accuracy, and poor adaptability to complex backgrounds and small defects in concrete defect detection. In particular, the YOLO series networks perform poorly in complex backgrounds, the U-Net network is susceptible to noise interference in concrete defect detection, and Faster R-CNN training costs are high and the adaptability is limited.
The improved YOLOv9 object detection module and SAM2 segmentation tracking module are adopted, combined with data preprocessing, dynamic weighted loss optimization and multi-scale feature extraction, and the improved YOLOv9 object detection module detects concrete defects in real time, and the improved SAM2 segmentation tracking module performs pixel-level segmentation and cross-frame feature matching to achieve continuous tracking of defects.
It significantly improves the accuracy and real-time nature of concrete defect detection, effectively overcomes the problems of missed detection and missed detection in complex backgrounds, realizes pixel-level segmentation of defects and continuous cross-frame tracking, and provides a high robustness and low latency automated monitoring solution.
Smart Images
Figure CN120471850A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a concrete defect detection method and system based on deep learning. Background Art
[0002] With the rapid development of infrastructure, concrete is widely used in various fields, including roads, bridges, tunnels, dams, and housing construction. However, due to various factors, including material properties, construction quality, and environmental factors, concrete structures inevitably suffer from defects such as concrete flaws. These defects not only affect the appearance and durability of concrete structures but can also seriously threaten their structural safety and service life. Therefore, timely and accurate identification and tracking of concrete defects is crucial to ensuring the safe operation of infrastructure.
[0003] Traditional concrete defect detection methods primarily include visual inspection, ultrasonic testing, steel needle testing, penetrant testing, and infrared thermal imaging. These methods each have their own advantages and disadvantages, but they all suffer from low efficiency, strong subjectivity, and insufficient accuracy. With the rapid development of computer vision and deep learning technologies, concrete defect detection methods based on image and video processing have become a research hotspot. Deep learning networks, particularly convolutional neural networks, have achieved remarkable results in image segmentation and object detection due to their powerful feature extraction and pattern recognition capabilities. However, existing deep learning network technologies still face challenges in concrete defect detection. For example, while YOLO networks offer real-time performance and accuracy, they may perform poorly in detecting complex backgrounds or small concrete defects. U-Net networks, while commonly used in medical image segmentation, can be affected by noise and complex backgrounds when detecting concrete defects. Object detection models such as Faster R-CNN, while offering high localization accuracy, are expensive to train and have limited adaptability to the irregular shapes and varying widths of concrete defects. Summary of the Invention
[0004] In order to solve the technical problems existing in the background technology, the present invention proposes a concrete defect detection method and system based on deep learning.
[0005] The present invention proposes a method for detecting concrete defects based on deep learning, comprising the following steps:
[0006] S1: collecting a video image sequence containing concrete defects from the surface of the infrastructure, and preprocessing the video image sequence to obtain a first sequence;
[0007] S2: The first sequence is input into a pre-trained joint neural network model to generate a final result containing defect location coordinates, category labels, unique tracking numbers, and cross-frame trajectories, and the final result is superimposed on the video image sequence in the form of visual annotations.
[0008] Preferably, the preprocessing specifically includes:
[0009] Adding a Gaussian noise matrix with a mean of 0 and a standard deviation of 50 to the video image sequence to obtain a first video sequence;
[0010] Convert the first video sequence from RGB space to HSV space, randomly adjust the hue, saturation and brightness, and then convert it back to RGB space to obtain a second video sequence;
[0011] A smooth deformation field is generated for the second video sequence by using a Gaussian function, and the image is deformed by using bilinear interpolation to obtain the first sequence.
[0012] Preferably, the joint neural network model includes an improved YOLOv9 target detection module and an improved SAM2 segmentation and tracking module; the improved YOLOv9 target detection module is used to detect concrete defects in video frames in real time and output defect categories, locations and detection frames; the improved SAM2 segmentation and tracking module performs pixel-level segmentation on defects based on the detection frames output by the improved YOLOv9 target detection module, and realizes continuous tracking of defects through cross-frame feature matching.
[0013] Preferably, the improvement process of the improved YOLOv9 target detection module specifically includes:
[0014] The default CIoU loss is replaced by DIoU loss, and the DIoU loss formula is:
[0015]
[0016] Where d is the target center point distance, α is the aspect ratio weight;
[0017] The k-nearest neighbor algorithm is used to calculate the anchor parameters that are suitable for the size of concrete defects;
[0018] In the early stage of training, the Adam optimizer is used to accelerate convergence, and in the later stage, it switches to stochastic gradient descent and uses the learning rate scheduler to smooth the transition.
[0019] Preferably, the improvement process of the improved SAM2 segmentation and tracking module specifically includes:
[0020] The pre-trained Hiera model is fine-tuned based on the preset concrete defect dataset. A dynamic weighted combination of cross entropy loss and Dice loss is used to balance the class imbalance. The loss function is expressed as:
[0021] L=λ1L CE +λ2L Dice ;
[0022] Among them, the cross entropy loss Dice loss y i is the true label; is the predicted probability of the model; ∈ is a small constant to prevent division by zero;
[0023] Insert a Gabor filter layer into the shallow encoder of the Hiera model;
[0024] An expanded self-attention mechanism with a dilation factor d is introduced to expand the receptive field to capture multi-scale defect features.
[0025] Preferably, before step S1, the method further includes:
[0026] After preprocessing the video image sequence used for concrete defect detection training, images with a size larger than 800×800 are directly used as training samples. For images with a size smaller than 800×800 in the first sequence, four images are randomly selected using the Mosaic method, compressed to 400×400 pixels, and then spliced into 800×800 training samples. The labels are adjusted to match the positions of the new images to serve as training samples, thereby obtaining a training sample set.
[0027] Preferably, the training sample set is divided into a training set, a validation set and a test set; the ratio of the training set, the validation set and the test set is 7:2:1.
[0028] Preferably, before step S1, the method further includes:
[0029] Construct a tracking list, and configure a one-to-one mapping relationship between the defect unique tracking number, defect category, detection box coordinates, segmentation mask, trajectory history, and status mark in the tracking list.
[0030] Preferably, the cross-frame feature matching specifically includes:
[0031] When the improved YOLOv9 object detection module detects a new defect, it assigns a globally unique tracking number to it and binds the new defect to the detection box and segmentation mask;
[0032] In subsequent frames, the intersection-over-union ratio and feature similarity of the current frame defect segmentation mask and the historical trajectory mask are calculated. If the intersection-over-union ratio and feature similarity meet the preset threshold, the tracking number is inherited; otherwise, it is considered a new defect;
[0033] If a defect is not matched for n consecutive frames, it is marked as "disappeared" and removed from the tracking list, where n≥5.
[0034] The present invention proposes a deep learning-based concrete defect detection system, comprising:
[0035] a data acquisition module for collecting a video image sequence containing concrete defects from a surface of the infrastructure and preprocessing the video image sequence to obtain a first sequence;
[0036] a data processing module, configured to input the first sequence into a pre-trained joint neural network model to generate a final result including defect location coordinates, category labels, unique tracking numbers, and cross-frame trajectories;
[0037] The output module is used to overlay the final results onto the video image sequence in the form of visual annotations.
[0038] In this invention, the proposed deep learning-based concrete defect detection method and system, through an improved YOLOv9 and SAM2 joint framework, combined with data preprocessing, dynamic weighted loss optimization and multi-scale feature extraction, significantly improves the accuracy and real-time performance of concrete defect detection, effectively overcomes the problems of missed detection and false detection in complex backgrounds of traditional methods, and realizes pixel-level segmentation and continuous tracking of defects across frames, providing a highly robust and low-latency solution for the automated monitoring of surface defects in infrastructure such as bridges and tunnels. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 This is a schematic diagram of the workflow of a deep learning-based concrete defect detection method proposed in the present invention;
[0040] Figure 2 This is a structural diagram of the training process of the improved Yolov9 target detection module of the deep learning-based concrete defect detection method proposed in the present invention;
[0041] Figure 3 This is a schematic diagram of the processing structure of the modified SAM2 image encoder of the concrete defect detection method based on deep learning proposed in the present invention;
[0042] Figure 4 This is a flowchart of concrete defect detection, segmentation and tracking of a concrete defect detection method based on deep learning proposed by the present invention;
[0043] Figure 5 This is a schematic diagram of the system architecture of a deep learning-based concrete defect detection system proposed in the present invention. DETAILED DESCRIPTION
[0044] Reference Figure 1-5 The present invention proposes a method for detecting concrete defects based on deep learning, comprising the following steps:
[0045] S00: After preprocessing the video image sequence used for concrete defect detection training, images with a size larger than 800×800 are directly used as training samples. For images with a size smaller than 800×800 in the first sequence, four images are randomly selected using the Mosaic method, compressed to 400×400 pixels, and then spliced into 800×800 training samples. The labels are adjusted to match the positions of the new images to serve as training samples, thereby obtaining a training sample set.
[0046] Specifically, a mosaic method is used to stitch four images into one, generating new training samples. Each sample is composed of four stitched images. Four original images are randomly selected and each is compressed or stretched to a size of 400×400 pixels. These four images are then placed at the four corners of a new 800×800 image, forming a 2×2 grid to ensure seamless stitching in the new image. Finally, the corresponding defect label for each image is adjusted to correspond to its location in the new image, ensuring label accuracy and consistency.
[0047] In this embodiment, the training sample set is divided into a training set, a validation set, and a test set; the ratio of the training set, validation set, and test set is 7:2:1.
[0048] S01: Construct a tracking list, in which a one-to-one mapping relationship between a defect's unique tracking number, defect category, detection frame coordinates, segmentation mask, trajectory history, and status mark is configured.
[0049] S1: A video image sequence containing concrete defects is collected from the surface of the infrastructure, and the video image sequence is preprocessed to obtain a first sequence.
[0050] In this embodiment, the preprocessing specifically includes: adding a Gaussian noise matrix with a mean of 0 and a standard deviation of 50 to the video image sequence to obtain a first video sequence; converting the first video sequence from the RGB space to the HSV space, randomly adjusting the hue, saturation, and brightness, and then converting it back to the RGB space to obtain a second video sequence; generating a smooth deformation field for the second video sequence using a Gaussian function, and deforming the image using bilinear interpolation to obtain the first sequence.
[0051] Specifically, the preprocessed image dataset, or the first sequence, was fed into the improved network for 400 training cycles. Starting from the first training cycle, the Adam optimizer was used for rapid convergence, with a learning rate of 0.01. Because the Adam optimizer excels at handling high-dimensional sparse gradients, it helps the model quickly learn effective feature representations. When the validation set loss stopped decreasing for a long time, stochastic gradient descent (SGD) was used for subsequent training. Before switching to SGD, the Adam learning rate was gradually reduced using a learning rate scheduler to ensure that Adam did not generate excessive updates in the later stages of training, thus ensuring a smoother transition. When switching to SGD, the learning rate was set to 0.001, the momentum to 0.937, the weight decay coefficient to 0.0005, and the training batch size to 16. Training continued until the model reached 400 epochs. To achieve optimal generalization and prevent overfitting, early stopping was introduced during training.
[0052] S2: The first sequence is input into a pre-trained joint neural network model to generate a final result containing defect location coordinates, category labels, unique tracking numbers, and cross-frame trajectories, and the final result is superimposed on the video image sequence in the form of visual annotations.
[0053] In this embodiment, the joint neural network model includes an improved YOLOv9 target detection module and an improved SAM2 segmentation and tracking module; the improved YOLOv9 target detection module is used to detect concrete defects in video frames in real time and output the defect category, location and detection frame; the improved SAM2 segmentation and tracking module performs pixel-level segmentation on the defects based on the detection frame output by the improved YOLOv9 target detection module, and realizes continuous tracking of the defects through cross-frame feature matching.
[0054] In this embodiment, the improvement process of the improved YOLOv9 target detection module specifically includes:
[0055] Replace the default CIoU loss with DIoU loss. The DIoU loss formula is:
[0056]
[0057] Where d is the target center point distance, α is the aspect ratio weight;
[0058] The k-nearest neighbor algorithm is used to calculate the anchor parameters that are suitable for the size of concrete defects;
[0059] In the early stage of training, the Adam optimizer is used to accelerate convergence, and in the later stage, it switches to stochastic gradient descent and uses the learning rate scheduler to smooth the transition.
[0060] In this embodiment, the improvement process of the improved SAM2 segmentation and tracking module specifically includes:
[0061] The pre-trained Hiera model is fine-tuned based on the preset concrete defect dataset. A dynamic weighted combination of cross entropy loss and Dice loss is used to balance the class imbalance. The loss function is expressed as:
[0062] L=λ1L CE +λ2L Dice ;
[0063] Among them, the cross entropy loss Dice loss y i is the true label; is the predicted probability of the model; ∈ is a small constant to prevent division by zero;
[0064] Insert a Gabor filter layer into the shallow encoder of the Hiera model;
[0065] An expanded self-attention mechanism with a dilation factor d is introduced to expand the receptive field to capture multi-scale defect features.
[0066] Specifically, the expression of the Gabor filter layer is:
[0067]
[0068] Among them, λ is the wavelength of the filter, which determines the filter's response to different frequencies; θ is the direction of the filter, which controls the angle of the filter; ψ is the phase offset, which controls the phase of the filter; σ is the standard deviation of the Gaussian envelope, which controls the scale of the filter; γ is the spatial aspect ratio, which controls the elliptical shape of the filter; (x′, y′) is the coordinate after rotating the input coordinate system.
[0069] Specifically, the expanded self-attention mechanism is as follows:
[0070]
[0071] Among them, M(d) is a mask matrix with a dilation factor, which helps to learn defect features across scales; Q, K, and V are Query, Key, and Value matrices, respectively.
[0072] In this embodiment, cross-frame feature matching specifically includes:
[0073] When the improved YOLOv9 object detection module detects a new defect, it assigns a globally unique tracking number to it and binds the new defect to the detection box and segmentation mask;
[0074] In subsequent frames, the intersection-over-union ratio and feature similarity of the current frame defect segmentation mask and the historical trajectory mask are calculated. If the intersection-over-union ratio and feature similarity meet the preset threshold, the tracking number is inherited; otherwise, it is considered a new defect.
[0075] If a defect is not matched for n consecutive frames, it is marked as "disappeared" and removed from the tracking list, where n≥5.
[0076] Specifically, the final concrete defect automatic detection, segmentation and tracking network is constructed as follows: Figure 4 As shown, combining the advantages of the improved YOLOv9 and improved SAM2, the system achieves automatic detection, image segmentation, and continuous tracking of concrete defects in video image sequences. The improved YOLOv9's detection prompt coordinates and corresponding labels serve as prompts and labels for the improved SAM2, which continuously segments and tracks the video image sequence. Simultaneously, the improved YOLOv9 continuously detects objects in the video frames and outputs detection boxes. When encountering a new concrete defect, it outputs a new detection box and number, allowing the improved SAM2 to add the new defect segmentation and tracking object. This achieves automated detection, segmentation, and tracking of concrete defects in video image sequences.
[0077] Reference Figure 1-5 The present invention proposes a deep learning-based concrete defect detection system, comprising:
[0078] a data acquisition module for collecting a video image sequence containing concrete defects from a surface of the infrastructure and preprocessing the video image sequence to obtain a first sequence;
[0079] a data processing module, configured to input the first sequence into a pre-trained joint neural network model to generate a final result including defect location coordinates, category labels, unique tracking numbers, and cross-frame trajectories;
[0080] The output module is used to overlay the final results onto the video image sequence in the form of visual annotations.
[0081] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.
Claims
1. A concrete defect detection method based on deep learning, characterized in that: The following steps are involved: S1: collecting a video image sequence containing concrete defects from the surface of the infrastructure, and preprocessing the video image sequence to obtain a first sequence; S2: The first sequence is input into a pre-trained joint neural network model to generate a final result containing defect location coordinates, category labels, unique tracking numbers, and cross-frame trajectories, and the final result is superimposed on the video image sequence in the form of visual annotations.
2. The method for detecting concrete defects based on deep learning according to claim 1, characterized in that: The pretreatment specifically includes: Adding a Gaussian noise matrix with a mean of 0 and a standard deviation of 50 to the video image sequence to obtain a first video sequence; Convert the first video sequence from RGB space to HSV space, randomly adjust the hue, saturation and brightness, and then convert it back to RGB space to obtain a second video sequence; A smooth deformation field is generated for the second video sequence by using a Gaussian function, and the image is deformed by using bilinear interpolation to obtain the first sequence.
3. The method for detecting concrete defects based on deep learning according to claim 1, characterized in that: The joint neural network model includes an improved YOLOv9 target detection module and an improved SAM2 segmentation and tracking module; the improved YOLOv9 target detection module is used to detect concrete defects in video frames in real time and output the defect category, location and detection frame; the improved SAM2 segmentation and tracking module performs pixel-level segmentation on defects based on the detection frame output by the improved YOLOv9 target detection module, and realizes continuous tracking of defects through cross-frame feature matching.
4. The method for detecting concrete defects based on deep learning according to claim 3, characterized in that: The improvement process of the improved YOLOv9 target detection module specifically includes: The default CIoU loss is replaced by DIoU loss, and the DIoU loss formula is: Where d is the target center point distance, α is the aspect ratio weight; The k-nearest neighbor algorithm is used to calculate the anchor parameters that are suitable for the size of concrete defects; In the early stage of training, the Adam optimizer is used to accelerate convergence, and in the later stage, it switches to stochastic gradient descent and uses the learning rate scheduler to smooth the transition.
5. The method for detecting concrete defects based on deep learning according to claim 3, characterized in that: The improvement process of the improved SAM2 segmentation and tracking module specifically includes: The pre-trained Hiera model is fine-tuned based on the preset concrete defect dataset. A dynamic weighted combination of cross entropy loss and Dice loss is used to balance the class imbalance. The loss function is expressed as: L=λ1L CE +λ2L Dice ; Among them, the cross entropy loss Dice loss y i is the true label; is the predicted probability of the model; ∈ is a small constant to prevent division by zero; Insert a Gabor filter layer into the shallow encoder of the Hiera model; An expanded self-attention mechanism with a dilation factor d is introduced to expand the receptive field to capture multi-scale defect features.
6. The method for detecting concrete defects based on deep learning according to claim 2, characterized in that: Before step S1, the method further includes: After preprocessing the video image sequence used for concrete defect detection training, images with a size larger than 800×800 are directly used as training samples. For images with a size smaller than 800×800 in the first sequence, four images are randomly selected using the Mosaic method, compressed to 400×400 pixels, and then spliced into 800×800 training samples. The labels are adjusted to match the positions of the new images to serve as training samples, thereby obtaining a training sample set.
7. The method for detecting concrete defects based on deep learning according to claim 6, characterized in that: The training sample set is divided into a training set, a validation set and a test set; the ratio of the training set, the validation set and the test set is 7:2:
1.
8. The method for detecting concrete defects based on deep learning according to claim 3, characterized in that: Before step S1, the method further includes: Construct a tracking list, and configure a one-to-one mapping relationship between the defect unique tracking number, defect category, detection box coordinates, segmentation mask, trajectory history, and status mark in the tracking list.
9. The method for detecting concrete defects based on deep learning according to claim 8, characterized in that: The cross-frame feature matching specifically includes: When the improved YOLOv9 object detection module detects a new defect, it assigns a globally unique tracking number to it and binds the new defect to the detection box and segmentation mask; In subsequent frames, the intersection-over-union ratio and feature similarity of the current frame defect segmentation mask and the historical trajectory mask are calculated. If the intersection-over-union ratio and feature similarity meet the preset threshold, the tracking number is inherited; otherwise, it is considered a new defect; If a defect is not matched for n consecutive frames, it is marked as "disappeared" and removed from the tracking list, where n≥5.
10. A concrete defect detection system based on deep learning, characterized in that: include: a data acquisition module for collecting a video image sequence containing concrete defects from a surface of the infrastructure and preprocessing the video image sequence to obtain a first sequence; a data processing module, configured to input the first sequence into a pre-trained joint neural network model to generate a final result including defect location coordinates, category labels, unique tracking numbers, and cross-frame trajectories; The output module is used to overlay the final results onto the video image sequence in the form of visual annotations.