Brick building crack detection method based on improved YOLOv5s

Abstract

The application relates to a masonry building crack detection method based on an improved YOLOv5. First, SPD-Conv is used on the backbone network to avoid the loss of fine-grained features of small cracks; second, in order to better fuse feature information of different scales, a weighted bidirectional feature pyramid network (BiFPN) is used in the network model Neck part, so that the model can learn features in a targeted manner; then, considering that the position of the crack is complex and interfered by many factors, a Coordinate Attention attention module is embedded on each branch of the Neck part to help the model accurately locate the crack position; finally, in order to further improve the mismatch between the real box and the predicted box, the SIoU Loss loss function is used, and the angle concept of the real box and the predicted box is introduced to accelerate the network convergence.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a masonry building crack detection method based on an improved YOLOv5. BACKGROUND

[0002] Masonry buildings often crack on the surface due to factors such as their own quality, temperature, humidity, and differential settlement of buildings. This not only affects the structural safety of the building, but also reduces the function of the building and shortens the service life of the building, which is a potential safety hazard in people's lives. Therefore, it is necessary to detect the cracks of the building in a timely and accurate manner.

[0003] Traditional crack detection is mainly based on manual detection methods. Manual detection methods are not only low in efficiency and high in cost, but also cannot guarantee the safety of the detection personnel. With the development of computers, researchers began to use digital image processing technology to assist in crack detection. Abudayyeh et al. compared the effects of four edge extraction algorithms in crack detection images; Ma Changxia et al. used a non-subsampled contourlet transform (NSTC) detection algorithm to enhance the contrast of small cracks and overcome the influence of light and noise, and used image morphological methods to effectively remove isolated noise points; Wang Yaodong et al. used a local and global combined preprocessing algorithm to suppress the influence of uneven light, combined with a multi-level filtering algorithm to filter image noise, and realized the detection of subway tunnel surface cracks; Zhou Shuyi et al. used a self-tunable edge filter (SEF) to detect cracks in green balls, which solved the interference of green ball surface reflection and contour, and used connected domain analysis method to detect whether there were cracks, which could effectively detect the cracks of small target surfaces. Li Na et al. used color and texture in the evaluation function of HSV to obtain evaluation parameters as a reference for manual detection. However, this method still has problems such as complicated detection steps, lack of automation, and subjective detection results.

[0004] Recently, machine learning, as a multi-disciplinary technology, is widely used in data analysis and mining, pattern recognition, bioinformatics and other fields, and has also been studied in the field of crack detection. Nguyen et al. segmented the crack image into small images for feature extraction and used machine learning methods for crack detection; Bai Biao used a multi-stage filter to remove a large amount of background and noise in the image, and proposed a crack detection algorithm based on SVM, which can effectively detect subway tunnel cracks; Kuchi et al. proposed a stacked machine learning method for levee and floodwall crack detection to improve detection accuracy. In addition, deep learning algorithms, as a new research direction in the field of machine learning, have been favored by many researchers and applied to practical scenarios of crack detection. Sun Zhe et al. used Faster RCNN combined with FPN network for target detection, and improved the detection accuracy by adding VAE; Wan Xiaodan used a target detection method combining adversarial networks and neural networks to improve detection accuracy. Zhang Xu et al. used a lightweight network Tiny-YOLOv3 to reduce power consumption on the terminal, and optimized the model on the OpenVINO platform to realize lightweight road crack detection. Wei Chuntao et al. proposed a lightweight grid bridge crack detection technology based on VGG, designed a grid processing method to realize high-precision detection of bridge cracks. Xie Yanghao et al. combined the optical flow model with the CT sequence to realize the segmentation of the target, and improved the automation level of crack segmentation. Feng Xiaochen et al. improved the U-Net and combined it with the attention mechanism to solve the problem of easy loss of crack width information. To some extent, these methods realize the automation of crack detection, but there is still room for improvement in terms of detection accuracy and speed. SUMMARY

[0005] The purpose of the present application is to provide a masonry building crack detection method based on improved YOLOv5 to solve the problems of low efficiency and poor accuracy in the prior art. YOLOv5 is used as the framework, SPD-Conv is used to retain the fine-grained features of small cracks, and multiple mechanisms are used for targeted optimization to build a method for masonry building crack detection.

[0006] To achieve the above purpose, the technical scheme of the present application is as follows: a masonry building crack detection method based on improved YOLOv5, an improved YOLOv5 crack detection model is constructed; first, SPD-Conv is used on Backbone; second, BiFPN structure is used on Neck, so that the model can learn features in a targeted manner; then, CoordinateAttention attention module is integrated into each branch of Neck to help the model locate the crack position more accurately; finally, SIoULoss is used in Head, the angle concept of real box and predicted box is introduced, the network convergence is accelerated, and the training speed and accuracy of the model are improved.

[0007] In an embodiment of the present application, the SPD-Conv is composed of a Space-to-depth (SPD) layer and a convolutional layer (stride=1) layer, wherein,

[0008] Space-to-depth (SPD) layer: When the Space-to-depth (SPD) layer is down-sampled, all information in the channel dimension is retained to avoid information loss, and the width and height dimensions are transferred to the channel dimension; for any feature map X of size SxSxC1, slicing is performed to obtain a sequence of sub-feature maps, each of which has a size of 2 The sequence of sub-feature maps is connected along the channel direction to form a new sequence of feature maps, which has a size of ; wherein S is the side length of the original feature map, scale is the scale factor used when down-sampling, and C1 is the number of channels of the original feature map;

[0009] Convolutional layer (stride=1): After the Space-to-depth (SPD) layer, a convolutional layer with stride=1 is used to transform the size of the sequence of feature maps to , wherein C2<scale 2 C1, and C2 is the number of channels of the sub-feature map.

[0010] In an embodiment of the present application, the BiFPN is a weighted bidirectional pyramid network, and the BiFPN adds a branch between the input and the output, so that the model can fuse more features without increasing the cost; the BiFPN assigns different weights to each layer for fusion, so that the model can pay attention to more important layers, and some unnecessary node connections are reduced, so that the model can better process features of different scales.

[0011] In an embodiment of the present application, the SIoU is composed of four parts, including an angle loss, a distance loss, a shape loss, and an IoU loss, wherein:

[0012] Angle loss

[0013]

[0014]

[0015]

[0016]

[0017] σ is the distance between the center points of the real box and the predicted box; c h ​is the height difference between the center points of the real box and the predicted box; and a is the included angle between the line connecting the center points of the real box and the predicted box and the horizontal line; is the center coordinate of the real box; is the center coordinate of the predicted box;

[0018] Distance loss

[0019]

[0020]

[0021] c w , c l are the width and height of the minimum circumscribed rectangle of the real box and the predicted box, respectively;

[0022] Shape loss

[0023]

[0024]

[0025] w, h, w gt , h gt are the width and height of the predicted box and the real box, respectively, and 0 controls the attention degree to the shape loss;

[0026] IoU loss

[0027]

[0028] A is the area of the candidate box region, and B is the area of the original labeled box region;

[0029] The final SIoU loss function is defined as follows

[0030]

[0031] Compared with the prior art, the present application has the following beneficial effects: the masonry building crack detection method based on improved YOLOv5 selects YOLOv5s as the framework, combines SPD-Conv to retain the fine-grained features of small cracks, and combines various mechanisms for targeted optimization to build an algorithm for masonry building crack detection. First, in order to better detect small cracks, we use SPD-Conv on Backbone to avoid the loss of fine-grained features of small cracks. Second, in order to better fuse feature information of different scales, we use BiFPN structure on Neck to enable the model to learn features specifically. Then, considering the problem that the location of the crack is complex and interfered by many factors, we integrate Coordinate Attention attention module in each branch of Neck to help the model locate the crack position more accurately. Finally, in order to further improve the problem of mismatch between real boxes and predicted boxes, we use SIoU Loss in Head, which introduces the concept of the angle between real boxes and predicted boxes, can accelerate the convergence of the network, and improve the training speed and accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The method flowchart of the present application.

[0033] Figure 2 The basic structure of the backbone network SPD-Conv (scale = 2).

[0034] Figure 3 The basic structure of BiFPN.

[0035] Figure 4 The basic structure of Coordinate Attention.

[0036] Figure 5 The visualization results of each model on different types of cracks (from left to right, fine cracks, coarse cracks and mixed cracks). DETAILED DESCRIPTION

[0037] The technical solutions of the present application will be specifically described below in conjunction with the drawings.

[0038] As Figure 1As shown, the application is based on an improved masonry crack detection method of YOLOv5, taking YOLOv5s as the framework, combining SPD-Conv to retain the fine-grained features of small cracks, and combining various mechanisms for targeted optimization to build an algorithm for masonry crack detection. First, in order to better detect small cracks, we use SPD-Conv on Backbone to avoid the loss of fine-grained features of small cracks. Second, in order to better fuse feature information of different scales, we use BiFPN structure on Neck to enable the model to learn features. Then, considering the complex and numerous problems of the location of the crack, we integrate the Coordinate Attention attention module in each branch of the Neck to help the model more accurately locate the crack position. Finally, in order to further improve the problem of mismatch between real boxes and predicted boxes, we use SIoU Loss in Head, which introduces the concept of angle between real boxes and predicted boxes, which can accelerate the convergence of the network and improve the training speed and accuracy of the model.

[0039] The technical scheme of the application is described in detail as follows:

[0040] 1. Improved YOLOv5 crack detection model

[0041] 1.1, SPD-Conv network structure

[0042] In the target detection task, the traditional CNN structure is an important network architecture. However, for the detection of low resolution and small objects, the performance of traditional CNN will decrease rapidly, which is due to the use of large step convolution and pooling operation which will cause the loss of fine-grained information, making the learning ability greatly discounted. Since the crack detection task has the characteristics of low resolution and small target, in order to solve the problem of down-sampling loss of fine-grained information, we choose to use SPD-Conv. Its basic structure is as shown in Figure 2 SPD-Conv completely replaces the large step convolution and pooling layer, eliminating the negative effects brought by the traditional CNN structure. By simply transforming the model to better perform the task of identifying low resolution or fine-grained crack images. SPD-Conv is composed of a Space-to-depth (SPD) layer and a convolution layer (stride = 1) layer.

[0043] Space-to-depth (SPD) layer: Space-to-depth (SPD) layer will retain all information in the channel dimension when down-sampling, avoiding information loss, and transferring the width and height dimensions to the channel dimension. For any feature map X of size SxSxC1, after slicing, scale2 sub-feature map sequences are obtained, each of which is of size The sub-feature sequences are connected along the channel direction to form a new feature map sequence with a size of

[0044] Convolution layer (stride = 1): After the Space-to-depth (SPD) layer, a convolution layer with stride = 1 is used to transform the feature map sequence size into where C2< scale2C1. This approach preserves as much feature information as possible.

[0045] In the above, S is the side length of the original feature map, C1 is the channel number of the original feature map, C2 is the channel number of the sub-feature map, and scale is the scale factor used in downsampling.

[0046] By using SPD-Conv, the model will better preserve the fine-grained features of the image, making the model have better detection ability for small cracks.

[0047] 1.2, BiFPN-Coordinate Attention network structure

[0048] The neck part adopts the BiFPN-Coordinate Attention structure, as shown in Figure 3 , 4 Early object detection algorithms usually directly use the last layer features of the Backbone to do object detection by bounding box detection heads. Such algorithms are called single-stage object detection algorithms. This algorithm cannot effectively represent objects of different scales. Therefore, object detection algorithms have gradually developed into using different stage feature maps to form a feature pyramid network (FPN). Since the FPN was proposed, many versions have been iterated. The PANet used by YOLOv5 is one of them. PANet is the first model to propose a bottom-up secondary fusion. This method proves the effectiveness of bidirectional fusion and is widely used in deep learning algorithms. However, since it simply sums up the features of different resolutions after adjusting them to a fixed size, it ignores the difference in the contribution of different resolutions to feature fusion. In order to solve these problems, we choose to use BiFPN, a weighted bidirectional pyramid network, whose basic structure is shown in Figure 3 BiFPN adds a branch between the input and the output, so that the model can fuse more features without increasing the cost. Moreover, BiFPN also removes unnecessary nodes, reducing the amount of calculation. BiFPN assigns different weights to each layer for fusion, allowing the model to pay more attention to more important layers and reducing some unnecessary node connections, allowing the model to better handle features of different scales.

[0049] 1.3, prediction stage based on SIoU loss

[0050] The prediction stage includes a bounding box loss function and non-maximum suppression (NMS). The previously proposed methods (the original YOLO model uses CIoU and other methods) all have the problem that the required real box and the predicted box do not match, which makes the convergence speed slower and the efficiency lower, and in the training process, the predicted box may float indefinitely and produce a model with worse effect. To solve this problem, we choose to use a bounding box regression loss function SIoU, which introduces a new idea of directionality, effectively reduces the degrees of freedom, and can greatly improve the training and inference of the target detection algorithm. Compared with CIoU, SIoU not only converges faster in the training stage, but also has better performance in the inference stage, and SIoU is superior to other bounding box regression loss functions. The loss function of the present application consists of four parts.

[0051] 1.3.1 Angle loss

[0052]

[0053]

[0054]

[0055]

[0056] σ is the distance between the center points of the real box and the predicted box. c h is the height difference between the center points of the real box and the predicted box. α is the included angle between the line connecting the center points of the real box and the predicted box and the horizontal line. is the center coordinate of the real box, is the center coordinate of the predicted box;

[0057] 1.3.2 Distance loss

[0058]

[0059]

[0060] c w , c l are the width and height of the minimum circumscribed rectangle of the real box and the predicted box respectively;

[0061] 1.3.3 Shape loss

[0062]

[0063]

[0064] w, h, w gt , h gtThe width and height of the predicted box and the real box, respectively, and θ controls the degree of attention to shape loss.

[0065] 1.3.4 IoU loss

[0066]

[0067] The area of intersection of two rectangular boxes / the area of combination of two rectangular boxes.

[0068] A is the area of the candidate box region, and B is the area of the original labeled box region

[0069] 1.3.5 The final SIoU loss function is defined as follows

[0070]

[0071] 2. Experimental examples:

[0072] In order to evaluate the performance of the masonry crack detection algorithm, a real masonry crack dataset with a resolution of 224x224 is used. We use horizontal flip, vertical flip, horizontal and vertical flip, etc. Data augmentation methods are used to preprocess the original dataset, and finally 960 masonry crack images are obtained. We use labelimg to label the dataset. Since the masonry cracks are mostly long and narrow, in order to avoid the area of the crack being too small in the prediction box, we use fine-grained labeling to cover the masonry cracks with as many small prediction boxes as possible to make the labeled area and positioning more accurate. After labeling, we divide the images and labels into training set and test set in the ratio of 8:2, of which the training set has 768 images and the test set has 192 images.

[0073] After training, the weight with the highest mAP@0.5:0.95 score in the test set is retained, which is used as the weight for detection. In addition, the values of Precision, Recall, mAP@0.5 and mAP@0.5:0.95 are recorded as evaluation criteria.

[0074]

[0075]

[0076] Among them, TP is true positive, FP is false positive, and FN is false negative.

[0077] Additionally, we used mAP@0.5 and mAP@0.5:0.95 to evaluate the detection precision of the categories. mAP refers to the area enclosed by plotting Precision and Recall on two axes, and @.X represents the threshold for determining whether an IoU is positive or negative. @0.5:0.95 represents the average mAP across different IoU thresholds (from 0.5 to 0.95, with a step size of 0.05). The training results were primarily observed for fluctuations in precision and recall; relatively small fluctuations indicated good training performance. The training results were then evaluated by observing the magnitudes of the four evaluation metrics.

[0078] 2.1 Implementation Details

[0079] The method of this invention is programmed using the PyTorch deep learning framework. The experimental environment consisted of a Windows 11 64-bit operating system, an Intel Core i7-11800H@2.3GHz processor, an NVIDIA GeForce RTX 3060 laptop, and 16GB of RAM. We specifically optimized the YOLOv5s6.0 model to construct a crack detection model. In the backbone network, we used the SPD-Conv module, the Neck part used a BiFPN-Coordinate Attention structure, and the Head part used the SIoULoss loss function. During training, our batch size was 16, using the Adam optimizer with an initial learning rate of 0.01, a learning rate momentum factor of 0.937, and a weight decay factor of 0.0005.

[0080] 2.2 Qualitative Comparison

[0081] To qualitatively compare the performance of our algorithm with other existing object detection methods on the masonry building dataset, we conducted comparative experiments, such as... Figure 5 As shown in the diagram, in the detection of fine cracks, our model is closest to the actual cracks in terms of localization and has the highest confidence score. Other models showed varying degrees of misclassification or missed detection. This demonstrates that our crack detection model is better suited for low-resolution or fine-grained crack detection. Furthermore, in the detection of coarse cracks, ObjectBox misclassified brick edges as cracks; in the detection of mixed coarse and fine cracks, Faster R-CNN misclassified gaps with color differences from the bricks as cracks; and in the detection of mixed cracks, YOLOv5s and Faster R-CNN misclassified larger and deeper gaps in the bricks as cracks. In contrast, our model correctly identified most cracks and generally achieved higher confidence scores than other methods. The visualization results clearly show that our model significantly outperforms other object detection models for the detection of all three types of cracks.

[0082] 2.3, Quantitative comparison

[0083] In this section, we show the quantitative comparison of the crack detection model and other existing object detection methods, and explain and analyze the advantages of targeted optimization through experiments to verify the effectiveness of our model. Our experimental results come from the average of three training results.

[0084] Table 1 Comparison of different algorithms

[0085] Method Precision Recall mAP@0.5 mAP@0.5:0.95 Faster RCNN 0.3 0.75 0.5 0.18 objectBox 0.941 0.91 0.936 0.607 yolov5s-SPD 0.939 0.824 0.921 0.624 yolov3-tiny 0.849 0.837 0.899 0.548 yolov5s 0.912 0.859 0.929 0.642 Ours 0.961 0.892 0.956 0.68

[0086] Table 1 shows the experimental results of various object detection methods on the masonry dataset. Our method achieves the best results in Precision, mAP@0.5, mAP@0.5:0.95, and is only slightly lower than ObjectBox in Recall. Compared with existing methods such as Faster RCNN, OjectBox, YOLO-SPD, our method improves mAP@0.5 by 45.6%, 2.0%, 3.5% respectively, and improves mAP@0.5:0.95 by 50.0%, 7.3%, 5.6% respectively, which proves that our method is superior to other object detection methods in architecture. In addition, compared with some mainstream models of YOLO series, YOLOv3-tiny, YOLOv5s, our method improves mAP@0.5 by 5.7%, 2.7% respectively, and improves mAP@0.5:0.95 by 13.2%, 3.8% respectively, which verifies the effectiveness of our optimization of YOLO model.

[0087] The above is the preferred embodiment of the present application, any changes made according to the technical solutions of the present application, as long as the generated function does not exceed the scope of the technical solutions of the present application, belongs to the protection scope of the present application.

Claims

1. A masonry crack detection method based on improved YOLOv5, characterized by, Applied to masonry crack detection, a crack detection model based on improved YOLOv5 is constructed; first, SPD-Conv is used on Backbone; Secondly, BiFPN structure is used on Neck, so that the model can learn features specifically; Then, CoordinateAttention attention module is integrated into each branch of Neck to help the model locate the crack position more accurately; finally, SIoULoss is used in Head, the angle concept between the real box and the predicted box is introduced, the network convergence is accelerated, and the training speed and accuracy of the model are improved; The SPD-Conv is composed of a Space-to-depth (SPD) layer and a convolution layer, wherein Space-to-depth (SPD) layer: During downsampling, Space-to-depth (SPD) retains all information in the channel dimension, avoiding information loss, and transfers the width and height dimensions to the channel dimension; for any size... Feature Mapping Row slicing, after slicing, we get Each of the following sub-feature mapping sequences has a size of ( ). , , The sub-feature mapping sequences are connected along the channel direction to form a new feature mapping sequence with a size of ( , , ), where S is the side length of the original feature map, scale is the scaling factor used during downsampling, and C1 is the number of channels in the original feature map; Convolutional layer: After the Space-to-depth (SPD) layer, a convolutional layer with is used to transform the feature map sequence size into ( , , ), where , and C2 is the number of channels of the sub-feature map. The BiFPN is a weighted bidirectional pyramid network, BiFPN adds a branch between input and output, so that the model can fuse more features without increasing cost; BiFPN assigns different weights to each layer for fusion, so that the model can pay attention to more important layers, and reduces some unnecessary node connections, so that the model can better handle different scale features.

2. The masonry crack detection method based on improved YOLOv5 according to claim 1, characterized in that, The SIoU is composed of four parts, including angle loss, distance loss, shape loss and IoU loss, wherein: Angle loss is the distance between the center points of the real and predicted bounding boxes; is the height difference between the center points of the real and predicted bounding boxes; is the angle between the line connecting the center points of the real and predicted bounding boxes and the horizontal line; , is the real bounding box center coordinate; , is the predicted bounding box center coordinate; Distance loss , , , are the width and height of the minimum bounding rectangle of the true and predicted boxes, respectively; Shape loss , , , , are the width and height of the predicted and ground truth boxes, respectively, controls the degree of attention to shape loss; IoU loss A is the area of the candidate box region, and B is the area of the original labeled box region; The final SIoU loss function is defined as follows 。

Images