A pavement crack detection method based on an improved ResNet-50 network structure

Abstract

The present invention relates to the field of deep learning technology, and in particular to a pavement crack detection method with an improved ResNet-50 network structure. The method comprises the following steps: removing a fully connected layer and an average pooling layer of the ResNet-50 network, integrating a hybrid dilated convolution into the last three layers of the network, adding a space-channel attention mechanism between the second layer and the third layer, the third layer and the fourth layer, and the fourth layer and the fifth layer, and using a multi-scale feature fusion method to fuse extracted features to form an improved ResNet-50 network. A pavement crack data set is used to train the improved network model to obtain a crack detection network model. Finally, the crack detection network model is used to classify cracks and evaluate their severity. The improved dilated convolution module can extract features from a larger range of pixels, ensuring that the network can obtain global feature information. The multi-scale feature fusion method improves the accuracy of the extracted feature fusion.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a pavement crack detection method with an improved ResNet-50 network structure. Background Art

[0002] Cracks are the most direct manifestation of evaluating the damage of the road surface and can reflect the actual condition of the current road surface in a timely manner. In the daily maintenance of roads, regular inspection of the road surface can detect cracks in time and repair them, reduce the impact of cracks on road safety, and increase the service life of the highway. The original manual detection method is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in low detection accuracy. In the early years, with the continuous development of computer vision, many researchers used digital image processing technology to automatically detect cracks. However, due to the heterogeneity of crack shapes, such as transverse cracks, longitudinal cracks, block cracks and cracks, and the complexity of the road environment, such as uneven light intensity and complex and changeable road environment, these detection methods cannot detect cracks from a global perspective, resulting in low crack detection accuracy. In recent years, many crack detection methods based on deep learning have also emerged, but there are still problems of low detection accuracy due to the complex and changeable crack topology and the influence of crack image noise. Summary of the invention

[0003] The purpose of the present invention is to provide a pavement crack detection method with an improved ResNet-50 network structure, so as to improve the technical problems of incomplete feature extraction and low detection accuracy in the existing crack detection method based on deep learning.

[0004] To achieve the above object, the present invention provides a pavement crack detection method with an improved ResNet-50 network structure, comprising the following steps:

[0005] Collecting road surface image sets;

[0006] Filter the pavement images containing cracks to build a pavement crack dataset;

[0007] Using the pavement crack dataset to train an improved ResNet-50 network model;

[0008] Test and evaluate the training results to obtain the crack detection network model;

[0009] The crack detection network model is used to perform crack classification and severity assessment.

[0010] Among them, in the process of screening pavement images with cracks to construct a pavement crack dataset, pavement images with cracks in the pavement image set are screened, the pavement crack images are cropped into small-resolution image blocks, the pavement crack images are annotated in the form of polygons and the actual pavement situation data is obtained, the training set and the test set are constructed, and the training set and the test set are annotated.

[0011] Among them, the improved ResNet-50 network model includes the following steps:

[0012] Remove the average pooling layer and the fully connected layer from the ResNet-50 network;

[0013] The hybrid dilated convolutional module is integrated into the last three layers of the network, and different dilation rates are set for each layer;

[0014] Integrate the spatial-channel attention mechanism module between the second and third layers, the third and fourth layers, and the fourth and fifth layers;

[0015] The features extracted from each layer are output to the side and processed using 1×1 convolution;

[0016] The processed features are fused using a multi-scale feature fusion module.

[0017] The hole rate arrangement of the hybrid dilated convolution module adopts a sawtooth waveform structure, that is, the continuous convolution layers are divided into a group, each group adopts a different hole rate, and the hole rate increases from small to large, and the next group adopts the same pattern. Convolution with a small hole rate can extract local information, and convolution with a large hole rate can extract long-distance information, so that the top convolution layer can extract features from a larger range of pixels, ensuring that the network can obtain global feature information.

[0018] Among them, the space-channel attention mechanism module is obtained by adding the space attention mechanism and the channel attention mechanism, which can obtain a more accurate feature map. The space attention mechanism first starts from the dimension of a channel to perform average pooling and maximum pooling on the feature F with an input size of H×W×C (where H represents the height of the input feature, W represents the width of the input feature, and C represents the number of channels of the input feature), and obtains two H×W×1 (where H represents the height of the output feature, W represents the width of the output feature, and 1 represents the number of channels of the output feature) features. Secondly, these two features are merged to obtain a feature with a channel number of 2, and finally the feature is processed using a convolution with a convolution kernel size of 7×7. Then, the weight coefficient Ms is obtained, and Ms is multiplied by the feature F to obtain the new feature F1. The channel attention mechanism performs global maximum pooling and average pooling on the feature F with an input size of H×W×C (where H represents the height of the input feature, W represents the width of the input feature, and C represents the number of channels of the input feature) from a spatial perspective, and obtains two 1×1×C (where 1 represents the height of the output feature, 1 represents the width of the output feature, and C represents the number of channels of the output feature) features. Then, the obtained features are processed using a two-layer neural network, and finally, the two features are processed using an activation function to obtain the weight coefficient Mc, and Mc is multiplied by the feature F to obtain the new feature F1. The spatial-channel attention mechanism can better enhance crack features, suppress non-crack features, and extract more crack features.

[0019] The multi-scale feature fusion module first processes the features output by each layer of the ResNet-50 network using a 1×1 convolution, then upsamples the obtained feature map to the same size as the input image and fuses it, and finally processes the fused features using a 1×1 convolution to obtain the final prediction result. The multi-scale feature fusion module integrates multi-scale and multi-level feature maps to make the final prediction result closer to the actual road condition.

[0020] In the process of testing and evaluating the training results and obtaining the crack detection network model, the trained network model is tested using the test set in the pavement crack data set. The accuracy is evaluated by calculating the precision, recall, and F-score based on the test results. The precision, recall, and F-score are shown in formulas (1) to (3), where TP is the number of pixels correctly detected and judged as cracks in the detection results, FP is the number of background pixels mistakenly detected as crack pixels, and FN is the number of crack pixels mistakenly detected as background pixels.

[0021]

[0022]

[0023]

[0024] If the final prediction results and evaluation index values ​​meet the requirements, save the output crack detection network model;

[0025] Otherwise, modify the hyperparameters of the network model and retrain.

[0026] The present invention provides a pavement crack detection method with an improved ResNet-50 network structure. The method comprises the following steps: removing the fully connected layer and the average pooling layer of the ResNet-50 network, integrating the hybrid dilated convolution into the last three layers of the network, adding a space-channel attention mechanism between the second layer and the third layer, the third layer and the fourth layer, and the fourth layer and the fifth layer, and using a multi-scale feature fusion method to fuse the extracted features to form an improved ResNet-50 network. The pavement crack data set is used to train the improved network model to obtain a crack detection network model. Finally, the crack detection network model is used to classify cracks and evaluate their severity. The improved dilated convolution module can extract features from a larger range of pixels, ensuring that the network can obtain global feature information. The multi-scale feature fusion method improves the accuracy of the extracted feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flow chart of a pavement crack detection method with an improved ResNet-50 network structure of the present invention.

[0029] Figure 2 It is a schematic diagram of the network structure of the crack detection network model of the present invention.

[0030] Figure 3 It is a schematic diagram of the hybrid dilated convolution module of the present invention.

[0031] Figure 4 It is a schematic diagram of the structure of the space-channel attention mechanism of the present invention.

[0032] Figure 5 It is a schematic diagram of the structure of the multi-scale feature fusion module of the present invention.

[0033] Figure 6It is a DeepCrack, CFD and CRACK500 data set test result image set of a specific embodiment of the present invention.

[0034] Figure 6 (a) is the original image of DeepCrack, CFD and CRACK500 datasets.

[0035] Figure 6 (b) Label map of DeepCrack, CFD and CRACK500 datasets.

[0036] Figure 6 (c) is the test result diagram of the HED method.

[0037] Figure 6 (d) is the test result diagram of the RCF method.

[0038] Figure 6 (e) is the test result diagram of the U-Net method.

[0039] Figure 6 (g) is the test result diagram of the CrackW-Net method.

[0040] Figure 6 (h) is the test result diagram of the DeepCrack method.

[0041] Figure 6 (i) is the test result diagram of the FPHBN method.

[0042] Figure 6 (j) is a graph showing the test results of the method of the present invention. DETAILED DESCRIPTION

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0044] See also Figure 1 , the present invention proposes a pavement crack detection method with an improved ResNet-50 network structure, comprising the following steps:

[0045] S1: Collect road image set;

[0046] S2: Screen the pavement images containing cracks to build a pavement crack dataset;

[0047] S3: Use the pavement crack dataset to train the improved ResNet-50 network model;

[0048] S4: Test and evaluate the training results to obtain the crack detection network model;

[0049] S5: Use the crack detection network model to perform crack classification and severity assessment.

[0050] In the process of screening pavement images with cracks to construct a pavement crack dataset, pavement images with cracks in the pavement image set are screened, the pavement crack images are cropped into small-resolution image blocks, the pavement crack images are annotated in the form of polygons and the actual pavement situation data is obtained, the training set and the test set are constructed, and the training set and the test set are annotated.

[0051] S21: The improved ResNet-50 network model includes the following steps:

[0052] S22: Remove the average pooling layer and the fully connected layer from the ResNet-50 network;

[0053] S23: Integrate the hybrid dilated convolution module into the last three layers of the network, and set different dilation rates for each layer;

[0054] S24: Integrate the spatial-channel attention mechanism module between the second and third layers, the third and fourth layers, and the fourth and fifth layers;

[0055] S25: Output the features extracted from each layer to the side and process them using 1×1 convolution;

[0056] S26: The processed features are fused using a multi-scale feature fusion module.

[0057] The hole rate arrangement method of the hybrid dilated convolution module adopts a sawtooth waveform structure, that is, the continuous convolution layers are divided into a group, each group adopts a different hole rate, and the hole rate increases from small to large, and the next group adopts the same pattern. Convolution with a small hole rate can extract local information, and convolution with a large hole rate can extract long-distance information. In this way, the top convolution layer can extract features from a larger range of pixels, ensuring that the network can obtain global feature information.

[0058] The spatial-channel attention mechanism module is obtained by adding the spatial attention mechanism and the channel attention mechanism, and can obtain a more accurate feature map. The spatial attention mechanism first calculates average pooling and maximum pooling for the feature F with an input size of H×W×C from the dimension of a channel, and obtains two features of H×W×1. Secondly, the two features are merged to obtain a feature with a channel number of 2. Finally, the feature is processed using a convolution with a convolution kernel size of 7×7 to obtain a weight coefficient Ms, and Ms is multiplied by the feature F to obtain a new feature F1. The channel attention mechanism calculates global maximum pooling and average pooling for the feature F with an input size of H×W×C from a spatial perspective, and obtains two features of 1×1×C. Secondly, the obtained features are processed using a two-layer neural network, and finally the two features are processed using an activation function to obtain a weight coefficient Mc, and Mc is multiplied by the feature F to obtain a new feature F1. The spatial-channel attention mechanism can better enhance crack features, suppress non-crack features, and extract more crack features.

[0059] The multi-scale feature fusion module first processes the features output by each layer of the ResNet-50 network using a 1×1 convolution, then upsamples the obtained feature map to the same size as the input image and fuses it, and finally processes the fused features using a 1×1 convolution to obtain the final prediction result. The multi-scale feature fusion module integrates multi-scale and multi-level feature maps to make the final prediction result closer to the actual road condition.

[0060] In the process of testing and evaluating the training results and obtaining the crack detection network model, the trained network model is tested using the test set in the pavement crack dataset. The accuracy of crack detection is evaluated by calculating the precision, recall, and F-score based on the test results.

[0061] If the final prediction results and evaluation index values ​​meet the requirements, save the output crack detection network model;

[0062] Otherwise, modify the hyperparameters of the network model and retrain.

[0063] The following is further described in conjunction with each module in the present invention:

[0064] The present invention divides crack detection into three parts: feature extraction, feature processing and feature fusion. In the feature extraction stage, the improved ResNet-50 network (average pooling layer and fully connected layer are removed) is first used to extract crack features. Because the semantic features extracted by the last three layers are relatively weak, a mixed dilated convolution module is integrated into the last three layers to improve the semantic features. The four blocks of the third layer are grouped together and their hole rates are changed to 1, 2, 5 and 9 respectively. The six blocks of the fourth layer are divided into two groups, and their hole rates are changed to 1, 2, 5 respectively, and the next group maintains the same mode. The three blocks of the fifth layer are grouped together, and their hole rates are changed to 5, 9 and 17. In addition, a space-channel attention mechanism module is added between the second layer and the third layer, the third layer and the fourth layer, and the fourth layer and the fifth layer. The space-channel attention mechanism module can effectively enhance crack features, suppress non-crack features, reduce the influence of noise, and improve the accuracy of crack detection in the feature extraction process. In the feature processing stage, a convolution kernel size of 1×1 is used to process crack features extracted at different levels. In the feature fusion stage, a multi-scale feature fusion module is introduced. First, the features extracted at different levels are processed using a convolution kernel size of 1×1. Then the processed features are upsampled to the size of the input image. Finally, the feature maps extracted at different levels are fused and processed using a 1×1 convolution to obtain the final prediction map. The crack detection model proposed in the present invention is as follows: Figure 2 As shown in the figure, the network model realizes pixel-level crack detection and improves the accuracy of crack detection.

[0065] (1) Hybrid dilated convolution module

[0066] The hole rate arrangement method of the hybrid dilated convolution adopts a sawtooth waveform, that is, multiple convolutional layers are divided into a group, and the hole rate of each group increases from small to large, and the next group adopts the same pattern, such as Figure 3 As shown in the figure. Convolution with a small dilation rate can extract local information, while convolution with a large dilation rate can extract long-distance information. In this way, the top convolution layer can extract features from a wider range of pixels, ensuring that the network can obtain global feature information. Hybrid dilated convolution naturally expands the receptive field of the network without adding additional modules, and can be naturally integrated into the original layer of the network, which can robustly extract contextual semantic information and different types of crack features.

[0067] The dilation rate of each group in the hybrid dilated convolution must meet the following conditions: (1) The dilation rate of each group of stacked convolutional layers cannot have a fixed transformation factor, that is, it should not have a common divisor greater than 1. For example, the common divisor of [2, 4, 8] is 2, which is not greater than 1; (2) The dilation rate of each group must be arranged from small to large, and the next group also adopts the same pattern, such as the [1, 2, 3, 1, 2, 3] cyclic structure; (3) The maximum distance between two non-zero points is defined as shown in formula (4),

[0068] M i =max[M i+1 -2r i ,M i+1 -2(M i+1 -r i ),r i ] (4)

[0069] Among them, M n =r n , i represents the i-th layer, K represents the convolution kernel size, and M2≤K must be satisfied. For example, the convolution kernel size is 3, that is, K=3, r=[1, 2, 5], M2=2, satisfying M2≤K. When the mixed dilated convolution meets the above conditions, the receptive field of the convolution calculation can completely cover the entire area without any holes or lost edges, so that every pixel of the convolution layer can participate in the convolution calculation. In this invention, the hole rate of the mixed dilated convolution integrated into the third layer is [1, 2, 5, 9], the hole rate of the mixed dilated convolution integrated into the fourth layer is [1, 2, 5], and the hole rate of the mixed dilated convolution integrated into the fifth layer is [5, 9, 17]. This mode enables the top layer of the convolutional neural network to obtain more feature information without changing the size of its receptive field, avoiding the chessboard problem caused by the dilated convolution.

[0070] (2) Spatial-Channel Attention Mechanism

[0071] like Figure 4 As shown in the figure, the spatial-channel attention mechanism module is obtained by adding the spatial attention mechanism and the channel attention mechanism. This module can effectively enhance crack features, suppress non-crack features, and reduce the interference of noise on the final prediction results in the feature extraction stage. The spatial attention mechanism first calculates average pooling and maximum pooling for the feature F with an input size of H×W×C from the dimension of a channel, and obtains two H×W×1 features. Then, these two features are merged to obtain a feature with a channel number of 2. Finally, the feature is processed using a convolution kernel size of 7×7 to obtain the weight coefficient Ms, and Ms is multiplied by the feature F to obtain a new feature F1. The channel attention mechanism calculates global maximum pooling and average pooling for the feature F with an input size of H×W×C from a spatial perspective, and obtains two 1×1×C features. Then, the obtained features are processed using a two-layer neural network. Finally, the two features are processed using an activation function to obtain the weight coefficient Mc, and Mc is multiplied by the feature F to obtain a new feature F1. The spatial-channel attention mechanism adds the two together. This module can not only be used for crack detection, but also can be used in any segmentation and detection tasks. It can effectively enhance useful features, suppress useless features, and improve detection accuracy.

[0072] (3) Multi-scale feature fusion

[0073] The multi-scale feature fusion module is mainly used to fuse multi-scale and multi-level crack features to make the final prediction result closer to the actual situation of the road surface. Figure 5 As shown, the multi-scale feature fusion module first processes the features output by each layer of the ResNet-50 network using a 1×1 convolution, then upsamples the obtained feature map to the same size as the input image and fuses it, and finally processes the fused features using a 1×1 convolution to obtain the final prediction result. The multi-scale feature fusion module is not only applicable to this invention, but can also be applied to various types of detection tasks. As long as the feature map extracted from each layer is input into the multi-scale feature fusion module, and then the upsampling size is set as needed, the final prediction result can be obtained. The multi-scale feature fusion module fuses multi-scale and multi-level features, so that the final prediction result is closer to the actual condition of the road surface.

[0074] The present invention also proposes a specific embodiment, which is verified on three public crack datasets DeepCrack, CFD and CRACK500, and compared with the currently popular edge detection methods HED (Holistically-nested Edge Detection), RCF (Richer Convolutional Feature for Edge Detection), image segmentation methods U-Net, CrackW-Net, DeepCrack and FPHBN (Feature Pyramid and Hierarchical BoostiongNetwork), and uses accuracy (Precision), recall (Recall), and F-score. The test results on the three datasets are as follows Figure 6 As shown, from a visual perspective, the crack features extracted by the present invention are clear and the feature map is less affected by noise. Some methods are affected by noise, resulting in blurred or even incomplete crack features. From an objective perspective, the present invention uses accuracy (Precision), recall (Recall), and F-score to evaluate the final prediction results. From Table 1, Table 2, and Table 3, it can be seen that the various indicators of the present invention are mostly higher than other methods, and the F-score value is improved by up to 12%. In summary, the crack detection method proposed by the present invention is better than other methods and has a better effect.

[0075] Table 1 Evaluation index values ​​on the DeepCrack dataset

[0076]

[0077]

[0078] Table 2 Evaluation index values ​​on CFD dataset

[0079]

[0080] Table 3 Evaluation index values ​​on the CRACK500 dataset

[0081]

[0082] In summary, the present invention has the following advantages:

[0083] 1. The improved dilated convolution module arranges the hole rate in a sawtooth waveform, that is, multiple convolutional layers are divided into a group, and the hole rate of each group increases from small to large, and the next group adopts the same pattern, so that the top convolutional layer can extract features from a larger range of pixels, ensuring that the network can obtain global feature information.

[0084] 2. The improved attention mechanism module combines the spatial and channel attention mechanisms to form a spatial-channel attention mechanism. In this mode, the network can enhance crack features, suppress non-crack features, reduce the impact of noise, and improve the accuracy of crack detection.

[0085] 3. The improved feature fusion module fuses multi-scale and multi-level features, extracts feature maps obtained at different levels, and fuses feature maps at different levels after a series of operations such as convolution and upsampling, making the final prediction result closer to the actual road condition.

[0086] 4. Improved ResNet-50 network, remove the fully connected layer and average pooling layer of the ResNet-50 network, integrate the hybrid dilated convolution into the last three layers of the network, add the spatial-channel attention mechanism between the second and third layers, the third and fourth layers, and the fourth and fifth layers, and use multi-scale feature fusion to fuse the extracted features, thus realizing pixel-level crack detection.

[0087] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A pavement crack detection method with an improved ResNet-50 network structure, characterized in that: The following steps are involved: Collecting road surface image sets; Filter the pavement images containing cracks to build a pavement crack dataset; Using the pavement crack dataset to train an improved ResNet-50 network model; The improved ResNet-50 network model includes the following steps: Remove the average pooling layer and the fully connected layer from the ResNet-50 network; The hybrid dilated convolutional module is integrated into the last three layers of the network, and different dilation rates are set for each layer; Integrate the spatial-channel attention mechanism module between the second and third layers, the third and fourth layers, and the fourth and fifth layers; The features extracted from each layer are output to the side and processed using 1×1 convolution; The processed features are fused using a multi-scale feature fusion module; The multi-scale feature fusion module first processes the features output by each layer of the ResNet-50 network using a 1×1 convolution, then upsamples the obtained feature map to the same size as the input image and fuses it, and finally processes the fused features using a 1×1 convolution to obtain the final prediction result; Test and evaluate the training results to obtain the crack detection network model; The crack detection network model is used to perform crack classification and severity assessment.

2. The pavement crack detection method with improved ResNet-50 network structure as claimed in claim 1, characterized in that: In the process of screening pavement images with cracks to construct a pavement crack dataset, pavement images with cracks in the pavement image set are screened, the pavement crack images are cropped into small-resolution image blocks, the pavement crack images are annotated in the form of polygons and the actual pavement situation data is obtained, the training set and the test set are constructed, and the training set and the test set are annotated.

3. The pavement crack detection method with improved ResNet-50 network structure as claimed in claim 1, characterized in that: The hole rate arrangement method of the hybrid dilated convolution module adopts a sawtooth waveform structure, that is, the continuous convolution layers are divided into a group, each group adopts a different hole rate, and the hole rate increases from small to large, and the next group adopts the same pattern.

4. The pavement crack detection method with improved ResNet-50 network structure as claimed in claim 1, characterized in that: The spatial-channel attention mechanism module is obtained by adding the spatial attention mechanism and the channel attention mechanism. The spatial attention mechanism first calculates average pooling and maximum pooling for the feature F with an input size of H×W×C from the dimension of a channel to obtain two H×W×1 features. Secondly, the two features are merged to obtain a feature with a channel number of 2. Finally, the feature is processed using a convolution with a convolution kernel size of 7×7 to obtain a weight coefficient Ms, and Ms is multiplied by the feature F to obtain a new feature F1. The channel attention mechanism calculates global maximum pooling and average pooling for the feature F with an input size of H×W×C from a spatial perspective to obtain two 1×1×C features. Secondly, the obtained features are processed using a two-layer neural network. Finally, the two features are processed using an activation function to obtain a weight coefficient Mc, and Mc is multiplied by the feature F to obtain a new feature F1.

5. The pavement crack detection method with improved ResNet-50 network structure as claimed in claim 1, characterized in that: In the process of testing and evaluating the training results and obtaining the crack detection network model, the trained network model is tested using the test set in the pavement crack dataset. The accuracy, recall rate, and F-score are calculated based on the test results to evaluate the accuracy of crack detection. If the final prediction results and evaluation index values ​​meet the requirements, save the output crack detection network model; Otherwise, modify the hyperparameters of the network model and retrain.

Images