A method for detecting cracks in vehicle-mounted front-view images

Abstract

The present invention provides a method for detecting cracks in a vehicle-mounted front-view image. The method includes semantic segmentation of the road surface of the front-view image, classification of road surface image blocks, crack detection, and crack type discrimination and statistics. The processing process of the method of the present invention has no approximation, and can effectively avoid the influence of weeds and debris on road surface detection. After introducing a histogram grayscale threshold segmentation algorithm based on significance detection, the cracks are clear and easy to extract, and can adapt to crack detection in complex images. It has universal applicability and can be widely used in the field of road disease detection.

Description

Technical Field

[0001] The invention relates to the technical field of road damage detection, and in particular to a method for detecting cracks in a vehicle-mounted front view image. Background Art

[0002] At present, the forward-looking road surface images have the characteristics of diverse background textures, multi-targets, complex lighting environments, and complex acquisition scenes, making cracks one of the most difficult targets to identify among the types of damage. The forward-looking images taken by the on-board camera have complex scenes, including not only the highway but also the ancillary facilities of the road construction and the off-road environment background; the noise is complex, including noise such as shadows, lane lines, oil spots, and water stains;

[0003] There are three main pavement crack detection technologies today: artificial visual detection, digital image processing, and three-dimensional laser detection methods. Among them, artificial visual detection has problems such as slow speed, low efficiency, high risk, and poor comprehensiveness. The detection steps of pavement crack recognition technology based on image processing generally include the following: first, the crack image is collected through the vehicle camera, and then the collected image is preprocessed, crack detection and feature extraction are performed to obtain information such as crack type, crack geometric characteristic parameters, and damage severity (Reference: Jiang Wenbo, Luo Qiurong, Zhang Xiaohua. A review of concrete road crack detection methods based on digital images [J]. Journal of Xihua University (Natural Science Edition), 2018 (1): 13.). Pavement crack detection technology based on image processing has become a common method in the field of pavement crack detection due to its accuracy, safety, robustness, and real-time performance. The pavement crack detection method based on three-dimensional laser is not affected by light and shadow, has high recognition accuracy, and is fast; however, the detection effect of small cracks is poor, the hardware cost is high, and it cannot be widely used.

[0004] Therefore, this paper's research on pavement crack detection technology based on vehicle-mounted forward-view images has good theoretical significance and practical application value. It can provide a more universal, efficient, stable and intelligent pavement crack identification technology for real-time pavement crack monitoring, and can also provide a scientific evaluation system and theoretical basis for pavement crack detection. Summary of the invention

[0005] In view of the above technical problems, the purpose of the present invention is to overcome the shortcomings of the existing technology and propose a method for detecting cracks in a vehicle-mounted front view image. The method includes road semantic segmentation of the front view image, road image block classification, crack detection, crack type discrimination and statistics.

[0006] To achieve the above object, the present invention proposes a method for detecting cracks in a vehicle-mounted front view image, comprising the following steps:

[0007] S1: For the vehicle-mounted front view image, the CNN road semantic segmentation method is used to separate the road surface and background areas to obtain the road surface image;

[0008] S2: Divide the road surface image obtained in step S1 into image blocks of regular size n×n, and use a ResNet network to perform binary classification on the image blocks to obtain cracked and non-cracked road surface image blocks;

[0009] S3: performing image enhancement and other preprocessing on the cracked pavement image block obtained in step S2, and performing crack detection using an improved HC saliency detection and grayscale histogram threshold segmentation method to obtain a crack segmentation image;

[0010] S4: extracting the crack contours by mathematical morphology from the crack detection image obtained in step S3, identifying the type of cracks by a histogram projection method, and establishing a two-dimensional coordinate system to calculate information such as the length and width of the cracks.

[0011] Preferably, the image preprocessing method in step S3 includes image grayscale correction processing, histogram equalization processing and median filtering denoising processing, and the improved HC significance detection method consists of two parts: HC significance detection and fine-scale enhancement. After detection, crack segmentation is performed using a grayscale histogram threshold segmentation method, including the following contents:

[0012] (1) HC-based saliency detection: The saliency value of a pixel is defined by the color difference with all other pixels in the image, as shown below:

[0013]

[0014] In the formula, D(I K ,I I ) is the color distance metric between two pixels in the space L·a·b. After expanding the pixel level, the above formula can obtain the saliency value of each color. The formula is as follows:

[0015]

[0016] Where c1 is the pixel I k The color value in, n is the number of different pixel colors, f j is the pixel color C in image I j The frequency of occurrence, the true color space contains 256*256*256 possible colors, which is more than the total pixels of the image, and the computational cost is high. Quantize each color channel to 12, reduce the colors to 12*12*12, discard the low-frequency colors, and retain the high-frequency colors. The RGB color space quantization method will produce defects. In order to reduce errors, a smoothing operation is used to replace the saliency value of each color with the weighted average of similar color saliency values, but similar colors need to measure the distance in the Lab color space. Let m=n / 4 nearest neighbor colors to improve the saliency value of color c1. The formula is as follows:

[0017]

[0018] In the formula, is the distance between color c and its m nearest neighbors;

[0019] (2) Fine-scale saliency enhancement: The fine-scale saliency enhancement algorithm can reduce the saliency value of image noise. Both cracks and background textures have a certain scale. Transforming the pavement crack image to a scale suitable for the crack will highlight the local features of the crack. The scale transformation formula is as follows:

[0020] I t+1 (x,y)=I(U N,t (x,y))×h(·)

[0021] In the formula, t represents the image scale, t = 0 is the original image, U N,t (x, y) is the M×N area around the pixel point (x, y) at scale t, and h(·) is the scale change kernel function. The crack scale changes with the image resolution. The crack shape is long and narrow, and the scale should not be too fine. Take N=2, t = 1;

[0022] The fine-scale saliency enhancement algorithm simulates the linear expansion process of cracks. The formula for fine-scale saliency enhancement of cracks is as follows:

[0023]

[0024] θ∈{0°,45°,90°,135°}

[0025] In the formula, I u is the grayscale mean of the crack scale image. Let S zu is the significant mean, w z (x,y) is the significance weight corresponding to the point (x,y), and the formula is as follows:

[0026]

[0027] In the formula, G N,θ (x, y) is the sum of the grayscale values ​​of the linear neighborhood in the θ direction centered at point (x, y), N = 3. Let the number of pixels in the original image be N. According to statistical laws, the proportion of candidate points to the original image is at most 10%. When the candidate point is a crack, the linear neighborhood points with a surrounding length of L do not need to be screened again;

[0028] Grayscale histogram threshold segmentation method: For pavement crack images, when the grayscale values ​​of the target area and the background area are different, the grayscale histogram appears as two peaks with a trough between the peaks. When the two peaks correspond to the central grayscale values ​​of the target area and the background area, respectively, the grayscale value corresponding to the trough can be used as the threshold for image segmentation. Assuming that the grayscale value corresponding to the trough is T, then T is used as the segmentation threshold, and the area composed of pixels with grayscale values ​​less than T is regarded as the crack area, and the area composed of pixels with grayscale values ​​greater than T is regarded as the background area. However, the pixel distribution of the crack area and the pavement area follows a normal distribution, that is:

[0029]

[0030]

[0031] Where f1(i) is the distribution function of the crack area;

[0032] From the normal distribution characteristics, it can be seen that in the above formula, a1 is the grayscale center value of all pixels in the crack area, u1 represents the average grayscale value of all pixels in the crack, and δ1 2 represents the mean square error of the gray value. Similarly, f2(i) represents the distribution function of the background area;

[0033] The limiting conditions are as follows:

[0034]

[0035] When T makes f(T) take the minimum value, T is the required threshold. Since the range of grayscale value variation in the crack image is limited, it will not cover the entire grayscale value range. Therefore, this can be used to improve the algorithm's operating efficiency by only reading the number of pixels corresponding to the grayscale value in the crack when designing the algorithm.

[0036] Preferably, the crack contour extraction in step S4 uses a projection method to identify the crack image and distinguish its crack type, establishes a two-dimensional coordinate system to calculate the length and width of the crack, and mainly includes the following methods and features:

[0037] (1) Crack feature extraction and type recognition based on projection method: Extract features from images containing cracks and identify different types of pavement cracks based on different feature values. Project the binary image after crack segmentation in different directions and determine the geometric features of the cracks by counting the number of pixels. The projection results have the following characteristics:

[0038] ① The horizontal projection amplitude of the transverse cracks varies greatly, and the data changes significantly. When projecting it in the vertical direction, there is still a significant difference in the projection amplitude, but it appears relatively smooth. The number of pixels in each row is roughly the same;

[0039] ② When the longitudinal crack is projected in the horizontal direction, there is still an obvious gap in its projection amplitude, but when it is projected in the vertical direction, there is an obvious maximum peak value in its projection amplitude;

[0040] ③ The horizontal and vertical projection curves of the network cracks are similar, and the amplitude fluctuations are roughly the same;

[0041] (2) Quantification of mesh cracks: The mesh cracks use an envelope rectangle to represent the crack periphery. The outer envelope rectangle is determined by finding the edge points of the crack target in the crack segmentation algorithm, and the four edge points of the envelope rectangle are obtained. Then, their extreme values ​​are taken as the geometric parameters of the envelope rectangle. Therefore, for the quantification of mesh cracks, the envelope rectangle method is used to calculate the damage area:

[0042] A=Hμ·Wμ

[0043] Considering the conversion parameter t between the front view image coordinate system and the road plane coordinate system, the formula is converted to:

[0044] A=Hμ·Wμ·t

[0045] (3) Calculation of linear crack length: Since the skeleton of a linear crack is composed of several small segments, the length of each small segment is calculated and summed up to get the length of the crack.

[0046] First, find the starting and ending points of the crack skeleton, assuming that their corresponding coordinates are (x1, y1) and (x m ,y n ), where the coordinates of any two adjacent points are (x t ,y t ) and (x t+1 ,y t+1 ), the total length L of the crack can be calculated according to the following formula:

[0047]

[0048]

[0049] Where D is the total length of the image, and Lt is the length between two adjacent points;

[0050] Taking into account the conversion parameter t between the front view image coordinate system and the road plane coordinate system, the formula conversion is as follows:

[0051]

[0052]

[0053] (4) Calculation of linear crack width: The crack width can be simply expressed as the ratio of the sum of the number of pixels in the skeleton to the crack length. The formula is as follows:

[0054] W=sum / L

[0055] Taking into account the conversion parameter t between the front view image coordinate system and the road plane coordinate system, the formula conversion is as follows:

[0056] W=sum / L·t

[0057] Where W is the width of the crack;

[0058] From the above, the present invention provides a method for detecting cracks in a vehicle-mounted front view image, which introduces a deep learning classification method, an HC saliency algorithm, and a grayscale histogram-based image threshold segmentation algorithm for combined use. The processing process of the method of the present invention is not approximated, and can be used in application fields such as road disease detection, which can effectively improve the accuracy and stability of road crack detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The description of the content of the present invention will become obvious and easy to understand in conjunction with the following drawings, in which:

[0060] Figure 1 This is a flow chart of a vehicle-mounted front view image crack detection method of the present invention

[0061] Figure 2 Three network verification results

[0062] Figure 3 Image preprocessing diagram

[0063] Figure 4 Comparison of saliency detection algorithms

[0064] Figure 5 Comparison of threshold segmentation methods

[0065] Figure 6 Binarization of significance detection results

[0066] Figure 7 Comparison between HC saliency detection algorithm after threshold segmentation and single histogram-based threshold segmentation

[0067] Figure 8 Comparison of skeleton extraction results DETAILED DESCRIPTION

[0068] See below Figures 1 to 8 Tables 1 to 5 provide detailed descriptions of a vehicle-mounted front view image crack detection method of the present invention.

[0069] like Figure 1 As shown, the present invention provides a vehicle-mounted front view image crack detection method, comprising the following steps:

[0070] S1: For the vehicle-mounted front view image, the CNN road semantic segmentation method is used to separate the road surface and background areas to obtain the road surface image;

[0071] S2: Divide the road surface image obtained in step S1 into image blocks of regular size of 10 cm×10 cm, and use the ResNet network to perform binary classification on the image blocks to obtain cracked and non-cracked road surface image blocks;

[0072] S3: performing image enhancement and other preprocessing on the cracked pavement image block obtained in step S2, and performing crack detection using an improved HC saliency detection and grayscale histogram threshold segmentation method to obtain a crack segmentation image;

[0073] S4: extracting the crack contours by mathematical morphology from the crack detection image obtained in step S3, identifying the type of cracks by a histogram projection method, and establishing a two-dimensional coordinate system to calculate information such as the length and width of the cracks.

[0074] The method of the present invention is further illustrated below by means of implementation examples.

[0075] Dataset: Based on the collected image data and grid numbers, the images after chessboard grid division are manually classified, and the divided images are classified into two categories: crack images and non-crack images. All images with cracks are in one category, and all images without cracks are in another category. They are divided into training set and validation set, and the number of images contained in the training set and validation set are 5000 and 400, respectively.

[0076] Evaluation indicators: Calculate precision (Pr), recall (Re), F-measure Implementation steps:

[0077] a) For the collected images, the CNN road semantic segmentation method is used to separate the road surface and background areas to obtain the road surface image. According to the camera calibration and measurement method, the segmented road area image is divided into 10cm×10cm grid blocks. For the grid-blocked images, the ResNet50 residual network, VGG network, and MobilenetV2 lightweight mobile network are used to perform binary classification on the training set image blocks under the same parameters. The parameters are shown in Table 1. The same image block is verified by the three methods of VGG convolutional network, MobilenetV2 lightweight mobile network, and ResNet50 residual network. The results are shown in the figure. Figure 2 400 images were selected and tested with four indicators: Top-1 image accuracy, precision, recall rate, and F-measure. The results are shown in Table 2, which proves that the ResNet50 residual network can effectively classify image blocks with cracks for subsequent crack image preprocessing and detection and recognition, meeting the requirements of the experiment.

[0078] Table 1

[0079]

[0080] Table 2

[0081]

[0082] b) Preprocess the classified images with cracks. First, grayscale transformation is used to improve the contrast of the crack area so that the image reaches the ideal grayscale range. Then, the image is preprocessed through image grayscale correction, histogram equalization and median filtering. The processing results are shown in Figure 2. Figure 3 , while retaining and highlighting the image details, it also removes the image noise, solving the contradiction between denoising and retaining details.

[0083] c) After image preprocessing, the image is segmented by grayscale histogram threshold. The improved HC saliency detection algorithm is compared with the FT saliency detection algorithm, the AC saliency detection algorithm, and the LC saliency detection algorithm to obtain their respective saliency maps and perform binarization, as shown in Figure 4 , Figure 6 ,The results show that the improved HC saliency detection algorithm of the present invention is more suitable for crack detection than the other three traditional saliency detection algorithms.

[0084] The same image is subjected to comparative experiments using iterative global threshold processing, Otsu global threshold processing, and threshold segmentation based on grayscale histogram, and the optimal segmentation method is obtained through analysis. The experimental results are shown in Figure 2. Figure 6 , which proves that using the bottom value of the grayscale histogram as the threshold to segment the image can achieve good results.

[0085] The grayscale histogram threshold segmentation algorithm based on the improved HC saliency detection algorithm is combined with the single grayscale histogram-based threshold segmentation to segment the image and calculate the three indicators of accuracy (Precision), recall (Recall), and F-measure (F-Measure). The processing results are as follows Figure 7 The accuracy, recall rate and F-measure corresponding to the crack detection algorithm are shown in Table 3. It can be clearly seen from the chart that the algorithm of the present invention has a huge improvement in the detection and segmentation of pavement cracks and can effectively segment the target area from the background area.

[0086] Table 3

[0087]

[0088] d) The image after threshold segmentation is subjected to crack contour extraction based on mathematical morphology. The results are shown in Figure 8 . By constructing the structural elements to obtain the connected domain and extract the skeleton of the image, the outline of the grayscale change edge of the binary image can be clearly obtained, the fracture area can be spliced, and the isolated points can be effectively removed. Then the projection method is used to extract the crack features and identify the type, distinguishing whether it is a horizontal crack, a longitudinal crack, or a mesh crack. And the crack geometric characteristic parameters are obtained, which is convenient for accurately evaluating the damage condition of the highway pavement and reflecting the actual pavement condition.

[0089] e) 40 test images were selected for the experiment. After a series of the above operations, the results of crack extraction were compared with the results of manual crack identification to illustrate the accuracy of the algorithm in this paper in identifying cracks. After manual detection, it was found that there were 74 transverse cracks, 53 longitudinal cracks, and 29 reticular cracks on the road surface of the test section. Based on this, the algorithm used in this paper was used for crack detection. Some experimental results are shown in Tables 4 and 5.

[0090] Table 4

[0091]

[0092] Table 5

[0093]

[0094] The method for detecting cracks in a vehicle-mounted front-view image of the present invention introduces a histogram grayscale threshold segmentation algorithm based on saliency detection, so that cracks are clear and easy to extract, can adapt to crack detection in complex images, and has universal applicability and can be widely used in the field of road disease detection.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting cracks in a vehicle-mounted front view image. It is characterized in that The steps include: S1: For the vehicle-mounted front view image, the CNN road semantic segmentation method is used to separate the road surface and background areas to obtain the road surface image; S2: Divide the road surface image obtained in step S1 into image blocks of regular size n×n, and use a ResNet network to perform binary classification on the image blocks to obtain cracked and non-cracked road surface image blocks; S3: performing image enhancement preprocessing on the cracked pavement image block obtained in step S2, and performing crack detection using an improved HC saliency detection and grayscale histogram threshold segmentation method to obtain a crack segmentation image; S4: extracting the crack contours by mathematical morphology from the crack detection image obtained in step S3, identifying the type of the crack by a histogram projection method, and establishing a two-dimensional coordinate system to calculate the length and width information of the crack; The image preprocessing method in step S3 includes image grayscale correction processing, histogram equalization processing and median filtering denoising processing. The improved HC saliency detection method consists of two parts: HC saliency detection and fine-scale enhancement. After detection, the grayscale histogram threshold segmentation method is used to perform crack segmentation, including the following methods and features: (1) HC-based saliency detection: The saliency value of a pixel is defined by the color difference with all other pixels in the image, as shown below: In the formula, D(I K ,I I ) is the color distance metric between two pixels in the space L·a·b. After expanding the pixel level, the above formula can obtain the saliency value of each color. The formula is as follows: In the formula, C 1 Pixel I K The color value in, n is the number of different pixel colors, f j is the pixel color C in image I j The frequency of occurrence, the true color space contains 256*256*256 possible colors, which is more than the total pixels of the image, and the computational cost is high. Each color channel is quantized to 12, and the colors are reduced to 12*12*12. The low-frequency colors are discarded and the high-frequency colors are retained. The RGB color space quantization method will produce defects. In order to reduce errors, a smoothing operation is used to replace the weighted average of the salient values ​​of similar colors with the salient value of each color. However, similar colors need to measure the distance in the Lab color space. Let m nearest neighbor colors to improve the salient value of color C. The formula is as follows: In the formula, is the distance between color C and its m nearest neighbors; (2) Fine-scale saliency enhancement: The fine-scale saliency enhancement algorithm can reduce the saliency value of image noise. Cracks and background textures have a certain scale. Transforming the pavement crack image to a scale suitable for the crack will highlight the local characteristics of the crack. The scale transformation formula is as follows: I t+1 (x,y)=I(U N,t (x,y))×h(·) In the formula, t represents the image scale, t = 0 is the original image, U N,t (x, y) is the M×N area around the pixel point (x, y) at scale t, h(·) is the scale change kernel function, the crack scale changes with the image resolution, the crack shape is long and narrow, the scale should not be too fine, The fine-scale saliency enhancement algorithm simulates the linear expansion process of cracks. The formula for fine-scale saliency enhancement of cracks is as follows: θ∈{0°,45°,90°,135°} In the formula, I U is the grayscale mean of the crack scale image, let S ZU is the significant mean, w z (x,y) is the significance weight corresponding to the point (x,y), and the formula is as follows: In the formula, G N,θ (x, y) is the sum of the grayscale values ​​of the linear neighborhood in the θ direction centered at the point (x, y), N = 3; let the number of pixels in the original image be N. According to statistical laws, the proportion of candidate points in the original image is at most 10%. When the candidate point is a crack, the linear neighborhood points with a surrounding length of L do not need to be screened again; Grayscale histogram threshold segmentation method: For pavement crack images, when the grayscale values ​​of the target area and the background area are different, the grayscale histogram shows two peaks with a valley between the peaks; when the two peaks correspond to the central grayscale values ​​of the target area and the background area, respectively, the grayscale value corresponding to the valley value can be used as the threshold for image segmentation; assuming that the grayscale value corresponding to the valley is T, then T is used as the segmentation threshold, and the area composed of pixels with grayscale values ​​less than T is regarded as the crack area, and the area composed of pixels with grayscale values ​​greater than T is regarded as the background area; however, the pixel distribution of the crack area and the pavement area obeys the normal distribution, that is: In the formula, f 1 (i) is the distribution function of the crack area; From the normal distribution characteristics, we can see that a in the above formula 1 is the grayscale center value of all pixels in the crack area, u 1 represents the average gray value of all pixels in the crack, δ 1 2 represents the mean square error of the gray value; similarly, f 2 (i) represents the distribution function of the background area; The limiting conditions are as follows: When T makes f(T) take the minimum value, T is the required threshold value; since the range of gray value variation in the crack image is limited, it will not cover all gray value ranges; therefore, this point can be used to design the algorithm by only reading the number of pixels corresponding to the gray value in the crack to improve the operation efficiency of the algorithm; The crack contour extraction in step S4 uses a projection method to identify the crack image and distinguish its crack type, establishes a two-dimensional coordinate system to calculate the length and width of the crack, including the following methods and features: (1) Crack feature extraction and type recognition based on projection method: Feature extraction is performed on images containing cracks, and different types of pavement cracks are identified based on different feature values. The binary image after crack segmentation is projected in different directions, and the geometric features of the cracks are determined by counting the number of pixels. The projection results have the following characteristics: ① The horizontal projection amplitude of the transverse cracks differs greatly, and the data changes significantly. When projecting it in the vertical direction, there is still a significant difference in the projection amplitude, but it appears relatively smooth, and the number of pixels in each row is roughly the same; ② When the longitudinal crack is projected in the horizontal direction, there is still an obvious gap in its projection amplitude, but when it is projected in the vertical direction, there is an obvious maximum peak value in its projection amplitude; ③ The horizontal and vertical projection curves of the network cracks are similar, and the amplitude fluctuations are roughly the same; (2) Quantification of network cracks: The network cracks use an envelope rectangle to represent the crack periphery. The outer envelope rectangle is determined by finding the edge points of the crack target in the crack segmentation algorithm, and the four edge points of the envelope rectangle are obtained. Then, their extreme values ​​are taken as the geometric parameters μ of the envelope rectangle. Therefore, for the quantification of network cracks, the envelope rectangle method is used to calculate the damage area A: A=Hμ·Wμ In the formula, H and W are the height and width of the rectangle. Considering the conversion parameter t between the front view image coordinate system and the road plane coordinate system, the formula is converted to: A=Hμ·Wμ·t (3) Calculation of linear crack length: Since the skeleton of a linear crack is composed of several small segments, the length of each small segment is calculated and summed up to get the length of the crack. First, find the starting and ending points of the crack skeleton, assuming that their corresponding coordinates are (x 1 ,y 1 ) and (x m ,y n ), where the coordinates of any two adjacent points are (x t ,y t ) and (x t+1 ,y t+1 ), the total length L of the crack can be calculated according to the following formula: Where D is the total length of the image, L t is the length of two adjacent points; Considering the conversion parameter t between the front view image coordinate system and the road plane coordinate system, the formula conversion is as follows: (4) Calculation of linear crack width: The crack width can be simply expressed by the ratio of the sum of the number of pixels in the skeleton to the crack length L, as shown in the following formula: W=sum / L Considering the conversion parameter t between the front view image coordinate system and the road plane coordinate system, the formula conversion is as follows: W=sum / L·t Where W is the width of the crack.

Images