An intelligent detection method for rural road pavement defects based on deep convolutional neural network

Abstract

The present invention discloses an intelligent detection method for rural road pavement diseases based on a deep convolutional neural network, and the steps are: preparing a multi-type rural road pavement disease training data set; constructing a semantic segmentation model based on a fully convolutional network; using the Adam optimizer to optimize the model training process; cropping images with overlaps and adopting an edge-ignoring strategy when splicing, cropping large images into a series of image blocks with specific repeated areas with adjacent image blocks, and storing them in a linked list, then creating a generator, and then sending them to the model of the training number for prediction; multi-type pavement disease classification and damage condition assessment; by developing a pavement technical condition evaluation system. The present invention can solve the problem that the context information of the edge area of ​​the cropped image is less, the prediction result accuracy is low, and then the extracted disease has obvious splicing marks, and achieve the accuracy and continuity of the disease prediction results in high-resolution images.

Description

Technical Field

[0001] The present invention belongs to the technical field of target recognition and information extraction, and in particular relates to an intelligent detection method for rural road pavement defects based on a deep convolutional neural network. Background Art

[0002] The Statistical Bulletin on the Development of the Transportation Industry in 2020 shows that by the end of 2020, the total mileage of highways in my country reached 5.1981 million kilometers, an increase of 185,600 kilometers over the previous year, of which the mileage of rural roads was 4.3823 million kilometers, accounting for 93%. The rural road network covering the whole country has basically been formed, and the era of highway maintenance is coming! Whether it is cement or asphalt pavement, in the actual operation process, due to the comprehensive influence of traffic volume, load, temperature, moisture, weathering, etc., one-way cracks, mesh cracks, broken plates, potholes (holes), repairs and other damages will appear one after another, collectively referred to as pavement diseases. The pavement damage index PCI is an important indicator for comprehensively calculating the performance of pavement using the type, area, range and density of pavement diseases. Pavement disease detection and quantitative evaluation are important components of intelligent road maintenance decision-making. Therefore, how to scientifically, quickly and quantitatively evaluate highway maintenance, reasonably formulate maintenance plans, and keep the road network at a high performance and service level has become a key technical problem that needs to be solved to achieve the sustainable development of rural roads.

[0003] In recent years, domestic and foreign researchers have conducted large-scale research based on computer vision in three main directions: image processing technology, machine learning, and deep learning. Faced with a large number of annual inspection tasks, the maintenance department traditionally uses automated equipment to complete large-scale collection of two-dimensional road images, and then manually judges, outlines, and calculates and evaluates the PCI index. This method has a long detection cycle, high cost, and strong subjectivity, making it difficult to achieve real-time, large-scale automatic detection and evaluation of road technical conditions.

[0004] Secondly, the disease extraction methods based on image processing techniques (IPTS) are mainly divided into five categories: threshold-based segmentation, edge detection-based segmentation, multi-scale segmentation, texture and fractal geometry feature-based segmentation, and other segmentation methods. Although the algorithm has a low computational cost, its universality is not high. Its detection performance depends largely on the quality of the image to be processed, which is difficult to meet engineering-level applications. With the rapid development of computer vision technology, domestic and foreign researchers have extracted pavement cracks through machine learning methods, mainly including: using support vector machine (SVM) and decision tree algorithm, using Candy operator and K-means clustering algorithm, and crack extraction based on random structure forest. Such methods simulate cracks by manually setting color or texture features, and rely heavily on the quality of intermediate features extracted by domain experts; in these methods, the manually set features can only meet the crack detection in certain specific situations. When a new crack environment appears, it needs to be reset, which cannot meet the detection requirements of pavement diseases in all situations. In recent years, deep learning technology has achieved great success in various computer vision tasks, replacing manual feature setting with automatic learning. This study mainly focuses on two aspects: pixel-level segmentation of pavement cracks and target detection. Object detection-based methods usually use bounding box methods to locate defects in pavement images. These methods first use CNN to extract different disease features from the image, then generate a region of interest (RoI), and finally detect the object through the bounding box coordinates. Pixel segmentation-based disease extraction algorithms often use the encoder-decoder architecture to build a deep neural network for disease detection. The encoder layer mainly implements the feature mapping of the image, and the decoder is used to fuse multi-scale semantic features during the segmentation process to generate a probability distribution for each pixel.

[0005] Rural road pavement damage has many types, complex topological structures, poor spatial continuity of damage pixels, variable shooting scenes, and low contrast. Although great progress has been made, the following problems still exist:

[0006] (1) Traditional image processing technology is greatly affected by the complexity of the image shooting environment and the multiple sources of noise. It requires manual feature extraction to distinguish multiple types of diseases. The algorithm fails to achieve satisfactory results in terms of accuracy, real-time and consistency.

[0007] (2) The existing public datasets of road damage have problems such as small scale and few scenarios, which makes it difficult to support the research and application of data-driven deep learning technology in intelligent detection of road damage. So far, there is no large-scale, public, category-labeled dataset for semantic segmentation of road damage images based on measured road images in the world.

[0008] (3) The size of road surface images collected is large, and there are few studies on end-to-end models that take into account the accuracy, speed and multi-type detection of disease prediction. Summary of the invention

[0009] Based on the above deficiencies in the prior art, the technical problem solved by the present invention is to provide an intelligent detection method for rural road pavement defects based on a deep convolutional neural network, which solves the problem that the context information in the edge area of ​​the cropped image is less and the prediction result accuracy is low, which leads to obvious splicing marks in the extracted defects, and achieves the accuracy and continuity of the defect prediction results in high-resolution images.

[0010] In order to solve the above technical problems, the present invention is implemented by the following technical solutions:

[0011] The present invention provides a method for intelligent detection of rural road pavement defects based on a deep convolutional neural network, comprising the following steps:

[0012] Step 1: Create a multi-type rural road pavement disease training dataset;

[0013] Step 2: Build a semantic segmentation model based on a fully convolutional network;

[0014] Step 3: Use the Adam optimizer to optimize the model training process;

[0015] Step 4: Crop the images with overlaps and ignore the edges when stitching. Crop the large image into a series of image blocks with specific overlapping areas with adjacent image blocks, store them in a linked list, create a generator, and then send it to the model of the training number for prediction;

[0016] Step 5: Classification of various types of pavement damage and assessment of damage conditions;

[0017] Step 6: By developing a pavement technical condition evaluation system, the geometric feature information and type information of various disease regression frames are counted, and then the pavement disease damage rate DR of the 100-meter evaluation unit is calculated. The evaluation model is shown in the following formula:

[0018]

[0019] Where DR is the pavement damage rate, which is the percentage of the sum of the converted areas of various diseases and the pavement survey area; A i is the area of ​​pavement damage of the i-th type; A is the investigated pavement area; ωi is the weight of the i-th type of pavement damage.

[0020] Preferably, the step 1 comprises:

[0021] Step 1.1: Collect multi-year and multi-scene pavement image data to build a pavement disease image benchmark library, and add normal pavement images as negative samples;

[0022] Step 1.2: Develop a targeted disease target image cropping program to complete the preprocessing of the original images of the dataset;

[0023] Step 1.3: Use Labelme to perform expert-level manual annotation on the disease dataset to construct a semantic segmentation dataset of multiple types of pavement diseases with rich scenarios and strong heterogeneity.

[0024] Furthermore, the step 2 comprises:

[0025] Step 2.1: Based on the U-shaped network architecture, extract the downsampling part and use VGG16 as the backbone feature extraction network to obtain five effective feature layers, namely f1, f2, f3, f4, and f5;

[0026] Step 2.2: f5 is sent to the context information extraction module CI EM, which obtains the context information extraction of pavement damage at multiple scales by providing hole convolution branches with different receptive fields, encodes the high-level semantic feature map, and the convolution operation with a large receptive field extracts and generates more abstract features for large targets;

[0027] Step 2.3: Send f1, f2, f3, and f4 to the attention mechanism module SE and the feature fusion module FFM to complete feature decoding;

[0028] Step 2.4: Finally, a pixel-by-pixel classification label map with the same size as the input image is output, and the value of each pixel shows the semantic category to which the pixel belongs.

[0029] Optionally, the step 2.3 includes:

[0030] Step 2.3.1: Send the four feature maps obtained by the encoder as input signals to the attention mechanism module, learn the feature weights through the Loss area, obtain the importance of each feature map, increase the channel weight of the effective target area, suppress the channel features of the non-target area, improve the target segmentation effect, and avoid increasing parameters and calculations;

[0031] Step 2.3.2: A feature fusion module FFM is proposed. The feature fusion module first connects the low-level features and the previous high-level features after deconvolution, uses global maximum pooling to extract feature texture, reduces the influence of useless information, and then uses BN to balance the feature scale to obtain an interpretable weight matrix corresponding to the scale of the encoder's shallow feature map; then multiply it with the high-order feature map to calculate the weighted feature vector, and finally add the output feature map to the low-order feature map, which can use the high-level and low-level mixed feature maps to restore the location of image pixels.

[0032] Optionally, step 2.3.1 has the following three operations:

[0033] The first operation is Squeeze: Through global average pooling, the two-dimensional features (H×W) of each channel are compressed into a real number. This real number is calculated based on all the values ​​of the two-dimensional features. Finally, it is transformed into 1×1×C through the Squeeze operation, where the calculation formula for the cth element of z is as follows:

[0034]

[0035] The second operation Excitation: The weight value is generated for each feature channel through the parameters. The correlation between each channel is formed by the bottleneck structure of two fully connected layers, and the output is the same as the number of input weight values. The threshold mechanism expression in the form of sigmoid function is shown as follows:

[0036] s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1 z))

[0037] Where δ is the ReLU function, Use two fully connected layers to filter the parameters using a threshold mechanism. The first fully connected layer reduces the dimension. The dimension reduction coefficient r is a hyperparameter. Then the ReLU function is used for activation. The second fully connected layer restores the original dimension.

[0038] The third operation Scale: assign the weight obtained by the previous normalization to the features of each channel, multiply each channel by its corresponding channel coefficient, introduce the attention mechanism into the channel dimension, activate the learned features of each channel using the sigmoid function, and then multiply it by the original features on U. The expression is shown in the following formula:

[0039]

[0040] In the formula and F scale (u c ,s c ) represents u c ∈R H×W and the scalar s c The channel multiplication between .

[0041] Optionally, the specific method of step 3 is: during the training process, the pre-trained weight model uses the optimal weights trained by the VOC dataset, and the parameters of the backbone feature extraction network are frozen in the freezing stage; after the frozen generation training is completed, the training unfreezing stage is started, multiple parameter combinations are tried through parallel calculation, and the best parameter combination is selected through verification.

[0042] Further, the step 5 comprises:

[0043] Step 5.1: Based on the multi-type disease classification samples, program to extract the statistical characteristics and structural characteristics of pavement diseases;

[0044] Step 5.2: Optimization design of pavement disease classification network, using radial basis probability neural network to build a multi-type disease classification model; the input layer is designed with 4 nodes corresponding to the feature vector, the first hidden layer neurons are determined by k-means clustering method, the second hidden layer sums the first hidden layer by category, the connection weights between the second hidden layer and the output layer are determined by recursive least squares method, and the output layer is designed with 6 nodes, corresponding to transverse cracks, longitudinal cracks, block cracks, cracks, potholes and repairs;

[0045] Step 5.3: Calculate the damage parameters and PCI evaluation index. The average width of the cracks is calculated by the following formula, and the damage level of the linear cracks is determined based on this:

[0046]

[0047] Among them, W c is the total number of crack pixels in the image, W s is the total number of crack pixels in the skeleton; the area of ​​mesh cracks, potholes and repairs is calculated using the optimal circumscribed rectangle, and finally the pavement disease parameters of a single image are output and stored in a txt file.

[0048] From the above, the rural road pavement disease intelligent detection method based on deep convolutional neural network of the present invention has at least the following beneficial effects:

[0049] (1) The present invention has produced a multi-type pavement disease semantic segmentation dataset (ISTD-RPDS8). The existing public pavement disease datasets have problems such as small scale and few scenes, which make it difficult to support the research and application of data-driven deep learning technology in intelligent disease detection. So far, there is no large-scale, public, category-labeled dataset for semantic segmentation of road disease images based on measured pavement images in the world. Therefore, the present invention uses shooting environment, pavement pavement type, sensor, and disease type as screening factors, relying on the multi-year measured rural road image library in Liaoning Province, and has successively constructed a multi-type pavement disease semantic segmentation dataset (ISTD-RPDS8) with rich scenes and strong heterogeneity.

[0050] (2) The present invention is based on a deep learning semantic segmentation model of a fully convolutional network, taking into account the fusion of multi-scale spatial feature information and long-distance contextual information. By constructing a context information extraction module CIEM, an attention mechanism module SE and a feature fusion module FFM, a trainable end-to-end disease segmentation network structure (ISTD-net) is established to achieve accuracy and continuity of disease prediction results in high-resolution images.

[0051] (3) The present invention adopts the strategy of cropping images with overlaps in the actual prediction stage and ignoring edges when fusing the prediction results. This reduces the risk of memory calculation overflow when predicting large images, and solves the problem of low prediction accuracy due to less contextual information in the edge areas of the cropped images, which in turn leads to obvious splicing marks in the extracted defects.

[0052] (4) The present invention extracts (α, d, D(1), D(0)) from the intelligent detection result diagram as a feature vector, and constructs a multi-type disease classification model by constructing a radial basis probability neural network RBPNN, so as to solve the classification of transverse cracks, longitudinal cracks, block cracks, cracks, pits and repairs with complex topological structures, and solve the problem of calculating PCI according to disease type.

[0053] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following is a detailed description in combination with the preferred embodiments and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings of the embodiment are briefly introduced below.

[0055] Figure 1 It is the technical roadmap of the intelligent detection method of pavement defects of the present invention;

[0056] Figure 2 It is a schematic diagram of the organization of the pavement disease database of the present invention;

[0057] Figure 3 A schematic diagram of a multi-type pavement disease benchmark data set of the present invention;

[0058] Figure 4 It is an overall schematic diagram of the disease extraction deep neural network of the present invention;

[0059] Figure 5 Schematic diagram of the core functional modules of the disease extraction network of the present invention, wherein (a) is a diagram of the construction of the attention mechanism module SE, (b) is a diagram of the construction of the context information extraction module CI EM, and (c) is a diagram of the construction of the feature fusion module FM;

[0060] Figure 6 It is the change curve of loss, loss, f_score and f_score of the training set and the validation set in the training process in the specific implementation mode of the present invention;

[0061] Figure 7 It is a comparison diagram of the results of disease extraction using different depth convolutional neural networks of the present invention, wherein (a) is the original image, (b) is the true value, (c) is the Unet extraction result, (d) is the PSPNet extraction result, (e) is the DeepLabv3+ extraction result, and (f) is the method of the present invention;

[0062] Figure 8 This is a schematic diagram of ignoring edge prediction of the present invention;

[0063] Fig. 9 Schematic diagram of the comparison results of direct prediction and block-wise edge-ignoring prediction of the present invention, wherein (a) is the original image, (b) is the true value, (c) is the UNet segmentation result, (d) is the UNet block segmentation result, (e) is the PSPNet segmentation result, (f) is the PSPNet block segmentation result, (g) is the DeepLabv3+ segmentation result, (h) is the DeepLabv3+ block segmentation result, (i) is the segmentation result of the model of the present invention, and (j) is the block segmentation result of the model of the present invention;

[0064] Fig.10 It is a schematic diagram of the distinguishability of different disease characteristics of the present invention;

[0065] Fig.11 It is the structure diagram of the RBPNN model of the present invention;

[0066] Fig.12 It is a statistical diagram of the PCI calculation results of the measured data of the present invention;

[0067] Fig.13 Comprehensive comparative analysis charts were made for the three groups of automatic extraction, manual software delineation, and assisted semi-automatic adjustment in terms of disease extraction quality, PCI calculation results, and time cost;

[0068] Fig.14 This is a statistical chart showing that the difference between intelligent detection and manual PC I is less than 5 points. DETAILED DESCRIPTION

[0069] The following is a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings, which is a part of this specification and illustrates the principles of the present invention through examples. Other aspects, features and advantages of the present invention will become clear from the detailed description. In the referenced drawings, the same or similar components in different figures are represented by the same reference numerals.

[0070] like Figures 1 to 14 As shown, the present invention provides an intelligent detection method for rural road pavement defects based on a deep convolutional neural network, comprising the following steps:

[0071] Step 1: Create a multi-type rural road pavement disease training dataset;

[0072] Step 1.1: Collect pavement image data from multiple years and multiple scenes to construct a pavement disease image benchmark library. The present invention uses image clarity, pavement light intensity, pavement dryness and incidental interference as standards to screen original pavement images under 9 types of scenes; the new data set consists of 9 types of common diseases: asphalt unidirectional cracks, asphalt block cracks, asphalt cracks, asphalt potholes, asphalt repairs, cement cracks, cement repairs, cement potholes, and cement broken plates. At the same time, normal pavement images are added as negative samples. The present invention collects measured pavement image data of rural roads in Liaoning Province from 2014 to 2021 to construct a pavement disease image benchmark library, such as Figure 2 As shown in the figure, the benchmark database contains four types of pavement: high-quality asphalt pavement, low-quality asphalt pavement, high-quality cement pavement, and low-quality cement pavement; the pavement images of different pavement types cover 9 common scenes: clear image, blurred image, strong light, weak light, dry pavement, slippery pavement, shadow interference, stain interference, and covering interference; the disease types include asphalt unidirectional cracks, asphalt block cracks, asphalt cracks, asphalt potholes, asphalt repairs, cement cracks, cement repairs, cement potholes, and cement broken boards. At the same time, normal pavement images are added as negative samples.

[0073] Step 1.2: Develop a targeted disease target image cropping program to complete the preprocessing of the original images of the dataset;

[0074] Step 1.3: If Figure 3 As shown in the figure, the present invention uses Labelme to perform expert-level manual annotation on the disease dataset; constructs a multi-type pavement disease semantic segmentation dataset (ISTD-RPDS8) with rich scenarios and strong heterogeneity, covering two types of pavement, asphalt and cement. The categories include transverse cracks, longitudinal cracks, turtle cracks, potholes, repairs, broken plates, cement cracks, potholes and negative samples, totaling 61,665, which can meet the requirements of binary and multi-classification pavement disease feature extraction; test set: 400 unidirectional cracks, 170 turtle cracks, 125 potholes, 79 block cracks, 55 broken plates, 171 repairs, totaling 1,000.

[0075] Step 2: If Figure 4 As shown, the present invention constructs a semantic segmentation model based on a fully convolutional network. The network takes into account the fusion of multi-scale spatial feature information and long-distance context information, and establishes a trainable end-to-end disease segmentation network structure (ISTD-net);

[0076] Step 2.1: Based on the U-shaped network architecture, in order to better utilize the pre-trained weight model for transfer learning, the feature extraction downsampling part of the present invention uses VGG16 as the backbone feature extraction network to obtain five effective feature layers, namely f1, f2, f3, f4, and f5.

[0077] Step 2.2: If Figure 5 As shown in (b), f5 is sent to the context information extraction module CI EM, which obtains the context information extraction of road damage at multiple scales by providing hole convolution branches with different receptive fields, and encodes the high-level semantic feature map. Convolution operations with large receptive fields can extract and generate more abstract features for large targets, while convolution operations with small receptive fields are better for small targets.

[0078] Step 2.3: Send f1, f2, f3, and f4 to the attention mechanism module SE (Squeeze and Excitation) and the feature fusion module FFM (Feature Fusion Mode l) to complete feature decoding.

[0079] Step 2.3.1: Figure 5 As shown in (a), the four feature maps obtained by the encoder are sent as input signals to the attention mechanism module. The feature weights are learned through the Loss area to obtain the importance of each feature map, increase the channel weight of the effective target area, suppress the channel features of the non-target area, and improve the segmentation effect of the target while avoiding increasing parameters and calculation amount. There are mainly three operations:

[0080] The first operation is Squeeze: considering the output characteristics of the signal to each channel, the two-dimensional features (H×W) of each channel are compressed into a real number through global average pooling. This real number is calculated based on all values ​​of the two-dimensional features and has a global receptive field to a certain extent. The number of channels remains unchanged and finally becomes 1×1×C through the Squeeze operation, where the calculation formula for the cth element of z is shown in (3.1).

[0081]

[0082] The second operation, Excitation, generates weight values ​​for each feature channel through parameters. The correlation between each channel is formed by a bottleneck structure of two fully connected layers, and outputs the same number of weight values ​​as the input. The threshold mechanism expression in the form of a sigmoid function is shown in formula (3.2):

[0083] s=F ex(z,W)=σ(g(z,W))=σ(W2δ(W1 z)) (3.2)

[0084] Where δ is the ReLU function, In order to reduce the complexity of the model and improve the generalization ability, two fully connected layers are used to filter the parameters using the threshold mechanism. The first fully connected layer reduces the dimension, and the dimension reduction coefficient r is a hyperparameter. Then the ReLU function is used for activation, and the second fully connected layer restores the original dimension.

[0085] The third operation Scale: assign the weight obtained by the previous normalization to the features of each channel, multiply each channel by its corresponding channel coefficient, introduce the attention mechanism into the channel dimension, activate the learned features of each channel using the sigmoid function, and then multiply it by the original features on U. The expression is shown in formula (3.3):

[0086]

[0087] In the formula and F scale (u c ,s c ) represents u c ∈R H×W and the scalar s c The channel multiplication between .

[0088] Step 2.3.2: If Figure 5 As shown in (c), in order to make full use of the information of high-level and low-level features, the present invention proposes a feature fusion module FFM (Feature Fusion Mode l). The feature fusion module fuses the feature map output by the SE module and the up-sampled output feature map, specifically: first, the low-level features and the previous high-level features after deconvolution are connected, and the global maximum pooling is used to extract the feature texture to reduce the influence of useless information, and then BN is used to balance the feature scale to obtain an interpretable weight matrix corresponding to the scale of the shallow feature map of the encoder. This weight vector can reselect and combine the features, refine the fused features, and improve the feature expression ability. Then, the weighted feature vector is calculated by multiplying the high-order feature map, and finally the output feature map is added to the low-order feature map, so that the location of the image pixels can be restored using the high-level and low-level mixed feature maps.

[0089] Step 2.4: Finally, a pixel-by-pixel classification label map with the same size as the input image is output, and the value of each pixel shows the semantic category to which the pixel belongs.

[0090] Step 3: The server used for ISTD-net model training of the present invention is CPU: Ryzen 7 2700x8-core processor×16, GPU: Nvi di a GeForce RTX 2080ti. The learning rate used is 10e -5 The Adam optimizer optimizes the model training process.

[0091] During the training process, the pre-trained weight model uses the optimal weights trained by the VOC dataset. In the freezing stage, the parameters of the backbone feature extraction network will be frozen. At this time, only the parameters of the second half of the model will be adjusted, not all parameters. Because the extracted features are universal, no adjustment will be made in the feature extraction part. The training of the freezing process will last for 50 generations, the batch size is set to 8, and the frozen learning rate is set to 10e -4 , because there are fewer parameters, the learning rate is set slightly larger. After the frozen generation training is completed, the training thawing phase begins. At this time, the main part of the model will not be frozen, and the parameters of the entire model can be adjusted. At this time, the parameters to be adjusted become more, the amount of calculation increases, the batch size is set to 4, and the learning rate is set to 10e -5 , try multiple parameter combinations through parallel computing, and select the best parameter combination through verification.

[0092] like Figure 6 As shown in the figure, the change curves of l oss, val_l oss, f_score, and val_f_score of the training set and validation set during this training process. As of 100 Epoch, the l oss of the training set is 0.1788, and the l oss of the validation set is 0.1899; the f_score of the training set is 0.8878, and the val_f_score of the validation set is 0.8856; the performance of the ISTD-net model in the 1000 test set is Precision=0.9545, Reca ll=0.8634, F-core=0.9217, and mIOU=0.9115.

[0093] like Figure 7As shown in the figure, in order to verify the accuracy of ISTD-net in identifying pavement defects, the trained UNet, PSPNet, DeepLabv3+ and ISTD-net were used to identify the I STD-RPDS8 test set. The experimental results show that the model of the present invention is superior to other networks in terms of overall, edge and detail image extraction, and can extract crack-type defects in a fine and continuous manner. The FFM module uses features between different pixels to effectively enhance the target features at the edge of the image, thereby accurately segmenting edge targets. I STD-net can effectively make up for the defects of the original network, and the target segmentation effect is better. When the segmentation target is too large, the local feature extraction is inconsistent, which easily leads to incomplete segmentation of large targets. By adding FFM, the context-related information between image positions can be effectively simulated, and different local feature information can be connected. The SE module can use the weights of different channels to improve the segmentation accuracy of target information. I STD-net is superior to other models in terms of overall, edge and detail.

[0094] Step 4: Figure 8 As shown, since the actual road surface image resolution is relatively large (3517*2193), directly sending it to the model for prediction will cause memory overflow, and if the resolution is 512×512 image for prediction, the small target disease information will be lost; if the conventional regular grid is used for cropping and then prediction and splicing, the edge area of ​​each image block has less context information, so the prediction result is less accurate, which leads to obvious splicing marks. The present invention adopts the strategy of cropping images with overlap and ignoring edges when splicing. The large image is cropped into a series of image blocks with specific repeated areas with adjacent image blocks, and they are stored in a linked list, and then a generator is created, and then sent to the model of the training number for prediction. The actual cropped image prediction result is A, and the splicing result is a. The percentage of a in the area of ​​A is r, and the overlap ratio of adjacent cropped images is Finally, only the middle part of the prediction results is taken for splicing.

[0095] like Fig. 9 As shown in the figure, the prediction algorithm of block-wise ignoring edges can basically segment the complete object area on the test set, improve the accuracy of crack disease extraction, and increase the integrity and continuity of crack extraction. Compared with direct prediction, the accuracy is significantly improved and it has strong robustness.

[0096] Step 5: Classification of various types of pavement damage and assessment of damage conditions;

[0097] Step 5.1: Based on the multi-type disease classification samples, program to extract the statistical and structural characteristics of pavement diseases.

[0098] Fig.10This is a schematic diagram of the distinguishability of different disease features. The present invention uses the box counting method for the first time to calculate the fractal statistical features such as the box dimension (D(1)) and information dimension (D(0)) of the disease binary image, so as to distinguish four types of diseases: unidirectional cracks, network cracks, pits and repairs; by extracting the geometric centroid of the single crack binary image, the minimum circumscribed rectangle is generated, and the structural features of transverse cracks and longitudinal cracks are extracted by calculating the single crack azimuth α and the regional crack pixel distribution density d.

[0099] Step 5.2: Optimization design of pavement damage classification network. Fig.11 As shown, the present invention adopts a radial basis probabilistic neural network (RBPNN) to construct a multi-type disease classification model; the input layer designs 4 nodes corresponding to the feature vector (α, d, D(1), D(0)), the first hidden layer neurons are determined by the k-means clustering method, the second hidden layer sums the first hidden layer by category, the connection weight between the second hidden layer and the output layer is determined by the recursive least squares method, and the output layer designs 6 nodes, corresponding to transverse cracks, longitudinal cracks, block cracks, cracks, pits and repairs respectively.

[0100] Step 5.3: Calculation of damage parameters and PCI evaluation index. Pavement damage parameters include the length and width of linear cracks, mesh cracks, potholes and repaired areas. According to the linear crack characteristics specified in the "Technical Specifications for Highway Asphalt Pavement Maintenance", the crack width is refined into single pixels, the crack skeleton line is extracted, and the average width of the crack is calculated using formula (5.1), and the damage level of the linear crack is determined based on this.

[0101]

[0102] Among them, W c is the total number of crack pixels in the image, W s is the total number of crack pixels in the skeleton; the area of ​​mesh cracks, potholes and repairs is calculated using the optimal circumscribed rectangle, and finally the pavement disease parameters of a single image are output and stored in a txt file in the following format: disease box display width; disease box display length; disease box display position X; disease box image is position Y; image control width; image control width; disease type; actual length of the disease; actual width of the disease; actual area of ​​the disease; corresponding pile number; disease serial number; evaluation value; pavement paving type.

[0103] Step 6: Intelligent road surface disease detection system

[0104] like Fig.12As shown in the figure, according to the detection results of the above model, the two main functional modules of intelligent detection of pavement defects and automatic classification are integrated. By developing a pavement technical condition evaluation system, the geometric feature information and type information of various disease regression frames are statistically analyzed, and then the pavement disease damage rate DR of the 100-meter evaluation unit is calculated (the percentage of the sum of the converted damaged areas of various damages and the pavement survey area). The evaluation model is shown in the following formula:

[0105] PCI=100-15DR 0.412 (6.1)

[0106]

[0107] Where DR is the pavement damage rate, which is the sum of the converted areas of various diseases and the percentage of the pavement survey area (%); A i is the area of ​​road surface damage of type i (m 2 ); A is the road surface area under investigation (the product of the investigation length and the effective road surface width, m 2 );ω i is the weight of the i-th type of pavement damage. Different pavement damage types have different values. The specific weight values ​​can be found in the Highway Technical Condition Evaluation Standard. The system can cope with complex and changeable shooting environments, overcome various interferences on rural roads, and solve the problem that intelligent detection algorithms are difficult to adapt to disease extraction and classification in all scenarios. The system adds an interactive inspection module to complete the rapid inspection, modification, and standardized report output of the extraction results, ensuring the reliability and accuracy of the PCI index calculation. At the same time, the system can realize interactive inspection, targeted expansion of data sets, and standardized output of evaluation information.

[0108] The following experiments verify the actual performance of the method proposed in the present invention. The measured rural road pavement data in Liaoning Province in 2020 were selected to verify the reliability of the algorithm results of the present invention from three experiments: the missed detection rate of intelligent disease detection, the comprehensive analysis of PCI calculation results and time cost, and the consistency of PCI evaluation levels.

[0109] (1) Experiment on missed detection rate of intelligent disease detection

[0110] Based on the field data collected on rural roads in Liaoning Province in 2020, four types of routes, namely excellent, good, medium and poor, were selected according to the road conditions to evaluate the missed detection rate in the NON file. The specific results are shown in Table 1.

[0111] Table 1 Detection accuracy evaluation table

[0112]

[0113] As can be seen from the above table, the system performs well in the classification and identification of pavement defects. The images in the NON files do not need to participate in interactive inspection, and the efficiency is significantly improved compared with the traditional full browsing working mode.

[0114] (2) Comprehensive analysis experiment of PCI calculation results reliability and time cost

[0115] In order to verify the effectiveness of the automatic crack detection method of the present invention, pavement images of three experimental lines (about 21 km) were randomly selected. The experimental grouping was: automatic extraction by the system, manual software delineation, and auxiliary semi-automatic adjustment. A comprehensive comparative analysis was conducted from the aspects of disease extraction quality, PCI calculation results, and time cost. Fig.13 shown.

[0116] The results of the analysis experiment show that the difference between pure AI automatic extraction and manual delineation in road PCI scores (per 100 meters) is small, with a maximum difference of 5.1 points and an average difference of 3.05 points, which is due to the many subjective judgment factors in manual delineation; the semi-automatic mode that assists manual re-judgment based on the automatic extraction results has a score between the other two methods, and the experimental statistics show an average difference of 1.81 points. In addition, the full cycle of calculation for each machine is 0.8s, and the average number of kilometers that can be calculated every 24 hours is 216km, which is much more efficient than manual delineation (9km / working day). Experiments have shown that the processing scheme of this system can be used for the detection and evaluation of pavement diseases on rural roads.

[0117] (3) PCI evaluation level consistency test

[0118] The Highway Technical Condition Evaluation Standard classifies the pavement damage index PCI into five levels: excellent, good, medium, poor, and bad according to the final score of the evaluation unit. See Table 2 for details.

[0119] Table 2 PCI indicator classification standards

[0120]

[0121] In order to be close to the actual application-level highway pavement disease detection scenario, Experiment 3 randomly selected 111 100-meter evaluation units on 8 highways in Liaoyang City and Shenyang City, Liaoning Province for detection, and compared and analyzed the PCI level consistency of the detection data with the manual delineation results of the city; Fig.14 It is the percentage of evaluation units with a difference of less than 5 points in the average PCI score of the 8 experimental routes. Compared with the manual results, the average consistency of the PCI evaluation level was 72.07%.

[0122] Therefore, the algorithm proposed by the present invention is completely reliable and has great advantages in the automatic detection of rural road surface diseases.

[0123] The above is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and changes can be made without departing from the principle of the present invention, and these improvements and changes are also regarded as the protection scope of the present invention.

Claims

1. An intelligent detection method for rural road pavement defects based on deep convolutional neural network, characterized in that: The following steps are involved: Step 1: Create a multi-type rural road pavement disease training dataset; Step 2: Build a semantic segmentation model based on a fully convolutional network; Step 3: Use the Adam optimizer to optimize the model training process; Step 4: Crop the images with overlaps and ignore the edges when stitching. Crop the large image into a series of image blocks with specific overlapping areas with adjacent image blocks, store them in a linked list, create a generator, and then feed it into the trained model for prediction; Step 5: Classification of various types of pavement damage and assessment of damage conditions; Step 6: By developing a pavement technical condition evaluation system, the geometric feature information and type information of various disease regression frames are counted, and then the pavement disease damage rate DR of the 100-meter evaluation unit is calculated. The evaluation model is shown in the following formula: Where DR is the pavement damage rate, which is the percentage of the sum of the converted areas of various diseases and the pavement survey area; A i is the area of ​​road surface damage of type i; A is the area of ​​road surface under investigation; ω i is the weight of the i-th type of pavement damage; The step 2 comprises: Step 2.1: Based on the U-shaped network architecture, extract the downsampling part and use VGG16 as the backbone feature extraction network to obtain 5 effective feature layers f1, f2, f3, f4, and f5 respectively; Step 2.2: Send f5 to the context information extraction module CIEM, and obtain the context information extraction of pavement damage at multiple scales by providing hole convolution branches with different receptive fields, encode the high-level semantic feature map, and the convolution operation with a large receptive field extracts and generates more abstract features for large targets; Step 2.3: Send f1, f2, f3, and f4 to the attention mechanism module SE and the feature fusion module FFM to complete feature decoding; Step 2.4: Finally, a pixel-by-pixel classification label map with the same size as the input image is output, and the value of each pixel shows the semantic category to which the pixel belongs; The step 2.3 comprises: Step 2.3.1: Send the four feature maps obtained by the encoder as input signals to the attention mechanism module, learn the feature weights through the Loss area, obtain the importance of each feature map, increase the channel weight of the effective target area, suppress the channel features of the non-target area, improve the target segmentation effect, and avoid increasing parameters and calculations; Step 2.3.2: A feature fusion module FFM is proposed. The feature fusion module first connects the low-level features and the previous high-level features after deconvolution, uses global maximum pooling to extract feature textures, reduces the influence of useless information, and then uses BN to balance the feature scale to obtain an interpretable weight matrix corresponding to the scale of the encoder's shallow feature map; then multiply it with the high-order feature map to calculate the weighted feature vector, and finally add the output feature map to the low-order feature map, which can use the high-level and low-level mixed feature maps to restore the location of image pixels; The step 2.3.1 has the following three operations: The first operation is Squeeze: Through global average pooling, the two-dimensional feature H×W of each channel is compressed into a real number. This real number is calculated based on all the values ​​of the two-dimensional feature. Finally, it is transformed into 1×1×C through the Squeeze operation, where the calculation formula for the cth element of z is as follows: The second operation, Excitation, generates weight values ​​for each feature channel through parameters. The correlation between each channel is formed by a bottleneck structure of two fully connected layers, and outputs the same number of weight values ​​as the input. The threshold mechanism expression in the form of a sigmoid function is shown as follows: s=F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z)) Where δ is the ReLU function, Use two fully connected layers to filter the parameters using a threshold mechanism. The first fully connected layer reduces the dimension. The dimension reduction coefficient r is a hyperparameter. Then the ReLU function is used for activation. The second fully connected layer restores the original dimension. The third operation Scale: assign the weight obtained by the previous normalization to the features of each channel, multiply each channel by its corresponding channel coefficient, introduce the attention mechanism into the channel dimension, activate the learned features of each channel using the sigmoid function, and then multiply it by the original features on U. The expression is shown in the following formula: In the formula and F scale (u c ,s c ) represents u c ∈R H×W and the scalar s c The channel multiplication between .

2. The rural road pavement disease intelligent detection method based on deep convolutional neural network according to claim 1 is characterized in that: The step 1 comprises: Step 1.1: Collect multi-year and multi-scene pavement image data to build a pavement disease image benchmark library, and add normal pavement images as negative samples; Step 1.2: Develop a targeted disease target image cropping program to complete the preprocessing of the original images of the dataset; Step 1.3: Use Labelme to perform expert-level manual annotation on the disease dataset and construct a semantic segmentation dataset of multiple types of pavement diseases with rich scenarios and strong heterogeneity.

3. The rural road pavement disease intelligent detection method based on deep convolutional neural network according to claim 1 is characterized in that: The specific method of step 3 is as follows: during the training process, the pre-trained weight model uses the optimal weights trained by the VOC dataset, and the parameters of the backbone feature extraction network are frozen in the freezing stage; after the frozen generation training is completed, the training unfreezing stage is started, multiple parameter combinations are tried through parallel calculation, and the best parameter combination is selected through verification.

4. The rural road pavement disease intelligent detection method based on deep convolutional neural network as claimed in claim 2 is characterized in that: The step 5 comprises: Step 5.1: Based on the multi-type disease classification samples, program to extract the statistical characteristics and structural characteristics of pavement diseases; Step 5.2: Optimization design of pavement disease classification network, using radial basis probability neural network to build a multi-type disease classification model; the input layer is designed with 4 nodes corresponding to the feature vector, the first hidden layer neurons are determined by k-means clustering method, the second hidden layer sums the first hidden layer by category, the connection weights between the second hidden layer and the output layer are determined by recursive least squares method, and the output layer is designed with 6 nodes, corresponding to transverse cracks, longitudinal cracks, block cracks, cracks, potholes and repairs; Step 5.3: Calculate the damage parameters and PCI evaluation index. The average width of the cracks is calculated by the following formula, and the damage level of the linear cracks is determined based on this: Among them, W c is the total number of crack pixels in the image, W s is the total number of crack pixels in the skeleton; the area of ​​mesh cracks, potholes and repairs is calculated using the optimal circumscribed rectangle, and finally the pavement disease parameters of a single image are output and stored in a txt file.

Images