Asphalt pavement crack detection method and related device thereof

Abstract

The invention discloses an asphalt pavement crack detection method and a related device thereof, and belongs to the technical field of asphalt pavement crack detection, and the method comprises the following steps: obtaining a driving video of a pavement; splicing crack video frames of the driving video to obtain an image containing complete crack features; deblurring the image containing the complete crack features by using an NAFNet network to obtain a clear RGB image; performing crack segmentation on the clear RGB image by using a DDNet network to obtain an image after crack segmentation processing; and calculating crack geometric parameters of the image after crack segmentation processing, and completing asphalt pavement crack detection. The intelligent mobile phone is adopted to collect the driving video, crack recognition can be performed on the driving video, and the crack extraction performance and recognition precision are effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of asphalt pavement crack detection, and in particular relates to an asphalt pavement crack detection method and a related device. Background Art

[0002] Affected by environmental climate, vehicle load, construction and other conditions, cracks have become the main surface disease of asphalt pavement, seriously affecting driving safety. Traditional asphalt pavement crack information is collected manually, which is not only inefficient and has large errors, but also poses safety hazards. The development and application of digital image processing technology has improved detection efficiency and safety, but it still requires manual intervention, which is relatively cumbersome.

[0003] With the rapid rise of deep learning algorithms in the field of computer vision, driven by the accumulation of large-scale data, the model can autonomously learn the feature information in the image, thereby realizing the automatic detection of road cracks, greatly improving the accuracy of detection. However, most of the crack recognition methods based on deep learning algorithms are currently based on still images. This method is inefficient and not suitable for large-scale crack collection. In addition, for cracks on highways, fixed-point still images require the closure of the road section, which brings great inconvenience to traffic.

[0004] At present, the equipment for collecting driving videos mainly includes comprehensive inspection vehicles, driving recorders and on-board cameras. Although comprehensive inspection vehicles and on-board cameras can obtain video images of cracks at high speeds and with high quality, their costs are relatively high. Although the driving recorder has a very low cost, the range of viewing angle adjustment when recording driving videos is limited, and the image quality is low. In general, the driving videos recorded by smartphones have high image quality and average cost, which is a better solution for collecting driving videos. However, since driving video images are collected during motion, motion blur and other problems often occur, resulting in a decrease in image quality, which in turn affects the recognition accuracy of cracks. In addition, when the driving video is framed, there will be a large number of similar images, that is, the same crack exists in multiple adjacent video images at the same time, and repeated statistical problems are prone to occur when calculating the geometric parameters of the cracks. There are also large differences between the crack recognition technologies based on static images and driving videos. The existing crack recognition technologies based on static images cannot be directly used for crack recognition based on driving videos. Therefore, it is of certain practical value to study the crack recognition technology based on driving videos. Summary of the invention

[0005] The purpose of the present invention is to provide an asphalt pavement crack detection method and related devices, which use a smart phone to collect driving videos, identify cracks based on the driving videos, and effectively improve the crack extraction performance and recognition accuracy.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, a method for detecting cracks in an asphalt pavement comprises the following steps: Get driving video of the road; splicing the crack video frames of the driving video to obtain an image containing complete crack features; Using the NAFNet network to deblur the image containing complete crack features to obtain a clear RGB image; Using the DDNet network to perform crack segmentation on the clear RGB image to obtain an image after crack segmentation processing; The crack geometric parameters of the image after the crack segmentation processing are calculated to complete the asphalt pavement crack detection.

[0007] In some embodiments, the crack video frames of the driving video are spliced ​​together through image registration and image fusion to obtain an image containing complete crack features.

[0008] In some embodiments, the step of splicing the crack video frames of the driving video by image registration and image fusion to obtain an image containing complete crack features specifically includes: Firstly, the feature points of the crack video frame are extracted by an accelerated robust feature algorithm, and the feature points are matched by using a K-nearest neighbor algorithm after constructing the descriptors of the feature points to obtain matching results; After the matching result is purified by using a random sampling consensus algorithm, the optimal stitching line of two crack video frames is determined in combination with a dynamic programming method, and a multi-resolution fusion method is used to fuse them to obtain an image containing complete crack features.

[0009] In some implementations, a NAFNet feature extraction module is obtained by using a simple gating module and a simplified channel attention module, and the NAFNet feature extraction module is combined with a U-Net architecture to obtain a NAFNet network.

[0010] In some embodiments, the step of using the NAFNet network to deblur the image containing complete crack features to obtain a clear RGB image specifically includes: The image containing complete crack features with a resolution of H×W is input into the NAFNet network, and after the number of channels is expanded by a 3×3 convolution block, it enters the encoding block for feature extraction to obtain a feature map, and the feature map is downsampled and reduced in size and increased in channels by a 2×2 convolution kernel and a convolution with a step size of 2; Then, upsampling is performed through the intermediate block transition, and the channels of the feature map are rearranged and increased according to 2×2 blocks through the PixelShuffle operation, and finally a clear RGB image is obtained through a 3×3 convolution block; the encoding block is the NAFNet feature extraction module.

[0011] In some implementations, the step of using the DDNet network to perform crack segmentation on the clear RGB image to obtain an image after crack segmentation processing specifically includes: First, the clear RGB image with a resolution of H×W is upsampled by a neighboring interpolation method to obtain a clear RGB image with a resolution of 2H×2W, and then the clear RGB image with a resolution of H×W is input to a feature extraction layer to obtain a first feature map, and the clear RGB image with a resolution of 2H×2W is input to the feature extraction layer to obtain a second feature map; Inputting the first feature map and the second feature map into a dual-scale cross attention module to perform weight balance distribution on the target object to obtain an integrated feature map; After downsampling the second feature map, the integrated feature map and the first feature map are fused, processed through a 1×1 convolution block and input into the supervised decoding module together with the feature map generated by Layer4. The result is then processed through a 1×1 convolution block and input into the supervised decoding module together with the feature map generated by Layer3. This operation is repeated by analogy until the module processing is completed with the feature map generated by Layer1, and the image after crack segmentation processing is output.

[0012] In some embodiments, the step of calculating the crack geometric parameters of the crack segmentation processed image specifically includes: Using a closing operation based on mathematical morphology to connect the break between two cracks in the image after the crack segmentation process; Calculating the number of background connected regions of the image after crack segmentation processing, and classifying the cracks to obtain crack types according to the number of background connected regions; The fracture geometric parameters of different fractures corresponding to the fracture types are calculated.

[0013] In a second aspect, an asphalt pavement crack detection system includes: Video acquisition module, used to obtain driving video of the road; A preprocessing module, used for splicing the crack video frames of the driving video to obtain an image containing complete crack features; An image deblurring module is used to deblur the image containing complete crack features by using a NAFNet network to obtain a clear RGB image; A crack segmentation module is used to perform crack segmentation on the clear RGB image using a DDNet network to obtain an image after crack segmentation processing; The crack geometric parameter calculation module is used to calculate the crack geometric parameters of the image after the crack segmentation processing to complete the asphalt pavement crack detection.

[0014] In a third aspect, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable in the processor, wherein the processor implements the steps of the asphalt pavement crack detection method when executing the computer program.

[0015] In a fourth aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the asphalt pavement crack detection method are implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides an asphalt pavement crack detection method, which obtains pavement driving videos and can detect pavement cracks based on driving videos; secondly, the present invention splices crack video frames to obtain complete crack features to avoid repeated crack calculations; finally, the present invention deblurs the image through the NAFNet network, which can visually deepen the cracks and improve the accuracy of subsequent semantic segmentation; finally, the present invention uses the DDNet network to perform crack segmentation on the image processed by NAFNet, thereby improving crack segmentation accuracy and crack continuity. The average error of the crack geometric parameters calculated by the present invention is less than 8%, and a complete and accurate pavement crack extraction result can be output, which effectively improves the crack extraction performance and recognition accuracy, and can meet the needs of actual engineering projects.

[0017] Furthermore, the present invention adopts image registration and image fusion when splicing crack video frames, which can reduce processing time and improve splicing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The video recording device provided in Example 1 is assembled into a diagram; Figure 2 This is a diagram of a simple gate control module used in Example 1; Figure 3 This is a simplified channel attention module diagram used in Example 1; Figure 4 A feature extraction module diagram of the NAFNet network provided in Example 1; Figure 5 A diagram of the NAFNet network model structure provided in Example 1; Figure 6 The DDNet network structure diagram provided for Example 1; Figure 7 Schematic diagram of linear crack geometric parameter calculation used in Example 1, wherein (a) is a schematic diagram of linear crack length calculation, and (b) is a schematic diagram of linear crack width calculation; Figure 8 This is a schematic diagram of the result after splicing the crack video frames provided in Example 1; Fig. 9 Schematic diagram of segmentation visualization results provided in Example 1, where (a), (b), (c), and (d) are visualization results after segmentation of different pavement crack images, respectively; Fig.10 Comparison diagram of crack segmentation visualization effects before and after deblurring provided in Example 1, wherein (a) is a visualization diagram of crack segmentation before deblurring, (b) is a visualization diagram of crack segmentation after deblurring using NAFNet, (c) is a visualization diagram of crack segmentation after deblurring using DeblurGANv2, and (d) is a visualization diagram of crack segmentation after deblurring using CDVD-TSP; Fig.11 The error analysis diagram of the linear crack calculation results provided in Example 1, wherein (a) is the error analysis diagram of the linear crack length calculation results, and (b) is the error analysis diagram of the linear crack maximum width calculation results; Fig.12 This is an experimental flow chart of the pavement crack detection method according to Example 1. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings, and the described content is intended to explain the present invention rather than to limit it.

[0020] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, systems, products or apparatus.

[0021] Embodiment 1 This embodiment provides a method for detecting cracks in an asphalt pavement based on driving videos recorded by a smartphone, comprising the following steps: S1, such as Figure 1 As shown, the video recording device modified according to the example vehicle model is assembled into a diagram, and the device can fix the smart phone, connect firmly, prevent shaking, and obtain clearer driving video; The equipment uses industrial material shelves with a square cross-section and grooves on all four sides, which is convenient for installation and fixation using nuts and screws. The mobile phone holder in the equipment also chooses an all-round restraint style to ensure that the mobile phone does not slip. The eight surrounding fixed claws are protected by soft pads, which can further reduce shock and prevent slipping. The connection part adopts a universal rotating ball structure design, which is convenient for adjusting the angle to obtain complete road information. The mobile phone holder is at an angle of about 30 degrees with the horizontal plane, 1.75 meters above the ground and 0.3 meters away from the rear of the vehicle.

[0022] The video recording equipment was used to record the test sections. One section was recorded on the school test road, with the vehicle speeds of 30 km / h, 40 km / h, and 50 km / h. The other section was recorded on the test road from Pengzhen to Huangling on the Yanxi Expressway, with the vehicle speeds of 60 km / h, 80 km / h, and 100 km / h.

[0023] S2, when processing the driving video of the road obtained by the smartphone, due to the limited viewing angle of the mobile phone camera and the narrow and long linear crack morphology, it is often impossible to collect all the morphological features of the cracks. Therefore, this embodiment splices the crack video frames obtained by the smartphone, removes repeated cracks and synthesizes them into an image containing complete crack features; Image stitching includes two key steps: image registration and image fusion. In image registration, the feature extraction of crack video images is first performed by using the Speed-up robust features (SURF) algorithm. After constructing descriptors for the extracted feature points, the K nearest neighbor algorithm is used to match the feature points. For mismatches in the matching results, the Random Sample Consensus (RANSAC) algorithm is used to purify the matching pairs. For the fusion of overlapping areas between images, the dynamic programming method is first used to determine the best stitching line in the two overlapping images, and then the multi-resolution fusion method is used to fuse the images, and finally an image containing complete crack features is obtained.

[0024] S3, uses the NAFNet network to deblur the image containing complete crack features. The NAFNet network mainly includes a simple gating module and a simplified channel attention module; The simple gating module is a simplification of the gated linear unit (GLU), which directly divides the input feature map into two parts in the channel dimension and multiplies them (such as Figure 2 As shown in (1), H, W, and C are the height, width, and number of channels of the image, respectively, where X and Y are feature maps of the same size. Represents element-wise multiplication.

[0025] (1) Channel attention can improve the network model's attention to key information, deblurring effect and efficiency by optimizing the weight distribution of feature channels. Figure 3 As shown, the expression is shown in formula (2), and W is a 1×1 convolutional layer.

[0026] (2) This embodiment obtains the feature extraction module of the NAFNet network based on the simple gating module and the simplified channel attention module (such as Figure 4 As shown in the figure, LN represents layer normalization, and Dconv represents a hole convolution layer), and then combined with the U-Net architecture to obtain the structure of the entire NAFNet network model (as shown in the figure, LN represents layer normalization, and Dconv represents a hole convolution layer). Figure 5 as shown).

[0027] First, the blurred image with a resolution of H×W and containing complete crack features is input into the NAFNet network, and then it is expanded through a 3×3 convolution block to expand the number of channels. Then it enters the four encoder blocks for feature extraction, and then after the transition of the middle block, it enters the four decoder blocks for image restoration. Figure 5 The blue arrow in the middle is a downsampling operation, which uses a 2×2 convolution kernel and a convolution with a stride of 2 to reduce the size of the feature map and increase the number of channels; the brown arrow is an upsampling operation, which uses the PixelShuffle operation to rearrange and increase the channels of the feature map in 2×2 blocks, thereby doubling the resolution of the feature map and reducing the number of channels to 1 / 4 of the original. Finally, a 3×3 convolution block is used to reduce the number of channels to obtain a deblurred RGB image, that is, a clear RGB image.

[0028] S4, using the DDNet network to segment the cracks in the clear RGB image, the DDNet network includes a double-scale cross attention module (Double-scale Cross Attention Module, DCA) and a supervised decoding module (Downsampling Guidance Decoding Module, DGD). Compared with the background, cracks often occupy a very small part in the road video image. This large size difference often causes the semantic segmentation model to be unable to take into account the segmentation accuracy of both at the same time. The double-scale cross attention module can process two images of different resolutions to achieve the purpose of balancing weights at the pixel level. The supervised decoding module can improve the learnability of the decoding process while introducing precise spatial detail information to improve the decoding accuracy.

[0029] like Figure 6 As shown in the figure, the H×W crack video images are first upsampled to 2H×2W images by the neighboring interpolation method, and then sent to the feature extraction layer (Layer1-Layer5) of the main network based on the VGG16 architecture. The results are further entered into the dual-scale cross-attention module to achieve a balanced distribution of the weight of the target object at the pixel level. Then, the two different sizes of feature maps are merged with the output results of the dual-scale cross-attention module, and 1×1 convolution is used to reduce the number of channels to achieve efficient integration of features. In the decoding stage of the DDNet network, the feature maps generated by each layer in the encoding process and the feature maps to be upsampled are input into the supervised decoding module for processing. The feature maps obtained by downsampling during the encoding process are used to guide the decoding process, and the lost spatial detail information is reintegrated to obtain the crack segmentation processed image, thereby further improving the accuracy of semantic segmentation.

[0030] Specifically, first, the clear RGB image with a resolution of H×W is upsampled by the neighboring interpolation method to obtain a clear RGB image with a resolution of 2H×2W, and then the clear RGB image with a resolution of H×W is input into the feature extraction layer to obtain a first feature map, and the clear RGB image with a resolution of 2H×2W is input into the feature extraction layer to obtain a second feature map; the first feature map and the second feature map are input into the dual-scale cross attention module to balance the weight distribution of the target object to obtain an integrated feature map; the second feature map is downsampled and then fused with the integrated feature map and the first feature map, and then processed by a 1×1 convolution block and the feature map generated by Layer4 is jointly input into the supervised decoding module to generate a result, and then processed by a 1×1 convolution block and the feature map generated by Layer3 is jointly input into the supervised decoding module to generate a result, and then processed by a 1×1 convolution block and the feature map generated by Layer2 is jointly input into the supervised decoding module to generate a result, and then processed by a 1×1 convolution block and the feature map generated by Layer1 is jointly input into the supervised decoding module for processing, and pixel-by-pixel addition is performed to output the crack segmentation processed image.

[0031] During the whole process, the feature map output by the encoding part guides the decoding, ensuring that the spatial detail information lost due to downsampling can be effectively restored while improving the segmentation accuracy.

[0032] S5, firstly, based on the closing operation in mathematical morphology, fill the empty area in the center of the crack after crack segmentation, and connect the breaks between two cracks; then calculate the number of background connected areas to roughly divide the cracks into linear cracks and network cracks, and use the projection method to subdivide the linear cracks into horizontal and vertical cracks; classify and calculate different crack geometric parameters, that is, for different types of cracks, the geometric characteristic parameters of interest are not exactly the same. For linear cracks, their length and width are the main considerations; for network cracks, more attention is paid to the size of their distribution range, which is quantified by the area and density of the circumscribed rectangle.

[0033] For the calculation of the length of linear cracks, the skeleton image of the crack is first obtained by the skeleton extraction method, and then the length between every two adjacent pixels in the skeleton image is calculated and summed up to obtain the length D of the linear crack, as shown in Figure 7 (a) and (4), where μ is the ratio between the actual size and the pixel size.

[0034] (4) For the calculation of the width of linear cracks, the maximum crack width of linear cracks is calculated based on the contour maximum inscribed circle algorithm of OpenCV, and the shortest distance between the point on the skeleton line and the contour is found using the cv2.pointPolygonTest function. , that is, the length in the normal direction, which is the radius of the maximum inscribed circle of the contour, such as Figure 7 (b) shows the maximum width of the crack. See formula (5). The average width of the crack The total number of pixels occupied by the cracks The crack length To calculate, as shown in formula (6). It is obtained by traversing the points with pixel value of 255 in the crack image.

[0035] (5) (6) The area of ​​the network cracks is described by the area of ​​its smallest circumscribed rectangle, that is, the area of ​​the smallest rectangle that can contain all the cracks. The width and height of the rectangle are recorded as and , the calculation formula is shown in formula (7), where: Indicates the area of ​​the smallest circumscribed rectangle. Crack density can provide an important reference for inspectors. When the area is similar, the greater the crack density, the greater the degree of damage in the area. Density of reticular cracks The total number of crack pixels The area of ​​the minimum circumscribed rectangle of the crack area It can be expressed as the ratio of , as shown in formula (8).

[0036] The following is a detailed description of the experimental process of the asphalt pavement crack detection method provided by this embodiment. Fig.12 shown.

[0037] 1. Experimental details This embodiment builds a network model based on Python 3.9, PaddlePaddle and Pytorch, and uses packages such as OpenCV, Numpy and Matplotlib in Python for digital image processing.

[0038] For the NAFNet crack deblurring model, the batch size (Batch size) of the training is selected as 16, the number of training rounds (epoch) is 550, the initial learning rate (Lr) is set to 1e-4, the cos decay method is used to adjust the learning rate during training, and the adam optimization algorithm is used to update the parameters. The loss function uses PSNRLoss, and the number of feature extraction blocks is 36.

[0039] For the DDNet crack segmentation model, VGG16 is used as the skeleton network for feature extraction during training, and the pre-trained weights of the network on ImageNet are loaded. Dice Loss plus Focal Loss are used as the loss function, and other hyperparameters such as Batch size are set to 8, the number of training rounds is set to 300, and Lr is set to 1e-4. The cosine decay method is also used with the adam optimization algorithm to update the learning rate and other parameters.

[0040] 2. Dataset In addition to training on an open source dataset, this embodiment also trains on a self-made dataset, which is derived from driving videos collected by video recording equipment on a test road. For the image deblurring task, this embodiment extracts clear frames in the video as clear images, and then synthesizes blurred images through data processing. The final self-made crack deblurring dataset has a total of 1209 pairs of images for training and a total of 134 pairs of images for testing. For the image segmentation task, the present invention uses the mainstream annotation tool Labelme to annotate the recorded video frames. The final dataset contains 2628 pictures, and 2082 and 270 images are randomly selected for training and verification, respectively, and the remaining 276 images are used in the testing phase.

[0041] 3. Evaluation indicators The evaluation indicators include precision, recall, F1-score and mean intersection over union (MIoU).

[0042] 4. Experimental results 4.1 Crack splicing effect In order to verify the effectiveness of the crack video image stitching method, this embodiment stitches the driving video collected on the test road. Some stitching schematic diagrams are shown in Figure 2. Figure 8 As shown. It can be found that the image stitching effect of repeated cracks is better. The longitudinal cracks that span multiple video frames can be completely spliced, and the detailed information of the cracks can be well preserved. For horizontal cracks that appear repeatedly in multiple videos, the redundant parts can be effectively removed after splicing, and finally only one is retained for calculation, and there is no blurring in the overlapping area, which helps to reduce the error of subsequent calculation of geometric parameter information of the cracks. 9 long cracks after splicing are randomly selected for calculation of geometric parameters, and the results are shown in Table 1. It can be found that the average error of the crack length after splicing does not exceed 8%, indicating that the crack splicing algorithm performed in this embodiment is effective.

[0043] Table 1 Calculation results of crack splicing

[0044] 4.2 Crack segmentation performance The segmentation visualization results after processing by DDNet, UNet, SegNet and DeepLabV3+ proposed in this invention are shown in the figure below: Fig. 9 As shown in the figure, DDNet can identify cracks well and is minimally affected by road markings and pedestrians, with only very slight false detections. In addition to comparing the visualization results subjectively, this paper also compares the model performance through objective evaluation indicators, and the results are shown in Table 2.

[0045] As shown in the table, DDNet of the present invention performs better in both F1-Score and MIoU. Compared with UNet, F1-Score is improved by 10.14% and MIoU is improved by 8.33%; compared with SegNet, F1-Score is improved by 9.14% and MIoU is improved by 7.07%; compared with DeepLabV3+, F1-Score is improved by 6.10% and MIoU is improved by 3.96%, which shows that the method proposed in this embodiment shows better performance in crack detection of CFD datasets.

[0046] Table 2 Comparison of indicators on different datasets

[0047] 4.3 Crack Deblurring Effect The deblurring effects of the NAFNet network model, DeblurGANv2, and CDVD-TSP are compared and analyzed on a self-made deblurring dataset. After deblurring different methods, semantic segmentation is performed. The index comparison results are shown in Table 3, and the visualization results are shown in Fig.10 shown.

[0048] It can be clearly seen from the figure that the crack image processed by NAFNet not only visually deepens the crack and forms a large difference with the surrounding area, but also has better semantic segmentation accuracy than the blurred image, with F1-score increased from 0.8172 to 0.8325, and MIoU value increased from 0.7295 to 0.7364. Compared with the image processed by DeblurGANv2, it has increased by 1.35% and 0.49%, respectively, and compared with the image processed by CDVD-TSP, it has increased by 1.79% and 1.10%, respectively. When the blurred image is directly segmented, a large part of the cracks will be missed, but after NAFNet processing, it can be clearly seen that the missed parts are detected. In addition, the images processed by the DeblurGANv2 method and the CDVD-TSP method can also identify the missed parts, but there are still some cracks that are incoherent. In comparison, the crack segmentation accuracy after NAFNet processing is the most improved, and the crack continuity is also the best. Combined with the visual effect, the effectiveness of the algorithm of this embodiment in the pavement crack scene is proved.

[0049] Table 3 Comparison of indicators after processing by different methods

[0050] 4.4 Geometric parameter calculation error In this embodiment, the calculated values ​​of the geometric parameters of some cracks at different vehicle speeds are compared with the manually measured values ​​on a test road on campus. The error analysis at different speeds is as follows: Fig.11 As shown. When the speed is 30 km / h, the error of the linear crack length is basically within 10%, with only a few errors exceeding 10%, but all are controlled within 12%. The average error is 5.33%, with a high accuracy rate. When the speed rises to 40 km / h and 50 km / h, the maximum error of the linear crack length increases, reaching 12.23% and 13.35% respectively; the average errors are 5.98% and 6.89% respectively, which are 0.65% and 1.56% higher than 30 km / h respectively. Therefore, this embodiment controls the maximum error and average error of the linear crack length calculation results within 14% and 7% respectively, proving that the algorithm for crack length calculation in this embodiment is effective. Similarly, by Fig.11(b) It can be seen that when the speed is 30 km / h, the error of the maximum width of the linear crack is within 12%, and the average error is 5.26%, which is a high accuracy. When the speed increases to 40 km / h and 50 km / h, the maximum error of the maximum width of the linear crack increases, reaching 12.51% and 13.15% respectively; the average error becomes 6.84% and 7.89% respectively, which is 1.58% and 2.63% higher than 30 km / h, respectively, proving that the algorithm for calculating the crack width in this embodiment is effective.

[0051] It can also be seen from Table 4 that the test error of the network crack area is very small, within 4%, which proves that the algorithm for calculating the network cracks in this embodiment is effective.

[0052] Table 4 Calculation results of network crack area and density

[0053] Then, the driving videos recorded at different speeds on the Yanxi Expressway were analyzed, and the calculated crack geometry parameter values ​​were compared with the manually measured values. The results are shown in Table 5. Results When the speed is 60 km / h, the error is still acceptable, but when the speed increases to 80 km / h and 100 km / h, the error is larger.

[0054] Table 5 Statistical results of cracks on Yanxi Expressway

[0055] In summary, the error of the crack geometric parameters increases with the increase of speed, but the increase is not large. In this embodiment, within the speed range of 80 km / h and below, the average error of the crack length, width and area meets the requirements.

[0056] Embodiment 2 This embodiment provides an asphalt pavement crack detection system, including a video acquisition module, a preprocessing module, an image deblurring module, a crack segmentation module and a crack geometric parameter calculation module; The video acquisition module obtains driving video of the road.

[0057] The preprocessing module splices the crack video frames of the driving video through image registration and image fusion to obtain an image containing complete crack features; firstly, the feature points of the crack video frame are extracted by an accelerated robust feature algorithm, and the descriptors of the feature points are constructed and then the feature points are matched using a K-nearest neighbor algorithm to obtain matching results; After the matching result is purified by using a random sampling consensus algorithm, the optimal stitching line of two crack video frames is determined in combination with a dynamic programming method, and a multi-resolution fusion method is used to fuse them to obtain an image containing complete crack features.

[0058] The image deblurring module uses the NAFNet network to deblur the image containing complete crack features to obtain a clear RGB image; the NAFNet feature extraction module is obtained through a simple gating module and a simplified channel attention module, and the NAFNet feature extraction module is combined with the U-Net architecture to obtain the NAFNet network; the image containing complete crack features with a resolution of H×W is input into the NAFNet network, and after the number of channels is expanded by a 3×3 convolution block, it enters the encoding block for feature extraction to obtain a feature map, the feature map is downsampled, and the size is reduced and the channels are increased by a 2×2 convolution kernel and a convolution with a step size of 2; then upsampling is performed through an intermediate block transition, and the channels of the feature map are rearranged and increased according to 2×2 blocks through a PixelShuffle operation, and finally a clear RGB image is obtained through a 3×3 convolution block; the encoding block is the NAFNet feature extraction module.

[0059] The crack segmentation module uses the DDNet network to perform crack segmentation on the clear RGB image to obtain an image after crack segmentation. First, the clear RGB image with a resolution of H×W is upsampled by the neighboring interpolation method to obtain a clear RGB image with a resolution of 2H×2W. Then, the clear RGB image with a resolution of H×W is input to the feature extraction layer to obtain a first feature map, and the clear RGB image with a resolution of 2H×2W is input to the feature extraction layer to obtain a second feature map. Inputting the first feature map and the second feature map into a dual-scale cross attention module to perform weight balance distribution on the target object to obtain an integrated feature map; After downsampling the second feature map, the integrated feature map and the first feature map are fused, processed by a 1×1 convolution block and input into a supervised decoding module. Meanwhile, the feature maps generated by each layer in the feature extraction layer and the feature maps to be upsampled are input into the supervised decoding module. Finally, the supervised decoding module performs pixel-by-pixel addition to output the crack segmentation processed image.

[0060] The crack geometry parameter calculation module calculates the crack geometry parameters of the image after crack segmentation processing, and uses a closing operation based on mathematical morphology to connect the breaks between two cracks in the image after crack segmentation processing; calculates the number of background connected areas of the image after crack segmentation processing, and classifies the cracks according to the number of background connected areas to obtain crack types, thereby completing asphalt pavement crack detection.

[0061] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0062] In this embodiment, a computer device is also provided, which includes a processor and a memory, wherein the memory is used to store a computer program (in this embodiment, the computer program includes a computing component and an iterative component, and can perform model calculation and model update), the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in a computer storage medium to implement a corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of an asphalt pavement crack detection method.

[0063] This embodiment also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of an asphalt pavement crack detection method in the above embodiment.

[0064] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0066] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting cracks in an asphalt pavement, characterized in that: The following steps are involved: Get driving video of the road; splicing the crack video frames of the driving video to obtain an image containing complete crack features; Using the NAFNet network to deblur the image containing complete crack features to obtain a clear RGB image; Using the DDNet network to perform crack segmentation on the clear RGB image to obtain an image after crack segmentation processing; The crack geometric parameters of the image after the crack segmentation processing are calculated to complete the asphalt pavement crack detection.

2. The asphalt pavement crack detection method according to claim 1, characterized in that: The crack video frames of the driving video are spliced ​​through image registration and image fusion to obtain an image containing complete crack features.

3. The asphalt pavement crack detection method according to claim 2, characterized in that: The step of splicing the crack video frames of the driving video by image registration and image fusion to obtain an image containing complete crack features specifically includes: Firstly, the feature points of the crack video frame are extracted by an accelerated robust feature algorithm, and the feature points are matched by using a K-nearest neighbor algorithm after constructing the descriptors of the feature points to obtain matching results; After the matching result is purified by using a random sampling consensus algorithm, the optimal stitching line of two crack video frames is determined in combination with a dynamic programming method, and a multi-resolution fusion method is used to fuse them to obtain an image containing complete crack features.

4. The asphalt pavement crack detection method according to claim 1, characterized in that: The NAFNet feature extraction module is obtained through a simple gating module and a simplified channel attention module, and the NAFNet feature extraction module is combined with the U-Net architecture to obtain the NAFNet network.

5. The asphalt pavement crack detection method according to claim 4, characterized in that: The step of using the NAFNet network to deblur the image containing complete crack features to obtain a clear RGB image specifically includes: The image containing complete crack features with a resolution of H×W is input into the NAFNet network, and after the number of channels is expanded by a 3×3 convolution block, it enters the encoding block for feature extraction to obtain a feature map, and the feature map is downsampled and reduced in size and increased in channels by a 2×2 convolution kernel and a convolution with a step size of 2; Then, upsampling is performed through the intermediate block transition, and the channels of the feature map are rearranged and increased in 2×2 blocks through the PixelShuffle operation, and finally a clear RGB image is obtained through a 3×3 convolution block; The encoding block is the NAFNet feature extraction module.

6. The asphalt pavement crack detection method according to claim 1, characterized in that: The step of using the DDNet network to perform crack segmentation on the clear RGB image to obtain an image after crack segmentation processing specifically includes: First, the clear RGB image with a resolution of H×W is upsampled by a neighboring interpolation method to obtain a clear RGB image with a resolution of 2H×2W, and then the clear RGB image with a resolution of H×W is input to a feature extraction layer to obtain a first feature map, and the clear RGB image with a resolution of 2H×2W is input to a feature extraction layer to obtain a second feature map, wherein the feature extraction layer includes Layer1, Layer2, Layer3, Layer4 and Layer5; Inputting the first feature map and the second feature map into a dual-scale cross attention module to perform weight balance distribution on the target object to obtain an integrated feature map; After downsampling the second feature map, the integrated feature map and the first feature map are fused, processed by a 1×1 convolution block and the feature map generated by Layer4 is input into the supervised decoding module to generate a result, then processed by a 1×1 convolution block and the feature map generated by Layer3 is input into the supervised decoding module to generate a result, then processed by a 1×1 convolution block and the feature map generated by Layer2 is input into the supervised decoding module to generate a result, then processed by a 1×1 convolution block and the feature map generated by Layer1 is input into the supervised decoding module for processing, and pixel by pixel addition is performed to output the crack segmentation processed image.

7. The asphalt pavement crack detection method according to claim 1, characterized in that: The step of calculating the crack geometric parameters of the crack segmentation processed image specifically includes: Using a closing operation based on mathematical morphology to connect the break between two cracks in the image after the crack segmentation process; Calculating the number of background connected regions of the image after crack segmentation processing, and classifying the cracks to obtain crack types according to the number of background connected regions; The fracture geometric parameters of different fractures corresponding to the fracture types are calculated.

8. An asphalt pavement crack detection system, characterized in that: include: Video acquisition module, used to obtain driving video of the road; A preprocessing module, used for splicing the crack video frames of the driving video to obtain an image containing complete crack features; An image deblurring module is used to deblur the image containing complete crack features by using a NAFNet network to obtain a clear RGB image; A crack segmentation module is used to perform crack segmentation on the clear RGB image using a DDNet network to obtain an image after crack segmentation processing; The crack geometric parameter calculation module is used to calculate the crack geometric parameters of the image after the crack segmentation processing to complete the asphalt pavement crack detection.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable in the processor, wherein the processor implements the steps of the asphalt pavement crack detection method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the asphalt pavement crack detection method according to any one of claims 1 to 7 are implemented.

Images