An interactive intelligent annotation method for pavement crack images

Through the VGG19-Unet interactive semantic segmentation model of pavement cracks, interactive semantic segmentation and error backpropagation training are used to solve the problem of low manual labeling efficiency in pavement crack detection, and efficient intelligent pavement crack annotation and model training are achieved, improving the Detection accuracy.

CN114022675BActive Publication Date: 2024-11-29SOUTHEAST UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111227838.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-21
Publication Date
2024-11-29
Estimated Expiration
2041-10-21

AI Technical Summary

Technical Problem

In the prior art, the pixel-level semantic segmentation model of pavement cracks requires a large amount of manual labeling data, and the labeling process lacks intelligence, resulting in inefficient efficiency and unable to effectively improve the recognition accuracy of pavement crack detection.

Method used

VGG19-Unet pavement crack interactive semantic segmentation model is adopted to perform interactive semantic segmentation through marked seed points and crack original images to generate seed candidate points in the target fracture area, and use error backpropagation training and similarity principles to perform the region. Growth, realize intelligent labeling of road cracks.

Benefits of technology

It greatly reduces the workload of manual labeling, improves the quality of image labeling, provides a large amount of labeling data for the pavement crack segmentation model, improves model performance, and promotes the intelligent development of pavement crack detection technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114022675B_ABST
    Figure CN114022675B_ABST
Patent Text Reader

Abstract

The present invention discloses an intelligent pavement crack labeling method based on interactive segmentation. First, a VGG19-Unet pavement crack interactive semantic segmentation model is designed to perform crack pixel-level segmentation. The model input end inputs the original pavement crack image, the crack labeling binary map, and the labeled crack area seed point map. During the training process, seed candidate points in the target crack area are generated, and the loss is calculated with the labeled crack seed points, and the positions of the seed candidate points in the target crack area are corrected by regression. Then, according to the four-neighborhood similarity criterion, the crack area is grown for the seed candidate points in the target crack area, and an interactive crack segmentation binary map is generated to complete the VGG19-Unet pavement crack interactive semantic segmentation model. The present invention makes up for the defect of insufficient sample size of existing deep learning crack pixel-level labeling images, is conducive to the training of the pavement crack semantic segmentation model, and provides a basis for rapid detection of pavement cracks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of road engineering detection and maintenance, and in particular to an interactive intelligent marking method for road surface crack images. Background Art

[0002] With economic development, my country's road mileage is increasing. By the end of 2020, the total mileage of national highways was 5.1981 million kilometers, and the road density was 54.15 kilometers per 100 square kilometers. However, with the increase in road mileage, there is an increasing number of road diseases. Under the influence of heavy vehicle loads and changing road environments, various road cracks, potholes and other diseases continue to appear on the road surface, greatly reducing driving comfort and safety. Since road cracks are common road diseases, timely road crack detection and maintenance are very important.

[0003] In recent years, computer vision methods based on deep learning have been continuously used in the automated detection of pavement cracks. In order to improve the recognition accuracy of pavement cracks, supervised learning is usually used to complete the recognition and positioning of pavement cracks. However, due to the large number of parameters in the deep learning model, a large amount of data is required for the training of the pavement crack segmentation model. The pavement crack pixel-level semantic segmentation model requires a large amount of manually annotated data sets. The current pixel-level crack segmentation task has very high requirements for the quality of the input annotated image, and requires a lot of manual work to perform pixel-level annotation of pavement cracks. However, manual annotation lacks intelligent and automated processes, and the annotation process is time-consuming and labor-intensive. Therefore, an intelligent pavement disease annotation method is urgently needed to improve the annotation efficiency of pavement crack images. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an interactive intelligent labeling method for pavement crack images, thereby realizing interactive segmentation of pavement cracks, which can not only effectively reduce the workload of manual labeling, but also improve the recognition of pavement crack segmentation models, thereby enabling pavement crack detection technology to continuously develop in the direction of intelligence.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] An interactive intelligent annotation method for pavement crack images proposed by the present invention comprises the following steps:

[0007] Step 1: Establish a VGG19-Unet pavement crack interactive semantic segmentation model;

[0008] Step 2: The seed points of the annotated cracks, the original crack images, and the crack segmentation binary images are used as inputs of the VGG19-Unet pavement crack interactive semantic segmentation model;

[0009] Step 3: Train the VGG19-Unet pavement crack interactive semantic segmentation model to generate seed candidate points in the target crack area;

[0010] Step 4: Calculate the coordinate errors between the seed points of the marked cracks and the seed candidate points in the target crack area, perform error back propagation training on the VGG19-Unet pavement crack interactive semantic segmentation model, and continuously reduce the errors between the seed points of the marked cracks and the seed candidate points in the target crack area, thereby completing the training of the seed candidate points in the target crack area;

[0011] Step 5: The seed candidate points in the target crack area obtained in step 4 are grown according to the similarity principle, and the target crack area is segmented, thereby generating a VGG19-Unet pavement crack interactive semantic segmentation model;

[0012] Step 6: Use the VGG19-Unet pavement crack interactive semantic segmentation model generated in step 5 to complete the pavement crack interactive intelligent labeling task.

[0013] As a further optimization scheme of the interactive intelligent annotation method for pavement crack images described in the present invention, the process of growing the target crack region in step 5 is as follows:

[0014] If the grayscale difference of the four neighborhoods of the seed candidate point in the target crack area is not greater than the preset grayscale difference threshold, the target crack area is grown, and the new target crack area and the seed candidate points in the new target crack area are expanded; then the grayscale difference of all the crack seed candidate points in the expanded new target crack area is calculated according to the four-neighborhood principle, and all points in the target crack area are traversed to end the growth of the target crack area.

[0015] As a further optimization scheme of the interactive intelligent labeling method for pavement crack images described in the present invention, in step 6, the VGG19-Unet pavement crack interactive semantic segmentation model generated in step 5 is used to form interactive seed points to complete the interactive intelligent labeling task of pavement cracks; the interactive seed points refer to the seed points of the VGG19-Unet pavement crack interactive semantic segmentation model.

[0016] As a further optimization scheme of the interactive intelligent labeling method for pavement crack images described in the present invention, the VGG19-Unet pavement crack interactive semantic segmentation model in step 1 extracts the features of pavement cracks by using 16 convolutional layers, and then restores the features through the upsampling layer of the pavement crack interactive semantic segmentation model, builds an encoding-decoding interactive semantic segmentation model, and uses shallow-deep feature fusion in the encoding-decoding interactive semantic segmentation model to complete the semantic segmentation task of pavement cracks.

[0017] As a further optimization scheme of the interactive intelligent annotation method for pavement crack images described in the present invention, the VGG19-Unet pavement crack interactive semantic segmentation model includes an encoding structure and a decoding structure, forming a symmetrical U-shaped structure;

[0018] The first 16 convolutional layers of the VGG19-Unet pavement crack interactive semantic segmentation model are selected as the encoding structure of the VGG19-Unet pavement crack interactive semantic segmentation model to complete the extraction of pavement crack texture characteristics and generate a concentrated feature map. The specific process is as follows: the image size of the input encoding structure is 512×512×3, and after 4 downsampling compressions of the VGG19-Unet pavement crack interactive semantic segmentation model, a 32×32×512 concentrated feature map is obtained; then the concentrated feature map passes through the decoding structure to restore the 32×32 feature map to a 512×512 crack output map to obtain the crack semantic features; bilinear interpolation is used in the decoding structure for upsampling; at the same time, there are feature fusion layers at the corresponding encoding and decoding structures to fuse the crack texture characteristics of the encoding structure with the crack semantic features of the decoding structure to achieve interactive semantic segmentation of pavement cracks.

[0019] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0020] The present invention can complete interactive intelligent labeling through the interaction of VGG19-Unet pavement crack interactive semantic segmentation model information, greatly reduce the current workload of labeling target cracks in supervised learning, effectively improve the image labeling quality, provide a large amount of labeled data for model training, and provide data support for improving the performance of the crack segmentation model. It can be promoted for automated detection of pavement cracks, meet actual engineering needs, and provide a basis for precise road maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the present invention.

[0022] Figure 2 This is the VGG19-Unet pavement crack interactive semantic segmentation model diagram provided by the present invention.

[0023] Figure 3 It is an input data diagram of the crack interactive segmentation model provided by the present invention; wherein, (a) is the original image of pavement cracks, (b) is the crack segmentation binary image, and (c) is the marked pavement crack seed points.

[0024] Figure 4It is a growth map of seed candidate points in the target crack area provided by the present invention; wherein, (a) is the seed candidate points in the initial target crack area, (b) is the first growth of seed candidate points in the target crack area, (c) is the second growth of seed candidate points in the target crack area, (d) is the third growth of seed candidate points in the target crack area, (e) the end of growth of seed candidate points in the target crack area, and (f) is a binary map of pavement crack segmentation.

[0025] Figure 5 It is an interactive pavement crack segmentation map provided by the present invention; wherein, (a) is an interactive seed point map of transverse cracks, (b) is an interactive seed point map of longitudinal cracks, (c) is an interactive segmentation binary map of transverse cracks, and (d) is an interactive segmentation binary map of longitudinal cracks. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 As shown, this embodiment provides an interactive intelligent marking method for pavement cracks, and the specific steps are:

[0028] Step 1: Build a VGG19-Unet road crack interactive semantic segmentation model based on a symmetric encoding-decoding structure;

[0029] Step 2: The seed points of the annotated cracks, the original crack image, and the crack segmentation binary map are used as inputs of the VGG19-Unet pavement crack interactive semantic segmentation model;

[0030] Step 3: Train the VGG19-Unet pavement crack interactive semantic segmentation model to generate seed candidate points in the target crack area;

[0031] Step 4: Calculate the coordinate errors between the seed points of the marked cracks and the candidate seed points in the target crack area, perform error back propagation training on the VGG19-Unet pavement crack interactive semantic segmentation model, and continuously reduce the errors between the seed points of the marked cracks and the candidate seed points in the target crack area, thereby completing the training of the candidate seed points in the target crack area;

[0032] Step 5: The seed candidate points in the target crack area obtained in step 4 are grown according to the similarity principle to segment the target crack area, thereby generating a VGG19-Unet pavement crack interactive semantic segmentation model;

[0033] Step 6: Use the VGG19-Unet pavement crack interactive semantic segmentation model generated in step 5 to form interactive seed points and complete the pavement crack interactive intelligent labeling task.

[0034] First, the VGG19-Unet pavement crack interactive semantic segmentation model is established. The specific structure is as follows: Figure 2 As shown in the figure. The interactive segmentation of pavement linear cracks (transverse cracks and longitudinal cracks) is studied. The VGG19-Unet pavement crack interactive semantic model is divided into an encoding structure and a decoding structure, forming a symmetrical U-shaped structure. The first 16 convolutional layers of the VGG19 pavement crack interactive semantic segmentation model are selected as the encoding structure to complete the extraction of pavement crack features. The size of the input image is 512×512×3. After 4 downsampling compressions of the VGG19-Unet pavement crack interactive semantic segmentation model, a 32×32×512 condensed feature map is obtained to obtain the crack texture characteristics. Then, after the decoding structure, the 32×32 condensed feature map is restored to a 512×512 crack segmentation binary map to obtain the crack semantic features. In the decoding part, bilinear interpolation is used for upsampling, which is simple to calculate and fast. At the same time, there are feature fusion layers at the corresponding encoding and decoding structures, which fuse the crack texture characteristics of the encoding structure and the crack semantic features of the decoding structure to achieve interactive semantic segmentation of pavement cracks. The encoding structure uses a stacking operation of convolution and pooling to extract road cracks. Maximum pooling can effectively reduce the size of the feature map and the amount of model calculation. The decoding structure uses a stacking operation of upsampling and convolution to restore the features. The convolution kernel size of 3×3 is used throughout the process. The use of this size of convolution kernel can increase the number of layers of the network structure, which is conducive to the segmentation network's learning of crack features.

[0035] Then, you need to prepare the input data of the VGG19-Unet pavement crack interactive segmentation model, that is, prepare the training image. In addition to the original crack image, you also need to prepare the annotated crack segmentation binary image and the marked crack seed point image, such as Figure 3 As shown, Figure 3 (a) is the original image of pavement cracks. Figure 3 (b) is the crack segmentation binary image. Figure 3 (c) in the figure is the annotated pavement crack seed point. The Labelme annotation tool is used to complete the pixel-level annotation of pavement cracks. The annotation time of a pavement crack binary image is about 1 minute, and the annotation time varies with the complexity of the cracks. When annotating, polygons are used to depict the boundaries of the cracks to form a closed area surrounding the target cracks, thereby achieving pixel-level segmentation of pavement diseases and obtaining a segmented binary image. On average, 120 annotation points are required for a crack binary image to form a polygon surrounding the target crack area, thereby obtaining a relatively accurate crack segmentation result.

[0036] The pavement crack interactive segmentation model is different from the general semantic segmentation model. The general semantic segmentation model only needs to input the crack original image and the annotated crack binary image to perform model training. However, the interactive model not only needs to input the pavement crack original image and the pixel-level annotated binary image, but also needs to input the annotated crack seed point map to achieve interactive segmentation. The target crack area seed candidate points generated during the training process are regressed and the annotated crack seed point (x i * ,y i * ) and the target crack region seed candidate point (x i ,y i ) is used to calculate the loss value of the VGG19-Unet road crack interactive semantic segmentation model, and then the loss value is continuously reduced by increasing the number of model training times to complete the training of the VGG19-Unet road crack interactive semantic segmentation model. During the training process, Adam is selected as the optimizer and the learning rate is adjusted to accelerate the training of the interactive model.

[0037]

[0038] After reducing the loss value through the training of the VGG19-Unet road crack interactive semantic segmentation model, the target crack region seed candidate points continue to approach the annotated crack seed points, thereby completing the road crack pixel-level segmentation task and realizing the interactive segmentation of cracks. The crack region seed candidate points include foreground crack region seed candidate points and road background seed candidate points. The two types of seed candidate points are adjusted through interactive training to achieve the ideal segmentation effect. Since the crack area is small, the points on the edge of the target crack area are selected as the foreground target road seed candidate points. Since the background accounts for a large proportion in the road crack image, it is usually only necessary to select four boundary extreme points (up, down, left, and right directions) and the midpoint of the region as the background region seed candidate points.

[0039] Through the generated crack region seed candidate points, the target segmentation region is expanded according to certain similarity criteria to complete the target region growth, such as Figure 4 As shown, Figure 4 (a) is the seed candidate point in the initial target crack area. Figure 4 (b) is the first growth of seed candidate points in the target crack area. Figure 4 (c) in the figure is the second growth of the seed candidate points in the target crack area. Figure 4 (d) in the figure is the third growth of the seed candidate points in the target crack area. Figure 4 (e) The seed candidate point in the target crack region ends the growth, Figure 4(f) Binary image of pavement crack segmentation.

[0040] The steps for growing seeds are:

[0041] 1. Take the pixel with gray value of 66 in the pavement crack image as the seed candidate point of the target crack area;

[0042] 2. Starting from the four neighborhoods of each target crack region seed candidate point, the target crack region is grown according to the similarity principle that the grayscale difference of adjacent pixels is within 3, otherwise the growth is stopped;

[0043] 3. Use the expanded area in step 2 as the starting point for the new crack seed growth. Then perform a four-neighborhood search again for all points in the new crack area. When the grayscale difference is 3, the crack area will grow, otherwise the growth will stop;

[0044] 4. Repeat the crack area growth iteration operation in step 3 until all pixels in the image are traversed, then the growth is terminated and finally pixel-level segmentation of the road surface cracks is performed.

[0045] In the interactive segmentation, the positions of the target crack region seed candidate points and the annotated crack seed points are continuously adjusted to complete the training of the pavement crack interactive segmentation model, thereby achieving pixel-level segmentation of pavement cracks. Finally, the trained model is used to complete the interactive intelligent annotation of cracks. Figure 5 As shown, Figure 5 (a) is the interactive seed point map of transverse cracks. Figure 5 (b) is the interactive seed point map of longitudinal cracks. Figure 5 (c) is the interactive segmentation binary image of the transverse cracks. Figure 5 (d) in the figure is a binary image of the interactive segmentation of longitudinal cracks. The trained interactive pavement disease segmentation network only needs to select a few crack seed points to form interactive seed points, and then the intelligent labeling of pavement cracks can be automatically completed. Compared with manual labeling, the average number of crack markers in an image can be reduced by 100, which greatly improves the efficiency of interactive labeling and can provide more semantic data for the training of crack semantic segmentation models, laying the foundation for rapid detection of pavement diseases in the future.

[0046] Finally, the code of the pavement crack interactive segmentation model can be encapsulated to produce the pavement crack interactive segmentation software, which can achieve a major breakthrough in the lack of labeled data in the pavement crack semantic segmentation model, and is conducive to the accurate segmentation of pavement cracks.

[0047] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. An interactive intelligent annotation method for pavement crack images, characterized in that: The following steps are involved: Step 1: Establish a VGG19-Unet pavement crack interactive semantic segmentation model; Step 2: The seed points of the annotated cracks, the original crack images, and the crack segmentation binary images are used as inputs of the VGG19-Unet pavement crack interactive semantic segmentation model; Step 3: Train the VGG19-Unet pavement crack interactive semantic segmentation model to generate seed candidate points in the target crack area; Step 4: Calculate the coordinate errors between the seed points of the marked cracks and the seed candidate points in the target crack area, perform error back propagation training on the VGG19-Unet pavement crack interactive semantic segmentation model, and continuously reduce the errors between the seed points of the marked cracks and the seed candidate points in the target crack area, thereby completing the training of the seed candidate points in the target crack area; Step 5: The seed candidate points in the target crack area obtained in step 4 are grown according to the similarity principle, and the target crack area is segmented, thereby generating a VGG19-Unet pavement crack interactive semantic segmentation model; Step 6: Use the VGG19-Unet pavement crack interactive semantic segmentation model generated in step 5 to complete the pavement crack interactive intelligent labeling task; The process of growing the target crack region in step 5 is as follows: If the grayscale difference of the four neighborhoods of the seed candidate point in the target crack area is not greater than the preset grayscale difference threshold, the target crack area is grown, and the new target crack area and the seed candidate points in the new target crack area are expanded; then the grayscale difference of all the crack seed candidate points in the expanded new target crack area is calculated according to the four-neighborhood principle, and all points in the target crack area are traversed, thereby terminating the growth of the target crack area; The VGG19-Unet pavement crack interactive semantic segmentation model includes an encoding structure and a decoding structure, forming a symmetrical U-shaped structure; The first 16 convolutional layers of the VGG19-Unet pavement crack interactive semantic segmentation model are selected as the encoding structure of the VGG19-Unet pavement crack interactive semantic segmentation model to complete the extraction of pavement crack texture characteristics and generate a concentrated feature map. The specific process is as follows: the image size of the input encoding structure is 512×512×3, and after 4 times of downsampling compression of the VGG19-Unet pavement crack interactive semantic segmentation model, a 32×32×512 concentrated feature map is obtained; Then the concentrated feature map passes through the decoding structure, and the 32×32 feature map is restored to a 512×512 crack output map to obtain the crack semantic features; Bilinear interpolation is used for upsampling in the decoding structure. At the same time, there are feature fusion layers in the corresponding encoding and decoding structures to fuse the crack texture characteristics of the encoding structure with the crack semantic characteristics of the decoding structure to achieve interactive semantic segmentation of road cracks.

2. The method for interactive intelligent labeling of pavement crack images according to claim 1, characterized in that: In step 6, the VGG19-Unet pavement crack interactive semantic segmentation model generated in step 5 is used to form interactive seed points to complete the pavement crack interactive intelligent labeling task; the interactive seed points refer to the seed points of the VGG19-Unet pavement crack interactive semantic segmentation model.

3. The method for interactive intelligent labeling of pavement crack images according to claim 1, characterized in that: In the step 1, the VGG19-Unet pavement crack interactive semantic segmentation model uses 16 convolutional layers to extract the features of pavement cracks, and then restores the features through the upsampling layer of the pavement crack interactive semantic segmentation model, builds an encoding-decoding interactive semantic segmentation model, and uses shallow-deep feature fusion in the encoding-decoding interactive semantic segmentation model to complete the semantic segmentation task of pavement cracks.

Citation Information

Patent Citations

  • Tunnel crack detection and measurement method based on dual-depth learning model

    CN112508030A

  • Asphalt pavement crack rapid classification method based on semantic segmentation

    CN112529092A