A road crack detection system based on semi-automatic annotation

Abstract

The application provides a kind of road crack detection system based on semi-automatic labeling, including information acquisition module, crack identification module, data management module and front end display module, the data management module uses distributed file management system, can dynamically store pavement information, and extract key frame and store.The application uses the key frame sample data set of data management module to roll optimization to the model of crack identification module, then uses the optimized model to realize semi-automatic labeling to video large-scale unlabeled image, and then obtains a large number of training data set and optimized crack identification model, reduces the labor cost, the degree of automation is high, and the labeling and identification detection precision is continuously improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of road quality detection, relates to intelligent detection of road cracks, and particularly relates to a road crack detection system based on semi-automatic marking. Background Art

[0002] As roads age, pavement defects frequently occur, necessitating regular maintenance and inspection to promptly identify damaged areas so that effective preventive or repair measures can be implemented to extend the service life of the roadbed and pavement structures and prevent catastrophic accidents. In the past, the detection and classification of road cracks often required visual assessment or the use of simple instruments, resulting in relatively backward detection methods. On the one hand, manual inspection methods are labor-intensive, inefficient, and costly, prone to missed and incorrect detections. On the other hand, road inspections often require road control operations, which adversely impacts normal traffic flow and is highly dangerous. Routine manual crack detection lacks initiative, and road information is unsystematic and incomplete, resulting in high costs, low accuracy, and poor timeliness.

[0003] The development of crack detection is fundamentally based on the application of data. The continuous expansion of datasets can improve the detection accuracy of models. In the field of artificial intelligence, data labeling is a crucial factor in determining data quality and parameter quality. However, manual labeling is not only subject to subjective factors and difficult to standardize, but also requires significant labor costs, is time-consuming, and expensive. This far exceeds the data volume required by deep learning networks, hindering the effective utilization of road surface video information. Summary of the Invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a road crack detection system based on semi-automatic annotation, so as to more effectively and comprehensively utilize the captured road surface video information to achieve more accurate, comprehensive and timely road crack detection.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A road crack detection system based on semi-automatic marking, comprising an information acquisition module, a crack identification module, a data management module and a front-end display module;

[0007] The information acquisition module is used to shoot the road surface to be inspected, obtain road surface video, determine the location information of the shot video through GPS, and upload the video and its location information to the cloud data management module;

[0008] The data management module extracts key frames from the video to obtain a key frame data set, and uses a distributed file management system to dynamically store the key frame data set, the video and its location information, manually marked samples, and automatically marked data sets;

[0009] The crack identification module is used to identify whether there are cracks on the road surface to be inspected and the types of cracks. The automatic labeling is performed simultaneously with the model optimization of the crack identification module.

[0010] The front-end display module is used to display the detection results in real time.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] 1. This invention uploads videos to the cloud for processing, rather than processing them directly on the terminal hardware. This solves the problems of insufficient local hardware resources and limited storage space, as well as insufficient processing speed and computing power. It enables the use of more accurate models for recognition on the server side, significantly improving detection accuracy.

[0013] 2. This invention utilizes a distributed file management system for data management, which not only ensures the efficiency of massive annotation resources but also ensures the reliability of file storage through the configuration of file replicas. In addition, HDFS and the Zstd compression algorithm are used to achieve the storage of massive large-file video data.

[0014] 3. This invention can automatically label all frames of a video and meet the labeling accuracy requirements, thereby greatly saving labor costs. During the iterative network training process of the crack recognition module, the results after customer verification and adjustment are repeatedly used for parameter optimization, thereby improving the detection accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a principle structure diagram of the present invention.

[0016] Figure 2 FIG. 4 is a flowchart of a model training method for a semi-automatic annotation method according to an exemplary embodiment.

[0017] Figure 3 This is a demonstration of the effect of the present invention on automatic annotation of video uploaded images. DETAILED DESCRIPTION

[0018] The present invention is further illustrated by the accompanying drawings. A brief introduction to the drawings required for the embodiments will be given below, but the drawings should not be used as a limitation. For ordinary technicians in this field, other drawings can be obtained based on the following drawings without paying any creative work.

[0019] See also Figure 1The present invention provides a road crack detection system based on semi-automatic labeling, which includes an information acquisition module, a crack identification module, a data management module and a front-end display module.

[0020] Among them, the information collection module is used to shoot the road surface to be inspected, obtain road surface video, determine the location information of the shot video through GPS, and upload the video and its location information to the data management module in the cloud.

[0021] The data management module extracts key frames from the videos uploaded by the information acquisition module to obtain a key frame dataset, and uses a distributed file management system to dynamically store the key frame dataset, the uploaded original video and its location information, manually marked samples, and automatically labeled datasets.

[0022] The crack identification module detects cracks on the road surface and their types by monitoring information from the data management module. By incorporating customer monitoring information, it optimizes the network and automatically labels massive data sets. The detection results are displayed in real time on the front-end display module.

[0023] In one embodiment of the present invention, the information acquisition module is installed on the patrol vehicle, and uses a high-definition camera to shoot dynamic video of the road surface and upload it to the data management module in real time. For example, it can be fixed in the center of the front module of the patrol vehicle or the inspection vehicle, and collect the current road video in real time as the vehicle moves. On the one hand, road video information can be collected by the on-board camera, and on the other hand, the geographical location information of the point can be transmitted in real time through GPS, and finally the information is transmitted to the data management module through the communication module. Overall, the front-end display module includes the functions of real-time video shooting, detection result query, detection record storage, crack location and maintenance suggestions. In terms of hardware, it includes a mobile client and a PC client. The PC client can view the road video and crack detection records in the data management module at any time, and perform related data export, data calculation, and data analysis operations.

[0024] The original video and its location information uploaded by the present invention mainly play the following roles:

[0025] 1. As a backup of original data, it can be provided to highway management agencies as file information storage.

[0026] 2. The original video is composed of all frames, and all frames can be used as data resources. By using the present invention to automatically label all frames of the video, the data set can be greatly expanded, making the model of the crack recognition module more accurate.

[0027] 3. The location information corresponds to the corresponding frame. When a crack is detected, a picture of the crack and related information such as the location will be displayed on the front end, helping highway management agencies to promptly detect road defects and take necessary measures such as repairs and maintenance based on the location, so as to achieve "targeted" results.

[0028] The front-end display module of the present invention can control the vehicle-mounted camera to collect videos, and can also control uploading of collected videos to the data management module, and view relevant crack detection results and historical information.

[0029] The data management module is a key component of the present invention, recording, cleaning, and organizing information. It divides files into four categories: original video, manually labeled samples, keyframe datasets, and automatically annotated datasets. Geographical location information is associated with each video frame using upload time tags, and each frame is saved. Preferably, each video frame is stored on the server's disk, with the data management module field storing the image's storage path on the server.

[0030] The distributed file management system can store massive video annotation resources completely and reliably, and classify and store the key frame data set, the video and its location information, the manually marked samples and the automatically marked data set. Preferably, the present invention adopts HDFS as the underlying file storage system for annotation resource management and annotation result management, adopts ElasticSearch as the statistical analysis engine for annotation results, and builds a 3-node HDFS cluster as the underlying storage support for the annotation resource management module and the annotation result management module. HDFS supports the storage of large files at the GB or even TB level to solve the problem of large video files. The Zstd algorithm is used to compress video files to realize large file and large data storage. A single-process multi-threaded background module written in Go language. The HFDS cluster is accessed using the WebHDFS interface by calling the gowfs library to realize the reading and writing of resource files.

[0031] Because uploaded videos contain a lot of redundant information, all frames can be replaced with keyframes. The crack recognition module only needs to identify the keyframe dataset to identify all cracks, significantly improving efficiency. Furthermore, the crack recognition results of the keyframe dataset uploaded in the i-th video are displayed on the front end. After manual user verification, this information can be used as the i-th manually labeled sample to further optimize the model.

[0032] In one embodiment of the present invention, the key frame is extracted using the frame difference method. Before the extraction, necessary image processing is performed, specifically:

[0033] 1. Perform grayscale processing on all frames of the captured video to obtain grayscale images. For example, the Python language can be used to call the OpenCV library to implement grayscale processing on all frame images.

[0034] 2. Perform bilateral filtering on the obtained image to reduce the impact of noise on image differentiation.

[0035] 3. Calculate the inter-frame difference between every two frames of the video, and then calculate the average inter-frame difference strength. Select the frame with the average inter-frame difference strength higher than the preset threshold as the key frame of the video.

[0036] For example, each pixel p in the obtained image with n pixels is differentiated, and the obtained results are summed. When the sum is higher than a threshold k, it is considered that the image is significantly different from the previous key frame and can be used as the next key frame.

[0037]

[0038] Where p s (i) represents the i-th pixel of the s-th image, and k is the set threshold. The s-th image that meets this formula is used as the key frame.

[0039] The large amount of automatically labeled data can be used as training data for the model of the crack recognition module, greatly improving the accuracy of crack detection. The automatically labeled data set of the present invention is obtained by the following method:

[0040] Step 1: Initial dataset construction.

[0041] Initially, m pieces of manually labeled, high-quality sample data are uploaded as the initial manually labeled samples. Here, manually labeled, high-quality sample data refers to the initial phase of system construction, where a batch of manually labeled sample data sets are introduced as samples for initial network parameter training. Later, using automatic labeling methods, a large amount of labeled data can be obtained, eliminating the need for manual labeling. This is also known as the "semi-automatic" labeling method of the present invention. This manually labeled sample data set constitutes the initial dataset.

[0042] Step 2: Initial model construction.

[0043] Build a model of the crack recognition module and feed the manually labeled sample data of the initial data set into the model for training or calibration.

[0044] Step 3: manual labeling for the i-th time (i=1, 2, 3, ...).

[0045] Take all frames of the uploaded video as the i-th sample resource and extract n iThe key frames are sent to the model of the crack recognition module for recognition, and the recognition results are transmitted to the front-end display module for customer verification and adjustment of the detection results, and the adjusted and marked n i Zhang sample data is uploaded and stored as manually labeled samples.

[0046] Manually labeled samples mainly play the following roles:

[0047] 1. Manually labeled samples can train the model of the crack recognition module and determine the relevant parameters of the model so that the model of the crack recognition module can automatically label all frames.

[0048] 2. Manually labeled samples can be used as sample data to optimize the training of the crack recognition module model and improve the recognition accuracy of the model.

[0049] Step 4: Optimize the model.

[0050] n i The manually labeled samples are fed into the model of the crack recognition module, retrained or calibrated, and the model parameters are updated.

[0051] Step 5: Update all sample annotations.

[0052] All video frames in the i-th sample resource are fed into the model to label all sample resources.

[0053] Step 6: Update the crack identification module parameters.

[0054] The model of the crack recognition module is retrained and adjusted based on a large number of automatically annotated data sets to improve recognition accuracy.

[0055] Through the above steps, the model of the present invention realizes the simultaneous execution of automatic labeling and model optimization of the crack identification module.

[0056] The crack recognition module of the present invention primarily performs image preprocessing, pavement crack detection, and classification algorithms. Image preprocessing includes grayscale correction and lane line removal. The Mask-RCNN algorithm first detects lane line areas, then removes these areas and inpaints the image using a modified Criminisi method.

[0057] The pavement crack detection and classification algorithm outputs detection and classification results based on Yolov5. It specifically includes the following steps:

[0058] Step 1: Upload the pre-collected road crack images to the data management module and perform image augmentation using Mosaic data augmentation to generate an enhanced dataset. Then, manually label the cracks using the annotation software LabelMe to generate the initial dataset.

[0059] Step two: build a convolutional neural network, and train the parameter weights using the initial data set described above. Preferably, the convolutional neural network adopts the structure of Yolov5, and the trained network is used as a crack classification and detection model. The Sobel operator is used as an edge detection algorithm and as a crack edge segmentation model.

[0060] Step three: input the key frame in the newly uploaded road video to be detected, i.e., the image to be recognized, into the crack classification and detection model to obtain the category and confidence of the cracks in the image to be recognized; then input the image to be recognized into the crack edge segmentation model to obtain a binary image of the cracks.

[0061] Step four: calculate the length and width of the cracks in the image to be recognized, and draw the category, confidence, coordinates, length, and width information of the cracks in the image to be recognized onto the image to be recognized.

[0062] Step five: according to the user's feedback, manually label the information of the detected crack image after adjustment, and add it to the training set of artificial labeling data.

[0063] Step six: update the network weights again according to the newly added image in the training set, so as to realize the rolling optimization of the model and continuously improve the accuracy of the model.

[0064] In this embodiment, the user compares the actual road surface condition obtained through field investigation with the crack recognition result in the image to be recognized. If they are consistent, it is determined to be accurate, if they are inconsistent, it is determined to be inconsistent, and appropriate adjustment and modification are made to realize the customer's verification, which serves as the artificial supervision information for automatic labeling.

[0065] The effect of the present application will be further described in combination with a simulation experiment.

[0066] Simulation conditions:

[0067] The running environment of the simulation experiment of the present application: NVIDIA GeForce RTX 3080, and the simulation experiment of the present application is completed under python3.8 and pycharm2022.1.2.

[0068] Simulation content:

[0069] The simulation experiment of the present application uses the semi-automatic labeling method of the present application to simulate the open source data set to obtain the automatic labeling result of the pavement crack image.

[0070] The model of the crack recognition module is trained using all artificial labeling data to obtain its accuracy. Then, the model of the crack recognition module is trained using the semi-automatic labeling method of the present application to obtain the automatically labeled data set and the accuracy of the model, and the effect of the current technical automatic labeling is obtained by comparison.

[0071] Experimental conditions

[0072] The data used in the experiment is an open-source dataset of measured road surfaces, consisting of 1,300 images. These images are categorized into two types: transverse cracks (D00) and longitudinal cracks (D44), with an average of two cracks per image. The dataset was augmented using methods such as rotation, folding, and color adjustment.

[0073] Before automatically labeling the road surface data set, the data is preprocessed. Image preprocessing includes grayscale correction and lane line removal. The lane line area is first detected using Mask-RCNN, and then the area is removed. The improved Criminisi method is then used to repair the image. The automatic labeling method of the present invention is then used to label the crack recognition module model, and a small amount of manual supervision information is added to finally obtain the automatic labeling effect. Figure 3 As shown, it can be seen that the automatic labeling method of the present invention can more accurately identify, classify and label the cracks in the image.

[0074] Experimental content and results:

[0075] AP represents the area enclosed by the precision-recall curve and the x-axis, and mAP is the average AP value for all categories. D00 represents the horizontal crack category, and D44 represents the vertical crack category. The table shows the AP values.

[0076] The crack recognition module's Yolov5 network was trained on a dataset containing 1,300 manually annotated images, and its effectiveness was tested using a validation set containing 130 images to obtain the accuracy of manual annotation. In addition, we used 500 manually annotated images from the same dataset as the initial dataset to train the initial parameters of the crack recognition module's Yolov5 network. The remaining 800 images were then automatically annotated in batches, with manual supervision information added and the model optimized. The same validation set was used for effectiveness testing to obtain the accuracy of automatic annotation.

[0077] Table 1

[0078] D00 D44 mAP Manual annotation 0.510 0.778 0.644 Automatic labeling 0.548 0.718 0.633

[0079] Referring to Table 1, a comparison of the AP values ​​for manual and automatic annotation shows that the accuracy of automatic annotation is similar to that of manual annotation, with the mAP for the crack identification model both achieving approximately 0.6. The semi-automatic annotation method employed in this paper reduces labor and improves efficiency while maintaining accuracy, significantly saving time and achieving excellent results.

[0080] In summary, the use of semi-automatic annotation can achieve an image dataset of all frames of massive videos, which can be used to train the network parameters of the crack recognition unit, improve detection accuracy, and greatly save costs.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention. Those skilled in the art can make corresponding modifications and variations based on the above description without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A road crack detection system based on semi-automatic annotation, characterized in that: It includes information acquisition module, crack identification module, data management module and front-end display module; The information acquisition module is used to shoot the road surface to be inspected, obtain road surface video, determine the location information of the shot video through GPS, and upload the video and its location information to the cloud data management module; The data management module extracts key frames from the video to obtain a key frame data set, and uses a distributed file management system to dynamically store the key frame data set, the video and its location information, manually marked samples, and automatically marked data sets; The crack identification module is used to identify whether there are cracks on the road surface to be inspected and the types of cracks. Automatic labeling and model optimization of the crack identification module are carried out simultaneously. The front-end display module is used to display the detection results in real time; The automatically annotated dataset is obtained by the following method: (1) Initial dataset construction: Initially, upload m manually annotated high-quality sample data as the initial manually labeled samples; (2) Initial model construction: Build a model of the crack recognition module and input manually labeled samples of the initial data set for training or calibration; (3) The i-th manual labeling: All frames of the i-th uploaded video are used as the i-th sample resources, and the The key frames are sent to the model of the crack recognition module for recognition, and the recognition results are transmitted to the front-end display module for customer verification and adjustment of the detection results, and the adjusted and marked Sample data is uploaded and stored as manually labeled samples, where i=1,2,3,...; (4) Optimization model: The manually labeled samples are fed into the model of the crack recognition module, retrained or calibrated, and the model parameters are updated; (5) Update all sample annotations: send all video frames in the i-th sample resource to the model of the crack recognition module to annotate all sample resources; (6) Update the parameters of the crack recognition module: retrain and adjust the model of the crack recognition module based on a large number of automatically annotated data sets to improve the recognition accuracy; The crack recognition module includes an image preprocessing submodule and a pavement crack detection and classification submodule. The image preprocessing submodule first detects the lane marking area using Mask-RCNN, then removes the area and then uses the improved Criminisi method to repair the image. The pavement crack detection and classification submodule outputs the detection and classification results based on Yolov5. The pavement crack detection and classification submodule implements crack detection and classification through the following process: Step 1: Upload the pre-collected road crack images to the data management module, and perform image augmentation on the images using Mosaic data enhancement to obtain an enhanced dataset. Then, use the labelimg annotation software to manually annotate the cracks to obtain the initial dataset. Step 2: Build a convolutional neural network and use the above initial data set to train parameter weights. The trained network is used as the crack classification detection model, and the Sobel operator is used as the edge detection algorithm and as the crack edge segmentation model. Step 3: Input the key frame of the newly uploaded road video to be detected, that is, the image to be identified, into the crack classification detection model to obtain the category and confidence of the crack in the image to be identified; then input the image to be identified into the crack edge segmentation model to obtain a crack binary image; Step 4: Calculate the length and width of the cracks in the image to be identified, and plot the category, confidence, coordinates, length, and width information of the cracks in the image to be identified on the image to be identified; Step 5: Based on user feedback, adjust the crack information of the detected image and manually annotate it in the image, and add it to the training set of manually annotated data; Step 6: Update the network weights again based on the newly added training set images to achieve rolling optimization of the model and continuously improve the accuracy of the model.

2. The road crack detection system based on semi-automatic marking according to claim 1, characterized in that: The information collection module is installed on the patrol vehicle, uses a high-definition camera to shoot dynamic video of the road surface, and uploads it to the data management module in real time.

3. The road crack detection system based on semi-automatic marking according to claim 1, characterized in that: The distributed file management system classifies and stores the key frame data set, the video and its location information, manually marked samples and automatically labeled data sets; the data management module matches the location information with each frame of the video through the upload time label, and saves each frame image of the video.

4. The road crack detection system based on semi-automatic marking according to claim 1 or 3, characterized in that: The method for extracting key frames is the frame difference method, and the steps are as follows: (1) grayscale processing is performed on all frames of the captured video to obtain grayscale images; (2) Perform bilateral filtering on the obtained grayscale image to reduce the impact of noise on image differentiation; (3) calculating the inter-frame difference between every two frames of the video in sequence, and then obtaining the average inter-frame difference intensity; (4) Select the frames whose average inter-frame difference intensity is higher than the preset threshold as the key frames of the video.

5. The road crack detection system based on semi-automatic marking according to claim 1, characterized in that: In step five, the user obtains the actual road conditions based on field investigations and compares them with the crack recognition results in the image to be recognized. If they are consistent, the results are determined to be accurate; if they are inconsistent, the results are determined to be inconsistent, and corresponding adjustments and modifications are made to serve as manual supervision information for automatic labeling.

6. The road crack detection system based on semi-automatic marking according to claim 1, characterized in that: The front-end display module includes functions of real-time video shooting, detection result query, detection record storage, crack location and maintenance suggestions; and includes mobile client and PC client.

7. The road crack detection system based on semi-automatic marking according to claim 6, characterized in that: The PC client can view the road video and crack detection records in the data management module at any time, and perform relevant data export, data calculation, and data analysis operations.

Images