Lightweight pavement crack detection method
By introducing the lightweight feature extraction module SandGlass and the coordinated attention mechanism into the pavement crack detection model, the problems of large model parameters and low detection accuracy are solved, and efficient and accurate pavement crack detection is achieved on mobile devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2023-03-24
- Publication Date
- 2026-06-09
AI Technical Summary
Existing road surface crack detection models have a large number of parameters and are computationally complex, making them difficult to apply on mobile devices. Furthermore, they have low detection accuracy in complex backgrounds, cannot effectively distinguish between cracks and noise, and lack detailed and semantic features.
We introduce a lightweight feature extraction module SandGlass and a coordinated attention mechanism. By combining depthwise separable convolution and small kernel convolution, we reduce the amount of computation and parameters, enhance the expression of crack features in the encoding and decoding structure, suppress noise interference, and fuse feature maps at different levels.
It enables efficient and accurate road surface crack detection on mobile devices, improves detection accuracy, reduces noise impact, and makes prediction results closer to the actual situation.
Smart Images
Figure CN116503327B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of pavement crack detection technology, and specifically to a lightweight pavement crack detection method. Background Technology
[0002] Cracks are an early manifestation of road surface damage. Timely and accurate detection of road cracks is crucial for routine road maintenance and extending the service life of the road surface. In routine road maintenance, inspecting and repairing cracked sections of the road can prevent further damage and significantly extend the road's lifespan. Early crack detection relied primarily on visual inspection by workers. This method is time-consuming, inefficient, and requires workers to be on the road during inspection, increasing the risk of accidents.
[0003] In recent years, with the improvement of computing power, deep learning-based crack detection methods have been applied. Deep learning-based road crack detection methods can also identify and detect cracks in images even in the presence of complex noise. As the number of model parameters increases, the performance of the model is also improved, achieving a high accuracy rate.
[0004] However, most current crack detection models have a large number of parameters and are computationally complex, lacking models specifically designed for mobile devices. Specifically, they have the following drawbacks:
[0005] (1) In the feature extraction stage, ordinary convolution operation itself has a large amount of computation, and the stacking of multiple convolutions will also lead to a large amount of time spent on the model. At the same time, the number of parameters required by ordinary convolution operation is large, which leads to a surge in the number of parameters of the entire model, which is not conducive to the application of mobile devices.
[0006] (2) When existing crack detection algorithms are used to detect road crack images with simple backgrounds and high crack contrast, they can achieve good crack image segmentation. However, once the background of the road crack image becomes complex, if there are noise interferences such as stains, leaves, lane lines and shadows in the road image, the current algorithm cannot distinguish cracks from noise well, resulting in a large amount of noise in the final prediction result and low detection accuracy.
[0007] (3) Low-level features are rich in detail and have high resolution, containing crack location and detailed information, but lack abstract semantic features and also contain background noise information. High-level features have strong semantic information, but have very low resolution, lack detailed information of crack pixels, and have poor detail perception. For crack detection, crack topology is complex and varied, such as transverse cracks, longitudinal cracks, block cracks, and alligator cracks, etc. Moreover, the number of crack pixels in actual road images accounts for a small proportion of the total number of pixels in the image. Using features extracted from only a single level will result in the final prediction result being far from the actual road surface condition. Summary of the Invention
[0008] The purpose of this invention is to provide a lightweight pavement crack detection method, which introduces a lightweight feature extraction module and improves the crack detection segmentation task, thereby reducing model complexity and improving crack detection accuracy.
[0009] To achieve the above objectives, the present invention provides a lightweight pavement crack detection method, comprising the following steps:
[0010] Step 1: Collect real road surface images and establish a road surface image database;
[0011] Step 2: Filter the images in the road surface image database and process them to obtain GroundTrue data, and construct training and testing sets;
[0012] Step 3: Train a lightweight cracked pavement detection network model using the training set;
[0013] Step 4: Test the trained network model using the test set, and calculate the evaluation metrics Precision, Recall, F-score, Prams, and FLOPs based on the test results;
[0014] If the evaluation indicators meet the requirements, it means that the network model meets the requirements. Save the network model for extracting pavement cracks.
[0015] Otherwise, modify the hyperparameters of the network model and retrain;
[0016] Step 5: Use the saved network model to detect road surface cracks, classify the cracks, and assess their severity.
[0017] Optionally, step 3 specifically includes the following steps:
[0018] Step 3.1: Use multiple convolution mode to extract features from the image to be detected. In the multiple convolution mode, the first layer is a normal convolution, and the second and last layers are depthwise separable convolutions.
[0019] Step 3.2: Use four lightweight feature extraction groups to extract image features in the encoder stage. Each lightweight feature extraction group contains two lightweight feature extraction modules and compresses the image size at the same time.
[0020] Step 3.3: Integrate the coordinated attention mechanism module into the first group of convolutional operations and each group of lightweight feature extraction;
[0021] Step 3.4: Recover the deep feature information of the network through upsampling and fuse it with the detailed features extracted in the encoder stage;
[0022] Step 3.5: Use the lightweight feature extraction group to further extract features from the fused feature image;
[0023] Step 3.6: Repeat steps 3.2 to 3.5 three times to form the final feature image;
[0024] Step 3.7: Use convolution operations to form the final predicted image;
[0025] Step 3.8: Train the network model.
[0026] Optionally, the lightweight feature extraction module performs two feature extraction operations and one dimensionality reduction and one dimensionality increase operation. In the feature extraction stage, two depthwise separable convolutions are used to replace the traditional convolution operation. The depthwise separable convolution reduces the increase in parameters and computation caused by using ordinary convolution operations by decomposing a single ordinary convolution into channel-wise convolution and point convolution. In the image feature dimensionality increase and reduction, two small kernel convolutions are used to perform dimensionality transformation.
[0027] Optionally, the coordinated attention mechanism module performs global pooling in both the horizontal X-axis and vertical Y-axis directions to preserve long-range dependency information in both directions. The horizontal and vertical information are then concatenated, and convolutional operations are used to interact the information in both directions. The resulting feature map is then segmented after being processed by Batch Normalization and a non-linear activation function. Convolution is performed on each segmented map, and attention is applied in both the horizontal and vertical directions. Finally, the feature map is fed into the Sigmoid function to obtain a feature map that can accurately locate the position of the target object.
[0028] Optionally, in the encoder stage, a set of convolutional operations and four sets of lightweight feature extraction module operation groups are used. A coordinating attention module is added at the end of each feature extraction group to enhance the expression of crack feature information. The first set of convolutional operations directly extracts image information using a combination of ordinary convolution and depthwise separable convolution. The latter four sets of lightweight feature extraction module operation groups each consist of two lightweight feature extraction modules. Each operation extracts image features while compressing the image size to reduce the subsequent computational load of the network model. After each set of operations, a coordinating attention mechanism is applied to the feature image to enhance the crack features.
[0029] Optionally, the execution process of steps 3.3 to 3.6 also includes a decoder stage. The detailed features retained by different stages of the encoder will be fused in the decoder stage. The final feature information of the encoder will be restored and semantically enhanced by upsampling in the decoder stage. By fusing with the features of different stages of the encoder, the semantic features of the crack image are enhanced. At the same time, the crack image details retained by the encoder stage are used to enhance the crack details of the fused image. The fused features are extracted by a lightweight feature extraction module. During the extraction process, a coordinated attention mechanism is applied to the fused feature map to strengthen the expression of crack information and filter out image noise. The crack prediction image is finally formed by the decoder extracting, restoring and fusing layer by layer.
[0030] Optionally, the process in step 3.8 includes the following steps:
[0031] Step 3.8.1: Reproduce the network model using PyTorch;
[0032] Step 3.8.2: Set the model parameters. The input image size is 256×256, the learning rate is le-4, the weight of the fusion layer is set to 1, the learning rate is reduced by 10 times every 100 epochs, the weight decay is 2e-4, the number of training epochs is 700, and the model is saved every 50 epochs.
[0033] Step 3.8.3: Conduct experiments on a server equipped with a Tesla-V100-SXM2-32GB GPU and a 4-core Intel(R)Xeno(R)Sliver4214 CPU;
[0034] Step 3.8.4: Adjust the parameters of the network model based on the experimental results and evaluate the crack detection effect;
[0035] Step 3.8.5: Repeat steps 3.8.2 to 3.8.4 to adjust the network model parameters to achieve the best crack detection effect.
[0036] This invention provides a lightweight pavement crack detection method. By integrating a lightweight feature extraction module (SandGlass) and a coordinated attention mechanism into the encoding-decoding structure, a lightweight pixel-level crack detection network is constructed. After training the network model using a pavement image database, the network model can achieve the same detection performance as a larger network model with less computational power. Furthermore, the introduction of a coordinated attention mechanism module suitable for mobile devices reduces the impact of noise and improves the accuracy of crack detection. Finally, by improving the encoding-decoding structure and fusing feature maps from different levels, the final prediction result more closely approximates the actual pavement condition. Moreover, this invention can effectively apply the improved method to mobile devices, achieving end-to-end real-time and effective detection. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0038] Figure 1 This is a schematic flowchart of a lightweight road surface crack detection method according to the present invention.
[0039] Figure 2 This is a schematic diagram of the lightweight feature extraction module structure of the present invention.
[0040] Figure 3 This is a schematic diagram of the coordinated attention mechanism of the present invention.
[0041] Figure 4 This is a schematic diagram of the network structure of the lightweight cracked pavement detection network model of the present invention.
[0042] Figure 5 This is a comparison image of the test results of specific embodiments of the present invention and other algorithms on the DeepCrack, CFD, and CRACK500 datasets. Detailed Implementation
[0043] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.
[0044] Please see Figure 1 This invention provides a lightweight pavement crack detection method, comprising the following steps:
[0045] S1: Collect real road surface images and establish a road surface image database;
[0046] S2: Filter the images in the road image database and process them to obtain GroundTrue data, and construct training and testing sets;
[0047] S3: Use the training set to train a lightweight cracked pavement detection network model;
[0048] S4: Test the trained network model using the test set, and calculate the evaluation metrics Precision, Recall, F-score, Prams, and FLOPs based on the test results;
[0049] If the evaluation indicators meet the requirements, it means that the network model meets the requirements. Save the network model for extracting pavement cracks.
[0050] Otherwise, modify the hyperparameters of the network model and retrain;
[0051] S5: Use the saved network model to detect road surface cracks, classify the cracks, and assess their severity.
[0052] The present invention will be further described below with reference to specific terminology and the steps of the embodiments:
[0053] (1) Lightweight Feature Extraction Module (SandGlass Module)
[0054] Most existing crack detection methods use ordinary convolution operations to extract image features. As network depth increases, the number of network parameters also increases, leading to a significant increase in computational cost and an excessively large parameter set, making them unsuitable for mobile devices. To avoid this, a lightweight feature extraction module (hereinafter referred to as the SandGlass module) is introduced to reduce the use of ordinary convolutions. The SandGlass module is as follows: Figure 2As shown, the entire module consists of two feature extraction operations, one dimensionality reduction operation, and one dimensionality increase operation. In the feature extraction stage, two depthwise separable convolutions replace traditional convolution operations. Depthwise separable convolutions reduce the number of parameters and computational cost associated with ordinary convolution operations by decomposing a single ordinary convolution into channel-wise convolutions and pointwise convolutions. During image feature dimensionality reduction and increase, two small-kernel convolutions are used for dimensionality transformation. Simultaneously, the SandGlass structure incorporates an identity mapping for image features. When the network model learns a significant feature loss through the SandGlass module, the identity mapping is directly applied without any processing of the feature image, allowing the SandGlass module to construct deeper networks. The SandGlass feature extraction module has significantly fewer parameters and lower computational cost than ordinary convolutions, making it suitable for building small mobile networks.
[0055] Step S3, the process of training the lightweight cracked pavement detection network model using the training set, includes the following steps:
[0056] Step 3.1: Use multiple convolution mode to extract features from the image to be detected. In the multiple convolution mode, the first layer is a normal convolution, and the second and last layers are depthwise separable convolutions.
[0057] Step 3.2: Use four lightweight feature extraction groups to extract image features in the encoder stage. Each SandGlass extraction group contains two SandGlass modules and performs image size compression at the same time.
[0058] Step 3.3: Integrate the coordinated attention mechanism module into the first group of convolutional operations and each SandGlass extraction group;
[0059] Step 3.4: Recover the deep feature information of the network through upsampling and fuse it with the detailed features extracted in the encoder stage;
[0060] Step 3.5: Use the lightweight feature extraction group to further extract features from the fused feature image;
[0061] Step 3.6: Repeat steps 3.2 to 3.5 three times to form the final feature image;
[0062] Step 3.7: Use convolution operations to form the final predicted image;
[0063] Step 3.8: Train the network model.
[0064] (2) Attention Coordination Mechanism Module
[0065] Please see Figure 3The CoordinateAttention (CA) module performs global pooling in both the horizontal (X-axis) and vertical (Y-axis) directions, preserving long-range dependency information in both directions. The horizontal and vertical information are then concatenated, and convolution operations are used to interact the information in both directions. The resulting feature map is then segmented after being processed by Batch Normalization (BN) and a non-linear activation function. Convolution is performed on each segment, and attention is applied in both the horizontal and vertical directions. Finally, the feature map is fed into the Sigmoid function to obtain a feature map that can accurately locate the position of the target object.
[0066] (3) Lightweight Crack Detection Network Model
[0067] This invention divides crack detection into two parts: an encoder stage and a decoder stage. In the encoder stage, a set of convolutional operations and four lightweight feature extraction groups are used. A coordinating attention module is added at the end of each feature extraction group to enhance the expression of crack feature information. The first set of convolutional operations directly extracts image information using a combination of ordinary convolution and depthwise separable convolution. The latter four lightweight feature extraction groups each consist of two SandGlass modules. Each operation extracts image features while compressing the image size, reducing the computational load on the subsequent network model. After each operation, a coordinating attention mechanism is applied to the feature image to enhance crack features. In the decoding stage, the detailed features retained by different stages of the encoder are fused in the decoder stage. The final feature information of the encoder is upsampled in the decoder stage for feature recovery and semantic enhancement. By fusing with features from different stages of the encoder, the semantic features of the crack image are enhanced. At the same time, the detailed features of the crack image retained in the encoder stage are used to enhance the crack details in the fused image. The fused features are extracted by a lightweight feature extraction module. During the extraction process, a coordinated attention mechanism is applied to the fused feature map to strengthen the expression of crack information and filter out image noise. Through layer-by-layer extraction, recovery, and fusion by the decoder, the crack prediction image is finally formed. The specific lightweight crack pavement detection network model structure diagram is shown below. Figure 4 As shown.
[0068] In addition, the training of the network model in step 3.8 specifically includes the following steps:
[0069] Step 3.8.1: Reproduce the network model using PyTorch;
[0070] Step 3.8.2: Set the model parameters. The input image size is 256×256, the learning rate is le-4, the weight of the fusion layer is set to 1, the learning rate is reduced by 10 times every 100 epochs, the weight decay is 2e-4, the number of training epochs is 700, and the model is saved every 50 epochs.
[0071] Step 3.8.3: Conduct experiments on a server equipped with a Tesla-V100-SXM2-32GB GPU and a 4-core Intel(R)Xeno(R)Sliver4214 CPU;
[0072] Step 3.8.4: Adjust the parameters of the network model based on the experimental results and evaluate the crack detection effect;
[0073] Step 3.8.5: Repeat steps 3.8.2 to 3.8.4 to adjust the network model parameters to achieve the best crack detection effect.
[0074] Furthermore, to verify the effectiveness of the proposed crack detection method, this invention also proposes a specific embodiment:
[0075] Validation was performed on three public crack datasets: DeepCrack, CFD, and CRACK500. Comparisons were made with popular edge detection methods SegNet and CrackSegNet, image segmentation methods U-Net and Deeplabv3+, and crack detection methods DeepCrack and FPHBN. Evaluation metrics included Precision, Recall, F-score, Params, and FLOPs. Test results on the three datasets are as follows: Figure 5 As shown, from a visual perspective, the crack features extracted by this invention are clear and the feature map is less affected by noise, while some methods suffer from blurred or even incomplete crack features due to noise. From an objective perspective, this invention uses the final prediction results of Precision, Recall, F-score, Params, and FLOPs for evaluation. Table 1 shows that the model parameter count and model complexity of this invention are much smaller than other methods, indicating significant potential for mobile applications. Tables 2, 3, and 4 show that the various indicators of this invention are close to other methods, achieving optimal accuracy and F-score in certain specific crack detection scenarios. In summary, the crack detection method proposed in this invention achieves crack image detection with high accuracy and low computational cost, demonstrating good performance.
[0076] In summary, compared with the prior art, the present invention has the following advantages:
[0077] 1. Improve the traditional convolution operation for feature extraction by using lightweight feature extraction. By combining depthwise separable convolution with small convolution kernels, the computational cost of the entire network model is reduced while the network model has good image feature extraction capabilities. This allows the network model to achieve the same detection effect as a larger network model with less computing power.
[0078] 2. A coordinated attention mechanism module suitable for mobile devices is introduced. Through the coordinated attention mechanism, the network in this mode can enhance crack features, suppress non-crack features, reduce the impact of noise, and improve the accuracy of crack detection.
[0079] 3. The encoder-decoder structure was improved by embedding the lightweight feature extraction module and the coordinated attention mechanism module into the encoder-decoder structure, which enhanced the feature extraction capability of the encoder and the size recovery capability of the decoder. Feature maps obtained at different levels were extracted separately. After a series of operations such as lightweight feature extraction module and upsampling, the feature maps at different levels were fused by the decoder, so that the final prediction result is closer to the real road conditions.
[0080] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.
Claims
1. A method for detecting cracks in lightweight pavement, characterized in that, Includes the following steps: Step 1: Collect real road surface images and establish a road surface image database; Step 2: Filter the images in the road surface image database and process them to obtain Ground True data, and construct the training set and test set; Step 3: Train a lightweight cracked pavement detection network model using the training set; Step 3 specifically includes the following steps: Step 3.1: Use multiple convolution mode to extract features from the image to be detected. In the multiple convolution mode, the first layer is a normal convolution, and the second and last layers are depthwise separable convolutions. Step 3.2: Use four lightweight feature extraction groups to extract image features in the encoder stage. Each lightweight feature extraction group contains two lightweight feature extraction modules and compresses the image size at the same time. Step 3.3: Integrate the coordinated attention mechanism module into the first group of convolutional operations and each group of lightweight feature extraction; Step 3.4: Recover the deep feature information of the network through upsampling and fuse it with the detailed features extracted in the encoder stage; Step 3.5: Use the lightweight feature extraction group to further extract features from the fused feature image; Step 3.6: Repeat steps 3.2 to 3.5 three times to form the final feature image; Step 3.7: Use convolution operations to form the final predicted image; Step 3.8: Train the network model; The lightweight feature extraction module consists of two feature extraction operations, one dimensionality reduction operation, and one dimensionality increase operation. In the feature extraction stage, two depthwise separable convolutions are used to replace the traditional convolution operation. The depthwise separable convolution reduces the number of parameters and computational cost caused by using ordinary convolution operations by decomposing a single ordinary convolution into channel-wise convolution and point convolution. In the image feature dimensionality increase and reduction, two small kernel convolutions are used to perform dimensionality transformation. The coordinated attention mechanism module performs global pooling in both the horizontal X-axis and vertical Y-axis directions to preserve long-range dependency information in both directions. The horizontal and vertical information is then concatenated, and convolution operations are used to interact the information in both directions. The resulting feature map is then segmented after being processed by BN and a non-linear activation function. Convolution is performed on each segmentation map, and attention is applied in both the horizontal and vertical directions. Finally, the feature map is fed into the Sigmoid function to obtain a feature map that can accurately locate the position of the target object. Step 4: Test the trained network model using the test set, and calculate the evaluation metrics Precision, Recall, F-score, Prams, and FLOPs based on the test results; If the evaluation indicators meet the requirements, it means that the network model meets the requirements. Save the network model for extracting pavement cracks. Otherwise, modify the hyperparameters of the network model and retrain; Step 5: Use the saved network model to detect road surface cracks, classify the cracks, and assess their severity.
2. The lightweight pavement crack detection method as described in claim 1, characterized in that, In the encoder stage, a set of convolutional operations and four sets of lightweight feature extraction module operations are used. A coordinating attention module is added at the end of each feature extraction group to enhance the expression of crack feature information. The first set of convolutional operations directly extracts image information using a combination of ordinary convolution and depthwise separable convolution. The latter four sets of lightweight feature extraction module operations each consist of two lightweight feature extraction modules. Each operation extracts image features while compressing the image size to reduce the subsequent computational load of the network model. After each set of operations, a coordinating attention mechanism is applied to the feature image to enhance crack features.
3. The lightweight pavement crack detection method as described in claim 2, characterized in that, During the execution of steps 3.3 to 3.6, a decoder stage is also included. The detailed features retained by different stages of the encoder will be fused in the decoder stage. The final feature information of the encoder will be restored and semantically enhanced by upsampling in the decoder stage. By fusing with the features from different stages of the encoder, the semantic features of the crack image are enhanced. At the same time, the crack image details retained by the encoder stage are used to enhance the crack details in the fused image. The fused features are extracted by a lightweight feature extraction module. During the extraction process, a coordinated attention mechanism is applied to the fused feature map to strengthen the expression of crack information and filter out image noise. The crack prediction image is finally formed by the decoder extracting, restoring and fusing layer by layer.
4. The lightweight pavement crack detection method as described in claim 3, characterized in that, Step 3.8 includes the following steps: Step 3.8.1: Reproduce the network model using PyTorch; Step 3.8.2: Set the model parameters. The input image size is 256×256, the learning rate is le-4, the weight of the fusion layer is set to 1, the learning rate is reduced by 10 times every 100 epochs, the weight decay is 2e-4, the number of training epochs is 700, and the model is saved every 50 epochs. Step 3.8.3: Conduct experiments on a server equipped with a Tesla-V100-SXM2-32GB GPU and a 4-core Intel(R) Xeno(R) Silver 4214 CPU; Step 3.8.4: Adjust the parameters of the network model based on the experimental results and evaluate the crack detection effect; Step 3.8.5: Repeat steps 3.8.2 to 3.8.4 to adjust the network model parameters to achieve the best crack detection effect.