A pavement crack segmentation method based on a dense connection strategy and image gradient

Abstract

The application provides a pavement crack segmentation method based on a dense connection strategy and an image gradient, and belongs to the technical field of pavement crack detection, and comprises the following steps: constructing a pavement crack segmentation model, wherein the pavement crack segmentation model comprises: a feature encoder module, which is used for outputting a feature map; a feature enhancement module, which is used for receiving the feature map and outputting an enhanced feature map; a feature decoder module, which is used for converting the enhanced feature map into a target image; an edge refiner module, which is used for converting a pavement image into a real edge gradient image by using a Sobel operator; and an image splicing module, which is used for splicing the real edge gradient image and a crack image to obtain a pavement crack segmentation image; and inputting a pavement original image into the edge refiner and the feature encoder module respectively to output a crack segmentation image. The method can segment a pavement crack image.

Description

Technical Field

[0001] This invention belongs to the field of crack segmentation technology, specifically relating to a road surface crack segmentation method based on a dense connection strategy and image gradient. Background Technology

[0002] Roads are a vital component of the transportation system, providing great convenience for people's daily travel. Cracks are the most common defects on road surfaces. Regular crack detection of roads is crucial to ensuring transportation safety. However, manual crack detection is time-consuming, inefficient, and prone to safety issues. Therefore, automated crack detection methods have become a hot research topic in recent years.

[0003] Initially, with the development of image processing technology, digital image processing-based methods were used to automatically detect cracks. Considering the difference between the crack and a background threshold, a threshold-based method was used to segment the crack region.

[0004] Due to the differences in grayscale gradients and morphological features between cracks and the background, edge detection and region growing methods have been proposed. However, these methods are suitable for high-contrast, continuous crack images and struggle to handle complex backgrounds and topological structures. To overcome these issues, a method for manually creating crack features has been proposed, employing algorithms such as support vector machines and random forests for crack detection. However, in practical detection, the complexity of crack features leads to significant discrepancies between manually created features and actual cracks, resulting in detection accuracy that fails to meet application requirements.

[0005] In recent years, deep learning methods have been widely applied in computer vision due to their ability to automatically learn features from data, such as image classification, object detection, and image segmentation. Researchers have applied deep learning methods to crack detection, improving its efficiency and accuracy. Some works treat crack detection as a classification task, combining CNN models and sliding window methods. However, the presence of fully connected layers results in low efficiency and high computational cost. Other works utilize object detection methods to locate crack position information, but cannot obtain the geometric features of the cracks. Since the introduction of fully convolutional neural networks (FCNs), the field of image segmentation has developed rapidly. Some works treat crack detection as a segmentation task, such as typical end-to-end encoder-decoder frameworks like SegNet and U-Net.

[0006] The study found that the Encoder-Decoder structure reduces the loss of crack features to some extent. However, due to the presence of pooling layers, some detailed feature information is still lost. Existing image segmentation methods have improved the accuracy of crack detection, but most network feature extraction methods lack sufficient non-local information, and the predicted crack edges are relatively thick. In addition, crack detection often encounters the problem of extreme class imbalance, making it difficult to improve the final performance. Summary of the Invention

[0007] To overcome the shortcomings of the existing technology, the present invention provides a road surface crack segmentation method based on dense connection strategy and image gradient.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A pavement crack segmentation method based on a dense connection strategy and image gradients includes:

[0010] Construct a pavement crack segmentation model;

[0011] The pavement crack segmentation model includes:

[0012] The feature encoder module is a five-layer ResNet50 network with the fully connected layers removed, and its layers are connected by a dense connection strategy; it is used to output feature maps.

[0013] The feature enhancement module is used to receive the feature map and output the enhanced feature map.

[0014] The feature decoder module, which includes five deconvolutional layers and is bridged with the five-layer structure of the feature encoder module, is used to convert the enhanced feature map into the target image.

[0015] The edge refiner module is used to transform an image into a true edge gradient image using the Sobel operator;

[0016] The image stitching module is used to stitch together the real edge gradient image and the target image to output the target segmentation image;

[0017] The original road surface image is input into the edge refiner module and feature encoder module of the trained road surface crack segmentation model, and the target segmentation image is output.

[0018] Furthermore, the feature encoder module and the feature decoder module are bridged in the following way:

[0019] The feature maps output by each layer of the feature encoding module from input to output are denoted as n1-n5, and the images output by each layer of the feature decoder module from the layer closest to the feature enhancement module to the layer furthest from the feature enhancement module are denoted as b1-b5. Among them, n5 is enhanced by the feature enhancement module and then input into the feature decoder module. After one deconvolution process, b5 is output. b5 is concatenated with n4 and used as the input for the next deconvolution operation, and so on.

[0020] Furthermore, the n1 and n2 feature maps output by the feature encoder module are processed by the SimAM attention module and then concatenated with b4 and b3 in the feature decoder module, respectively.

[0021] Furthermore, the n3 and n4 feature maps output by the feature encoder module are processed by Dropout and then concatenated with b2 and b1 in the feature decoder module, respectively.

[0022] Furthermore, the edge refiner module also includes:

[0023] The channel enhancement module is used to enhance the channels of the real edge gradient image. It includes: a 1×1 convolutional layer; a global average pooling layer, whose input is connected to the output of the 1×1 convolutional layer; a 2×2 hybrid pooling layer, whose input is connected to the output of the 1×1 convolutional layer; a 4×4 hybrid pooling layer, whose input is connected to the output of the 1×1 convolutional layer; a feature map concatenation layer, whose input is connected to the outputs of the global average pooling layer, the 2×2 hybrid pooling layer, and the 4×4 hybrid pooling layer; two cascaded fully connected layers, whose input is connected to the output of the feature map concatenation layer, used to output the channel weights; and a feature map generation layer, whose input is connected to the outputs of the two cascaded fully connected layers, used to multiply the weights and features to obtain the enhanced channel real edge gradient image.

[0024] The spatial enhancement module is used to spatially enhance the real edge gradient image after channel enhancement. It includes three parallel branches: a 3×3 convolutional layer; a 3×3 convolutional layer connected in series with a 4×4 hybrid pooling layer; and a 3×3 convolutional layer connected in series with a 2×2 hybrid pooling layer.

[0025] Furthermore, the loss function for training the road surface crack segmentation model is:

[0026] L = L side (X,Y,W,w)+L fuse (X,Y,W)

[0027] Among them, L side L represents the loss function of the five outputs during the decoding stage. fuseX is the loss function output by the road surface crack segmentation model, where X is the original image, Y is the label, W is the parameters learned by the network, and w is the weight.

[0028] further:

[0029] L fuse (X,Y,W)=L′ WCE (X,Y,W)+L Dice (X,Y,W)

[0030] Among them, L' WCE For the improved weighted cross-entropy loss function, L Dice This is the dice loss function.

[0031] Furthermore, the improved weighted cross-entropy loss function is:

[0032]

[0033] Where α satisfies:

[0034]

[0035] Where β=|G - | / (|G + |+|G - |), G - and G + These represent the edge and non-edge real label sets, respectively.

[0036] Furthermore, the feature enhancement module includes: a deformable convolution module and an enhanced convolution module connected to the output of the deformable convolution module; the enhanced convolution module includes:

[0037] A 1×1 convolutional layer; a 1×1 convolutional layer connected in series with a 3×3 dilated convolutional layer; a 1×1 convolutional layer connected in series with a 3×3 dilated convolutional layer; a 1×1 convolutional layer connected in series with a 3×3 dilated convolutional layer; a hybrid pooling layer connected in series with a 1×1 convolutional layer; the hybrid pooling layer includes three parallel pooling layers, which include: a k×k average pooling layer; a 1×k average pooling layer; and a k×1 average pooling layer.

[0038] The road surface crack segmentation method based on dense connection strategy and image gradient provided by this invention has the following beneficial effects:

[0039] This invention uses ResNet50 as the feature extraction network, employs a dense connection strategy in each encoding stage to reuse low-level features to preserve edge detail information; it uses deformable convolution to more effectively extract topological crack features; it uses an enhanced convolution module to replace traditional convolution to obtain multi-scale information features; and it integrates feature information from each decoding stage in the final prediction result, using the Sobel operator to generate a true edge gradient image, further refining the network prediction results. Attached Figure Description

[0040] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of a road crack segmentation model structure according to an embodiment of the present invention, which is a road crack segmentation method based on dense connection strategy and image gradient.

[0042] Figure 2 This is a schematic diagram illustrating the dense connection strategy of an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating the deformable convolution feature extraction process according to an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram comparing standard convolution and dilated convolution in an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the enhanced convolution module according to an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the mixing pool module according to an embodiment of the present invention;

[0047] Figure 7 This is a schematic diagram of the SimAM attention module in an embodiment of the present invention.

[0048] Figure 8 This is a schematic diagram of the channel enhancement module according to an embodiment of the present invention;

[0049] Figure 9 This is a schematic diagram of the space enhancement module according to an embodiment of the present invention;

[0050] Figure 10 This is a schematic diagram of the variation curves of parameter β and F1-score on the CFD dataset in this embodiment of the invention;

[0051] Figure 11This is a schematic diagram of the linear function variation curve of β according to an embodiment of the present invention;

[0052] Figure 12 This is a schematic diagram illustrating the evaluation metrics for different periods on a CFD dataset according to an embodiment of the present invention.

[0053] Figure 13 This is a schematic diagram of the F1-score curves of different methods on the CFD dataset in this embodiment of the invention;

[0054] Figure 14 This is a schematic diagram of qualitative comparison results on a CFD dataset according to an embodiment of the present invention;

[0055] Figure 15 This is a schematic diagram of the F1-score curves of different methods on the Crack500 dataset in an embodiment of the present invention.

[0056] Figure 16 This is a schematic diagram of the qualitative comparison results on the Crack500 dataset in an embodiment of the present invention;

[0057] Figure 17 Visualization of PA, MIoU, and F1-score evaluation metrics under different loss functions in embodiments of the present invention;

[0058] Figure 18 The dataset is GAPS384;

[0059] Figure 19 This represents the prediction results for the network dataset. Detailed Implementation

[0060] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0061] Example:

[0062] This invention provides a method for road surface crack segmentation based on a dense connection strategy and image gradients, specifically as follows: Figure 1-19 As shown, the process includes: constructing a road surface crack segmentation model; inputting the original road surface image into the edge refiner module and feature encoder module of the trained road surface crack segmentation model, and outputting a road surface crack segmentation image.

[0063] The pavement crack segmentation model includes:

[0064] The feature encoder module is a five-layer ResNet50 network with the fully connected layers removed, and its layers are connected by a dense connection strategy; it is used to output feature maps.

[0065] The feature enhancement module receives the feature map and outputs the enhanced feature map; the feature decoder module includes 5 deconvolutional layers and is bridged with the five-layer structure of the feature encoder module; it is used to convert the enhanced feature map into the target image.

[0066] The edge refiner module is used to transform the image into a true edge gradient image using the Sobel operator; the image stitching module is used to stitch the true edge gradient image and the target image together to output the target segmentation image.

[0067] The feature enhancement module includes: a deformable convolution module and an enhancement convolution module connected to the output of the deformable convolution module; the enhancement convolution module includes: a 1×1 convolutional layer; a 1×1 convolutional layer connected in series with a 3×3 dilated convolutional layer; a 1×1 convolutional layer connected in series with a 3×3 dilated convolutional layer; a 1×1 convolutional layer connected in series with a 3×3 dilated convolutional layer; a hybrid pooling layer connected in series with a 1×1 convolutional layer; the hybrid pooling layer includes three parallel pooling layers, which include: a k×k average pooling layer; a 1×k average pooling layer; and a k×1 average pooling layer.

[0068] The following are the specific implementation details of the present invention:

[0069] 1) Feature encoder module

[0070] This invention uses ResNet50 as the baseline network for the feature encoder. Considering the loss of detail information in ResNet50 due to pooling and multiple convolutions, and that the network can only propagate features layer by layer, this invention employs a dense connection strategy to enhance feature propagation. For example... Figure 2 As shown, the dense connection strategy is a module in DenseNet. Let X0 represent the input layer, X... l This represents the output of each stage. The output of each stage of the feature encoder module can be represented as:

[0071] X l =H l ([X0,X1,...,X l-1 ])

[0072] Where l represents the stage, H l This represents a nonlinear transformation function, which is a combination of operations including a series of 1×1 Conv, BN, ReLU, and Average Pooling operations.

[0073] 2) Feature enhancer

[0074] 2.1 Deformable Convolution Module

[0075] Existing CNNs, due to the specific geometric structure of convolutions, lack a mechanism to adapt to geometric deformations within the network. Therefore, for complex geometric linear structures like cracks, using traditional convolutions to extract crack features retains a significant amount of background information. To alleviate this problem, deformable convolutions are introduced to replace traditional convolutions. Due to the characteristics of deformable convolutions, this invention stacks multiple deformable convolutions to construct a deformable convolution module.

[0076] like Figure 3 The diagram illustrates the complete process of deformable convolutional feature extraction. It utilizes standard convolution to learn corresponding offsets, then superimposes these learned offsets onto the corresponding pixel positions of the input features, allowing the convolution kernel to diverge and sample across the input feature map, thereby achieving focal length targeting of the target features. Mathematically, this invention defines the convolution kernel as R, x... n ∈R. Then the output feature map y of the input feature X after x0 undergoes deformable convolution can be expressed as:

[0077]

[0078] In the formula, ω(*) represents the sampling point weight, and Δx n This represents the offset of the sampling point.

[0079] like Figure 3 The process of deformable convolutional feature extraction involves applying a convolution to the feature map to obtain an offset. This offset is learned using an interpolation algorithm through backpropagation.

[0080] 2.2 Enhanced Convolution Module

[0081] To reduce the number of network parameters, convolutional kernels are typically designed with small receptive fields of 1×1 and 3×3. Small kernels limit the receptive field area when the network extracts features, making it difficult to improve crack detection accuracy. Inspired by DeeplabV3, this invention introduces dilated convolution. Dilated convolution adds a Dilation Rate parameter to standard convolution to control the sampling stride of the receptive field area; increasing the Dilation Rate increases the receptive field area. Figure 4 As shown, standard convolution ( Figure 4 (a) and dilated convolution ( Figure 4 (b) Comparison of the convolution process. This invention designs a final enhanced convolution module based on the Inception architecture.

[0082] like Figure 5As shown, this module consists of five sub-networks. Three of them are composed of 3×3 dilated convolutions. A 1×1 convolution is used before the dilated convolutions to reduce the feature channels. Using dilated convolutions with different dilation rates is beneficial for handling crack features of different sizes. Additionally, this invention adds a 1×1 convolution to correct linear activation. Simultaneously, to better capture contextual information, this invention adds pooling. Since conventional pooling for irregular cracks will contain a large amount of background information, this invention introduces a hybrid pooling module to replace conventional pooling. Figure 6 As shown, the mixed pool module consists of k×k average pooling, 1×k average pooling, and k×1 average pooling.

[0083] Figure 5 The enhanced convolutional module consists of five parts, with dilated convolutions at different dilation rates, each capturing non-local information. Hybrid pooling acquires multi-scale features. 1×1 convolutions are used to reduce feature channels and decrease the number of network parameters.

[0084] 3) Feature Decoder Module

[0085] In the feature decoding stage, this invention uses the transpose of a learnable parameter matrix to gradually recover crack features. Some crack information is lost during resolution restoration. Therefore, this invention uses a bridging mechanism to connect the features from the encoding and decoding stages, directly preserving detailed information.

[0086] Considering that crack detection is a small object segmentation task, this invention uses an attention mechanism in the low-level feature map to capture rich detail information.

[0087] To prevent overfitting, a Droupt operation is added during the bridging connection of high-level feature maps. For better network training, this invention introduces deep supervision at each decoding stage to generate prediction results. Finally, to further utilize information from different layers effectively, this invention fuses features from each decoding stage.

[0088] Attention mechanisms are widely used in computer vision. In recent years, many researchers have applied attention mechanisms to crack detection. Because the output of the feature encoding stage integrates multi-scale features, information redundancy occurs. To address this issue, this invention introduces an attention mechanism. Common attention mechanisms include spatial, channel, and hybrid attention mechanisms. Spatial and channel attention focus only on one aspect, space or channel. Hybrid attention, while considering both, directly embeds itself into the network, increasing the number of network parameters. Therefore, this invention introduces the attention module SimAM. It is a parameter-free 3D attention mechanism that embeds itself into the network without increasing network parameters and can assign different weights to both channel and spatial attention. The principle is as follows: Figure 7 As shown.

[0089] SimAM is based on visual neuroscience theory. It specifically designs an energy function to calculate weights and selects operators by defining the solution of the energy function. The closed-form solution of the energy function for each pixel t is as follows:

[0090]

[0091] in λ represents the mean and variance of all pixels except t in the channel dimension. In this paper, λ is set to 0.0001.

[0092] From the closed-form expression, it can be seen that the smaller e(t) is, the greater the difference between pixel t and its surrounding pixels. Therefore, the importance of each pixel can be obtained by 1 / e(t). Mathematically speaking, the pixels in the input feature X, after passing through SimAM, correspond to the output feature. The pixel relationships are as follows:

[0093]

[0094] Where S(x) represents the Sigmoid activation function, x i ∈X,

[0095] 4) Edge refiner

[0096] Deep learning has become the mainstream method for crack detection. However, the thick and blurry edges of crack predictions using convolutional neural networks have been a persistent drawback. This invention proposes an edge refiner that combines image gradients with coarse prediction results to refine the edge prediction results. Figure 1 As shown, the Sobel operator

[37] is first used to perform edge detection and generate a real edge gradient image. At this time, the gradient image contains many irrelevant edges.

[0097] To reduce the impact of these irrelevant edges, this invention proposes a spatial enhancement module and a channel enhancement module composed of CNNs. These two modules not only enable image gradients to participate in training, but also allow for analysis of the relationship between coarse prediction results and image gradients from both spatial and channel perspectives, thereby better adapting to real crack edge features.

[0098] 4.1 Channel Enhancement Module

[0099] The purpose of the channel enhancement module proposed in this invention is to learn the relationships between input feature channels. For example... Figure 8As shown, firstly, to reduce computational complexity, a 1×1 convolution is used to decrease the number of channels. Then, the features are input into global average pooling, 2×2 hybrid pooling, and 4×4 hybrid pooling respectively, and the outputs are merged. The merged features are then passed through two fully connected layers to obtain the channel weights of the input features. Finally, the input features are multiplied by the channel weights. This can be expressed by the following formula:

[0100]

[0101] Where X represents the input feature, and CA represents the feature mapping of the channel enhancement module. φ C , Let f represent a fully connected layer, channel concatenation, global average pooling, a 1×1 convolutional layer, and hybrid pooling, respectively. Then the output f is:

[0102] f = CA * x

[0103] 4.2 Space Enhancement Module

[0104] To adapt to the real-world crack edge features, this invention utilizes a CNN model to analyze the crack's shape and location. However, due to the fixed receptive field of CNNs, it is difficult to capture crack features of different sizes. Therefore, multi-scale feature maps are generated using different object sizes to capture crack features. The spatial augmentation module proposed in this invention is as follows: Figure 9 As shown, its output is the sum of three. To better adapt to the characteristics of irregular cracks, a hybrid pooling module is used instead of average pooling.

[0105] 5) Loss Function

[0106] Crack detection is a typical binary classification task. The commonly used loss function is the cross-entropy loss function

[38] . In this paper, the proportion of cracks in each crack image on the CFD and Crack500 datasets is between 0.01 and 0.12, which is a dataset with extremely imbalanced classes. The cross-entropy loss function pays equal attention to each class, and the model training is biased towards the background, which cannot solve the problem of class imbalance. The proposed weighted cross-entropy loss function reduces the impact of positive and negative sample imbalance to a certain extent.

[0107] The training set for this article is S = (X n ,Y n ), n=1...N, where X n and Y n These represent the original image and the label, respectively. Y pred This represents the prediction result. For convenience, the subscript n will be omitted below. In this invention, W represents the parameters that the entire network needs to learn. The five side outputs in the decoding stage can be considered as a single weight w. mFor classifiers with m = 1...5, the loss function can be defined as:

[0108]

[0109] The weighted cross-entropy loss function is defined as follows:

[0110]

[0111] Where β=|G - | / (|G + |+|G - |). G - and G + Let represent the edge and non-edge true label sets, respectively. The parameter β can balance the loss function between positive and negative samples within a certain range.

[0112] This invention conducts experiments on a CFD dataset to verify the impact of parameter β on the experimental results. For example... Figure 10 As shown, the experimental results are not ideal when the parameter β is extremely small or extremely large. The best results are achieved when β is around 0.25. The crack ratio has been statistically estimated to be between 0.01% and 0.12%. Figure 10 It was found that β did not reach its optimal value within this range, and most of the crack proportions were concentrated around 0.01, which resulted in the weighted cross-entropy training model being less than ideal. This invention is based on... Figure 10 The experiments showed that β increased between 0.1 and 0.25, reaching its optimum near 0.25, indicating that appropriately increasing the β value is beneficial for model training. Therefore, this invention utilizes a linear function to increase the β value. Assuming extreme cases, when β = 0, a linear function is used to achieve β = 0.1; when β = 0.12, a linear function is used to achieve β = 0.25. When β is greater than 0.12, this invention fixes β at 0.25. That is, the linear function is as follows... Figure 11 As shown, its formula is as follows:

[0113]

[0114] Therefore, this invention proposes the following method to improve the weighted cross-entropy loss:

[0115]

[0116] This paper incorporates a dice loss function into the improved weighted cross-entropy loss. The dice loss function is beneficial not only for imbalanced positive and negative samples but also for considering the overall situation of crack prediction. Its definition is as follows:

[0117]

[0118] This is a local minimum to prevent the denominator from being zero. Based on the above, the expression for Δ is:

[0119] Δ=L′ WCE (X,Y,W,w)+L Dice (X,Y,W,w)

[0120] The five side output layers are connected together to obtain the final fusion layer, where the loss function is expressed as:

[0121] L fuse (X,Y,W)=L′ WCE (X,Y,W)+L Dice (X,Y,W)

[0122] In summary, this paper improves the model by combining the weighted cross-entropy loss function with the dice loss function. The overall loss function for model training is:

[0123] L = L side (X,Y,W,w)+L fuse (X,Y,W)

[0124] This paper uses ResNet50 as the feature extraction network. While high-level features in the encoding stage contain rich semantic information, they lack edge detail. This invention reuses low-level features in each encoding stage using a dense connection strategy. A 3D attention mechanism is used in the low-level feature map to capture subtle crack features. Deformable convolutions are employed for more efficient extraction of topological crack features. An enhanced convolutional block replaces traditional convolutions to obtain multi-scale information features. The final prediction result integrates feature information from each decoding stage. Furthermore, the Sobel operator is used to generate true edge gradients, further refining the network prediction results. Additionally, this paper proposes an improved weighted cross-entropy function combined with a dice loss function to address class imbalance and incorporates deep supervision into the network.

[0125] The main contributions of this paper can be summarized as follows:

[0126] 1. A road surface crack segmentation method based on dense connection strategy and image gradient encoder-decoder structure is proposed, which can detect cracks under complex background conditions.

[0127] 2. This invention employs a dense connection strategy to reuse low-level features, enhance feature transmission, and prevent the loss of detailed information.

[0128] 3. Incorporate image gradients into the training process to refine the network's prediction results.

[0129] 4. This invention proposes an improved weighted cross-entropy function combined with a dice loss function to solve the class imbalance problem.

[0130] The following are specific embodiments and comparisons of the method of the present invention:

[0131] 5) Experiments and Results

[0132] 5.1 Implementation Details:

[0133] This paper implements the network based on the TensorFlow 2 deep learning framework in Python. Model training and prediction were performed on a Linux system using a 14-core Intel(R) Xeon(R) Gold 6330 CPU at 2.00GHz and an RTX 3090 GPU. LabelMe software was used for annotation. During training, early stopping, a batch size of 16, and a training epoch of 200 were implemented. Cosine decay was used to decrease the learning rate. A momentum of 0.9 was chosen as the parameter for the Adam optimizer to update the network parameters.

[0134] 5.2. Dataset:

[0135] 1) Crack500: This contains 500 images of concrete pavement cracks at a resolution of 1440×2560. These images have complex background noise and are often mixed together by similar backgrounds. This invention crops the images into 3364 images of size 360×640. 2692 images are used as the training set, 336 images as the validation set, and 336 images as the test set.

[0136] 2) CFD: It contains 118 images of asphalt pavement cracks at a resolution of 480×320. The cracks in the images are relatively small and contain background noise. Due to the limited dataset, this invention performs data augmentation operations such as rotation and mirroring on the images, ultimately obtaining 1000 images. This invention uses 800 images as the training set, 100 images as the validation set, and 100 images as the test set.

[0137] 3) GAPS384: This dataset contains 509 images of road surface cracks at two resolutions: 640×540 and 440×540. The dataset focuses on extreme lighting variations on the road surface. Cracked areas are divided into overly dark and overly bright regions. This is used to evaluate the model of this invention.

[0138] 5.3. Indicators

[0139] To evaluate the work of this invention, commonly used evaluation metrics in segmentation tasks are introduced: Precision, Recall, F1-score, PA, MPA, and MIoU. Their formulas are as follows:

[0140]

[0141]

[0142]

[0143]

[0144]

[0145]

[0146] Wherein, TP, FP, TN and FN represent predicted as a crack and actually a crack, predicted as a crack and actually not a crack, predicted as not a crack and actually not a crack, and predicted as not a crack and actually a crack, respectively.

[0147] like Figure 12 The results show the changes in F1-score, Precision, Recall, and MIoU during the iteration process on the CFD dataset. It can be seen that the evaluation metrics increase with the number of iterations, but gradually plateau after 125 iterations.

[0148] 5.4. Comparison Methods

[0149] 1) SegNet: A classic encoder-decoder network architecture. It employs a pooling index strategy to perform non-linear upsampling. During training, the batch size is set to 16, the initial learning rate is set to 1e-4, and the Adam optimizer is used.

[0150] 2) U-Net: Similar to SegNet, it uses a classic encoder-decoder network structure. A bridging structure connects the encoder and decoder to prevent loss of detailed features. Hyperparameter settings are the same as SegNet.

[0151] 3) DeeplabV3: DeeplabV3 balances accuracy and processing time through dilated convolutions. It introduces a large number of dilated convolutions in the encoder part to increase the receptive field without loss of information, allowing each convolution output to contain a wider range of information. The initial learning rate is set to 5e-4, and the remaining parameters are the same as SegNet.

[0152] 4) FCN: The first network model to propose a fully convolutional neural network. Utilizing an integrated structure of upsampling and downsampling, it can detect cracks of different proportions. It employs the SGD optimizer with an initial learning rate set to 0.001.

[0153] 5) HED: VGG16 is used as the backbone network. The network uses deep supervision and fuses the side outputs of each stage for the final prediction. During training, the batch size is set to 16, the initial learning rate is set to 0.001, and the Adam optimizer is used.

[0154] 6) FPHBN: Based on the HED model. It combines a feature pyramid and a hierarchical enhancement module for crack detection. The hyperparameter settings are the same as those for the HED network.

[0155] 7) DeepCrack: Based on the HED model. It incorporates conditional random fields and guided filtering to refine crack detection results. Hyperparameter settings are the same as for the HED network.

[0156] 5.5. Experimental Results and Discussion

[0157] CFD Results: On the CFD dataset, this invention records the changes in F1-score for different methods during training. For example... Figure 13 As shown, the algorithm in this paper achieves the highest F1-score after iteration convergence. FCN has the lowest F1-score, similar to SegNet and HED.

[0158] As shown in Table 1, the method proposed in this invention has certain advantages. The F1-score reaches 0.875, while FCN performs the worst. The MIoU value is also higher than other methods, reaching 0.801. Furthermore, the evaluation metrics PA, MPA, and Recall of this invention are all superior to other methods.

[0159] like Figure 14 As shown, SegNet, FCN, and HED yielded poor detection results, exhibiting discontinuities, gaps, and false positives, especially for... Figure 14 (vi) In general, cracks are not detectable. Figure 14 (iii) UNet, FPHBN, and DeepCrack misdetect scratches as crack pixels. For blocky cracks and crazing cracks, Deeplabv3 and DeepCrack also show discontinuities in their detection results. For more complex... Figure 14 (vi) In comparison, the detection results of the methods described in this paper are not ideal. The method proposed in this paper can detect cracks under complex conditions better. Furthermore, the method proposed in this invention can also detect minute cracks. For example... Figure 14 (one), Figure 14 (vi) Detection results. Compared with the comparative method, the present invention can accurately detect minute cracks. Therefore, the method proposed in this invention has certain advantages.

[0160] Figure 14 The first and second columns represent the original image and the actual ground reality, respectively. The first column is from top to bottom: Figure 14 (a) to Figure 14 (iii) The background of the crack image includes markings, scratches, and low light. Figure 14 (Four), Figure 14 (v) The cracks are blocky or crazing-like. Figure 14 (vi) The cracks are small and contain background elements such as manhole covers.

[0161] Table 1 Quantitative evaluation on the CFD dataset

[0162]

[0163] Results from Crack500: The proposed method is compared with other methods on the Crack500 dataset. Figure 15 As shown, the method of the present invention has the highest F1-score value, while FCN has the lowest F1-score value.

[0164] As shown in Table 2, the method proposed in this invention has certain advantages. The F1-score reaches 0.892, followed by UNet and FPHBN, with F1-scores of 0.881 and 0.879 respectively. The MIoU value is also higher than other methods, reaching 0.821. Furthermore, the evaluation metrics PA, MPA, and Recall of this invention are all superior to other methods.

[0165] like Figure 16 As shown, for simpler crack images, both the comparison method and the proposed method can detect the cracks. FCN and HED are relatively less effective. The proposed method's detection results are closer to real road surface cracks compared to other methods. For more complex crack images, SegNet, FCN, and HED are largely unable to detect blurred crack images, such as... Figure 16 (iv) Detection results. Although UNet, FPHBN, DeepCrack, and Deeplabv3 detected cracks, the detection results showed discontinuities. From... Figure 16 (iv) The test results show that the method proposed in this paper can be well adapted to crack detection in fine and ambiguous backgrounds.

[0166] Table 2 Quantitative evaluation on the Crack500 dataset

[0167]

[0168] Figure 16 This is a qualitative comparison result on the Crack500 dataset. The first and second columns represent the original image and the ground truth, respectively. The first column is from top to bottom: Figure 16 (a) to Figure 16 (iii) A simpler crack image. Figure 16 (iv) to Figure 16 (vi) Images of more complex cracks.

[0169] 4.6. Ablation Test

[0170] This invention uses the proposed method without adding a dense connection strategy (DCS), a feature enhancer (FI), and an edge refiner (ER) as the baseline model (B). Other parameter settings are the same as the proposed method. The baseline model performs the worst on the CFD dataset. Based on the baseline model, DCS, FI, and ER are added sequentially. As shown in Table 3, the proposed modules improve crack detection capabilities. Among them, ER shows the best improvement in crack detection. Furthermore, combining the three modules in pairs yields results that are better than the original modules, indicating that these modules are compatible and can mutually enhance each other. Finally, this invention combines these three modules together to form the proposed method, which achieves better results than the original individual modules.

[0171] Table 3 Ablation experiments on the CFD dataset

[0172]

[0173] 5.7. Loss Function Analysis

[0174] This paper employs an improved weighted cross-entropy loss function combined with a dice loss function to train the model. To verify the feasibility of the proposed loss function, weighted cross entropy loss (WCE), Dice loss (Dice), weighted cross entropy loss + Dice loss (WCE + Dice), and the proposed loss function were used as the objective function of the model on a CFD dataset, as shown in Table 4.

[0175] Table 4 Quantitative Comparison of Different Loss Functions

[0176]

[0177] To more clearly compare the data in Table 4, this invention uses three important indicators—PA, MIoU, and F1-score—to create graphs. For example... Figure 17As shown, using only weighted cross-entropy loss to optimize the model yields the worst results. When weighted cross-entropy loss is combined with Dice loss, the PA value improves by 9.6%, and MIoU and F1-score also improve. The above analysis demonstrates that optimizing the model using only weighted cross-entropy loss is ineffective; combining weighted cross-entropy loss with Dice loss improves the model. Therefore, this invention improves upon weighted cross-entropy loss by combining it with Dice loss. As shown in Table 4, the improved objective function effectively improves the model. The improved weighted cross-entropy loss improves PA, MIoU, and F1-score by 1.4%, 1.2%, and 0.8%, respectively.

[0178] 5.8. Robustness Analysis

[0179] To verify the robustness of the proposed model, this invention performs a robustness analysis. Experiments are conducted on crack images obtained from the GAPS384 dataset and web crawlers. The model's ability to detect cracks is tested without being used in training.

[0180] like Figure 18 As shown, Figure 18 Using the GAPS384 dataset, this invention selected crack images with different backgrounds for testing. It can be seen that, even without training, the model proposed in this invention can effectively identify cracks in blurred and shadowed environments. To further verify the robustness of the model, some crack images obtained from web scraping are shown below. Figure 19 As shown, the model of the present invention can also effectively extract cracks. Therefore, the robustness of the model of the present invention is verified.

[0181] 6. Conclusion

[0182] This paper proposes a road crack segmentation network based on a dense connection strategy and an image gradient encoder-decoder structure. The network employs a dense connection strategy to reuse low-level features and enhance feature transfer. A feature enhancer is proposed, introducing variable convolution to better extract crack features. An enhanced convolutional module composed of dilated convolution and hybrid pooling acquires multi-scale information features. An attention mechanism is used in the low-level feature map to capture detailed feature information. Gradient images are used to refine the network's prediction results. Extensive experiments demonstrate the superiority and generalization of this method. Even under environmental conditions such as low contrast between cracks and background, blurriness, and darkness, the proposed method maintains better segmentation results than other methods, meeting certain engineering inspection requirements.

[0183] In the future, we can consider combining object detection algorithms to build an end-to-end crack detection system. For example, we could use object detection algorithms to locate the crack region, and then use image segmentation algorithms to extract the crack. This would avoid the influence of background complexity and reduce network parameters. Furthermore, manually labeled cracks are not only time-consuming but also result in thicker edges in the detection results; therefore, unsupervised learning methods could be considered for crack detection.

[0184] The above-described embodiments are merely preferred embodiments of the present invention, and the scope of protection of the present invention is not limited thereto. Any simple changes or equivalent substitutions of the technical solutions that can be obviously obtained by those skilled in the art within the scope of the technology disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A method for pavement crack segmentation based on dense connection strategy and image gradient, characterized in that, The method comprises the following steps: constructing a pavement crack segmentation model; wherein the pavement crack segmentation model comprises: a feature encoder module, which is a resnet50 network with a five-layer structure after discarding a full connection layer, and each layer thereof is connected by using a dense connection strategy; and is used for outputting a feature map; a feature enhancement module, which is used for receiving the feature map and outputting an enhanced feature map; a feature decoder module, which comprises five inverse convolution layers and is bridged with the five-layer structure of the feature encoder module; and is used for converting the enhanced feature map into a target image; an edge refiner module, which is used for converting an image into a real edge gradient image by using a Sobel operator; an image splicing module, which is used for splicing the real edge gradient image and the target image to output a target segmentation image; the bridging manner of the feature encoder module and the feature decoder module is that the feature map output by each layer of the feature encoder module from an input end to an output end is recorded as n1-n5, and the image output by each layer of the feature decoder module from a position close to the feature enhancement module to a position far away from the feature enhancement module is recorded as b1-b5; wherein n5 is input into the feature decoder module after being enhanced by the feature enhancement module, and b5 is output after one inverse convolution processing; b5 and n4 are spliced to serve as an input of the next inverse convolution operation; and the above operations are sequentially performed; the n1 and n2 feature maps output by the feature encoder module are spliced with b4 and b3 in the feature decoder module after being processed by a SimAM attention module; and the n3 and n4 feature maps output by the feature encoder module are spliced with b2 and b1 in the feature decoder module after being processed by a Dropout; a pavement original image is input into the edge refiner module and the feature encoder module in the trained pavement crack segmentation model to output a target segmentation image.

2. The method according to claim 1, wherein, The edge refiner module comprises: The channel enhancement module for performing channel enhancement on the real edge gradient image comprises a 1*1 convolution layer, a global average pooling layer, a mixed pooling layer, a feature map splicing layer, two serial full connection layers and a feature map generation layer. The input end of the mixed pooling layer is connected with the output end of the 1*1 convolution layer. The input end of the mixed pooling layer is connected with the output end of the 1*1 convolution layer. The input end of the mixed pooling layer is connected with the output end of the global average pooling layer, The input end of the mixed pooling layer is connected with the output end of the global average pooling layer. The input end of the two serial full connection layers is connected with the output end of the feature map splicing layer, and the two serial full connection layers are used for outputting channel weights. The input end of the feature map generation layer is connected with the output end of the two serial full connection layers, and the feature map generation layer is used for multiplying the weights and the features to obtain the real edge gradient image after channel enhancement. a spatial enhancement module for spatially enhancing the channel-enhanced real edge gradient image, which includes three parallel branches, including: a convolutional layer; a convolutional layer in series a mixed pooling layer; a convolutional layer in series a mixed pooling layer.

3. The method of claim 1, wherein the method is characterized by: a loss function used for training the pavement crack segmentation model is: where L side is the loss function of the five side outputs of the decoding stage, L fuse is the loss function of the pavement crack segmentation model output, X is the original image, Y is the label, W is the network learning parameter, and w is the weight.

4. The pavement crack segmentation method based on a dense connection strategy and an image gradient according to claim 3, characterized in that: where L WCE is the improved weighted cross-entropy loss function, L Dice is the dice loss function.

5. The method of claim 4, wherein the improved weighted cross-entropy loss function is: wherein α satisfies: wherein, , and denote the edge and non-edge ground truth sets, respectively.

6. The method of claim 1, wherein the feature enhancement module comprises a deformable convolution module and an enhanced convolution module connected with an output end of the deformable convolution module; and the enhanced convolution module comprises: convolutional layer; convolutional layers in series dilated convolutional layer; convolutional layers in series dilated convolutional layers in series dilated convolutional layer; convolutional layers in series dilated convolutional layer; mixed pooling layer in series convolutional layer; the mixed pooling layer comprises three parallel pooling layers comprising: average pooling layer; average pooling layer; average pooling layer.

Images