Pavement crack detection method based on adversarial and adaptive cross-consistency training

Abstract

The present application relates to a kind of road surface crack detection methods based on adversarial and adaptive cross-consistency training, the method includes the following steps: obtaining road surface crack image data;Build segmentation network model and discriminator, wherein the segmentation network model includes main segmentation network and auxiliary segmentation network, the main segmentation network includes main encoder and main decoder;Using adversarial training strategy to train the main segmentation network and discriminator;Increase the calculation weight of high confidence prediction area at the output end of discriminator, adaptively guide cross-consistency training, train the mapping relationship of main segmentation network learning from input road surface crack image data to crack segmentation result, save the best main segmentation network model parameter in the performance of verification set;In test stage, input road surface crack image to trained main segmentation network, obtain pixel-level road surface crack detection result.Compared with prior art, the present application has the advantages of reducing manual labeling, detecting more accurately and the like.

Description

Technical Field

[0001] This invention relates to the field of semi-supervised learning technology, and in particular to a semi-supervised method for detecting pavement cracks based on adversarial and cross-consistency training. Background Technology

[0002] In the context of digital twins and smart city construction, intelligent road damage detection is a crucial link in ensuring road quality, safety, traffic capacity, and comfort. Road damage that is not detected and repaired in a timely manner not only inconveniences the driving experience but can also pose serious risks to vehicle condition, traffic safety, and even personal safety, sometimes leading to irreversible personal injury and property damage. Currently, road damage identification still relies on routine manual maintenance inspections, requiring structural engineers and certified inspectors to regularly inspect roads and record damage locations. However, this manual inspection method is often affected by factors such as vehicle speed, personnel experience, and attention, resulting in low maintenance efficiency, high operating costs, and difficulty in ensuring personnel safety. Furthermore, the results of manual inspections often lack objectivity because they rely heavily on the inspector's subjective judgment and experience. Therefore, with the rapid development of artificial intelligence technology, many internet technology companies, such as Baidu and Alibaba Cloud, are dedicated to developing intelligent road damage detection algorithms to promote the application of artificial intelligence in road inspection systems. Among these, intelligent detection of road cracks is one of the research hotspots.

[0003] Before the widespread application of deep learning technology, intelligent detection of road cracks mainly relied on a series of traditional two-dimensional image processing techniques. These techniques can be categorized into edge detection, threshold segmentation, texture analysis, wavelet analysis, and minimum path search, based on different principles. Although these traditional methods can achieve certain results in handling some simple scenarios, they are often computationally intensive, sensitive to changes in environmental factors such as lighting and weather, and lack robustness. Furthermore, due to the irregular shape of road cracks, some geometric assumptions in these classic algorithms are not always applicable, which limits the practicality of these methods to some extent.

[0004] To overcome the limitations of traditional methods, researchers have begun to employ various machine learning algorithms to detect road surface cracks, thanks to advancements in artificial intelligence. These include ensemble learning methods such as Support Vector Machines (SVM), Random Forests, Markov Random Fields, and Adaboost. In recent years, with breakthroughs in deep learning technology, Deep Convolutional Neural Networks (DCNNs) have become a research hotspot in the field of intelligent road surface crack detection due to their superior performance. Unlike traditional algorithms that segment road surface crack images to identify cracks by manually setting parameters and thresholds, DCNNs are typically trained using large amounts of manually labeled road surface data. This data-driven approach eliminates the need for manual parameter adjustments or assumptions about the shape of road surface cracks, while also exhibiting greater robustness to various environmental factors. These data-driven road surface crack detection algorithms are mainly divided into three categories: image classification networks, object detection networks, and semantic segmentation networks. Image classification networks are trained to classify images of healthy road surfaces and cracked road surfaces; object detection networks are used for instance-level road surface crack detection, typically marking crack regions and their categories in the form of bounding boxes; and semantic segmentation networks are trained to achieve pixel-level crack detection in road surface crack images.

[0005] While existing DCNN algorithms have demonstrated superior performance in road surface crack detection, most of these algorithms are supervised learning networks, meaning they require large amounts of manually labeled datasets during training. However, preparing pixel-level fine annotations is a time-consuming and labor-intensive task, especially for labeling fine cracks. With smart city construction becoming a national strategy, researchers need to address real-world challenges and explore optimization principles and methods for intelligent road surface damage recognition algorithms to reduce costs, improve efficiency, and accelerate their application in digital twins and smart city construction.

[0006] A search revealed Chinese invention patent publication number CN 116993684 A, which discloses an unsupervised method, device, and storage medium for detecting road surface cracks. The method includes the following steps: randomly removing cracks from a healthy road surface image using a multi-scale random square mask; constructing a conditional generative adversarial network (GAN) model to extract information and feature distributions from the randomly removed healthy road surface crack image, and reconstructing a new healthy road surface crack image based on the extracted feature distributions; training the network model using a pixel-wise loss function, a structural consistency loss function, a feature consistency loss function, and an adversarial loss function; inputting the road surface crack image to obtain the image reconstructed by the GAN model, comparing it with the input road surface crack image to obtain an error map; and filtering and binarizing the error map to obtain the road surface crack detection result. This existing patent suffers from insufficient detection accuracy, particularly in detecting small cracks.

[0007] How to achieve accurate detection of road surface cracks with a small number of manual markings has become a technical problem that needs to be solved. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a road surface crack detection method based on adversarial and adaptive cross-consistency training.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] According to one aspect of the present invention, a method for detecting road surface cracks based on adversarial and adaptive cross-consistency training is provided, the method comprising the following steps:

[0011] Step S1: Obtain road surface crack image data and divide it into training set and validation set;

[0012] Step S2: Construct a segmentation network model and a discriminator, wherein the segmentation network model includes a main segmentation network and an auxiliary segmentation network, and the main segmentation network includes a main encoder and a main decoder;

[0013] Step S3: Train the main segmentation network and the discriminator using an adversarial training strategy; add computational weights for high-confidence prediction regions to the output of the discriminator to adaptively guide cross-consistency training; train the main segmentation network to learn the mapping relationship from the input road surface crack image data to the crack segmentation result, and save the parameters of the main segmentation network model that performs best on the validation set.

[0014] Step S4: During the testing phase, the road surface crack image is input into the trained main segmentation network, and the parameters of the best-performing main segmentation network model from step S5 are loaded to obtain pixel-level road surface crack detection results.

[0015] Preferably, the road surface crack image data includes road surface crack image data with ground truth labels and road surface crack image data without labels.

[0016] More preferably, the main encoder uses a ResNet-50 pre-trained on ImageNet as the backbone;

[0017] The auxiliary segmentation network includes multiple auxiliary decoders;

[0018] The main segmentation network processes the road surface crack image data with ground truth labels from step S1, and then the discriminator separates the fake images from the real images.

[0019] Preferably, the discriminator is a U-Net-based discriminator network architecture, which includes an encoder and a decoder, and transmits information through hop connections;

[0020] The discriminator concatenates two pairs of images as input and simultaneously outputs global and local decision results, where local results are for each pixel and global results are for the entire image pair.

[0021] Preferably, the encoder comprises five convolutional blocks consisting of convolution operations, batch regularization operations, and ReLU activation;

[0022] The decoder contains five convolutional blocks consisting of deconvolution operations, batch regularization operations, and ReLU activation.

[0023] More preferably, the adversarial training strategy utilizes the discriminator feature matching loss function, the discriminator global loss function, and the discriminator local loss function to train the discriminator and the main segmentation network;

[0024] The discriminator feature matching loss function is defined as follows:

[0025]

[0026] Where R is the number of convolutional layers in the discriminator. For the i-th convolutional layer, The segmentation result of the main segmentation network is represented by I, where I represents the labeled input image and Y represents the ground truth labels in the dataset. This represents the expected function.

[0027] More preferably, the global loss function and the local loss function of the discriminator are defined as follows:

[0028]

[0029]

[0030] Among them, D enc The encoder represents the discriminator. and Let D(Y,I) represent the expectation function. i,j and This represents the discriminator's judgment result at pixel (i,j). This represents the segmentation result of the main segmentation network, where I represents the labeled input image and Y represents the ground truth labels in the dataset.

[0031] More preferably, the cross-consistency is the primary decoder M. d and auxiliary decoder Cross-consistency between feature mappings and segmentation results;

[0032] Using the aforementioned cross-consistency training strategy, training signals are extracted from pavement crack image data without ground truth annotations, specifically as follows:

[0033] First, input an unlabeled dataset of road surface crack images. Image I U The intermediate layer representation is obtained from the main encoder and is defined as Z. U =M e (I U );

[0034] Next, a random perturbation function p is introduced. f Where f∈[1,F], and F represents the number of perturbation functions used; a random perturbation function is used in each iteration of the cross-consistency training to obtain a perturbation version of the intermediate layer representation, denoted as... in Representative group, Z U For the intermediate layer representation, the f-th perturbation version p f The random selection is used as the input to the auxiliary decoder, and the segmentation result of the auxiliary decoder is represented as...

[0035] The random perturbation function includes characteristic noise and feature loss.

[0036] More preferably, the step of adding computational weights for high-confidence prediction regions to the output of the discriminator to adaptively guide cross-consistency training specifically involves:

[0037] An unlabeled road surface crack image is input into the main decoder. The segmentation results of the main decoder and the auxiliary decoders are input into the discriminator. The output decision results are used as the confidence distribution of these segmentation results.

[0038] The confidence distribution is used as a weight map to emphasize the high-confidence regions of the segmentation results of the main decoder and the auxiliary decoders;

[0039] The weight graphs of the main decoder and the auxiliary decoder are defined as follows:

[0040]

[0041]

[0042] in, Represents the output weight map of the main decoder. The output weight map represents the auxiliary decoder, D represents the design discriminator, and I represents the output weight map. U Z represents an unlabeled input image. U This is represented as an intermediate layer. M is the perturbation version represented by the intermediate layer. d Main decoder, As an auxiliary decoder;

[0043] The loss function used to measure the difference between the output of the main decoder and the output of the auxiliary decoder is defined as follows:

[0044]

[0045] Where K represents the number of auxiliary decoders, ⊙ represents the Hardman product, and M d Main decoder, As an auxiliary decoder, Z U This is represented as an intermediate layer. This is the perturbation version represented by the intermediate layer. Represents the output weight map of the main decoder. Represents the weight map output by the auxiliary decoder;

[0046] Auxiliary feature matching loss The feature maps generated by the intermediate convolutional layers used to compare the main decoder and the auxiliary decoder are defined as follows:

[0047]

[0048] in, Let T represent the expectation function, and T represent the number of convolutional layers in the decoder. and Z represents the i-th convolutional layer of the main decoder and the auxiliary decoder, respectively. U This is represented as an intermediate layer. This is the perturbation version represented by the intermediate layer.

[0049] Preferably, the parameters of the main segmentation network model that perform best on the validation set are specifically: if the performance on the validation set does not improve after a set number of m epochs, then training is terminated and the parameters of the main segmentation network model that perform best on the validation set are saved.

[0050] The parameters of the main segmentation network model include the weights and biases of each neuron in each layer.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1) The method of this invention combines adversarial training strategy with adaptive cross-consistency training to intelligently detect road surface cracks. It effectively solves the problem that existing deep learning-based supervised road surface crack detection algorithms rely on a large amount of detailed and complete labeled data, resulting in a large consumption of manpower and material resources. It greatly reduces the need for manual labeling, which is conducive to fully automated intelligent detection and is more suitable for smart city construction in the field of digital twins.

[0053] 2) This invention designs a discriminator based on skip connections and uses an adversarial training strategy. The discriminator feature matching loss function, the discriminator global loss function, and the discriminator local loss function are used to train the discriminator and the main segmentation network, maintaining local consistency, global image consistency, and network feature consistency, thereby enhancing the detection accuracy of the segmentation network model.

[0054] 3) This invention designs multiple auxiliary decoders and uses a cross-consistency training strategy. By constraining the consistency between the network features and segmentation results of the auxiliary decoders and the main decoder, the representation ability of the encoder is enhanced, thereby achieving the purpose of extracting additional training signals from data without ground truth labels, so as to enhance the crack segmentation performance of the main segmentation network.

[0055] 4) This invention uses the output of the designed discriminator as the weight graph. By increasing the computational weight of high-confidence prediction regions, it adaptively guides cross-consistency training, enabling the network to learn more high-confidence prediction regions. This allows for better utilization of information from unlabeled data, enhances the model's invariance to disturbances, makes the model robust to changes in data distribution after training, and improves the crack segmentation performance of the main segmentation network.

[0056] 5) Experiments were conducted on datasets such as Crack500, Deepcrack, and CFD. The experimental results show that the proposed algorithm only requires 10%-20% of the training data with ground truth labels to achieve detection performance comparable to supervised algorithms that use 100% of the training data with ground truth labels, and outperforms advanced semi-supervised segmentation algorithms and unsupervised crack segmentation algorithms. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the framework of the road surface crack detection method in this invention;

[0058] Figure 2 This is a schematic diagram of a road surface crack image used during testing in one embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of the road surface crack segmentation results output during testing in one embodiment of the present invention;

[0060] Figure 4 This is a schematic flowchart of the road surface crack detection method in this invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0062] Current supervised algorithms based on deep learning suffer from drawbacks such as requiring large amounts of manually labeled datasets and consuming significant human and material resources. This invention proposes a semi-supervised intelligent pavement crack detection method based on adversarial and adaptive cross-consistency training. The core of this method lies in using an adversarial training strategy to improve the training effect of labeled data and using an adaptive cross-consistency training strategy to extract additional training signals from unlabeled data, ultimately significantly improving the detection performance of the segmentation network model.

[0063] This embodiment relates to a road surface crack detection method based on adversarial and adaptive cross-consistency training, which includes the following steps:

[0064] Step S1, pre-training phase, involves constructing the validation set and training data. The training data is divided into two types: only 10% has ground truth labels, and only 20% has ground truth labels.

[0065] Step S2: Construct a segmentation network model and design a discriminator based on skip connections, wherein the segmentation network model includes a main segmentation network and an auxiliary segmentation network, and the main segmentation network includes a main encoder and a main decoder;

[0066] The discriminator structure consists of an encoder and a decoder; an adversarial training strategy is used to enhance the detection results of the segmentation network model by maintaining pixel-wise consistency, global image consistency, and network feature consistency.

[0067] An auxiliary decoder is designed and a cross-consistency training strategy is used. By constraining the consistency between the network features and segmentation results of the auxiliary decoder and the main decoder, the representation ability of the encoder is enhanced. This achieves the goal of extracting additional training signals from data without ground truth labels, thereby enhancing the crack segmentation performance of the main segmentation network.

[0068] Step S3 involves inputting the combined predictions from each decoder and the unlabeled image into the discriminator. The output is used as a weight map. By increasing the computational weights of high-confidence prediction regions, the cross-consistency training of the unlabeled data is adaptively guided, enabling the network to learn more high-confidence prediction regions and thus better utilize the information from the unlabeled data. This enhances the detection results of the segmentation network model and ultimately improves the crack segmentation performance of the main segmentation network. During training, a validation set is used for verification, and the parameters of the main segmentation network model that performs best on the validation set are saved for the testing phase. These parameters include the weights and biases of each neuron in each layer.

[0069] Step S4: In the testing phase, input the road surface crack image and use the trained segmentation network model to obtain pixel-level crack detection results.

[0070] In step S1, a dataset of road surface crack images is obtained. The training data is divided into two types: 10% with ground truth labels and 90% without labels, and 20% with ground truth labels and 80% without labels. These two types of training data are used as input to the main segmentation network, which learns the mapping from the input road surface crack images to the crack segmentation results. For example, for the Deepcrack dataset, if the training data consists of 300 images, it is divided into two combinations: 30 labeled images and 270 unlabeled images, and 60 labeled images and 240 unlabeled images.

[0071] In step S2, based on the PyTorch framework, this invention constructs a segmentation network model, which includes a main segmentation network and auxiliary segmentation networks. The main segmentation network is composed of a main encoder and a main decoder. We construct the main encoder based on the PSP module, using a ResNet-50 pre-trained on ImageNet as the backbone. Compared to the main decoder, this invention designs multiple auxiliary decoders in the auxiliary segmentation network. Considering efficiency and the number of network parameters, we use 1×1 convolutions to adjust the depth of the encoded feature map to the number of classes to be segmented, and use a series of sub-pixel convolutions to upsample the output to the original input size.

[0072] Unlike the discriminator framework of classic GANs used for classification networks, this invention proposes a U-Net-based discriminator network architecture, comprising an encoder and a decoder, with information passed through skip connections. The encoder contains five convolutional blocks consisting of convolutional operations, batch regularization, and ReLU activation, while the decoder contains five convolutional blocks consisting of deconvolutional operations, batch regularization, and ReLU activation. The proposed discriminator takes two pairs of images concatenated together as input and simultaneously outputs global (the entire image pair) and local (pixel-by-pixel) decisions. Specifically, during the training phase, conditioned on an input image of road surface cracks, the discriminator is trained to distinguish between fake images (results generated by the segmentation network model) and real images (labeled ground truth). Furthermore, to fully utilize information from deep features, the feature mapping differences of the discriminator's intermediate layers under different inputs are further constrained. Finally, the adversarial training strategy utilizes the discriminator feature matching loss function, the discriminator global loss function, and the discriminator local loss function to train the discriminator and the main segmentation network. By maintaining pixel-wise consistency, image global consistency, and network feature consistency, the detection results of the segmentation network model are enhanced.

[0073] The standard cross-entropy loss function used is defined as follows:

[0074]

[0075] in Indicates a labeled dataset, I i and Y i Let represent the images in the dataset and their corresponding ground truth labels, respectively. S represents the designed main segmentation network. The discriminator feature matching loss function used is defined as follows:

[0076]

[0077] Where R is the number of convolutional layers in the discriminator. For the i-th convolutional layer, The segmentation result of the main segmentation network is represented by I, where I represents the labeled input image and Y represents the ground truth labels in the dataset. This represents the expected function.

[0078] The global loss and local loss of the discriminator used are defined as follows:

[0079]

[0080]

[0081] Where D enc The encoder represents the discriminator. and Let D(Y,I) represent the expectation function.i,j and This represents the discriminator's judgment result at pixel (i,j). This represents the segmentation result of the main segmentation network, where I represents the labeled input image and Y represents the ground truth labels in the dataset.

[0082] Convolution is a mathematical concept defined as the sequential integration of an input signal by a convolution kernel (also called a filter) function. Batch regularization is a layer of computation added before the nonlinear activation function of each layer of a neural network. Its operation normalizes the input values ​​and maps them to the desired range. ReLU activation function, short for Linear Rectification Function, is a commonly used nonlinear activation function in artificial neural networks. When the input is greater than 0, this function is completely identity, equivalent to no calculation being performed. Therefore, compared to the traditional neural network activation function Sigmoid, it has advantages such as faster computation and easier backpropagation of error. Furthermore, because it splits into two discontinuous parts at the 0 position, it possesses the same nonlinear characteristics as the Sigmoid function, making it more suitable for deep feedforward neural networks.

[0083] In step S2, the deconvolution operation can be viewed as a mirror image of the convolution operation; each convolution operation has a corresponding deconvolution operation. First, the convolution kernel of the convolution operation is converted into the deconvolution kernel, which is essentially a "vertical and horizontal flip" operation relative to the original kernel. Second, the input image corresponding to the deconvolution operation is padded with zeros on both sides to create a larger image, so that when the deconvolution kernel is applied to this padded image, the output image is the same size as the input image of the convolution operation. Third, the deconvolution kernel is convolved with the padded input image to obtain the convolution result of the deconvolution kernel.

[0084] In step S3, an auxiliary decoder is designed to perform semi-supervised learning by constraining the main decoder M. d and auxiliary decoder The cross-consistency between feature maps and segmentation results is used to extract training signals from unlabeled data to train the segmentation network model, thereby enhancing the main encoder M. e The feature representation is used to improve the detection performance of the main segmentation network. The purpose of cross-consistency training is to enhance the invariance of the main segmentation network's predictions to perturbations, making the main segmentation network robust to changes in data distribution after training, and ultimately achieving better performance on test data compared to using supervised learning alone.

[0085] Input unlabeled dataset Image I U The intermediate layer representation, defined as Z, can be obtained from the main encoder. U =M e (I U ).

[0086] The input to the auxiliary decoder is a perturbed feature map, which includes perturbations such as feature noise and feature loss.

[0087] Next, we introduce a random perturbation function (including feature noise, feature loss, etc.), denoted as p. f , where f∈[1,F], and F represents the number of perturbation functions used. In each iteration of training, we use these perturbations to obtain the perturbation version of the intermediate layer representation, denoted as . in Representative group, Z U For the intermediate layer representation, the f-th perturbation version p f It is randomly selected as the input to the auxiliary decoder. Therefore, the segmentation result of the auxiliary network is represented as...

[0088] In step S4, this invention innovatively proposes using the designed discriminator to adaptively guide the training process on unlabeled data. Specifically, under the condition of inputting an unlabeled road surface crack image, the segmentation results of the main decoder and auxiliary decoders are input into the discriminator to obtain the decision results for all pixels, which serve as the confidence distribution of these segmentation results. Then, these confidence results are used as weight maps, and the Hadamard product between them and the corresponding segmentation results is calculated to emphasize the high-confidence regions of the segmentation results of the main decoder and auxiliary decoders, thereby improving the training effect. The weight maps of the main decoder and auxiliary decoder are defined as follows:

[0089] W Md =D(M d (Z U ),I U )

[0090]

[0091] in Represents the output weight map of the main decoder. The output weight map represents the auxiliary decoder, D represents the design discriminator, and I represents the output weight map. U Z represents an unlabeled input image. U This is represented as an intermediate layer. M is the perturbation version represented by the intermediate layer. d Main decoder, For auxiliary decoders.

[0092] In step S4, the loss function used to measure the difference between the output of the main decoder and the outputs of the auxiliary decoders is defined as:

[0093]

[0094] Where K represents the number of auxiliary decoders, and ⊙ represents the Hardman product.

[0095] Furthermore, we designed an auxiliary feature matching loss. To compare the feature maps generated by the intermediate convolutional layers of the main decoder and the auxiliary decoder, this can further enhance the cross-consistency of feature maps between the main decoder and the auxiliary encoder, defined as:

[0096]

[0097] in Let T represent the expectation function, and T represent the number of convolutional layers in the decoder. and These represent the i-th convolutional layer of the main decoder and the auxiliary decoder, respectively.

[0098] In step S5, training was performed for 200 epochs. Training was terminated if the performance on the validation set did not improve for 20 consecutive epochs. The model parameters that performed best on the validation set were retained, including the weights and biases of each neuron in each layer. This was done using the `torch.save()` function in the PyTorch framework. During training, data augmentation techniques, including scaling and flipping, were applied. Stochastic gradient descent (SGD) was used to optimize the network with a momentum value of 0.9 and a weight decay of 10. -4 The initial learning rate was set to 0.01 and dynamically adjusted using a poly strategy. In actual training, we used seven perturbation functions and set the number of auxiliary decoders to three. A single forward and backward propagation computation using the entire training dataset to update the model weights is considered a complete epoch.

[0099] In step S6, during the testing phase, road surface crack images from the Crack500, Deepcrack, and CFD test sets are input. Pre-trained main segmentation network model parameters are loaded, and the segmentation network model (main decoder, main encoder) outputs pixel-level crack detection results. Crack500, Deepcrack, and CFD all represent road surface crack images. The quantitative evaluation metrics used include precision, recall, accuracy, IoU, and F1-Score.

[0100] Testing was conducted on three datasets: Crack500, Deepcrack, and CFD. Crack500 consisted of 500 2000×1500 pixel images of road surface cracks, collected using smartphones at Temple University's main campus and manually labeled with pixel-level tags. Each image was cropped into 16 non-overlapping regions, retaining only regions containing crack images larger than 1000 pixels. Therefore, Crack500 contained 1896 training images, 348 validation images, and 1124 test images. Notably, the images in Crack500 present challenges for real-world crack segmentation due to factors such as shadows, occlusion, uneven lighting conditions, and noise.

[0101] The DeepCrack dataset contains 537 images of concrete surfaces, each 544×384 pixels, featuring cracks at multiple scales and in various scenes. All images were manually annotated with pixel-level labels and were divided into two main subsets: 300 images for training and 237 images for testing.

[0102] The CFD dataset contains 118 images of concrete surface cracks, with manually annotated pixel-level labels. All images are 480×320 pixels in size. Because these images contain various lighting conditions, shadows, stains, and lane lines, it is difficult to detect the cracks. They are divided into two sets: 70 images for training and 48 images for testing.

[0103] This invention proposes an intelligent road surface crack detection method based on adversarial and adaptive cross-consistency training. Based on semi-supervised learning, it can effectively alleviate the problems of existing supervised road surface crack detection algorithms based on deep learning (training relies entirely on detailed and complete labeled data, and these pixel-level detailed labels require a lot of manpower and resources). It helps to achieve fully automated intelligent detection and is more suitable for smart city construction in the field of digital twins. During the training phase, the developed algorithm first designed a discriminator based on skip connections, with an encoder-decoder structure. An adversarial training strategy was employed to enhance the training performance of the segmentation network model on labeled data by maintaining pixel-wise consistency, global image consistency, and network feature consistency. Then, multiple auxiliary decoders were designed, using a cross-consistency training strategy. This strategy enhanced the encoder's representational ability by constraining the consistency of the auxiliary decoders' network features and segmentation results with the main decoder, thereby extracting additional training signals from unlabeled data. Next, the prediction results of each decoder were combined with the unlabeled image and input into the discriminator. The output was used as a weight map. By increasing the computational weight of high-confidence prediction regions, the cross-consistency training on unlabeled data was adaptively guided, allowing the network to learn more high-confidence prediction regions and further improving the crack segmentation performance of the main segmentation network. During the testing phase, the model parameters that performed best on the saved validation set were loaded, and a road crack image was input. The pre-trained segmentation network model was then used to obtain pixel-level crack detection results.

[0104] Figure 1 A flowchart illustrating the intelligent detection method for road surface cracks is presented. Figure 2 and Figure 3 Examples of road surface crack images and corresponding segmentation results used in the tests are shown. Experiments were conducted on datasets such as Crack500, Deepcrack, and CFD. Experimental results show that the proposed algorithm requires only 10%–20% ground-labeled training data to achieve detection performance comparable to supervised algorithms using 100% ground-labeled training data, and outperforms advanced semi-supervised segmentation algorithms and unsupervised crack segmentation algorithms. Table 1 shows the quantitative comparison results of the SP-CrackNet method of this invention with other algorithms on the Crack500 dataset; Table 2 shows the quantitative comparison results of the method of this invention with other algorithms on the DeepCrack dataset.

[0105] Table 1

[0106]

[0107] Table 2

[0108]

[0109] This embodiment also relates to the application of a road surface crack detection algorithm.

[0110] In the construction of smart cities oriented towards digital twins, the proposed method can serve as an intelligent detection tool for road damage. Combined with inspection robots and sensing devices such as cameras, it can execute a novel semi-supervised intelligent road crack detection algorithm based on adversarial and adaptive cross-consistency training, and accumulate road damage reports. Compared to existing supervised detection algorithms, this method significantly reduces the need for manual labor, saving time and budget, and truly achieving fully automated intelligent detection. Furthermore, after establishing a comprehensive and scientific road damage database, it enables intelligent data sharing among road construction, management, and maintenance systems, further establishing a road damage prediction model to make road damage evolution "measurable." This allows for the development of reasonable maintenance plans, early prediction of maintenance, and reduction of maintenance cycles.

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for road crack detection based on adversarial and adaptive cross-consistency training, characterized in that, The method includes the following steps: Step S1: Obtain road surface crack image data and divide it into training set and validation set; Step S2: Construct a segmentation network model and a discriminator, wherein the segmentation network model includes a main segmentation network and an auxiliary segmentation network, the main segmentation network includes a main encoder and a main decoder; the main encoder uses a ResNet-50 pre-trained on ImageNet as the backbone; the auxiliary segmentation network includes multiple auxiliary decoders; the main segmentation network processes the ground truth labeled road surface crack image data from step S1, and then the discriminator separates the fake images and real images; Step S3, training the main segmentation network and the discriminator using an adversarial training strategy; adding a calculation weight of high-confidence prediction regions at the output end of the discriminator to adaptively guide cross-consistency training; training the main segmentation network to learn a mapping relationship from input road crack image data to crack segmentation results, and saving the main segmentation network model parameters with the best performance on the validation set; the cross-consistency is cross-consistency between feature mappings and segmentation results of the main decoder and the auxiliary decoder ; using the cross-consistency training strategy to extract training signals from road crack image data without true value labels, specifically: First, input an unlabeled dataset of road surface crack images. Images in The main encoder obtains the intermediate layer representation, defined as follows: ; Next, random perturbation functions are introduced where , denotes the number of perturbation functions used; at each iteration of the cross-consistency training, a perturbed version of the intermediate layer representation is obtained using a random perturbation function, denoted as where represents the combination, is the intermediate layer representation, the th perturbed version is randomly selected as the input of the auxiliary decoder, and the segmentation result of the auxiliary decoder is denoted as ; the random perturbation function contains feature noise and feature loss. Step S4: During the testing phase, the road surface crack image is input into the trained main segmentation network, and the parameters of the best-performing main segmentation network model from step S3 are loaded to obtain pixel-level road surface crack detection results.

2. The road surface crack detection method based on adversarial and adaptive cross-consistency training according to claim 1, characterized in that, The road surface crack image data includes road surface crack image data with ground truth labels and road surface crack image data without labels.

3. The road surface crack detection method based on adversarial and adaptive cross-consistency training according to claim 1, characterized in that, The discriminator described is a U-Net-based discriminator network architecture, which includes an encoder and a decoder, and transmits information through skip connections; The discriminator concatenates two pairs of images as input and simultaneously outputs global and local decision results, where local results are for each pixel and global results are for the entire image pair.

4. The road surface crack detection method based on adversarial and adaptive cross-consistency training according to claim 3, characterized in that, The encoder contains five convolutional blocks consisting of convolution operations, batch regularization operations, and ReLU activation. The decoder contains five convolutional blocks consisting of deconvolution operations, batch regularization operations, and ReLU activation.

5. The road surface crack detection method based on adversarial and adaptive cross-consistency training according to claim 3, characterized in that, The adversarial training strategy described above uses the discriminator feature matching loss function, the discriminator global loss function, and the discriminator local loss function to train the discriminator and the main segmentation network. The discriminator feature matching loss function is defined as follows: in The number of convolutional layers in the discriminator. For the i-th convolutional layer, This represents the segmentation result of the main segmentation network, where I represents the labeled input image and Y represents the ground truth labels in the dataset. This represents the expected function.

6. The road surface crack detection method based on adversarial and adaptive cross-consistency training according to claim 5, characterized in that, The global loss function and local loss function of the discriminator are defined as follows: in, The encoder represents the discriminator. and Represents the expectation function, and This indicates that the discriminator is at the pixel point. The judgment result above, This represents the segmentation result of the main segmentation network, where I represents the labeled input image and Y represents the ground truth labels in the dataset.

7. The road surface crack detection method based on adversarial and adaptive cross-consistency training according to claim 2, characterized in that, The specific steps of adding computational weights for high-confidence prediction regions to the output of the discriminator to adaptively guide cross-consistency training are as follows: An unlabeled road surface crack image is input into the main decoder. The segmentation results of the main decoder and the auxiliary decoders are input into the discriminator. The output decision results are used as the confidence distribution of these segmentation results. The confidence distribution is used as a weight map to emphasize the high-confidence regions of the segmentation results of the main decoder and the auxiliary decoders; The weight graphs of the main decoder and the auxiliary decoder are defined as follows: in, Represents the output weight map of the main decoder. This represents the weight map output by the auxiliary decoder. Representative design discriminator, This indicates that the input is an unlabeled image of road surface cracks. This is represented as an intermediate layer. This is the perturbation version represented by the intermediate layer. Main decoder, As an auxiliary decoder; The loss function used to measure the difference between the output of the main decoder and the output of the auxiliary decoder is defined as follows: in, Represents the number of auxiliary decoders. Represents the Hartmann product. Main decoder, As an auxiliary decoder, This is represented as an intermediate layer. This is the perturbation version represented by the intermediate layer. Represents the output weight map of the main decoder. Represents the output weight map of the auxiliary decoder; Auxiliary feature matching loss The feature maps generated by the intermediate convolutional layers of the main decoder and the auxiliary decoder are defined as follows: in, Represents the expected function, This represents the number of convolutional layers in the decoder. and These represent the i-th convolutional layers of the main decoder and the auxiliary decoder, respectively. This is represented as an intermediate layer. This is the perturbation version represented by the intermediate layer.

8. The road surface crack detection method based on adversarial and adaptive cross-consistency training according to claim 1, characterized in that, The parameters of the main segmentation network model that perform best on the validation set are specifically as follows: if the performance on the validation set does not improve after a set number of m epochs, then training is terminated and the parameters of the main segmentation network model that performs best on the validation set are saved. The parameters of the main segmentation network model include the weights and biases of each neuron in each layer.

Images