Lightweight real-time pavement crack intelligent detection method

Abstract

The invention relates to the field of image processing, in particular to a lightweight real-time pavement crack intelligent detection method, which comprises the following steps of: standardizing an acquired pavement image and converting the standardized pavement image into a tensor; the training set data is used to train a lightweight generator network, the lightweight generator network comprises a U-shaped main segmentation network and an adaptive boundary extraction module, and the U-shaped main segmentation network comprises a first encoder, a fast vision converter and a first decoder; the adaptive boundary extraction module is used for dynamically capturing boundary features; inputting the output result of the lightweight generator network into a self-attention discriminator network, training the self-attention discriminator network from global and local levels to supervise the output result of the lightweight generator network, and storing model parameters with the best effect; and inputting the tensor image into the trained U-shaped main segmentation network, and outputting a pavement crack detection result in real time. Compared with the prior art, the method has the advantages of realizing real-time and accurate pavement crack detection and the like.

Description

Technical Field

[0001] The present invention relates to the field of image processing and artificial intelligence technology, and in particular to a lightweight real-time intelligent detection method for road cracks. Background Art

[0002] In the context of the development of digital twin technology and smart cities, intelligent detection of road damage has become an important part of ensuring road quality, safety, traffic efficiency and driving comfort. If road damage is not identified and repaired in a timely manner, it will not only affect the driving experience, but may also pose a serious threat to vehicle performance, traffic safety and even life and property safety, and even in some cases lead to irreversible personal injury and economic losses. At present, the detection of road damage still mainly relies on manual inspections, that is, structural engineers and certified inspectors conduct regular on-site inspections and record the specific location of the damage. However, this traditional manual inspection method is easily affected by factors such as driving speed, experience level of inspectors and concentration, resulting in low detection efficiency, high operating costs, and difficulty in effectively ensuring the safety of inspectors. In addition, since manual inspection is highly dependent on the subjective judgment and experience accumulation of inspectors, its results often lack objectivity and consistency. Therefore, with the rapid development of artificial intelligence technology, some technology companies are actively developing road damage detection technology based on intelligent algorithms to promote the application of artificial intelligence in the field of road inspection. Among them, intelligent detection of pavement cracks based on deep learning has become one of the research hotspots in this field.

[0003] Before deep learning technology was widely used, intelligent detection of pavement cracks mainly relied on the traditional two-dimensional image analysis and processing system. Its method framework can be summarized into typical paradigms such as feature extraction (including gradient feature extraction, grayscale threshold segmentation), texture feature analysis, frequency domain analysis, and minimum path search algorithm. Although such algorithms are applicable in specific scenarios (such as linear crack recognition under simple backgrounds), they generally have technical defects such as large amount of calculation, sensitivity to changes in environmental factors such as lighting and weather, and lack of robustness in actual engineering applications. In addition, since the shape of pavement cracks is not regular, the hypothetical models used by traditional image processing methods often have significant deviations from the actual morphological characteristics of cracks, which fundamentally restricts the engineering application value of traditional algorithms.

[0004] In the process of breaking through the bottleneck of traditional detection methods, the rapid evolution of artificial intelligence technology has prompted researchers to introduce multivariate machine learning solutions for road damage identification, such as integrated algorithms such as support vector machine (SVM), random forest (RandomForest), Markov random field (Markov Random Field) and Adaboost. The paradigm innovation of deep learning has promoted the innovative application of convolutional deep neural network (DCNN) in road damage detection. With its end-to-end feature learning ability, it has become the core breakthrough direction in this field. Compared with traditional image segmentation methods that rely on manual parameter adjustment and threshold setting, DCNN realizes the automation of parameter optimization process through the adaptive feature extraction mechanism of massive labeled data, gets rid of the prior restrictions on crack geometric features, and shows better environmental adaptability in complex scenes such as lighting changes and stain interference. From the perspective of technical implementation, the data-driven road damage detection system mainly covers three branch architectures: image classification model based on global feature discrimination, target detection framework focusing on local damage positioning, and semantic segmentation network for pixel-level analysis. The image classification model focuses on the macroscopic discrimination between sound and damaged roads; the object detection system realizes instance recognition of multi-category cracks through annotation boxes; and the semantic segmentation technology accurately depicts the morphological characteristics of cracks through pixel-level mask generation.

[0005] Although existing deep learning-based intelligent pavement crack detection algorithms, especially semantic segmentation algorithms, have demonstrated excellent performance, all publicly available algorithms in the field cannot achieve a balance between detection performance, model size, and model processing speed, which limits the further deployment of these algorithms on automatic road inspection vehicles / robots. In the context of the accelerated digitalization of smart cities, researchers need to face practical challenges, explore the optimization principles and methods of algorithms for the problems encountered in the practical application of intelligent pavement damage identification algorithms, and develop intelligent pavement damage detection algorithms that are more suitable for automatic inspection vehicles and robots to accelerate the application of these algorithms in the fields of digital twins and smart city construction.

[0006] After searching, the Chinese invention patent application publication number CN116758507A discloses a road surface quality analysis method, device and program based on disease image acquisition and segmentation, including the following steps: the car starts; the car wheel drives the wheel speed encoder, triggers the industrial camera to collect road surface image data, and at the same time, the satellite positioning navigation system obtains the latitude and longitude positioning information of the car, and then the road surface image data is matched with the positioning information one by one; the industrial computer in the car uses the large convolution U-shaped network structure of the road surface disease segmentation model to perform real-time road surface disease detection on the road surface image; the road surface disease segmentation model is trained using a three-stage training method; the large convolution kernel U-shaped network structure is deployed and inferred using the re-parameterization technology to obtain the overall road surface defect detection result; the road surface condition index PCI is calculated based on the detection result; the road surface analysis report is automatically generated to provide road surface maintenance suggestions. This existing patent application has the problems of large model scale, too high requirements on the hardware configuration of the inspection vehicle, and slow detection speed.

[0007] How to achieve real-time and accurate detection of road cracks based on a lightweight model has become a technical problem that needs to be solved. Summary of the invention

[0008] The purpose of the present invention is to provide a lightweight real-time intelligent detection method for road cracks in order to overcome the defects of the above-mentioned prior art.

[0009] The purpose of the present invention can be achieved by the following technical solutions: According to one aspect of the present invention, a lightweight real-time intelligent detection method for pavement cracks is provided, the method comprising the following steps: Step 1: Standardize the acquired road surface image and convert it into a tensor; Step 2: In the training phase, the lightweight generator network is trained with the training set data, wherein the lightweight generator network includes a U-shaped main segmentation network and an adaptive boundary extraction module, wherein the U-shaped main segmentation network includes a first encoder for extracting local features and improving the network inference speed, a fast visual transformer for extracting global features, and a first decoder; the adaptive boundary extraction module is used to dynamically capture the boundary features of the training set data; Step 3: In the training phase, the output of the lightweight generator network is input into the self-attention discriminator network, and the self-attention discriminator network is trained from both global and local levels to supervise the output of the lightweight generator network, and the model parameters with the best effect on the validation set are saved; Step 4: Input the processed result of step 1 into the trained U-shaped main segmentation network, load the saved model parameters, and output the pixel-level pavement crack detection result in real time; The first encoder includes two residual convolution double-layer down-sampling modules and two convolution double-layer down-sampling modules, and the first decoder includes two residual convolution double-layer up-sampling modules and two convolution double-layer up-sampling modules.

[0010] Preferably, each of the residual convolution double-layer downsampling modules comprises a first maximum pooling layer, a first depth-separable convolution layer, a first batch of regularization operation layers, a first ReLU activation layer, a second depth-separable convolution layer, a second batch of regularization operation layers and a second ReLU activation layer, wherein the output of the first depth-separable convolution layer is cross-layer connected to the output of the second depth-separable convolution layer; The structure of each of the convolutional double-layer downsampling modules is similar to that of the residual convolutional double-layer downsampling module, wherein the convolutional layer adopts a standard convolutional layer.

[0011] Preferably, each of the residual convolution double-layer upsampling modules comprises a first bilinear upsampling layer, a third depthwise separable convolution layer, a fifth batch regularization operation layer, a fifth ReLU activation layer, a fourth depthwise separable convolution layer, a sixth batch regularization operation layer and a sixth ReLU activation layer, wherein the output of the third depthwise separable convolution layer is cross-layer connected to the output of the fourth depthwise separable convolution layer; The structure of each of the convolutional double-layer upsampling modules is similar to that of the residual convolutional double-layer upsampling module, wherein the convolutional layer adopts a standard convolutional layer.

[0012] Preferably, the fast visual transformer comprises a spatial attention layer, a standard convolution layer, a spatial attention layer, a channel attention layer, a standard convolution layer and a random inactivation layer connected in sequence.

[0013] Preferably, the adaptive boundary extraction module includes a boundary extraction network submodule, and the boundary extraction network submodule is trained and constrained so that the extracted boundary information is consistent with the boundary truth value. Figure 1 To.

[0014] Preferably, the adaptive boundary extraction module adopts an adaptive erosion and expansion iterative algorithm to obtain the boundary truth map, and the specific method includes: First, the first channel image matrix of the input road surface image tensor data is extracted and the data type is standardized; If the core radius is not initialized, the adaptive erosion iteration is performed to calculate the adaptive radius value; Based on the adaptive radius value, the kernel radius is calculated using an empirical coefficient : , Among them, round(x) is a rounding function that returns the integer closest to x; max(a,b) is to take the larger value of a and b; is the adaptive radius value; A diamond kernel is generated based on the kernel radius and Manhattan distance principle, and then the diamond kernel is used to dilate the original image matrix. The dilation result is differentially calculated with the original image matrix to obtain the boundary truth map corresponding to each image.

[0015] More preferably, if the kernel radius has been initialized, a diamond kernel is generated directly based on the final kernel radius and the Manhattan distance principle, and then the diamond kernel is used to perform an expansion operation on the original image matrix, and the expansion result is differentiated from the original image matrix to obtain a boundary truth map corresponding to each image.

[0016] More preferably, the process of performing adaptive erosion iteration to calculate the adaptive radius value includes: Initialization process: Initialize the current image matrix to be a copy of the original image, set the initial value of the adaptive radius to 0, and create an elliptical erosion kernel; Iterative erosion operation: an elliptical erosion kernel is used to iteratively erode the image matrix, and the radius value is adaptive after each iteration. Increment by 1 until the image matrix is ​​completely zeroed and the iteration is terminated.

[0017] Preferably, the self-attention discriminator network is a U-shaped structure, comprising a second encoder and a second decoder, and transmits information through a jump connection; The second encoder is composed of four self-attention convolutional downsampling modules, each of which includes an efficient multi-scale self-attention layer, a second batch normalization layer, and a tenth ReLU activation layer connected in sequence; The second decoder is composed of four convolutional downsampling modules, each of which includes a deconvolution layer, a third batch normalization layer and an eleventh ReLU activation layer.

[0018] More preferably, the input of the self-attention discriminator network includes a fake image pair consisting of an output of the lightweight generator network and an input road image, and a real image pair consisting of an image truth value and an input road image, the self-attention discriminator network is trained by a discriminator local loss and a discriminator global loss, and pixel-by-pixel local consistency and image global consistency are constrained by adversarial training with the lightweight generator network; The efficient multi-scale self-attention layer is based on cross-space learning, which reshapes part of the channel dimensions into batch dimensions, builds a parallel network, designs local cross-channel interactions, and uses a cross-space learning method to fuse the output feature maps of the parallel networks.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) The lightweight generator network of the present invention extracts global features and local features through a U-shaped main segmentation network, dynamically captures boundary features through an adaptive boundary extraction module, and combines the lightweight generator network to supervise the output results of the lightweight generator network from both global and local levels. It has the characteristics of lightweight, high performance, and high real-time performance, which effectively overcomes the difficulty of the existing technology in balancing detection performance, model scale, and model processing speed, making it suitable for on-board deployment of intelligent inspection vehicles, and realizing real-time and accurate fully automated intelligent road crack detection.

[0020] (2) The U-shaped main segmentation network of the present invention includes a residual convolution double-layer downsampling module and a residual convolution double-layer upsampling module. While extracting local features of the input image based on the convolutional neural network, the double-layer sampling further improves the network's reasoning speed; the fast visual transformer can also improve the network's reasoning speed while extracting global features of the input image. The lightweight network further improves the real-time and detection performance of road cracks.

[0021] (3) The adaptive boundary extraction module of the present invention realizes the dynamic capture of crack contour features through the adaptive corrosion and expansion iterative algorithm and the boundary extraction network sub-module, enhances the learning ability of the U-shaped main segmentation network for crack boundary features, can effectively deal with cracks of different shapes and thicknesses, and improves the accuracy and robustness of network detection.

[0022] (4) The present invention designs a self-attention discriminator network, adopts a self-attention convolutional downsampling module to enhance the feature extraction capability of the input image pairs, so as to enhance the discriminative ability of the discriminator network, and designs local discriminative loss and global discriminative loss, thereby realizing high-dimensional supervision of the lightweight generator network composed of a U-shaped main segmentation network and an adaptive boundary extraction module, and further improving the crack detection performance of the main segmentation network.

[0023] (5) The present invention conducts experiments on the public datasets UDTIRI-Crack and Deepcrack. The experimental results show that compared with the existing public crack detection algorithms, the present method achieves the best in terms of detection performance and inference speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the process framework of the intelligent detection method for pavement cracks in the present invention; Figure 2 It is a schematic diagram of the process of the adaptive boundary extraction algorithm in the present invention; Figure 3 A road surface crack image sample used in testing the road surface crack intelligent detection method of the present invention; Figure 4This is a sample of the pavement crack segmentation results tested by the pavement crack intelligent detection method in the present invention; Figure 5 It is a schematic diagram of the process of the intelligent detection method of pavement cracks in the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] The purpose of the present invention is to overcome the defects of the above-mentioned existing deep learning-based pavement crack detection algorithm (inability to balance detection performance, model scale and model processing speed), and propose a lightweight and efficient new real-time pavement crack intelligent detection algorithm suitable for vehicle-mounted deployment.

[0027] Example 1 The present embodiment relates to a lightweight real-time intelligent detection method for road cracks, the core of which is to design a lightweight generator network and a self-attention discriminator network. Among them, the lightweight generator network includes a U-shaped main segmentation network and an adaptive boundary extraction module. The U-shaped main segmentation network includes a new residual convolution double-layer downsampling module and a residual convolution double-layer upsampling module, which are used to extract local features and improve the network reasoning speed. The adaptive boundary extraction module adopts an adaptive corrosion and expansion iterative algorithm to realize the dynamic capture of crack contour features. The self-attention discriminator network includes a new self-attention convolution downsampling module, which is used to enhance the network's feature extraction ability for input image pairs. The lightweight generator network is trained through an adversarial training strategy, and finally a road crack segmentation network with high detection performance, lightweight scale and fast reasoning speed is obtained.

[0028] like Figure 1 and Figure 5 , the method comprises the following steps: Step S1, standardize the training data and convert it into a tensor as the input of the U-shaped main segmentation network; Step S2: In the training phase, a new U-shaped main segmentation network is designed, and a new residual convolution double-layer downsampling module and a residual convolution double-layer upsampling module are proposed. While extracting local features of the input image based on the convolutional neural network, the network's reasoning speed is improved; at the same time, a new fast visual transformer is proposed. While extracting global features of the input image based on the visual transformer, the network's reasoning speed is improved; Step S3: During the training phase, an adaptive boundary extraction module is designed, and an adaptive corrosion and expansion iterative algorithm and a new boundary extraction network sub-module are proposed to dynamically capture crack contour features (i.e., boundary features), enhance the segmentation model's learning ability for crack boundary features, and effectively cope with cracks of different shapes and thicknesses, achieve better detection results for cracks with diverse shapes and thicknesses, and improve the robustness of crack detection.

[0029] like Figure 2 As shown in the figure, the input of the adaptive boundary extraction module is the true image value of the training set. After the true image value is preprocessed, the dynamic structure element parameters are calculated by dilation, and the kernel radius is obtained according to the empirical coefficient. According to the obtained kernel radius, a diamond kernel is constructed, and the diamond kernel is used to dilate the input image true value. The dilation operation result is then differentially calculated with the input image to extract boundary features and reconstruct the output boundary true value map. At the same time, a new boundary extraction network submodule is designed inside the model, and the boundary information extracted by the training constraint new boundary extraction network submodule is consistent with the boundary true value. Figure 1 This can enhance the model’s ability to learn crack boundary features.

[0030] Step S4: A self-attention discriminator network is designed, and a new self-attention convolutional downsampling module is proposed to enhance the feature extraction capability of the input image pairs, so as to enhance the discriminative ability of the discriminator network. Based on the adversarial training strategy, by constraining the pixel-by-pixel local consistency and the image global consistency, high-dimensional supervision of the generator network composed of the U-shaped main segmentation network and the adaptive boundary extraction module is achieved from both local and global supervision, further improving the crack detection performance of the main segmentation network.

[0031] Specifically, the output of the main segmentation network and the annotations corresponding to the input image are used as image pairs with the input image as the input of the self-attention discriminator network. The designed discriminator network also adopts a U-shaped structure, with the encoder part consisting of four self-attention convolutional downsampling modules and the decoder part consisting of four convolutional upsampling modules.

[0032] Step S5: training the U-shaped main segmentation network, the adaptive boundary extraction module and the self-attention discriminator network, using the validation set for validation during the training process, and saving the model parameters with the best effect on the validation set for use in the test phase, including the weights and bias values ​​of each neuron in each layer; Step S6: In the test phase, a road crack image is input, and only the trained U-shaped main segmentation network is used to obtain a pixel-level road crack detection result; In the construction of smart cities for digital twins, the method proposed in this invention application can be used as a new type of intelligent road damage detection tool. Its lightweight, high-performance, and high-real-time characteristics make it suitable for deployment on intelligent inspection vehicles, which in turn helps to accumulate back-end reports on road damage and realize intelligent data sharing between road construction, management, and maintenance systems. Compared with existing deep learning-based road crack detection algorithms, this algorithm achieves a balance between detection performance, model scale, and model processing speed.

[0033] In step S1, the pre-training stage, the training data is standardized and converted into tensors as the input of the subsequent network; the training generator network learns the mapping from the input road crack map to the crack segmentation result. For the Deepcrack dataset, since it has no validation set, the original training set is randomly divided into a training set and a validation set in a ratio of 7:3. For example, for Deepcrack, the training data is 300, and 210 of them are randomly selected as the training set, and the remaining 90 are used as the validation set.

[0034] In step S2, the main segmentation network is a U-shaped network built based on the pytorch framework, including a first encoder and a first decoder, wherein the first encoder includes: two residual convolutional double-layer downsampling modules and two convolutional double-layer downsampling modules.

[0035] Each residual convolution double-layer downsampling module includes a first maximum pooling layer, a first depth-separable convolution layer, a first batch of regularization operation layers, a first ReLU activation layer, a second depth-separable convolution layer, a second batch of regularization operation layers, and a second ReLU activation layer, wherein a cross-layer connection structure from the output of the first depth-separable convolution layer to the output of the second depth-separable convolution layer is designed; Each convolutional double-layer downsampling module includes a second maximum pooling layer, a first standard convolution layer, a third batch normalization operation layer, a third ReLU activation layer, a second standard convolution layer, a fourth batch normalization operation layer and a fourth ReLU activation layer.

[0036] The first decoder includes: two residual convolutional double-layer upsampling modules and two convolutional double-layer upsampling modules.

[0037] Each residual convolution double-layer upsampling module includes a first bilinear upsampling layer, a third depth-separable convolution layer, a fifth batch regularization operation layer, a fifth ReLU activation layer, a fourth depth-separable convolution layer, a sixth batch regularization operation layer and a sixth ReLU activation layer, wherein a cross-layer connection structure from the output of the third depth-separable convolution layer to the output of the fourth depth-separable convolution layer is designed; Each convolutional double-layer upsampling module includes a second bilinear upsampling layer, a third standard convolution layer, a seventh batch regularization operation layer, a seventh ReLU activation layer, a fourth standard convolution layer, an eighth batch regularization operation layer, and an eighth ReLU activation layer.

[0038] The first encoder and the first decoder are connected by a fast visual transformer module to further extract the global features of the input image and improve the reasoning speed of the network. The fast visual transformer module includes a spatial attention layer, a standard convolution layer, a spatial attention layer, a channel attention layer, a standard convolution layer, and a random dropout layer connected in sequence.

[0039] Through the designed residual convolution double-layer downsampling module, residual convolution double-layer upsampling module, and fast visual transformer, the main segmentation network is able to extract the global and local features of the input image, thereby achieving high detection performance and maintaining high inference speed.

[0040] The segmentation loss function of the main segmentation network used in this step is defined as follows: , in, Represents BCE loss function (Binary Cross-Entropy Loss, binary cross entropy loss function), Represents the Dice loss function, α and β are weight coefficients, which are used to balance the importance of the two losses. Among them, the BCE loss function is used to improve the accuracy of single pixel classification, and the Dice loss function is used to optimize the overall overlap between the predicted area and the true label. The two can be used together to improve the model robustness and segmentation accuracy.

[0041] Specifically, the formula of the BCE loss function is as follows: , in, represents the crack dataset; Respectively represent the images in the dataset and the corresponding image truth values; S represents the generator network; is the Sigmoid function, which converts the output of the network into a probability value; log is the natural logarithm.

[0042] The Dice loss function is as follows: , in, represents the crack dataset; Represents the images in the dataset and the corresponding true values ​​of the images; S represents the generator network; is a smoothing term used to avoid the denominator being zero.

[0043] In addition, during the training phase, we designed a self-attention discriminator network and used an adversarial training strategy. For the lightweight generator network consisting of the main segmentation network and the adaptive boundary extraction module, the generator loss function is It can be written as follows: , , , in, represents the encoder of the self-attention discriminator; Represents the judgment result of the discriminator at the pixel point (i, j); Represents the output of the lightweight generator network; Represents the global loss function of the generator, which is used to constrain the lightweight generator network to output detection results equivalent to the true value at the global level; represents the local loss function of the generator, which is used to constrain the lightweight generator network to output detection results equivalent to the true value at the local pixel level; γ represents the weight coefficient; Represents the expected function.

[0044] In step S2, convolution is a mathematical concept, which is defined as the serialized integral operation of a convolution kernel (also called filter) function on the input signal. Depthwise separable convolution is an optimized convolution operation, which is widely used in lightweight neural networks. It decomposes the standard convolution into two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs convolution operations on each input channel independently without mixing between channels; pointwise convolution performs linear combinations of the results of depthwise convolution across channels through 1x1 convolution. This method significantly reduces the amount of calculation and the number of parameters while maintaining model performance. Batch normalization technology is a layer of operation nodes added before each layer of nonlinear activation function of the neural network. The operation performed is to normalize the input value and then map it to the range we want. The full name of ReLU activation function is Linear Rectification Function, also known as Rectified Linear Unit, which is a commonly used nonlinear activation function in artificial neural networks. This function is completely an identity function when the input is greater than 0, which is equivalent to not completing any calculation. Therefore, compared with the traditional neural network activation function Sigmoid, it has excellent characteristics such as fast calculation and convenient reverse error propagation. At the same time, because it is divided into two discontinuous parts at the position of 0, it has the same nonlinear characteristics as the Sigmoid function, so it is more suitable for deep feedforward neural networks. The spatial attention mechanism aims to enhance the model's ability to pay attention to important areas in the image. It dynamically adjusts the importance of different positions by weighting the spatial dimensions (width and height) of the feature map. The specific implementation is usually to extract channel information through global average pooling and global maximum pooling, and then generate a spatial attention map, which is multiplied with the original feature map to highlight the key areas and suppress irrelevant areas. The channel attention mechanism focuses on adjusting the importance of different channels in the feature map. It learns the correlation between channels by analyzing global information and generates a channel attention weight vector. The mechanism usually includes two steps: feature squeezing, obtaining channel descriptors through global pooling operations; feature excitation, using fully connected layers or convolutional layers to model inter-channel dependencies and generate attention weights. Ultimately, these weights are used to recalibrate channel features and improve the model's ability to capture key information. The core component of the Visual Transformer is the self-attention mechanism, which allows the model to pay attention to the information of all other positions in the sequence when processing each position of the input sequence. This mechanism generates a new representation by calculating the relevance of each element in the input sequence to other elements (attention weights) and weighted summing the inputs according to these weights. In order to capture different features and patterns, the Fast Visual Transformer uses a multi-head attention mechanism. It divides the input into multiple subspaces and applies the self-attention mechanism independently in each subspace. The outputs of these subspaces are then concatenated and linearly transformed to obtain the final attention output.This approach enhances the expressiveness of the model.

[0045] In step S3, in order to improve the segmentation performance of the generator, the present application designs an adaptive boundary extraction module so that the model pays more attention to boundary features after training, thereby improving the detection performance. The execution of this module is divided into two steps. First, before starting training, the boundary of the image truth value in the training set is extracted by the designed adaptive corrosion and expansion iterative algorithm to obtain the boundary truth map corresponding to each image in the training set; second, a new boundary extraction network submodule is designed to extract the boundary information of the input image during training. On the basis of these two steps, the present invention designs a boundary-aware loss function, trains the boundary extraction effect of the boundary extraction network submodule, and fuses its output boundary information into the main segmentation network to enhance the detection performance of the main segmentation network.

[0046] In step S3, specifically, the designed adaptive corrosion and expansion iterative algorithm is as follows: Figure 2 As shown, it includes: obtaining the input image tensor data and the kernel radius, extracting the first channel of the input image tensor data (i.e. Figure 2 The first channel in the image matrix and normalize its data type (refer to Figure 2 in to an unsigned 8-bit integer matrix) to meet the requirements of subsequent morphological operations, expressed as .

[0047] 1) If the kernel radius has been initialized (i.e., the kernel radius required for the specified dilation operation is entered When , a diamond kernel is generated based on the Manhattan distance principle, and its principle can be expressed as: , in, r is the radius parameter of the rhombus core; x and y They represent the coordinate offset of a point in the matrix with the center of the matrix as the origin. The non-zero points of the generated kernel need to satisfy the requirement that the sum of the absolute values ​​of the offsets in the two axis directions is less than the radius parameter. r .

[0048] 2) If the kernel radius is not initialized (that is, when the kernel radius parameter is not specified in the input), the algorithm performs the designed adaptive erosion iterative calculation, uses the erosion kernel to erode the image matrix, and calculates the kernel radius. First, the current image matrix is ​​initialized to a copy of the original image, and the initial value of the adaptive radius is set to zero. Then, an elliptical structure element with a size of 3×3 is used for iterative erosion. After each round of iteration, the adaptive radius value is Increment by 1 until the image matrix is ​​completely zeroed (i.e. empty) to terminate the iteration. Based on the obtained adaptive radius value, the kernel radius is calculated using the empirical coefficient: , Among them, round(x) is a rounding function that returns the integer closest to x; max(a,b) takes the larger value of a and b.

[0049] In getting the nuclear radius , also based on the Manhattan distance principle to generate a diamond kernel. After generating the diamond kernel, the diamond kernel is then used to dilate the original image matrix: , in, D represents the expansion operation, represents the above operation of obtaining a diamond core according to the core radius, represents the image after the dilation operation. Then, the dilation result is differentially calculated with the original image matrix to obtain the boundary truth map corresponding to each image, which is expressed as: .

[0050] In step S3, specifically, the designed new boundary extraction network submodule includes a deformable convolution layer, a first batch of normalization layers, a ninth ReLU activation layer and a labeled convolution layer connected in sequence. During the training process, the present invention obtains the boundary prediction result of the input image from this module, which is expressed as: , in, , represents the feature map of the boundary prediction result, H and W are the spatial dimensions of the feature map respectively; x is the feature map output by the first residual convolution double-layer downsampling module in the first encoder, ; Dconv represents a deformable convolution of size 3×3, Conv represents a convolutional layer of size 1×1, It means that after the activation function sigmoid is processed, it can be expressed as: .

[0051] Next, expand Channel Get , and compare it with the feature map x Combined to get a new feature map It is expressed as: , The new feature map Transmitting it back to the first encoder can increase the model's perception of the boundary information of the input image.

[0052] At the same time, the present invention designs a boundary loss function based on the Dice function, and extracts the prediction results of the network submodules and the boundary truth value by constraining the boundary. Figure 1The loss function used is defined as follows: ,

[0053] in, is the boundary truth map of the input image, The boundary extraction prediction results for the boundary extraction network.

[0054] In step S4, different from the discriminator framework of the classic GAN as a classification network, we designed a U-shaped self-attention discriminator network, which consists of two parts: the second encoder and the second decoder, and transmits information through jump connections. The second encoder consists of four new self-attention convolutional downsampling modules, each of which includes an efficient multi-scale self-attention layer, a second batch normalization layer, and a tenth ReLU activation layer connected in sequence. Compared with the standard convolutional downsampling module, it can effectively enhance the feature extraction ability of the input image pair, thereby enhancing the discriminative ability of the self-attention discriminator network. Among them, the designed efficient multi-scale self-attention layer is based on cross-space learning. By reshaping part of the channel dimension into the batch dimension, it avoids the dimensionality reduction brought by the standard convolutional layer, and constructs a parallel network, designs local cross-channel interactions, and uses cross-space learning methods to fuse the output feature maps of the parallel network.

[0055] The second decoder consists of four convolutional downsampling modules, each of which includes a deconvolution layer, a third batch normalization layer, and an eleventh ReLU activation layer. The inputs of the self-attention discriminator network are fake image pairs consisting of the output of the lightweight generator network and the input image, and real image pairs consisting of the image truth value and the input image. In addition, we propose a discriminator global loss and a discriminator local loss to train the discriminator to distinguish fake image pairs from real image pairs at both the global and local levels. Finally, through the process of adversarial training with the lightweight generator network, the pixel-by-pixel local consistency and image global consistency are constrained, and the lightweight generator network is supervised from a high-dimensional level to output higher-performance detection results.

[0056] The discriminator loss function used is defined as follows, including the discriminator global loss and the discriminator local loss: , , in, represents the second encoder of the self-attention discriminator network; and They represent the judgment results of the self-attention discriminator network on the true image pair and the false image pair at the pixel point (i, j); represents the output result of the lightweight generator network; I represents the input image, Represents the expected function of the true image pair consisting of the true image value Y and the input image I, Represents the output of the lightweight generator network The expected function of the fake image pair composed of and input image I, represents the discriminator global loss, represents the local loss of the discriminator.

[0057] In step S5, the graphics card used in the training process is NVIDIA RTX 3090, the video memory is 24GB, the system used for training is Ubuntu system, the programming language and version used are Python3.8, and the deep learning framework used is Pytorch 1.11.0.

[0058] In step S5, a total of 200 epochs were trained, and we started to add the adversary at the 50th round. We finally retained the model parameters with the best effect on the validation set, the model parameters of the 50th, 100th, 150th, and 200th rounds, including the weights and bias values ​​of each neuron in each layer. The command used was the torch.save() function in the pytorch framework. During the training process, we used adaptive moment estimation weight decay (AdamW) to optimize the generator, with a momentum value of 0.9 and a weight decay of 10. -4 The initial learning rate is set to 0.001, and the learning rate is exponentially decayed in each round with a decay coefficient of 0.99. After 50 rounds, we added the discriminator and adjusted the learning rate to exponentially decay from 0.0001 with a decay coefficient of 0.99. For the discriminator, we use adaptive moment estimation (Adam) for optimization, where the first-order momentum parameter of the discriminator optimizer is set to 0.9 and the second-order momentum parameter is set to 0.99.

[0059] In step S6, tests are performed on two datasets, including UDTIRI-Crack and Deepcrack. UDTIRI-Crack contains 2500 high-quality pavement crack images with a size of 320×320 pixels. These images are collected based on seven public datasets, namely Crack500, CrackLS315, CrackSC, CrackTree260, CRKWH100, DeepCrack537 and ShadowCrack. They are cropped, scaled and divided into 1500 training data images, 400 verification data images and 600 test data images. Since the UDTIRI-Crack dataset comes from different public datasets, it has different characteristics such as crack type, crack morphology, and coarseness, and is affected by different degrees of shadows, occlusions, lighting conditions, noise, etc.

[0060] In step S6, the DeepCrack dataset contains 537 544×384 pixel images of concrete surfaces with cracks of multiple scales and scenes. All images are manually annotated with pixel-level labels, and they are divided into two main subsets: 300 images for training and 237 images for testing.

[0061] In step S6, the road crack image in the test set is input into the main segmentation network, and the pre-trained model parameters are loaded, and the model can output the pixel-level crack detection results. The quantitative evaluation indicators used include precision, recall, accuracy, IoU, F1-Score, and AIoU.

[0062] Compared with the existing pavement crack detection algorithm based on deep learning, this method achieves a balance between detection performance, model scale and model processing speed. After establishing a complete scientific pavement damage database, it is possible to realize intelligent data sharing between pavement construction, management and maintenance systems, and further build a pavement damage prediction model to achieve the "predictability" of pavement damage evolution. In this way, a more reasonable maintenance plan can be formulated, maintenance measures can be predicted and implemented in advance, and the maintenance cycle can be reduced.

[0063] Example 2 This embodiment relates to a lightweight, real-time intelligent detection method for road cracks. The method has the characteristics of lightweight, high performance, and high real-time performance, making it suitable for on-board deployment of intelligent patrol vehicles, facilitating fully automated intelligent detection, and more suitable for smart city construction in the field of digital twins.

[0064] In the training stage, the present invention includes a designed U-shaped main segmentation network and an adaptive boundary extraction module to form a lightweight generator network, and a novel self-attention discriminator network, and trains the U-shaped main segmentation network to learn the mapping from the input image to the pixel-level crack detection result. Among them, we proposed a new residual convolution double-layer downsampling module and a residual convolution double-layer upsampling module, which not only extracts local features of the input image based on the convolutional neural network, but also improves the network's reasoning speed; we proposed a new fast visual transformer, which not only extracts global features of the input image based on the visual transformer, but also improves the network's reasoning speed; we proposed an adaptive erosion and expansion iterative algorithm and a new boundary extraction network sub-module, which realizes the dynamic capture of crack contour features, enhances the learning ability of the designed U-shaped main segmentation network for crack boundary features, and can effectively deal with cracks of different shapes and thicknesses, thereby improving the robustness of network detection; we proposed a new self-attention convolution downsampling module to enhance the feature extraction ability of the input image pairs, so as to enhance the discrimination ability of the discriminator network, and proposed local discrimination loss and global discrimination loss, which realizes high-dimensional supervision of the generator network composed of the U-shaped main segmentation network and the adaptive boundary extraction module, further improving the crack detection performance of the main segmentation network.

[0065] During the testing phase, the best model parameters on the saved validation set are loaded, the road crack image is input, and the pre-trained main segmentation network model is used to obtain pixel-level crack detection results.

[0066] Figure 3 , Figure 4 The road crack images used in algorithm testing and the corresponding segmentation result samples are shown respectively.

[0067] The invention was experimented on the UDTIRI-Crack and Deepcrack datasets. The quantitative experimental results of performance comparison are shown in Tables 1 and 2, respectively. The quantitative experimental results of model parameter quantity and running speed are shown in Table 3.

[0068] Table 1 method Accuracy(%) Recall rate (%) Accuracy(%) F1-Score(%) IoU(%) AIoU(%) Deep-Convolutional Crack Detection Network (Deepcrack18) 74.736 58.702 98.457 65.756 48.982 73.708 Deep Hierarchical-Feature-Learning Crack Detection Network (Deepcrack19) 72.831 57.795 98.390 64.448 47.544 72.956 Skip Connected Crack Detection Network (SCCDNet) 71.294 58.371 98.356 64.189 47.263 72.797 Crack-Attention-Network (Crack-Att) 68.790 66.447 98.392 67.598 51.055 74.710 Crack-Dense-Link-Network (CDLN) 56.947 82.918 97.986 67.522 50.968 74.456 Locally Enhanced Cross-shaped Windows Transformer (LECSFormer) 74.711 65.341 98.567 69.712 53.506 76.025 Convolutional-Transformer Crack Segmentation Network (CT-crackseg) 75.019 66.694 98.599 70.612 54.573 76.574 The present invention 73.435 71.225 98.623 72.313 56.633 77.616 Table 2 method Accuracy(%) Recall rate (%) Accuracy(%) F1-Score(%) IoU(%) AIoU(%) Deep-Convolutional Crack Detection Network (Deepcrack18) 91.793 68.036 98.354 78.149 64.135 81.219 Deep Hierarchical-Feature-Learning Crack Detection Network (Deepcrack19) 90.300 73.891 98.527 81.276 68.458 83.468 Skip Connected Crack Detection Network (SCCDNet) 70.567 82.726 97.760 76.164 61.504 79.590 Crack-Attention-Network (Crack-Att) 86.868 71.090 98.284 78.191 64.191 81.210 Crack-Dense-Link-Network (CDLN) 78.939 85.833 98.396 82.242 69.839 84.087 Locally Enhanced Cross-shaped Windows Transformer (LECSFormer) 85.935 79.124 98.536 82.389 70.052 84.268 Convolutional-Transformer Crack Segmentation Network (CT-crackseg) 93.733 71.043 98.541 80.825 67.821 83.158 The present invention 89.081 78.795 98.665 83.623 71.856 85.237 Table 3 method Parameter quantity (M) FPS Deep-Convolutional Crack Detection Network (Deepcrack18) 30.905 43.935 深度层次特征学习裂缝检测网络(Deepcrack19) 14.72 276.695 跳跃连接裂缝检测网络(SCCDNet) 31.705 113.317 裂缝注意力网络(Crack-Att) 45.804 58.257 裂缝密集连接网络(CDLN) 19.151 45.934 局部增强十字形窗口变换器(LECSFormer) 16.528 65.46 卷积变换器裂缝分割网络(CT-crackseg) 22.882 24.926 The present invention 3.856 338.64 Experimental results show that compared with other public pavement crack detection algorithms in the field, the proposed method has achieved leading results in segmentation performance (precision, recall, accuracy, F1-score, IOU, AIoU) and inference speed (FPS) on these two datasets.

[0069] Example 3 The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0070] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0071] The processing unit performs the various methods and processes described above. For example, in some embodiments, the method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the method in any other appropriate manner (e.g., by means of firmware).

[0072] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0073] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0074] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0075] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A lightweight real-time intelligent road crack detection method, characterized in that: The method comprises the following steps: Step 1: Standardize the acquired road surface image and convert it into a tensor; Step 2: training a lightweight generator network using training set data, wherein the lightweight generator network includes a U-shaped main segmentation network and an adaptive boundary extraction module, wherein the U-shaped main segmentation network includes a first encoder for extracting local features and improving network inference speed, a fast visual transformer for extracting global features, and a first decoder; the adaptive boundary extraction module is used to dynamically capture boundary features of the training set data; Step 3: Input the output of the lightweight generator network into the self-attention discriminator network, train the self-attention discriminator network from both global and local levels to supervise the output of the lightweight generator network, and save the model parameters with the best effect on the validation set; Step 4: Input the processed result of step 1 into the trained U-shaped main segmentation network, load the saved model parameters, and output the pixel-level pavement crack detection result in real time; The first encoder includes two residual convolution double-layer down-sampling modules and two convolution double-layer down-sampling modules, and the first decoder includes two residual convolution double-layer up-sampling modules and two convolution double-layer up-sampling modules.

2. A lightweight real-time intelligent detection method for pavement cracks according to claim 1, characterized in that: Each of the residual convolutional double-layer downsampling modules includes a first maximum pooling layer, a first depth-separable convolutional layer, a first batch of regularization operation layers, a first ReLU activation layer, a second depth-separable convolutional layer, a second batch of regularization operation layers, and a second ReLU activation layer, wherein the output of the first depth-separable convolutional layer is cross-layer connected to the output of the second depth-separable convolutional layer; The structure of each of the convolutional double-layer downsampling modules is similar to that of the residual convolutional double-layer downsampling module, wherein the convolutional layer adopts a standard convolutional layer.

3. A lightweight real-time intelligent detection method for pavement cracks according to claim 1, characterized in that: Each of the residual convolutional double-layer upsampling modules includes a first bilinear upsampling layer, a third depth-separable convolutional layer, a fifth batch regularization operation layer, a fifth ReLU activation layer, a fourth depth-separable convolutional layer, a sixth batch regularization operation layer and a sixth ReLU activation layer, wherein the output of the third depth-separable convolutional layer is cross-layer connected to the output of the fourth depth-separable convolutional layer; The structure of each of the convolutional double-layer upsampling modules is similar to that of the residual convolutional double-layer upsampling module, wherein the convolutional layer adopts a standard convolutional layer.

4. A lightweight real-time intelligent detection method for pavement cracks according to claim 1, characterized in that: The fast visual transformer includes a spatial attention layer, a standard convolution layer, a spatial attention layer, a channel attention layer, a standard convolution layer and a random inactivation layer connected in sequence.

5. The lightweight real-time intelligent detection method for pavement cracks according to claim 1, characterized in that: The adaptive boundary extraction module includes a boundary extraction network submodule, and the boundary extraction network submodule is trained and constrained so that the extracted boundary information is consistent with the boundary truth map.

6. A lightweight real-time intelligent road crack detection method according to claim 5, characterized in that: The adaptive boundary extraction module adopts an adaptive corrosion and expansion iterative algorithm to obtain the boundary truth map, and the specific method includes: First, the first channel image matrix of the input road surface image tensor data is extracted and the data type is standardized; If the core radius is not initialized, the adaptive erosion iteration is performed to calculate the adaptive radius value; Based on the adaptive radius value, the kernel radius is calculated using an empirical coefficient : ， Among them, round(x) is a rounding function that returns the integer closest to x; max(a,b) is to take the larger value of a and b; is the adaptive radius value; A diamond kernel is generated based on the kernel radius and Manhattan distance principle, and then the diamond kernel is used to dilate the original image matrix. The dilation result is differentially calculated with the original image matrix to obtain the boundary truth map corresponding to each image.

7. A lightweight real-time intelligent detection method for road cracks according to claim 6, characterized in that: If the kernel radius has been initialized, a diamond kernel is directly generated based on the final kernel radius and the Manhattan distance principle, and then the diamond kernel is used to dilate the original image matrix, and the dilation result is differentially calculated with the original image matrix to obtain the boundary truth map corresponding to each image.

8. A lightweight real-time intelligent road crack detection method according to claim 7, characterized in that: The process of performing adaptive erosion iteration to calculate the adaptive radius value includes: Initialization process: Initialize the current image matrix to be a copy of the original image, set the initial value of the adaptive radius to 0, and create an elliptical erosion kernel; Iterative erosion operation: an elliptical erosion kernel is used to iteratively erode the image matrix, and the radius value is adaptive after each iteration. Increment by 1 until the image matrix is ​​completely zeroed and the iteration is terminated.

9. A lightweight real-time intelligent detection method for pavement cracks according to claim 1, characterized in that: The self-attention discriminator network is a U-shaped structure, comprising a second encoder and a second decoder, and transmits information through a jump connection; The second encoder is composed of four self-attention convolutional downsampling modules, each of which includes an efficient multi-scale self-attention layer, a second batch normalization layer, and a tenth ReLU activation layer connected in sequence; The second decoder is composed of four convolutional downsampling modules, each of which includes a deconvolution layer, a third batch normalization layer and an eleventh ReLU activation layer.

10. A lightweight real-time intelligent road crack detection method according to claim 9, characterized in that: The input of the self-attention discriminator network includes a fake image pair consisting of the output of the lightweight generator network and the input road image, and a real image pair consisting of the image truth value and the input road image, the self-attention discriminator network is trained by the discriminator local loss and the discriminator global loss, and the pixel-by-pixel local consistency and the image global consistency are constrained by adversarial training with the lightweight generator network; The efficient multi-scale self-attention layer is based on cross-space learning, which reshapes part of the channel dimensions into batch dimensions, builds a parallel network, designs local cross-channel interactions, and uses a cross-space learning method to fuse the output feature maps of the parallel networks.

Images