Pavement crack detection method and system

Abstract

The invention provides a pavement crack detection method and system. The method comprises the following steps: acquiring a plurality of pavement crack images; a Swinin-Unet model is constructed; determining the total loss of the Swinin-Unet model by using the plurality of pavement crack images; using the total loss of the Swinin-Unet model to train the Swinin-Unet model, and using the trained Swinin-Unet model as a pavement crack detection model; and inputting a to-be-detected pavement crack image into the pavement crack detection model to obtain a pavement crack detection result output by the pavement crack detection model. According to the method, the comprehensive capture capability on the global and local features of the crack and the segmentation effect on crack details are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a road surface crack detection method and system. Background Art

[0002] As the most important part of transportation infrastructure, roads are subjected to continuous heavy traffic and heavy freight during their service life. They are prone to various diseases and damages, including different types of cracks. The efficiency of disease detection is crucial for the evaluation of road usage status and durability.

[0003] With the development and popularization of artificial intelligence, related detection methods based on deep learning are gradually used in pavement crack detection tasks. Deep learning does not require manual design of feature extractors, and can automatically perform abstract expressions based on the original image features to determine whether a specific pixel is part of a crack.

[0004] However, cracks have various shapes and directions, and their colors are often highly similar to the background. These uncertainties significantly increase the recognition difficulty of the segmentation model. In addition, the lighting conditions in the natural environment may vary significantly, resulting in fluctuations in the visibility of cracks. Cracks usually only occupy a small area in the image, while most of the area is crack-free background, resulting in a serious imbalance in the ratio of positive and negative samples. Acquiring a large number of high-quality annotated datasets is not only time-consuming and laborious, but the data features obtained in different regions or environments may be different, further limiting the generalization ability of the model. Summary of the invention

[0005] The present invention aims at the deficiencies in the prior art and provides a road surface crack detection method and system.

[0006] In a first aspect, the present invention provides a pavement crack detection method, comprising:

[0007] Acquire multiple pavement crack images;

[0008] Build the Swin-Unet model;

[0009] Determine the total loss of the Swin-Unet model using multiple pavement crack images;

[0010] The total loss of the Swin-Unet model is used to train the Swin-Unet model, and the trained Swin-Unet model is used as a pavement crack detection model;

[0011] The pavement crack image to be detected is input into the pavement crack detection model, and the pavement crack detection model outputs the pavement crack detection result.

[0012] Optionally, constructing the Swin-Unet model includes:

[0013] Constructing an encoder module, which uses a Swin Transformer block to extract features from the pavement crack image, segmenting the pavement crack image into non-overlapping image blocks, and then processing each image block to obtain its embedding vector sequence, and combining them to obtain the overall sequence embedding to form an overall feature representation;

[0014] Constructing an intermediate connection module, wherein the intermediate connection module is used to further perform feature learning and extract deep features, and the intermediate connection module is composed of two consecutive Swin Transformer blocks;

[0015] Construct a decoder module, which includes a Swin Transformer block and an image block expansion layer. The decoder is used to process the decoding task of sequence embedding and gradually restore the resolution of the original image to generate a segmentation result.

[0016] By introducing skip connections in the model structure, features at different levels are fused and integrated to fuse the extracted contextual features with the multi-scale features of the encoder.

[0017] Optionally, constructing an encoder module, wherein the encoder module uses a Swin Transformer block to extract features from a pavement crack image, comprises:

[0018] Step 211, dividing the road crack image into image blocks of a preset pixel size, and the image blocks do not overlap each other;

[0019] Step 212, mapping the feature dimension of the image block to the spatial dimension C by applying a linear embedding layer;

[0020] Step 213, using two consecutive Swin Transformer blocks to process the feature map, in the process keeping the feature dimension and resolution unchanged;

[0021] Step 214, merging adjacent image blocks through an image merging layer to reduce computational complexity and further extract multi-scale features, reducing the spatial resolution to a multiple of the original, while increasing the number of feature channels to generate a feature map with a more compact representation;

[0022] Step 215, repeating steps 213 and 214 three times in the encoder to generate a hierarchical feature representation;

[0023] Step 216, applying a linear layer to expand the feature dimension, applying a linear layer to the connected features, introducing a linear transformation, and expanding the connected feature dimension to a preset multiple of the original dimension.

[0024] Optionally, constructing a decoder module, the decoder module includes two parts: a Swin Transformer block and an image block expansion layer, the decoder is used to process the decoding task of sequence embedding, and gradually restores the resolution of the original image to generate a segmentation result, including:

[0025] Step 231, using the Swin Transformer block and the image block expansion layer to process the sequence embedding decoding task;

[0026] Step 232, fusing features at different levels through skip connections, combining the encoder's multi-scale features with context information to compensate for the spatial information lost during the downsampling process;

[0027] Step 233, using an image block expansion layer to process the input feature map, gradually increasing its spatial resolution to restore the original spatial resolution of the feature map;

[0028] Step 234, using the Swin Transformer block to extract high-level features and gradually restore the spatial information of the input image to generate an accurate segmentation result;

[0029] Step 235, repeating step 232, step 233 and step 234 three times to gradually restore the resolution of the feature map and extract richer context information;

[0030] Step 236, applying a linear projection layer on the obtained feature map to generate the final pixel-level segmentation prediction result.

[0031] Optionally, determining the total loss of the Swin-Unet model using a plurality of pavement crack images includes:

[0032] The binary Dice loss L is calculated according to the following formula Binary Dice :

[0033]

[0034] Where N is the total number of pixels in the pavement crack image; P i is the predicted category probability corresponding to pixel i in the pavement crack image; T i is the category label corresponding to pixel i in the pavement crack image; smooth represents the smoothness term;

[0035] The edge-aware loss L is calculated according to the following formula Edge Aware :

[0036]

[0037] Where Sobel(·) represents the Sobel operator, which is used to extract edge information of the image; Sobel(P) i Represents the edge strength of the i-th pixel in the predicted image; Sobel(T) i Represents the edge strength of the i-th pixel of the true label;

[0038] The total loss L of the Swin-Unet model is calculated according to the following formula Swin-Unet :

[0039] L Swin-Unet =α·L Binary Dice +β·L Edge Aware ;

[0040] Among them, α is the weight of the binary Dice loss; β is the weight of the edge-aware loss.

[0041] In a second aspect, the present invention provides a pavement crack detection system, comprising:

[0042] An acquisition module, used for acquiring a plurality of pavement crack images;

[0043] Building module, used to build Swin-Unet model;

[0044] a determination module for determining a total loss of a Swin-Unet model using a plurality of pavement crack images;

[0045] A model training module, used to train the Swin-Unet model using the total loss of the Swin-Unet model, and use the trained Swin-Unet model as a pavement crack detection model;

[0046] The crack detection module is used to input the pavement crack image to be detected into the pavement crack detection model, and obtain the pavement crack detection result output by the pavement crack detection model.

[0047] Optionally, the building blocks include:

[0048] A first construction unit is used to construct an encoder module, wherein the encoder module uses a Swin Transformer block to extract features from a pavement crack image, divides the pavement crack image into non-overlapping image blocks, and then processes each image block to obtain its embedding vector sequence, and obtains the overall sequence embedding by combining to form an overall feature representation;

[0049] A second construction unit is used to construct an intermediate connection module, wherein the intermediate connection module is used to further perform feature learning and extract deep features, and the intermediate connection module is composed of two consecutive Swin Transformer blocks;

[0050] The third construction unit is used to construct a decoder module, which includes a Swin Transformer block and an image block expansion layer. The decoder is used to process the decoding task of sequence embedding and gradually restore the resolution of the original image to generate a segmentation result.

[0051] The feature fusion unit is used to fuse and integrate features at different levels by introducing skip connections in the model structure to fuse the extracted context features with the multi-scale features of the encoder.

[0052] Optionally, the second building unit includes:

[0053] An image segmentation device, used for segmenting the road crack image into image blocks of preset pixel size, wherein the image blocks do not overlap each other;

[0054] Dimension mapping means for mapping the feature dimension of the image block to a spatial dimension C by applying a linear embedding layer;

[0055] a first image block processing device for processing the feature map using two consecutive Swin Transformer blocks, while maintaining the feature dimension and resolution unchanged;

[0056] The second image block processing device is used to merge adjacent image blocks through a picture merging layer to reduce the computational complexity and further extract multi-scale features, reduce the spatial resolution to a multiple of the original, and increase the number of channels of the feature to generate a feature map with a more compact representation;

[0057] A feature representation generating device, configured to repeat the operations of the first image block processing device and the second image block processing device three times in the encoder to generate a hierarchical feature representation;

[0058] The feature dimension expansion device is used to apply a linear layer to expand the feature dimension. The linear layer is applied to the connected features, and a linear transformation is introduced to expand the connected feature dimension to a preset multiple of the original dimension.

[0059] Optionally, the third building block comprises:

[0060] A decoding task processing device for processing a decoding task of sequence embedding using a Swin Transformer block and an image block expansion layer;

[0061] A feature fusion device for fusing features at different levels through skip connections, combining the multi-scale features of the encoder with context information to compensate for the spatial information lost during the downsampling process;

[0062] A first resolution recovery device is used to process the input feature map using an image block expansion layer to gradually increase its spatial resolution to restore the original spatial resolution of the feature map;

[0063] A segmentation result generating device, for extracting high-level features using a Swin Transformer block and gradually restoring spatial information of an input image to generate an accurate segmentation result;

[0064] A second resolution recovery device, used to repeat the operations of the feature fusion device, the first resolution recovery device and the segmentation result generation device three times to gradually recover the resolution of the feature map and extract richer context information;

[0065] The prediction result generating device is used to apply a linear projection layer on the obtained feature map to generate a final pixel-level segmentation prediction result.

[0066] Optionally, the determining module includes:

[0067] The first calculation unit is used to calculate the binary Dice loss L according to the following formula Binary Dice :

[0068]

[0069] Where N is the total number of pixels in the pavement crack image; P i is the predicted category probability corresponding to pixel i in the pavement crack image; T i is the category label corresponding to pixel i in the pavement crack image; smooth represents the smoothness term;

[0070] The second calculation unit is used to calculate the edge perception loss L according to the following formula Edge Aware :

[0071]

[0072] Where Sobel(·) represents the Sobel operator, which is used to extract edge information of the image; Sobel(P) i Represents the edge strength of the i-th pixel in the predicted image; Sobel(T) i Represents the edge strength of the i-th pixel of the true label;

[0073] The third calculation unit is used to calculate the total loss L of the Swin-Unet model according to the following formula Swin-Unet :

[0074] L Swin-Unet =α·L Binary Dice +β·L Edge Aware ;

[0075] Among them, α is the weight of the binary Dice loss; β is the weight of the edge-aware loss.

[0076] The present invention provides a pavement crack detection method and system. In the method, the Swin-UNet model can accurately capture the morphological characteristics of cracks at different scales through its layered architecture and sliding window mechanism. Compared with the local receptive field limitation of the convolutional network, the Swin-UNet model significantly improves the comprehensive capture capability of the global and local characteristics of cracks.

[0077] In the loss function design, a combination of binary Dice loss and edge-aware loss is adopted. The binary Dice loss effectively handles the class imbalance problem in the crack segmentation task by optimizing the Dice coefficient, making the model pay more attention to small crack areas and improving the accuracy of segmentation. At the same time, the edge-aware loss extracts the edge information of the predicted image and the real label by introducing the Sobel operator, thereby enhancing the model's ability to recognize crack boundaries. The combination of the two loss functions not only balances the needs of overall region segmentation and edge detail detection, but also significantly improves the segmentation effect of the model on crack details. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 A schematic diagram of a process flow of a road surface crack detection method provided by an embodiment of the present invention;

[0080] Figure 2 Flow chart of segmentation by the Swin-UNet model provided in an embodiment of the present invention;

[0081] Figure 3 A detection effect diagram of the first pavement crack map provided by an embodiment of the present invention;

[0082] Figure 4 A detection effect diagram of a second pavement crack map provided by an embodiment of the present invention;

[0083] Figure 5 A detection effect diagram of a third pavement crack map provided by an embodiment of the present invention;

[0084] Figure 6 A detection effect diagram of the fourth pavement crack map provided by an embodiment of the present invention;

[0085] Figure 7A fifth pavement crack detection effect diagram provided by an embodiment of the present invention;

[0086] Figure 8 A schematic structural diagram of a pavement crack detection system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] Example 1

[0089] like Figure 1 As shown, this embodiment provides a road surface crack detection method, including:

[0090] S1, obtaining multiple pavement crack images.

[0091] In this step, the public datasets CFD and CRACK500 were used. The CFD dataset contains 118 annotated crack images with a resolution of 480×320, and the CRACK500 dataset contains 471 annotated crack images with a resolution of 1440×2560. The dataset contains a total of 589 images. Among them, there are 156 vertical crack images and 433 horizontal crack images. The dataset is divided into 475 training set images and 114 test set images in a ratio of 8:2.

[0092] Since the shape, color and background of cracks are complex and changeable, factors such as lighting and contrast will directly affect the segmentation effect of the model. Preprocessing operations can reduce the interference caused by differences in shooting environments, thereby improving the robustness of the model. In addition, cracks usually occupy a small part of the image and are distributed in different positions and directions. This imbalance will cause the model to tend to focus on the background area and ignore the cracks. Preprocessing can enrich the data set, increase the frequency of cracks, and help the model learn crack features more effectively.

[0093] In this step, brightness adjustment, rotation, horizontal mirroring and contrast adjustment are performed on the image to improve the robustness and generalization ability of the model.

[0094] First, the brightness adjustment is divided into two parts: increasing and decreasing the brightness, which aims to simulate different lighting conditions and enhance the model's adaptability to changes in ambient lighting. Specifically, the brightness coefficient is set to 1.5 to increase the brightness and to 0.5 to decrease the brightness through the ImageEnhance.Brightness module. This adjustment allows the model to perform stably in both well-lit and under-lit scenes.

[0095] Secondly, the directionality and diversity of the dataset are enhanced by rotation operations. The image is rotated 180 degrees using OpenCV’s cv2.rotate function, and the label image is processed synchronously to ensure the spatial consistency of the image. This enhancement strategy helps the model better adapt to crack morphologies in different directions.

[0096] In addition, the horizontal mirroring operation further enriches the diversity of the data. The cv2.flip function is used to flip the image and label horizontally to generate left-right symmetrical training samples, thereby enhancing the model's ability to identify cracks in different symmetrical environments.

[0097] Finally, contrast adjustment further optimizes the recognition effect of the model by improving the distinction between cracks and background. The ImageEnhance.Contrast module is used to enhance the image contrast by 1.5 times to highlight the significance of the crack edge, making it easier for the model to capture the boundary information of the crack. This enhancement method effectively improves the segmentation accuracy of the model.

[0098] After image preprocessing, we finally obtained 2850 training set images and 684 test set images.

[0099] S2, build the Swin-Unet model.

[0100] The Swin-Unet model is an image segmentation network based on the Swin Transformer structure, which is used for the segmentation of images with complex structures and rich details. The Swin-Unet model combines the global modeling capabilities of the Transformer with the hierarchical feature extraction structure of the Unet, enabling it to demonstrate excellent expressiveness and robustness in refined segmentation tasks such as crack detection. Compared with traditional convolutional neural networks (CNNs), the Swin-Unet model is more effective in capturing local and global features of images through a hierarchical self-attention mechanism, while maintaining computational efficiency and spatial continuity, thereby achieving high-precision segmentation without sacrificing resolution.

[0101] The Swin-Unet model introduces the Swin Transformer as the backbone network, and effectively overcomes the limitations of traditional convolutional neural networks in terms of receptive field through a hierarchical self-attention mechanism. The Swin Transformer can capture the global contextual information of crack images at different scales, and combined with the advantages of Unet in local detail recovery, the Swin-Unet model can accurately identify the shape and size of cracks and subtle features in complex backgrounds. In addition, the hierarchical architecture and sliding window mechanism of the Swin-Unet model improve the model's ability to process large-size images, making it more generalizable and adaptable in crack detection tasks.

[0102] Exemplarily, this step includes constructing an encoder module, which uses a Swin Transformer block to extract features from a pavement crack image, divides the pavement crack image into non-overlapping image blocks, and then processes each image block to obtain its embedding vector sequence, obtains the overall sequence embedding through combination, and forms an overall feature representation.

[0103] An intermediate connection module is constructed, wherein the intermediate connection module is used to further perform feature learning and extract deep features, and the intermediate connection module is composed of two consecutive Swin Transformer blocks.

[0104] Construct a decoder module, which includes a Swin Transformer block and an image block expansion layer. The decoder is used to process the decoding task of sequence embedding and gradually restore it to the resolution of the original image to generate a segmentation result.

[0105] By introducing skip connections in the model structure, features at different levels are fused and integrated to fuse the extracted contextual features with the multi-scale features of the encoder.

[0106] The encoder module is constructed, and the encoder module uses the Swin Transformer block to extract features from the pavement crack image, including:

[0107] Step 211 : segment the road surface crack image into image blocks of a preset pixel size, and the image blocks do not overlap with each other.

[0108] Step 212, mapping the feature dimension of the image block to the spatial dimension C by applying a linear embedding layer.

[0109] In step 213, the feature map is processed using two consecutive Swin Transformer blocks, while keeping the feature dimension and resolution unchanged.

[0110] Step 214, merge adjacent image blocks through a picture merging layer to reduce computational complexity and further extract multi-scale features, reduce the spatial resolution to a multiple of the original, and increase the number of feature channels to generate a feature map with a more compact representation.

[0111] Step 215, repeating steps 213 and 214 three times in the encoder to generate a hierarchical feature representation.

[0112] Step 216, applying a linear layer to expand the feature dimension, applying a linear layer to the connected features, introducing a linear transformation, and expanding the connected feature dimension to a preset multiple of the original dimension.

[0113] like Figure 2 As shown in the figure, in the encoder part of the Swin-Unet model, Swin Transformer is used as the core structure of feature extraction. Swin Transformer captures the local features of the image through a sliding window mechanism, and gradually extracts the different scale features of the image through a hierarchical self-attention mechanism to obtain global information. Each layer of self-attention includes multiple layers of self-attention modules and multi-layer perceptrons (MLPs), thereby realizing the fusion of local and global information.

[0114] First, we input an H×W×3 image, where H represents the height of the image, W represents the width of the image, and 3 represents the number of channels of the image, i.e., there are three color channels: R (red), G (green), and B (blue). Then, through the PatchPartition process, where the Patch Size is 4*4, each patch will be mapped into a feature vector, so the size of the feature map becomes That is, the resolution of the image is reduced to 1 / 4 of the original image, and each pixel has 3 channels, so the number of channels is 48. Next, after the Linear Embedding layer, the channel dimension data is linearly transformed from 48 to C' (C' is a hyperparameter, here 96), generating a size of The feature map retains part of the spatial information of the original image and increases the feature dimension to better describe the image content.

[0115] Then it enters two Swin Transformer blocks for processing. Unlike the traditional multi-head self-attention (MSA) module, the Swin Transformer is built based on a shifted window. Each Swin Transformer module consists of a LayerNorm (LN) layer, a multi-head self-attention module, a residual connection, and a 2-layer MLP with GELU nonlinearity. The window-based multi-head self-attention (W-MSA) module and the shifted window-based multi-head self-attention (SW-MSA) module are applied to two consecutive Transformer blocks respectively. Based on such a window partitioning mechanism, the consecutive Swin Transformer blocks can be formulated as:

[0116]

[0117] W-MSA(·) represents window-based multi-head self-attention; SW-MSA(·) represents shifted window multi-head self-attention.

[0118] First, input z l-1 Perform layer normalization, then calculate multi-head attention in each window, and finally compare the result with the input z l-1 Add together to achieve residual connection.

[0119] The results after W-MSA processing The layer is normalized again, and then nonlinear mapping is performed through a multi-layer perceptron (MLP). Finally, the output of the MLP is compared with z^ l Add together to form the output z after residual connection l .

[0120] Right l After layer normalization, multi-head self-attention (SW-MSA) is calculated within the shifted window.

[0121] Finally, compare the result with z l Add together to achieve residual connection.

[0122] The results after SW-MSA treatment Perform layer normalization, and then use MLP to perform nonlinear transformation. Finally, the output of MLP is compared with Add together to form the final output z after residual connection l+1 .

[0123] Among them, z l Represents the output of the (S)W-MSA module and the MLP module of the lth block.

[0124] The core formula of self-attention is:

[0125] Q=XW Q ,K=XW K ,V=XW V .

[0126]

[0127] X is the input feature, W Q , W K , W V are the linear projection matrices that map the input features into Q, K, and V, respectively. represents query, key and value, where M 2 represents the number of image patches in the window; d represents the dimension of the query or key, which is used for scaling to prevent the attention value from being too large, and the values ​​in B are taken from the bias matrix B′∈R (2M-1)× ( 2M+1 ).

[0128] The core formula of the multi-layer perceptron (MLP) is:

[0129] MLP(x)=σ(xW1+b1)W2+b2.

[0130] W1 and W2 both represent weights, b1 and b2 both represent biases, and σ is the activation function.

[0131] After passing through two consecutive Swin Transformer blocks, the amount of calculation is reduced while the information transfer between windows is realized, and the correlation between distant pixels is realized, thereby extracting global feature information. Then, the model performs a Patch Merging operation, and the resolution of the feature map is further reduced to This process merges two adjacent image blocks to reduce the spatial dimension of the feature map, thereby effectively compressing the information. The feature map is then further processed by two more identical Patch Merging operations and two SwinTransformer blocks, and the image is finally compressed to This series of operations significantly reduces the computational complexity and enhances the model's representation capabilities by extracting multi-scale features. This gradual compression process can effectively retain key information in the image, thereby improving the model's robustness in dealing with complex scenes and subtle changes.

[0132] In the middle connection layer of the model, two Swin Transformer blocks are included for deep feature extraction of highly compressed feature maps. Through the two attention mechanisms of W-MSA and SW-MSA, the receptive field of the model is further expanded to further capture global contextual information, compensate for the loss of local feature information caused by the compression process, and ensure that the model can obtain richer semantic information and global structural features, thereby improving the overall segmentation accuracy and generalization ability.

[0133] In the decoder part, the decoder is responsible for restoring the compressed feature map processed by the encoder and bottleneck layer to the resolution of the original image, thereby generating accurate segmentation results.

[0134] Exemplarily, the construction of a decoder module, the decoder module includes two parts: a Swin Transformer block and an image block expansion layer, the decoder is used to process the decoding task of sequence embedding, and gradually restores the resolution of the original image to generate a segmentation result, including:

[0135] Step 231, using the Swin Transformer block and the image block expansion layer to process the sequence embedding decoding task.

[0136] Step 232, features at different levels are fused through skip connections, and the multi-scale features of the encoder are combined with context information to compensate for the spatial information lost during the downsampling process.

[0137] Step 233, using the image block expansion layer to process the input feature map, gradually increasing its spatial resolution to restore the original spatial resolution of the feature map.

[0138] In step 234, a Swin Transformer block is used to extract high-level features and gradually restore the spatial information of the input image to generate an accurate segmentation result.

[0139] Step 235, repeat steps 232, 233 and 234 three times to gradually restore the resolution of the feature map and extract richer context information.

[0140] Step 236, applying a linear projection layer on the obtained feature map to generate the final pixel-level segmentation prediction result.

[0141] To achieve this goal, the decoder uses a multi-level feature map upsampling step by step. By hierarchically fusing the feature information of different levels in the encoding stage, the decoder can reconstruct the detailed structure of the image. The specific upsampling and feature fusion process is as follows:

[0142] First, the resolution of the feature map is increased from 100 to 1000 by Patch Expanding and two Swin Transformer blocks. Upsample to This process gradually restores the detailed information in the feature map and provides rich contextual information through the self-attention mechanism, so that the model can better understand the complex relationships in the image, thereby improving the accuracy of the segmentation task. After each upsampling, the decoder fuses the feature map of the corresponding layer in the encoder through the Skip Connection. The role of the skip connection is to pass the features of different scales in the encoder directly to the decoder to retain multi-scale information. This feature fusion not only enables the model to retain rich spatial information when restoring details, but also improves the model's expressiveness in segmentation tasks. Next, the feature map is restored to the original state through two more similar operations. The upsampled feature maps of each layer are concatenated or added element by element to the features of the previous layer, which can gradually restore the image structure and improve the model's ability to capture target details.

[0143] Finally, in the last step of the decoder, LinearProjection is used to perform a 1×1 convolution on the gradually recovered feature map to restore it to the original input H×W×Class. This linear projection generates the classification result for each pixel and outputs a segmentation mask, where each pixel value corresponds to a specific classification label. This complete segmented image output represents the model's accurate segmentation of different areas in the image, achieving high-precision detection of tiny details such as cracks.

[0144] S3, Determine the total loss of the Swin-Unet model using multiple pavement crack images.

[0145] In the training process of the Swin-Unet model, the choice of loss function is crucial. Crack segmentation has the characteristic of feature imbalance, that is, cracks usually only occupy a small part of the image, and most of the image area is a crack-free background. Therefore, this embodiment selects a combination of binary Dice loss (BinaryDice Loss) and edge-aware loss (EdgeAware Loss) to balance the overall region segmentation accuracy and edge recognition ability. Binary Dice Loss helps to deal with the problem of class imbalance, and EdgeAware Loss strengthens the model's attention to the edges of cracks, allowing the model to capture the details of the cracks more accurately.

[0146] Exemplarily, the binary Dice loss L is calculated according to the following formula: Binary Dice :

[0147]

[0148] Where N is the total number of pixels in the pavement crack image; P i is the predicted category probability corresponding to pixel i in the pavement crack image; T i is the category label corresponding to pixel i in the pavement crack image; smooth represents the smoothing term to avoid the denominator being 0; It represents the intersection of the predicted value and the true label, that is, the overlap between the prediction and the label is calculated at all pixel positions. The 2x coefficient is used to emphasize the importance of the intersection. It represents the sum of the p-th power of the predicted value and the p-th power of the true label. Usually p=2, that is, the sum of squares is used to calculate the total number of pixels of the prediction and label. Finally, the 1-Dice coefficient is used to get the Binary Dice Loss value. The higher the Dice coefficient, the lower the Binary Dice Loss, indicating that the model's prediction is closer to the true label.

[0149] Edge Aware Loss uses the Sobel operator to extract the edge information of the image and measures the difference between the predicted edge and the real edge through the mean square error (MSE) to improve the edge recognition ability of the model. The specific formula is as follows:

[0150]

[0151] Among them, I is the given image, G x is the horizontal gradient of the image; G y is the vertical gradient of the image; * represents the convolution operation; G represents the edge amplitude of the image.

[0152] The edge-aware loss L is calculated according to the following formula Edge Aware :

[0153]

[0154] Where Sobel(·) represents the Sobel operator, which is used to extract edge information of the image; Sobel(P) i Represents the edge strength of the i-th pixel in the predicted image; Sobel(T) i Represents the edge strength of the i-th pixel of the true label; the value of EdgeAware Loss is used to measure the difference between the model prediction result and the true label in the edge area.

[0155] The total loss L of the Swin-Unet model is calculated according to the following formula Swin-Unet :

[0156] L Swin-Unet =α·L Binary Dice +β·LEdge Aware .

[0157] Wherein, α is the weight of the binary Dice loss, and β is the weight of the edge-aware loss. In this embodiment, α=0.6, β=0.4.

[0158] S4, using the total loss of the Swin-Unet model to train the Swin-Unet model, and using the trained Swin-Unet model as a pavement crack detection model.

[0159] In this step, the trained Swin-Unet model is tested using the test set images in S1.

[0160] S5, inputting the pavement crack image to be detected into a pavement crack detection model, and obtaining a pavement crack detection result output by the pavement crack detection model.

[0161] like Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, the ground truth images (Ground Truth) of 5 different crack images obtained by image processing methods and the segmentation result images using the Swin-UNet model are obtained. Folder1 represents the original image, Folder2 represents the ground truth image obtained by the image processing method, and Folder3 represents the segmentation result image using the Swin-UNet model.

[0162] In summary, the pavement crack detection method provided in this embodiment introduces the self-attention mechanism of Swin Transformer into crack detection. Unlike the traditional UNet, the Swin-UNet model can accurately capture the morphological characteristics of cracks at different scales through its layered architecture and sliding window mechanism. Compared with the local receptive field limitation of the convolutional network, the Swin-UNet model significantly improves the comprehensive capture capability of the global and local characteristics of cracks.

[0163] Multiple image preprocessing techniques were introduced in the crack segmentation task to enhance the robustness and generalization ability of the Swin-UNet model. Through operations such as brightness increase and decrease, image rotation and horizontal mirroring, and contrast enhancement, the diversity of the data set was enriched, so that the Swin-UNet model can better adapt to the significant changes of cracks in different directions and environments, and the segmentation accuracy of the Swin-UNet model was enhanced. In the design of the loss function, a combination of Binary Dice Loss and Edge Aware Loss was adopted. Binary Dice Loss effectively handles the category imbalance problem in the crack segmentation task by optimizing the Dice coefficient, so that the Swin-UNet model pays more attention to small crack areas and improves the accuracy of segmentation. At the same time, Edge Aware Loss extracts the edge information of the predicted image and the true label by introducing the Sobel operator, and uses the mean square error to measure the edge difference between the two, thereby enhancing the model's ability to recognize crack boundaries. This combination of loss functions not only balances the needs of overall region segmentation and edge detail detection, but also significantly improves the segmentation effect of the Swin-UNet model on crack details.

[0164] Example 2

[0165] Based on the same inventive concept as Example 1, this embodiment provides a pavement crack detection system. Since the principle of solving the problem by this system is similar to the aforementioned pavement crack detection method, the implementation of this system can refer to the implementation of the pavement crack detection method.

[0166] like Figure 8 As shown, the pavement crack detection system comprises:

[0167] The acquisition module 10 is used to acquire a plurality of pavement crack images.

[0168] The construction module 20 is used to construct the Swin-Unet model.

[0169] The determination module 30 is used to determine the total loss of the Swin-Unet model using a plurality of pavement crack images.

[0170] The model training module 40 is used to train the Swin-Unet model using the total loss of the Swin-Unet model, and use the trained Swin-Unet model as a pavement crack detection model.

[0171] The crack detection module 50 is used to input the pavement crack image to be detected into the pavement crack detection model, and obtain the pavement crack detection model to output the pavement crack detection result.

[0172] Exemplarily, the building blocks include:

[0173] A first construction unit is used to construct an encoder module, wherein the encoder module uses a Swin Transformer block to extract features from a pavement crack image, divides the pavement crack image into non-overlapping image blocks, and then processes each image block to obtain its embedding vector sequence, and obtains the overall sequence embedding by combining to form an overall feature representation;

[0174] A second construction unit is used to construct an intermediate connection module, wherein the intermediate connection module is used to further perform feature learning and extract deep features, and the intermediate connection module is composed of two consecutive Swin Transformer blocks;

[0175] The third construction unit is used to construct a decoder module, which includes a Swin Transformer block and an image block expansion layer. The decoder is used to process the decoding task of sequence embedding and gradually restore the resolution of the original image to generate a segmentation result.

[0176] The feature fusion unit is used to fuse and integrate features at different levels by introducing skip connections in the model structure to fuse the extracted context features with the multi-scale features of the encoder.

[0177] Exemplarily, the second building unit includes:

[0178] The image segmentation device is used to segment the road crack image into image blocks with a size of preset pixels, and the image blocks do not overlap each other.

[0179] A dimension mapping device is used to map the feature dimension of the image block to a spatial dimension C by applying a linear embedding layer.

[0180] The first image block processing device is used to process the feature map using two consecutive Swin Transformer blocks, while keeping the feature dimension and resolution unchanged.

[0181] The second image block processing device is used to merge adjacent image blocks through a picture merging layer to reduce the computational complexity and further extract multi-scale features, reduce the spatial resolution to a multiple of the original, and increase the number of feature channels to generate a feature map with a more compact representation.

[0182] The feature representation generating device is used to repeat the operations of the first image block processing device and the second image block processing device three times in the encoder to generate a hierarchical feature representation.

[0183] The feature dimension expansion device is used to apply a linear layer to expand the feature dimension. The linear layer is applied to the connected features, and a linear transformation is introduced to expand the connected feature dimension to a preset multiple of the original dimension.

[0184] Exemplarily, the third building block comprises:

[0185] The decoding task processing device is used to process the decoding task of sequence embedding using the Swin Transformer block and the image block expansion layer.

[0186] The feature fusion device is used to fuse features of different levels through jump connections, combining the multi-scale features of the encoder with context information to compensate for the spatial information lost during the downsampling process.

[0187] The first resolution recovery device is used to process the input feature map using the image block expansion layer to gradually increase its spatial resolution so as to recover the original spatial resolution of the feature map.

[0188] The segmentation result generating device is used to extract high-level features using the Swin Transformer block and gradually restore the spatial information of the input image to generate an accurate segmentation result.

[0189] The second resolution recovery device is used to repeat the operations of the feature fusion device, the first resolution recovery device and the segmentation result generation device three times to gradually recover the resolution of the feature map and extract richer context information.

[0190] The prediction result generating device is used to apply a linear projection layer on the obtained feature map to generate a final pixel-level segmentation prediction result.

[0191] Exemplarily, the determining module includes:

[0192] The first calculation unit is used to calculate the binary Dice loss L according to the following formula Binary Dice :

[0193]

[0194] Where N is the total number of pixels in the pavement crack image; P i is the predicted category probability corresponding to pixel i in the pavement crack image; T i is the category label corresponding to pixel i in the pavement crack image; smooth represents the smoothness term;

[0195] The second calculation unit is used to calculate the edge perception loss L according to the following formula Edge Aware :

[0196]

[0197] Where Sobel(·) represents the Sobel operator, which is used to extract edge information of the image; Sobel(P) iRepresents the edge strength of the i-th pixel in the predicted image; Sobel(T) i Represents the edge strength of the i-th pixel of the true label.

[0198] The third calculation unit is used to calculate the total loss L of the Swin-Unet model according to the following formula Swin-Unet :

[0199] L Swin-Unet =α·L Binary Dice +β·L Edge Aware .

[0200] Among them, α is the weight of the binary Dice loss; β is the weight of the edge-aware loss.

[0201] For more specific working processes of the above modules, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0202] Example 3

[0203] This embodiment provides a computer device, including a processor and a memory; wherein the processor implements the steps of the pavement crack detection method described in Embodiment 1 when executing a computer program stored in the memory.

[0204] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0205] Example 4

[0206] This embodiment provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the pavement crack detection method described in Embodiment 1 are implemented.

[0207] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0208] Example 5

[0209] This embodiment provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the steps of the pavement crack detection method described in Embodiment 1 are implemented.

[0210] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0211] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the systems, devices, storage media, and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0212] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0213] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0214] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0215] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0216] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A road crack detection method, characterized in that: include: Acquire multiple pavement crack images; Build the Swin-Unet model; Determine the total loss of the Swin-Unet model using multiple pavement crack images; The total loss of the Swin-Unet model is used to train the Swin-Unet model, and the trained Swin-Unet model is used as a pavement crack detection model; The pavement crack image to be detected is input into the pavement crack detection model, and the pavement crack detection model outputs the pavement crack detection result.

2. The pavement crack detection method according to claim 1, characterized in that: The construction of the Swin-Unet model includes: Constructing an encoder module, which uses a Swin Transformer block to extract features from the pavement crack image, segmenting the pavement crack image into non-overlapping image blocks, and then processing each image block to obtain its embedding vector sequence, and combining them to obtain the overall sequence embedding to form an overall feature representation; Constructing an intermediate connection module, wherein the intermediate connection module is used to further perform feature learning and extract deep features, and the intermediate connection module is composed of two consecutive Swin Transformer blocks; Construct a decoder module, which includes a Swin Transformer block and an image block expansion layer. The decoder is used to process the decoding task of sequence embedding and gradually restore the resolution of the original image to generate a segmentation result. By introducing skip connections in the model structure, features at different levels are fused and integrated to fuse the extracted contextual features with the multi-scale features of the encoder.

3. The pavement crack detection method according to claim 2, characterized in that: The encoder module is constructed, and the encoder module uses the Swin Transformer block to extract features from the pavement crack image, including: Step 211, dividing the road crack image into image blocks of a preset pixel size, and the image blocks do not overlap each other; Step 212, mapping the feature dimension of the image block to the spatial dimension C by applying a linear embedding layer; Step 213, using two consecutive SwinTransformer blocks to process the feature map, in the process keeping the feature dimension and resolution unchanged; Step 214, merging adjacent image blocks through an image merging layer to reduce computational complexity and further extract multi-scale features, reducing the spatial resolution to a multiple of the original, while increasing the number of feature channels to generate a feature map with a more compact representation; Step 215, repeating steps 213 and 214 three times in the encoder to generate a hierarchical feature representation; Step 216, applying a linear layer to expand the feature dimension, applying a linear layer to the connected features, introducing a linear transformation, and expanding the connected feature dimension to a preset multiple of the original dimension.

4. The pavement crack detection method according to claim 2, characterized in that: The decoder module is constructed, and the decoder module includes two parts: a Swin Transformer block and an image block expansion layer. The decoder is used to process the decoding task of sequence embedding and gradually restore the resolution of the original image to generate the segmentation result, including: Step 231, using the Swin Transformer block and the image block expansion layer to process the sequence embedding decoding task; Step 232, fusing features at different levels through skip connections, combining the encoder's multi-scale features with context information to compensate for the spatial information lost during the downsampling process; Step 233, using an image block expansion layer to process the input feature map, gradually increasing its spatial resolution to restore the original spatial resolution of the feature map; Step 234, using the Swin Transformer block to extract high-level features and gradually restore the spatial information of the input image to generate an accurate segmentation result; Step 235, repeating step 232, step 233 and step 234 three times to gradually restore the resolution of the feature map and extract richer context information; Step 236, applying a linear projection layer on the obtained feature map to generate the final pixel-level segmentation prediction result.

5. The pavement crack detection method according to claim 1, characterized in that: The method of using a plurality of pavement crack images to determine the total loss of the Swin-Unet model includes: The binary Dice loss L is calculated according to the following formula BinaryDice : Where N is the total number of pixels in the pavement crack image; P i is the predicted category probability corresponding to pixel i in the pavement crack image; T i is the category label corresponding to pixel i in the pavement crack image; smooth represents the smoothness term; The edge-aware loss L is calculated according to the following formula EdgeAware : Where Sobel(·) represents the Sobel operator, which is used to extract edge information of the image; Sobel(P) i Represents the edge strength of the i-th pixel in the predicted image; Sobel(T) i Represents the edge strength of the i-th pixel of the true label; The total loss L of the Swin-Unet model is calculated according to the following formula Swin-Unet : L Swin-Unet =α·L BinaryDice +β·L EdgeAware ; Among them, α is the weight of the binary Dice loss; β is the weight of the edge-aware loss.

6. A road crack detection system, characterized in that: include: An acquisition module, used for acquiring a plurality of pavement crack images; Building module, used to build Swin-Unet model; a determination module for determining a total loss of a Swin-Unet model using a plurality of pavement crack images; A model training module, used to train the Swin-Unet model using the total loss of the Swin-Unet model, and use the trained Swin-Unet model as a pavement crack detection model; The crack detection module is used to input the pavement crack image to be detected into the pavement crack detection model, and obtain the pavement crack detection result output by the pavement crack detection model.

7. The pavement crack detection system according to claim 6, characterized in that: The building blocks include: A first construction unit is used to construct an encoder module, wherein the encoder module uses a Swin Transformer block to extract features from a pavement crack image, divides the pavement crack image into non-overlapping image blocks, and then processes each image block to obtain its embedding vector sequence, and obtains the overall sequence embedding by combining to form an overall feature representation; A second construction unit is used to construct an intermediate connection module, wherein the intermediate connection module is used to further perform feature learning and extract deep features, and the intermediate connection module is composed of two consecutive Swin Transformer blocks; The third construction unit is used to construct a decoder module, which includes a Swin Transformer block and an image block expansion layer. The decoder is used to process the decoding task of sequence embedding and gradually restore the resolution of the original image to generate a segmentation result. The feature fusion unit is used to fuse and integrate features at different levels by introducing skip connections in the model structure to fuse the extracted context features with the multi-scale features of the encoder.

8. The pavement crack detection system according to claim 7, characterized in that: The second building block comprises: An image segmentation device, used for segmenting the road crack image into image blocks of preset pixel size, wherein the image blocks do not overlap each other; Dimension mapping means for mapping the feature dimension of the image block to a spatial dimension C by applying a linear embedding layer; a first image block processing device for processing the feature map using two consecutive Swin Transformer blocks, while maintaining the feature dimension and resolution unchanged; The second image block processing device is used to merge adjacent image blocks through a picture merging layer to reduce the computational complexity and further extract multi-scale features, reduce the spatial resolution to a multiple of the original, and increase the number of channels of the feature to generate a feature map with a more compact representation; A feature representation generating device, configured to repeat the operations of the first image block processing device and the second image block processing device three times in the encoder to generate a hierarchical feature representation; The feature dimension expansion device is used to apply a linear layer to expand the feature dimension. The linear layer is applied to the connected features, and a linear transformation is introduced to expand the connected feature dimension to a preset multiple of the original dimension.

9. The pavement crack detection system according to claim 7, characterized in that: The third building block comprises: A decoding task processing device for processing a decoding task of sequence embedding using a Swin Transformer block and an image block expansion layer; A feature fusion device for fusing features at different levels through skip connections, combining the multi-scale features of the encoder with context information to compensate for the spatial information lost during the downsampling process; A first resolution recovery device is used to process the input feature map using an image block expansion layer to gradually increase its spatial resolution to restore the original spatial resolution of the feature map; A segmentation result generating device, for extracting high-level features using a SwinTransformer block and gradually restoring the spatial information of the input image to generate an accurate segmentation result; A second resolution recovery device, used to repeat the operations of the feature fusion device, the first resolution recovery device and the segmentation result generation device three times to gradually recover the resolution of the feature map and extract richer context information; The prediction result generating device is used to apply a linear projection layer on the obtained feature map to generate a final pixel-level segmentation prediction result.

10. The pavement crack detection system according to claim 6, characterized in that: The determination module comprises: The first calculation unit is used to calculate the binary Dice loss L according to the following formula BinaryDice : Where N is the total number of pixels in the pavement crack image; P i is the predicted category probability corresponding to pixel i in the pavement crack image; T i is the category label corresponding to pixel i in the pavement crack image; smooth represents the smoothness term; The second calculation unit is used to calculate the edge perception loss L according to the following formula EdgeAware : Where Sobel(·) represents the Sobel operator, which is used to extract edge information of the image; Sobel(P) i Represents the edge strength of the i-th pixel in the predicted image; Sobel(T) i Represents the edge strength of the i-th pixel of the true label; The third calculation unit is used to calculate the total loss L of the Swin-Unet model according to the following formula Swin-Unet : L Swin-Unet =α·L BinaryDice +β·L EdgeAware ; Among them, α is the weight of the binary Dice loss; β is the weight of the edge-aware loss.

Images