A highway pavement crack detection method based on improved RT-DETR

Abstract

The present application discloses a method for detecting cracks in a highway pavement based on an improved RT-DETR, and relates to the field of pavement disease detection. The method includes constructing a pavement crack sample set based on a pavement crack image; improving the RT-DETR model based on a high- and low-frequency attention mechanism; training the improved RT-DETR model based on the pavement crack sample set to obtain a crack detection model; detecting the collected pavement image based on the crack detection model to determine the target crack in the pavement image. The present invention effectively solves the problem of difficulty in extracting crack features under complex pavement conditions, and at the same time, it is necessary to process interference factors such as noise, shadows, and textures in the image, which leads to poor versatility and accuracy in pavement crack recognition, and achieves the technical effect of improving the accuracy of pavement crack feature extraction and unifying pavement disease detection standards.

Description

Technical Field

[0001] The present application relates to the field of pavement disease detection, and in particular to a highway pavement crack detection method based on improved RT-DETR. Background Art

[0002] With the increasing traffic volume on highways, as well as the impact of rolling, impact, wear and tear of vehicles, weather changes, etc., a series of road surface defects have occurred, such as cracks, fissures and repairs. These defects have a great impact on the stability of the road surface.

[0003] Cracks are common and can be divided into transverse cracks and longitudinal cracks. In related technologies, in order to identify road cracks, digital image processing technology is used to perform preprocessing, segmentation, feature extraction and recognition on road images, thereby achieving automatic detection and analysis of cracks. Alternatively, the image is gridded and cut into many small blocks to detect whether a small block in the image has cracks, and then the detection results of all small blocks are mapped to the entire image to detect the location and type of cracks on the entire image.

[0004] However, the above technologies have the problem of difficulty in extracting crack features in complex road environments. At the same time, it is necessary to deal with interference factors such as noise, shadows, and textures in the image, resulting in poor versatility and accuracy in road crack recognition. Summary of the invention

[0005] The main purpose of this application is to provide a highway pavement crack detection method based on improved RT-DETR, aiming to solve the problem of difficulty in crack feature extraction in complex road environments. At the same time, it is necessary to deal with interference factors such as noise, shadows, textures in the image, which leads to technical problems such as poor versatility and accuracy of pavement crack recognition.

[0006] To achieve the above objectives, the present application proposes a road pavement crack detection method based on improved RT-DETR, the road pavement crack detection method based on improved RT-DETR comprising:

[0007] Construct a pavement crack sample set based on pavement crack images;

[0008] Improve the RT-DETR model based on high and low frequency attention mechanism;

[0009] The improved RT-DETR model is trained based on the pavement crack sample set to obtain a crack detection model;

[0010] The collected road surface image is detected based on the crack detection model to determine the target cracks in the road surface image.

[0011] In one embodiment, the step of constructing a pavement crack sample set based on the pavement crack image includes:

[0012] Determine a grayscale image corresponding to the pavement crack image;

[0013] Performing light uniformity processing on the grayscale image based on an image threshold of brightness balance;

[0014] The processed grayscale image is labeled to construct the pavement crack sample set.

[0015] In one embodiment, the pavement crack sample set is divided into a training set, a validation set, and a test set according to a preset ratio;

[0016] In one embodiment, the step of improving the RT-DETR model based on the high- and low-frequency attention mechanism includes:

[0017] Construct an improved high- and low-frequency attention module;

[0018] Determine a replacement module based on replacing the AIFI attention module in the RT-DETR model with the improved high- and low-frequency attention module;

[0019] Construct an improved multi-scale feature fusion module;

[0020] The CCFM feature fusion module in the RT-DETR model is replaced by the improved multi-scale feature fusion module, and the replacement module is determined.

[0021] In one embodiment, the step of constructing an improved high- and low-frequency attention module includes:

[0022] Build a dot-product attention module;

[0023] Build a high-frequency attention module based on the dot-product attention module;

[0024] Construct a low-frequency attention module based on the dot-product attention module;

[0025] Construct a high- and low-frequency feature fusion module based on an iterative fusion strategy;

[0026] Construct high- and low-frequency attention modules.

[0027] In one embodiment, the step of constructing an improved multi-scale feature fusion module includes:

[0028] Construct an iterative feature fusion module;

[0029] An improved feature fusion module is constructed based on the iterative feature fusion module.

[0030] In one embodiment, the step of training the improved RT-DETR model based on the pavement crack sample set to obtain a crack detection model includes:

[0031] Determining training parameters, wherein the training parameters include batch size, learning rate, number of detection categories, and maximum number of iterations;

[0032] Inputting the pavement crack sample set into the improved RT-DETR model, calculating the gradient based on the loss function and back propagation to update the gradient in real time;

[0033] When the maximum number of iterations is reached, the crack detection model is determined based on the iteration result.

[0034] In one embodiment, the step of inputting the pavement crack sample set into the improved RT-DETR model, calculating the gradient based on the loss function and back propagation, and updating the gradient in real time includes:

[0035] Determining a feature map of a preset scale based on the pavement crack sample set and the backbone of the improved RT-DETR model;

[0036] Determining tensor data based on the feature map and the middle portion of the improved RT-DETR model;

[0037] The tensor data is input into the output part of the improved RT-DETR model, and the gradient is calculated in combination with the loss function and the back propagation to update the gradient in real time.

[0038] One or more technical solutions proposed in this application have the following technical effects or advantages:

[0039] The pavement crack sample set is constructed based on the pavement crack image; the RT-DETR model is improved based on the high-frequency attention module, and the high-frequency and low-frequency attention mechanisms are used to extract and fuse high-frequency features that are beneficial to strip crack detection and low-frequency features that are beneficial to block crack detection, respectively. Then, a high-frequency and low-frequency feature fusion module based on an iterative fusion strategy is constructed to fuse high-frequency and low-frequency features; the improved RT-DETR model is trained based on the pavement crack sample set to obtain a crack detection model; the collected pavement image is detected based on the crack detection model to determine the target crack in the pavement image. The problem of difficulty in crack feature extraction in a complex pavement environment is effectively solved, and the technical problem of poor versatility and accuracy of pavement crack recognition due to interference factors such as noise, shadows, and textures in the image is required to be processed, achieving the technical effect of improving the accuracy of pavement crack feature extraction and unifying the pavement disease detection standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of an embodiment of a method for detecting cracks on a highway pavement based on improved RT-DETR of the present application;

[0041] Figure 2 This is a comparison diagram of the effects before and after the uniform light operation in an embodiment of the highway pavement crack detection method based on the improved RT-DETR of this application. Figure 2 (a) is the effect picture before the uniform light operation. Figure 2 (b) is the effect diagram after the uniform light operation;

[0042] Figure 3 This is a schematic diagram of the high and low frequency attention modules in an embodiment of the highway pavement crack detection method based on improved RT-DETR of this application;

[0043] Figure 4 This is a schematic diagram of an iterative high- and low-frequency feature fusion module in an embodiment of a highway pavement crack detection method based on improved RT-DET in this application;

[0044] Figure 5 This is a transverse crack detection effect diagram of an embodiment of a highway pavement crack detection method based on improved RT-DET in this application. Figure 5 (a) is the RT-DETR detection effect diagram, Figure 5 (b) is the improved RT-DETR detection effect diagram;

[0045] Figure 6 This is a longitudinal crack detection effect diagram of an embodiment of a highway pavement crack detection method based on improved RT-DET in this application. Figure 6 (a) is the RT-DETR detection effect diagram, Figure 6 (b) is the improved RT-DETR detection effect diagram;

[0046] Figure 7 This is a diagram showing various crack detection effects in an embodiment of a road pavement crack detection method based on improved RT-DET in this application. Figure 7 (a) is the RT-DETR detection effect diagram, Figure 7 (b) is the improved RT-DETR detection effect diagram. DETAILED DESCRIPTION

[0047] At present, the detection methods for road pavement cracks mainly include road pavement crack detection methods based on manual annotation, detection methods based on traditional image processing, detection methods based on computer vision classification, detection methods based on traditional target detection, etc. However, there are some problems with the above detection methods. When facing crack detection in complex road conditions, the crack detection effect is poor due to the insufficient crack feature extraction ability of the model. For example, it is necessary to deal with interference factors such as noise, shadows, and textures in the image, and there are certain assumptions and restrictions on the shape and type of cracks. At the same time, the minimum unit of detection is the size of a single small block. When the difference between the crack area and the area of ​​the smallest block is large or the crack is just on the dividing line, the error will be relatively large.

[0048] The main technical solutions adopted by the embodiment of the present application are: constructing a pavement crack sample set based on pavement crack images; improving the RT-DETR model based on the high- and low-frequency attention modules; training the improved RT-DETR model based on the pavement crack sample set to obtain a crack detection model; detecting the collected pavement images based on the crack detection model to determine the target cracks in the pavement images. This achieves the technical effect of strengthening the model's ability to extract crack features and effectively improving crack detection accuracy.

[0049] In order to better understand the above technical solution, exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0050] The first embodiment of the present application discloses a road pavement crack detection method based on improved RT-DETR, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of a highway pavement crack detection method based on improved RT-DETR of the present application.

[0051] In this embodiment, the highway pavement crack detection method based on improved RT-DETR includes steps S100 to S400:

[0052] Step S100: constructing a pavement crack sample set based on the pavement crack image.

[0053] In this embodiment, crack image data is collected, a large number of road pavement crack images are collected, and annotated to construct a road pavement crack sample set. Traditional image processing methods or deep learning image segmentation methods can be used for annotation to obtain the location and shape information of each crack.

[0054] In one embodiment, step S100 includes:

[0055] Step S110: determining a grayscale image corresponding to the pavement crack image.

[0056] In this embodiment, image segments of crack areas are extracted from the road surface image as training samples. The image can be collected using a drone, a driving recorder, or a handheld camera. The collected road crack image is converted into a grayscale image, that is, a grayscale image is obtained using image processing software or a programming library.

[0057] Step S120: performing light homogenization processing on the grayscale image based on the image threshold of brightness balance.

[0058] In this embodiment, the grayscale image is enhanced based on the brightness equalization method to improve the contrast and details of the image. The brightness equalization is performed by using methods such as histogram equalization. The enhanced grayscale image is threshold processed to divide the image into two categories: background and foreground. The crack area is separated from the background by setting a suitable threshold.

[0059] Step S130: annotating the processed grayscale image to construct the pavement crack sample set.

[0060] In this embodiment, the grayscale image after threshold processing is annotated, and the pixels in the crack area are marked as crack categories. The outline or boundary box of the crack is drawn using the drawing function in the image processing software or programming library. The image fragments after crack annotating and the corresponding labels are saved to construct a pavement crack sample set. The sample set is saved as an image file and a label file, or stored in a specific data format.

[0061] As an optional implementation, a grayscale image of a pavement crack image with a size of 2048*2048 is obtained, and the image is homogenized using an image threshold segmentation method based on brightness balance; the homogenized image is annotated using LabelImg software to construct a pavement crack sample set. LabelImg is an open source image annotation tool used to manually mark objects in an image.

[0062] For example, a road image collection vehicle equipped with a high-definition camera collects road images of different road types, different weather conditions, and different time periods to increase the richness of the original image. Then, the image is uniformed using the image threshold segmentation method based on brightness balance, where the principle of the image threshold segmentation method based on brightness balance is as follows:

[0063] Set a Image, grayscale level (0, ..., L). The average brightness of the image Then:

[0064] (Formula 1)

[0065] in, The coordinates in the image are The pixel brightness value of The image is divided into sub-blocks, is the image height, is the image width. In this embodiment = , then the mean brightness of each sub-block is for:

[0066] (Formula 2)

[0067] Then the difference between the mean brightness of the sub-block and the whole image is , it is obvious that the high brightness sub-block in an image is positive, and the low brightness sub-block To make the brightness of the whole image equal, the brightness of each sub-block needs to be adjusted. The positive sub-blocks are attenuated, and The negative sub-blocks are enhanced.

[0068] The comparison of the effects before and after the uniform light operation in this step is shown in the figure below. Figure 2 shown.

[0069] As an optional implementation, after the pavement crack sample set is determined, the pavement crack sample set is divided into a training set, a validation set, and a test set according to a ratio of 7:2:1.

[0070] Step S200: Improve the RT-DETR model based on the high and low frequency attention mechanism.

[0071] In one embodiment, step S200 includes:

[0072] Step S210: Construct an improved high- and low-frequency attention module, whose network structure is as follows Figure 3 shown.

[0073] In this embodiment, the improved high-low frequency attention module is a multi-head attention mechanism, the core of which is to use part of the attention heads to extract high-frequency features, and the remaining attention heads to extract low-frequency features, and then fuse the extracted features.

[0074] As an optional implementation, an improved high- and low-frequency attention module is designed and constructed according to the needs and actual conditions. The purpose of this module is to extract high-frequency features that are beneficial to strip crack detection through the high-frequency attention submodule, and to extract low-frequency features that are beneficial to crack detection using the low-frequency attention submodule, and to improve the performance and effect of the RT-DETR model in crack detection tasks through the high- and low-frequency attention modules.

[0075] Step S220: Determine the replacement module based on replacing the AIFI attention module in the RT-DETR model with the improved high- and low-frequency attention module.

[0076] As an optional implementation, the target module in the RT-DETR model is replaced, and by analyzing the structure and function of the RT-DETR model, it is determined that the target module to be replaced is the AIFI attention module.

[0077] In this embodiment, after completing the above replacement, an improved RT-DETR model is generated based on the original RT-DETR model.

[0078] Similarly, the code is modified and adjusted as needed to replace the AIFI attention module. After the AIFI attention module is replaced, the improved RT-DETR model is generated by saving the modified network structure and parameters or retraining the network.

[0079] Step S230: constructing an improved multi-scale feature fusion module.

[0080] Step S240: replacing the multi-scale feature fusion module (CCF) in the RT-DETR model based on the improved multi-scale feature fusion module, and determining a replacement module.

[0081] Through the above optional implementation methods and embodiments, the RT-DETR model can be improved and optimized, and the performance and accuracy of the crack detection model can be improved. The introduction of the improved multi-scale feature fusion module can increase the expression and generalization capabilities of the network, thereby improving the detection and identification capabilities of cracks.

[0082] Further, step S210 includes:

[0083] Step S211: Construct a dot product attention module.

[0084] In this embodiment, a dot product attention module is first constructed, and then a high-frequency and low-frequency attention module are constructed based on the dot product attention module. The algorithm principle is as follows:

[0085] (Formula 3)

[0086] Where Q (query), K (key), V (value) are the input vectors of the dot product attention mechanism DA; softmax represents the normalized exponential function, represents the transpose of K, d k Represents the dimension of the K vector.

[0087] Step S212: Construct a high-frequency attention module based on the dot product attention module.

[0088] In this embodiment, a high-frequency attention submodule is constructed based on the dot product attention mechanism. First, each attention head is queried by inputting feature X. ,key Sum ( ), the algorithm principle is as follows:

[0089] (Formula 4)

[0090] in, are learnable network parameters, is the learnable network parameter used by the h-th attention head to map the input feature X to the query subspace, is the learnable network parameter used by the h-th attention head to map the input feature X to the key subspace, are the learnable network parameters used by the h-th attention head to map the input feature X to the value subspace.

[0091] Then use multi-head attention to calculate high-frequency features , the algorithm principle is as follows:

[0092] (Formula 5)

[0093] Among them, N h represents the total number of heads of high and low frequency attention mechanisms, It means to traverse and calculate the high-frequency attention of each attention head, and then concatenate the calculation results. DA means to use the dot product attention mechanism to perform dot product attention operation. Indicates using a linear layer to project the calculation result to the same dimension as the input.

[0094] Step S213: Construct a low-frequency attention module based on the dot product attention module.

[0095] In this embodiment, a low-frequency attention submodule is constructed based on the dot product attention mechanism. First, a low-frequency signal of the input feature X is obtained through a low-frequency filter. , and then each attention head uses the input feature X to calculate the query, using the low-frequency signal Calculate the keys and values, and finally use the dot product attention mechanism to calculate and concatenate the low-frequency features of each attention head.

[0096] The algorithm principle for obtaining low-frequency signals is as follows:

[0097] (Formula 6)

[0098] Avgpool represents the average pooling operation, k and s represent the window and step size of the average pooling operation; Conv represents the use of a 1×1 convolutional layer to extract the features of the average pooling calculation results to improve the nonlinear expression ability.

[0099] Calculate low-frequency queries ,key ,value The algorithm is as follows:

[0100] (Formula 7)

[0101] in is a learnable parameter. Finally, the dot product attention mechanism is used to calculate and concatenate the low-frequency features of each attention head. , the specific calculation method is as follows:

[0102] (Formula 8)

[0103] in, It means to traverse and calculate the high-frequency attention of each attention head, and then concatenate the calculation results.

[0104] Step S214: construct a high- and low-frequency feature fusion module based on an iterative fusion strategy.

[0105] By building a feature fusion module to completely fuse the input features. Specifically, Figure 4 As shown in the figure, the input features are first concatenated, and then the features are fused using a 1x1 convolution layer and N cascaded iterative blocks. Subsequently, a residual structure is constructed to prevent feature loss caused by the network being too deep. Compared with simple average fusion, the iterative feature fusion module can iteratively fuse features and prevent the loss of useful information.

[0106] Step S215: Construct high and low frequency attention modules.

[0107] The high- and low-frequency attention module uses the low-frequency attention sub-module to extract low-frequency features from the input features, uses the high-frequency attention sub-module to extract high-frequency features, and then uses the iterative feature fusion module to fuse the high- and low-frequency features.

[0108] Further, step S230 includes:

[0109] Step S231: multi-scale feature selection. According to a large number of experiments, it is concluded that pavement cracks are mainly medium-scale targets and large-scale targets. Based on this, the outputs of the last two residual blocks of the backbone network backbone (the feature sizes are: 40*40*512; 20*20*1024) are selected for multi-scale feature fusion.

[0110] Step S232: top-down feature fusion, upsampling the small-scale features and then fusing them with the large-scale features, aiming to integrate the semantic information of the small-scale features into the large-scale features.

[0111] Step S233: bottom-up feature fusion, down-sampling the features fused from top to bottom, and then fusing them with small-scale features, aiming to fuse the detail information of large-scale features into small-scale features.

[0112] Step S300: training the improved RT-DETR model based on the pavement crack sample set to obtain a crack detection model.

[0113] As an optional implementation, based on the improved RT-DETR network architecture, RT-DETR is improved and optimized for the highway pavement crack detection task. The performance of the model in detecting highway pavement cracks can be improved by adjusting the network parameters such as the number of layers, number of channels, and receptive field. Then the crack detection model is trained, and the improved RT-DETR model is trained using the crack sample set. An end-to-end training method is used to input the image into the network, and the parameters are updated and optimized by calculating the loss function until the model converges. Common optimizers, such as Adam, and loss functions, such as cross entropy loss, can be used for training.

[0114] Furthermore, when improving the RT-DETR model, snake convolution is used to improve the RT-DETR model based on the characteristics of pavement cracks such as linear structure and long-distance dependence.

[0115] In one embodiment, step S300 includes:

[0116] Step S310: determining training parameters, wherein the training parameters include batch size, learning rate, number of detection categories and maximum number of iterations.

[0117] Step S320: input the pavement crack sample set into the improved RT-DETR model, calculate the gradient based on the loss function and back propagation, and update the gradient in real time.

[0118] In this embodiment, the pavement crack sample set is divided into a training set, a validation set and a test set in advance, and the part corresponding to the training set is input into the improved RT-DETR model.

[0119] Step S330: When the maximum number of iterations is reached, the crack detection model is determined based on the iteration result.

[0120] As an optional implementation, after dividing the pavement crack sample set into a training set, a validation set, and a test set, the corresponding part of the training set is input into the improved RT-DETR model, and according to the needs and computing resources, a suitable improved RT-DETR model is selected as the backbone network of the crack detection model. Prepare pre-trained weights, and use weights pre-trained on large-scale image data, or use weights pre-trained on other similar tasks, and determine training parameters such as batch size, learning rate, number of detection categories, and maximum number of iterations. Set the data enhancement method, and use enhancement techniques such as random cropping, translation, and rotation to increase the generalization ability of the model. Define a loss function suitable for the crack detection task, and commonly use a combination of IOU loss function and classification loss function. Set different weight coefficients to balance the loss of the bounding box position and the loss of classification to improve the performance of the model. In the model training process, the pavement crack sample set is input into the improved RT-DETR model. The gradient is calculated based on the loss function and back propagation to update the gradient in real time. The training samples are continuously iterated until the maximum number of iterations is reached. In the model evaluation process, an independent validation set is used to evaluate the performance of the model on the crack detection task. Calculate the model's accuracy, recall, F1 score and other indicators to evaluate the model's performance in crack detection. According to the evaluation results, adjust the training parameters and data enhancement methods, and repeat the above steps until the optimal model performance is achieved. When the maximum number of iterations is reached and the performance requirements are met, save the weights and configuration of the crack detection model. Apply the model to the crack detection scenario, and by inputting a new road image, use the crack detection model to detect and mark the crack area in the image.

[0121] Further, step S320 includes:

[0122] Step S321: determining a feature map of a preset scale based on the pavement crack sample set and the main part of the improved RT-DETR model;

[0123] Step S322: determining tensor data based on the feature map and the middle part of the improved RT-DETR model;

[0124] Step S323: Input the tensor data into the output part of the improved RT-DETR model, and calculate the gradient in combination with the loss function and the back propagation to update the gradient in real time.

[0125] As an optional implementation method, set the training parameters: batch size, learning rate, number of detection categories and maximum number of iterations; input the images of the training set in the pavement crack sample set into the backbone network to obtain feature maps of two different scales (feature size is 40*40*512; 20*20*1024); input the feature maps of two different scales into the improved multi-scale feature fusion module for upsampling and feature fusion, and obtain tensor data of two different scales (tensor size is 40*40*512; 20*20*1024); input the tensor data of two different scales into the Head part, calculate the gradient based on the loss function and back propagation, update the gradient in real time and use the verification set in the pavement crack sample set to verify the network accuracy. Repeat the steps, and after reaching the set maximum number of iterations, the improved RT-DETR model completes the learning and obtains the pavement crack detection model and weights.

[0126] Further, a feature map of a preset scale is determined, and the pavement crack sample set is input into the backbone of the improved RT-DETR model. In the backbone, the spatial size of the input image is gradually reduced through operations such as convolution and pooling, and a hierarchical feature map is extracted. The feature map of the preset scale required for crack detection is determined, and multiple feature map scales can be selected to detect different sizes of cracks. Tensor data is determined, and based on the feature map, the feature map is converted into tensor data through the middle part of the improved RT-DETR model. In the middle part, the feature map can be further processed using operations such as convolution and full connection to extract higher-level feature representations. The gradient is calculated and updated, and the tensor data is input into the output part of the improved RT-DETR model. In the output part, the tensor data is usually mapped to a preset number of categories using operations such as convolution and full connection. Combined with the loss function, the difference between the predicted result and the actual label is calculated. Using the back-propagation algorithm, the gradient is calculated, and the network parameters are updated according to the gradient. An optimization algorithm such as stochastic gradient descent is used to guide the gradient update process. The above steps are repeated until all samples are used to update the parameters of the network.

[0127] Specifically, before training the improved RT-DETR model, it is necessary to adjust the parameters involved according to the characteristics of different sample sets to find the optimal parameter values, which is very important for the performance of the detection model and the speed of training. The following are the hyperparameters set based on the crack sample set:

[0128] First, set the batch size. Batch size (BS) refers to the number of samples in each iteration, which determines the direction of gradient descent. Generally, a larger BS leads to faster network convergence, but it is easy to fall into the local optimal solution, resulting in poor model generalization; a smaller BS leads to greater randomness, and the loss function is prone to oscillation and difficult to converge. Therefore, after multiple rounds of parameter adjustment, the optimal BS is 8.

[0129] Set the learning rate. The learning rate is a critical hyperparameter in model training. It is used to update the weight step size and balance the convergence speed and whether it converges. If the learning rate is too small, the gradient descent step is small and the network converges slowly; if the learning rate is too high, the loss function will oscillate. Usually, the learning rate is initialized to 0.001, and a variable learning rate is used to train the model.

[0130] Set the number of detection categories. The number of detection categories is the number of detection types involved in the model. The pavement crack sample set involves three types of cracks: transverse cracks, longitudinal cracks, and block cracks, so the number of detection categories is set to 3.

[0131] Set the maximum number of iterations. The maximum number of iterations (Max epochs) refers to the maximum number of iterations that can be performed during model training. After exceeding the maximum number of iterations, model training is terminated and the weights are no longer updated. In this embodiment, it is set to 300.

[0132] Step S400: Detecting the collected road surface image based on the crack detection model to determine target cracks in the road surface image.

[0133] As an optional implementation, the crack detection model is tested and verified, and the trained model is tested and verified using a verification set to evaluate the model's crack detection accuracy and recall rate on the highway pavement image. The accuracy and stability of the detection results can be improved by adjusting the model's threshold or using a non-maximum suppression method. Then, the crack detection system is used to apply the trained crack detection model to the actual highway pavement image. The collected pavement image is input, and the crack detection model is used to detect and locate and mark the target cracks in the image.

[0134] For example, highway pavement images are collected from different locations, different angles and different lighting conditions. Cracks in the images are manually annotated using professional image processing software and saved as crack images. The RT-DETR model is improved by adding specific convolutional layers to the backbone network to extract more targeted features. The number of channels and receptive field size are adjusted to adapt to the characteristics of cracks in highway pavement. The crack image set is divided into a training set and a validation set. The images are preprocessed, including image size unification, data augmentation (such as random cropping, flipping, rotation, etc.) and normalization operations. The improved RT-DETR model is trained using the training set. Appropriate hyperparameters such as learning rate, batch size, etc. are set, and the model is optimized iteratively using the training set. The trained model is tested and verified using the validation set. Evaluation indicators such as the model's precision, recall and F1 value are calculated to evaluate the model's performance on the crack detection task. The trained model is applied to actual highway pavement images.

[0135] Furthermore, a road surface image is input, and the model is used to detect and locate the target cracks. The location and size of the cracks can be marked with a box. The performance of the crack detection system is evaluated using a real highway road surface image dataset. The differences between the detection results and the annotation results are compared, and the evaluation indicators are calculated to evaluate the accuracy and robustness of the system.

[0136] Specifically, as an example of step S400 in this embodiment, the road surface crack detection model is used to detect the collected road surface image. If cracks are found in the recognition result, the crack position and crack type are marked in the image, and the crack position information and crack type are written into a txt file for storage for subsequent use. The transverse crack detection effect of the image is as follows: Figure 5 As shown in Figure 1, Figure (a) is the RT-DETR detection effect diagram, and Figure (b) is the improved RT-DETR detection effect diagram. The longitudinal crack detection effect is shown in Figure 1. Figure 6 As shown in Figure 1, Figure (a) is the RT-DETR detection effect diagram, and Figure (b) is the improved RT-DETR detection effect diagram. Figure 7 As shown in the figure, Figure (a) is the RT-DETR detection effect diagram, and Figure (b) is the improved RT-DETR detection effect diagram. Finally, the model was tested on two different data sets. Data set 1: mAP (average precision of all categories) increased from 0.532 to 0.577. Data set 2: mAP increased from 0.698 to 0.729.

[0137] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A highway pavement crack detection method based on improved RT-DETR, characterized in that: The highway pavement crack detection method based on improved RT-DETR includes: Construct a pavement crack sample set based on pavement crack images; Improve the RT-DETR model based on high and low frequency attention mechanism; The improved RT-DETR model is trained based on the pavement crack sample set to obtain a crack detection model; Detecting the collected road surface image based on the crack detection model to determine target cracks in the road surface image; The steps of improving the RT-DETR model based on the high and low frequency attention mechanism include: Step S210: constructing an improved high- and low-frequency attention module; The improved high-low frequency attention module is a multi-head attention mechanism, the core of which is to use part of the attention heads to extract high-frequency features, and the remaining attention heads to extract low-frequency features, and then fuse the extracted features; Step S220: replacing the AIFI attention module in the RT-DETR model based on the improved high- and low-frequency attention module, and determining the replacement module; Replace the target module in the RT-DETR model. By analyzing the structure and function of the RT-DETR model, it is determined that the target module to be replaced is the AIFI attention module. After completing the above replacement, an improved RT-DETR model is generated based on the original RT-DETR model; Modify and adjust the code as needed, replace the AIFI attention module, and after completing the replacement of the AIFI attention module, generate an improved RT-DETR model by saving the modified network structure and parameters, or retraining the network; Step S230: constructing an improved multi-scale feature fusion module; Step S240: replacing the multi-scale feature fusion module in the RT-DETR model based on the improved multi-scale feature fusion module, and determining a replacement module; The step S210 includes: Step S211: construct a dot product attention module; First, we build a dot product attention module, and then build high-frequency and low-frequency attention modules based on the dot product attention module, as follows: ； Where Q, K, V are the input vectors of the dot product attention mechanism DA; softmax represents the normalized exponential function, represents the transpose of K, d k Represents the dimension of the K vector; Step S212: constructing a high-frequency attention module based on the dot product attention module; First, each attention head calculates the query by inputting feature X ,key Sum , as follows: ； in, are learnable network parameters, is the learnable network parameter used by the h-th attention head to map the input feature X to the query subspace, is the learnable network parameter used by the h-th attention head to map the input feature X to the key subspace, is the learnable network parameter used by the h-th attention head to map the input feature X to the value subspace; Then use multi-head attention to calculate high-frequency features , as follows: ； Among them, N h represents the total number of heads of high and low frequency attention mechanisms, It means to traverse and calculate the high-frequency attention of each attention head, and then concatenate the calculation results. DA means to use the dot product attention mechanism to perform dot product attention operation. Indicates that a linear layer is used to project the calculation result to the same dimension as the input; Step S213: constructing a low-frequency attention module based on the dot product attention module; First, a low-frequency filter is used to obtain the low-frequency signal of the input feature X. , and then each attention head uses the input feature X to calculate the query, using the low-frequency signal Calculate the keys and values, and finally use the dot product attention mechanism to calculate and concatenate the low-frequency features of each attention head; The specific details of obtaining the low-frequency signal are as follows: ； Avgpool represents the average pooling operation, k and s represent the window and step size of the average pooling operation; Conv represents the use of a 1×1 convolutional layer to extract the features of the average pooling calculation results to improve the nonlinear expression ability; Calculate low-frequency queries ,key ,value The details are as follows: ； in is a learnable parameter, and finally the dot product attention mechanism is used to calculate and concatenate the low-frequency features of each attention head , the specific calculation method is as follows: ； Then concatenate the calculation results; Step S214: constructing a high- and low-frequency feature fusion module based on an iterative fusion strategy; First, the input features are concatenated, and then a 1x1 convolution layer and N cascaded iterative blocks are used to fuse the features. Finally, a residual structure is constructed to prevent feature loss caused by too deep a network. Step S215: construct high and low frequency attention modules; The high- and low-frequency attention module uses the low-frequency attention submodule to extract low-frequency features from the input features, uses the high-frequency attention submodule to extract high-frequency features, and then uses the iterative feature fusion module to fuse the high- and low-frequency features; The step S230 includes: Step S231: Multi-scale feature selection. Road cracks are mainly medium-scale targets and large-scale targets. Based on this, the outputs of the last two residual blocks of the backbone network are selected to perform multi-scale feature fusion. The feature sizes are: 40*40*512; 20*20*1024, Step S232: top-down feature fusion, upsampling the small-scale features, and then fusing them with the large-scale features, so as to integrate the semantic information of the small-scale features into the large-scale features; Step S233: bottom-up feature fusion, down-sampling the features fused from top to bottom, and then fusing them with small-scale features to fuse the detail information of large-scale features into small-scale features.

2. The highway pavement crack detection method based on improved RT-DETR as claimed in claim 1, characterized in that: The step of constructing a pavement crack sample set based on the pavement crack image comprises: Determine a grayscale image corresponding to the pavement crack image; Performing light uniformity processing on the grayscale image based on an image threshold of brightness balance; The processed grayscale image is labeled to construct the pavement crack sample set.

3. The highway pavement crack detection method based on improved RT-DETR as claimed in claim 1, characterized in that: The step of training the improved RT-DETR model based on the pavement crack sample set to obtain a crack detection model comprises: Determining training parameters, wherein the training parameters include batch size, learning rate, number of detection categories, and maximum number of iterations; Inputting the pavement crack sample set into the improved RT-DETR model, calculating the gradient based on the loss function and back propagation to update the gradient in real time; When the maximum number of iterations is reached, the crack detection model is determined based on the iteration result.

4. The highway pavement crack detection method based on improved RT-DETR as claimed in claim 3, characterized in that: The step of inputting the pavement crack sample set into the improved RT-DETR model, calculating the gradient based on the loss function and back propagation, and updating the gradient in real time includes: Determining a feature map of a preset scale based on the pavement crack sample set and the backbone of the improved RT-DETR model; Determining tensor data based on the feature map and the middle portion of the improved RT-DETR model; The tensor data is input into the output part of the improved RT-DETR model, and the gradient is calculated in combination with the loss function and the back propagation to update the gradient in real time.

Images