Lightweight highway pavement defect detection method based on improved RT-DETR

Abstract

The invention discloses a lightweight highway pavement defect detection method based on improved RT-DETR. The method comprises the following steps: acquiring a highway pavement defect data set and dividing the highway pavement defect data set into a training set, a verification set and a test set; designing a lightweight highway pavement defect detection model of the improved RT-DETR; training the obtained highway pavement defect data set by using the improved RT-DETR network to obtain a trained weight; and using the trained weight for image processing of the test set by using the improved RT-DETR network, and obtaining a detection result. According to the invention, the problem of slow detection speed caused by large calculated amount and parameter amount of the existing highway pavement defect detection model is solved, and the success rate and accuracy of the highway pavement defect detection task are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and relates to road defect detection using target detection, deep learning and other technologies, and in particular to a lightweight highway pavement defect detection method based on improved RT-DETR. Background Art

[0002] Computer vision technology can achieve automatic analysis and understanding of images by simulating the human visual system. In highway pavement defect detection, computer vision technology can automatically identify and locate defects such as longitudinal cracks, transverse cracks, potholes, cracks, and repair areas on the road surface, improving the accuracy and efficiency of detection; target detection technology is an important branch in the field of computer vision, and has strong robustness for many complex images. By training deep learning models, the target detection system will learn the characteristics of these defects and accurately identify them in actual detection; deep learning technology is an important tool in the field of computer vision. It uses the constructed deep neural network model to automatically learn the feature representation in the image, automatically analyze and process the images in the highway pavement defect detection task, extract the feature information of the defects, and realize the automatic identification and location of the defects.

[0003] In the actual highway pavement defect detection task, researchers at home and abroad also face many difficulties. First, the current highway pavement defect detection dataset has problems such as insufficient data volume and lack of data diversity, which affects the generalization ability of the model; second, the traditional detection model has a large amount of calculation and parameters, which will lead to a slow detection speed and cannot meet the needs of real-time detection; at the same time, due to the complex and changeable traffic conditions, the traditional detection model is prone to misjudgment and missed judgment; finally, small target defects (such as tiny cracks) on the highway pavement are difficult to detect and identify in traditional detection models.

[0004] In recent years, the method of deep fusion of regression-based object detection algorithms with other technologies has been widely used. Although the YOLO series has become the most popular framework for real-time object detection, non-maximum suppression has a negative impact on its accuracy and speed. RT-DETR based on the Transformer architecture reduces the computational cost of DETR with its end-to-end approach and intra-scale feature interaction and cross-scale feature fusion modules, and outperforms the YOLO detector of the same scale in both speed and accuracy. However, despite the good performance of the RT-DETR model in defect detection tasks, it still has certain limitations in the detection of complex traffic conditions and small target defects. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a lightweight highway pavement defect detection method based on improved RT-DETR, so as to improve the detection speed and detection accuracy at the same time and avoid misjudgment and missed judgment.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A lightweight highway pavement defect detection method based on improved RT-DETR includes the following steps:

[0008] S1, construct a highway pavement defect dataset and annotate the image samples in the dataset;

[0009] S2, designing a lightweight highway pavement defect detection model based on improved RT-DETR; the improved RT-DETR refers to making one or more of the following improvements to the RT-DETR model:

[0010] Improvement 1: In the backbone network of the RT-DETR model, the ResNet18 network is rebuilt using the lightweight module RGCSPELAN;

[0011] Improvement 2: In the hybrid encoder of the RT-DETR model, the efficient additive attention module EAA is introduced to improve the multi-head self-attention mechanism of the intra-scale feature interaction module AIFI, thereby replacing the matrix multiplication operation with the element operation and eliminating the explicit key-value interaction;

[0012] Improvement 3: introduce feature pyramid CGRFPN as the cross-scale feature fusion module of RT-DETR;

[0013] S3, using the highway pavement defect dataset to train the lightweight highway pavement defect detection model, collecting images and using the trained model to perform highway pavement defect detection.

[0014] In one embodiment, the road surface defect dataset S1 is from various road surface defect image data collected by different types of vehicles. The defect types include longitudinal cracks, transverse cracks, potholes, cracks and repair areas. Labelimg is used to annotate the dataset and divide it into training set, validation set and test set.

[0015] In one embodiment, the improvement 1, the ResNet18 network reconstructed using the lightweight module RGCSPELAN is composed of several lightweight modules RGCSPELAN and Conv modules alternately connected, the lightweight module RGCSPELAN uses RepConv for efficient convolution on the gradient flow branch, is fused during inference, and the size is controlled using a scaling factor.

[0016] In one embodiment, the lightweight module RGCSPELAN first performs a 1×1 convolution operation on the input defect image, and then performs a Split operation to divide it into two parts. One part first performs a RepConv convolution operation, then performs n-1 3×3 convolution operations, and finally performs a 1×1 convolution operation. The image processed by the RepConv convolution, n-1 3×3 convolution operations and one 1×1 convolution operation is spliced ​​with the other part of the image after Split; finally, a 1×1 convolution operation is performed on the spliced ​​image.

[0017] In one embodiment, the improvement 2 introduces an efficient additive attention module EAA to improve the multi-head self-attention mechanism of the intra-scale feature interaction module AIFI, by introducing EAA in AIFI, where EAA uses linear element multiplication operations instead of quadratic matrix multiplication operations to perform key-value interactions without sacrificing accuracy; the query matrix is ​​multiplied by learnable weights and aggregated to generate a global query; then, the matrix K is element-wise multiplied by the broadcasted global query to obtain a global context representation.

[0018] In one embodiment, the efficient additive attention module EAA encodes the relevance score of the input sequence context information based on the interaction of the three attention components Q, K, and V; first, the key-value interaction is removed without sacrificing performance, and only the relationship between the query key interaction learning tags is effectively encoded by merging the linear projection layer; second, the input embedding matrix x is used by two matrices W q , W k Transformed into query Q and key K, where Q, K∈R n×d , W q ,W k ∈R d×d , n is the token length, d is the dimension of the embedding vector; Next, multiply the query Q by the learnable parameter vector w α ∈R d , learn the query attention weight to obtain the global attention query vector α∈R n ; Then, the query Q is pooled according to the learned attention weights to obtain a single global query vector q∈R d , the formula is as follows:

[0019]

[0020] Next, we use the element-wise product to solve the global query vector q∈R d and the key matrix, i.e., key K∈R n×d The interactions between them are encoded to form a global context representation R n×d Finally, a linear transformation layer is used to query the key interaction to learn the hidden representation of the token, effectively adding attention The output formula is as follows:

[0021]

[0022] in, represents the normalized query matrix and T represents the linear transformation.

[0023] In one embodiment, the feature pyramid CGRFPN is used for spatial feature reconstruction and pyramid context extraction, capturing global context in the horizontal and vertical directions of the road defect detection task, and obtaining the axial global context to explicitly model the rectangular key area; through multi-scale feature fusion and dynamic interpolation, the feature expression ability of the model is further improved, and its recognition ability for defect detection tasks under complex backgrounds is enhanced.

[0024] In one embodiment, the feature pyramid CGRFPN first uses a matrix self-calibration module RCM to capture axial context in the horizontal and vertical directions, gradually downsamples the encoder to generate features of different scales [F1, F2, F3, F4] and resolutions [H / 4×H / 4, H / 8×H / 8, H / 16×H / 16, H / 32×H / 32], and abandons the largest feature F1; generates a pyramid feature F5 through downsampling and concatenation operations, and inputs it into multiple stacked RCMs for pyramid feature interaction; secondly, uses RCM to capture the axial global context to model the rectangular key area, and uses the shape self-calibration function to convert The attention area is adjusted to the foreground; then, the matrix self-calibration attention RCA uses horizontal pooling and vertical pooling to capture the axial global context in two directions, generates two different axis vectors, uses horizontal strip convolution to calibrate the shape in the horizontal direction, BN normalizes the features, ReLU adds nonlinearity, and uses vertical strip convolution to perform shape self-calibration in the vertical direction. Then, a 3×3 depth convolution is used to further extract the local details of the input features, and the corrected attention features are weighted to the refined input features; finally, the rectangular self-calibration attention is combined with the MetaNeXt structure, and the residual connection is used to further enhance feature reuse.

[0025] In one embodiment, the shape self-calibration is expressed as:

[0026]

[0027] Among them, ψ represents large kernel strip convolution, k represents the kernel size of strip convolution, φ represents batch normalization, and δ represents the Sigmoid function;

[0028] The corrected attention features are weighted onto the refined input features through Hadamard convolution, and the formula is expressed as:

[0029] ξ F (x,y)=ψ 3×3 (x)⊙y (9)

[0030] Among them, ψ 3×3 represents a 3×3 depth convolution with a kernel; Y is the attention feature obtained in the previous step; ⊙ represents a Hadamard convolution;

[0031] The final output of the feature pyramid CGRFPN is expressed as:

[0032]

[0033] in, Indicates broadcast addition; H p and V p denote horizontal pooling and vertical pooling respectively; ρ denotes BN and MLP operations, and x denotes the input feature map.

[0034] In one embodiment, the ResNet18 network reconstructed using the lightweight module RGCSPELAN is sequentially connected by Conv1, Conv2, RGCSPELAN1, Conv3, RGCSPELAN2, Conv4, RGCSPELAN3, Conv5, and RGCSPELAN4; the hybrid encoder uses the outputs of RGCSPELAN2, RGCSPELAN3, and RGCSPELAN4 as input features;

[0035] The hybrid encoder after introducing the efficient additive attention module EAA includes:

[0036] RCM1, DIF1, FBM1, PCE, GIO1, GIO2, GIO3, FBM2, RCM2, DIF2, Conv6, AIFI-EAA, Conv7, RCM3, FBM3, Conv8, Concat1, RepConv1, Conv9, Concat2, and RepConv2;

[0037] The output of RGCSPELAN2 is divided into two paths, one of which is connected to the input of RCM1. The output of RGCSPELAN3 is divided into two paths, one of which is connected to the input of RCM2. The output of RGCSPELAN4 is connected to the input of Conv6. The output of Conv6 is connected to AIFI-EAA. The output of AIFI-EAA is connected to Conv7. The output of Conv7 is divided into two paths, one of which is connected to the input of RCM3, and the other path and the other path of the output of RGCSPELAN2 and the other path of the output of RGCSPELAN3 are all connected to the input of PCE. The output of PCE is divided into three paths, which are respectively connected to the inputs of GIO1, GIO2, and GIO3. The output of GIO3 and the output of RCM3 are connected to FBM3. The output of FBM3 is divided into two paths, one of which is connected to the output of RCM2. DIF2, the output of DIF2 and the output of GIO2 are connected to FBM2, the output of FBM2 is divided into two paths, one path and the output of RCM1 are connected to DIF1, the output of DIF1 and the output of GIO1 are connected to FBM1, the output of FBM1 is divided into two paths, one path is used as decoder input, and the other path is connected to the input of Conv8, the other path of FBM2 output and the output of Conv8 are connected to Concat1, the output of Concat1 is connected to RepConv1, the output of RepConv1 is divided into two paths, one path is used as decoder input, and the other path is connected to the input of Conv9, the other path of FBM3 output and the output of Conv9 are connected to Concat2, the output of Concat2 is connected to RepConv2, and the output of RepConv2 is used as decoder input.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The present invention reconstructs the backbone network of ResNet18 through the RGCSPELAN module to achieve lightweight model. In highway pavement defect detection, the complexity of the detection environment (such as complex traffic conditions and diverse road conditions) places high demands on the real-time and accuracy of the detection model. The RGCSPELAN module significantly reduces the amount of calculation and parameters of the traditional detection model, improves the detection speed, and enables the model to respond quickly and accurately identify road defects in complex traffic environments. At the same time, while maintaining high-precision feature extraction capabilities, the lightweight model also reduces the demand for computing resources, making the application of non-contact detection technology (such as cameras, lidar, etc.) in highway pavement defect detection more extensive and efficient.

[0040] 2. The efficient additive attention EAA introduced in the present invention reduces the computational cost and improves the detection accuracy by improving the intra-scale feature interaction module. This feature enables the model to more accurately capture the characteristics of small target defects (such as fine cracks, small potholes, etc.) in complex and changeable road pavement environments, and improves the detection and recognition capabilities of small target defects. In addition, the introduction of EAA also enhances the model's adaptability to changes in pavement materials, lighting conditions, etc. In emergency detection, this high-precision and rapid response capability can quickly locate and deal with potential safety hazards.

[0041] 3. The present invention uses the efficient feature pyramid CGRFPN to redesign the intra-scale feature interaction module to achieve global context information extraction and multi-scale feature fusion. On the one hand, this feature enables the model to more comprehensively capture the context information of highway pavement defects during the detection process, and improves the ability to recognize defect features of different scales. On the other hand, it enhances the model's ability to distinguish target features in complex backgrounds, and reduces misjudgments and missed judgments caused by background interference from pavement defects. In regular inspections, this module can ensure that the model accurately identifies various types of pavement defects and provide strong support for highway maintenance and repair. At the same time, it reduces the workload of manual review and improves detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the implementation of the present invention or the existing technical solutions, the following briefly introduces the drawings required to be used in the embodiments or the description of the prior art.

[0043] Figure 1 It is a flow chart of the detection method of the present invention.

[0044] Figure 2 It is the structural diagram of the improved RT-DETR of the present invention.

[0045] Figure 3 It is a structural diagram of the lightweight module RGCSPELAN designed in the present invention.

[0046] Figure 4 It is a structural diagram of the efficient additive attention EAA of the present invention.

[0047] Figure 5 It is a structural diagram of CGRSeg of the present invention.

[0048] Figure 6 It is a structural diagram of the feature pyramid CGRFPN designed in the present invention.

[0049] Figure 7 It is a comparison diagram of the actual scene detection effects of the baseline model and the improved model in the embodiment of the present invention.

[0050] Figure 8It is the result of the thermal map experiment in the embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following is a clear and complete description of the technical solutions in the embodiments of the invention in conjunction with the drawings in the embodiments of the invention. Obviously, the described embodiments are only part of the embodiments of the invention, not all of the embodiments. Based on the embodiments of the invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the invention.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0053] refer to Figure 1 The present invention provides a lightweight highway pavement defect detection method based on improved RT-DETR, the main steps of which are as follows:

[0054] Step 1: Construct a highway pavement defect dataset and annotate the image samples in the dataset.

[0055] Step 2: Improve the original RT-DETR model by making it lightweight, and obtain a lightweight highway pavement defect detection model based on the improved RT-DETR, which is applied to the highway pavement defect detection task.

[0056] Step 3: Use the highway pavement defect dataset constructed in step 1 to train the lightweight highway pavement defect detection model designed in step 2 to obtain the trained weights.

[0057] Step 4: In the lightweight highway pavement defect detection model, the weights obtained in step 3 are used to detect and process the images of the newly acquired test set and obtain the detection results.

[0058] As a preferred embodiment of the present invention, in step 1, a data set can be acquired by shooting during movement through a vehicle-mounted camera or a vehicle-mounted mobile phone device, or a mature existing data set can be directly used. In this embodiment, the highway pavement defect data set ROAD DEFECT is used, which is a self-built data set after processing a public data set and collecting a data set by itself, and the data set is labeled using LabelImg. The data set comes from various types of highway pavement defect image data collected by different types of cars, including longitudinal cracks, transverse cracks, potholes, cracks, repair areas and other categories, and is divided into training set, verification set and test set according to 7:2:1 or other feasible ratios. The training set is used to train the model; the verification set is used to provide feedback on the training of the model, and the weight parameters of the model are saved according to the performance of the model on the verification set, which is used to load during testing and reasoning detection; the test set is mainly used for the final evaluation of the trained model.

[0059] As a preferred embodiment of the present invention, refer to Figure 2 In step 2, the lightweight highway pavement defect detection model based on the improved RT-DETR of the present invention makes one or more of the following improvements to the RT-DETR model:

[0060] Improvement 1: In the backbone network of the RT-DETR model, the lightweight module RGCSPELAN is used to rebuild the ResNet18 network to improve the feature extraction capability of the model and reduce the amount of calculation and parameters.

[0061] The present invention uses the newly constructed ResNet18 network as the backbone network of the detection model, which is composed of several lightweight modules RGCSPELAN and Conv modules connected alternately, such as Figure 2 The backbone network in has 4 lightweight modules RGCSPELAN and 5 Conv modules. The first two Conv modules are connected in sequence. Starting from the second Conv module, the 4 Conv modules and the 4 lightweight modules RGCSPELAN are alternately arranged and connected in sequence. Starting from the second lightweight module RGCSPELAN, the three lightweight modules RGCSPELAN provide one output respectively.

[0062] The lightweight module RGCSPELAN uses RepConv to perform efficient convolution on the gradient flow branch, performs fusion during inference, and uses a scaling factor to control the size. The lightweight module RGCSPELAN has a CSP-like structure for Split operations. As a preferred embodiment of the present invention, refer to Figure 3, its execution process includes: first, due to the limitation of storage and computing resources, the application of convolutional neural networks (CNNs) on embedded devices is very difficult. Not only does it cause redundancy in the intermediate feature maps of mainstream CNN calculations, but it also increases the amount of calculation and parameters. Therefore, this module refers to the idea of ​​GhostNet, uses cheap operations to generate a part of redundant feature maps, reduces the negative impact of extensive redundant information generated by the intermediate feature maps in mainstream CNN calculations, and reduces the amount of calculation and parameters of the model. Specifically, a 1×1 convolution operation is first performed on the input defect image, and then a Split operation is performed to divide it into two parts. Secondly, BottleNeck, which is commonly used in YOLOv5 and YOLOv8, is discarded. In order to compensate for the performance loss caused by discarding the residual block, RepConv is used on the gradient flow branch for efficient convolution to enhance the ability of feature extraction and gradient flow, and RepConv can be fused during inference to avoid the performance loss caused by discarding the residual block and enhance the ability of feature extraction and gradient flow. BottleNeck is often used in YOLOv5 and YOLOv8 modules. BottleNeck in YOLOv5 module has only 2 layers of convolution, and the last layer no longer uses 1x1 convolution to increase dimension, but directly outputs. Specifically, a part of it first performs RepConv convolution operation on the gradient flow branch to replace the residual block structure, which is used for lightweight processing. RepConv provides a good backbone design idea. During training, multiple branches are used to improve feature extraction capabilities, so that gradients can flow better, and during reasoning, multiple branches are re-parameterized and merged into one path. Therefore, in order to compensate for the performance loss caused by discarding the residual block, RepConv convolution operation is used; then n-1 3×3 convolution operations are performed, n-1 because the previous RepConv has already occupied one; finally, using the idea that there is extensive redundancy in the intermediate feature map calculated by the mainstream CNN in GhostNet, a 1×1 convolution operation is performed to reduce the amount of calculation and the amount of parameters. The image processed by RepConv convolution, n-1 3×3 convolution operations and one 1×1 convolution operation is concatenated with the other part of the image after Split; finally, a 1×1 convolution operation is performed on the concatenated image.

[0063] Taking n=3 as an example, the module first performs a 1×1 convolution operation on the input defect image. Then the Split operation is performed to divide it into two parts. One part performs a RepConv convolution operation, then two 3×3 convolution operations, and then a 1×1 convolution operation. Among them, the image processed by the RepConv convolution, two 3×3 convolution operations and one 1×1 convolution operation will be spliced ​​with the other part of the image after the Split operation. Finally, the spliced ​​image is subjected to a 1×1 convolution operation.

[0064] According to the above theory, when outputting n feature maps, RGCSPELAN has an identity mapping (i.e., m feature maps are obtained by traditional convolution) and linear operations (n=m·s), and the convolution kernel size of each linear operation is d·d, k=d, s is much smaller than c, so the theoretical speedup ratio of RGCSPELAN is as follows:

[0065]

[0066] Its parameter compression ratio is:

[0067]

[0068] Among them, s represents the saving multiple of training time, s and k are adjustable hyperparameters, s is used to adjust the number of feature maps of the traditional convolution method, and d is the size of the linear transformation convolution kernel.

[0069] As a preferred embodiment of the present invention, Figure 2 In order to increase operability, the size of RGCSPELAN is controlled by the scaling factor. Specifically, each RGCSPELAN in the backbone network can modify the parameters according to the effect of the modified defect image detection, that is, scale. When the modified parameters are small, the corresponding amount of calculation and parameters will be reduced, the reasoning time will be faster, but the accuracy may be reduced. When the modified parameters are large, the corresponding amount of calculation and parameters will increase, the reasoning time will be slower, but the accuracy will be improved. This design can take into account both small and large models.

[0070] Improvement 2: In the hybrid encoder of the RT-DETR model, the efficient additive attention module EAA is introduced to improve the multi-head self-attention mechanism of the intra-scale feature interaction module AIFI, thereby replacing expensive matrix multiplication operations with element operations and eliminating explicit key-value interactions, thereby improving the ability to detect and identify small target defects on highway pavements.

[0071] As a preferred embodiment of the present invention, refer to Figure 4, Since RT-DETR's AIFI still uses a multi-head self-attention structure, the expensive matrix multiplication operation limits its use in real-time applications when performing defect detection tasks on many resource-constrained mobile devices. In this regard, the efficient additive attention module EAA is introduced in the multi-head self-attention mechanism of the intra-scale feature interaction module AIFI, that is, EAA is introduced in AIFI. In the standard form, the attention mechanism performs three steps to model query, key, and value interactions. Each step provides information for the subsequent steps, so it needs to be processed sequentially. EAA improves the multi-head self-attention mechanism of AIFI, which can use linear element multiplication operations instead of quadratic matrix multiplication operations, reducing the need to calculate explicit interactions between key values, and performing key-value interactions without sacrificing accuracy; the query matrix is ​​multiplied by learnable weights and pooled to produce a global query; then, the matrix K is element-wise multiplied by the broadcasted global query to obtain a global context representation. EAA only encodes the global query and key interactions to learn global context information, and then performs a linear transformation to calculate the global context-aware attention weights. At the same time, it has linear complexity in terms of token length and can be merged in all stages to learn consistent global context at each scale. This consistency improves the performance of the model, making it more general and scalable to high-resolution images, and better encodes contextual representations by replacing them with simple linear transformations.

[0072] The introduction process of this step includes: Self-attention has become the de facto choice for capturing global context in various visual applications. However, its quadratic computational complexity relative to the image resolution limits its use in real-time applications, especially deployment on resource-constrained mobile devices. Although hybrid methods have been proposed to combine the advantages of convolution and self-attention to achieve a better speed-accuracy trade-off, the expensive matrix multiplication operation in self-attention remains a bottleneck. The typical additive attention mechanism in NLP exploits pairwise interactions between tokens through element-wise multiplication instead of using dot product operations to capture global context. The efficient additive attention module EAA encodes the relevance score of the contextual information of the input sequence based on the interaction of three attention components (Q, K, V). Figure 1 In the AIFI-EAA module obtained by introducing EAA into AIFI, the key-value interaction can be removed without sacrificing performance, and it is sufficient to learn the relationship between tags by effectively encoding the query-key interaction by merging linear projection layers. The efficient additive attention module EAA has faster inference speed and produces more robust contextual representation. Specifically, the input embedding matrix x is represented by two matrices W q , W k Transformed into query (Q) and key (K), where Q, K∈R n×d , W q ,W k ∈Rd×d , n is the token length, and d is the dimension of the embedding vector. Next, the query matrix Q is multiplied by the learnable parameter vector w α ∈R d , learn the attention weights of the query and obtain the global attention query vector α∈R n as follows:

[0073]

[0074] The query matrix Q is then pooled according to the learned attention weights to obtain a single global query vector q∈R d , as shown below:

[0075]

[0076] Next, we use the element-wise product to solve the global query vector q = R d and the key matrix K∈R n×d The interactions between n×d ). This matrix has similarities to the attention matrix in MHSA, can capture information from each token, and can flexibly learn correlations in the input sequence. However, compared with MHSA, it is relatively computationally expensive and its complexity is linear in the token length. To this end, the present invention adopts a linear transformation layer to query the key interaction to learn the hidden representation of the token. Effective Additional Attention The output can be described as:

[0077]

[0078] in, represents the normalized query matrix and T represents the linear transformation.

[0079] Improvement 3: introduce the feature pyramid CGRFPN as the cross-scale feature fusion module CCFM of RT-DETR.

[0080] In order to solve the problems of insufficient global context information capture and poor multi-scale feature expression ability of the cross-scale feature fusion module when facing complex backgrounds and defect detection tasks of different scales, the present invention uses the feature pyramid CGRFPN to reconstruct spatial features and extract pyramid context, captures the global context in the horizontal and vertical directions of the pavement defect detection task, and obtains the axial global context to explicitly model the rectangular key area; through multi-scale feature fusion and dynamic interpolation, the feature expression ability of the model is further improved, and its recognition ability for defect detection tasks under complex backgrounds is enhanced.

[0081] The feature pyramid CGRFPN has the characteristics of global context information extraction and multi-scale feature fusion. As a preferred embodiment of the present invention, refer to Figure 5 and Figure 6 The execution process of the designed feature pyramid CGRFPN is as follows: First, a rectangular self-calibration module RCM is designed for spatial feature reconstruction and pyramid context extraction. RCM captures axial global context in horizontal and vertical directions to explicitly model the rectangular key area; secondly, a shape self-calibration function, namely the shape self-calibration attention RCA of the present invention, is designed to make the key area closer to the foreground object. Pyramid context extraction uses RCM to extract pyramid context, and uses horizontal pooling and vertical pooling to capture axial context from two directions; in addition, RCM uses MLP to further enhance feature representation. The rectangular self-calibration module RCM is designed for spatial feature reconstruction and pyramid context extraction. The rectangular self-calibration attention RCA explicitly models the rectangular area and calibrates the attention shape. The different scale features generated by the progressively down-sampling encoder are [F1, F2, F3, F4], with resolutions of [H / 4×H / 4, H / 8×W / 8, H / 16×W / 16, H / 32×W / 32], respectively. In order to ensure the efficiency of the entire network, the decoder abandons the largest scale feature, namely F1. Then, the low-scale features F2, F3, and F4 are downsampled to the size of H / 64×W / 64 by average pooling, and they are concatenated together to generate the pyramid feature F5. F5 is input into multiple stacked RCMs for pyramid feature interaction to extract scale-aware semantic features; finally, after pyramid feature extraction, the features are segmented and upsampled to the original scale. This process can be expressed as:

[0082] P=RCM(AP(F2,8),AP(F3,4),AP(F4,2)) (16)

[0083] Where AP(F, x) represents the average pooling of feature F downsampled by factor x. P is the pyramid context.

[0084] In order to make the decoder features more focused on the foreground, RCM is used to reconstruct the spatial features. RCM captures the axial global context to model the rectangular key area, and then the shape self-calibration function is used to adjust the attention area to the foreground. Rectangular Self-Calibration Attention RCA uses horizontal pooling and vertical pooling to capture the axial global context in two directions and generate two different axis vectors. By broadcasting and adding these two vectors, RCA can effectively model the rectangular region of interest. Then, a shape self-calibration function is designed to calibrate the region of interest so that the region of interest is closer to the foreground target. The shape is calibrated in the horizontal direction using horizontal strip convolution, and each row element is adjusted to make the horizontal shape closer to the foreground object. Then, the features are normalized using BN and nonlinearity is added using ReLU. Subsequently, the shape is calibrated in the vertical direction using vertical strip convolution. The shape self-calibration function can be formulated as follows:

[0085]

[0086] Where ψ represents the large kernel strip convolution, k represents the kernel size of the strip convolution. φ represents batch normalization followed by the ReLU function, and δ represents the Sigmoid function. In addition, a feature fusion function is designed to fuse the attention features with the input features. Specifically, a 3×3 depth convolution is used to further extract the local details of the input features.

[0087] The corrected attention features are weighted onto the refined input features through Hadamard convolution, and the formula is as follows:

[0088] ξ F (x,y)=ψ 3×3 (x)⊙y (18)

[0089] where ψ 3×3 Represents a deep convolution with a kernel of 3×3. Y is the attention feature obtained in the previous step. ⊙ is the Hadamard convolution. Rectangular self-calibration attention is combined with the MetaNeXt structure to further improve the feature representation. Specifically, after the rectangular self-calibration attention, batch normalization and MLP are added to refine the features. Finally, residual connections are used to further enhance feature reuse.

[0090] according to Figure 6 , the output process of CGRFPN is:

[0091] The pyramid context feature P in formula (6) is split again according to the number of channels before merging, and the number of channels is the same as the original, but the resolution is downsampled to the original The fusion features P3′, P4′ and P5′

[0092] First, P5′ is upsampled to the input Then perform FMB operation (dot multiplication) with P5 after RCM module. The process can be expressed as: Output1 = upsample(P5′)*RCM(P5)

[0093] Then, Output1 is upsampled to the input And perform DIF (addition) operation with P4 processed by RCM module, and then upsample to P4′ of the same resolution through RCM module for FBM operation (dot multiplication). The process can be expressed as: Output2 = upsample(P4′)*RCM(upsample(Output1)+RCM(P4))

[0094] Finally, Output2 is upsampled to the input And perform DIF (addition) operation with P3 processed by RCM module, and then upsample to P3′ of the same resolution through RCM module for FBM operation (dot multiplication). The process can be expressed as:

[0095] Output3=upsample(P3′)*RCM(upsample(Output2)+RCM(P3))

[0096] The final feature pyramid CGRFPN output is divided into three parts from top to bottom:

[0097] The first part is Output3;

[0098] The second part is the result of performing Conv convolution on Output3, concatenating it with Output2, and then processing it with reparameterized convolution RepC3.

[0099] The third part is the result of Conv convolution on the second part, Concat concatenation with Output1, and processing with reparameterized convolution RepC3.

[0100] It can also be described as:

[0101]

[0102] in Indicates broadcast addition. P and V P Represents horizontal pooling and vertical pooling respectively. ρ is the BN and MLP operations.

[0103] In a specific embodiment of the present invention, the constructed detection model refers to Figure 2As shown in the figure, it mainly includes a backbone network, a hybrid encoder and a decoder. The backbone network is a ResNet18 network reconstructed using the lightweight module RGCSPELAN, which is connected in sequence by Conv1, Conv2, RGCSPELAN1, Conv3, RGCSPELAN2, Conv4, RGCSPELAN3, Conv5, and RGCSPELAN4. The hybrid encoder after introducing the efficient additive attention module EAA includes: RCM1, DIF1, FBM1, PCE, GIO1, GIO2, GIO3, FBM2, RCM2, DIF2, Conv6, AIFI-EAA, Conv7, RCM3, FBM3, Conv8, Concat1, RepConv1, Conv9, Concat2, and RepConv2. Among them, in the hybrid encoder, except for Conv6, AIFI-EAA and Conv7, the remaining modules constitute the main components of the feature pyramid CGRFPN.

[0104] The hybrid encoder takes the output of RGCSPELAN2, RGCSPELAN3, and RGCSPELAN4 as input features. The specific process is as follows:

[0105] The output of RGCSPELAN2 is divided into two paths, one of which is connected to the input of RCM1. The output of RGCSPELAN3 is divided into two paths, one of which is connected to the input of RCM2. The output of RGCSPELAN4 is connected to the input of Conv6. The output of Conv6 is connected to AIFI-EAA. The output of AIFI-EAA is connected to Conv7. The output of Conv7 is divided into two paths, one of which is connected to the input of RCM3, and the other path and the other path of the output of RGCSPELAN2 and the other path of the output of RGCSPELAN3 are all connected to the input of PCE. PCE is used for pyramid context extraction, which effectively integrates different levels of feature information of highway pavement defects (such as subtle texture and overall structure, etc.), improves the context perception ability of the model, and enables the model to understand the road conditions more accurately.

[0106] Since the operation of the PCE module is relatively complex, the present invention uses three GIOs, namely GIO1, GIO2, and GIO3, and uses the GIO function to obtain the three outputs of PCE. That is, the output of PCE is divided into three paths, which are respectively connected to the inputs of GIO1, GIO2, and GIO3. The output of GIO3 and the output of RCM3 are connected to FBM3. The output of FBM3 is divided into two paths, one of which is connected to DIF2 with the output of RCM2, and the output of DIF2 and the output of GIO2 are connected to FBM2. The output of FBM2 is divided into two paths, one of which is connected to DIF1 with the output of RCM1, and the output of DIF1 and the output of GIO1 are connected to FBM1. DIF is used here for dynamic interpolation processing to solve the problem of different input content sizes. It ensures that no matter how the size of the road surface image changes, it can be effectively processed and is not affected by the difference in image size.

[0107] FBM is used for the fusion of multiple features, and can efficiently integrate feature information from different sources and dimensions (such as color, texture, shape, etc.), realize comprehensive and detailed capture of road surface defect features, and improve the accuracy and robustness of detection. In the present invention, the output of FBM1 is divided into two paths, one path is used as the decoder input, and the other path is connected to the input of Conv8, the other path of FBM2 output and the output of Conv8 are connected to Concat1, the output of Concat1 is connected to RepConv1, the output of RepConv1 is divided into two paths, one path is used as the decoder input, and the other path is connected to the input of Conv9, the other path of FBM3 output and the output of Conv9 are connected to Concat2, the output of Concat2 is connected to RepConv2, and the output of RepConv2 is used as the decoder input.

[0108] Therefore, the decoder has three inputs, from top to bottom: ① It focuses on the detection of large defect targets. Large defects (such as large-scale damage, severe cracks, etc.) occupy a large pixel area in the image, with obvious features and easy to identify. Through lower resolution representation and strong feature expression capabilities, it can quickly capture the overall features of large defects and achieve efficient detection in the decoder. ② It focuses on the detection of medium-sized defect targets. Medium-sized defects (such as medium-width cracks, large potholes, etc.) occupy a certain pixel area in the image, have sufficient feature information, and require certain contextual information to assist judgment. By balancing resolution and feature expression capabilities, it can well capture the features of medium-sized defects and accurately identify the location and type of these defects in the decoder. ③ It focuses on the detection of small defect targets. Since small defects (such as fine cracks, small potholes, etc.) occupy fewer pixels in the image, the details are rich but the overall features are not obvious, so through high-resolution representation capabilities, it can capture these subtle feature changes, and can accurately locate and identify small defects in the decoder, providing key information for subsequent defect classification and repair.

[0109] As a preferred embodiment of the present invention, refer to Figure 7 , in step 4, (a) and (b) represent the actual scene detection effect diagrams of the baseline model and the improved model respectively. In Figure (a), the interference of external conditions such as dense vehicles, volatile weather conditions, and more small target defects will cause the baseline model to have false detection and missed detection when performing defect detection tasks, and the confidence is low. In Figure (b), although the backbone network abandons the common residual block, which will cause performance loss, the use of RepConv on the gradient flow branch will enhance the ability of feature extraction and gradient flow, allowing them to be integrated during reasoning, while reducing the amount of calculation and the number of parameters, maintaining performance and speeding up reasoning; the introduction of the EAA attention mechanism uses element operations instead of expensive matrix multiplication operations, and eliminates explicit key-value interactions, which improves the improved model's ability to detect and identify small target defects; the designed CGRFPN module captures global context in the horizontal and vertical directions, and obtains axial global context to explicitly model rectangular key areas. By effectively integrating feature information at different levels and multi-scale feature fusion and dynamic interpolation, the probability of false detection and missed detection of the improved model is further reduced.

[0110] As a preferred embodiment of the present invention, refer to Figure 8 ,In order to more intuitively analyze the emphasis of the model on the target features, ,the heat map of the improved model is drawn.,The improved model can pay more attention to the defective parts of the road surface, ,and the dark red area fits the target contour better, ,while extracting the target features, it reduces the extraction of background ,information.

Claims

1. A lightweight highway pavement defect detection method based on improved RT-DETR, characterized in that: The steps include: S1, construct a highway pavement defect dataset and annotate the image samples in the dataset; S2, designing a lightweight highway pavement defect detection model based on improved RT-DETR; the improved RT-DETR refers to making one or more of the following improvements to the RT-DETR model: Improvement 1: In the backbone network of the RT-DETR model, the ResNet18 network is rebuilt using the lightweight module RGCSPELAN; Improvement 2: In the hybrid encoder of the RT-DETR model, the efficient additive attention module EAA is introduced to improve the multi-head self-attention mechanism of the intra-scale feature interaction module AIFI, thereby replacing the matrix multiplication operation with the element operation and eliminating the explicit key-value interaction; Improvement 3: introduce feature pyramid CGRFPN as the cross-scale feature fusion module of RT-DETR; S3, using the highway pavement defect dataset to train the lightweight highway pavement defect detection model, collecting images and using the trained model to perform highway pavement defect detection.

2. The lightweight highway pavement defect detection method based on improved RT-DETR according to claim 1 is characterized in that: The S1, highway pavement defect dataset comes from various types of highway pavement defect image data collected by different types of vehicles. The defect types include longitudinal cracks, transverse cracks, potholes, cracks and repair areas. Labelimg is used to annotate the dataset and divide it into training set, validation set and test set.

3. The lightweight highway pavement defect detection method based on improved RT-DETR according to claim 1 is characterized in that: The improvement 1, the ResNet18 network reconstructed using the lightweight module RGCSPELAN, is composed of several lightweight modules RGCSPELAN and Conv modules alternately connected. The lightweight module RGCSPELAN uses RepConv for efficient convolution on the gradient flow branch, which is fused during inference and the size is controlled using a scaling factor.

4. The lightweight highway pavement defect detection method based on improved RT-DETR according to claim 3 is characterized in that: The lightweight module RGCSPELAN first performs a 1×1 convolution operation on the input defect image, and then performs a Split operation to divide it into two parts. One part first performs a RepConv convolution operation, then performs n-1 3×3 convolution operations, and finally performs a 1×1 convolution operation. The image processed by the RepConv convolution, n-1 3×3 convolution operations and a 1×1 convolution operation is spliced ​​with the other part of the image after Split; finally, a 1×1 convolution operation is performed on the spliced ​​image.

5. The lightweight highway pavement defect detection method based on improved RT-DETR according to claim 1 is characterized in that: The improvement 2 introduces the efficient additive attention module EAA to improve the multi-head self-attention mechanism of the intra-scale feature interaction module AIFI, which introduces EAA in AIFI. EAA uses linear element multiplication operations instead of quadratic matrix multiplication operations to perform key-value interactions without sacrificing accuracy; the query matrix is ​​multiplied by learnable weights and aggregated to generate a global query; then, the matrix K is element-wise multiplied by the broadcasted global query to obtain a global context representation.

6. The lightweight highway pavement defect detection method based on improved RT-DETR according to claim 5 is characterized in that: The proposed efficient additive attention module EAA encodes the relevance score of the contextual information of the input sequence based on the interaction of the three attention components Q, K, and V. First, the key-value interaction is removed without sacrificing performance, and only the relationship between the query key interaction learning tokens is effectively encoded by merging the linear projection layer. Second, the input embedding matrix x is transformed into two matrices W. q , W k Transformed into query Q and key K, where Q, K∈R n×d , W q ,W k ∈R d×d , n is the token length, d is the dimension of the embedding vector; Next, multiply the query Q by the learnable parameter vector w α ∈R d , learn the query attention weight to obtain the global attention query vector α∈R n ; Then, the query Q is pooled according to the learned attention weights to obtain a single global query vector q∈R d , the formula is as follows: Next, we use the element-wise product to solve the global query vector q∈R d and the key matrix, i.e., key K∈R n×d The interactions between them are encoded to form a global context representation R n×d ; Finally, a linear transformation layer is used to query the key interaction to learn the hidden representation of the token. The output formula of the effective additional attention x is as follows: x=Q+T(K*q) (2) Where Q represents the normalized query matrix and T represents the linear transformation.

7. The lightweight highway pavement defect detection method based on improved RT-DETR according to claim 1 is characterized in that: The feature pyramid CGRFPN is used for spatial feature reconstruction and pyramid context extraction, capturing global context in the horizontal and vertical directions of the road defect detection task, and obtaining axial global context to explicitly model the rectangular key area; through multi-scale feature fusion and dynamic interpolation, the feature expression ability of the model is further improved, and its recognition ability for defect detection tasks under complex backgrounds is enhanced.

8. The lightweight highway pavement defect detection method based on improved RT-DETR according to claim 1 or 7, characterized in that: The feature pyramid CGRFPN first uses a matrix self-calibration module RCM to capture axial context in the horizontal and vertical directions, gradually downsamples the encoder to generate features of different scales [F1, F2, F3, F4] and resolutions [H / 4×H / 4, H / 8×H / 8, H / 16×H / 16, H / 32×H / 32], and abandons the largest feature F1; generates a pyramid feature F5 through downsampling and connection operations, and inputs it into multiple stacked RCMs for pyramid feature interaction; Secondly, RCM is used to capture the axial global context to model the rectangular key area, and the shape self-calibration function is used to adjust the attention area to the foreground; then, the matrix self-calibration attention RCA adopts horizontal pooling and vertical pooling to capture the axial global context in two directions, generates two different axis vectors, uses horizontal strip convolution to calibrate the shape in the horizontal direction, BN normalizes the features, ReLU adds nonlinearity, and uses vertical strip convolution to perform shape self-calibration in the vertical direction. Then, 3×3 depth convolution is used to further extract local details of the input features, and the corrected attention features are weighted to the refined input features; finally, the rectangular self-calibration attention is combined with the MetaNeXt structure, and the residual connection is used to further enhance feature reuse.

9. The lightweight highway pavement defect detection method based on improved RT-DETR according to claim 8 is characterized in that: The shape self-calibration is expressed as: Among them, ψ represents large kernel strip convolution, k represents the kernel size of strip convolution, φ represents batch normalization, and δ represents the Sigmoid function; The corrected attention features are weighted onto the refined input features through Hadamard convolution, and the formula is expressed as: x F (x,y)=ψ 3×3 (x)⊙y (4) Among them, ψ 3×3 represents a 3×3 depth convolution with a kernel; Y is the attention feature obtained in the previous step; ⊙ represents a Hadamard convolution; The final output of the feature pyramid CGRFPN is expressed as: in, Indicates broadcast addition; H p and V p denote horizontal pooling and vertical pooling respectively; ρ denotes BN and MLP operations, and x denotes the input feature map.

10. The lightweight highway pavement defect detection method based on improved RT-DETR according to claim 1, characterized in that: The ResNet18 network reconstructed using the lightweight module RGCSPELAN is sequentially connected and composed of Conv1, Conv2, RGCSPELAN1, Conv3, RGCSPELAN2, Conv4, RGCSPELAN3, Conv5, and RGCSPELAN4; the hybrid encoder uses the outputs of RGCSPELAN2, RGCSPELAN3, and RGCSPELAN4 as input features; The hybrid encoder after introducing the efficient additive attention module EAA includes: RCM1, DIF1, FBM1, PCE, GIO1, GIO2, GIO3, FBM2, RCM2, DIF2, Conv6, AIFI-EAA, Conv7, RCM3, FBM3, Conv8, Concat1, RepConv1, Conv9, Concat2, and RepConv2; The output of RGCSPELAN2 is divided into two paths, one of which is connected to the input of RCM1. The output of RGCSPELAN3 is divided into two paths, one of which is connected to the input of RCM2. The output of RGCSPELAN4 is connected to the input of Conv6. The output of Conv6 is connected to AIFI-EAA. The output of AIFI-EAA is connected to Conv7. The output of Conv7 is divided into two paths, one of which is connected to the input of RCM3, and the other path and the other path of the output of RGCSPELAN2 and the other path of the output of RGCSPELAN3 are all connected to the input of PCE. The output of PCE is divided into three paths, which are respectively connected to the inputs of GIO1, GIO2, and GIO3. The output of GIO3 and the output of RCM3 are connected to FBM3. The output of FBM3 is divided into two paths, one of which is connected to the output of RCM2. DIF2, the output of DIF2 and the output of GIO2 are connected to FBM2, the output of FBM2 is divided into two paths, one path and the output of RCM1 are connected to DIF1, the output of DIF1 and the output of GIO1 are connected to FBM1, the output of FBM1 is divided into two paths, one path is used as decoder input, and the other path is connected to the input of Conv8, the other path of FBM2 output and the output of Conv8 are connected to Concat1, the output of Concat1 is connected to RepConv1, the output of RepConv1 is divided into two paths, one path is used as decoder input, and the other path is connected to the input of Conv9, the other path of FBM3 output and the output of Conv9 are connected to Concat2, the output of Concat2 is connected to RepConv2, and the output of RepConv2 is used as decoder input.

Images