Road defect detection method under complex weather based on improved RT-DETR

Abstract

A road defect detection method in complex weather based on improved RT-DETR belongs to the technical field of road intelligent perception and computer vision detection, and comprises the following steps: 1) obtaining a road defect image data set; 2) dividing a road defect image data set; 3) performing data enhancement on the training set image to form an enhanced training data set; 4) constructing an improved RT-DETR road defect detection model; 5) inputting the enhanced training data set obtained in the step 3) into the road defect detection model in the step 4) for training and storing a final model; according to the road defect detection method, through multi-meteorological data enhancement, statistical characteristic self-attention scale internal modeling, frequency domain modulation multi-scale fusion and scale dynamic regression, high-precision and real-time detection of road defects is achieved, the calculation overhead is reduced, and the deployment efficiency in actual road inspection is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of road intelligent perception and computer vision detection, and particularly relates to a complex weather road defect detection method based on improved RT-DETR, which can be used for automatic identification and positioning of multiple types of diseases such as road cracks, potholes and repairs in complex weather. BACKGROUND

[0002] Road defect detection is an important part of urban road maintenance and traffic safety guarantee. With the acceleration of urbanization, the number of road cracks, potholes, damage and other defects is increasing, and if it is not found and repaired in time, it will have a serious impact on vehicle driving safety, traffic efficiency and road life. The traditional manual inspection method has the problems of low efficiency, high omission rate and dependence on manual experience, and cannot meet the needs of large-scale road inspection. Therefore, the automatic road defect detection method based on visual sensing technology has gradually become a research hotspot.

[0003] The existing target detection algorithm based on convolutional neural network has good performance in detecting general targets, but still has obvious shortcomings in road scenes: first, the road surface is obviously affected by light changes, rain and fog weather, noise interference and other factors, and the detection robustness of the traditional model under multiple weather conditions is insufficient; second, the defect morphology of slender cracks and small potholes is complex and the scale difference is large, and the deep feature is easy to appear broken expression or local blur, which leads to incomplete detection or omission; third, the fusion between different scale features often relies on simple splicing or convolution, and it is difficult to simultaneously consider high-frequency details and low-frequency background information; fourth, the target scale is sensitive to the traditional bounding box regression loss function, which easily leads to unstable positioning of small-scale cracks; fifth, the calculation amount of part attention or cross-scale fusion structure is large, which is difficult to meet the real-time requirements of road inspection and other scenes.

[0004] Based on the above problems, it is necessary to design an end-to-end road defect detection method that can adapt to multiple weather environments, has structure-sensitive modeling capability, considers multi-scale and multi-band feature fusion, and has high computational efficiency, in order to improve the detection accuracy while meeting the real-time requirements of practical applications. SUMMARY

[0005] The purpose of the present application is to provide a complex weather road defect detection method based on improved RT-DETR, which realizes high-precision and strong-robust detection of road defects through multi-weather data enhancement, scale modeling based on statistical feature self-attention, frequency domain modulation multi-scale cross fusion and scale dynamic regression, and reduces redundant calculation in structure design, improves the real-time performance and deployment efficiency of the model in the actual road inspection scene.

[0006] The application is realized by the following method: a complex weather-based road defect detection method based on improved RT-DETR, the steps of which are: Step 1) Obtain the road defect image dataset: obtain the image dataset for road defect detection, which is composed of multiple road surface scene images and their corresponding defect labels, and the defect types include longitudinal cracks, transverse cracks, network cracks, potholes and repair.

[0007] Step 2) Divide the road defect image dataset: divide the road defect image dataset into training set, validation set and test set according to the ratio of 8:1:1.

[0008] Step 3) Data enhancement on road defect image data: in order to improve the robustness of the model in detecting road defects under different weather conditions, the training set images obtained in step 2) are subjected to data enhancement to construct corresponding rain, snow and fog scene images, forming an enhanced training dataset .

[0009] Step 3.1) Rain enhancement

[0010] For the original image , a convolution kernel simulating rain stripes is constructed to perform convolution operation on the image, and the original image is superimposed to obtain the rain enhanced image. Its form can be expressed as:

[0011] Wherein, represents a two-dimensional convolution operation, is a rain stripe kernel. In order to control the direction and thickness of the rain stripe, the can be constructed as a kernel approximately distributed along a certain direction, and then by randomly sampling different images and different parameters, a diversified set of rain scene road defect images .

[0012] Step 3.2) Snow enhancement

[0013] In order to simulate the snowfall environment, sparse highlight snowflake points or small patches are superimposed on the original image. A snowflake mask is constructed, and a Bernoulli random variable is sampled for each pixel position :

[0014] Wherein represents the probability that the pixel is covered by snowflakes, which can be taken as [0.01, 0.05] to maintain the sparse distribution of snowflakes. The enhanced snow image can be expressed as:

[0015] Wherein For the brightness value of the snowflake region, it can be approximately taken as 255 or the brightness value close to white. In implementation, the snowflake can be further slightly expanded or blurred in space to be expressed as a small block rather than a single point.

[0016] Step 3.3) Fog enhancement

[0017] The fog enhancement can adopt an atmospheric scattering model to perform fog processing on the image. The original image The fogged image can be expressed as:

[0018] wherein is the atmospheric light intensity, which is generally a constant close to the background brightness; it can be selected in the range of [0.7, 1.0]; is the transmittance function, is the attenuation coefficient related to the fog density, which can be selected in the range of [0.5, 1.5] to adjust the thickness of the fog; is the approximate "depth" or distance field, which can be constructed as a two-dimensional field that changes smoothly or changes slowly at random according to the image size to simulate the change of fog intensity caused by the distance from the camera.

[0019] By changing the values of and , a set of foggy road defect images under different density and brightness conditions can be obtained .

[0020] Step 3.4) Construction of enhanced training data set

[0021] For each original road defect image in the training set, it is selected whether to perform rain, snow or fog enhancement according to a predetermined probability, and the generated enhanced image is combined with the original image to form the final training data set, which can be expressed as:

[0022] Compared with the model trained only using the original sunny data, the enhanced training data set can maintain more stable road defect detection performance under various weather conditions.

[0023] Step 4) Construction of road defect detection model based on improved RT-DETR: RT-DETR-R18 including backbone network, efficient hybrid encoder and transformation decoder with auxiliary prediction head is selected as the basic detection model for improvement. First, the multi-scale features are extracted by the backbone network and the deep semantic feature map ; then in the high-efficiency mixed encoder, the multi-head self-attention in the original AIFI module is replaced by the statistical feature-based self-attention to construct a scale-in feature interaction module based on statistical feature-based self-attention for realizing scale-in structural modeling of deep features; on this basis, a frequency domain modulation multi-scale cross-fusion network is further designed to spatially align, bidirectionally interact and frequency domain weightedly fuse features from different scales to obtain neck features with multi-scale semantic and frequency band adaptive capabilities, thereby providing a more stable and fine-grained road defect expression for the transposed decoder.

[0024] Step 4.1) Scale-in feature interaction module based on statistical feature self-attention

[0025] For the highest layer feature map output by the backbone network After flattening, converted into a vector sequence , wherein represents the semantic information of the region corresponding to the i-th position. In order to model the structural differences of these regions in different directions within the scale, based on the statistical feature self-attention mechanism, each feature vector is projected into several subspaces through a plurality of low-rank matrices to form corresponding subspace responses:

[0026] , wherein is the projection matrix of the i-th subspace, represents the response intensity of the i-th position in the subspace. The response size after projection can reflect the sensitivity of the local region in the subspace. In order to realize the selectivity of the subspace, the self-attention module constructs a membership matrix based on these projections , and calculates the membership coefficients of each position in different subspaces through :

[0027] , wherein is a temperature parameter, represents the probability weight of the feature vector belonging to the i-th subspace, which is a membership distribution constructed by the projection norm. After obtaining the membership of each subspace, in order to further depict the overall structural features within the subspace, the projection values of all positions within the same subspace are weighted and counted to obtain the second-order statistics of the subspace:

[0028]

[0029] The statistics reflect the amplitude of feature changes within the subspace, which is used to construct a diagonal weighting matrix ​​​, thereby adjusting the contribution strength of the subspace to subsequent feature updates. Based on the above membership and statistics, the self-attention and residual connection are unified as an update operator for the feature, which updates the form of the first

[0030] wherein is a step coefficient, used to control the overall update amplitude and ensure the stability of the update process.

[0031] The updated feature sequence enters the feedforward network to further enhance the nonlinear expression capability, and through layer normalization and linear mapping, the output structure consistent with the original AIFI module is maintained. Finally, the feature representation with the same spatial size as the original feature map is restored , as the input of subsequent cross-scale feature fusion, it can reduce background interference while strengthening the local structure consistency and linear form of continuous features to completely extract the crack form.

[0032] Step 4.2.1) Multi-scale cross-fusion module

[0033] Let the three scale feature maps output by the backbone network be , wherein , are low-level and middle-level semantic features, is the high-level semantic feature processed in 4.1). In the adjacent scale interaction stage, the multi-scale cross-fusion module performs spatial alignment on different scale features, upsamples the upper layer features, downsamples the lower layer features, and concatenates them with the current scale features, then integrates them through convolution mapping,

[0034] wherein is channel concatenation, is a convolution or lightweight convolution block used to compress channels and complete local fusion. After the completion of adjacent interaction, the multi-scale cross-fusion module further maps the fused features of each scale to the same spatial size, and the aligned feature set is represented as:

[0035] wherein denotes scale alignment operation, i.e., fusing features of different resolutions after mapping them to a unified spatial scale through upsampling or downsampling to construct shallow texture and high-frequency edge information to high-level semantic structure, creating a stable multi-scale input basis for subsequent frequency domain modulation.

[0036] Step 4.2.2) Frequency domain modulation feature fusion module ​

[0037] In the global fusion stage of step 4.2.1), a frequency domain modulation module is added to adaptively weight the features of different frequency bands in order to address the problem of poor low-frequency brightness detection under complex weather conditions such as haze or uneven lighting.

[0038] The specific steps are as follows: First, the multi-scale fusion features obtained by the multi-scale cross-fusion module... After being divided into low-frequency components based on frequency domain mask decomposition or filtering operations. With high frequency components .

[0039] The signals are then fed into the HL and LH units of the frequency domain modulation feature fusion module to achieve bidirectional modulation of "high frequency guiding low frequency" and "low frequency supplementing high frequency", respectively.

[0040] The HL unit is responsible for extracting high-frequency characteristics. Extract spatial attention maps to enhance low-frequency features. The positional response related to the defective structure. Global average pooling and global max pooling are performed along the channel dimension to obtain two single-channel spatial feature maps, which are then concatenated along the channel dimension. Convolution and Activation generates spatial attention map Then, the low-frequency features are weighted pixel by pixel, in the following form:

[0041] in , Let $\mathbf$ and $\mathbf$ represent global average pooling and global max pooling along the channel dimension, respectively. For convolution kernels with channel compression ratio, express function, This indicates element-wise multiplication. This unit utilizes high-frequency edge and detail information to selectively enhance low-frequency brightness and structural components in the spatial dimension, resulting in a clearer response of low-frequency features in defect areas.

[0042] The LH unit is characterized by low frequency. As input, construct channel attention descriptors to focus on high-frequency features. Each channel was recalibrated. Its structure is a dual-branch system: the upper branch... Perform global average pooling, and then perform lower branch pairing. Perform global max pooling, with both branches passing through two layers. Convolution and Activate and extract channel descriptions, then sum them up and pass through... get the channel attention vector , which is used to modulate the high-frequency features, and the expression is:

[0043] wherein , is the global average pooling and global maximum pooling along the spatial dimension, is the activation function, , is the convolution kernel with channel compression ratio, , is the convolution kernel with channel expansion, is the function. The L-H unit reweights the importance of high-frequency channels using the global structure and brightness information in the low-frequency component, and suppresses invalid high-frequency components introduced by noise, rain streaks, etc.

[0044] Finally, after being modulated by the H-L and L-H units, the obtained and have completed bidirectional information interaction in the frequency domain. By concatenating the two in the channel dimension and fusing them through convolution, the multi-scale features modulated in the frequency domain can be obtained, wherein represents convolution and nonlinear activation combination.

[0045] Through this structure, the model can realize adaptive weight adjustment of different frequency bands by explicitly modeling the low-frequency energy and high-frequency excitation distribution, effectively suppressing low-frequency noise interference such as rain and fog light variation.

[0046] Step 4.2.3) Frequency domain modulation multi-scale cross-fusion network

[0047] The multi-scale cross-fusion module and the frequency domain modulation feature fusion module are combined to form a frequency domain modulation multi-scale cross-fusion network. This network first uses the multi-scale cross-fusion module to build bidirectional interaction between different scales, and fuses texture, edge and semantic information. Then, the frequency domain modulation feature fusion module is used to modulate the fused multi-scale features in the frequency domain, so that the network can automatically select important frequency bands according to the current scene. The overall output can be represented as:

[0048] wherein represents the channel compression and nonlinear mapping layer, which is used to adjust the final neck features to the required dimension of the decoder.

[0049] The structure can adaptively enhance key defect structures such as crack profile and pit boundary in the fusion process of different scales and frequency bands, effectively suppress noise interference caused by complex weather, and improve the accuracy and robustness of road defect detection.

[0050] Step 5) build the enhanced training data set in step 3) The road defect detection model in input 4) is trained using the validation set divided in step 2) And the test set The model performance is evaluated and tested, and the best model is saved.

[0051] Step 5.1) SD Loss-based target frame regression loss construction

[0052] During the training process, scale dynamic loss (SD Loss) is introduced as the core constraint of the boundary box regression to further regulate the regression process of the boundary box.

[0053] For a pair of predicted frame And the real First, calculate the scale term and the position term, and then construct the scale factor On this basis, adaptive weights are given to the scale term and the position term, and the boundary box regression form of SD Loss is finally obtained. The specific process is as follows:

[0054] Where Represents the long and short ratio deviation term of the predicted frame and the real frame, Represents the center point distance, Represents the diagonal length of the circumscribed rectangle of the two frames, The real frame area, Is the reference scale of the target area normalization, Is the scale ratio of the original image and the current feature map, Is the upper limit factor.

[0055] Step 5.2) overall loss function and training strategy

[0056] After introducing SD Loss, the model's total loss function is designed as a weighted sum of classification loss and scale dynamic regression loss, while retaining auxiliary supervision terms for each layer of the decoder to ensure end-to-end training stability.

[0057] in, For classification loss; The scale-based dynamic bounding box regression loss is defined above; This is the sum of auxiliary losses applied to the prediction results of each intermediate layer of the decoder; , , The corresponding weighting coefficients can be adjusted based on the performance of the validation set.

[0058] During training, the validation set is used The metrics were used as the basis for model selection, and finally, the model was selected from the test set. The performance of the trained model is evaluated using average accuracy. Mean Precision Accuracy Recall rate The performance of a trained model is evaluated using the metrics of parameters (parameters) and floating-point operations (FLOPs), expressed as follows:

[0059] Where TP represents the number of positive samples correctly detected by the model, FP represents the number of positive samples incorrectly detected by the model, FN represents the number of positive samples that the model failed to detect, and N is the total number of object categories.

[0060] Step 6) Obtain the road scene image to be detected, input it into the final model obtained in Step 5) for detection, and output the detection results of the type and location of road defects.

[0061] The beneficial effects of this invention are as follows: 1. By introducing multiple meteorological data such as rainy days, snowy days, and foggy days into the training set, the model gains a richer ability to learn appearance changes, improves feature adaptability in complex environments such as uneven lighting and rain and fog interference, and thus maintains stable detection performance under various road conditions.

[0062] 2、The scale internal feature interaction module based on statistical feature self-attention is adopted, the structured modeling of deep features is realized through the subspace projection and the feature updating mechanism guided by the statistical quantity, the consistency expression of the local region is enhanced while the calculation amount is reduced, and the slender defects such as cracks can be more completely presented.

[0063] 3、The frequency domain modulation multi-scale cross fusion network is constructed, the spatial alignment and the frequency domain weighting of multi-scale information are carried out, the adaptive adjustment of high-frequency structures and low-frequency backgrounds is realized, noise interference such as rain and fog light variation is effectively inhibited, and the detection robustness in a complex scene is improved.

[0064] 4、In the training process, the scale dynamic loss (SD Loss) is used to carry out scale adaptive adjustment on the boundary box regression, so that defects of different sizes can obtain more reasonable regression constraints in the training process, and thus the overall positioning accuracy of the model on multi-scale road defects is improved. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 It is a flowchart of the method of the present application.

[0066] Figure 2 It is a model structure diagram of the improved RT-DETR in the method of the present application.

[0067] Figure 3 It is a scale internal feature interaction module diagram based on statistical feature self-attention.

[0068] Figure 4 It is a multi-scale cross fusion module diagram.

[0069] Figure 5 It is a frequency domain modulation feature fusion module diagram.

[0070] Figure 6 It is a model training result diagram.

[0071] Figure 7 It is a detection result diagram of the present application. DETAILED DESCRIPTION

[0072] The present application provides a road defect detection method under complex weather based on an improved RT-DETR, which specifically comprises the following steps: Step 1) Obtain a road defect image dataset: obtain an image dataset for road defect detection, which is composed of multiple road surface scene images and their corresponding defect labels, and the defect types include longitudinal cracks, transverse cracks, network cracks, potholes and repair.

[0073] Step 2) Divide the road defect image dataset: divide the road defect image dataset into a training set, a validation set and a test set according to the ratio of 8:1:1.

[0074] Step 3) Data augmentation on road defect image data: To improve the robustness of the model in detecting road defects under different weather conditions, data augmentation is performed on the training set images obtained in step 2) to construct corresponding rain, snow, and fog scene images, forming an augmented training data set .

[0075] Step 3.1) Rain enhancement

[0076] For the original image , a convolution kernel simulating rain streaks is constructed to perform convolution operation on the image, and the original image is superimposed to obtain a rain-enhanced image. Its form can be expressed as:

[0077] where represents a two-dimensional convolution operation, is a rain streak kernel. To control the direction and thickness of the rain streaks, the kernel can be constructed to be approximately distributed along a certain direction, and then by randomly sampling different images and different parameters, a diversified set of rain scene road defect images can be generated .

[0078] Step 3.2) Snow enhancement

[0079] To simulate a snowfall environment, sparse high-light snowflake points or small patches are superimposed on the original image. A snowflake mask is constructed, and a Bernoulli random variable is sampled for each pixel position :

[0080] where represents the probability of the pixel being covered by snowflakes, which can be taken as [0.01, 0.05] to maintain sparse distribution of snowflakes. The enhanced snow image can be represented as:

[0081] where is the brightness value of the snowflake region, which can be approximately taken as 255 or a brightness value close to white. In implementation, the snowflakes can be further slightly expanded or blurred in space to appear as small blocks rather than single points.

[0082] Step 3.3) Fog enhancement

[0083] Fog enhancement can use the atmospheric scattering model to fog the image. For the original image , the fogged image can be represented as:

[0084] wherein is the atmospheric light intensity, generally a constant close to the background brightness; can be selected in the range [0.7, 1.0]; is the transmittance function, is the fog density dependent attenuation coefficient, which can be selected in the range [0.5, 1.5] to adjust the thickness of the fog; is the approximate "depth" or distance field, which can be constructed as a smoothly varying or slowly randomly varying two-dimensional field according to the image size to simulate the fog intensity variation caused by the distance of the camera.

[0085] By changing the values of and , a set of foggy road defect images under different density and brightness conditions can be obtained .

[0086] Step 3.4) Constructing the enhanced training data set

[0087] For each original road defect image in the training set, according to the preset probability, it is determined whether to perform rain, snow or fog enhancement, and the generated enhanced image is combined with the original image to form the final training data set, which can be represented as:

[0088] Compared with the model trained only using original sunny data, the enhanced training data set can maintain more stable road defect detection performance under various weather conditions.

[0089] Step 4) Constructing a road defect detection model based on improved RT-DETR: Select RT-DETR-R18 including backbone network, efficient hybrid encoder and transform decoder with auxiliary prediction head as the basic detection model for improvement. First, the multi-scale features are extracted by the backbone network and the deep semantic feature map is obtained; then in the efficient hybrid encoder, the multi-head self-attention in the original AIFI module is replaced by the statistical feature self-attention to construct a scale-in feature interaction module based on statistical feature self-attention for realizing scale-in structure modeling of deep features; on this basis, a frequency domain modulation multi-scale cross fusion network is further designed to spatially align, bidirectionally interact and frequency domain weightedly fuse the features from different scales to obtain a neck feature with multi-scale semantic and frequency band adaptive capabilities, providing a more stable and fine-grained road defect expression for the transform decoder.

[0090] Step 4.1) Scale-in feature interaction module based on statistical feature self-attention

[0091] For the highest layer feature map output by the backbone network Flattened, converted to a sequence of vectors where denotes the semantic information of the region corresponding to the th position. To model the structural differences of these regions in different directions within the scale, each feature vector is projected into several subspaces by a low-rank matrix based on the statistical feature self-attention mechanism, forming the corresponding subspace response:

[0092] where is the projection matrix of the th subspace, denotes the response intensity of the th position in this subspace. The response size after projection can reflect the sensitivity of the local region in this subspace. To achieve the selectivity of the subspace, the self-attention module constructs the membership matrix based on these projections, and calculates the membership coefficient of each position in different subspaces by :

[0093] where is the temperature parameter, denotes the probability weight of the feature vector belonging to the th subspace, which is constructed by the projection norm.

[0094] After obtaining the membership of each subspace, to further depict the overall structural features within the subspace, the weighted statistics of the projection values of all positions within the same subspace are obtained, and the second-order statistics of the subspace are obtained:

[0095] This statistic reflects the amplitude of feature changes within the subspace, which is used to construct a diagonal weighting matrix , so as to adjust the contribution strength of this subspace to the subsequent feature update. Based on the above membership and statistics, the self-attention and residual connection are unified to represent an update operator for the feature, and the update form of the feature of the th position is:

[0096] where is the step coefficient, which is used to control the overall update amplitude and ensure the stability of the update process.

[0097] The updated feature sequence enters the feedforward network to further enhance the nonlinear expression ability, and through layer normalization and linear mapping, the output structure consistent with the original AIFI module is maintained. Finally, it is restored to the original feature map Feature representation of the same spatial size As the input of subsequent cross-scale feature fusion, it can reduce background interference while strengthening the continuity of local structure consistency and linear form to completely extract the crack form.

[0098] Step 4.2.1) Multi-scale cross-fusion module

[0099] Let the three scale feature maps output by the backbone network be , wherein , are low-level and middle-level semantic features, is the high-level semantic feature processed in 4.1). In the adjacent scale interaction stage, the multi-scale cross-fusion module performs spatial alignment on different scale features, upsamples the upper layer features, downsamples the lower layer features, and integrates them after being spliced with the current scale features through convolution mapping,

[0100] , wherein is channel splicing, is convolution or lightweight convolution block, used to compress channels and complete local fusion. After the completion of adjacent interaction, the multi-scale cross-fusion module further maps the features fused by each scale to the same spatial size, and the aligned feature set is represented as:

[0101] , wherein represents the scale alignment operation, that is, the features of different resolutions are fused after being mapped to a unified spatial scale through upsampling or downsampling to construct shallow texture and high-frequency edge information to effectively transfer to high-level semantic structure, creating a stable multi-scale input basis for subsequent frequency domain modulation.

[0102] Step 4.2.2) Frequency domain modulation feature fusion module

[0103] In the global fusion stage of step 4.2.1), a frequency domain modulation module is added to adaptively weight different frequency band features to address the problem of poor low-frequency brightness detection effect under complex weather such as fog and uneven light.

[0104] The specific method is: First, the multi-scale fused features obtained by the multi-scale cross-fusion module are divided into low-frequency components and high-frequency components based on frequency domain mask decomposition or filtering operation.

[0105] The H-L unit and the L-H unit are then sent to the frequency domain modulation feature fusion module to realize the bidirectional modulation of "high frequency guiding low frequency" and "low frequency supplementing high frequency", respectively.

[0106] The H-L unit is responsible for extracting the spatial attention map from the high-frequency feature to enhance the position response of the low-frequency feature related to the defect structure. The global average pooling and the global maximum pooling are performed along the channel dimension, respectively, to obtain two single-channel spatial feature maps, which are spliced in the channel dimension, and then convolved and activated to generate the spatial attention map , and then the low-frequency feature is pixel-wise weighted, which is expressed as:

[0107] wherein , represent the global average pooling and the global maximum pooling along the channel dimension, respectively, is a convolution kernel with a channel compression ratio, represents a function, represents an element-wise multiplication. This unit uses the high-frequency edge and detail information to selectively enhance the low-frequency brightness and structure component in the spatial dimension, so that the low-frequency feature has a clearer response in the defect area.

[0108] The L-H unit takes the low-frequency feature as the input to construct the channel attention descriptor, which is used to re-label the channels of the high-frequency feature . Its structure is a double branch: the upper branch is subjected to global average pooling, and the lower branch is subjected to global maximum pooling. Both branches are extracted through two layers convolution and activation to obtain the channel descriptor, which is finally added and subjected to to obtain the channel attention vector , which is then used to modulate the high-frequency feature, which is expressed as:

[0109] wherein , are the global average pooling and the global maximum pooling along the spatial dimension, is an activation function, , are convolution kernels with a channel compression ratio,​​ , For channel-expanded convolution kernels, for The LH unit utilizes global structure and brightness information in low-frequency components to reweight the importance of high-frequency channels, suppressing invalid high-frequency components introduced by noise, rain streaks, etc.

[0110] Finally, after modulation by HL and LH units, the result is... and Two-way information exchange has been completed in the frequency domain. The two are then spliced ​​together in the channel dimension and... By performing convolutional fusion and channel compression, multi-scale features modulated in the frequency domain can be obtained. ,in express Combination of convolution and nonlinear activation.

[0111] Through this structure, the model can explicitly model the distribution of low-frequency energy and high-frequency excitation, thereby achieving adaptive weight adjustment for different frequency bands and effectively suppressing low-frequency noise interference such as rain, fog, and light variations.

[0112] Step 4.2.3) Frequency Domain Modulation Multi-Scale Cross-Fusion Network

[0113] A multi-scale cross-fusion module and a frequency-domain modulation feature fusion module are combined to form a frequency-domain modulation multi-scale cross-fusion network. This network first utilizes the multi-scale cross-fusion module to build bidirectional interactions between different scales, fusing texture, edge, and semantic information. Then, the frequency-domain modulation feature fusion module modulates the fused multi-scale features in the frequency domain, enabling the network to automatically select important frequency bands based on the current scene. Its overall output can be represented as:

[0114] in This represents a channel compression and nonlinear mapping layer used to adjust the final neck features to the dimensions required by the decoder.

[0115] This structure can adaptively enhance key defect structures such as crack contours and pothole boundaries during the fusion process at different scales and frequency bands, while effectively suppressing noise interference caused by complex weather conditions, thereby improving the accuracy and robustness of road defect detection.

[0116] Step 5) Use the augmented training dataset constructed in Step 3) The road defect detection model in input 4) is used to train it, and the validation set partitioned in step 2) is used. With test set Evaluate and test the model performance, and save the best model.

[0117] Step 5.1) SD Loss based bounding box regression loss construction

[0118] In the training process, to further regulate the regression process of the bounding box, the scale dynamic loss (SD Loss) is introduced as the core constraint of the bounding box regression.

[0119] For a pair of predicted box and the real , first calculate the scale term and the position term, and then construct the scale factor on the basis of the area of the real target at the current feature scale, respectively give adaptive weights to the scale term and the position term, and finally get the bounding box regression form of SD Loss. The specific process is as follows:

[0120] Among them represents the deviation term of the length-width ratio of the predicted box and the real box, represents the center distance, represents the diagonal length of the circumscribed rectangle of the two boxes, is the area of the real box, is the reference scale of the target area normalization, is the scale ratio of the original image and the current feature map, is the upper limit factor.

[0121] Step 5.2) Overall loss function and training strategy

[0122] After introducing the SD Loss, the total loss function of the model is designed as the weighted sum of the classification loss and the scale dynamic regression loss, and the auxiliary supervision term of each layer of the decoder is retained to ensure the stability of the end-to-end training:

[0123] Among them, is the classification loss; is the scale dynamic bounding box regression loss defined above; is the sum of the auxiliary losses imposed on the prediction results of each intermediate layer of the decoder; , , are the corresponding weight coefficients, which can be adjusted according to the performance of the validation set.

[0124] In the training process, the performance of the validation set the indicators as the basis for model selection, and finally the average precision , average precision mean , precision , recall , parameter quantity Params and floating point operation number FLOPs, etc. The specific calculation formula is as follows:

[0125] Among them, represents the number of positive samples correctly detected by the model, represents the number of positive samples incorrectly detected by the model, represents the number of positive samples that the model fails to detect, is the total number of object categories.

[0126] Step 6) Obtain the road scene image to be detected, input it into the final model obtained in step 5) for detection, and output the category and position detection result of the road defect.

[0127] Example 1: The RDD_2022 road defect detection data set is used, and the overall method process shown in Figure 1 is used for training and testing.

[0128] Step S1: Import road defect image data and perform data preprocessing; The data set is converted in format, normalized in size, and some data with annotation errors are filtered out, and then divided into training set, validation set and test set according to 8:1:1. The training set images are enhanced in rainy, snowy and foggy days to improve the adaptability of the model to complex weather.

[0129] Step S2: Set the model running environment and parameters; The software environment for method implementation is Ubuntu20.04, Python3.9.0, and Pytorch2.0.1. The hardware environment includes NVIDIARTX 4090D 24GB GPU and Intel Xeon (Ice Lake) Platinum 8369B CPU, and CUDA11.8 is used to accelerate the GPU. The specific running environment and parameter configuration are shown in Table 1.

[0130] Table 1: Experimental parameter configuration table ​

[0131] Step S3: Constructing a road defect detection model based on improved RT-DETR; The overall structure of the model is shown in Figure 2 The present application adopts RT-DETR-R18 as the basic framework, and extracts multi-scale features from the backbone network. Then, a scale-in feature interaction module based on statistical feature self-attention is introduced in the efficient hybrid encoder, which is used to replace the original AIFI multi-head self-attention to enhance the continuity of linear defects such as cracks in deep features and reduce the computational complexity. The structure of the module is shown in Figure 3 On this basis, a frequency domain modulation multi-scale cross fusion network is further designed to fuse multi-scale and multi-band information, and the structure is shown in Figure 4 (multi-scale cross fusion module) and Figure 5 (frequency domain modulation feature fusion module). Through spatial scale alignment, high-low frequency information bidirectional modulation and semantic feature enhancement, the model still maintains high robustness in complex rain and fog interference environments. Finally, the defect category and position are output by the transform decoder, and auxiliary prediction heads are added to each layer of the decoder to improve the training stability.

[0132] In the training phase, the model prediction results are compared with the real labels, and the scale dynamic loss (SD Loss) is introduced to adaptively adjust the scale sensitivity of the regression box, so that small cracks and small holes can obtain more stable gradient constraints. This training strategy is reflected in the training result diagram of Figure 6 .

[0133] Step S4: Verify and test the model and compare it with existing detection models; After adjusting the model hyperparameters using the validation set, the final model is evaluated on the test set, and the current common target detection model is selected as the control object. The model parameters, computational complexity (FLOPs), precision, recall, and mAP are compared. The experimental results of each model are shown in Table 2. From the table, it can be seen that the improved RT-DETR of the present application performs better than the mainstream single-stage and lightweight detectors in complex weather and multi-scale scenes, with the highest precision and mAP among all models, reaching 87.4% and 85.5% respectively, which is significantly better than YOLOv5m, YOLOv7, YOLO11m and the original RT-DETR-18 model. At the same time, compared with similar lightweight models, the parameter quantity and FLOPs of the present application are further reduced, realizing the optimization of computational efficiency on the premise of significant improvement in accuracy.

[0134] Table 2 Comparison of different detection model effects

[0135] It should be noted that although the model of the present application is slightly lower than the Faster-RCNN two-stage detector in the recall rate index, the present application presents overall advantages in key indicators such as comprehensive accuracy, lightweight degree and stability under complex weather environment, realizes good balance of detection accuracy and real-time performance, and is more suitable for deployment in scenes such as road inspection which require speed and performance. The model detection effect is as shown in Figure 7 It can be seen that the model detection effect is as shown in the figure, which can accurately identify cracks, pits and repair areas in various environments such as rainy days, snowy days and foggy days, verifying the practicability and engineering application value of the method of the present application.

Claims

1. A method for detecting road defects under complex weather conditions based on improved RT-DETR, characterized in that, The steps are as follows: Step 1) Obtain the road defect image dataset: Obtain the image dataset for road defect detection. The image dataset consists of multiple road surface scene images and their corresponding defect annotations. The defect types include five features: longitudinal cracks, transverse cracks, network cracks, potholes, and repair. Step 2) Divide the road defect image dataset: Divide the road defect image dataset into training set, validation set, and test set according to the proportions; Step 3) Perform data augmentation on the training set images obtained in Step 2) to construct corresponding rainy, snowy, and foggy scene images, forming an augmented training dataset. ; Step 4) Construct a road defect detection model based on the improved RT-DETR: Select RT-DETR-R18, which includes a backbone network, an efficient hybrid encoder, and a transform decoder with an auxiliary prediction head, as the basic detection model. First, the backbone network extracts multi-scale features and obtains a deep semantic feature map. Subsequently, in the high-efficiency hybrid encoder, the multi-head self-attention in the original AIFI module was replaced with a statistical feature self-attention-based module to construct an intra-scale feature interaction module based on statistical feature self-attention to realize intra-scale structural modeling of deep features; a frequency-domain modulation multi-scale cross-fusion network was designed to spatially align, bidirectionally interact and frequency-domain weighted fusion of features from different scales to obtain neck features with both multi-scale semantics and frequency band adaptability. Step 5) Use the augmented training dataset constructed in Step 3) Train the road defect detection model from step 4) using the validation set partitioned in step 2). With test set Evaluate and test the model's performance, and save the final model; Step 6) Obtain the road scene image to be detected, input it into the final model obtained in Step 5) for detection, and output the detection results of the type and location of road defects.

2. The method for detecting road defects under complex weather conditions based on improved RT-DETR according to claim 1, characterized in that, In step 3), the specific steps are as follows: Step 3.1) Enhance rainy weather For the original image By constructing a convolution kernel that simulates rain streaks to perform convolution operations on the image and then superimposing it with the original image, an enhanced rainy day image is obtained, represented as: in, This represents a two-dimensional convolution operation. This is the core of the rain streaks. To control the direction and thickness of the rain streaks, The kernel is constructed to be approximately distributed along a certain direction. Then, by randomly sampling different images and different parameters, a diverse set of road defect images in rainy weather scenes is generated. ; Step 3.2) Enhanced snow weather To simulate a snowfall environment, sparse, bright snowflake dots or small patches are superimposed on the original image to construct a snowflake mask. For each pixel position Sample a Bernoulli random variable: in This represents the probability that a pixel is covered by snowflakes. The enhanced snowy image is represented as: in The brightness value of the snowflake area is used to further slightly expand or blur the snowflake in space so that it appears as small patches rather than single points; Step 3.3) Fog Enhancement Fog enhancement uses an atmospheric scattering model to perform fogging processing on the original image. Image after fogging Represented as: in Atmospheric light intensity, generally a constant close to the background brightness; It is a function of transmittance. The attenuation coefficient is related to fog concentration. This is an approximate "depth" or distance field; By changing and By taking values ​​of , a set of images of road defects in foggy weather under different concentrations and brightness conditions is obtained. ; Step 3.4) Construct the augmented training dataset For each original road defect image in the training set, enhancement is performed based on a preset probability, choosing whether to enhance the image for rainy, snowy, or foggy days. The enhanced images are then combined with the original images to form the final training dataset, represented as follows: 。 3. The method for detecting road defects under complex weather conditions based on improved RT-DETR according to claim 1, characterized in that, In step 4), the specific steps are as follows: Step 4.1) Intra-scale feature interaction module based on statistical feature self-attention The highest-level feature map output by the backbone network After flattening, it is converted into a vector sequence. ,in Indicates the first To model the structural differences of these regions in different directions within a given scale, each feature vector is projected onto several subspaces through multiple low-rank matrices based on a statistical feature self-attention mechanism, forming corresponding subspace responses. in For the first The projection matrix of each subspace, Indicates the first The response intensity of each location in the subspace, and the magnitude of the projected response, can reflect the sensitivity of the local region in the subspace. To achieve selectivity for the subspace, the self-attention module constructs a membership matrix based on these projections. ,pass Calculate the membership coefficients of each position in different subspaces: in For temperature parameters, Representing the eigenvector Belongs to the The probability weights of each subspace are distributed by the membership degree constructed from the projection norm; After obtaining the membership degrees of each subspace, in order to characterize the overall structural features within the subspace, the projection values ​​of all positions within the same subspace are weighted and statistically analyzed to obtain the second-order statistics of that subspace: Statistics reflect the magnitude of feature changes within a subspace and are used to construct a diagonal weighted matrix. Based on the aforementioned membership degrees and statistics, self-attention and residual connections are uniformly represented as an update operator for the features. The update form for each location feature is as follows: in This is the step size coefficient, used to control the overall update magnitude; The updated feature sequence is fed into the feedforward network to further enhance its nonlinear expressive power. Through layer normalization and linear mapping, the output structure is kept consistent with the original AIFI module, ultimately restoring the original feature map. Feature representation of the same spatial size This serves as the input for subsequent cross-scale feature fusion; Step 4.2.1) Multi-scale cross-fusion module Let the three scale feature maps output by the backbone network be respectively ,in , These are low-level and mid-level semantic features, respectively. For the high-level semantic features processed in 4.1), during the adjacent scale interaction stage, the multi-scale cross-fusion module spatially aligns features at different scales, upsamples upper-level features and downsamples lower-level features, concatenates them with the current scale features, and integrates them through convolutional mapping. in For channel splicing, for Convolutional or lightweight convolutional blocks are used to compress channels and complete local fusion; after adjacent interactions are completed, the multi-scale cross-fusion module further maps the fused features from each scale to the same spatial size, and the aligned feature set is represented as: in This refers to the scale alignment operation, which maps features of different resolutions to a unified spatial scale through upsampling or downsampling and then fuses them to construct shallow textures and effectively transfer high-frequency edge information to higher-level semantic structures. Step 4.2.2) Frequency Domain Modulation Feature Fusion Module In the global fusion stage of step 4.2.1), a frequency domain modulation module is added to adaptively weight the features of different frequency bands: First, the multi-scale fusion features obtained by the multi-scale cross-fusion module... After being divided into low-frequency components based on frequency domain mask decomposition or filtering operations. With high frequency components ; The signals are then fed into the HL and LH units of the frequency domain modulation feature fusion module to achieve bidirectional modulation of "high frequency guiding low frequency" and "low frequency supplementing high frequency", respectively. The HL unit is responsible for extracting high-frequency characteristics. Extract spatial attention maps to enhance low-frequency features. The positional response related to the defect structure in the middle, for Global average pooling and global max pooling are performed along the channel dimension to obtain two single-channel spatial feature maps, which are then concatenated along the channel dimension. Convolution and Activation generates spatial attention map Then, the low-frequency features are weighted pixel by pixel, in the following form: in , Let $\mathbf$ and $\mathbf$ represent global average pooling and global max pooling along the channel dimension, respectively. For convolution kernels with channel compression ratio, express function, This represents element-wise multiplication; The LH unit is characterized by low frequency. As input, construct channel attention descriptors to focus on high-frequency features. Each channel is recalibrated, and its structure is a dual-branch system: the upper branch... Perform global average pooling, and then perform lower branch pairing. Perform global max pooling, with both branches passing through two layers. Convolution and Activate and extract channel descriptions, then sum them up and pass through... Obtain the channel attention vector Then it is used to modulate high-frequency characteristics, and the expression is: in , For global average pooling and global max pooling along the spatial dimension, for Activation function , For convolution kernels with channel compression ratio, , For channel-expanded convolution kernels, for Function; LH unit utilizes global structure and brightness information in low-frequency components; Finally, after modulation by HL and LH units, the result is... and Two-way information exchange has been completed in the frequency domain; the two will be spliced ​​together in the channel dimension and then... Convolution is used for fusion and channel compression to obtain multi-scale features after frequency domain modulation. ,in express Combination of convolution and nonlinear activation; Step 4.2.3) Frequency Domain Modulation Multi-Scale Cross-Fusion Network By combining a multi-scale cross-fusion module and a frequency-domain modulation feature fusion module, a frequency-domain modulation multi-scale cross-fusion network is formed. This network first utilizes the multi-scale cross-fusion module to build bidirectional interactions between different scales, fusing texture, edge, and semantic information. Then, the frequency-domain modulation feature fusion module modulates the fused multi-scale features in the frequency domain. The overall output is represented as follows: in This represents a channel compression and nonlinear mapping layer used to adjust the final neck features to the dimensions required by the decoder.

4. The method for detecting road defects under complex weather conditions based on improved RT-DETR according to claim 1, characterized in that, In step 5), the specific steps are as follows: Step 5.1) Constructing the target bounding box regression loss based on SD Loss During training, the scale dynamic loss (SD Loss) is introduced as the core constraint for bounding box regression. For a pair of prediction boxes With reality First, the scale term and the location term are calculated. Then, the scale factor is constructed based on the area of ​​the true target at the current feature scale. Based on this, adaptive weights are assigned to the scale and location terms respectively, and the bounding box regression form of SD Loss is finally obtained. The specific process is as follows: in This represents the aspect ratio deviation between the predicted bounding box and the ground truth bounding box. Indicates the distance from the center point. This represents the length of the diagonal of the bounding rectangle of the two frames. This represents the actual area of ​​the frame. It is a reference scale for normalizing the target area. The scale ratio between the original image and the current feature map. Upper limit factor; Step 5.2) Overall Loss Function and Training Strategy After introducing SD Loss, the model's total loss function is designed as a weighted sum of the classification loss and the scaling dynamic regression loss, while retaining the auxiliary supervision terms for each layer of the decoder: in, For classification loss; The scale-based dynamic bounding box regression loss is defined above; This is the sum of auxiliary losses applied to the prediction results of each intermediate layer of the decoder; , , These are the corresponding weighting coefficients; During training, the validation set is used The metrics were used as the basis for model selection, and finally, the model was selected from the test set. The performance of the trained model is evaluated using average accuracy. Mean Precision Accuracy Recall rate The performance of a trained model is evaluated using the metrics of parameters (parameters) and floating-point operations (FLOPs), expressed as follows: in, This represents the number of positive samples correctly detected by the model. This indicates the number of positive samples that the model incorrectly detected. This indicates the number of positive samples that the model failed to detect. This represents the total number of object categories.

Images