A Road Defect Detection Method Based on Reparameterized Multi-Scale Fusion

Abstract

The present invention discloses a road defect detection method based on reparameterized multi-scale fusion, belonging to the technical field of object detection in deep learning neural networks. A dataset of road defects is selected from public data sources and divided into a training set, a validation set, and a test set. A lightweight road defect detection neural network model based on reparameterized multi-scale fusion is constructed. This model adopts reparameterization technology and multi-scale feature fusion strategy, and can identify and locate various road defects in images. The constructed neural network model is trained using the divided training set and validation set, and the best lightweight road defect detection model is obtained through index evaluation. Subsequently, the inference acceleration of the lightweight detection model is carried out and embedded into the edge computing terminal for deployment. The vehicle-mounted camera is used to collect image data, and the collected road images are input into the edge computing terminal for processing, and the types and positions of road defects existing in the vehicle-mounted images are output.

Description

Technical Field

[0001] The present invention belongs to the technical field of object detection in deep learning neural networks, and specifically relates to a road defect detection method based on reparameterized multi-scale fusion. Background Art

[0002] In the field of computer vision, the development of lightweight object detection algorithms aims to meet the requirements of efficient and accurate object recognition in resource-constrained environments, especially suitable for application scenarios with limited computing resources such as mobile devices and embedded systems. These algorithms optimize the network structure, reduce the number of parameters and floating-point operation counts (FLOPs), and introduce efficient feature extraction methods to significantly reduce the model complexity while ensuring the detection accuracy. Traditional high-precision object detectors are difficult to meet the requirements of real-time inference scenarios such as autonomous driving and security monitoring due to their large model size and low execution efficiency. Summary of the Invention

[0003] To solve the deficiencies of the prior art and achieve the purpose of improving the speed and accuracy of road defect detection, the present invention adopts the following technical solutions:

[0004] A road defect detection method based on reparameterized multi-scale fusion, comprising the following steps:

[0005] Step 1: Obtain road defect data;

[0006] Step 2: Construct a road defect detection neural network model, which adopts reparameterization technology and multi-scale feature fusion strategy to identify and locate various road defects in the image; the construction of the road defect detection neural network model includes the following steps:

[0007] Step 2.1: Sequentially add a group of reparameterizable partial convolution modules to the backbone network for hierarchical feature extraction, optimize the convolutional kernel weights through the dynamic parameter recombination mechanism, and construct a hierarchical feature reuse channel for cross-layer feature interaction and information complementation;

[0008] Step 2.2: Perform in-scale feature interaction on the outputs of the first and last reparameterizable partial convolution modules to obtain the corresponding interaction features of the last layer;

[0009] Step 2.3: Construct a multi-path scale feature fusion network to perform multi-scale feature fusion on the corresponding interaction features of the last layer and the outputs of the previous reparameterizable partial convolution modules, and extract a set of features for decoding;

[0010] Step 2.4: Through the IoU-aware query mechanism, a certain number of image features are screened out from the encoded output sequence as the initial object queries for decoding. Subsequently, through the decoding process equipped with an auxiliary prediction head, the object queries are iteratively optimized, and finally the target boundaries and their corresponding categories are generated.

[0011] Step 3: Use the road defect data to divide the training set and the validation set, train the road defect detection neural network model, and use the trained model for road defect detection.

[0012] Furthermore, for the reparameterizable partial convolution module in Step 2.1, feature extraction of a multi-branch structure is performed on some of the input channels, effectively extracting target features while reducing the number of parameters. Reparameterize the reparameterization block RepVGG Block used in the detection head during training, convert the 1x1 convolution and two branches without data processing into 3x3 convolutions, and then fuse them. To simplify the operations in the inference stage, it is necessary to fuse the batch normalization (BN) layer into the corresponding convolution layer. The formula is as follows:

[0013]

[0014] where x represents the input feature, represents the output feature, μ, σ, γ, and β respectively represent the mean, variance, scaling factor, and offset factor of the normalization layer, and ε represents a small constant to avoid division by zero errors;

[0015] When fusing the BN and Conv layers, the new convolution kernel W’ and bias b’ can be calculated by the following formula:

[0016]

[0017]

[0018] where W represents the weight matrix of the original convolution layer, and W’ and b’ respectively represent the fused convolution kernel and bias term;

[0019] Finally, fuse the reparameterizable partial convolution module PRConv with the basic block BasicBlock in the backbone network ResNet to obtain a lightweight backbone network based on reparameterizable partial convolution.

[0020] The reparameterizable partial convolution module combines the advantages of partial convolution (PConv) and reparameterizable convolution (RepConv), aiming to reduce the model parameters while improving the feature extraction ability and detection accuracy. It divides the input channels and only performs feature extraction on a part of them, while the rest remains unchanged.

[0021] Further, in step 2.2, a scale-internal feature interaction module is adopted. After the output of the first layer of the reparameterizable partial convolution module passes through the variability attention module and the first dropout layer, it is connected with the output of the last layer of the reparameterizable partial convolution module in a residual connection and then normalized. The normalized feature map is connected with the first convolutional layer, activation layer, second dropout layer, and second convolutional layer in a residual connection, and then normalized to obtain the output feature map as the feature after interaction.

[0022] Further, in step 2.3, based on four layers of reparameterizable partial convolution modules, the outputs S2, S3, and S4 of the first three layers and the corresponding feature X5 after interaction of the last layer are input into the efficient multi-path scale feature fusion network (EMBS-FFN). The input feature map first undergoes a series of convolutional operations, including multiple 3x3 convolutional layers, for extracting the basic features of the image. X5 passes through a 1×1 convolution and then is feature-fused with S4 through a multi-scale feature weighted fusion module (BiFusion), and then is input into the multi-scale efficient convolutional module (CSP-MSDC) to obtain the F1 feature map. After passing through the efficient upsampling module (E-Upsample), F1 obtains , Then, it is feature-fused with S3 and S4 through the multi-scale feature weighted fusion module. After passing through the multi-scale efficient convolutional module, the fused feature map is obtained. After passing through the efficient upsampling module (E-Upsample), F2 obtains , Then, it is feature-fused with S2 and S3 through the multi-scale feature weighted fusion module. After passing through the multi-scale efficient convolutional module, the fused feature map is obtained, the feature map is then feature-fused with to obtain , and finally it passes through the multi-scale efficient convolutional module to obtain the first output , which is input into the decoder (Decoder); and are feature-fused through the multi-scale feature weighted fusion module and then pass through the multi-scale efficient convolutional module to obtain the second output , which is input into the decoder (Decoder); Then, it is feature-fused with and F1 through the multi-scale feature weighted fusion module and then passes through the multi-scale efficient convolutional module to obtain the third output , which is input into the decoder (Decoder).

[0023] The efficient multi-path scale feature fusion network combines a multi-scale efficient convolution module and a global heterogeneous kernel selection mechanism, which can intelligently select the most suitable convolution kernel for feature layers of different scales in the intra-scale feature fusion stage, so as to obtain the best multi-scale perception field information. At the same time, a multi-scale feature weighted fusion architecture is adopted, and the Add operation is used instead of the traditional Concat method to reduce the number of parameters and computational complexity. Without affecting the performance, the model size is further compressed, and at the same time, it can adaptively select weighted fusion according to the importance of each scale feature, improving the quality of feature representation. Finally, an efficient upsampling module is also adopted in the efficient multi-path scale feature fusion network, which can maintain a relatively high operation efficiency while achieving certain effects, which is crucial for real-time processing of a large number of road images.

[0024] The multi-scale feature weighted fusion module introduces bidirectional information flow, enabling the feature maps of each layer to exchange information from top to bottom and from bottom to top, ensuring that the feature maps of each layer can fully capture information at different scales. This bidirectional information flow mechanism makes the feature fusion more comprehensive and refined.

[0025] Furthermore, in the multi-scale feature weighted fusion module in step 2.3, during the fusion process, the weighting coefficient of the feature map is obtained through neural network training and can be optimized through backpropagation. The weighted feature maps are fused by summation, and the formula is as follows:

[0026]

[0027] where, represents the feature map of the i-th layer, represents the weighting coefficient related to the feature map of the i-th layer, which is used to represent the contribution degree of this feature map in the fusion, and O represents the final fused feature map;

[0028] To ensure the effectiveness of the weights, the multi-scale feature weighted fusion normalizes the weights so that the sum of all weights is 1. The formula for the normalized weights is as follows:

[0029]

[0030] where, represents the normalized weighting coefficient, represents applying the ReLU activation function to the original weight w i to ensure that the weight is positive; ε represents a small constant to avoid division by zero errors, w j represents the weighting coefficient related to the feature map of the j-th layer;

[0031] The multi-scale feature weighted fusion module is not just a simple feature fusion. It adopts a feature fusion process with multiple iterations, enabling the features of each layer to be fully optimized. After each fusion, the feature map will be propagated through a two-way information flow to further optimize the feature representation of each layer. Through this multi-level feature fusion, the output of each layer not only depends on the current features but is also affected by the features of the upper and lower layers, thus enhancing the expressive power of the features; the output feature map of each layer can be expressed by the following formula:

[0032]

[0033] where, represents the i-th feature map of the k-th layer, represents the weighting coefficient of the i-th layer feature map, represents the output feature map of the k-th layer after weighted fusion.

[0034] The multi-scale feature weighted fusion module enables better fusion of low-level features (with rich detailed information) and high-level features (with strong semantic information) through a two-way information flow, thus providing a more accurate feature representation when detecting small objects; through a learnable weighting mechanism, it can dynamically adjust the weights of different-scale features according to the feedback during the training process, enabling the network to flexibly optimize the feature fusion strategy according to the actual task requirements; although it adds a two-way information flow and a weighted fusion mechanism, its design still focuses on computational efficiency, avoiding excessive computation and reducing the computational overhead while ensuring performance.

[0035] Furthermore, the multi-scale efficient convolution module in step 2.3 combines multi-scale depth convolution (MSDC) and cross-stage partial connection (CSP) designs. By replacing the Bottleneck block in the traditional C2f module with a multi-scale depth convolution module, it aims to enhance the network's multi-scale feature extraction ability, strengthen the perception ability for targets of different scales, and maintain a low computational complexity. In the multi-scale efficient convolution module, first, the input feature map is divided into two parts, x1 and x2, through cross-stage partial connection. The x1 part is directly passed to the subsequent layer through a skip connection, while the x2 part enters the multi-scale depth convolution module for processing. The main advantage of this segmentation design is to reduce the computational amount and retain rich information in the input feature map through partial connection, avoiding excessive information loss. Next, the multi-scale depth convolution module performs multi-scale feature extraction on x2. After being processed by the multi-scale depth convolution module, the number of channels of the feature map is restored to the original dimension through a new 1×1 point convolution (PWC2), followed by a batch normalization (BN) layer and a ReLU6 activation function to help the network maintain a stable training process and introduce non-linear features. In this way, the features processed by the multi-scale depth convolution module can not only contain more scale information but also enhance the interaction between channels and improve the network's ability to express complex features. Finally, in the fusion stage, the features of the x1 and x2 parts are fused through concatenation, and the fused features are further processed by a convolution layer to output the final feature map. Through this structural design, the multi-scale efficient convolution module can effectively combine low computational cost and powerful feature representation ability to adapt to the detection requirements of targets of different scales.

[0036] Furthermore, inside the multi-scale depth convolution module, first, a 1×1 point convolution (PWC1) is used to expand the channels of x2 to increase the number of channels and enhance the network's representation ability. Then, multi-scale depth convolution (DWConv) is used to perform convolution operations on different convolution kernel sizes ks (such as 3×3, 5×5, 7×7, etc.) to extract features from different scales. The design of multi-scale convolution makes the network more sensitive when processing targets of different sizes, thereby enhancing the detection ability for small and large targets. To further improve the feature expression ability, a channel rearrangement operation is introduced into the multi-scale depth convolution module. Channel rearrangement can effectively break the independence between channels, enhance the information interaction between channels, and thus improve the diversity and representation ability of the feature map. The formula is as follows:

[0037]

[0038]

[0039]

[0040] Among them, PWC1 represents a 1×1 point convolution operation, BN represents a batch normalization operation, and R6 represents a ReLU6 activation operation. represents a depth convolution block, which uses a depth convolution operation for each convolution kernel size ks and combines batch normalization (BN) and ReLU6 activation. CS represents a channel rearrangement operation.

[0041] Furthermore, for the efficient upsampling module in step 2.3, its design goal is to improve the spatial resolution of the feature map while maintaining computational efficiency. First, the size of the input feature map is scaled up by a factor of 2 through an upsampling operation. Then, depthwise separable convolution (DWConv) is applied. Different from traditional convolution methods, depthwise separable convolution performs convolution operations independently on each input channel, thus significantly reducing the amount of computation and the number of parameters, thereby improving computational efficiency. Next, the convolution result is processed through a batch normalization (BN) layer to stabilize the training process. Subsequently, the ReLU activation function is applied to introduce non-linearity. Finally, a 1×1 convolution is used to adjust the number of channels of the output feature map to match the input requirements of the next decoding stage.

[0042] Furthermore, in step 2.4, the traditional method screens the top K high-scoring samples from the encoded layer features based on the classification confidence to generate initial query samples. However, due to the evaluation bias between the classification confidence and the localization accuracy, the spatial overlap between some prediction boxes with high classification scores and the annotation boxes is low, resulting in the screening mechanism being prone to missing high-quality candidates with low classification scores but high IoU. Therefore, this invention adopts an IoU-aware candidate screening mechanism. By imposing optimization constraints during the model training stage, the model is forced to learn to map feature representations with high spatial overlap IoU to high classification scores, while suppressing the classification confidence of features with low spatial overlap IoU. After this joint optimization, the system finally extracts the optimal K candidate samples from the encoded layer features based on the optimized classification scores. The optimization objectives are as follows:

[0043]

[0044]

[0045] Among them, and respectively represent the predicted value and the true value. and , c and b respectively represent the class confidence and the target bounding box. L box represents the target bounding box prediction loss function. L cls represents the class prediction loss function.

[0046] Based on the above-mentioned road defect detection method based on reparameterized multi-scale fusion, in the model training of step 3, through index evaluation, the best lightweight road defect detection neural network model is obtained. Subsequently, the inference acceleration of the lightweight model is carried out and embedded into the edge computing terminal for deployment; it also includes step 4, using an on-vehicle camera to collect road image data and inputting it into the deployed edge computing terminal to obtain the road defect categories and positions existing in the on-vehicle images.

[0047] The advantages and beneficial effects of the present invention are as follows:

[0048] The present invention optimizes road defect detection by integrating the reparameterized multi-scale technology. Among them, by introducing partial convolution and reparameterizable convolution (PRConv), the multi-scale feature capture ability is enhanced; by constructing an efficient multi-path scale feature fusion network (EMBS-FFN), combining the CSP-MSDC module and the global heterogeneous kernel selection mechanism, the in-scale feature adaptive fusion is realized, and the Add operation is adopted to reduce the complexity; by an efficient upsampling module (E-Upsample), the detection effect and computational efficiency are balanced. While inheriting the advantages of lightweight design, the present invention significantly improves the accuracy and efficiency of road defect detection. Brief Description of the Drawings

[0049] Figure 1 is the flowchart of the method in the embodiment of the present invention.

[0050] Figure 2 is the schematic diagram of the network structure of the road defect detection model in the embodiment of the present invention.

[0051] Figure 3 is the schematic diagram of the structure of the reparameterizable partial convolution module in the embodiment of the present invention.

[0052] Figure 4 is the schematic diagram of the structure of the multi-scale efficient convolution module in the embodiment of the present invention.

[0053] Figure 5 is the schematic diagram of the structure of the efficient upsampling module in the embodiment of the present invention. Detailed Description of the Specific Embodiment

[0054] The following details the specific embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0055] As Figure 1 shown, a road defect detection method based on reparameterized multi-scale fusion specifically includes the following steps:

[0056] Step 1: Select a dataset containing typical road defects such as longitudinal cracks, transverse cracks, reticulated cracks, and potholes from public data sources, and divide it into a training set, a validation set, and a test set;

[0057] Step 2: Construct a lightweight road defect detection neural network model based on reparameterized multi-scale fusion. This model adopts reparameterization technology and multi-scale feature fusion strategy, and can identify and locate various road defects in images;

[0058] Among them, the specific structure of the lightweight road defect detection neural network model based on reparameterized multi-scale fusion is implemented as follows: Figure 2 as shown:

[0059] Step 2.1: The lightweight road defect detection neural network model based on reparameterized multi-scale fusion is based on the ResNet18 network architecture. In the backbone network, a reparameterizable partial convolution module (PRConv) is innovatively proposed: adopting a phased feature extraction strategy, the convolution kernel weights are optimized through a dynamic parameter recombination mechanism, and at the same time, a hierarchical feature reuse channel is constructed to achieve cross-layer feature interaction and information complementation. Reparameterizable partial convolution modules are added to the last four layers of the backbone network. Finally, the input image will output four feature maps of different scales after entering the backbone network, which are respectively labeled as S2, S3, S4, and S5. Among them, S2 comes from the fifth layer of the backbone network, S3 comes from the sixth layer of the backbone network, S4 comes from the seventh layer of the backbone network, and S5 comes from the eighth layer of the backbone network.

[0060] The specific structure of the reparameterizable partial convolution module is as follows: Figure 3 as shown. The input of the module is divided into a backbone and a branch. The input feature passes through a convolution module and a partial convolution PRConv module in sequence on the backbone. The input feature passes through an identity mapping module Identity on the branch and is concatenated with the output of the partial convolution PRConv module, and then outputs through a Relu activation module. The implementation process is as follows:

[0061] The reparameterizable partial convolution module combines the advantages of partial convolution (PConv) and reparameterizable convolution (RepConv), aiming to reduce the model parameter quantity while improving the feature extraction ability and detection accuracy. It divides the input channels and only extracts features from a part of them, and the rest remains unchanged. Specifically, the original convolution operation calculates using different convolution kernels for each input channel, and its parameters are A 2 ×C, while the PRConv module only takes a part of them (usually 1 / 4) for calculation, and its parameter quantity is reduced to A 2×C / 4, and at the same time, to alleviate the problem of the accuracy decline caused by the reduction of the number of parameters. It effectively extracts target features through a multi-branch structure and plays a role in improving the model accuracy in the PRConv module. Specifically, PRConv reparameterizes the RepVGG Block used in the detection head during training, converts the 1x1 convolution and two branches without data processing into 3x3 convolutions, and then fuses them. To simplify the operations in the inference stage, the batch normalization (BN) layer needs to be fused into the corresponding convolutional layer. Given the parameters μ (mean), σ (variance), γ (scaling factor), and β (offset factor) of a BN layer, and the input , its formula can be expressed as:

[0062]

[0063] where ε is a very small constant to avoid division by zero error.

[0064] When fusing the BN and Conv layers, the new convolutional kernel W’ and bias b’ can be calculated by the following formula:

[0065]

[0066]

[0067] where W represents the weight matrix of the original convolutional layer, and W’ and b’ represent the fused convolutional kernel and bias term respectively.

[0068] Finally, PRConv is fused with the BasicBlock in the backbone network ResNet to obtain a lightweight backbone network based on reparameterized partial convolution.

[0069] Step 2.2: Input the feature map S5 extracted in Step 2.1 into the intra-scale feature interaction module AIFI (Attention-based Intra-scale Feature Interaction). In the intra-scale feature interaction module, the feature map S2 passes through the deformable attention module, dropout, and then is connected in series with S5 and then normalized. The feature map after the first normalization is connected in series with the convolutional layer fc1, the activation function layer GELU, the dropout layer, the convolutional layer fc2, and then passes through the normalization layer and finally outputs the feature map X5.

[0070] Step 2.3: Input S2, S3, S4, and X5 into the Efficient Multi-Path Scale Feature Fusion Network (EMBS-FFN). The input feature maps first go through a series of convolutional operations, including multiple 3×3 convolutional layers, which are used to extract the basic features of the image. After X5 goes through a 1×1 convolution, it is feature-fused with S4 through the BiFusion (Bilateral Fusion) module for multi-scale feature weighting, and then input into the CSP-MSDC (Cross Stage Partial Multi-Scale Dense Convolution) module to obtain the F1 feature map. After F1 passes through the E-Upsample (Efficient Upsampling) module, it gets , Then, through the multi-scale feature weighting and fusion module, it is feature-fused with S3 and S4. After the fused features pass through the multi-scale efficient convolutional module, the F2 fused feature map is obtained. After F2 passes through the E-Upsample (Efficient Upsampling) module, it gets , Then, through the multi-scale feature weighting and fusion module, it is feature-fused with S2 and S3. After passing through the multi-scale efficient convolutional module, the F3 fused feature map is obtained. The F3 feature map is then multi-scale feature-weighted and fused with to obtain , and finally after passing through the multi-scale efficient convolutional module, the first output P1 is obtained and input into the decoder (Decoder); P1 is then multi-scale feature-weighted and fused with , F2, and After multi-scale feature weighting and fusion, the second output P2 is obtained through the multi-scale efficient convolutional module and input into the decoder (Decoder); P2 is then multi-scale feature-weighted and fused with F2 and F1, and after passing through the multi-scale efficient convolutional module, the third output P3 is obtained and input into the decoder (Decoder).

[0071] The structural diagram of the Efficient Multi-Path Scale Feature Fusion Network is implemented as shown in Figure 2 .

[0072] The Efficient Multi-Path Scale Feature Fusion Network combines the multi-scale efficient convolutional module and the global heterogeneous kernel selection mechanism, which can intelligently select the most suitable convolutional kernel for feature layers of different scales during the intra-scale feature fusion stage, thereby obtaining the best multi-scale perception field information. At the same time, it adopts a multi-scale feature weighting and fusion architecture, and reduces the number of parameters and computational amount by using the Add operation instead of the traditional Concat method. Without affecting performance, it further compresses the model size, and can also adaptively select weighted fusion according to the importance of each scale feature, improving the quality of feature representation. Finally, the Efficient Multi-Path Scale Feature Fusion Network also adopts an efficient upsampling module, which can maintain a relatively high operation efficiency while achieving a certain effect, which is crucial for real-time processing of a large number of road images.

[0073] The multi-scale feature weighted fusion module enables information exchange from top to bottom and from bottom to top for the feature maps of each layer by introducing bidirectional information flow, ensuring that the feature maps of each layer can fully capture information at different scales. This bidirectional information flow mechanism makes feature fusion more comprehensive and refined.

[0074] Meanwhile, a weighted fusion strategy is introduced so that when the feature maps of each layer are fused, different weights are assigned according to their importance. The idea of weighted fusion is to dynamically adjust the contributions of feature maps at different scales through learnable weights, thereby improving the fusion effect. During the fusion process, the weighting coefficients of the feature maps are obtained through neural network training and can be optimized through backpropagation. The weighted feature maps are fused by summation, and the formula is as follows:

[0075]

[0076] where, represents the feature map of the i-th layer, represents the weighting coefficient related to the feature map of the i-th layer, indicating the contribution degree of this feature map in the fusion, and O represents the final fused feature map.

[0077] To ensure the effectiveness of the weights, the multi-scale feature weighted fusion performs normalization on the weights so that the sum of all weights is 1. The formula for the normalized weights is as follows:

[0078]

[0079] where, represents the normalized weighting coefficient, represents applying the ReLU activation function to the original weight w i to ensure that the weight is positive; ε is a small constant to avoid division by zero errors, w j represents the weighting coefficient related to the feature map of the j-th layer.

[0080] The multi-scale feature weighted fusion is not just a simple feature fusion. It adopts a multi-iteration feature fusion process, enabling the features of each layer to be fully optimized. After each fusion, the feature maps will be propagated through bidirectional information flow to further optimize the feature representation of each layer. Through this multi-level feature fusion, the output of each layer not only depends on the current features but is also affected by the features of the upper and lower layers, thereby enhancing the expression ability of the features.

[0081] The output feature map of each layer can be represented by the following formula:

[0082]

[0083] Among them, represents the i-th feature map of the k-th layer, represents the weighting coefficient of the i-th layer feature map, represents the output feature map of the k-th layer after weighted fusion.

[0084] The multi-scale feature weighted fusion module enables better fusion of low-level features (with rich detail information) and high-level features (with strong semantic information) through bidirectional information flow, thereby providing a more accurate feature representation when detecting small objects; through a learnable weighting mechanism, it can dynamically adjust the weights of features at different scales according to the feedback during the training process, enabling the network to flexibly optimize the feature fusion strategy according to the requirements of the actual task; although it adds bidirectional information flow and a weighted fusion mechanism, its design still focuses on computational efficiency, avoiding excessive computation, and can reduce the computational overhead while ensuring performance.

[0085] Among them, the multi-scale efficient convolution module is implemented as Figure 4 shown below:

[0086] The multi-scale efficient convolution module is an innovative combination of multi-scale depth convolution (MSDC) and cross-stage partial connection (CSP) designs based on the C2f module in YOLO. It aims to enhance the multi-scale feature extraction ability of the network and the perception ability of targets at different scales by replacing the Bottleneck block in the traditional C2f module with a multi-scale depth convolution module, while maintaining a low computational complexity.

[0087] In the multi-scale efficient convolution module, the input feature map is first divided into two parts through cross-stage partial connection. Specifically, the input feature map x is split into two parts, x1 and x2. The x1 part is directly passed to the subsequent layer through a skip connection, while the x2 part enters the multi-scale depth convolution module for processing. The main advantage of this splitting design is to reduce the computational amount and retain the rich information in the input feature map through partial connection, avoiding excessive information loss.

[0088] Next, the multi-scale depth convolution module performs multi-scale feature extraction on x2. Inside the multi-scale depth convolution module, first, a 1×1 pointwise convolution (PWC1) is used to expand the channels of x2, increasing the number of channels to enhance the network's representation ability. Then, multi-scale depth convolutions (DWConv) are performed on different convolution kernel sizes ks (such as 3×3, 5×5, 7×7, etc.) to extract features from different scales. The design of multi-scale convolutions makes the network more sensitive when processing targets of different sizes, thus enhancing the detection ability for small and large targets. To further improve the feature expression ability, a channel rearrangement operation is introduced into the multi-scale depth convolution module. Channel rearrangement can effectively break the independence between channels, enhance the information interaction between channels, and thus improve the diversity and representation ability of feature maps. Its formula can be expressed as:

[0089]

[0090]

[0091]

[0092] where PWC1 represents the 1×1 pointwise convolution operation, BN represents the batch normalization operation, R6 represents the activation operation, represents the depth convolution block, which uses the depth convolution operation for each convolution kernel size ks and combines batch normalization (BN) and ReLU6 activation, and CS represents the channel rearrangement operation.

[0093] After being processed by the multi-scale depth convolution module, the number of channels of the feature map is restored to the original dimension, which is achieved through a new 1×1 pointwise convolution (PWC2), followed by a batch normalization (BN) layer and a ReLU6 activation function, helping the network maintain a stable training process and introducing non-linear features. In this way, the features processed by the multi-scale depth convolution module can not only contain more scale information but also enhance the interaction between channels, improving the network's ability to express complex features.

[0094] At the end of the multi-scale efficient convolution module, the fusion stage combines the features of x1 and x2 parts, and this fusion operation is achieved through concatenation. The fused features will be processed by a further convolution layer to output the final feature map. Through this structural design, the multi-scale efficient convolution module can effectively combine low computational cost and powerful feature representation ability to meet the detection requirements of targets of different scales.

[0095] where the efficient upsampling module is implemented as Figure 5 shown:

[0096] The efficient upsampling module is designed to efficiently upsample feature maps. The module's design goal is to increase the spatial resolution of feature maps while maintaining computational efficiency. Its structure consists of several key steps. First, the efficient upsampling module scales the input feature map by a factor of 2 through upsampling. This upsampling operation is followed by a depthwise separable convolution (DWConv). Unlike traditional convolution methods, depthwise separable convolution performs convolution operations independently on each input channel, significantly reducing the amount of computation and the number of parameters, thereby improving computational efficiency. Next, the convolution result is processed through a batch normalization (BN) layer to stabilize the training process, followed by a ReLU activation function to introduce nonlinear characteristics. Finally, a 1×1 convolution is performed to adjust the number of channels in the output feature map to match the input requirements of the next decoding stage.

[0097] Step 2.4: Through the IoU-aware query mechanism, a certain number of image features are selected from the encoder's output sequence as the decoder's initial object query. Subsequently, the decoder equipped with an auxiliary prediction head iteratively optimizes these object queries to ultimately generate object bounding boxes and their corresponding confidence scores.

[0098] The optimization principle of the IoU-aware candidate screening mechanism is as follows: the traditional method screens the top K high-scoring samples from the coding layer features based on classification confidence to generate the initial query sample. However, due to the evaluation deviation between classification confidence and positioning accuracy, some prediction boxes with high classification scores have low spatial overlap with the annotation boxes, which makes the screening mechanism easily miss high-quality candidates with low classification scores but high IoU. To this end, the IoU-aware strategy imposes optimization constraints during the model training phase, forcing the model to learn to map feature representations with high spatial overlap (IoU) to high classification scores, while suppressing the classification confidence of low IoU features. After this joint optimization, the system finally extracts the best K candidate samples from the coding layer features based on the optimized classification scores. The optimization objectives of the detector are as follows:

[0099]

[0100]

[0101] in, and denote the predicted value and the true value respectively, and , c and b represent the category and bounding box respectively, L box represents the target bounding box prediction loss function, L cls Represents the category prediction loss function.

[0102] Step 3: Use the training set and validation set divided in Step 1 to train the constructed neural network model. After metric evaluation, obtain the best lightweight road defect detection model. Subsequently, perform inference acceleration on the lightweight detection model and embed it for deployment on edge computing terminals;

[0103] To evaluate the performance of the object detection model, the metric mean average precision (mAP) is used. It comprehensively considers the accuracy and coverage rate of the model in different categories and averages the detection effects of each category. Among them, accuracy (Precision) refers to the proportion of samples that are actually positive among all samples predicted as positive by the model; while recall (Recall) measures the proportion of actual positive samples that the model can identify, that is, the ratio of the number of samples successfully predicted as positive to the total number of all actual positive samples. mAP50 specifically refers to using a standard with an IoU (Intersection over Union) threshold of 50% when calculating mAP to determine whether the prediction box is correct. Specifically, only when the overlap degree (IoU) between the prediction box and the true annotation box reaches or exceeds 50%, the prediction is considered valid. This setting ensures that the evaluation not only considers the accuracy of model classification but also examines the precision of its positioning. The calculation formulas for recall (R) and precision (P) are as follows:

[0104]

[0105]

[0106] Among them, R represents recall, P represents precision, F represents the number of object detection errors, T represents the number of correctly detected objects, and G represents the total number of objects in the test set.

[0107] A set of comprehensive evaluation metrics is also used to evaluate the lightweight performance of the model, including three key metrics: the number of parameters (Params), the amount of computation (GFLOPs), and the number of frames detected per second (FPS). Params refers to the total number of parameters of the model, reflecting the spatial complexity of the model. The larger the number of parameters, usually the more storage space and memory resources the model occupies; GFLOPs represents the floating-point operation volume of the model and is an important metric for measuring the computational performance of the model. A lower GFLOPs value means that the model requires less computational resources during operation, which helps improve the energy efficiency ratio; FPS (Frames Per Second) represents the detection speed of the model, that is, the number of image frames that can be processed per second. A higher FPS indicates that the model can complete the detection task faster, which is particularly important for application scenarios with high real-time requirements.

[0108] This evaluation system not only focuses on the lightweight characteristics of the model (measured by Params and GFLOPs) to ensure its efficient operation in resource-constrained environments, but also attaches importance to the speed of the model (measured by FPS) and accuracy (measured by mAP) to guarantee performance in practical applications.

[0109] To illustrate the effectiveness of this method, the complete network was trained on the classic road defect data GRDDC2020. Through the test set, comparative experiments were carried out with 8 real-time object detection methods including YOLOv5-s, YOLOv6-s, YOLOv8-s, DINO-Deformable-DETR, and RT-DETR in terms of four indicators: the number of parameters (Params), the amount of computation (GFLOPs), the number of frames detected per second (FPS), and the mean average precision (mAP). The experimental environment used the Linux operating system, was implemented in the Pytorch deep learning development framework, the development language was Python, the training strategy and hyperparameters of the decoder basically followed the DINO settings, and 4 NVIDIA GeForce GTX 1080 Ti graphics cards with 11 GB of video memory were used for training and testing. The batch size was set to 40 for the GRDDC2020 dataset, and 250 epochs were iterated; the AdamW optimizer was used, the base learning rate was 0.0001, and the weight decay coefficient was also 0.0001. A global gradient clipping norm of 0.1 and a linear learning rate warm start for the first 2,000 steps were applied; the learning rate adjustment strategy for the backbone network referred to the experiments in the previous chapter, and the exponential moving average (EMA) technique was used with a decay rate of 0.9999 to improve the model performance; during the training process, data augmentation operations such as color distortion, dilation, cropping, flipping, and resizing were randomly applied to increase the robustness and generalization ability of the model. The comparison results on the GRDDC2020 dataset are shown in Table 1:

[0110] Table 1 Comparative experiment table of the present invention and 8 other methods on the classic road defect data GRDDC2020

[0111]

[0112] Step 4: Use an in-vehicle camera to collect image data, and input the collected road images into the edge computing terminal in Step 3 for processing, and output the road defect categories and locations existing in the in-vehicle images.

[0113] Specifically, the road images captured by the in-vehicle camera are input into the road defect detection model with the highest comprehensive evaluation index in the test set. After the operation and processing of this model, the road defect categories existing in the in-vehicle images and the locations of these defects are finally output.

[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A road defect detection method based on heavy parameter multi-scale fusion, characterized by The steps include: Step 1: Obtain road defect data; Step 2: Build a road defect detection neural network model. This model uses reparameterization technology and multi-scale feature fusion strategy to identify and locate various road defects in the image. The construction of the road defect detection neural network model includes the following steps: Step 2.1: A set of reparameterizable partial convolutional modules are sequentially added to the backbone network to perform hierarchical feature extraction. The convolution kernel weights are optimized through a dynamic parameter reorganization mechanism. At the same time, a hierarchical feature reuse channel is constructed to perform cross-layer feature interaction and information complementation. Step 2.2: Perform intra-scale feature interaction on the outputs of the first and last layers of reparameterizable partial convolutional modules to obtain the corresponding interactive features of the last layer; Using the intra-scale feature interaction module, the output of the first layer of reparameterized partial convolutional module is passed through the variable attention module and the first random dropout layer, and then the residual connection is made with the output of the last layer of reparameterized partial convolutional module before normalization. The normalized feature map is residually connected with the first convolutional layer, activation layer, second random dropout layer, and second convolutional layer, and the output feature map obtained by normalization is used as the interactive feature. Step 2.3: Construct a multi-path scale feature fusion network, fuse the interactive features corresponding to the last layer with the output of the previous reparameterized partial convolution module, and extract a set of features for decoding; Based on the four-layer reparameterized partial convolution module, the outputs S2, S3, and S4 of the first three layers and the corresponding interactive features X5 of the last layer are input into the multi-path scale feature fusion network. The input feature map first undergoes a series of convolution operations to extract the basic features of the image. X5 and S4 are fused through the multi-scale feature weighted fusion module and then input into the multi-scale convolution module to obtain the F1 feature map. F1 is obtained after the upsampling module. , Then, the multi-scale feature weighted fusion module is used to fuse the features with S3 and S4. The fused features are then passed through the multi-scale efficient convolution module to obtain The fusion feature map, F2, is obtained after passing through the upsampling module , Then, the multi-scale feature weighted fusion module is used to fuse the features with S2 and S3, and the multi-scale convolution module is used to obtain Fusion feature map, The feature map is then combined with Perform multi-scale feature weighted fusion to obtain ,at last The first output is obtained after the multi-scale convolution module , input to the decoder; Then with 、 and After multi-scale feature weighted fusion, the second output is obtained through the multi-scale convolution module , input to the decoder; Then with After multi-scale feature weighted fusion with F1, the third output is obtained through the multi-scale convolution module. , input to the decoder; The multi-scale feature weighted fusion module is the weighted coefficient of the feature map during the fusion process. It is obtained through neural network training and can be optimized through back propagation. The weighted feature maps are fused by summing. The formula is as follows: ， in, represents the feature map of the i-th layer, represents the weighted coefficient associated with the feature map of the i-th layer, and O represents the final fused feature map; Multi-scale feature weighted fusion will normalize the weights, and the formula is as follows: ， in, represents the normalized weighting coefficient, Represents the original weight w i Perform ReLU activation function processing; ε represents a small constant, w j Represents the weighted coefficient related to the j-th layer feature map; The multi-scale feature weighted fusion module adopts a multi-iteration feature fusion process. After each fusion, the feature map will be propagated through a bidirectional information flow. The output feature map of each layer It can be expressed by the following formula: ， in, represents the i-th feature map of the k-th layer, represents the weighted coefficient of the feature map of the i-th layer, Represents the k-th layer output feature map after weighted fusion; The multi-scale efficient convolution module combines multi-scale depth convolution and cross-stage partial connection. First, the input feature map is divided into two parts, x1 and x2, through the cross-stage partial connection. The x1 part is directly passed to the subsequent layer through the jump connection, while the x2 part enters the multi-scale depth convolution module for processing; Next, the multi-scale depth convolution module performs multi-scale feature extraction on x2; After processing by the multi-scale depth convolution module, the number of channels of the feature map is restored to the original dimension through point convolution, followed by a batch normalization layer and an activation function; Finally, in the fusion stage, the features of the x1 and x2 parts are fused by splicing. The fused features are further processed by the convolution layer to output the final feature map; Inside the multi-scale depth convolution module, the channel of x2 is first expanded through point convolution to increase the number of channels. Then, multi-scale depth convolution is used to perform convolution operations on different convolution kernel sizes ks. The design of multi-scale convolution makes the network more sensitive when processing targets of different sizes. The channel rearrangement operation is introduced into the multi-scale depth convolution module, and its formula is as follows: ， ， ， Among them, PWC1 represents a 1×1 point convolution operation, BN represents a batch normalization operation, and R6 represents an activation operation. represents the depth convolution block, which uses depth convolution operation for each convolution kernel size ks, combined with batch normalization and activation, and CS represents the channel reordering operation; Step 2.4: Through the perceptual query mechanism, features are selected from the encoding output as the initial object query for decoding. Subsequently, through the decoding process equipped with an auxiliary prediction head, the object query is iteratively refined to finally generate the object boundary and its corresponding category; Step 3: Using the road defect data, train the road defect detection neural network model, and use the trained model for road defect detection.

2. The road defect detection method based on heavy parameter multi-scale fusion according to claim 1, characterized in that: The reparameterizable partial convolution module in step 2.1 performs feature extraction of the multi-branch structure for some input channels, reparameterizes the reparameterized block used in the detection head during training, and integrates the batch normalization layer into the corresponding convolution layer. The formula is as follows: ， Among them, x represents the input feature, Represents the output features, μ, σ, γ, β represent the mean, variance, scaling factor and offset factor of the normalization layer respectively, and ε represents a small constant; When the BN and Conv layers are fused, the new convolution kernel W' and bias b' can be calculated by the following formula: ， ， Among them, W represents the weight matrix of the original convolution layer, W' and b' represent the fused convolution kernel and bias term respectively; Finally, the reparameterizable partial convolution module is fused with the basic block in the backbone network to obtain a lightweight backbone network based on reparameterized partial convolution.

3. The road defect detection method based on heavy parameter multi-scale fusion according to claim 1, characterized in that: The upsampling module in step 2.3 first scales up the size of the input feature map through an upsampling operation; then, depthwise separable convolution is applied to perform convolution operations independently on each input channel; next, the convolution result is processed by a batch normalization layer; then, an activation function is applied to introduce nonlinear characteristics; finally, convolution is used to adjust the number of channels of the output feature map to match the input requirements of the next decoding stage.

4. The road defect detection method based on heavy parameter multi-scale fusion according to claim 1, characterized in that: In step 2.4, an overlap-aware candidate screening mechanism is used. By imposing optimization constraints during the model training phase, the model is forced to learn to map feature representations with high spatial overlap into high classification scores, while suppressing the classification confidence of features with low spatial overlap. The optimal K candidate samples are extracted from the coding layer features based on the optimized classification scores. The optimization objectives are as follows: ， ， in, and denote the predicted value and the true value respectively, and , c and b represent the category confidence and target bounding box respectively, L box represents the target bounding box prediction loss function, L cls Represents the category prediction loss function.

5. The road defect detection method based on heavy parameter multi-scale fusion according to claim 1, characterized in that: In the model training of step 3, the optimal lightweight road defect detection neural network model is obtained through indicator evaluation. Subsequently, the lightweight model's inference is accelerated and embedded in the edge computing terminal for deployment. It also includes step 4, using the on-board camera to collect road image data and input it into the deployed edge computing terminal to obtain the road defect category and location in the on-board image.

Images