Lightweight road defect detection method based on multi-scale layered fusion

Abstract

The invention discloses a lightweight road defect detection method based on multi-scale hierarchical fusion, and the method specifically comprises the following steps: 1, constructing a data set, and dividing the data set into a training set, a test set and a verification set; 2, constructing a multi-scale layered fusion lightweight target detection model; 3, training the model constructed in the step 2 by adopting the training set divided in the step 1; and 4, testing and verifying the model trained in the step 3 by using the test set and the verification set divided in the step 1, and outputting a detection result. According to the invention, the problem of small target missing detection caused by the adoption of a single-scale feature extraction mode in the existing detection method is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and relates to a lightweight road defect detection method based on multi-scale hierarchical fusion. Background Art

[0002] In intelligent transportation and autonomous driving systems, automatic detection of road defects is a crucial task. Road defects (such as cracks, potholes, and dents) not only affect the smooth driving and driving safety of vehicles, but may also pose potential safety hazards to pedestrians and non-motor vehicles. The characteristics of road defects are diverse and multi-scale, and different types of defects have significant differences in size, shape, and texture. Traditional single-scale feature extraction methods are difficult to deal with multi-scale road defects, and there will be problems with missed detection of small targets (the area of ​​the target is less than 2%-5% of the total area of ​​the image) during road defect detection. Summary of the invention

[0003] The purpose of the present invention is to provide a lightweight road defect detection method based on multi-scale hierarchical fusion, which solves the problem of small target missed detection in existing detection methods using single-scale feature extraction.

[0004] The technical solution adopted by the present invention is a lightweight road defect detection method based on multi-scale hierarchical fusion, which specifically includes the following steps:

[0005] Step 1: construct a data set and divide it into training set, test set and validation set;

[0006] Step 2: Build a multi-scale hierarchical fusion lightweight target detection model;

[0007] Step 3, using the training set divided in step 1 to train the model constructed in step 2;

[0008] Step 4: Use the test set and validation set divided in step 1 to test and validate the model trained in step 3, and output the detection results.

[0009] The present invention is also characterized in that:

[0010] In step 1, CRACK500, RSDDs and self-photographed road defect images are used to construct a detection dataset, which contains 40,000 annotated defect instances, including transverse cracks, transverse splicing cracks, longitudinal cracks, longitudinal splicing cracks, potholes, cracks, manhole covers, blurred intersections, and blurred white lines.

[0011] The specific process of step 2 is:

[0012] Firstly, YOLOv7-tiny is used as the backbone network, and the EMA attention mechanism is introduced into the backbone network. The global context information of the feature map is first captured through the long-distance branch to establish the long-distance dependency. Then the local information is captured through the short-distance branch to extract the detail features. The long-distance branch and the short-distance branch adopt a parallel processing mechanism to obtain semantic feature maps of different resolutions. Secondly, a multi-scale feature pyramid FPN network is designed, and the CARAFE upsampling module is introduced to perform upsampling operation to fuse the semantic feature maps of different resolutions.

[0013] The specific process of step 2 is:

[0014] Step 2.1: Input the feature map X in the dataset extracted by the backbone network into the EMA attention mechanism to obtain the enhanced feature map X out ;

[0015] Step 2.2, the feature map X obtained in step 2.1 out Input into the feature pyramid network FPN, and extract multi-scale feature maps F at different levels 1 、F 2 、F 3 ,…,F N , where F 1 is a high-resolution and low-semantic information feature map; F 2 ~F N It is a low-resolution high-semantic information feature map;

[0016] Step 2.3, the CARAFE upsampling module introduced in the multi-scale feature pyramid FPN network is applied to each low-resolution feature map F 2 ~F N , the upsampling kernel is dynamically generated through the CARAFE upsampling module, and then upsampling and reorganization are performed to keep the feature map sizes of all scales consistent;

[0017] Step 2.4: Perform pixel-by-pixel weighted fusion on the upsampled feature maps of different scales, and generate a fused feature map F by weighted average fusion operation. fused (x,y), for the fusion feature map F fused (x,y) undergoes batch normalization and nonlinear activation.

[0018] The specific process of step 2.1 is:

[0019] Perform average pooling and maximum pooling operations on the feature map X to extract spatial information, and generate spatial attention weights As through convolution and activation functions;

[0020] The average pooling operation is shown in the following formula (1):

[0021]

[0022] Among them, X(i+m,j+m) is the pixel value of the input feature map X in the data set; i is the index of the convolution kernel in the vertical direction; j is the index of the convolution kernel in the horizontal direction; m is the relative offset of the convolution kernel in the vertical direction; n is the relative offset of the convolution kernel in the horizontal direction; k is the size of the pooling window; M avg is the feature map after average pooling;

[0023] The maximum pooling operation is shown in the following formula (2):

[0024]

[0025] Among them, M max is the feature map after maximum pooling;

[0026] The spatial attention weight As is shown in the following formula (3):

[0027] A s =Sigmoid(Conv([M avg ,M max ])) (3)

[0028] Among them, the Sigmoid activation function maps the result to the interval [0,1]; Conv is a convolution operation using a 1×1 convolution kernel;

[0029] And generate the channel attention weight Ac through the fully connected layer and activation function:

[0030] The operation of the fully connected layer is shown in the following formula (4):

[0031]

[0032] in, is the input feature vector; W is the weight matrix; is the bias vector; FC is the fully connected layer;

[0033] The channel attention weight Ac is shown in the following formula (5):

[0034] A c =Sigmoid(FC([V avg ,V max ])) (5)

[0035] Among them, the Sigmoid activation function maps the result to the [0,1] interval; FC is the fully connected layer;

[0036] Apply the spatial attention weights and channel attention weights to the input feature map X respectively, and finally output the weighted enhanced feature map:

[0037] X out =A c ·(A s ·X) (6)

[0038] Among them, X out is the output feature of the EMA module, A s represents the spatial attention weight, A c represents the channel attention weight.

[0039] The specific process of step 2.3 is as follows: 1 is the target size, for the feature map First, the upsampling kernel is generated by convolution of K·K K·K refers to the size of the convolution kernel, where K is a positive integer representing the width and height of the convolution kernel. The generated upsampling kernel is used to perform pixel-by-pixel convolution on the feature map output in step 2.2. Reorganized into upsampled feature maps

[0040]

[0041] Among them, n 1 =2~N, Conv is the convolution operation;

[0042]

[0043] Among them, x represents the pixel position in the vertical direction of the output feature map, y represents the pixel position in the horizontal direction of the output feature map, and K x,y (k,l) is the upsampling kernel dynamically generated according to the content of the feature map; (k,l) represents the offset position of the kernel; F n1 (x+k,y+l) is the local region of the feature map in step 2.2.

[0044] In step 2.4, the weighted average fusion process is shown in the following formula (9):

[0045]

[0046] Among them, ω 1 represents the weight of the first feature map; F 1 (x, y) represents the value of the first feature map at pixel (x, y); Indicates the nth 1 The weight of the feature map; Indicates the nth 1 The value of the upsampled feature map at pixel (x, y); F fused (x, y) represents the value of the fused feature map at pixel (x, y);

[0047] The normalization process is shown in the following formula (10):

[0048]

[0049] Where μ, σ are the mean and standard deviation of the feature map output in step 2.3 respectively; γ, β are learnable scaling and offset parameters; F norm (x, y) represents the normalized feature map;

[0050] The nonlinear activation process is shown in the following formula (11):

[0051] F activated (x,y)=ReLU(F norm (x,y))(11)

[0052] Among them, ReLU=max(0,z) is the activation function, z represents the input value of the neuron in the convolution kernel, and F activated (x,y) represents the pixel value at position (x,y) in the feature map after activation.

[0053] The final multi-scale fusion feature map is shown in the following formula (12):

[0054] F final (x,y)=Activation(BatchNorm(F fused (x,y)))(12)

[0055] Among them, Activation is the activation function; BatchNorm is quasi-normalization.

[0056] The training process in step 3 is as follows:

[0057] Step 3.1, prepare the predicted box and the true box, specifically:

[0058] For each target, predict box B p and the real frame B t Respectively expressed as:

[0059] B p =(x p ,y p ,w p ,h p ),B t =(x t ,y t ,w t ,h t )(13)

[0060] Among them, (x p ,y p)、(x t ,y t ) represents the center coordinates of the predicted box and the real box; w p ,h p Indicates the width and height of the prediction box; w t ,h t Indicates the width and height of the real box;

[0061] Step 3.2, calculate the prediction box B p and the real frame B t The intersection area A intersection , the union area A union , where the intersection area A intersection As shown in the following formula (14):

[0062] A intersection =max(0,x right -x left )·max(0,y top -y bottom )(14)

[0063] Among them, x right With x left Respectively represent the right and left boundaries of the intersection area, specifically expressed as:

[0064] x right =min(x p +w p / 2,x t +w t / 2) (15)

[0065] x left =max(x p -w p / 2,x t -w t / 2) (16)

[0066] y top With y bottom Respectively represent the upper and lower boundaries of the intersection area, specifically expressed as:

[0067] y top =min(y p +h p / 2,y t +h t / 2) (17)

[0068] y bottom =max(y p -h p / 2,y t -h t / 2) (18)

[0069] Union area A union The following formula (19) is used for calculation:

[0070] A union =w p ·h p +w t ·h t -A intersection (19)

[0071] Step 3.3, calculate CIoU loss L CIoU :

[0072]

[0073] Among them, IoU is expressed as the intersection-over-union ratio, that is, the intersection area A intersection and the area of ​​the union A union The ratio of the predicted box to the true value is the ratio of the overlapping area; p 2 (b,b gt ) is represented as the square Euclidean distance between the center point of the predicted box and the real box, b represents the coordinates of the center point of the predicted box, b gt Indicates the coordinates of the center point of the real box; c 2 It is expressed as the square of the diagonal length of the minimum enclosing rectangle that encloses the predicted box and the true box; av is represented as the shape penalty term, a is the weighting coefficient used to adjust the impact of the penalty term v, and v is the penalty term for the aspect ratio;

[0074] Step 3.4, calculate NWD loss L NWD :

[0075]

[0076] In step 3.5, the CIoU-NWD weighted loss function is defined as follows:

[0077] L CIoU-NWD =a'·L CIoU +(1-a')·L NWD (twenty two)

[0078] Among them, a' is the balance factor;

[0079] Step 3.6: Input the images in the training set into the multi-scale hierarchical fusion lightweight target detection model, and use back propagation to calculate the gradient of the model parameters according to the loss value.

[0080]

[0081] in, represents the operation of finding partial derivatives; θ represents the current model parameters;

[0082] Use the optimizer to update the model parameters:

[0083]

[0084] Among them, η represents the learning rate; θ t represents the model parameters of the current round; and θ t+1 It represents the model parameters of the next round;

[0085] The model parameters are continuously updated and iterated until the set number of training rounds or the CIoU-NWD weighted loss function L is reached. CIoU-NWD No longer changes, the optimal model is obtained through training.

[0086] The beneficial effect of the present invention is that the improved yolov7-tiny algorithm for lightweight road detection based on multi-scale hierarchical fusion proposed by the present invention effectively improves the detection accuracy of small targets in remote sensing data sets by introducing the EMA attention mechanism in the backbone network, improving the FPN-CARAFE structural feature upsampling design, and adopting the CIoU-NWD weighted loss function. Compared with the baseline model, the detection accuracy is significantly improved with a slight increase in the number of parameters and the average detection time per frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a photo of a crossroad taken on the spot in the data set used by the lightweight road defect detection method based on multi-scale hierarchical fusion of the present invention;

[0088] Figure 2 It is a pothole photo taken on site in the lightweight road defect detection method based on multi-scale hierarchical fusion of the present invention;

[0089] Figure 3 It is a photo of a crossroad taken on site in the lightweight road defect detection method based on multi-scale hierarchical fusion of the present invention;

[0090] Figure 4 It is a defect photo in the CRACK500 data set in the lightweight road defect detection method based on multi-scale hierarchical fusion of the present invention;

[0091] Figure 5 It is the defect photo in the RSDDs data set in the lightweight road defect detection method based on multi-scale hierarchical fusion of the present invention. DETAILED DESCRIPTION

[0092] The following describes it in detail in conjunction with specific implementation methods.

[0093] Example 1

[0094] The present invention is based on a lightweight road defect detection method based on multi-scale hierarchical fusion. On the basis of YOLOv7-tiny, a new multi-scale (Multi-Scale mainly refers to the ability to effectively detect and extract features of targets of different sizes (from small to large)) hierarchical fusion lightweight target detection model is designed; the model effectively improves the model's ability to extract multi-scale road features by introducing deep separable convolution and attention mechanism.

[0095] The design idea of ​​the present invention is: first, start with the data set, mark the data set, and randomly divide the data set into a training set, a test set, and a validation set. First, by introducing an efficient multi-scale attention mechanism-EMA (Efficient Multi-scale Attention), the learning and detection capabilities of features are enhanced. Then, a content-aware feature reorganization module-CARAFE (Content-Aware ReAssembly of FEatures), i.e., an upsampling module, is introduced into the original feature pyramid (FPN) structure to better adapt to the features of different regions. Finally, the complete intersection over Union-normalized Wasserstein distance CIoU-NWD (Complete Intersection over Union-Normalized Wasserstein Distance) weighted loss function is used to improve the detection results of small targets.

[0096] Example 2

[0097] The lightweight road defect detection method based on multi-scale hierarchical fusion of the present invention specifically includes the following steps:

[0098] Step 1, build a data set;

[0099] Step 2: Use the random partitioning algorithm to randomly divide the data set into three parts: 60% training set, 30% test set, and 10% validation set.

[0100] Step 3: construct a multi-scale hierarchical fusion lightweight target detection model;

[0101] Step 4: Use the training set divided in step 2 to train the model constructed in step 3;

[0102] Step 5: Use the test set and validation set divided in step 2 to test and validate the model trained in step 4, and output the detection results.

[0103] Example 3

[0104] The specific process of step 1 is: Use CRACK500, RSDDs and self-taken road defect images to build a detection dataset. The dataset contains 15,000 images, including 40,000 annotated defect instances, with fine classification, including transverse cracks, transverse splicing cracks, longitudinal cracks, longitudinal splicing cracks, potholes, cracks, manhole covers, blurred intersections, and blurred white lines, which are used for road defect detection and segmentation tasks.

[0105] Example 4

[0106] The specific process of step 3 is as follows: Backbone design: Introduce the EMA (Efficient Multi-scale Attention) attention mechanism in the network that defines YOLOv7-tiny as the backbone. This mechanism improves the feature extraction capability in the following ways: Long-distance branch: Capture the global context information of the feature map and establish long-distance dependencies; Short-distance branch: Focus on capturing local information and extracting detailed features. The two branches use a parallel processing mechanism to retain the complementary characteristics between global and local information through the fused semantic feature map, while suppressing irrelevant features; The final output semantic feature map will be used as the input of subsequent modules to provide a basis for multi-scale feature processing. Multi-scale feature pyramid (FPN) module: Based on the output of the backbone network, a lightweight multi-scale feature pyramid structure is designed to fuse feature maps of different resolutions to generate feature representations suitable for multi-scale target detection; Multi-scale fusion and dynamic reorganization of feature maps (introduced by the CARAFE module): Dynamic reorganization: The CARAFE (Content-Aware ReAssembly of FEatures) module dynamically generates a weight matrix, reorganizes feature maps based on contextual information, and captures the subtle relationship between local and context; Multi-scale fusion: By combining CARAFE with upsampling operations in the feature pyramid, efficient fusion of feature maps of different resolutions is achieved, enhancing the model's perception of small targets.

[0107] Example 5

[0108] A new and efficient multi-scale attention mechanism (EMA) is introduced in the YOLOv7-tiny backbone network; EMA has parallel branches for establishing long-distance dependencies and short-distance dependencies, and adopts the method of encoding global information to reveal the pairwise relationship at the pixel level, and can embed precise location information into EMA. The improved EMA mechanism in the backbone network can not only enhance the network's multi-scale feature information extraction capability, but also obtain more contextual information, enhance the network's feature learning and representation capabilities, and effectively improve the model's ability to detect small targets.

[0109] The feature map X in the dataset extracted by the backbone network is input into the EMA attention mechanism. The feature map size is C×H×W, where: C is the number of channels of the feature map; H is the height of the feature map; W is the width of the feature map. The spatial information is extracted through average pooling and maximum pooling operations, and the spatial attention weight As is generated through convolution and activation functions:

[0110] Average Pooling:

[0111]

[0112] Among them, X(i+m,j+m) is the pixel value of the input feature map in the data set; i is the index of the convolution kernel in the vertical direction; j is the index of the convolution kernel in the horizontal direction; m is the relative offset of the convolution kernel in the vertical direction; n is the relative offset of the convolution kernel in the horizontal direction; k is the size of the pooling window; M avg is the feature map after average pooling.

[0113] Max Pooling:

[0114]

[0115] Among them, M max It is the feature map after maximum pooling.

[0116] Spatial attention weights:

[0117] A s =Sigmoid(Conv([M avg ,M max ]))(3)

[0118] Among them, the Sigmoid activation function maps the result to the [0,1] interval; and Conv is a convolution operation using a 1×1 convolution kernel.

[0119] Pool the feature map X in the spatial dimension, extract channel information, and generate channel attention weights Ac through a fully connected layer and activation function:

[0120] Fully connected layer:

[0121]

[0122] in, is the input feature vector; W is the weight matrix; is the bias vector; FC is the fully connected layer.

[0123] Channel attention weight:

[0124] A c =Sigmoid(FC([V avg ,V max])) (5)

[0125] The Sigmoid activation function maps the result to the interval [0,1]; FC is the fully connected layer; V avg, Represents the channel features after average pooling; V max Represents the channel features after maximum pooling.

[0126] Apply spatial and channel weights to the input feature map respectively, and finally output the weighted enhanced feature map:

[0127] X out =A c ·(A s ·X) (6)

[0128] Among them, X out is the output feature of the EMA module, A s represents the spatial attention weight, A c represents the channel attention weight.

[0129] Example 6

[0130] The feature map X out Input into the Feature Pyramid Network (FPN) and extract multi-scale feature maps F at different levels 1 、F 2 、F 3 ,…,F N , where each feature map F n (n=1~N) represents features of different scales: F 1 It is a low-level feature map with high resolution and low semantic information, suitable for extracting small target details; F n1 (n 1 =2~N) is a high-level feature map, which is a feature map with low resolution and high semantic information, suitable for capturing global context information. Because according to the characteristics of the convolution operation, the low-level feature map is usually processed by a smaller convolution kernel to capture finer local features. These feature maps contain rich spatial information and therefore have a higher resolution. 1 (First layer feature map): usually from the primary layer of the convolutional network, with higher resolution and more detailed information. For example, if the input image is 640×640, after convolution and downsampling, the higher resolution feature map may be 160×160, and the lower resolution feature map may be 20×20 or 10×10. As shown in the following table: (In this embodiment, N is 5)

[0131] Tiers Resolution (width*height) Features enter 640*640 Initial resolution The first layer feature map 320*320 High resolution, rich details The second layer feature map 160*160 Medium resolution, more details The third layer feature map 80*80 Medium to low resolution, reduced detail The fourth layer feature map 40*40 Low resolution, rich semantics The fifth layer feature map 20*20 Very low resolution, strong semantics

[0132] Apply the CARAFE upsampling module introduced in the feature pyramid (FPN) to each low-resolution feature map For each low-resolution feature map The upsampling kernel is dynamically generated through the CARAFE upsampling module to perform upsampling, with F 1 is the target size. Dynamically generate upsampling kernels: for feature maps First, the upsampling kernel is generated by convolution of K·K K·K refers to the size of the convolution kernel, where K is a positive integer representing the width and height of the convolution kernel. The upsampling kernel has adaptive characteristics and can adjust the weights according to the input features. The generated upsampling kernel is used to perform pixel-by-pixel convolution on the feature map output in step 4. Reorganized into upsampled feature maps The generation of each pixel takes into account the neighborhood characteristics and can effectively expand the receptive field.

[0133]

[0134] Where Conv is a convolution operation.

[0135]

[0136] Among them, x represents the pixel position in the vertical direction (height) of the output feature map, y represents the pixel position in the horizontal direction (width) of the output feature map, and K x,y (k,l) is the upsampling kernel dynamically generated according to the content of the feature map; (k,l) represents the offset position of the kernel; F n1 (x+k,y+l) is the feature map local area.

[0137] After the upsampling and reorganization of the CARAFE module is completed in the feature maps of different scales, the feature maps of all scales The sizes are consistent, so that features of different resolutions can be fused at the same scale. Next, perform the following operations: perform pixel-by-pixel weighted fusion on the upsampled feature maps of different scales, and generate a fused feature map F through a weighted average fusion operation. fused (x,y). Activation and normalization: fusion feature map F fused (x,y) is batch normalized and nonlinearly activated to enhance the expressiveness of multi-scale features. This includes weighted average fusion and activation and normalization. The weighted average fusion is as follows:

[0138]

[0139] Among them, ω 1 represents the weight of the first feature map; F 1(x, y) represents the value of the first feature map at pixel (x, y); Indicates the nth 1 The weight of the feature map; Indicates the nth 1 The value of the upsampled feature map at pixel (x, y); F fused (x, y) represents the value of the fused feature map at pixel (x, y).

[0140] The proposed normalization is as follows:

[0141]

[0142] Where μ, σ are the mean and standard deviation respectively; γ, β are learnable scaling and offset parameters; F norm (x,y) represents the normalized feature map.

[0143] The nonlinear activation is as follows:

[0144] F activated (x,y)=ReLU(F norm (x,y))(11)

[0145] Among them, ReLU=max(0,z) is the activation function, z represents the input value of the neuron in the convolution kernel, and F activated (x,y) represents the pixel value at position (x,y) in the feature map after activation.

[0146] The final multi-scale fusion feature map is as follows:

[0147] F final (x,y)=Activation(BatchNorm(F fused (x,y)))(12)

[0148] Among them, Activation is the activation function; BatchNorm is quasi-normalization.

[0149] Example 7

[0150] The specific process of step 4 is:

[0151] CIoU-NWD weighted loss function optimization: CIoU (Complete Intersection over Union): optimizes the overlap rate, center point distance, and aspect ratio similarity of the predicted box and the real box at the same time;

[0152] NWD (Normalized Wasserstein Distance): The Wasserstein distance is used to describe the distribution difference between the predicted box and the real box, which has stronger optimization capabilities especially for small targets.

[0153] Step 4.1, for each target, predict box B p and the real frame B t Respectively expressed as:

[0154] B p =(x p ,y p ,w p ,h p ),B t =(x t ,y t ,w t ,h t )(13)

[0155] Among them, (x p ,y p )、(x t ,y t ) represents the center coordinates of the predicted box and the real box; w p ,h p w t ,h t Represents the width and height of the predicted box and the true box.

[0156] Step 4.2, calculate the prediction box B p and the real frame B t Key parameters: intersection area A intersection , the union area A union .

[0157] A intersection =max(0,x right -x left )·max(0,y top -y bottom )(14)

[0158] Among them, x right With x left Respectively represent the right and left boundaries of the intersection area, specifically expressed as:

[0159] x right =min(x p +w p / 2,x t +w t / 2) (15)

[0160] x left =max(xp -w p / 2,x t -w t / 2) (16)

[0161] y top With y bottom Respectively represent the upper and lower boundaries of the intersection area, specifically expressed as:

[0162] y top =min(y p +h p / 2,y t +h t / 2) (17)

[0163] y bottom =max(y p -h p / 2,y t -h t / 2) (18)

[0164] A union =w p ·h p +w t ·h t -A intersection (19)

[0165] Step 4.3, calculate CIoU loss L CIoU :

[0166]

[0167] Among them, IoU is expressed as the intersection-over-union ratio, that is, the intersection area A intersection and the area of ​​the union A union The ratio of the predicted box to the true value is the ratio of the overlapping area; p 2 (b,b gt ) represents the square Euclidean distance between the center point of the predicted box and the real box, b represents the coordinates of the center point of the predicted box, and b gt Indicates the coordinates of the center point of the real box; c 2 It is expressed as the square of the diagonal length of the minimum enclosing rectangle that encloses the predicted box and the true box; av is represented as the shape penalty term, a is the weighting coefficient used to adjust the impact of the penalty term v, and v is the penalty term for the aspect ratio.

[0168] Step 4.4, calculate NWD loss L NWD :

[0169]

[0170] In step 4.5, the CIoU-NWD weighted loss function is defined as follows:

[0171] L CIoU-NWD =a'·L CIoU +(1-a')·L NWD (twenty two)

[0172] Among them, a' is a balancing factor, which is used to adjust the relative importance of CIoU and NWD.

[0173] Step 4.6, Observation results: When the CIoU-NWD weighted loss function (L CIoU-NWD ) is closer to 0, the more similar the position, size, shape and distribution of the predicted box and the true box are, and the more accurate the model prediction is; when the result is larger, it means that the gap between the predicted box and the true box is large, and the model parameters need to be optimized through training.

[0174] Configuration parameters: The operating environment of the experiment is set as follows: the operating system is Microsoft Windows 11 Professional Edition, the graphics card model is NVIDIA GeForce RTX3090, the server running memory size is 16GB, CUDA version 11.7, PyTorch version 2.0, Python version 3.9. This experiment conducted a total of 300 training cycles (Epochs), the size of each batch (Batch) was set to 16, the optimizer selected was the SGD optimizer, the momentum parameter was set to 0.937, and the initial learning rate was set to 0.01.

[0175] Input the images in the training set into the network defined as YOLOv7-tiny as the backbone, train according to the above steps, calculate the loss, and adjust the training parameters. Use back propagation to calculate the gradient of the model parameters based on the loss value:

[0176]

[0177] in, Represents the operation of finding partial derivatives; θ represents the current model parameters.

[0178] Use the optimizer to update the model parameters:

[0179]

[0180] Among them, η represents the learning rate; θ t represents the model parameters of the current round; and θ t+1 It represents the model parameters for the next round.

[0181] By repeating the above steps, until the set number of training rounds or the CIoU-NWD weighted loss function (L CIoU-NWD) remains basically unchanged (i.e. the loss value converges).

[0182] Example 8

[0183] The specific process of step 5 is as follows:

[0184] After the model training is completed, the divided test set and validation set are used for testing and verification to observe whether the trained model can perfectly predict the road defect type in the feature map. The feature maps of the test set and validation set are input into the multi-scale hierarchical fusion lightweight object detection model to obtain the prediction box and the corresponding loss function.

[0185] Example 9

[0186] The predicted results are as follows Figure 1 As shown in the figure, there are two types of road defects, among which D43 is intersection blur, and 0.79 represents the CIoU-NWD weighted loss function (L CIoU-NWD ) results; D50 is the manhole cover, 0.40 and 0.36 represent the CIoU-NWD weighted loss function (L CIoU-NWD ) results.

[0187] Figure 1 The loss values ​​of the two D50 road defect types are 0.40 and 0.36 respectively; the loss value of the D43 road defect type is 0.79. It is very close to 0. It can be seen that the position of the predicted box is very close to the position of the real box at this time, and the prediction accuracy is very high.

[0188] Figure 2 is a photo of a pothole taken on site, D40 is a pothole, and 0.51 and 0.48 represent the CIoU-NWD weighted loss function (L CIoU-NWD ) results.

[0189] Figure 3 is a photo of the intersection taken on site. D10 is a horizontal crack, and 0.10 and 0.31 represent the CIoU-NWD weighted loss function (L CIoU-NWD ) result; D44 is the blurred white line, and 0.19 represents the CIoU-NWD weighted loss function (L CIoU-NWD ) result; D50 is the manhole cover, and 0.65 represents the CIoU-NWD weighted loss function (L CIoU-NWD ) results;

[0190] Figure 4 is a defect photo in the CRACK500 dataset. D20 is a crack, and 0.72 represents the CIoU-NWD weighted loss function (L CIoU-NWD ) results.

[0191] Figure 5is a defect photo in the RSDDs dataset. D00 is a longitudinal crack, and 0.28 represents the CIoU-NWD weighted loss function (L CIoU-NWD ) results; D10 is a horizontal crack, 0.23 represents the CIoU-NWD weighted loss function (L CIoU-NWD ) result; D44 is the blurred white line, and 0.38 represents the CIoU-NWD weighted loss function (L CIoU-NWD ) results.

[0192] Figure 1 It shows that this method can accurately detect small targets in complex intersection environments, and the matching degree between the predicted frame and the real frame is high.

[0193] Figure 2 The results show that this method performs well in detecting small potholes and can effectively reduce missed detections.

[0194] Figure 3 It shows that this method has strong detection ability in multi-target complex scenes and high adaptability.

[0195] Figure 4 The results show that this method also performs well in the crack detection task in the standard dataset and can accurately detect road defects with complex textures.

[0196] Figure 5 The results show that this method has significant effect in detecting crack types of small targets in standard data sets and has strong generalization ability.

[0197] In summary, it can be seen that this method can efficiently detect small targets in complex scenes. The CIoU-NWD weighted loss value between the predicted box and the true box is close to 0, and the positioning accuracy is high. It has strong capabilities for multi-target and multi-type defect detection, and is particularly sensitive to small targets. The performance in small target detection tasks is better than that of traditional detection methods.

[0198] The following table 1 compares the evaluation indicators of existing models:

[0199] Table 1

[0200]

[0201]

[0202] 1. Accuracy comparison (mAP@0.5):

[0203] In small target detection and complex scenes, the mAP of the improved YOLOv7-tiny reaches 44.3%, which is 4.7 percentage points higher than the standard YOLOv7-tiny. This improvement is mainly due to the introduction of multi-scale feature fusion and adaptive NMS.

[0204] 2. Comparison of parameter quantity (M) and computational quantity (GFLOPs):

[0205] The improved version of YOLOv7-tiny has 5.9M parameters and 11.5GFLOPs of computation, which is slightly less than that of YOLOv7-tiny. This optimization is due to the introduction of lightweight convolution modules.

[0206] YOLOv5-s and EfficientDet-D0 have relatively small parameters and computational complexity, but are slightly inferior to the improved YOLOv7-tiny in detection accuracy, mainly in the detection of complex scenes and small objects. Figure 3 and Figure 5 ,exist Figure 3 There are four types of defects in Figure 5 There are three types of defects, all of which are typical complex scenarios; Figure 3 In the figure, the loss value of D50 (manhole cover) is 0.65, the loss value of D43 (blurred intersection) is 0.77, the loss values ​​of the two D10 (transverse cracks) are 0.10 and 0.31 respectively, and the loss value of D44 (blurred white line) is 0.19. It can be seen that the accuracy and loss value of these road defect types have reached a state that is more suitable for current road detection.

[0207] 3. Comparison of inference time and FPS:

[0208] The improved YOLOv7-tiny has an inference time of 1.9ms and an FPS of 526, which performs well on mobile or embedded devices. Compared with the standard YOLOv7-tiny, its inference time is reduced by about 10%, while the detection accuracy is greatly improved.

Claims

1. A lightweight road defect detection method based on multi-scale hierarchical fusion, characterized by: The specific steps include: Step 1: construct a data set and divide it into training set, test set and validation set; Step 2: Build a multi-scale hierarchical fusion lightweight target detection model; Step 3, using the training set divided in step 1 to train the model constructed in step 2; Step 4: Use the test set and validation set divided in step 1 to test and validate the model trained in step 3, and output the detection results.

2. The lightweight road defect detection method based on multi-scale hierarchical fusion according to claim 1 is characterized in that: In step 1, CRACK500, RSDDs and self-photographed road defect images are used to construct a detection dataset, which contains 40,000 labeled defect instances, including transverse cracks, transverse splicing cracks, longitudinal cracks, longitudinal splicing cracks, potholes, cracks, manhole covers, blurred intersections, and blurred white lines.

3. The lightweight road defect detection method based on multi-scale hierarchical fusion according to claim 2 is characterized by: The specific process of step 2 is: Firstly, YOLOv7-tiny is used as the backbone network, and the EMA attention mechanism is introduced into the backbone network. The global context information of the feature map is first captured through the long-distance branch to establish the long-distance dependency. Then the local information is captured through the short-distance branch to extract the detail features. The long-distance branch and the short-distance branch adopt a parallel processing mechanism to obtain semantic feature maps of different resolutions. Secondly, a multi-scale feature pyramid FPN network is designed, and the CARAFE upsampling module is introduced to perform upsampling operation to fuse the semantic feature maps of different resolutions.

4. The lightweight road defect detection method based on multi-scale hierarchical fusion according to claim 3 is characterized in that: The specific process of step 2 is: Step 2.1: Input the feature map X in the dataset extracted by the backbone network into the EMA attention mechanism to obtain the enhanced feature map X out ; Step 2.2, the feature map X obtained in step 2.1 out Input into the feature pyramid network FPN, and extract multi-scale feature maps F1, F2, F3, ..., F at different levels N , where F1 is a high-resolution low-semantic information feature map; F2~F N It is a low-resolution high-semantic information feature map; Step 2.3: The CARAFE upsampling module introduced in the multi-scale feature pyramid FPN network is applied to each low-resolution feature map F2~F N , the upsampling kernel is dynamically generated through the CARAFE upsampling module, and then upsampling and reorganization are performed to keep the feature map sizes of all scales consistent; Step 2.4: Perform pixel-by-pixel weighted fusion on the upsampled feature maps of different scales, and generate a fused feature map F by weighted average fusion operation. fused (x,y), for the fusion feature map F fused (x,y) undergoes batch normalization and nonlinear activation.

5. The lightweight road defect detection method based on multi-scale hierarchical fusion according to claim 4 is characterized in that: The specific process of step 2.1 is as follows: Perform average pooling and maximum pooling operations on the feature map X to extract spatial information, and generate spatial attention weights As through convolution and activation functions; The average pooling operation is shown in the following formula (1): Among them, X(i+m,j+m) is the pixel value of the input feature map X in the data set; i is the index of the convolution kernel in the vertical direction; j is the index of the convolution kernel in the horizontal direction; m is the relative offset of the convolution kernel in the vertical direction; n is the relative offset of the convolution kernel in the horizontal direction; k is the size of the pooling window; M avg is the feature map after average pooling; The maximum pooling operation is shown in the following formula (2): Among them, M max is the feature map after maximum pooling; The spatial attention weight As is shown in the following formula (3): A s =Sigmoid(Conv([M avg ,M max ])) (3) Among them, the Sigmoid activation function maps the result to the interval [0,1]; Conv is a convolution operation using a 1×1 convolution kernel; And generate the channel attention weight Ac through the fully connected layer and activation function: The operation of the fully connected layer is shown in the following formula (4): in, is the input feature vector; W is the weight matrix; is the bias vector; FC is the fully connected layer; The channel attention weight Ac is shown in the following formula (5): A c =Sigmoid(FC([V avg ,V max ])) (5) Among them, the Sigmoid activation function maps the result to the [0,1] interval; FC is the fully connected layer; Apply the spatial attention weights and channel attention weights to the input feature map X respectively, and finally output the weighted enhanced feature map: X out =A c ·(A s ·X) (6) Among them, X out is the output feature of the EMA module, A s represents the spatial attention weight, A c represents the channel attention weight.

6. The lightweight road defect detection method based on multi-scale hierarchical fusion according to claim 5 is characterized by: The specific process of step 2.3 is as follows: Taking F1 as the target size, for the feature map First, the upsampling kernel is generated by convolution of K·K K·K refers to the size of the convolution kernel, where K is a positive integer representing the width and height of the convolution kernel. The generated upsampling kernel is used to perform pixel-by-pixel convolution on the feature map output in step 2.

2. Reorganized into upsampled feature maps Where n1=2~N, Conv is a convolution operation; Among them, x represents the pixel position in the vertical direction of the output feature map, y represents the pixel position in the horizontal direction of the output feature map, and K x,y (k,l) is the upsampling kernel dynamically generated according to the content of the feature map; (k,l) represents the offset position of the kernel; F n1 (x+k,y+l) is the local region of the feature map in step 2.

2.

7. The lightweight road defect detection method based on multi-scale hierarchical fusion according to claim 6 is characterized by: In step 2.4, the weighted average fusion process is shown in the following formula (9): Among them, ω1 represents the weight of the first feature map; F1(x, y) represents the value of the first feature map at pixel (x, y); Represents the weight of the n1th feature map; Represents the value of the n1th upsampled feature map at pixel (x, y); F fused (x, y) represents the value of the fused feature map at pixel (x, y); The normalization process is shown in the following formula (10): Where μ, σ are the mean and standard deviation of the feature map output in step 2.3 respectively; γ, β are learnable scaling and offset parameters; F norm (x, y) represents the normalized feature map; The nonlinear activation process is shown in the following formula (11): F activated (x,y)=ReLU(F norm (x,y)) (11) Among them, ReLU=max(0,z) is the activation function, z represents the input value of the neuron in the convolution kernel, and F activated (x,y) represents the pixel value at position (x,y) in the activated feature map; The final multi-scale fusion feature map is shown in the following formula (12): F final (x,y)=Activation(BatchNorm(F fused (x,y))) (12) Among them, Activation is the activation function; BatchNorm is quasi-normalization.

8. The lightweight road defect detection method based on multi-scale hierarchical fusion according to claim 7 is characterized in that: The training process in step 3 is as follows: Step 3.1, prepare the predicted box and the true box, specifically: For each target, predict box B p and the real frame B t Respectively expressed as: B p =(x p ,y p ,w p ,h p ),B t =(x t ,y t ,w t ,h t ) (13) Among them, (x p ,y p )、(x t ,y t ) represents the center coordinates of the predicted box and the real box; w p ,h p Indicates the width and height of the prediction box; w t ,h t Indicates the width and height of the real box; Step 3.2, calculate the prediction box B p and the real frame B t The intersection area A intersection , the union area A union , where the intersection area A intersection As shown in the following formula (14): TO intersection =max(0,x right -x left )·max(0,y top -and bottom ) (14) Among them, x right With x left Respectively represent the right and left boundaries of the intersection area, specifically expressed as: x right =min(x p +w p / 2,x t +w t / 2) (15) x left =max(x p -w p / 2,x t -w t / 2) (16) y top With y bottom Respectively represent the upper and lower boundaries of the intersection area, specifically expressed as: y top =min(y p +h p / 2,y t +h t / 2) (17) and bottom =max(y p -h p / 2,and t -h t / 2) (18) Union area A union The following formula (19) is used for calculation: A union =w p ·h p +w t ·h t -A intersection (19) Step 3.3, calculate CIoU loss L CIoU : Among them, IoU is expressed as the intersection-over-union ratio, that is, the intersection area A intersection and the area of ​​the union A union The ratio of the predicted box to the true value is the ratio of the overlapping area; p 2 (b,b gt ) represents the square Euclidean distance between the center point of the predicted box and the real box, b represents the coordinates of the center point of the predicted box, and b gt Indicates the coordinates of the center point of the real box; c 2 It is expressed as the square of the diagonal length of the minimum enclosing rectangle that encloses the predicted box and the true box; av is represented as the shape penalty term, a is the weighting coefficient used to adjust the impact of the penalty term v, and v is the penalty term for the aspect ratio; Step 3.4, calculate NWD loss L NWD : In step 3.5, the CIoU-NWD weighted loss function is defined as follows: L CIoU-NWD =a'·L CIoU +(1-a')·L NWD (22) Among them, a' is the balance factor; Step 3.6: Input the images in the training set into the multi-scale hierarchical fusion lightweight target detection model, and use back propagation to calculate the gradient of the model parameters according to the loss value. in, represents the operation of finding partial derivatives; θ represents the current model parameters; Use the optimizer to update the model parameters: Among them, η represents the learning rate; θ t represents the model parameters of the current round; and θ t+1 It represents the model parameters of the next round; The model parameters are continuously updated and iterated until the set number of training rounds or the CIoU-NWD weighted loss function L is reached. CIoU-NWD No longer changes, the optimal model is obtained through training.

Images