A method for detecting road potholes and small roadblocks based on YOLOv8

Abstract

The application provides a road surface pit and small roadblock detection method based on YOLOv8. An ultra-small target detection model is constructed based on a YOLOv8 model. In the ultra-small target detection model, a fuzzy adaptive C3 module is used to replace a C2f module in a backbone network of the YOLOv8 model. The input image is deblurred based on the fuzzy correction module, and then the image features are extracted. The image features are processed through a convolution attention mechanism CBAM, so that the detection accuracy can be maintained in a fuzzy scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly to a road surface pit and small barrier detection method based on YOLOv8. BACKGROUND

[0002] Real vehicle road testing is needed in the later stage of intelligent vehicle development. Usually, researchers will select suitable test sections around the city area where the research and development base is located. In order to ensure the smooth implementation of vehicle testing, suitable sections are usually selected according to the test content. There are some pressure tests that need to run for a long time in the vehicle test for the specified module, such as the reliability test of some signal collection functions in unmanned driving within the designed service life. This requires to avoid sections prone to accidents as much as possible, and to keep the vehicle in a stable long-time running state as much as possible.

[0003] Usually, before the test starts, the test personnel will first use a drone to capture the actual road surface conditions in the test area, and then plan the test path. In some fields, technical personnel have applied drones to collect road surface conditions, such as the patent documents with application numbers CN118096654A, CN117576073A, CN117078591A and CN117765373B, which disclose a scheme of analyzing the road surface conditions and detecting the road surface disaster conditions based on the road surface pictures collected by the drone. However, most of these road surface detection work is for the preliminary preparation of road maintenance, so the scheme does not need to consider smaller road pits and road obstacles, and only needs to detect road damage above a certain size.

[0004] The detection requirements of the vehicle test environment for the road surface are different from those of road maintenance. For example, if there are many small and sharp objects on some sections, there is a certain probability that the test vehicle's tires will be damaged, thereby interrupting the test work; or too many small pits will also cause continuous deviation of some sensor data, which will affect the overall test effect of the pressure test when accumulated to a certain extent. However, the existing road surface detection algorithm based on a drone cannot accurately identify the detection of road pits and small barriers, which cannot meet the needs of vehicle test road surface detection. SUMMARY

[0005] In order to solve the problem that the existing road surface detection algorithm based on a drone cannot accurately identify the detection of road pits and small barriers, the present application provides a road surface pit and small barrier detection method based on YOLOv8, which can improve the detection accuracy of road pits and small barriers and meet the detection needs of small obstacles on the road surface.

[0006] The technical scheme of the present application is as follows: a road surface pit and small roadblock detection method based on YOLOv8, characterized in that it comprises the following steps:

[0007] S1: constructing a blur correction module;

[0008] The blur correction module; the input feature map F is processed to extract image features F H×W×C after deblurring; conv ; wherein C is the number of channels, H and W are the height and width;

[0009] S2: constructing a blur adaptive C3 module;

[0010] The blur adaptive C3 module is constructed on the basis of the CSPDarknet of YOLOv8; the blur adaptive C3 module extracts features after the input feature map is corrected by the blur correction module, and then outputs after attention enhancement processing by CBAM;

[0011] S3: constructing a cross-scale self-attention module CSSA;

[0012] The cross-scale self-attention module CSSA enhances global context information through inter-scale feature interaction;

[0013] S4: constructing a super-small target detection model based on the YOLOv8 model;

[0014] The super-small target detection model comprises a backbone network, a neck, and a head network;

[0015] The backbone network comprises 3 CBS convolution modules, 3 blur adaptive C3 modules, and 1 SPPF module; each CBS module is followed by a blur adaptive C3 module, and the SPPF module is connected at the end; the network layers in the backbone network are labeled as P1~P7 from top to bottom;

[0016] The network structure of the neck comprises 4 up-sampling Upsample layers, 4 C2f modules, and 4 cross-scale self-attention modules CSSA; the up-sampling Upsample layer comprises Upsample1, Upsample2, Upsample3, and Upsample4 arranged from bottom to top, and each up-sampling layer corresponds to a scale branch, and a C2f module is arranged after each up-sampling layer;

[0017] The output of the P7 layer of the backbone network is sent to the Upsample1 layer. The output of the Upsample1 layer is concatted with the output of the P6 layer and then sent to the C2f layer. The output of the Upsample2 layer is concatted with the output of the P4 layer and then sent to the C2f layer. The output of the Upsample3 layer is concatted with the output of the P2 layer and then sent to the C2f layer.

[0018] Each CSSA module receives the output feature map from a C2f module, performs cross-scale feature fusion processing, and then connects it to a detection head; the output feature maps of the first three C2f modules are fed into the CSSA module and simultaneously fed into the next Upsample layer.

[0019] The head network includes four detection heads, each corresponding to a detection result at one of four scales.

[0020] S5: Train the ultra-small target detection model to obtain the trained ultra-small target detection model;

[0021] S6: Deploy the trained ultra-small target detection model onto the UAV and control the aircraft to fly over the preset detection area; send the image acquisition device on the UAV as the image to be identified into the ultra-small target detection model and output the road surface detection result.

[0022] Its further features are:

[0023] The blur correction module includes: sequentially arranged 1×1 convolutional layers. Reshape layer, 2D convolutional layer and main convolutional layer K∈R k×k×C_in×C_out Where k is the kernel of the main convolutional layer, C_in is the input channel, C_out is the output channel, and k b For convolutional layer W b The output mapped size;

[0024] After the input feature map F is fed into a 1×1 convolutional layer, the 1×1 convolutional layer maps the feature map F to H×W×k. b ×k b The feature matrix is ​​then fed into a reshape layer to reshape the matrix in dimensions, resulting in a pixel-by-pixel blur kernel. ;

[0025] The input feature map F and the blur kernel B are simultaneously fed into a two-dimensional convolutional layer. The blur kernel B is used to deblur the input feature map F, resulting in the deblurred feature map F. deblur =B F, where It is a two-dimensional convolution;

[0026] The deblurred features are sent to the main convolutional layer K to extract target features F conv = K F deblur ;

[0027] The blur adaptation C3 module includes a CBS convolutional block, a blur correction module, N Bottieneck-CBAM blocks connected in series, a Concat block, and a CBS convolutional block. In each Bottieneck-CBAM block, a Bottieneck block and a CBAM module are connected in series.

[0028] The feature map sent to the blur adaptation C3 module extracts an intermediate feature map via the CBS convolutional block. The intermediate feature map is split into two parts. One part is sent to the Concat block, and the other part is processed by the blur correction module and then sent to the Bottieneck-CBAM block to extract a feature map. The Bottieneck-CBAM block extracts a feature map, which is sent to the Concat block for splicing. The spliced feature map is processed by the last CBS convolutional block and then output.

[0029] The following operations are performed in the cross-scale self-attention module CSSA.

[0030] a1: feature flattening and projection

[0031] Each scale feature map F s is flattened into an N s ×C sequence and multiplied by W Q , W K , and W V , respectively, to obtain

[0032] The query matrix Q s = flatten(F s )·W Q , the key matrix K tt = flatten(F tt )·W K , and the value matrix V tt = flatten(F tt )·W V ;

[0033] where F s ∈R H_s×W_s×C , F s is the feature map of the s-th scale, F tt is the feature map of the t-th scale, H s , and W sH, W, C are height, width, and channel number of the feature map; s and tt are scale indexes of the feature map, s, tt e {1, 2, 3, 4}, {1, 2, 3, 4} correspond to extra-small, small, medium, large scale respectively; W Q , W K , W V e R C×d_k , are projection matrix of query Query, key Key, value Value respectively, d k is projection dimension;

[0034] a2: attention weight calculation: calculate the similarity between scales by scaling dot product attention formula, and then normalize by Softmax;

[0035] ;

[0036] A s,tt e R N_s×N_tt is the attention weight matrix from scale s to scale tt, N s = H s x W s is the number of feature map pixels; N tt = H tt x W tt is the number of feature map pixels;

[0037] a3: cross-scale feature fusion: weighted sum of value matrix V s,tt of each scale by attention weight A tt , reshape to the original feature map size, and finally add residual connection to retain the original feature information:

[0038] ;

[0039] wherein, is the feature map after cross-scale fusion;

[0040] The detection head includes three parallel task heads, which are respectively a classification head, a positioning head and a confidence calibration head;

[0041] The classification head performs the following operations:

[0042] ;

[0043] In the formula, sigma is a Sigmoid function, Conv1 is a 1x1 convolution, t is a target category, and p t is the probability that the target belongs to the category t;

[0044] The positioning head outputs 5-dimensional parameters (x, y, w, h, p) of the target prediction box, wherein (x, y) is the center coordinate of the prediction box, (w, h) is the width and height of the box, and p is the rotation angle of the prediction box;

[0045] The confidence calibration head outputs a confidence: score = p t × scene;

[0046] In the formula, score is the confidence, and scene∈[0,1] is a scene factor;

[0047] It also includes a preprocessing module. The pictures collected by the unmanned aerial vehicle are processed by the preprocessing module and then sent to the ultra-small target detection model for detection. The following steps are performed in the preprocessing module:

[0048] b1: data cleaning and standardization;

[0049] Remove samples with missing or mislabeled targets, and retain valid samples. Resize the image to 1280*1280. Normalize the mean and standard deviation of the RGB three channels of the input image respectively to eliminate the influence of brightness difference on the model;

[0050] b2: For the cleaned pictures, radial and tangential distortion correction is performed on the images collected by the unmanned aerial vehicle;

[0051] b3: For the corrected pictures, a pre-trained lightweight semantic segmentation model is used to segment the regions in the pictures and only keep the road surface region, outputting a 1280*1280 image focusing on the road surface;

[0052] b4: For the focused image, perform bilateral filtering denoising processing to output an image that retains road surface details after denoising;

[0053] It also includes a coordinate conversion module that maps the road target pixel coordinates in the detection results output by the ultra-small target detection model to GPS coordinates;

[0054] The following operations are performed in the coordinate conversion module:

[0055] c1: Obtain the unmanned aerial vehicle pose parameters;

[0056] The unmanned aerial vehicle pose parameters include GPS position (lat0, lon0), attitude angle (ψ, ω, φ), and flight height H;

[0057] Wherein, lat0 is the latitude coordinate, lon0 is the longitude coordinate, ψ is the heading angle, ω is the pitch angle, and φ is the roll angle;

[0058] c2: Obtain the hardware parameters of the camera mounted on the unmanned aerial vehicle;

[0059] The hardware parameters include flight height H, camera intrinsic matrix K, and pixel size (dx, dy);

[0060] c3: converting the pixel coordinates of the detected target output by the model into camera coordinates;

[0061] The pixel coordinates (u, v) are converted into camera coordinates (X c ,Y c ,Z c ) by using the inverse matrix of the camera intrinsic matrix K.

[0062] c4: converting the camera coordinates of the detected target into world coordinates;

[0063] The camera coordinates are rotated to the direction of the world coordinate system by using the heading angle ψ, the pitch angle ω and the roll angle φ of the UAV, and then the position of the camera in the world coordinate system is subtracted from the rotated camera coordinates to obtain the world coordinates of the detected target.

[0064] c5: converting the world coordinates of the detected target into GPS coordinates;

[0065] The world coordinates of the detected target are converted into global GPS coordinates (lat, lon) of the detected target in combination with the GPS coordinates (lat0, lon0) of the UAV itself.

[0066] In step S5, the loss function L total used in the training process of the ultra-small target detection model is:

[0067] L total =L cls +L loc ;

[0068] Wherein, L cls is a classification loss function, and L loc is a positioning loss function.

[0069] The classification loss function L cls :

[0070] ;

[0071] In the formula, λ1 is a weight for controlling the proportion of the classification loss in the total loss, N is the total number of detected targets included in a single image, W i s is the scale weight corresponding to the i-th target at the s-th scale; α t is a balance factor corresponding to the class t, p i t is the probability that the model predicts that the i-th target belongs to the class t, and γ is a focusing parameter.

[0072] The positioning loss function L loc :

[0073] ;

[0074] wherein λ2 is a weight of a proportion of a positioning loss in a total loss, b i is a model predicted bounding box of the i-th target, b i t is a real labeled bounding box of the i-th target, and SIoU() is a bounding box regression loss function;

[0075] In step S5, in the training process of the super-small target detection model, a data enhancement module is used to generate training data;

[0076] The operations performed in the data enhancement module include the following contents:

[0077] d1: performing scale enhancement on sample data;

[0078] A random scaling factor α ~ U ∈ (0.5, 2.0) is set, the sample data is scaled, and then a cutting region is randomly selected from the scaled image in a preset fixed size to obtain training samples reflecting different scales and perspectives;

[0079] d2: performing perspective enhancement on sample data;

[0080] A perspective matrix M p ∈ R 3×3 is generated, P' = M p · P is solved by least squares, and the coordinates (u', v') of the pixel (u, v) in the sample data are transformed;

[0081] ;

[0082] wherein P is four vertices of image data, and P' is four offset vertices;

[0083] P = [(0, 0), (W, 0), (W, H), (0, H)],

[0084] P' = [(d1, d2), (W-d3, d4), (W-d5, H-d6), (d7, H-d8)];

[0085] The offset d i ~ U (-0.15W, 0.15W), i = 1 ~ 8;

[0086] m ij is an element of the perspective matrix M p ;

[0087] d3: performing motion blur enhancement on sample data;

[0088] Motion blur processing is performed on the sample data I by two-dimensional convolution:

[0089] I blur =B motion I,

[0090] in, For two-dimensional convolution operations; B motion For the fuzzy kernel ∈ R L×L ,

[0091] B motion The diagonal elements along the θ direction are 1, and the elements in the other directions are 0. After normalization, this satisfies B. motion (i,j)=1;

[0092] Where L is the fuzzy length L~U(1,5) and θ is the fuzzy direction θ~U(0,π);

[0093] d4: Perform occlusion and background enhancement processing on the sample data;

[0094] The road surface texture is cropped from the non-target area of ​​the sample data, and the target area is occluded by 10% to 30% to obtain the sample data of the target being occluded;

[0095] The target is cropped from the original image and randomly pasted onto a preset highway background to obtain various types of road surface background data.

[0096] This application provides a method for detecting road potholes and small road obstacles based on YOLOv8. A small target detection model is constructed based on the YOLOv8 model. In this model, the C2f module in the backbone network of the YOLOv8 model is replaced with the fuzzy adaptation C3 module. The input image is deblurred using a fuzz correction module, and then image features are extracted. Finally, a convolutional attention mechanism (CBAM) is applied to maintain detection accuracy even in blurred scenes. The small target detection model builds a small target branch on top of the YOLOv8 model. This branch uses the feature map of the last layer of the backbone network and upsamples it to obtain a high-resolution feature map of 1280×1280. The receptive field of the small target branch in the model is only 32×32, specifically adapted to small targets with pixel sizes <32×32, such as small gravel and tiny road potholes, thus solving the problem of high false negative rates for small targets in the original YOLOv8 architecture. The neck network in the ultra-small target detection model breaks through the limitation of independent feature extraction at each scale in YOLOv8. By adding ultra-small branches to complete the scale and combining self-attention association features, it achieves ultra-small target detection and linkage of multi-scale features, providing the output layer with globally correlated and full-scale coverage fused features, ensuring that this method is applicable to road pothole and small obstacle detection tasks. This application also includes a coordinate transformation module to convert the detection records of the ultra-small target detection model into GPS coordinates. After combining the GPS coordinates of the detected targets with electronic maps, the road surface conditions of the detected area can be reconstructed more accurately, providing more precise and easier-to-understand data support for road surface detection in vehicle test road surface detection, ensuring that this method is more practical. Attached Figure Description

[0097] Figure 1 This is a schematic diagram of the network structure of the ultra-small target detection model in this application;

[0098] Figure 2 This is a schematic diagram of the network structure of the fuzzy adaptation C3 module in this application;

[0099] Figure 3 This is a schematic diagram of the output head structure in this application;

[0100] Figure 4 This is a schematic diagram of the data processing flow for this application. Detailed Implementation

[0101] This application includes a method for detecting potholes and small obstacles on the road surface based on YOLOv8, which includes the following steps.

[0102] S1: Construct a fuzzy correction module;

[0103] Blur correction module; for input feature map F∈R H×W×C Image features F are extracted after deblurring.conv ; wherein: C is the number of channels, H and W are high and wide.

[0104] As shown in Figure 2 , the blur correction module includes: a 1x1 convolution layer , a reshape layer, a two-dimensional convolution layer and a main convolution layer K∈R k×k×C_in×C_out , wherein k is the convolution kernel of the main convolution layer, C_in is the input channel, C_out is the output channel, k b is the mapping size of the output of the convolution layer W b , k b reflects the blur direction and intensity.

[0105] After the input feature map F is sent into the 1x1 convolution layer, the 1x1 convolution layer maps the feature map F into a feature matrix of HxWxk b x k b , and sends the matrix into the reshape layer for dimension reshaping to obtain a blur kernel corresponding to each pixel ; in this embodiment, k b =5.

[0106] The input feature map F and the blur kernel B are simultaneously sent into the two-dimensional convolution layer for pixel-by-pixel blur kernel convolution, the input feature map F is deblurred by using the predicted blur kernel B, that is, deblurring correction is realized, and a deblurred feature F deblur =B F, wherein is two-dimensional convolution.

[0107] The deblurred feature is sent into the main convolution layer K, and the target feature F conv =K F deblur , K∈R k×k×C_in×C_out , in this embodiment, k=3.

[0108] S2: Construct a blur-adaptive C3 module. In this application, the feature extraction capability of the motion blur target is enhanced through the blur-adaptive C3 module, and the joint process of "deblurring + feature extraction" is simulated.

[0109] On the basis of the CSPDarknet of YOLOv8, a blur-adaptive C3 module is constructed. In this application, the convolution attention mechanism (Convolutional Block Attention Module, CBAM) and the blur correction module are introduced into the C2f module to obtain the blur-adaptive C3 module. The blur-adaptive C3 module extracts features after the input feature map is corrected by the blur correction module, and then outputs after attention enhancement processing by CBAM.

[0110] As shown in Figure 2 , the blur adaptive C3 module includes a CBS convolution block, a blur correction module, N Bottieneck-CBAM blocks connected in series, a Concat block, and a CBS convolution block. In each Bottieneck-CBAM block, a Bottieneck block and a CBAM module are connected in series. After blur adaptive convolution output, CBAM filters effective features and focuses on the target area through the series connection of channel attention and spatial attention.

[0111] The feature map input into the blur adaptive C3 module extracts an intermediate feature map via the CBS convolution block. The intermediate feature map is split into two parts, one of which is input into the Concat block, and the other is processed via the blur correction module and then input into the Bottieneck-CBAM block to extract a feature map. The Bottieneck-CBAM block extracts a feature map, which is input into the Concat block for splicing operation. The spliced feature map is output after being processed by the last CBS convolution block.

[0112] The blur adaptive C3 module deeply integrates deblurring and feature extraction, breaks through the limitation of the traditional YOLOv8 backbone network that only extracts features without processing blur, provides clear, focused, and high-recognizability basic features for the subsequent neck network, and ensures that the super-small target detection model can maintain high detection accuracy in a blur scene.

[0113] S3: Construct a cross-scale self-attention module based on a cross-scale self-attention mechanism (CSSA).

[0114] The cross-scale self-attention module CSSA enhances global context information through inter-scale feature interaction. The following operations are performed in the cross-scale self-attention module CSSA.

[0115] a1: Feature flattening and projection

[0116] The feature maps F s of each scale are flattened into a sequence of N s ×C, and are multiplied by W Q , W K , and W V , respectively, to obtain

[0117] The query matrix Q s =flatten(F s )·W Q , the key matrix K tt =flatten(F tg )·W K , and the value matrix V tt =flatten(F tt)·W V ;

[0118] where F s ∈R H_s×W_s×C , F s is the feature map of the s-th scale, F tt is the feature map of the tt-th scale, H s and W s are the height and width, and C is the number of channels; s and tt are the scale indexes of the feature map, s, tt ∈ {1, 2, 3, 4}, {1, 2, 3, 4} respectively correspond to the ultra-small, small, medium, and large scales; W Q , W K , W V ∈R C×d_k , are the projection matrices of the query Query, the key Key, and the value Value respectively, and d k is the projection dimension.

[0119] a2: attention weight calculation: calculate the similarity between scales by scaling the dot product attention formula, and then normalize by Softmax;

[0120] ;

[0121] A s,tt ∈R N_s×N_tt is the attention weight matrix from scale s to scale tt, N s =H s ×W s is the total number of pixels of the s-th feature map; N tt =H tt ×W tt is the total number of pixels of the tt-th feature map. The dimension of A s,tt is “N s rows × N tt columns”, and each element represents the association weight between a pixel in scale s and a pixel in scale tt. d k is the projection dimension, and the processing of d k root is used to alleviate the gradient disappearance problem caused by too high dimension.

[0122] a3: cross-scale feature fusion: weighted sum of the value matrix V s,tt of each scale using the attention weight A tt , reshape to the original feature map size, and finally add a residual connection to retain the original feature information:

[0123] ;

[0124] where, is the feature map after cross-scale fusion.

[0125] In the cross-scale self-attention mechanism, channel compression is realized, and the channel number of the feature map after attention fusion is d k = 256, which is compressed to the original channel number (such as the channel number of the super-small target branch remaining 256) through a 1x1 convolution, ensuring compatibility with the subsequent network structure. At the same time, the fused features are added to the original features F s element-wise through a residual connection, avoiding gradient vanishing caused by cross-scale interaction and improving training stability.

[0126] S4: Based on the YOLOv8 model, a super-small target detection model is constructed.

[0127] As shown in Figure 1 , the super-small target detection model in the method includes a backbone network, a neck, and a head network.

[0128] In the standard YOLOv8 model, the Backbone network is a CSPDarknet architecture. In the method, three key C2F modules in the CSPDarknet of YOLOv8 are replaced with fuzzy adaptive C3 modules. The original C2F module contains three convolution layers, and after modification, only the convolution layer structure at both ends is retained, and the middle core convolution layer is replaced with the fuzzy adaptive convolution of the application. At the same time, a CBAM attention module is added at the output end of the module, forming a three-order processing flow of fuzzy correction, feature extraction, and attention enhancement executed in turn, optimizing feature extraction, fusion, and output layer, and enhancing multi-scale perception and anti-fuzzy ability.

[0129] The backbone network includes three CBS convolution modules, three fuzzy adaptive C3 modules, and one SPPF module; each CBS module is followed by a fuzzy adaptive C3 module, and finally connected with an SPPF module; the network layers in the backbone network are labeled as P1~P7 from top to bottom.

[0130] The application realizes multi-scale feature fusion in the neck network (Neck). Based on the original FPN-PANet structure of YOLOv8, a cross-scale attention is added and integrated through a new branch to strengthen the detection ability of super-small targets and improve the global correlation of features of different scales. After each layer output of FPN, a cross-scale self-attention module CSSA (4 layers corresponding to 4 scales) is inserted to enhance the global context information through scale-to-scale feature interaction. The specific implementation is as follows.

[0131] The neck structure of the neck part includes four upsample layers, four C2f modules and four cross-scale self-attention modules CSSA; the upsample layers include Upsample1, Upsample2, Upsample3 and Upsample4 arranged from bottom to top, each upsample layer corresponds to a scale branch, and a C2f module is arranged behind each upsample layer.

[0132] The P7 layer output of the backbone network is input into the Upsample1 layer, the output of the Upsample1 layer is subjected to a Concat operation with the output of the P6 layer and then input into the subsequent C2f layer, the output of the Upsample2 layer is subjected to a Concat operation with the output of the P4 layer and then input into the subsequent C2f layer, and the output of the Upsample3 layer is subjected to a Concat operation with the output of the P2 layer and then input into the subsequent C2f layer.

[0133] Each CSSA module receives the output feature map of a C2f module to perform cross-scale feature fusion processing and then connects a detection head detect; the output feature map of each of the first three C2f modules is input into the subsequent Upsample layer in addition to being input into the CSSA module.

[0134] In the present application, on the basis of the original three branches of YOLOv8: large targets (160x160), medium targets (320x320) and small targets (640x640), a super-small target branch is newly added, the super-small target branch is based on the last layer feature map (original resolution is low) of the backbone network, and a high-resolution feature map of 1280x1280 is obtained through upsample operation (interpolation enlargement); the receptive field of the super-small target branch is only 32x32, and is specially adapted to super-small targets such as small gravel, road micro-pits and the like with a pixel size of <32x32, so as to solve the problem of high missing detection rate of the original architecture for super-small targets.

[0135] The neck network of the super-small target detection model breaks the limitation of independent extraction of features of different scales in the original YOLOv8, fills in the scale through the super-small branch, correlates the features through self-attention, realizes the detection of super-small targets and the linkage of multi-scale features, and provides the output layer with fused features with global correlation and full-scale coverage.

[0136] The head network includes four detection heads detect, which correspond to the detection results of four scales respectively. The structure of each detection head is as shown in the description accompanying Figure 3 drawings.

[0137] The present application is targeted at the target characteristics of multi-scale, irregular shape, scene adaptability requirements of the road detection scene. An adaptive anchor box and a multi-task output head are designed to realize the integrated output of "classification + positioning + confidence calibration". Based on the head network as the output layer and the multi-scale features fused based on the neck network, target classification, accurate positioning, and confidence calibration are realized, and finally the detection results adapted to the road scene are output.

[0138] Based on the four-scale detection branches constructed in the present application, the present application uses the original detection head structure of YOLOv8 to construct four detection heads. Specifically, through statistical analysis of the real labeled boxes by K-means clustering algorithm, four-scale anchor boxes of different types of targets adapted to the road are generated: dead animals, truck litter, and obstacles; each scale branch corresponds to three parallel task heads, which respectively realize classification, positioning, and confidence calibration.

[0139] Size distribution of different types of targets: dead animals (10x10~50x50), truck litter (20x20~100x100), and obstacles (50x50~200x200).

[0140] Clustering objective function: minimize the IoU loss of anchor boxes and labeled boxes to ensure the shape matching degree of anchor boxes and real targets:

[0141] ;

[0142] K=12, i.e. 3 anchor boxes for each of the 4 scales; r i,j =1 indicates that the labeled box i belongs to cluster j; (a j w ,a j h ) is the anchor box width and height of cluster j.

[0143] Final anchor box parameters:

[0144] Super-small branch (1280x1280): (8, 10), (12, 15), (16, 20);

[0145] Small branch (640x640): (22, 25), (30, 35), (40, 45);

[0146] Medium branch (320x320): (50, 60), (70, 80), (90, 100);

[0147] Large branch (160x160): (120, 130), (150, 160), (180, 200).

[0148] The classification head compresses the channel number of the feature map after cross-scale fusion through 1x1 convolution, and then outputs the class probability through Sigmoid.

[0149] ;

[0150] In the formula, σ is a Sigmoid function, Conv1 is a 1x1 convolution, and t is a target category. The specific category t of the detected target is obtained by training the training data according to actual test needs; in the present embodiment, the classification result output by the model includes three categories, that is, t takes values 1, 2 and 3, which respectively correspond to dead animals, truck sundries and obstacles. t is the probability that the target belongs to the category t.

[0151] The regression head: output 5-dimensional parameters (x, y, w, h, p) of the prediction box, where (x, y) is the center coordinate of the prediction box, (w, h) is the width and height of the box, and p is the rotation angle of the prediction box, which is used to adapt to irregular targets such as inclined roadblocks and irregular potholes.

[0152] Coordinate offset: △x and △y respectively represent the offset of the center of the prediction box relative to the center of the anchor box;

[0153] , ;

[0154] Width and height scaling: △w and △h represent the scaling ratio of the width and height of the prediction box and the anchor box of the current scale;

[0155] △w=log(w / a w s ),△h=log(h / a h s );

[0156] In the formula, a w s and a h s represent the width and height of the anchor box of the current scale s. The scaling ratio of the width and height relative to the anchor box is calculated by logarithmic transformation to avoid negative prediction values.

[0157] Rotation angle calculation: determine p by the direction angle of the long side of the prediction box, and the calculation method is:

[0158] p=arctan(△y long / △x long );

[0159] In the formula, △y long and △x long are the offset of the long side of the prediction box in the x and y directions respectively.

[0160] The confidence calibration head dynamically adjusts the confidence according to the scene complexity (such as night, blur, backlight), and reduces the false detection rate. After the feature map fused across scales is processed by 1x1 convolution and Sigmoid, a scene factor scene [0, 1] is output, and the more complex the scene is, the smaller the scene is, such as the night scene scene can be as low as 0.3.

[0161] The final confidence score is set as: score = p t x scene, ensuring that false detection boxes with low confidence in complex scenes are filtered.

[0162] In order to ensure that the output result of the detection head is available, a post-processing module is designed for the output of the detection head in the present application. The post-processing module solves the problems of repeated boxes and false detection boxes in the original detection results output by the detection head through a redundant box filtering operation, and outputs accurate and stable road target detection results. Dynamic Non-Maximum Suppression (DynamicNMS) in the post-processing module realizes redundant box filtering, and a differentiated strategy is designed according to the characteristics of the multi-scale target coexistence of the super-small target detection model road detection, the IoU threshold and the confidence threshold are adaptively adjusted according to the target scale, and the redundant boxes of road targets of different scales, from super-small gravel to large roadblocks, can be accurately filtered, while the effective target boxes are reserved. After the output result of the detection head is processed by the post-processing module, the redundant box set is output, which reduces the confusion problem of box matching in time sequence tracking and improves the tracking stability.

[0163] Specifically, the DynamicNMS is realized by sequentially performing threshold matching, sorting, and iterative filtering, ensuring that each step is adapted to the target scale characteristics. Among them, for each prediction box b i : coordinates (x, y, w, h, p), confidence score i , where p is the direction angle of the prediction box, first judge the scale level according to the product of the width and height of the box area = w x h, and then match the corresponding threshold:

[0164] Super-small target: area < 32x32 = 1024, match t IoU = 0.2, t score = 0.3;

[0165] Small target: 1024 ≤ area < 64x64 = 4096, match t IoU = 0.3, t score = 0.35;

[0166] Medium target: 4096 ≤ area < 128x128 = 16384, match t IoU = 0.4, t score = 0.4;

[0167] Large target: area >= 16384, match t IoU = 0.5, t score = 0.45.

[0168] At the same time, directly filter the confidence score score i <t score The prediction box, such as the super small target score i <0.3 box is directly deleted, and the false detection is preliminarily reduced.

[0169] The remaining calculation process based on dynamic NMS to improve the detection accuracy can be realized based on the prior art.

[0170] For the data characteristics of the highway photographed by the unmanned aerial vehicle, such as distortion, multiple backgrounds, resolution fluctuation, etc., a preprocessing module is also provided in the application, and operations such as noise reduction, correction, focusing effective area, etc. are realized in the preprocessing module to provide high-quality data for model input. The following steps are performed in the preprocessing module.

[0171] b1: data cleaning and standardization. Including the following operations:

[0172] Remove the samples with PSNR < 20 dB, missing or mislabeled samples, and keep the effective samples; resize the image to 1280x1280 to make the image consider the details of the super small target and the calculation efficiency, and specifically avoid the sawtooth in real-time process through bilinear interpolation; the mean and standard deviation of the RGB three channels of the input image are normalized, and the influence of brightness difference on the model is eliminated. The specific implementation process can be processed based on the prior art.

[0173] b2: For the cleaned picture, radial and tangential distortion correction is performed on the image collected by the unmanned aerial vehicle.

[0174] Generally, the camera carried on the unmanned aerial vehicle, such as a fisheye lens, has radial and tangential distortion, which needs to be corrected through camera calibration parameters. The correction process is:

[0175] The pixel coordinates (u, v) are normalized to image coordinates through the camera intrinsic parameters K, focal length f x and f y and principal point coordinates (c x , c y ). Then, the distortion coefficient of the camera is used to calculate the normalized coordinates after distortion, and the thinnest distortion is corrected to obtain the corrected pixel coordinates.

[0176] Specifically, the camera intrinsic parameters K, focal length f x and f y and principal point coordinates (c x , cy )For the camera with known parameters, the conversion method is based on the existing technology. The distortion coefficient and the specific calculation method of correcting the distorted coordinates are also based on the existing technology.

[0177] b3: Focus on the road surface area, extract ROI. For the corrected picture, use the pre-trained lightweight semantic segmentation model to segment the area in the picture and only keep the road surface area. Output the 1280*1280 image of the focused road surface.

[0178] Specifically, the following steps are included:

[0179] Step 1: Generate semantic segmentation mask:

[0180] For the picture I after distortion correction, output the mask M∈{0,1,2} through the semantic segmentation model, where 0=sky, 1=road surface, and 2=guardrail. Keep the road surface area M(u,v)=1 to get the road surface mask M road ;

[0181] That is, the value method of M road is: when M(u,v)=1, M road =1, and in other cases, M road =0.

[0182] Step 2: Crop and fill of ROI:

[0183] Input: the minimum bounding rectangle (u road ,v min ,u min ,v max ) of the mask M max ;

[0184] The cropped area is I crop =I[u min ∶u max ,v min ∶v max ], and it is filled to 1280×1280 by zero padding.

[0185] That is, in the cropped area: I roi (u,v)=I crop (u-u pad ,v-v pad ); in the padding area outside the cropped area: I roi (u,v)=0.

[0186] Where u pad ,v pad are padding offsets to ensure center alignment, and I roi (u,v) is the image output of the focused road surface area.

[0187] The input of the road area ROI operation is the corrected picture; the output is a 1280*1280 image of the focused road surface, which ensures the elimination of invalid backgrounds and improves the detection efficiency and accuracy of the subsequent detection because only the road area is saved, thereby effectively reducing the calculation amount.

[0188] The specific lightweight semantic segmentation model can be realized based on various semantic segmentation models in the prior art. In the present embodiment, a lightweight semantic segmentation model is constructed using MobileNetV2 and U-Net, and the specific construction details of the model are referred to: Computer Engineering and Applications, 2021, Vol. 57, Issue (17): 175-180. Fast Semantic Segmentation Network Fusing U-Net and MobileNet-V2.

[0189] The training sample data is constructed by collecting historical image data by the unmanned aerial vehicle. In the sample data, the image area is divided into three types of areas: "road surface", "sky", and "guardrail". Through semantic segmentation assisted ROI extraction, the "road surface" area (pixels with a mask of 1) is retained, cropped and filled to 1280*1280, the background interference is reduced, and the background proportion is reduced from 60% to below 30%.

[0190] b4: bilateral filtering denoising processing is performed on the focused image, and an image retaining road surface details after denoising is output.

[0191] For the noise generated by the sensor when the unmanned aerial vehicle collects pictures, such as Gaussian noise, salt and pepper noise, and mixed noise of the two, adaptive bilateral filtering is adopted to accurately suppress the mixed noise specific to the unmanned aerial vehicle, retain the key details required for road surface detection, and reduce the interference of background noise on detection.

[0192] The data size after the above three steps is 1280*1280, and it has three RGB channels. Adaptive bilateral filtering is adopted:

[0193] For low-noise areas (small gradient): a small filter kernel (3*3) is used for filtering processing to ensure that the details are retained.

[0194] For high-noise areas (large gradient): a large filter kernel (5*5) is used for filtering processing to ensure that the noise is smoothed.

[0195] The specific bilateral filtering operation includes:

[0196] First, the input picture is converted into a single-channel grayscale image I gray , which is used to calculate I x ,I y ;

[0197] I gray = 0.299R + 0.587G + 0.114B;

[0198] ;

[0199] ;

[0200] In the formula, Conv() is a convolution operation;

[0201] Input pixel gradient G(x, y):

[0202] ;

[0203] I x ,I y is the gradient in the x and y directions;

[0204] Set the filter kernel k of the bilateral filtering operation:

[0205] In the embodiment, T is 50;

[0206] Perform bilateral filtering calculation:

[0207] For the pixels (u', v') in the window W, calculate the spatial weight w s and the similarity weight w r :

[0208] ;

[0209] ;

[0210] Bilateral filtering result I denoise (u, v):

[0211] ;

[0212] In the formula, σ s is the standard deviation of the current pixel position, σ r is the standard deviation of the current pixel information, in the embodiment, σ s = 2, σ r = 30.

[0213] S5: training the super-small target detection model to obtain a trained super-small target detection model.

[0214] The loss function L total used in the training process of the super-small target detection model is:

[0215] L total = L cls + L loc ;

[0216] wherein, L clsFor the classification loss function, solve the problem of target class judgment error; L loc For the positioning loss function, solve the problem of prediction box and real box offset; the weights of the two are complementary to each other, and together guarantee the detection performance.

[0217] The core goal of the classification loss L cls is to make the model accurately distinguish the three types of road targets "dead animals, obstacles, and truck debris", and at the same time solve the following problems:

[0218] The scale of the road target is unbalanced: the number of small target samples is small and easy to miss, and the number of large target samples is large and easy to overfit; the class of the sample is unbalanced: for example, the "obstacle" sample accounts for a high proportion, and the "dead animal" sample accounts for a low proportion, which leads the model to be biased towards the majority class; the difficult and easy samples are unbalanced: the clear and large target simple sample is easy to learn, and the fuzzy and small target difficult sample is difficult to optimize.

[0219] The classification loss function L cls is realized based on Focal Loss:

[0220] ;

[0221] In the formula, λ1 is a weight for controlling the proportion of the classification loss in the total loss, which cooperates with the positioning loss to ensure that the class judgment and the position prediction are optimized synchronously; in this embodiment, λ1=1.0.

[0222] N is the total number of detection targets included in a single image.

[0223] p i t is the probability that the target i predicted by the small target detection model belongs to the class t, and after Sigmoid activation, p i t The value range is [0, 1];

[0224] γ is a focusing parameter, which reduces the weight of simple samples and increases the loss proportion of difficult samples (fuzzy and occluded targets); the focusing parameter γ is the classical value of FocalLoss, which has the optimal focusing effect and avoids that simple samples dominate the loss update. In this embodiment, γ=2.

[0225] (1-p i t ) γ is the focusing term; if the simple sample prediction is accurate, p i t is close to 1, the focusing term tends to 0, and the loss proportion decreases; if the difficult sample prediction is wrong, p i t is close to 0, the focusing term tends to 1, the loss proportion increases, and the model is forced to pay attention to such samples.

[0226] Wi s W i is the scale weight corresponding to the i-th target at the s-th scale; the method assigns different weights to targets of different scales, W s is set as: super-small target (s = 1): W i 1=1.5, small target (s = 2): W i 2=1.3, medium target (s = 3): W i 3=1.1, large target (s = 4): W i 4=1.0; priority is given to optimizing the super-small and small targets that are prone to be missed in road detection. Weight: W i s =1+λ·(1 / s i ), λ=0.5; target scale level s i ∈{1,2,3,4} respectively correspond to: super-small, small, medium and large.

[0227] α t is the balance factor corresponding to the t-th category, which balances the sample proportion of the three categories of targets to avoid model bias to the majority class; the specific value is adjusted according to the sample proportion, and in this embodiment, α 死亡动物 =0.3, α 障碍物 =0.4, α 货车杂物 =0.3; in this embodiment, based on the statistics of 10k+ labeled samples, the sample proportion of obstacles is the highest, so the weight is slightly higher.

[0228] Through the classification loss function L cls , the scale imbalance problem is solved, and the super-small target weight is the highest (1.5), which ensures that the model prioritizes learning the class features of small pits, gravel and other easily missed targets in training; the class distribution is balanced, and α t is adjusted to avoid the model ignoring the minority class targets such as dead animals due to the large number of obstacle samples; the model focuses on difficult samples: the loss proportion of fuzzy and occluded road targets (such as pits covered with fallen leaves) is increased, which promotes the model to optimize the classification accuracy in such scenarios.

[0229] The positioning loss function L loc is realized based on SIoU Loss.

[0230] The core goal of the positioning loss is to make the model's predicted bounding box (enclosing the pit / barrier) highly coincide with the true labeled box, especially suitable for the "irregular shape" of road targets (such as inclined barriers, elliptical pits). The algorithm does not use the traditional IoU Loss (only considers the overlapping area, ignoring the position and direction offset), but chooses SIoU Loss (Scaled IoU Loss), which quantifies the positioning error through "overlapping area, center distance, and direction angle" in three dimensions.

[0231] Direction angle θ calculation: θ = arctan ((y2-y1) / (x2-x1)) is determined by the long side direction of the minimum circumscribed rectangle of the bounding box. Where (x1, y1), (x2, y2) are the coordinates of the long side end points, describing the inclined direction of the target bounding box; adapt to irregular road targets, such as inclined roadblocks.

[0232] Weight λ2 = 2.0; control the proportion of positioning loss in the total loss through λ2, and give priority to guarantee the positioning accuracy. The position accuracy directly affects the subsequent GPS coordinate conversion in road detection.

[0233] The overlap area loss is used to measure the basic index of the overlap degree of the bounding box, and is calculated by IoU: IoU = |b∩b t | / |b∪b t |。

[0234] b is the model prediction box, represented by the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2); b t is the real label box, represented by the upper left and right corner coordinates (x t1 ,y t1 ,x t2 ,y t2 ). |b∩b t | is the intersection area of the prediction box and the real box; |b∪b t | is the union area; the larger the IoU value (close to 1), the higher the overlap degree, and the more accurate the positioning.

[0235] The traditional IoU ignores the center offset of the box, such as the high overlap area of the two boxes, but the center distance is far, and the actual positioning is still not accurate; SIoU quantifies this error by normalizing the center distance △:

[0236] ;

[0237] In the formula, (x c ,y c ) is the center coordinates of the prediction box, x c = (x1+x2) / 2, yc=(y1+y2) / 2; (x tc ,y tc ) is the center coordinates of the real box; w, h: prediction box width and height; w t ,h t : real box width and height. The denominator of △ is the maximum possible distance between the centers of the two boxes, that is, the Euclidean distance of the sum of half of the width and height of the two boxes, which ensures that △ is in the range [0, 1]: the larger the △, the more serious the center offset, and the higher the loss.

[0238] The road surface target is mostly irregular in shape, such as an inclined roadblock and a long strip-shaped pothole; the traditional IoU ignores the problem of frame direction offset, such as a predicted frame and a real frame being highly overlapped but vertically oriented, and the target is not actually framed; the SIoU quantifies this error through the direction angle difference: Ω = 1-cos(ρ-ρ t ); ρ is the direction angle of the predicted frame, and ρ t is the direction angle of the real frame; cos(ρ-ρ t ) ranges from -1 to 1, so Ω ranges from 0 to 2, when the directions are the same, ρ = ρ t , Ω = 0; when the difference between ρ and ρ t is large (such as vertical), Ω tends to 2, and the loss is significantly increased.

[0239] ;

[0240] In the formula, λ2 is the proportion weight of the positioning loss in the total loss, b i is the model predicted bounding box of the i-th target, b i t is the real labeled bounding box of the i-th target; the SIoU() is a bounding box regression loss function, which combines the overlap, center and direction errors through SIoU, and the SIoU ranges from -1 to 1, 1-SIoU ensures that the loss is non-negative, and the larger the value, the more serious the positioning error.

[0241] In step S5, before training the ultra-small target detection model, a data augmentation module is used to generate sample data and construct a training data set; the operations performed in the data augmentation module include the following.

[0242] d1: scale augmentation is performed on the sample data;

[0243] A random scaling factor α ~ U (0.5, 2.0) is set, and the sample data is scaled; then a fixed size is preset, and a cropping region is randomly selected from the scaled image to obtain training samples reflecting different scales and perspectives.

[0244] The random scaling factor α ~ U (0.5, 2.0) is uniformly distributed in the range (0.5, 2.0), and the scaled image size (H', W') = (αH, αW), where H and W are the original image height and width. When α < 1.0, the image is scaled down to simulate high-altitude shooting by a UAV, and the target scale becomes smaller; when α > 1.0, the image is scaled up to simulate low-altitude shooting by a UAV, and the target scale becomes larger.

[0245] In this embodiment, the preset fixed size is 1280*1280, which is consistent with the image size after preprocessing; by randomly selecting a clipping area from the scaled image, the size after clipping is fixed, covering the full scale range of the road target from ultra-small scale (<32*32 pixels) to large scale (50*50~200*200 pixels).

[0246] During flight, the UAV has problems such as target scale fluctuation caused by height change and shooting angle deviation caused by attitude tilt. The data enhancement module covers multi-scale road targets through "random scaling" and simulates the image perspective distortion (such as trapezoidal road edges) caused by the flight attitude tilt (such as roll and pitch) of the UAV through "perspective transformation", realizes the simulation of image features under different flight states, and ensures the detection robustness of the model to multi-scale and multi-angle targets.

[0247] d2: perspective enhancement of sample data;

[0248] Generate perspective matrix M p ∈R 3×3 , solve P'=M p ·P by least squares; transform the coordinates (u', v') of pixel (u, v) in sample data;

[0249] ;

[0250] wherein P is the four vertices of the image data, including: top left corner, top right corner, bottom right corner, bottom left corner; P' is the four vertices after offset;

[0251] P=[(0,0),(W,0),(W,H),(0,H)],

[0252] P'=[(d1,d2),(W-d3,d4),(W-d5,H-d6),(d7,H-d8)];

[0253] Offset d i ~U(-0.15W,0.15W), i=1~8, ensure that the offset range is controlled within 15% of the image width and height, and avoid excessive distortion of the target;

[0254] m ij is an element of the perspective matrix M p , the pixel value is filled by interpolation after conversion, ensuring the integrity of the image.

[0255] In the perspective enhancement operation, 4 vertex offsets (±15% image width and height) are randomly generated to simulate the perspective distortion (e.g., the road edge appears trapezoidal) caused by the tilt (e.g., roll, pitch) of the UAV flight attitude, avoiding missing detection in actual tilt scenes due to the fixed perspective of the model. Simulate common tilt shooting scenes of UAVs, such as roll over bending and pitch down, to ensure that the model can accurately detect targets in perspective distorted images

[0256] d3: motion blur enhancement on sample data.

[0257] Simulate motion blur (e.g., blurred edges of potholes when moving quickly) caused by UAV flight jitter to avoid missing detection in actual jitter scenes due to feature blurring. Calculate the original blurriness of the image by Laplacian variance:

[0258] Low blur (variance > 100): high image clarity, add 3-5 pixel blur kernel (probability 60%), simulate severe jitter;

[0259] High blur (variance < 50): high image clarity, add 1-2 pixel blur kernel (probability 30%), avoid feature loss due to excessive blur.

[0260] Motion blur processing on sample data I is achieved through two-dimensional convolution:

[0261] I blur =B motion I,

[0262] where, is a two-dimensional convolution operation; B motion is a blur kernel ∈ R L×L ,

[0263] B motion The diagonal elements in the direction of θ are 1, i.e., B motion (i,j)=1 / L, and the rest are 0; after normalization, B motion (i,j)=1.

[0264] where L is the blur length L~U(1,5), L is in pixels, corresponding to the jitter amplitude; θ is the blur direction, θ~U(0,π), randomly simulating the jitter direction, such as horizontal, vertical, and diagonal.

[0265] In the motion blur enhancement operation, cover the full scene from slight jitter (1-2 pixel blur) to severe jitter (3-5 pixel blur) of the UAV, to improve the feature extraction capability of the model for blurred targets.

[0266] d4: Occlusion and background enhancement processing is performed on the sample data, which extends the scene adaptation range of the model from a single background to multiple types of road surface backgrounds, avoiding accuracy degradation in special scenes such as tunnels and bridges.

[0267] The specific operation is:

[0268] Crop the road surface texture from the target-free area of the sample data, occlude 10% to 30% of the target area, and obtain target-occluded sample data;

[0269] Cut the target from the original image and randomly paste it on the preset highway background to obtain multi-type road surface background data.

[0270] In this application, the data enhancement module is used to expand the diversity and complexity of the training data, simulate the extreme scenarios that the unmanned aerial vehicle may encounter when shooting the road surface (such as inclined view angle, night lighting, target occlusion, etc.), and ultimately improve the generalization ability and anti-interference ability of the model, avoiding overfitting during training. The specific calculation details in the data enhancement module can be realized based on the existing technology.

[0271] S6: Deploy the trained ultra-small target detection model to the unmanned aerial vehicle, control the aircraft to fly over the preset detection area; use the image acquisition device arranged on the unmanned aerial vehicle as the input of the ultra-small target detection model, and output the road surface condition detection result.

[0272] In order to use the detection result output by the ultra-small target detection model in combination with the electronic map, a coordinate conversion module is also provided in this application, which maps the road target pixel coordinates in the detection result output by the ultra-small target detection model to GPS coordinates; after combining the GPS coordinates of the road target with the electronic map, the road surface condition of the detected area can be reconstructed more accurately.

[0273] The coordinate conversion module performs the following operations:

[0274] c1: Obtain the pose parameters of the unmanned aerial vehicle;

[0275] The pose parameters of the unmanned aerial vehicle include: GPS position (lat0, lon0), attitude angle (ψ, ω, φ) and flight height H;

[0276] Wherein, lat0 is the latitude coordinate, lon0 is the longitude coordinate; ψ is the heading angle, rotating around the z-axis, 0° is north; ω is the pitch angle, rotating around the y-axis, looking up is positive; φ is the roll angle, rotating around the x-axis, right side tilt is positive. The attitude angle is output in real time based on the inertial measurement unit (IMU) of the unmanned aerial vehicle. The flight height H is measured by the laser radar or barometric altimeter carried by the unmanned aerial vehicle.

[0277] c2: Obtain the hardware parameters of the camera carried by the unmanned aerial vehicle;

[0278] The hardware parameters include: flight altitude H, camera intrinsic parameter matrix K, and pixel size (dx, dy). The camera intrinsic parameter matrix K is obtained in advance through camera calibration (e.g., Zhang Zhengyou calibration method). The pixel size (dx, dy) is the physical size (meters / pixel) of a single pixel on the camera sensor, defaulting to the image resolution (1280×1280), and is the camera's factory parameter. In this embodiment, carrier aircraft such as the DJI M300 and NVIDIA Jetson are suitable for this method; only recalibration of the camera's intrinsic parameters and IMU parameters is required for rapid deployment.

[0279] c3: Converts the pixel coordinates of the detected target output by the model into camera coordinates;

[0280] Using the inverse of the camera intrinsic matrix K, the pixel coordinates (u,v) are converted to camera coordinates (X). c ,Y c Z c );

[0281] c4: Converts the camera coordinates of the detected target to world coordinates;

[0282] Using the UAV's heading angle ψ, pitch angle ω, and roll angle φ, the camera coordinates are rotated to the world coordinate system. Then, the camera's position in the world coordinate system is subtracted from the rotated camera coordinates to obtain the world coordinates of the detected target.

[0283] c5: Convert the world coordinates of the detected target to GPS coordinates;

[0284] By combining the UAV's own GPS coordinates (lat0, lon0), the world coordinates of the detected target are converted into the global GPS coordinates (lat, lon) of the detected target.

[0285] The specific methods for converting between world coordinates, camera coordinates, and image coordinate systems, as well as the method for converting world coordinates to GPS coordinates, can be implemented based on existing technologies.

[0286] like Figure 4 As shown, the images collected by the UAV are processed by the preprocessing module and then sent to the ultra-small target detection model, which is the core network module, for road target detection. The prediction results output by the ultra-small target detection model are sent to the post-processing module to improve the accuracy of the detection results, and then sent to the coordinate transformation module to convert the image coordinates of the detected targets into GPS coordinates.

[0287] The trained ultra-small target detection model can be adapted to most unmanned aerial vehicle embedded platforms, and the embodiment uses an NVIDIA Jetson AGX Orin; the C3 module of the Backbone is pruned by 30% of the low-weight channels (an accuracy loss of <2%) by calculating the channel importance through L1 regularization; post-training quantization (PTQ): use 1000 validation set images to calibrate, and quantize the model from FP32 to INT8, which improves the inference speed by 2 times and reduces the memory usage by 75%. The deployment framework during deployment is TensorRT8.6, which enables FP16 mode and Tensor Core acceleration; the preprocessing (distortion correction, ROI extraction) and model inference are processed in parallel, and the end-to-end delay is controlled within 40ms.

[0288] The KITTI and COCO2017 public datasets are used below to verify the generalization ability of the method in non-road scenarios. The original standard YOLOv8 (labeled as original YOLOv8 in the table) and the ultra-small target detection model constructed in the application are selected for comparison, and the ultra-small target detection model is labeled as the method model in the following table. To avoid overfitting of the improved algorithm to the self-defined road dataset, the generalization ability of the improved algorithm is tested on the COCO2017 and KITTI public datasets, and the universality of the core optimization such as multi-scale target detection and anti-fuzzy interference is verified.

[0289] The multi-scale generalization level is tested on the COCO2017 dataset, and small targets similar to road scenarios in COCO are selected, such as rock, bottle, and obstacle classes such as traffic cone and chair. The detection accuracy of the two algorithms is compared, and the verification results are shown in Table 1 below.

[0290] Table 1: Comparison of test results based on COCO2017 dataset

[0291]

[0292] From the data in Table 1 above, it can be seen that the "ultra-small target branch" and "cross-scale self-attention" of the improved algorithm in the application can still effectively improve the multi-scale detection accuracy on the COCO dataset, proving the universality of the optimization design, rather than just adapting to the self-defined road data.

[0293] The anti-fuzzy ability of the improved algorithm in the moving scene is tested on the KITTI dataset. The KITTI dataset contains fuzzy and dynamic scenes captured by a vehicle-mounted mobile camera, which is similar to the "motion blur" characteristics of unmanned aerial vehicle road detection. The anti-interference ability of the improved algorithm is verified, and the test result comparison is shown in Table 2 below.

[0294] Table 2: Comparison of test results based on KITTI dataset

[0295]

[0296] As can be seen from the data in Table 2 above, the Blur-AdaptiveC3 module in the super-small target detection model of the application performs outstandingly in the blur scene of KITTI, and the anti-blur ability is generalized to the vehicle mobile scene, proving that the solution to the motion blur problem has universality.

[0297] On the COCO and KITTI public data sets, the multi-scale detection accuracy and anti-blur ability of the model of the application are better than those of the original YOLOv8, proving that the optimization design is not “overfitting to specific data” and can be migrated to similar mobile detection scenes; through lightweight optimization (pruning, quantization), the inference speed of the improved algorithm on the unmanned aerial vehicle embedded platform (Jetson AGX Orin) reaches 32 FPS, meeting the real-time requirement of 25 FPS, the model volume is reduced by 75%, and there is no significant loss in accuracy.

[0298] Therefore, the super-small target detection model in the application is superior to the original YOLOv8 in the road pit and small roadblock detection tasks, and has cross-scene generalization potential, which is an engineering algorithm scheme that takes into account accuracy, efficiency and practicality.

Claims

1. A pothole and small barrier detection method based on YOLOv8, characterized by, It comprises the following steps: S1: constructing a blur correction module; The blur correction module; the input feature map F is extracted after deblurring processing H×W×C The image feature F is extracted conv ; wherein C is the number of channels, H and W are high and wide; S2: constructing a blur adaptive C3 module; On the basis of the CSPDarknet of YOLOv8, the blur adaptive C3 module is constructed; the blur adaptive C3 module extracts features after the input feature map is corrected by the blur correction module, and then outputs after attention enhancement processing by CBAM; S3: constructing a cross-scale self-attention module CSSA; The cross-scale self-attention module CSSA enhances global context information through inter-scale feature interaction; S4: based on the YOLOv8 model, constructing a super-small target detection model; The super-small target detection model comprises a backbone network, a neck and a head network; The backbone network comprises three CBS convolution modules, three blur adaptive C3 modules and one SPPF module; each CBS module is followed by a blur adaptive C3 module, and the SPPF module is connected at the end; the network layers in the backbone network are marked as P1-P7 from top to bottom; The network structure of the neck comprises four up-sampling Upsample layers, four C2f modules and four cross-scale self-attention modules CSSA; the up-sampling Upsample layer comprises Upsample1, Upsample2, Upsample3 and Upsample4 arranged from bottom to top, each up-sampling layer corresponding to a scale branch, and each up-sampling layer being followed by a C2f module; The output of the P7 layer of the backbone network is input into the Upsample1 layer, the output of the Upsample1 layer is concatenated with the output of the P6 layer and then input into the subsequent C2f layer, the output of the Upsample2 layer is concatenated with the output of the P4 layer and then input into the subsequent C2f layer, and the output of the Upsample3 layer is concatenated with the output of the P2 layer and then input into the subsequent C2f layer; Each CSSA module receives the output feature map of one C2f module for cross-scale feature fusion processing and then connects a detection head; the output feature maps of the first three C2f modules are input into the next up-sampling layer in addition to being input into the CSSA module; The head network comprises four detection heads corresponding to the detection results of four scales respectively; S5: training the super-small target detection model to obtain a trained super-small target detection model; S6: deploying the trained super-small target detection model to a UAV, controlling the aircraft to fly over a preset detection area; an image acquisition device arranged on the UAV is taken as a to-be-recognized image and input into the super-small target detection model, and a road condition detection result is output.

2. The pothole and small barrier detection method based on YOLOv8 according to claim 1, characterized in that: The blur correction module: 1x1 convolution layer arranged in sequence , reshape layer, two-dimensional convolution layer and main convolution layer K∈R k×k×C_in×C_out ; wherein, k is the main convolution kernel, C_in is the input channel, C_out is the output channel, k b is the convolution layer W b output mapping size; After the input feature map F is sent into the 1x1 convolution layer, the 1x1 convolution layer maps the feature map F into a feature matrix of HxWxk b ; and the reshape layer is sent into to reshape the matrix in dimension to obtain a blur kernel corresponding to each pixel b . ; The input feature map F and the blur kernel B are simultaneously sent into a two-dimensional convolution layer, and the input feature map F is deblurred by using the blur kernel B to obtain a deblurred feature F deblur =B F, wherein is a two-dimensional convolution; The deblurred features are sent to a main convolutional layer K to extract target features F conv =K F deblur .

3. The method according to claim 2, wherein the method is based on YOLOv8. The blur adaptive C3 module comprises a CBS convolution block, a blur correction module, N Bottieneck-CBAM blocks connected in series, a Concat block and a CBS convolution block; each Bottieneck-CBAM block comprises a Bottieneck block and a CBAM module connected in sequence. The feature map sent into the fuzzy adaptive C3 module extracts an intermediate feature map via a CBS convolution block, the intermediate feature map is split into two parts, one part is sent into a Concat block, and the other part is sent into a Bottieneck-CBAM block in series after being processed by the fuzzy correction module to extract a feature map, the feature map extracted by the Bottieneck-CBAM block is sent into the Concat block for splicing operation, and the spliced feature map is output after being processed by a last CBS convolution block.

4. The method according to claim 1, wherein the method is based on YOLOv8. The following operations are performed in the cross-scale self-attention module CSSA; a1: feature flattening and projection; The feature maps F s are flattened into a sequence of N s × C, respectively multiplied by W Q , W K , W V , resulting in: Query matrix Q s = flatten(F s ) · W Q , key matrix K tt = flatten(F tt ) · W K , value matrix V tt = flatten(F tt ) · W V ; wherein F s ∈ R H_s×W_s×C , F s is the feature map of the s-th scale, F tt is the feature map of the tt-th scale, H s and W s are height and width, and C is the number of channels; s and tt are scale indexes of the feature map, s, tt ∈ {1, 2, 3, 4}, {1, 2, 3, 4} respectively correspond to extra-small, small, medium, and large scales; W Q , W K , W V ∈ R C×d_k , are projection matrices of the query Query, the key Key, and the value Value respectively, d k is the projection dimension; a2: attention weight calculation: the similarity between scales is calculated by a scaled dot-product attention formula, and then normalized by Softmax; ； where A s,tt ∈ R N_s×N_tt is the attention weight matrix from scale s to scale tt, N s = H s × W s is the number of feature map pixels; N tt = H tt × W tt is the number of feature map pixels; a3: cross-scale feature fusion: with attention weight A s,tt value matrix V for each scale tt weighted sum, reshape to the original feature map size, and finally stack the residual connection to retain the original feature information: ； wherein, is the fused feature map across scales.

5. The method according to claim 1, wherein the method is based on YOLOv8. The detection head includes three parallel task heads, which are respectively a classification head, a positioning head and a confidence calibration head. The following operations are performed in the classification head: ； where σ is a sigmoid function, Conv1 is a 1x1 convolution, t is a target class, p t is a probability that the target belongs to class t. The positioning head outputs 5-dimensional parameters (x, y, w, h, ρ) of a target prediction box, wherein (x, y) is the center coordinate of the prediction box, (w, h) is the width and height of the box, and ρ is the rotation angle of the prediction box. The confidence calibration head outputs a confidence: score = p t x scene; In the formula, score is the confidence, and scene ∈ [0, 1] is the scene factor.

6. The method according to claim 1, wherein the method is based on YOLOv8. It also includes a preprocessing module, the pictures collected by the unmanned aerial vehicle are processed by the preprocessing module, and then sent into the ultra-small target detection model for detection; the following steps are performed in the preprocessing module: b1: data cleaning and standardization; Remove samples with missing or mislabeled fuzzy excessive targets, and retain valid samples; resize the image to 1280*1280; perform mean and standard deviation normalization on the RGB three channels of the input image respectively to eliminate the influence of brightness difference on the model; b2: For the cleaned pictures, radial and tangential distortion correction is performed on the images collected by the unmanned aerial vehicle; b3: For the corrected pictures, a pre-trained lightweight semantic segmentation model is used to segment the regions in the pictures and only keep the road surface region, outputting a 1280*1280 image focusing on the road surface; b4: For the focused image, a bilateral filter denoising process is performed to output an image that retains the road surface details after denoising.

7. The method according to claim 1, wherein the method is based on YOLOv8. It also includes a coordinate conversion module, which maps the road surface target pixel coordinates in the detection results output by the ultra-small target detection model to GPS coordinates; The following operations are performed in the coordinate conversion module: c1: obtain the unmanned aerial vehicle pose parameters; The unmanned aerial vehicle pose parameters include: GPS position (lat0, lon0), attitude angle (ψ, ω, φ) and flight height H; Wherein, lat0 is the latitude coordinate, lon0 is the longitude coordinate, ψ is the heading angle, ω is the pitch angle, and φ is the roll angle; c2: obtain the hardware parameters of the camera carried on the unmanned aerial vehicle; The hardware parameters include: flight height H, camera intrinsic matrix K and pixel size; c3: convert the pixel coordinates of the detection target output by the model into camera coordinates; The pixel coordinates (u, v) are converted into camera coordinates (X c ,Y c ,Z c ) using the inverse of the camera intrinsic matrix K. c4: convert the camera coordinates of the detection target into world coordinates; The camera coordinates are rotated to the direction of the world coordinate system by using the heading angle ψ, the pitch angle ω and the roll angle φ of the unmanned aerial vehicle, and then the position of the camera in the world coordinate system is subtracted from the rotated camera coordinates to obtain the world coordinates of the detection target; c5: converting the world coordinates of the detection target into GPS coordinates; In combination with the GPS coordinates (lat0, lon0) of the unmanned aerial vehicle itself, the world coordinates of the detection target are converted into global GPS coordinates of the detection target.

8. The method according to claim 1, wherein the method is based on YOLOv8. In step S5, the loss function L used in the training process of the super-small target detection model total is: L total =L cls +L loc ; wherein L cls is a classification loss function, L loc is a localization loss function; The classification loss function L cls : ； In the formula, λ1 is a weight for controlling the proportion of the classification loss in the total loss, N is the total number of detection targets included in a single image, W i s is the scale weight corresponding to the i-th target at the s-th scale; a t is the balance factor corresponding to the class t, p i t is the probability that the model predicts that the target i belongs to the class t, and γ is a focus parameter. The positioning loss function L loc : ； where λ2is the proportion weight of the positioning loss in the total loss, b i is the model predicted bounding box of the ith target i t is the real labeled bounding box of the ith target, and SIoU() is the bounding box regression loss function.

9. The method according to claim 1, wherein the method is based on YOLOv8. In step S5, during the training of the ultra-small target detection model, a data enhancement module is used to generate training data; The operations performed in the data enhancement module include the following: d1: scale enhancement of sample data; A random scaling factor α~U∈(0.5, 2.0) is set to scale the sample data, and then a preset fixed size is used to randomly select a cropping region from the scaled image to obtain training samples reflecting different scales and perspectives; d2: perspective enhancement of sample data; Generate perspective matrix M p ∈R 3×3 , solve P'=M p ·P by least squares; transformed coordinates (u', v') of pixel (u, v) in sample data; ； Wherein, P is the four vertices of the image data, P' is the four vertices after offset; P=[(0,0),(W,0),(W,H),(0,H)], P'=[(d1,d2),(W-d3,d4),(W-d5,H-d6),(d7,H-d8)]; Offset d i U(-0.15W, 0.15W), i = 1-8; m ij is an element of the perspective matrix M p ; d3: motion blur enhancement of sample data; Motion blur processing is performed on the sample data I by two-dimensional convolution: I blur =B motion I, wherein is a two-dimensional convolution operation; B motion is a blur kernel ∈ R L×L , B motion The diagonal elements in the θ direction are 1, and the rest are 0, and after normalization, B motion (i,j) = 1; Wherein, L is the blur length L~U(1, 5), θ is the blur direction θ~U(0, π); d4: occlusion and background enhancement processing of sample data; The road surface texture is cropped from the non-target area of the sample data, and the target is occluded by 10%~30%, to obtain sample data in which the target is occluded. The target is cropped from the original image and randomly pasted on the preset highway background to obtain multi-type road surface background data.

Images