Road defect detection method and system based on unmanned aerial vehicle inspection image

Abstract

The invention provides a road defect detection method and system based on an unmanned aerial vehicle inspection image, and relates to the technical field of image data processing, and the method comprises the steps: obtaining an original road image shot by an unmanned aerial vehicle; preprocessing the original road image to obtain a preprocessed road image; carrying out splicing processing on the preprocessed road image to obtain a complete road image; performing segmentation processing on the complete road image according to a preset size to obtain a plurality of road images to be detected; a road defect detection model based on improved YOLOv5 is constructed; and inputting the to-be-detected road image into the road defect detection model, and outputting a road defect detection result. The method can make up for the deficiency of single view angle shooting of the unmanned aerial vehicle, eliminates the shielding influence of vehicles running on the road, provides a more complete and clearer road image, and improves the accuracy of road defect recognition.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method and system for detecting road defects based on images from unmanned aerial vehicle (UAV) inspections. Background Technology

[0002] Drone road inspection is of great significance. Equipped with high-definition cameras and sensors, it can efficiently and accurately monitor and assess roads in real time. Compared with traditional manual inspections, drones can cover a wider area, reduce labor costs and risks, and improve inspection efficiency. Especially on complex terrain or busy roads, drones can quickly and reliably acquire high-definition image data, promptly identify road defects, damage, or safety hazards, support road maintenance and repair decisions, and thus effectively improve road safety and maintenance management.

[0003] Currently, the detection of road defects mainly relies on manual screening. Based on video images transmitted by drones, people manually inspect the roads for defects, but this method still takes a lot of time.

[0004] With the rapid development of image recognition technology, there are already technical solutions that apply image recognition technologies such as YOLO and U-Net to road defect detection. In actual shooting, due to factors such as the drone's flight altitude and angle, as well as the presence of vehicles on the road, image distortion or partial occlusion may occur, which may lead to inaccurate defect detection results. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a road defect detection method based on UAV inspection images, which can solve the technical problem that in the actual shooting process, due to factors such as the flight altitude and angle of the UAV and the presence of vehicles on the road, the image may be distorted or partially obscured, resulting in inaccurate defect detection results.

[0006] A first aspect of this invention provides a road defect detection method based on UAV inspection images, comprising:

[0007] S1: Acquire raw road images taken by a drone;

[0008] S2: Preprocess the original road image to obtain a preprocessed road image;

[0009] S3: The preprocessed road images are stitched together to obtain complete road images;

[0010] S4: The complete road image is segmented according to a preset size to obtain multiple road images to be detected;

[0011] S5: Construct a road defect detection model based on the improved YOLOv5;

[0012] S6: Input the road image to be detected into the road defect detection model and output the road defect detection result.

[0013] A second aspect of this invention provides a road defect detection system based on UAV inspection images, comprising: a processor and a memory;

[0014] The memory stores programs or instructions that can run on the processor, which, when executed by the processor, implement the steps of the road defect detection method based on UAV inspection images as described in the first aspect.

[0015] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0016] In this embodiment of the invention, a complete road image is obtained by stitching together road images. The complete road image is then segmented according to a preset size to obtain multiple road images to be detected. The road images to be detected are input into the road defect detection model, and the road defect detection results are output. The stitching process can make up for the deficiencies of a single viewpoint, eliminate the occlusion effect of vehicles on the road, provide a more complete and clear road image, and thus improve the accuracy of road defect recognition. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0018] Figure 1 This is a schematic flowchart of a road defect detection method based on UAV inspection images provided by an embodiment of the present invention.

[0019] Figure 2 This invention provides a road defect detection model based on an improved YOLOv5.

[0020] Figure 3 This is a schematic diagram of a road defect detection system based on UAV inspection images provided in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] The road defect detection method based on UAV inspection images provided by the present invention will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0023] Reference manual attached Figure 1 The diagram illustrates a flowchart of a road defect detection method based on UAV inspection images provided by an embodiment of the present invention.

[0024] This invention provides a road defect detection method based on UAV inspection images, which may include the following steps:

[0025] S1: Acquire raw road images taken by a drone.

[0026] S2: Preprocess the original road image to obtain a preprocessed road image.

[0027] Optionally, preprocessing includes brightness adjustment, contrast adjustment, and parallax correction.

[0028] S3: Stitch the preprocessed road images to obtain a complete road image.

[0029] In one possible implementation, S3 specifically includes sub-steps S301 to S308:

[0030] S301: Use the SURF (Speeded-Up Robust Features) feature point detection algorithm to extract image feature points from each preprocessed road image.

[0031] The SURF feature point detection algorithm, used to extract stable local feature points from images, is widely applied in tasks such as image matching, object recognition, and image stitching. SURF detects feature points by finding local extrema in scale space and uses the determinant of the Hessian matrix to measure the saliency of each point. Compared to the traditional SIFT algorithm, SURF optimizes computation by using integral images to accelerate image gradient calculation, thus significantly improving detection speed. SURF feature points are robust to rotation, scale changes, and some viewpoint transformations, enabling stable extraction of key features in complex environments. The SURF feature point detection algorithm is a mature existing technology and will not be elaborated further in this invention.

[0032] S302: Based on the quadtree partitioning method, image feature points are filtered, and the image feature points with the highest saliency within the node are selected, while redundant and invalid image feature points in the image are removed.

[0033] Specifically, the entire preprocessed road image is recursively divided into four sub-regions. When the number of feature points in a sub-region is greater than 1, the current sub-region is further divided into four sub-regions, and each image feature point is mapped to the sub-region as a child node according to its position. The splitting stops when the number of feature points in a node is greater than the desired number of feature points, forming a quadtree structure. The most representative feature point in each node is selected and retained.

[0034] In this embodiment of the invention, by filtering image feature points using a quadtree partitioning method, feature points can be effectively and evenly distributed. Furthermore, through a recursive segmentation and filtering process, redundant or invalid feature points are eliminated, thereby improving the quality and detection accuracy of feature points. The hierarchical partitioning of the quadtree avoids overly concentrated or uneven distribution of feature points in the image, ensuring that each region retains representative and significant feature points. This method enhances the stability of feature points, improves the reliability of image matching and stitching, while reducing the waste of computational resources and increasing detection efficiency.

[0035] Specifically, the image feature points with the highest saliency can be selected for retention. Saliency is usually a measure of the saliency of an image feature point in the image, representing the contribution of the local features of that point to the matching.

[0036] Alternatively, the significance can be calculated using the determinant of the Hessian matrix:

[0037]

[0038] Where R represents the significance, H represents the Hessian matrix, and det represents the determinant operation.

[0039] It's important to note that calculating saliency using the determinant of the Hessian matrix effectively measures the local curvature at a given location in an image, thus determining whether that point is a significant feature point. The Hessian matrix contains information about the second derivative of the image at that point, and the determinant value reflects the degree of change in the image at that point. A larger value indicates a more pronounced local change at that point, making it more likely to be a stable and identifiable feature point. Using the determinant as a measure of saliency not only improves the accuracy of feature point selection but also enhances the algorithm's robustness to transformations such as scaling and rotation, making feature points more reliable in tasks like image matching and stitching.

[0040] Alternatively, for each image feature point, the saliency can be represented by calculating the brightness difference between its surrounding pixels. If the brightness difference between the feature point and the brightness of other neighboring pixels is large enough, the point will be considered to have higher saliency.

[0041] It's important to note that using brightness difference to represent saliency effectively distinguishes points with distinct local features from background or low-contrast areas in an image. When the brightness of a feature point differs significantly from its neighboring pixels, it indicates that the point possesses strong local contrast and uniqueness, making it more representative of the image's salient features. This saliency calculation method helps identify stable and highly recognizable feature points, thereby improving the accuracy of image matching and object recognition, and better handling noise or lighting variations in complex environments.

[0042] S303: Separate the foreground and background of each preprocessed road image using the U-Net network.

[0043] U-Net is a convolutional neural network used for image segmentation tasks. Its network structure is characterized by a symmetrical encoder-decoder architecture, where the encoder progressively extracts high-level features from the image, while the decoder gradually restores the image's spatial resolution through upsampling. U-Net's core advantage lies in its skip connections, which directly pass low-level features from the encoder to the decoder, helping to recover image details and improve segmentation accuracy. U-Net can effectively handle complex image segmentation problems, especially performing exceptionally well with limited training samples, thus finding wide application in tasks such as foreground and background separation. U-Net is a mature existing technology and will not be elaborated upon further in this invention.

[0044] S304: Delete image feature points of moving objects in the foreground region, and retain feature points of static objects in the background region.

[0045] In this embodiment of the invention, deleting feature points of moving objects in the foreground region while retaining static objects in the background region effectively reduces the interference of dynamic objects (such as moving vehicles and pedestrians) on image processing and defect detection results. Dynamic objects may cause mismatches or unstable feature points, affecting subsequent image stitching and defect detection. By focusing on static feature points in the background region, the stability and accuracy of feature points can be improved, thereby enhancing the reliability of detection results, especially in complex environments or dynamic scenes, ensuring more accurate and consistent detection results.

[0046] In one possible implementation, after S304 and before S305, the following is also included:

[0047] Based on the foreground and background separation results, calculate the motion disturbance index of the current image caused by moving objects:

[0048]

[0049] Where MDI represents the motion interference index, A moving A represents the total area of ​​moving objects within the foreground region. total N represents the total area of ​​the image. moving N represents the number of feature points of a moving object within the foreground region. total This represents the total number of feature points in the image.

[0050] It should be noted that the motion interference index is a metric that measures the impact of moving objects in an image on the image processing process. It calculates a comprehensive value by combining the area of ​​the moving object within the foreground region and the number of feature points to quantify the degree of interference of the moving object on image detection.

[0051] Determine if the motion interference index of the current image is greater than the preset interference index. If so, trigger the multi-frame compensation mechanism. Otherwise, proceed to the next step.

[0052] Those skilled in the art can set the value of the preset interference index according to the actual situation, and the present invention does not limit it.

[0053] In this embodiment of the invention, the impact of moving objects on image quality and detection accuracy can be quantitatively assessed by calculating the Motion Interference Index (MDI). A high MDI value indicates significant interference from dynamic objects in the image, which may affect subsequent detection accuracy. In this case, triggering a multi-frame compensation mechanism utilizes information from multiple frames to reduce interference from moving objects in a single frame, thereby improving detection stability.

[0054] S305: Use the image feature points of static objects within the retained background area and perform feature point matching using the nearest neighbor matching algorithm.

[0055] The nearest neighbor matching algorithm is a simple and commonly used feature point matching method used to find the most similar pair of feature points between two images. In this algorithm, for each feature point in one image, the point in the other image that is most similar to its feature descriptor is searched. Similarity is typically determined by calculating the Euclidean distance between the feature descriptors. The point with the smallest distance is considered the match. This algorithm is simple to implement, but it is sensitive to noise and false matches. Therefore, it is often combined with other methods (such as distance ratio testing) to improve matching accuracy. The nearest neighbor matching algorithm is a very mature existing technology, and will not be elaborated upon further in this invention.

[0056] S306: Use the random sampling consensus algorithm to delete incorrectly matched feature point pairs.

[0057] Random Sample Consensus (RANSAC) is an algorithm used to estimate model parameters from data containing outliers, often used for error removal in feature point matching. In image matching, RANSAC randomly selects a small sample set from the matched point pairs and calculates the corresponding geometric transformation model (such as homography matrix, fundamental matrix, etc.). Then, it verifies whether the remaining matching points are consistent with the model, i.e., whether they meet a predetermined threshold condition. By iterating this process repeatedly, RANSAC can remove incorrectly matched point pairs and retain only those correctly matched points that are consistent with the model, thereby improving the accuracy and robustness of image registration. The Random Sample Consensus algorithm is a very mature existing technology and will not be elaborated upon in this invention.

[0058] Existing random sample consensus algorithms often use a fixed number of iterations. This fixed iteration strategy lacks adaptability and cannot automatically adjust the computational load based on the complexity of the image content (such as the magnitude of motion interference). Therefore, when processing different types of images, the algorithm may over-compute (low-interference images) or under-compute (high-interference images), thus affecting the overall detection efficiency and accuracy. This invention improves the iteration count of the random sample consensus algorithm.

[0059] In one possible implementation, the number of iterations of the random sample consensus algorithm is adjusted according to the motion disturbance index:

[0060]

[0061] Among them, T r T represents the number of iterations in the random sample consensus algorithm, min indicates taking the smaller value, and T d T represents the minimum number of iterations. s This represents the baseline iteration number, T. s Greater than at least the number of iterations T d .

[0062] Optionally, T d Take 200 times, T s Take 1000 times.

[0063] In this embodiment of the invention, when motion interference in the image is low (low MDI value), the impact of dynamic objects in the image on feature point matching is small, and RANSAC can achieve better matching results with fewer iterations, thereby improving efficiency. Conversely, when the MDI is high, it indicates greater interference from dynamic objects in the image, which may lead to more false matches. In this case, increasing the number of RANSAC iterations helps improve matching accuracy and reduce false matches. Through this dynamic adjustment, unnecessary computational resources can be saved while ensuring matching accuracy, thus improving overall processing efficiency.

[0064] S307: Based on the retained feature point pairs, match each preprocessed road image to determine the overlapping areas in each preprocessed road image.

[0065] It should be noted that, firstly, feature point matching algorithms (such as nearest neighbor matching) are used to find corresponding feature point pairs between different road images. Then, these matched feature point pairs are used to estimate the geometric transformation relationships between the images (such as homography matrices or affine transformation matrices), thereby determining the spatial alignment between the images. By calculating the geometric transformation model, overlapping areas between images, i.e., regions of the same road segment, can be accurately identified, providing a foundation for subsequent image stitching.

[0066] S308: A weighted fusion algorithm is used to stitch the images together by utilizing overlapping areas and optimizing the stitching gaps to obtain a complete road image.

[0067] Weighted fusion is a commonly used method in image stitching. It reduces stitching gaps and unnatural transitions by weighting pixels in overlapping areas. This algorithm assigns different weights to each pixel in the overlapping region, typically giving smaller weights to image edges and larger weights to the center, resulting in a smoother transition. Weighted fusion optimizes the boundaries of image fusion, making the transition between different images more natural and reducing stitching gaps caused by exposure differences, lighting variations, or color inconsistencies. This method effectively improves the quality of stitched images, making the resulting image more realistic and seamless. Weighted fusion is a mature existing technology and will not be elaborated upon further in this invention.

[0068] S4: Segment the complete road image according to a preset size to obtain multiple road images to be detected.

[0069] Those skilled in the art can set the specific size of the preset dimensions according to the actual situation, and the present invention does not limit it.

[0070] S5: Construct a road defect detection model based on the improved YOLOv5.

[0071] YOLOv5 (You Only Look Once version 5) is a high-efficiency real-time object detection algorithm and the fifth version in the YOLO series. It uses a single convolutional neural network (CNN) architecture and can simultaneously perform image classification and localization tasks.

[0072] In one possible implementation, the road defect detection model specifically includes: a backbone network, a neck network, and a detection head.

[0073] The main improvements in the upgraded YOLOv5 include:

[0074] (1) Introduce a deep fusion mechanism and a dual-channel differential mechanism in the backbone network.

[0075] (2) Introduce the sensory visual center mechanism into the neck network.

[0076] (3) Introduce a dynamic detection head into the detection head.

[0077] Reference manual attached Figure 2 This illustrates a road defect detection model based on an improved YOLOv5 provided by an embodiment of the present invention.

[0078] Figure 2 In this context, Backbone represents the backbone network, Neck represents the neck network, Head represents the detection head, Conv represents the Conv convolutional unit, DI represents the deep fusion unit of the deep fusion mechanism, CBS represents the CBS unit, C3Ghost represents the C3Ghost unit, SPPF represents the SPPF unit, PVC represents the perceptual visual central mechanism based on the perceptual visual central mechanism, Upsample represents the Upsample unit, Concat represents the Concat unit, and Dynamic Head represents the dynamic detection head.

[0079] In one possible implementation, the backbone network is structured as follows:

[0080] The first Conv convolutional unit, the second Conv convolutional unit, the third Conv convolutional unit, and the first deep fusion unit using a deep fusion mechanism are connected in series to form the input channel of the image to be detected. The input channel of the image to be detected is used to input the road image to be detected.

[0081] The fourth, fifth, and sixth Conv convolutional units, along with the second deep fusion unit employing a deep fusion mechanism, are connected in series to form a standard image input channel. This standard image input channel is used to input defect-free standard road images.

[0082] The input channel of the image to be detected and the input channel of the standard image are differentially connected to extract the differential features of the road image to be detected, thus forming a dual-channel differential mechanism.

[0083] In this embodiment of the invention, the dual-channel difference mechanism can effectively enhance the model's sensitivity to subtle differences between the image to be detected and the standard image. The image to be detected input channel processes the road image to be detected, while the standard image input channel processes the defect-free standard image. The two are differentially connected to extract the differences, thereby helping the model to more accurately identify defects in the image to be detected. This method can utilize the stable features of the standard image to compare with the differences in the image to be detected, making defect detection more accurate and effectively improving the model's recognition ability and robustness when processing road images.

[0084] Optionally, the first Conv convolutional unit has the same convolutional parameters as the fourth Conv convolutional unit, the second Conv convolutional unit has the same convolutional parameters as the fifth Conv convolutional unit, and the third Conv convolutional unit has the same convolutional parameters as the sixth Conv convolutional unit.

[0085] In this embodiment of the invention, by sharing convolution parameters between two channels, the network can perform efficient feature extraction based on multi-channel input, making the detection process more stable, avoiding overfitting, and improving the robustness of the network in different image scenarios.

[0086] Optionally, the deep integration mechanism is as follows:

[0087] A feature pyramid network structure is used to fuse the features extracted by each Conv convolutional unit:

[0088]

[0089] Among them, X DFFM This represents the output of the deep information feature fusion unit. This represents the features extracted by the second Conv convolutional unit. Upsample indicates the upsampling operation, ReLU indicates the ReLU activation function, BN indicates batch normalization, and Conv indicates the convolution operation. This represents the features extracted by the third Conv convolutional unit.

[0090] In this embodiment of the invention, the advantage of using a feature pyramid network structure for deep fusion is that it can effectively fuse features at different scales, improving the model's performance when handling multi-scale targets. By fusing the features extracted by the second convolutional unit with those of the third convolutional unit through upsampling, high-level semantic information and low-level detailed information can be fully utilized. Using ReLU activation, batch normalization, and convolution operations to process features helps enhance the network's non-linear expressive power and improve feature robustness. This mechanism, while preserving the detailed information of low-level features, can also capture a wider range of global features, thereby improving the model's accuracy and generalization ability, especially showing significant advantages in complex and variable detection tasks.

[0091] After differentially connecting the input channel of the image to be detected with the input channel of the standard image to extract differential features, the first CBS unit, the first C3Ghost unit, the second CBS unit, the second C3Ghost unit, and the SPPF unit are connected in sequence to form the backbone network.

[0092] The CBS (Convolution-Batch Normalization-Swish) unit is a common convolutional module used for image feature extraction. It consists of convolutional layers, batch normalization layers, and the Swish activation function.

[0093] The C3Ghost unit is an innovative convolutional structure designed to reduce computational cost and memory usage while maintaining good performance. It consists of multiple lightweight convolutional operations (such as Ghost convolutions) that extract more features with fewer parameters. The core idea of ​​Ghost convolutions is to enhance the network's feature extraction capabilities by generating more feature maps without significantly increasing computational overhead. The C3Ghost unit further optimizes computational efficiency by stacking multiple Ghost convolutional layers, making it suitable for resource-constrained devices such as embedded systems or mobile devices while maintaining high detection accuracy.

[0094] The SPPF (Spatial Pyramid Pooling Fast) unit is a module for multi-scale feature fusion, designed to enhance the network's ability to perceive targets at different scales. It performs spatial pyramid pooling on the input feature map, pooling information from different scales to improve the model's robustness when handling targets of varying sizes. By fusing multi-level information within pooling windows of different sizes, the SPPF unit enables the model to capture more detailed local features, effectively addressing the scale invariance problem and improving the accuracy and efficiency of target detection.

[0095] In one possible implementation, the neck network is specifically structured as follows:

[0096] The SPPF unit is sequentially connected to the third CBS unit, the perceptual visual center unit based on the perceptual visual center mechanism, the first upsampling unit, the first stitching unit, the third C3Ghost unit, the fourth CBS unit, the second upsampling unit, the second stitching unit, the fourth C3Ghost unit, the fifth CBS unit, the third stitching unit, the fifth C3Ghost unit, the sixth CBS unit, the fourth stitching unit, and the sixth C3Ghost unit. The first C3Ghost unit is connected to the first stitching unit. The differential features extracted from the differential connection between the input channel of the image to be detected and the standard image input channel are connected to the second stitching unit. The fourth CBS unit is connected to the third stitching unit, and the third CBS unit is connected to the fourth stitching unit, forming the neck network.

[0097] The third, fourth, and fifth C3Ghost units are connected to the dynamic detection head. The dynamic detection head is a module used in object detection tasks, designed to dynamically adjust the detection strategy based on the features of different inputs, thereby improving detection accuracy and flexibility. Unlike traditional fixed-structure detection heads, dynamic detection heads can adaptively adjust their output based on image content and contextual information, optimizing the detection process. This is typically achieved by introducing learnable parameters or mechanisms, allowing the network to flexibly select appropriate detection methods and output formats when processing different types of targets. Dynamic detection heads can improve the model's adaptability in complex scenes, especially when dealing with significant variations in target size, shape, and background complexity, enabling more accurate target location and recognition, and enhancing the overall performance of the object detection system. Dynamic detection heads have been widely used in other network architectures, and will not be elaborated upon here.

[0098] In one possible implementation, the sensory visual central mechanism specifically includes:

[0099] This invention creatively employs a lightweight multilayer perceptron and a learnable visual center to construct a perceptual visual central unit, thereby simulating the perceptual visual central unit.

[0100] Smooth the raw features input to the visual cortex:

[0101]

[0102] Among them, X p Represents the smoothed features, ReLU represents the ReLU activation function, and Conv... 7×7 This represents a 7×7 convolution operation, where X... r Indicates original features.

[0103] In this embodiment of the invention, smoothing the input features helps remove noise and enhance the local structural information of the image. Through this smoothing operation, the network can obtain a more stable feature representation, avoiding the impact of overly subtle local fluctuations on subsequent operations, thereby improving the robustness and consistency of the features. This provides higher quality input for subsequent multilayer perceptron and visual center feature extraction.

[0104] The smoothed features are then input into the multilayer perceptron and the vision center, respectively.

[0105] The multilayer perceptron employs a combination of depthwise separable convolutional residual modules and channel-wise MLP residual modules. The input of the channel-wise MLP residual module is the output of the depthwise separable convolutional residual module. The multilayer perceptron captures the global long-range dependencies of top-level features and extracts global features.

[0106]

[0107]

[0108] Where MLP stands for Multilayer Perceptron, MLP(X) p ) represents the global features extracted by the multilayer perceptron. This represents the output of the depthwise separable convolution residual module, which is also the input of the channel MLP residual module. DConv represents the depthwise separable convolution operation, GN represents normalization processing, and CMLP represents the channel MLP operation.

[0109] In this embodiment of the invention, the Multilayer Perceptron (MLP) employs a combination of depthwise separable convolutional residual modules and channel-based MLP residual modules, which effectively captures global long-range dependencies and extracts high-level global features. Depthwise separable convolutions reduce computational complexity, while channel-based MLP residual modules enhance the network's expressive power. This combination of MLPs improves the model's ability to capture global information, especially under complex features and contextually varying conditions.

[0110] The visual center encodes the smoothed features using a CBR module. The encoded features are then input into the codebook to extract local region features. These local region features are fed into a fully connected layer and a 1×1 convolutional layer to predict significant key features. These key features are then multiplied channel-wise with the smoothed features to obtain local corner features. Finally, these local corner features are added channel-wise with the smoothed features to extract local features.

[0111]

[0112]

[0113] Where VC represents the visual center, VC(X) p X represents the local features extracted from the visual center. z Indicates local corner features, This indicates channel-by-channel multiplication, where Y represents the local region features extracted from the code. This indicates channel-by-channel addition.

[0114] In this embodiment of the invention, the visual center encodes the smoothed feature map using the CBR module, and then inputs the encoded features into the codebook to extract local region features. This step helps to extract key detail features from local regions of the image. By focusing on the salient features of local regions, the model's ability to perceive local structures is enhanced, thereby improving the accuracy of key feature recognition.

[0115] The CBR module (Convolution-Batch Normalization-ReLU module) is a commonly used convolutional neural network (CNN) module structure, typically used in image feature extraction.

[0116] Furthermore, by multiplying the local region features with the smoothed features channel by channel (to obtain local corner features), and then adding the local corner features with the smoothed features channel by channel, local features and corners in the image can be effectively extracted. This process enhances the model's sensitivity to local details and helps improve the model's ability to detect small targets such as road defects.

[0117] Global and local features are concatenated along the channel dimension to form fused features:

[0118]

[0119] Among them, X PVC This represents the fusion characteristics of the visual perception center unit, and Concat represents the splicing operation.

[0120] In this embodiment of the invention, global and local features are concatenated along the channel dimension to form fused features, which can simultaneously preserve the global contextual information and local detail information of the image. This fusion not only enhances the model's ability to comprehensively utilize global and local features, but also improves the model's adaptability at different scales and levels of detail, thereby improving the overall performance of object detection, especially in complex scenes where it is more stable and accurate.

[0121] S6: Input the road image to be detected into the road defect detection model and output the road defect detection results.

[0122] The main categories of road defect detection include: cracks, potholes, ruts, peeling, damage, vegetation invasion, subsidence, and uplift.

[0123] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0124] In this embodiment of the invention, a complete road image is obtained by stitching together road images. The complete road image is then segmented according to a preset size to obtain multiple road images to be detected. These road images are then input into a road defect detection model, which outputs the road defect detection results. The stitching process can compensate for the deficiencies of a single viewpoint, eliminate the occlusion effect of vehicles on the road, and provide a more complete and clear road image, thereby improving the accuracy of road defect identification.

[0125] Reference manual attached Figure 3 The diagram shows a schematic representation of a road defect detection system based on UAV inspection images provided by an embodiment of the present invention.

[0126] This invention provides a road defect detection system 20 based on UAV inspection images, including: a processor 201 and a memory 202;

[0127] The memory 202 stores programs or instructions that can run on the processor 201. When the program or instructions are executed by the processor 201, they implement the steps of the road defect detection method based on UAV inspection images described above and achieve the same technical effect. To avoid repetition, the present invention will not elaborate further.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A road defect detection method based on UAV inspection images, characterized in that, include: S1: Acquire raw road images taken by a drone; S2: Preprocess the original road image to obtain a preprocessed road image; S3: The preprocessed road images are stitched together to obtain complete road images; S4: The complete road image is segmented according to a preset size to obtain multiple road images to be detected; S5: Construct a road defect detection model based on the improved YOLOv5; S6: Input the road image to be detected into the road defect detection model and output the road defect detection result.

2. The road defect detection method based on UAV inspection images according to claim 1, characterized in that, S3 specifically includes: S301: Use the SURF feature point detection algorithm to extract image feature points from each of the preprocessed road images; S302: Based on the quadtree partitioning method, the image feature points are filtered, the image feature points with the highest saliency within the node are selected, and redundant and invalid image feature points in the image are removed. S303: Separate the foreground and background of each of the preprocessed road images using the U-Net network; S304: Delete image feature points of moving objects in the foreground region, and retain image feature points of static objects in the background region; S305: Use the image feature points of static objects within the retained background area and perform feature point matching using the nearest neighbor matching algorithm; S306: Use the random sampling consensus algorithm to remove incorrectly matched feature point pairs; S307: Based on the retained feature point pairs, match each of the preprocessed road images to determine the overlapping areas in each of the preprocessed road images; S308: Using a weighted fusion algorithm, the overlapping areas are stitched together, and the stitching gaps of the stitched images are optimized to obtain the complete road image.

3. The road defect detection method based on UAV inspection images according to claim 2, characterized in that, After S304 and before S305, the following is also included: Based on the foreground and background separation results, calculate the motion interference index of the current image caused by moving objects; Determine if the motion interference index of the current image is greater than the preset interference index; if so, trigger the multi-frame compensation mechanism; otherwise, proceed to the next step.

4. The road defect detection method based on UAV inspection images according to claim 3, characterized in that, The number of iterations of the random sampling consensus algorithm is adjusted according to the motion interference index.

5. The road defect detection method based on UAV inspection images according to claim 1, characterized in that, The road defect detection model specifically includes: a backbone network, a neck network, and a detection head; A deep fusion mechanism and a dual-channel differential mechanism are introduced into the backbone network. A sensory visual center mechanism is introduced into the neck network; A dynamic detection head is introduced into the detection head.

6. The road defect detection method based on UAV inspection images according to claim 5, characterized in that, The specific structure of the backbone network is as follows: The first Conv convolutional unit, the second Conv convolutional unit, the third Conv convolutional unit, and the first deep fusion unit using the aforementioned deep fusion mechanism are connected in series to form the input channel for the image to be detected. The input channel for the image to be detected is used to input the image of the road to be detected. The fourth Conv convolutional unit, the fifth Conv convolutional unit, the sixth Conv convolutional unit, and the second deep fusion unit employing the aforementioned deep fusion mechanism are connected in series to form a standard image input channel, which is used to input a defect-free standard road image. The input channel of the image to be detected and the input channel of the standard image are differentially connected to extract the differential features of the road image to be detected, thus forming the dual-channel differential mechanism; After the differential features are extracted by differentially connecting the input channel of the image to be detected with the standard image input channel, the first CBS unit, the first C3Ghost unit, the second CBS unit, the second C3Ghost unit, and the SPPF unit are connected in sequence to form the backbone network.

7. The road defect detection method based on UAV inspection images according to claim 6, characterized in that, The deep fusion mechanism is specifically as follows: A feature pyramid network structure is adopted to fuse the features extracted by each Conv convolutional unit.

8. The road defect detection method based on UAV inspection images according to claim 6, characterized in that, The specific structure of the neck network is as follows: The SPPF unit is sequentially connected to the third CBS unit, the perceptual visual center unit based on the perceptual visual center mechanism, the first upsampling unit, the first stitching unit, the third C3Ghost unit, the fourth CBS unit, the second upsampling unit, the second stitching unit, the fourth C3Ghost unit, the fifth CBS unit, the third stitching unit, the fifth C3Ghost unit, the sixth CBS unit, the fourth stitching unit, and the sixth C3Ghost unit; the first C3Ghost unit is connected to the first stitching unit, the differential features extracted from the differential connection between the input channel of the image to be detected and the input channel of the standard image are connected to the second stitching unit, the fourth CBS unit is connected to the third stitching unit, and the third CBS unit is connected to the fourth stitching unit, thus forming the neck network; The third, fourth, and fifth C3Ghost units are connected to the dynamic detection head.

9. The road defect detection method based on UAV inspection images according to claim 8, characterized in that, The specific mechanisms of the visual perception center include: A lightweight multilayer perceptron and a learnable visual center are used to construct the perceptual visual center unit, simulating the perceptual visual center. The raw features input to the visual perception center are smoothed. The smoothed features are input into the multilayer perceptron and the visual center, respectively. The multilayer perceptron employs a combination of a depthwise separable convolutional residual module and a channel-wise MLP residual module. The input of the channel-wise MLP residual module is the output of the depthwise separable convolutional residual module. The multilayer perceptron captures the global long-range dependencies of the top-level features and extracts global features. The visual center encodes the smoothed features using the CBR module; the encoded features are input into the codebook to extract local region features; the local region features are input into a fully connected layer and a 1×1 convolutional layer to predict significant key features; the key features are multiplied channel-by-channel with the smoothed features to obtain local corner features; then the local corner features are added channel-by-channel with the smoothed features to extract local features. The global features and the local features are connected along the channel dimension to form a fused feature.

10. A road defect detection system based on UAV inspection images, characterized in that, include: Processor and memory; The memory stores programs or instructions that can run on the processor, which, when executed by the processor, implement the steps of the road defect detection method based on UAV inspection images as described in any one of claims 1 to 9.

Images