Unmanned aerial vehicle bridge slope disease detection method and system based on AI image recognition

Abstract

The invention belongs to the technical field of image recognition, and particularly relates to an unmanned aerial vehicle bridge slope disease detection method based on AI image recognition, and the method comprises the following steps: S1, carrying multi-sensor data collection by an unmanned aerial vehicle; s2, carrying out data standardization processing, and carrying out image fusion by using a weighted average method; s3, preprocessing the fused data; s4, expanding the training set through data enhancement, and enhancing the adaptability of the model to different environmental conditions; s5, training a deep learning model through enhanced data; s6, deploying a deep learning model by using edge computing equipment, and performing real-time data processing and reasoning; and S7, the severity is preset, and early warning is carried out. An efficient and accurate bridge disease automatic detection scheme is provided by integrating multi-modal data acquisition, a deep learning model and an edge computing technology; according to the method, the efficiency and accuracy of bridge disease detection can be greatly improved, and the method is suitable for large-scale bridge detection tasks and has wide application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and system for detecting bridge slope diseases by using AI image recognition for unmanned aerial vehicles (UAVs). Background Art

[0002] With the improvement of China's economy and national strength, the scale of bridge construction has been continuously expanding. By the end of 2020, the number of highway bridges in the country reached 912,800, an increase of 34,500 compared with the previous year. During the use of bridges, under the influence of external forces and natural aging, various surface diseases are likely to occur on the slopes. Therefore, it is crucial to detect and accurately evaluate bridge diseases in a timely manner to ensure the safe operation of bridges. Non-destructive testing techniques can obtain damage information of bridges without damaging the structure by using methods such as infrared rays and ultrasonic waves.

[0003] However, traditional non-destructive testing techniques have the problem of limited detection areas in bridge disease detection. Although manual measuring tools such as scales can detect bridge diseases, there are risks of insufficient accuracy and missed detections. With the development of UAV shooting technology, the manual evaluation method has significantly improved the detection efficiency. However, with the increase in the number of bridges, the limitations of the manual method have become increasingly apparent, resulting in problems such as increased workload and reduced efficiency.

[0004] To overcome the deficiencies of traditional detection methods, using the convolutional neural network (CNN) in deep learning to process a large number of bridge disease images, automatically extract image features, and locate the disease positions and categories has become a research hotspot for intelligent detection of bridge diseases. For example, in CN11985499B, a method for accurately identifying apparent diseases of bridges based on computer vision preprocesses a disease image training set, including Gaussian filtering and image pixel equalization, removes local abnormal noise points in the image, and at the same time merges pixels to improve the clarity of the training set images; by jointly using the characteristics of the multi-level edge detection algorithm of the Canny operator and the discrete difference algorithm characteristics of the Sobel operator, the disease boundaries of the images are optimized to achieve the accuracy of the weight coefficients during the training process of the YOLO neural network model; using the image simulation performance of the generative adversarial network, the generator is trained to generate a large number of simulated disease images to improve the disease recognition accuracy after the training of the neural network model.

[0005] However, this method relies on large datasets, and the neural network structure is complex, which may result in a large computational overhead during the training process; at the same time, this method may still be affected by environmental changes, especially in practical applications, and changes in image quality may lead to a decrease in recognition accuracy. Summary of the Invention

[0006] The object of the present invention is to provide a method and system for detecting diseases of bridge slopes by drones based on AI image recognition. This method uses drones and a variety of sensors, such as high-definition cameras, infrared thermal imagers, and lidar LiDAR, to collect multi-modal data of bridge slopes, ensuring that disease information on the surface and structure of the bridge is obtained from multiple dimensions.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The method for detecting diseases of bridge slopes by drones based on AI image recognition includes the following steps:

[0009] S1. Use a drone equipped with several sensors to collect multi-dimensional data of the bridge, ensuring that disease information on the surface and structure of the bridge is obtained from multiple dimensions;

[0010] S2. Align the data obtained by different sensors to the same coordinate system to ensure the precise matching of multi-dimensional collected data information; data fusion includes standardizing the data, performing feature point matching on multi-dimensional data using the SIFT-based image registration algorithm, calculating the transformation matrix between the data, and using the weighted average method for image fusion;

[0011] S3. After the multi-modal data fusion is completed, perform unified preprocessing on the fused data; remove image noise through Gaussian filtering, and enhance the contrast and details of the image through histogram equalization and adaptive light adjustment;

[0012] S4. After image preprocessing, expand the training set through data augmentation to enhance the model's adaptability to different environmental conditions and improve the model's generalization ability; use GANs to generate simulation data under different environments, and enrich the data set and enhance the robustness of the model by synthesizing disease images with different weather and light conditions;

[0013] S5. Train a deep learning model with the preprocessed and augmented data, and use depthwise separable convolution to reduce the computational amount by splitting the convolution operation; the formula is as follows:

[0014] y i =w i *x i

[0015]

[0016] where x i is the input feature map, w i is the convolution kernel, and y i is the convolution output;

[0017] Prune the model, remove unimportant neurons, and convert the network weights from floating-point numbers to low-precision integers to reduce memory occupancy and improve computing speed;

[0018] S6. Deploy the deep learning model using edge computing devices for real-time data processing and inference to reduce dependence on cloud computing and improve real-time response capabilities;

[0019] S7. Use the deep learning model YOLOX to analyze multi-modal data and detect the types of diseases on the bridge surface; the model outputs the position, category, and confidence of each detection box;

[0020] Calculate the severity of the disease:

[0021] Severity = A × C

[0022] where A is the area of the disease area and C is the confidence of the disease. When Severity exceeds the preset threshold, an alarm is automatically triggered.

[0023] Preferably, in S1, the drone is equipped with but not limited to a high-definition camera, an infrared thermal imager, and a lidar to collect image data, infrared thermal images, and depth maps of the bridge surface.

[0024] Preferably, in S2, the data normalization process includes processing the images to have a consistent size and color space, normalizing the temperature of the infrared thermal images, and mapping temperature values in different measurement units to a unified range; compressing the depth values to a specific range through linear normalization, set in the interval [0,1];

[0025]

[0026] where Dnorm is the normalized depth value, usually in the range [0,1];

[0027] D is the original depth value in the depth map;

[0028] Dmin and Dmax are the minimum and maximum depth values in the depth map, respectively.

[0029] Preferably, use the SIFT-based image registration algorithm to perform feature point matching on the images, thermal images, and depth maps, and calculate the transformation matrix between them; the specific process is as follows:

[0030] Extract the feature points of each image, perform matching, and find the corresponding relationships in the same area;

[0031] Use the RANSAC algorithm to remove outliers in the matching and calculate the transformation matrix H;

[0032] Apply a transformation matrix to each sensor data to ensure they are aligned in the same coordinate system;

[0033] Transformation formula:

[0034] H·p1 = p2

[0035] Where H is the homography matrix, P1 is the feature point in the reference image, and P2 is the corresponding feature point in the thermal image or depth map.

[0036] Preferably, use the weighted average method for image fusion, combining the data of RGB images, thermal images, and depth maps,

[0037]

[0038] Where i is the data of different sensors, RGB image, thermal image, depth map, W i is the corresponding weight, and Ifused is the result after fusion.

[0039] Preferably, in S4, image preprocessing includes performing Gaussian filtering on the image to remove noise and smooth the image; histogram equalization to improve image contrast and enhance details; using the adaptive histogram equalization AHE method to improve the details of the image under low light conditions.

[0040] Preferably, in S4, expand the dataset through the following operations to improve the robustness of the model; perform transformations such as rotation and translation on the image to increase the diversity of training data; brightness / contrast changes, randomly adjust the brightness and contrast of the image to simulate different environmental conditions; generate adversarial networks GANs, use the generative adversarial network to generate disease data under different environmental conditions, and enhance the generalization ability of the model;

[0041] The generation formula of GAN is as follows:

[0042]

[0043] Where G is the generator, D is the discriminator, x is the real sample, z is the noise input, p data is the data distribution.

[0044] A detection system using the above-mentioned UAV bridge slope disease detection method.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] 1. By equipping an unmanned aerial vehicle (UAV) with multiple sensors such as a high-definition camera, an infrared thermal imager, and a light detection and ranging (LiDAR), multimodal data of the bridge can be obtained from different dimensions, such as image data, thermal images, and depth maps. This multimodal data provides more comprehensive information, enabling the detection system to obtain more accurate disease identification results in different environments;

[0047] 2. Through the image registration algorithm based on Scale-Invariant Feature Transform (SIFT) and image fusion using the weighted average method, this method can align data from different sensors to the same coordinate system; this fused multimodal data enhances the accuracy of disease detection. For example, the fusion of thermal images and RGB images can help the system better distinguish surface diseases and deep diseases, thereby reducing false detections and missed detections.

[0048] 3. By performing image preprocessing through techniques such as Gaussian filtering, histogram equalization, and adaptive illumination adjustment, image noise can be effectively removed, image contrast can be improved, and details can be enhanced, thereby helping the model identify more subtle disease features. In addition, data augmentation (such as rotation, translation, brightness adjustment, etc.) expands the training set, enhances the robustness of the model, and further improves the generalization ability and detection accuracy of the model.

[0049] 4. This method uses lightweight neural networks such as YOLOX, enabling them to maintain high detection accuracy under conditions of low computing resources, and finding a good balance between speed and accuracy.

[0050] 5. This method deploys a deep learning model through an edge computing device for real-time data processing, reducing the dependence on cloud computing, thereby reducing latency and computational burden; further improving the speed of image processing and inference to ensure real-time detection requirements. Specific implementation manners

[0051] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0052] In the description of the present invention, it should be understood that the orientation or positional relationships indicated by the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention.

[0053] Example 1

[0054] With the increase in bridge infrastructure, bridge disease monitoring has become increasingly important. Traditional manual inspection methods have the disadvantages of low efficiency, serious missed inspections, and difficulty in coping with large-scale inspections. This example proposes an intelligent detection system and detection method based on drones and deep learning, which combines multi-sensor data acquisition and real-time data processing technology to achieve efficient and accurate detection of bridge surface and slope diseases.

[0055] System composition: Drone: Equipped with a high-definition camera, an infrared thermal imager, and a LiDAR (Light Detection and Ranging) sensor. Edge computing device: Used for real-time data processing, deep learning model inference, and result output. Data processing and fusion module: Responsible for the acquisition, registration, fusion, and preprocessing of multi-modal data. Deep learning model: A convolutional neural network (CNN) optimized based on YOLOX, used for disease detection. Intelligent warning system: Triggers an alarm when serious diseases are detected and notifies maintenance personnel. User interface and data visualization system: Provides visualization of real-time images and detection results and generates disease reports.

[0056] Implementation steps:

[0057] S1: Multi-sensor data acquisition. The drone is equipped with sensors, including but not limited to a high-definition camera, an infrared thermal imager, and a LiDAR.

[0058] Use the high-definition camera to obtain high-resolution images of the bridge surface, which can identify cracks, corrosion, and other surface diseases.

[0059] Use the infrared thermal imager to detect temperature anomalies on the bridge surface and identify potential water accumulation, temperature differences inside cracks, etc.

[0060] Use the LiDAR to generate a depth map, providing accurate bridge structure information to help analyze problems such as deformation and settlement.

[0061] Flight path planning: According to the geometric structure and disease characteristics of the bridge, plan the flight path of the drone to ensure comprehensive coverage of each important area of the bridge.

[0062] S2: Data registration and fusion, standardize the sensor data; the data standardization process includes processing the images to make them have a consistent size and color space, normalizing the temperature of the infrared thermal images, mapping temperature values in different measurement units to a unified range; compressing the depth values to a specific range through linear normalization, set in the interval [0,1].

[0063]

[0064] Among them, Dnorm is the normalized depth value, and its range is usually [0, 1];

[0065] D is the original depth value in the depth map;

[0066] Dmin and Dmax are the minimum and maximum depth values in the depth map respectively.

[0067] Extract the feature points of each sensor image (RGB image, thermal image, depth map) through the SIFT feature point matching algorithm, and calculate the transformation matrix between them;

[0068] The specific process is as follows:

[0069] Extract the feature points of each image, perform matching, and find the corresponding relationship of the same area;

[0070] Use the RANSAC algorithm to remove the outliers in the matching and calculate the transformation matrix H;

[0071] Apply the transformation matrix to each sensor data to ensure that they are aligned in the same coordinate system;

[0072] Transformation formula:

[0073] H·p1=p2

[0074] Among them, H is the homography matrix, P1 is the feature point in the reference image, and P2 is the corresponding feature point in the thermal image or depth map.

[0075] Use the RANSAC algorithm to remove the abnormal matches, ensure the accurate alignment of different sensor data, and form a unified multi-modal dataset.

[0076] Image fusion: Use the weighted average method to fuse the aligned images, combining the information of the RGB image, thermal image and depth map. Through weighted fusion, a fused image with more dimensional information can be obtained, making the disease information more abundant;

[0077] The weighted formula is as follows:

[0078]

[0079] Among them, i is the data of different sensors, RGB image, thermal image, depth map, W i is the corresponding weight, and Ifused is the fused result.

[0080] S4. Image preprocessing. Perform Gaussian filtering on the fused image to remove noise, use histogram equalization to enhance the image contrast, enhance the visibility of details such as cracks, and perform adaptive histogram equalization (AHE) on the images in low-light environments to improve the image details.

[0081] Gaussian Filtering: Apply Gaussian filtering to all types of images (RGB images, thermal images, depth maps) to remove noise, smooth the images, and reduce detail loss. This operation helps eliminate unnecessary image noise and makes subsequent feature extraction more accurate.

[0082] The Gaussian filtering formula is:

[0083] I smooth = I * G σ where I is the original image, and G σ is the Gaussian kernel, and σ smooth is the standard deviation of the Gaussian kernel.

[0084] Histogram Equalization: Improve the image contrast to make image details (such as crack and corrosion disease features) more obvious, especially under low-light conditions. Apply histogram equalization to RGB images and thermal images to enhance the visual effect of the images.

[0085] The histogram equalization formula is:

[0086]

[0087] where p(k) is the probability density of the pixel value k, and H(i) is the cumulative distribution function.

[0088] Adaptive Illumination Adjustment: Perform adaptive illumination adjustment on the images, especially in cases of low light or large illumination changes. Enhance image details through the Adaptive Histogram Equalization (AHE) method, which is particularly useful for bridge disease detection.

[0089] The Adaptive Histogram Equalization (AHE) formula is:

[0090]

[0091] where H AHE is the image processed by AHE.

[0092] Data Augmentation:

[0093] Geometric Transformation: Perform rotation, translation, scaling, and flipping operations on the images to increase the diversity of training data and help the model adapt to different bridge postures or directions;

[0094] Rotation formula (taking the angle θ as an example)

[0095]

[0096] where x, y are the coordinates in the original image, and x', y' are the coordinates after rotation.

[0097] Generate simulated disease data under different weather and lighting conditions using generative adversarial networks (GANs) to expand the training set and improve the robustness of the model.

[0098] Generate disease data under different environmental conditions using generative adversarial networks to enhance the generalization ability of the model;

[0099] The generation formula of GAN is as follows:

[0100]

[0101] Among them, G is the generator, D is the discriminator, x is the real sample, z is the noise input, and p data is the data distribution.

[0102] S5: Training and optimization of lightweight neural network models

[0103] Select the YOLOX model:

[0104] Use the YOLOX model for disease detection. It is superior to the traditional YOLO series in detection accuracy and has a faster inference speed.

[0105] Adopt depthwise separable convolution instead of traditional convolution to reduce the computational amount and maintain high accuracy.

[0106] Train the deep learning model with the preprocessed and enhanced data, and use depthwise separable convolution to reduce the computational amount by splitting the convolution operation; the formula is as follows:

[0107] y i = w i * x i

[0108]

[0109] Among them, x i is the input feature map, w i is the convolution kernel, and y i is the convolution output;

[0110] And prune the model, remove unimportant neurons, and convert the network weights from floating-point numbers to low-precision integers to reduce memory occupancy and improve computational speed;

[0111] Perform model pruning and quantization, remove unnecessary neurons, and convert the model weights from floating-point numbers to low-precision integers (such as 8 bits) to reduce memory occupancy and improve inference speed.

[0112] And prune the model, remove unimportant neurons, and convert the network weights from floating-point numbers to low-precision integers to reduce memory occupancy and improve computational speed;

[0113] S6. Deploy the deep learning model using edge computing devices for real-time data processing and inference to reduce reliance on cloud computing and improve real-time response capabilities;

[0114] S7. Use the deep learning model YOLOX to analyze multimodal data and detect the types of diseases on the bridge surface; the model outputs the position, category, and confidence of each detection box;

[0115] Calculate the severity of the disease:

[0116] Severity = A × C

[0117] where A is the area of the disease area and C is the confidence of the disease. When Severity exceeds the preset threshold, an alarm is automatically triggered.

[0118] Field test:

[0119] Conduct tests on an actual bridge to verify the accuracy, real-time performance, and stability of the system. During the test, use a drone to fly and inspect the bridge, collect and process data, and compare the detection results with the manual detection results to ensure the detection accuracy.

[0120] The above shows and describes the basic principles, main features, and advantages of the present invention. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic features of the present invention; therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, it is intended to include all changes falling within the meaning and scope of the equivalent elements of the claims in the present invention, and the claims should not be regarded as limiting the claims involved.

[0121] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting diseases of bridge slopes by drones based on AI image recognition, characterized in that, It includes the following steps: S1. Use a drone carrying several sensors to collect multi-dimensional data of the bridge, ensuring the acquisition of disease information on the bridge surface and structure from multiple dimensions; S2. Align the data obtained by different sensors to the same coordinate system to ensure the precise matching of multi-dimensional collected data information; perform data standardization processing, perform feature point matching on multi-dimensional data based on the SIFT-based image registration algorithm, calculate the transformation matrix between the data, and use the weighted average method for image fusion; S3. After the multi-modal data fusion is completed, perform unified preprocessing on the fused data; remove image noise through Gaussian filtering, and enhance the contrast and details of the image through histogram equalization and adaptive illumination adjustment; S4. After image preprocessing, expand the training set through data augmentation to enhance the model's adaptability to different environmental conditions and improve the model's generalization ability; use GANs to generate simulation data under different environments, and enrich the data set and enhance the model's robustness by synthesizing disease images with different weather and lighting conditions; S5. Train a deep learning model with the preprocessed and augmented data, and use depthwise separable convolution to reduce the computational amount by splitting the convolution operation; the formula is as follows: y i = w i * x i Among them, x i is the input feature map, w i is the convolution kernel, and y i is the convolution output; And prune the model, remove unimportant neurons, and convert the network weights from floating-point numbers to low-precision integers to reduce memory occupancy and improve computational speed; S6. Deploy the deep learning model using an edge computing device for real-time data processing and inference to reduce dependence on cloud computing and improve real-time response capabilities; S7. Use the deep learning model YOLOX to analyze multi-modal data and detect the disease types on the bridge surface; the model outputs the position, category, and confidence of each detection box; Calculate the severity of the disease: where A is the area of the disease area, C is the confidence of the disease, and when Severity exceeds the preset Severity = A × C threshold, an alarm is automatically triggered.

2. The method for detecting bridge slope diseases of an unmanned aerial vehicle according to claim 1, wherein: In S1, the drone carries but is not limited to a high-definition camera, an infrared thermal imager, and a lidar to collect image data, infrared thermal images, and depth maps of the bridge surface.

3. The method for detecting diseases of the bridge slope by using a drone according to claim 2, wherein: In S2, the data standardization processing includes processing the image to make it have a consistent size and color space, performing temperature standardization on the infrared thermal image, and mapping temperature values in different measurement units to a unified range; compressing the depth value to a specific range through linear normalization, and setting it in the range of [0,1]; where Dnorm is the normalized depth value, and the range is usually [0,1]; D is the original depth value in the depth map; Dmin and Dmax are the minimum and maximum depth values in the depth map respectively.

4. The method for detecting diseases of the bridge slope by using an unmanned aerial vehicle according to claim 3, wherein: Use the SIFT-based image registration algorithm to perform feature point matching on the image, thermal image, and depth map, and calculate the transformation matrix between them; the specific process is as follows: Extract the feature points of each image and perform matching to find the corresponding relationship in the same area; Use the RANSAC algorithm to remove outliers in the matching and calculate the transformation matrix H; Apply the transformation matrix to each sensor data to ensure that they are aligned in the same coordinate system; Transformation formula: H·p1 = p2 Among them, H is the homography matrix, P1 is the feature point in the reference image, and P2 is the corresponding feature point in the thermal image or depth map.

5. The method for detecting diseases of the bridge slope by using a drone according to claim 4, characterized in that: The weighted average method is used for image fusion, combining the data of RGB images, thermal images, and depth maps. Among them, i is the data of different sensors, RGB image, thermal image, depth image, and W i is the corresponding weight, and Ifused is the result after fusion.

6. The method for detecting diseases of the bridge slope by using an unmanned aerial vehicle according to claim 1, characterized in that: In S4, image preprocessing includes performing Gaussian filtering on the image to remove noise and smooth the image; histogram equalization to improve image contrast and enhance details; using the adaptive histogram equalization AHE method to improve the details of the image under low-light conditions.

7. The method for detecting diseases of the bridge slope by using a drone according to claim 1, characterized in that: In S4, the dataset is expanded through the following operations to improve the robustness of the model: performing transformations such as rotation and translation on the image to increase the diversity of training data; brightness / contrast variation, randomly adjusting the brightness and contrast of the image to simulate different environmental conditions; generative adversarial networks GANs, using generative adversarial networks to generate disease data under different environmental conditions to enhance the generalization ability of the model. The generation formula of GAN is as follows: Among them, G is the generator, D is the discriminator, x is the real sample, z is the noise input, and p data is the data distribution.

8. A detection system using the drone bridge slope disease detection method according to any one of claims 1-7.

Images