A UAV bridge crack detection method and system based on deep learning

Abstract

The present invention discloses a deep learning-based UAV bridge crack detection method and system. The method obtains more accurate environmental parameters through the cooperation of UAV A and UAV B. At the same time, UWB-assisted positioning is used to more conveniently obtain the image position of the UAV. The SSD algorithm is improved through shear-blending data enhancement, making the UAV crack detection more accurate.

Description

Technical Field

[0001] The present invention relates to the field of drone bridge detection, and in particular to a drone bridge crack detection method and system based on deep learning. Background Art

[0002] Transportation is a vital link connecting cities and has a decisive impact on the flow of production factors. Bridges, as an integral component of the transportation system, play a crucial role in complex areas such as rivers and ravines, becoming a crucial element in ensuring people's mobility. Due to years of exposure to external pressure and tension, as well as external factors such as overloading and exceeding limits, bridges inevitably suffer damage. Bridge defects primarily include cracks, weathering, spalling, exposed reinforcement, and rust. Cracks, as a major structural defect of bridges, can irreversibly damage their load-bearing capacity if not promptly repaired, severely impacting their safe operation and potentially even leading to major accidents. Therefore, effective bridge inspections are crucial.

[0003] Traditional bridge inspections rely primarily on manual testing, which has numerous shortcomings. First, due to external influences such as weather and temperature, as well as the necessary steps for precise installation and removal of inspection equipment, manual inspections are extremely inefficient. Second, manual inspections rely primarily on visual observation, which is susceptible to environmental and subjective factors, resulting in low accuracy. Third, bridges are often located in complex areas such as rivers and gullies. Manual inspections are not only limited by the location but also pose safety risks to inspectors. Fourth, manual inspections are costly and inefficient.

[0004] In recent years, with the significant increase in computing power, deep learning-based drone-based bridge defect detection has gained popularity in related fields, and bridge crack detection technology has developed rapidly. This method can effectively alleviate the shortcomings of traditional bridge inspection, but two major issues still exist: First, this method requires operators to be proficient in operating drones, which can easily lead to crashes in complex areas or areas with difficult visual access. Second, it is prone to flight blind spots, resulting in detection loopholes. Third, at the bottom of the bridge, there may be weak GPS signals, making it impossible for drones to locate. Finally, various target detection algorithms that have emerged in recent years, such as two-stage target detection algorithms represented by R-CNN, Fast-RCNN, and Faster-RCNN, and one-stage target detection algorithms represented by SSD and Yolo, can identify most cracks. However, for fine cracks, the detection accuracy of existing algorithms is low and cannot achieve the expected results. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method and system for detecting bridge cracks using a drone based on deep learning, which improves the detection accuracy by improving the SSD algorithm.

[0006] Technical solution: The present invention provides a deep learning-based UAV bridge crack detection method, which includes the following steps:

[0007] 1) Use surveying drone A to collect data on the bridge and its surrounding terrain, and transmit the data to the ground processing platform in real time;

[0008] 2) The ground processing platform performs 3D modeling based on the data transmitted by the surveying and mapping UAV A, obtaining a 3D coordinate real-scene model of the bridge to be measured and its surrounding environment;

[0009] 3) The ground station plans the flight path of the image acquisition drone B based on the 3D coordinate real-scene model and transmits the command to the flight control module of the image acquisition drone B;

[0010] 4) Simulate the flight of image acquisition drone B along the planned path. During the simulation, when the drone enters the bottom of the bridge, the UWB-assisted positioning module obtains the drone's position, transmits it to the ground station, and the ground station continuously corrects it to obtain the actual flight path. Image acquisition drone B actually flies along the actual flight path and transmits the collected images to the ground station in real time. The images are also stored in the storage module for offline processing.

[0011] 5) Preprocessing the collected image, which includes normalization, Gaussian filtering, and gamma correction;

[0012] 6) Use the improved SSD detection algorithm to detect the preprocessed image and complete the defect detection.

[0013] Furthermore, in step 3), the image acquisition drone B includes a flight control module, a ranging sensor module, an acceleration sensor module, a GPS positioning module, a UWB auxiliary positioning tag, a first data transmission radio, a fill light module, a gimbal camera, a storage module, a first image transmission radio, and an onboard data processing unit;

[0014] The flight control module is used to receive the flight path; the ranging sensor module is used to measure the distance between the UAV and the bridge; the acceleration sensor module is used to detect the acceleration of the UAV; the GPS positioning module is used to locate the UAV; the UWB auxiliary positioning tag is used to assist in positioning when the GPS signal at the bottom of the bridge is weak; the first data transmission radio is used to transmit data between the UAV and the second data transmission radio of the ground station; the gimbal camera is used to collect images; the fill light module is used to supplement the light source for the gimbal camera in low-light environments; the storage module is used to store the collected data for offline processing; the first image transmission radio is used to transmit data between the UAV and the second image transmission radio of the ground station; the onboard data processing unit is used to control the posture of the gimbal camera and send the collected signals.

[0015] Furthermore, in step 5), the collected image is preprocessed, wherein the normalization formula is as follows:

[0016]

[0017] where x i Represents the image pixel value, max(x) represents the maximum pixel value, Represents the minimum pixel value.

[0018] Furthermore, in step 5), gamma correction is applied to the image captured in dark conditions, and the formula is as follows:

[0019]

[0020] Where γ is an adjustable coefficient and I is the current image.

[0021] Furthermore, in step 6), the improved SSD model includes the following steps:

[0022] 6.1) Use the shear mixing data augmentation method to create a dataset, where the shear mixing formula is as follows:

[0023] α=M☉x A +(1-M)☉x B

[0024] β=μy A +(1-μ)y B

[0025] Among them, x A and x B are two different training samples, y A and y B is the corresponding label value. Cutting and mixing need to generate new training samples and corresponding labels α, β, M∈(0,1) W×His a binary mask, ☉ is pixel-by-pixel multiplication, and μ belongs to the Beta distribution: μ ~ Beta(a,a), a=1 obeys the (0, 1) uniform distribution;

[0026] The bounding box of the clipping area B = (r x ,r y ,r w ,r h ) is used to sample the sample x A and x B To calibrate the indication of the cropping area, the formula is as follows:

[0027] r x ~Unif(0,W);

[0028] r y ~Unif(0,H);

[0029] After determining the cropping area B, set the cropping area B in M to 0 and other areas to 1 to complete the mask sampling, remove the cropping area B in sample A, crop the cropping area B in sample B and then fill it into sample A to obtain the data set;

[0030] 6.2) Fusing the feature maps of high and low layers, performing CBL operations on fully connected layers 6 and 7 respectively and then downsampling operations to improve the SSD backbone network;

[0031] 6.3) Calculate the loss function of the improved SSD. The formula is as follows:

[0032]

[0033] The first part of the loss function is the category loss, the second part is the position loss, and N is the number of matched prior boxes.

[0034] The present invention provides a deep learning-based UAV bridge crack detection system, which includes an acquisition module, a three-dimensional modeling module, a path planning module, a flight simulation module, a preprocessing module, and a detection module;

[0035] The acquisition module is used to use the surveying drone A to collect data about the bridge and its surrounding terrain, and transmit the data to the ground processing platform in real time;

[0036] The 3D modeling module is used to perform 3D modeling based on the data transmitted by the surveying and mapping UAV A using a ground processing platform, thereby obtaining a 3D coordinate real-scene model of the bridge to be measured and its surrounding environment.

[0037] The path planning module is used to plan the flight path of the image acquisition drone B based on the 3D coordinate real-scene model through the ground station and transmit the instructions to the flight control module of the image acquisition drone B;

[0038] The flight simulation module is used to simulate the flight of image acquisition UAV B according to the planned path. During the simulation, when the UAV enters the bottom of the bridge, the UWB-assisted positioning module obtains the UAV's position, which is then transmitted to the ground station. The ground station then continuously corrects the position to obtain the actual flight path. Image acquisition UAV B actually flies according to the actual flight path and transmits the collected images to the ground station in real time. The images are then stored in the storage module for offline processing.

[0039] The preprocessing module is used to preprocess the collected images, and the preprocessing methods include normalization, Gaussian filtering, and gamma correction;

[0040] The detection module is used to detect the pre-processed image through the improved SSD detection algorithm to complete the detection of defects.

[0041] Furthermore, in the path planning module, the image acquisition drone B includes a flight control module, a ranging sensor module, an acceleration sensor module, a GPS positioning module, a UWB auxiliary positioning tag, a first data transmission radio, a fill light module, a gimbal camera, a storage module, a first image transmission radio, and an onboard data processing unit;

[0042] The flight control module is used to receive the flight path; the ranging sensor module is used to measure the distance between the UAV and the bridge; the acceleration sensor module is used to detect the acceleration of the UAV; the GPS positioning module is used to locate the UAV; the UWB auxiliary positioning tag is used to assist in positioning when the GPS signal at the bottom of the bridge is weak; the first data transmission radio is used to transmit data between the UAV and the second data transmission radio of the ground station; the gimbal camera is used to collect images; the fill light module is used to supplement the light source for the gimbal camera in low-light environments; the storage module is used to store the collected data for offline processing; the first image transmission radio is used to transmit data between the UAV and the second image transmission radio of the ground station; the onboard data processing unit is used to control the posture of the gimbal camera and send the collected signals.

[0043] Furthermore, in the preprocessing module, the collected image is preprocessed, wherein the normalization formula is as follows:

[0044]

[0045] where x i Represents the image pixel value, max(x) represents the maximum pixel value, Represents the minimum pixel value.

[0046] Furthermore, in the pre-processing module, gamma correction is applied to images collected in dark conditions. The formula is as follows:

[0047]

[0048] Where γ is an adjustable coefficient and I is the current image.

[0049] Furthermore, in the detection module, the improved SSD model includes a dataset production unit, an improved SSD backbone network unit, and a loss function unit;

[0050] The data set production unit is used to produce the data set using the shear mixing data enhancement method, wherein the shear mixing formula is as follows:

[0051] α=M☉x A +(1-M)☉x B

[0052] β=μy A +(1-μ)y B

[0053] Among them, x A and x B are two different training samples, y A and y B is the corresponding label value. Cutting and mixing need to generate new training samples and corresponding labels α, β, M∈(0,1) W×H is a binary mask, ☉ is pixel-by-pixel multiplication, and μ belongs to the Beta distribution: μ ~ Beta(a,a), a=1 obeys the (0, 1) uniform distribution;

[0054] The bounding box of the clipping area B = (r x ,r y ,r w ,r h ) is used to sample the sample x A and x B To calibrate the indication of the cropping area, the formula is as follows:

[0055] r x ~Unif(0,W);

[0056] r y ~Unif(0,H);

[0057] After determining the cropping area B, set the cropping area B in M to 0 and other areas to 1 to complete the mask sampling, remove the cropping area B in sample A, crop the cropping area B in sample B and then fill it into sample A to obtain the data set;

[0058] Improve the SSD backbone network unit to fuse the feature maps of high and low layers, perform CBL operation on the fully connected layer 6 and the fully connected layer 7 respectively, and then perform downsampling operation to improve the SSD backbone network;

[0059] The loss function unit is used to calculate the loss function of the improved SSD. The formula is as follows:

[0060]

[0061] The first part of the loss function is the category loss, the second part is the position loss, and N is the number of matched prior boxes.

[0062] Beneficial effect: Compared with the existing technology, the significant advantage of the present invention is that more accurate environmental parameters can be obtained through the cooperation of UAV A and UAV B. At the same time, the UAV image position can be more conveniently obtained through UWB-assisted positioning, and the SSD algorithm can be improved through shear blending data enhancement, making the detection of UAV cracks more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a diagram of the three-dimensional modeling process of the bridge in the present invention;

[0064] Figure 2 Schematic diagram of the structure of the UAV B in the present invention;

[0065] Figure 3 It is the route planning diagram of UAV B in the present invention;

[0066] Figure 4 This is a flow chart of the simulated flight route of UAV B in the present invention;

[0067] Figure 5 This is an image gamma correction effect diagram in the present invention;

[0068] Figure 6 This is a diagram showing the effect of shear mixing data enhancement in the present invention;

[0069] Figure 7 This is the improved SSD backbone network diagram of the present invention;

[0070] Figure 8 This is a sample diagram of the bridge defect detection results in the present invention. DETAILED DESCRIPTION

[0071] The specific embodiments of the present invention are described below with reference to the accompanying drawings.

[0072] Example 1

[0073] The present invention provides a deep learning-based UAV bridge crack detection method, which includes the following steps:

[0074] 1) Use surveying drone A to collect data on the bridge and its surrounding terrain, and transmit the data to the ground processing platform in real time.

[0075] Surveying drone A is equipped with an oblique aerial camera and a communication device. The oblique aerial camera consists of five cameras, each facing forward, backward, left, right, and vertically downward. It simultaneously captures images from both vertical and oblique angles, providing more accurate information. The film captured perpendicular to the ground is called a positive film. The four sets of images captured at an angle to the ground are called oblique films, which provide a better view of the model under test than perpendicular images. Phased control points are deployed after the flight plan is developed.

[0076] First: To ensure the accuracy of the 3D model, the heading overlap and lateral overlap must be at least 80%; second: the distance between the camera and the model cannot be too large; third: the model to be measured must occupy more than half of the area of the photo, and each part must appear in at least 3 images; the communication device records the images and position information collected by the aerial camera and transmits them to the processing platform in real time for 3D modeling.

[0077] 2) The ground processing platform performs three-dimensional modeling based on the data sent back by surveying and mapping UAV A to obtain a 3D coordinate real-scene model of the bridge to be measured and its surrounding environment.

[0078] See also Figure 1 As shown in the figure, the processing platform is a 3D modeling software developed based on the principle of oblique photography. When images with a certain degree of overlap are imported, the modeling software will automatically perform aerial triangulation based on the position information contained in the image to generate a dense point cloud to form a triangular mesh model. Finally, the 3D model will be generated based on the pixel information of the oblique image.

[0079] 3) The ground station plans the flight path of the image acquisition drone B based on the 3D coordinate real-scene model and transmits the instructions to the flight control module of the image acquisition drone B.

[0080] See also Figure 2 As shown, the image acquisition drone B is also called a drone system, which includes a flight control module, a fuselage stabilization module, a ranging sensor module, an acceleration sensor module, a GPS positioning module, a UWB auxiliary positioning tag, a first data transmission radio, a fill light module, a gimbal camera, a storage module, a first image transmission radio, and an airborne data processing unit.

[0081] The ground station is called the ground end system, which includes the ground station, the second data transmission radio, the second image transmission radio, and the UWB positioning base station.

[0082] The flight control module is used to receive the flight path; the fuselage stabilization module is used to keep the fuselage stable when collecting images; the ranging sensor module is used to measure the distance between the UAV and the bridge; the acceleration sensor module is used to detect the acceleration of the UAV; the GPS positioning module is used to locate the UAV; the UWB auxiliary positioning tag is used to assist in positioning when the GPS signal at the bottom of the bridge is weak; the first data transmission radio is used to transmit data between the UAV and the second data transmission radio of the ground station; the gimbal camera is used to collect images; the fill light module is used to supplement the light source for the gimbal camera in low-light environments; the storage module is used to store the collected data for offline processing; the first image transmission radio is used to transmit data between the UAV and the second image transmission radio of the ground station; the onboard data processing unit is used to control the posture of the gimbal camera and send collection signals.

[0083] 4) Simulate the flight of image acquisition UAV B according to the planned path. During the simulation, when the UAV enters the bottom of the bridge, the UWB-assisted positioning module obtains the UAV's position, transmits it to the ground station, and the ground station continuously corrects it to obtain the actual flight path. Image acquisition UAV B actually flies according to the actual flight path and transmits the collected images to the ground station in real time. At the same time, it is saved by the storage module for offline processing.

[0084] The drone system includes a fuselage stabilization device mounted on the drone B and connected to the onboard processing unit. The main function of the fuselage stabilization device is to ensure that the fuselage can meet the stability conditions when collecting pictures. Since most bridges are built in open areas that span streams and are higher than the horizontal plane, there are large wind loads around them and airflow disturbances caused by vehicles passing through them, which will cause significant interference to the flight state of the drone. When the onboard camera collects pictures, it will produce large vibrations, and the collected pictures will produce a lot of noise and missing edges. The edge information of the picture is particularly important for the subsequent feature extraction. Therefore, before collecting images, the drone is gravity-adjusted. By adjusting the angle between the axial gravity direction of the gyroscope and the direction of the collected image, the stability of the fuselage during detection is guaranteed and the error caused by vibration is reduced. At the same time, the initially collected image is judged. If the image meets the requirements, subsequent collection is carried out. Otherwise, adjustments are continued until the requirements are met.

[0085] The drone system includes a sensor group mounted on the drone B and connected to an onboard processing unit. The sensor group is arranged in the middle of the drone. In order to avoid collisions between the drone and the bridge body or piers during flight, a maximum contact distance with a radius of 1m is set in advance. The entire sensor group includes a ranging sensor and an acceleration sensor, both of which transmit data to the fuselage stabilization device via a wired connection. The ranging sensor is used to measure the distance between the drone and the bridge to prevent the camera lens from losing focus due to being too far or too close. The acceleration sensor is used to detect the acceleration of the drone, so that the drone changes the acceleration when it is about to reach the designated area or the maximum distance to avoid possible collisions.

[0086] The UAV system includes a GPS and UWB module mounted on the UAV B and connected to the airborne processing unit. The GPS module is composed of a GPS positioning system, an electronic map system and a navigation system. The UWB auxiliary positioning module plays a greater role when the GPS signal of the UAV is weak. Considering that the problem of weak GPS signal often occurs at the bottom of the bridge, 4 base stations are arranged around the bridge according to the actual environment, and the number of base stations can be considered according to the actual signal conditions. This embodiment adopts the UWB positioning method of arrival time difference. The tag card sends a UWB signal to the outside once, and all base stations within the wireless coverage range of the tag will receive the wireless signal. If there are two base stations with known coordinates that receive the signal, and the distance between the tag and the two base stations is different, then the time points when the two base stations receive the signal are different. Suppose the coordinates of the UAV at a certain moment are (x i ,y i , z i ), the coordinates of the four ground base stations are (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), (x4, y4, z4), the reception times are t1, t2, t3, t4, and the distances between the drone and the ground base stations are r1, r2, r3, r4. The coordinate calculation formula is as follows:

[0087]

[0088]

[0089] The drone uses the GPS module and UWB auxiliary positioning module to accurately locate the drone according to the pre-set synchronization time, and displays the positioning results on the electronic map. The drone's flight is corrected according to the deviation between the drone's dynamic track and the planned route.

[0090] The first data transmission radio is connected to the airborne processing unit via wireless, and the second data transmission radio is connected to the ground station. The control instructions and flight status data of the UAV are interactively transmitted between the UAV system and the ground end system through the first data transmission radio and the second data transmission radio.

[0091] The UAV system includes a fill light module mounted on the UAV B and connected to the onboard processing unit. The fill light module provides light for the PTZ camera in low-light environments, thereby improving image acquisition quality.

[0092] The UAV system includes a pan-tilt camera mounted on the UAV B and connected to the airborne processing unit. The module consists of two cameras located on the upper and lower parts of the UAV. Each camera has its own independent pan-tilt, which can capture images of the bridge from multiple angles and multiple positions.

[0093] The bridge data collected by the pan-tilt camera is sent to the second image transmission station on the ground via the first image transmission station.

[0094] The ground center's primary function is to plan UAV B's flight path and receive data from the primary and secondary image transmitters via the secondary data and image transmitters. This route planning considers multiple factors, including the bridge's piers, the length, width, and height of the bridge deck, and nearby obstacles, to create a reasonable flight path. The transmitted image information is also saved in real time for post-processing and the locations of bridge defects are recorded.

[0095] See also Figure 4 As shown in FIG, UAV B obtains the actual flight path after the simulated flight.

[0096] 5) Preprocessing the captured image, which includes normalization, Gaussian filtering, and gamma correction.

[0097] Due to the influence of the external environment and camera vibration, the collected images will contain noise. The collected images need to be preprocessed. The specific process is as follows:

[0098] 5.1) Normalize the image first: Image normalization refers to the process of performing a series of standard transformations on the image to transform it into a fixed standard form. The main functions of image normalization are: first, converting it into a standard mode to prevent the influence of affine transformation; second, ensuring that small values in the output data are not swallowed; third, image normalization can accelerate the convergence of subsequent neural networks. The formula is as follows

[0099]

[0100] where x i Represents the image pixel value, max(x) represents the maximum pixel value, Represents the minimum pixel value.

[0101] 5.2) Next, Gaussian filtering is performed. Gaussian filtering is a smoothing filter widely used in image denoising. The specific operation of Gaussian filtering is to scan each pixel in the image using a template (also called a convolution or mask) and replace the value of the central pixel in the template with the weighted average grayscale value of the pixels in the neighborhood determined by the template.

[0102] 5.3) For dark images, gamma correction is used. The gamma correction formula is as follows:

[0103]

[0104] Where γ is an adjustable coefficient, I is the current image, and when γ is 0.4, the correction effect is as follows: Figure 5 shown.

[0105] 6) Use the improved SSD detection algorithm to detect the preprocessed image and complete the defect detection.

[0106] 6.1) Dataset Creation: Data augmentation is performed on the open-source bridge crack dataset using the shear mixing data augmentation method. The principle of shear mixing is as follows: A and x B are two different training samples, y A and y B is the corresponding label value, and the shear mixing needs to generate new training samples and corresponding labels α and β.

[0107] α=M☉x A +(1-M)☉x B

[0108] β=μy A +(1-μ)y B

[0109] Where M∈(0,1) W×H It is a binary mask used to remove some areas and fill them. ☉ is pixel-by-pixel multiplication, and μ belongs to the Beta distribution: μ~Beta(a, a), and a=1 obeys the (0, 1) uniform distribution.

[0110] In order to sample the binary mask M, we first need to calculate the bounding box B of the cropped area = (r x , r y , r w , r h ) is used to sample the sample x A and x B To calibrate the indication of the cropping area, the formula is as follows:

[0111] r x ~Unif(0,W);

[0112] r y ~Unif(0,H);

[0113] After determining the cropping area B, set the cropping area B in M to 0 and the other areas to 1. The sampling of the mask is completed, and then the cropping area B in sample A is removed, and the cropping area B in sample B is cropped and then filled into sample A. The effect is as follows Figure 6 shown.

[0114] 6.2) Improved SSD Backbone Network: The traditional SSD algorithm uses six different feature maps to detect targets of different scales. Its low-level localization effect is good but the classification accuracy is low. Since the six different feature maps are independent of each other, the traditional SSD algorithm has a general detection effect on small targets.

[0115] See also Figure 7 As shown in the figure, in the neural network, the bottom feature map contains rich position information, and the high-level feature map contains rich semantic information. However, in the traditional SSD network, the feature maps are independent of each other and there is no connection between them. Therefore, the network will lose a lot of information when detecting the model. For this reason, based on the original network, the feature maps of the high and low layers are fused, and the fully connected layer 6 and the fully connected layer 7 are respectively subjected to CBL (convolution->batch normalization->non-linear activation) operations and then down-sampled. The feature map size is adjusted to be consistent with the size and number of channels of the fully connected layer 8. Then, the fully connected layer 6, the fully connected layer 7, and the convolution layer 8 are stacked and used as a feature map to replace the original convolution layer 8 feature map; the convolution layer 9 and the convolution layer 10 are respectively subjected to CBL operations and then up-sampled, and stacked with the convolution layer 11 feature map to replace the original convolution layer 11 feature map. Finally, the obtained 6 feature maps are sent to the detector. The improved network integrates the low-level position information and the high-level semantic information, thereby improving the accuracy of bridge crack detection.

[0116] 6.3) Loss Function: SSD defines the overall target loss function as the weighted sum of the positioning loss loc and the confidence loss conf, as follows:

[0117]

[0118] The first part of the loss function is the category loss, and the second part is the position loss. N is the number of matched prior boxes.

[0119] For class loss:

[0120]

[0121] in:

[0122] For confidence loss:

[0123]

[0124] in It is an indicator function, indicating whether the i-th prior box matches the true value, if it matches, it is 1, otherwise it is 0.

[0125] For the crack detection results, this embodiment uses Precision, Recall, F1 score The detection results are evaluated by using the value and mAP (since there is only one category, mAP = AP).

[0126]

[0127]

[0128]

[0129]

[0130] Among them, TP (True positive) means that the prediction is positive and the label is also positive; TN (Truenegative) means that the prediction is negative and the label is also negative; FP (False positive) means that the prediction is positive and the label is negative; FN (False Negative) means that the prediction is negative but the label is positive. Its evaluation indicators are shown in Table 1. The network with the feature fusion module has significantly higher accuracy, recall rate (Recall), and F1_score than the original network, which proves the effectiveness of the improved network from the data perspective. The sample results of bridge defect detection are shown in Figure 1. Figure 8 shown.

[0131] Table 1 Comparison between SSD-300 and improved SSD

[0132]

[0133] Example 2

[0134] Corresponding to the deep learning-based drone bridge crack detection method provided in Example 1, this embodiment provides a deep learning-based drone bridge crack detection system, which includes an acquisition module, a three-dimensional modeling module, a path planning module, a simulation flight module, a preprocessing module, and a detection module.

[0135] The acquisition module is used to use the surveying and mapping drone A to collect the measured bridge and its surrounding terrain environment, and transmit the data to the ground processing platform in real time.

[0136] Surveying drone A is equipped with an oblique aerial camera and a communication device. The oblique aerial camera consists of five cameras, each facing forward, backward, left, right, and vertically downward. It simultaneously captures images from both vertical and oblique angles, providing more accurate information. The film captured perpendicular to the ground is called a positive film. The four sets of images captured at an angle to the ground are called oblique films, which provide a better view of the model under test than perpendicular images. Phased control points are deployed after the flight plan is developed.

[0137] First: To ensure the accuracy of the 3D model, the heading overlap and lateral overlap must be at least 80%; second: the distance between the camera and the model cannot be too large; third: the model to be measured must occupy more than half of the area of the photo, and each part must appear in at least 3 images; the communication device records the images and position information collected by the aerial camera and transmits them to the processing platform in real time for 3D modeling.

[0138] The three-dimensional modeling module is used to perform three-dimensional modeling based on the data sent back by the surveying and mapping drone A using a ground processing platform to obtain a 3D coordinate real-scene model of the bridge to be measured and its surrounding environment.

[0139] See also Figure 1 As shown in the figure, the processing platform is a 3D modeling software developed based on the principle of oblique photography. When images with a certain degree of overlap are imported, the modeling software will automatically perform aerial triangulation based on the position information contained in the image to generate a dense point cloud to form a triangular mesh model. Finally, the 3D model will be generated based on the pixel information of the oblique image.

[0140] The path planning module is used to plan the flight path of the image acquisition drone B according to the 3D coordinate real scene model through the ground station, and transmit the instructions to the flight control module of the image acquisition drone B.

[0141] See also Figure 2 As shown, the image acquisition drone B is also called a drone system, which includes a flight control module, a fuselage stabilization module, a ranging sensor module, an acceleration sensor module, a GPS positioning module, a UWB auxiliary positioning tag, a first data transmission radio, a fill light module, a gimbal camera, a storage module, a first image transmission radio, and an airborne data processing unit.

[0142] The ground station is called the ground end system, which includes the ground station, the second data transmission radio, the second image transmission radio, and the UWB positioning base station.

[0143] The flight control module is used to receive the flight path; the fuselage stabilization module is used to keep the fuselage stable when collecting images; the ranging sensor module is used to measure the distance between the UAV and the bridge; the acceleration sensor module is used to detect the acceleration of the UAV; the GPS positioning module is used to locate the UAV; the UWB auxiliary positioning tag is used to assist in positioning when the GPS signal at the bottom of the bridge is weak; the first data transmission radio is used to transmit data between the UAV and the second data transmission radio of the ground station; the gimbal camera is used to collect images; the fill light module is used to supplement the light source for the gimbal camera in low-light environments; the storage module is used to store the collected data for offline processing; the first image transmission radio is used to transmit data between the UAV and the second image transmission radio of the ground station; the onboard data processing unit is used to control the posture of the gimbal camera and send collection signals.

[0144] The simulation flight module is used to simulate the flight of the image acquisition drone B according to the planned path. During the simulation process, when the drone enters the bottom of the bridge, the UWB-assisted positioning module obtains the drone's position, which is transmitted to the ground station and continuously corrected by the ground station to obtain the actual flight path. The image acquisition drone B actually flies according to the actual flight path and transmits the collected images to the ground station in real time. At the same time, it is saved through the storage module for offline processing.

[0145] The drone system includes a fuselage stabilization device mounted on the drone B and connected to the onboard processing unit. The main function of the fuselage stabilization device is to ensure that the fuselage can meet the stability conditions when collecting pictures. Since most bridges are built in open areas that span streams and are higher than the horizontal plane, there are large wind loads around them and airflow disturbances caused by vehicles passing through them, which will cause significant interference to the flight state of the drone. When the onboard camera collects pictures, it will produce large vibrations, and the collected pictures will produce a lot of noise and missing edges. The edge information of the picture is particularly important for the subsequent feature extraction. Therefore, before collecting images, the drone is gravity-adjusted. By adjusting the angle between the axial gravity direction of the gyroscope and the direction of the collected image, the stability of the fuselage during detection is guaranteed and the error caused by vibration is reduced. At the same time, the initially collected image is judged. If the image meets the requirements, subsequent collection is carried out. Otherwise, adjustments are continued until the requirements are met.

[0146] The drone system includes a sensor group mounted on the drone B and connected to an onboard processing unit. The sensor group is arranged in the middle of the drone. In order to avoid collisions between the drone and the bridge body or piers during flight, a maximum contact distance with a radius of 1m is set in advance. The entire sensor group includes a ranging sensor and an acceleration sensor, both of which transmit data to the fuselage stabilization device via a wired connection. The ranging sensor is used to measure the distance between the drone and the bridge to prevent the camera lens from losing focus due to being too far or too close. The acceleration sensor is used to detect the acceleration of the drone, so that the drone changes the acceleration when it is about to reach the designated area or the maximum distance to avoid possible collisions.

[0147] The UAV system includes a GPS and UWB module mounted on the UAV B and connected to the airborne processing unit. The GPS module is composed of a GPS positioning system, an electronic map system and a navigation system. The UWB auxiliary positioning module plays a greater role when the GPS signal of the UAV is weak. Considering that the problem of weak GPS signal often occurs at the bottom of the bridge, 4 base stations are arranged around the bridge according to the actual environment, and the number of base stations can be considered according to the actual signal conditions. This embodiment adopts the UWB positioning method of arrival time difference. The tag card sends a UWB signal to the outside once, and all base stations within the wireless coverage range of the tag will receive the wireless signal. If there are two base stations with known coordinates that receive the signal, and the distance between the tag and the two base stations is different, then the time points when the two base stations receive the signal are different. Suppose the coordinates of the UAV at a certain moment are (x i ,y i , z i ), the coordinates of the four ground base stations are (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), (x4, y4, z4), the reception times are t1, t2, t3, t4, and the distances between the drone and the ground base stations are r1, r2, r3, r4. The coordinate calculation formula is as follows:

[0148]

[0149]

[0150] The drone uses the GPS module and UWB auxiliary positioning module to accurately locate the drone according to the pre-set synchronization time, and displays the positioning results on the electronic map. The drone's flight is corrected according to the deviation between the drone's dynamic track and the planned route.

[0151] The first data transmission radio is connected to the airborne processing unit via wireless, and the second data transmission radio is connected to the ground station. The control instructions and flight status data of the UAV are interactively transmitted between the UAV system and the ground end system through the first data transmission radio and the second data transmission radio.

[0152] The UAV system includes a fill light module mounted on the UAV B and connected to the onboard processing unit. The fill light module provides light for the PTZ camera in low-light environments, thereby improving image acquisition quality.

[0153] The UAV system includes a pan-tilt camera mounted on the UAV B and connected to the airborne processing unit. The module consists of two cameras located on the upper and lower parts of the UAV. Each camera has its own independent pan-tilt, which can capture images of the bridge from multiple angles and multiple positions.

[0154] The bridge data collected by the pan-tilt camera is sent to the second image transmission station on the ground via the first image transmission station.

[0155] The ground center's primary function is to plan UAV B's flight path and receive data from the primary and secondary image transmitters via the secondary data and image transmitters. This route planning considers multiple factors, including the bridge's piers, the length, width, and height of the bridge deck, and nearby obstacles, to create a reasonable flight path. The transmitted image information is also saved in real time for post-processing and the locations of bridge defects are recorded.

[0156] See also Figure 4 As shown in FIG, UAV B obtains the actual flight path after the simulated flight.

[0157] The preprocessing module is used to preprocess the collected images, and the preprocessing method includes normalization, Gaussian filtering, and gamma correction.

[0158] Normalization: Image normalization refers to the process of performing a series of standard transformations on an image to transform it into a fixed standard form. The main functions of image normalization are: first, converting it into a standard mode to prevent the influence of affine transformation; second, ensuring that small values in the output data are not swallowed; third, image normalization can accelerate the convergence of subsequent neural networks. The formula is as follows

[0159]

[0160] where x i Represents the image pixel value, max(x) represents the maximum pixel value, Represents the minimum pixel value.

[0161] Gaussian filtering is a smoothing filter widely used in image denoising. It involves scanning each pixel in the image with a template (also called a convolution or mask) and replacing the value of the central pixel with the weighted average grayscale value of the pixels in the neighborhood defined by the template.

[0162] For dark images, gamma correction is used. The gamma correction formula is as follows:

[0163]

[0164] Where γ is an adjustable coefficient, I is the current image, and when γ is 0.4, the correction effect is as follows: Figure 5 shown.

[0165] The detection module is used to detect the preprocessed image through the improved SSD detection algorithm to complete the detection of defects; the detection module includes a data set production unit, an improved SSD backbone network unit, and a loss function unit.

[0166] The dataset production unit is used to expand the open source bridge crack dataset and use the shear mixing data enhancement method. The principle of shear mixing is as follows: A and x B are two different training samples, y A and y B is the corresponding label value, and the shear mixing needs to generate new training samples and corresponding labels α and β.

[0167] α=M☉x A +(1-M)☉x B

[0168] β=μy A +(1-μ)y B

[0169] Where M∈(0,1) W×H It is a binary mask used to remove some areas and fill them. ☉ is pixel-by-pixel multiplication, and μ belongs to the Beta distribution: μ~Beta(a, a), and a=1 obeys the (0, 1) uniform distribution.

[0170] In order to sample the binary mask M, we first need to calculate the bounding box B of the cropped area = (r x , r y , r w , r h ) is used to sample the sample x A and x B To calibrate the indication of the cropping area, the formula is as follows:

[0171] r x ~Unif(0,W);

[0172] r y ~Unif(0,H);

[0173] After determining the cropping area B, set the cropping area B in M to 0 and the other areas to 1. The sampling of the mask is completed, and then the cropping area B in sample A is removed, and the cropping area B in sample B is cropped and then filled into sample A. The effect is as follows Figure 6 shown.

[0174] The improved SSD backbone network is used to improve the traditional SSD algorithm. The traditional SSD algorithm uses six different feature maps to detect objects of different scales. Its low-level localization effect is good, but the classification accuracy is low. Because the six different feature maps are independent of each other, the traditional SSD algorithm has a poor detection effect on small objects.

[0175] See also Figure 7 As shown in the figure, in the neural network, the bottom feature map contains rich position information, and the high-level feature map contains rich semantic information. However, in the traditional SSD network, the feature maps are independent of each other and there is no connection between them. Therefore, the network will lose a lot of information when detecting the model. For this reason, based on the original network, the feature maps of the high and low layers are fused, and the fully connected layer 6 and the fully connected layer 7 are respectively subjected to CBL (convolution->batch normalization->non-linear activation) operations and then down-sampled. The feature map size is adjusted to be consistent with the size and number of channels of the fully connected layer 8. Then, the fully connected layer 6, the fully connected layer 7, and the convolution layer 8 are stacked and used as a feature map to replace the original convolution layer 8 feature map; the convolution layer 9 and the convolution layer 10 are respectively subjected to CBL operations and then up-sampled, and stacked with the convolution layer 11 feature map to replace the original convolution layer 11 feature map. Finally, the obtained 6 feature maps are sent to the detector. The improved network integrates the low-level position information and the high-level semantic information, thereby improving the accuracy of bridge crack detection.

[0176] The loss function unit is used to calculate the loss function of the improved SSD algorithm. SSD defines the overall target loss function as the weighted sum of the positioning loss loc and the confidence loss conf, as follows:

[0177]

[0178] The first part of the loss function is the category loss, and the second part is the position loss. N is the number of matched prior boxes.

[0179] For class loss:

[0180]

[0181] in:

[0182] For confidence loss:

[0183]

[0184] in It is an indicator function, indicating whether the i-th prior box matches the true value, if it matches, it is 1, otherwise it is 0.

[0185] For the crack detection results, this embodiment uses Precision, Recall, F1 score The detection results are evaluated by using the value and mAP (since there is only one category, mAP = AP).

[0186]

[0187]

[0188]

[0189]

[0190] Among them, TP (True positive) means that the prediction is positive and the label is also positive; TN (Truenegative) means that the prediction is negative and the label is also negative; FP (False positive) means that the prediction is positive and the label is negative; FN (False Negative) means that the prediction is negative but the label is positive. Its evaluation indicators are shown in Table 1. The network with the feature fusion module has significantly higher accuracy, recall rate (Recall), and F1_score than the original network, which proves the effectiveness of the improved network from the data perspective. The sample results of bridge defect detection are shown in Figure 1. Figure 8 shown.

[0191] Table 2 Comparison between SSD-300 and improved SSD

[0192]

Claims

1. A deep learning-based UAV bridge crack detection method, characterized in that: The following steps are involved: 1) Use surveying drone A to collect data on the bridge and its surrounding terrain, and transmit the data to the ground processing platform in real time; 2) The ground processing platform performs 3D modeling based on the data transmitted by surveying and mapping UAV A, obtaining a 3D coordinate real-scene model of the bridge to be measured and its surrounding environment; 3) The ground station plans the flight path of image acquisition drone B based on the 3D coordinate real-scene model and transmits the command to the flight control module of image acquisition drone B; 4) Simulate the flight of image acquisition drone B along the planned path. During the simulation, when the drone enters the bottom of the bridge, the UWB-assisted positioning module obtains the drone's position, transmits it to the ground station, and the ground station continuously corrects it to obtain the actual flight path. Image acquisition drone B actually flies along the actual flight path and transmits the collected images to the ground station in real time. The images are also stored in the storage module for offline processing. 5) Preprocessing the collected images, including normalization, Gaussian filtering, and gamma correction; 6) Use the improved SSD detection algorithm to detect the preprocessed image and complete the defect detection; the improved SSD detection algorithm includes the following steps: 6.1) Use the shear mixing data augmentation method to create a dataset. The shear mixing formula is as follows: ， ， in, and are two different training samples. and is the corresponding label value. Cutting and mixing need to generate new training samples and corresponding labels. 、 , is a binary mask, is pixel-by-pixel multiplication, Belongs to Beta distribution: , Then it obeys the (0, 1) uniform distribution; Bounding box for the clipping area Sampling, used to sample and To calibrate the indication of the cropping area, the formula is as follows: ， ， After determining the cropping area B, set the cropping area B in M to 0 and other areas to 1 to complete the mask sampling, remove the cropping area B in sample A, crop the cropping area B in sample B and then fill it into sample A to obtain the data set; 6.2) Fusing the feature maps of high and low layers, performing CBL operations on fully connected layers 6 and 7 respectively and then downsampling them to improve the SSD backbone network; 6.3) Calculate the loss function of the improved SSD. The formula is as follows: = ， The first part of the loss function is the category loss, and the second part is the position loss. is the number of matching prior boxes.

2. The deep learning-based UAV bridge crack detection method according to claim 1 is characterized in that: In step 3), the image acquisition drone B includes a flight control module, a range sensor module, an acceleration sensor module, a GPS positioning module, a UWB auxiliary positioning tag, a first data transmission radio, a fill light module, a gimbal camera, a storage module, a first image transmission radio, and an onboard data processing unit; The flight control module is used to receive the flight path; the ranging sensor module is used to measure the distance between the UAV and the bridge; the accelerometer module is used to detect the acceleration of the UAV; the GPS positioning module is used to locate the UAV; the UWB auxiliary positioning tag is used to assist in positioning when the GPS signal under the bridge is weak; the first data transmission radio is used to transmit data between the UAV and the second data transmission radio of the ground station; the gimbal camera is used to capture images; the fill light module is used to supplement the light source for the gimbal camera in low-light environments; and the storage module is used to store the collected data for offline processing. The first image transmission radio is used to transmit data between the UAV and the second image transmission radio of the ground station; the onboard data processing unit is used to control the posture of the gimbal camera and send acquisition signals.

3. The deep learning-based UAV bridge crack detection method according to claim 1 is characterized in that: In step 5), the collected image is preprocessed, wherein the normalization formula is as follows: , in Represents the image pixel value, represents the maximum pixel value, Represents the minimum pixel value.

4. The deep learning-based UAV bridge crack detection method according to claim 1 is characterized in that: In step 5), gamma correction is used for images collected in dark conditions. The formula is as follows: ， in is the adjustable coefficient, is the current image.

5. A deep learning-based UAV bridge crack detection system, characterized by: It includes acquisition module, 3D modeling module, path planning module, flight simulation module, pre-processing module and detection module; The acquisition module is used to use the surveying drone A to collect data about the bridge and its surrounding terrain, and transmit the data to the ground processing platform in real time; The 3D modeling module is used to perform 3D modeling based on the data transmitted by the surveying and mapping UAV A using a ground processing platform, thereby obtaining a 3D coordinate real-scene model of the bridge to be measured and its surrounding environment. The path planning module is used to plan the flight path of the image acquisition drone B based on the 3D coordinate real-scene model through the ground station and transmit the instructions to the flight control module of the image acquisition drone B; The flight simulation module is used to simulate the flight of image acquisition UAV B according to the planned path. During the simulation, when the UAV enters the bottom of the bridge, the UWB-assisted positioning module obtains the UAV's position, which is then transmitted to the ground station. The ground station then continuously corrects the position to obtain the actual flight path. Image acquisition UAV B actually flies according to the actual flight path and transmits the collected images to the ground station in real time. The images are then stored in the storage module for offline processing. The preprocessing module is used to preprocess the collected images, and the preprocessing methods include normalization, Gaussian filtering, and gamma correction; The detection module is used to detect defects in pre-processed images using the improved SSD detection algorithm. The improved SSD detection algorithm includes a dataset creation unit, an improved SSD backbone network unit, and a loss function unit. The data set production unit is used to produce the data set using the shear mixing data enhancement method, wherein the shear mixing formula is as follows: ， ， in, and are two different training samples. and is the corresponding label value. Cutting and mixing need to generate new training samples and corresponding labels. 、 , is a binary mask, is pixel-by-pixel multiplication, Belongs to Beta distribution: , Then it obeys the (0, 1) uniform distribution; Bounding box for the clipping area Sampling, used to sample and To calibrate the indication of the cropping area, the formula is as follows: ， ， After determining the cropping area B, set the cropping area B in M to 0 and other areas to 1 to complete the mask sampling, remove the cropping area B in sample A, crop the cropping area B in sample B and then fill it into sample A to obtain the data set; Improve the SSD backbone network unit to fuse the feature maps of high and low layers, perform CBL operation on the fully connected layer 6 and the fully connected layer 7 respectively, and then perform downsampling operation to improve the SSD backbone network; The loss function unit is used to calculate the loss function of the improved SSD. The formula is as follows: = ， The first part of the loss function is the category loss, and the second part is the position loss. is the number of matching prior boxes.

6. The deep learning-based UAV bridge crack detection system according to claim 5 is characterized in that: In the path planning module, the image acquisition drone B includes a flight control module, a ranging sensor module, an acceleration sensor module, a GPS positioning module, a UWB auxiliary positioning tag, a first data transmission radio, a fill light module, a gimbal camera, a storage module, a first image transmission radio, and an onboard data processing unit; The flight control module is used to receive the flight path; the ranging sensor module is used to measure the distance between the UAV and the bridge; the accelerometer module is used to detect the acceleration of the UAV; the GPS positioning module is used to locate the UAV; the UWB auxiliary positioning tag is used to assist in positioning when the GPS signal under the bridge is weak; the first data transmission radio is used to transmit data between the UAV and the second data transmission radio of the ground station; the gimbal camera is used to capture images; the fill light module is used to supplement the light source for the gimbal camera in low-light environments; and the storage module is used to store the collected data for offline processing. The first image transmission radio is used to transmit data between the UAV and the second image transmission radio of the ground station; the onboard data processing unit is used to control the posture of the gimbal camera and send acquisition signals.

7. The deep learning-based UAV bridge crack detection system according to claim 5 is characterized in that: In the preprocessing module, the collected image is preprocessed, where the normalization formula is as follows: , in Represents the image pixel value, represents the maximum pixel value, Represents the minimum pixel value.

8. The deep learning-based UAV bridge crack detection system according to claim 5 is characterized in that: In the pre-processing module, gamma correction is used for images collected in dark conditions. The formula is as follows: ， in is the adjustable coefficient, is the current image.

Images