Unmanned aerial vehicle bridge displacement detection method based on target detection

Abstract

The invention discloses an unmanned aerial vehicle bridge displacement detection method based on target detection, establishes a solution of bridge displacement detection based on unmanned aerial vehicle photography, takes a bridge picture by using an unmanned aerial vehicle, and extracts a grid center position and performs unmanned aerial vehicle image correction through a target detection method based on deep learning. And the bridge displacement is calculated with high precision. Specifically, a grid pattern is designed, an unmanned aerial vehicle is used for shooting grid photos under different conditions, the position of a grid region is positioned through a target tracking technology based on YOLOv8 deep learning, and a central point is accurately determined by using grid color features and image gradient information; the influence of vibration of the unmanned aerial vehicle is eliminated through a proposed vector alignment method and phase information fine compensation, and finally bridge displacement is calculated according to coordinates. The method has the characteristics of high precision, high robustness, no contact, low cost and the like, and can be suitable for bridge displacement detection under different complex backgrounds.

Description

Technical Field

[0001] The present invention relates to the field of bridge displacement detection, and in particular to a method for detecting bridge displacement using an unmanned aerial vehicle (UAV) based on target detection. Background Art

[0002] With the rapid development of bridge construction in my country, the construction volume of large-scale super engineering bridges is increasing day by day. In the whole life cycle of bridges, the maintenance period accounts for 90% of the life of bridges, and the long-term normal use of bridges depends not only on the construction quality, but also on the health monitoring and maintenance level in the later stage. As the proportion of bridge structures in transportation construction continues to increase, the aging of bridges will lead to the continuous destruction of bridge structures, and the load capacity will also be weakened. In order to ensure the safe operation of bridges and avoid catastrophic collapse, it is necessary to monitor the health and safety of bridges.

[0003] The traditional use of contact measurement systems to obtain the dynamic displacement of structures is an important method for bridge damage identification and bridge safety assessment. However, the installation process of contact sensors is relatively cumbersome, requiring the laying of a large number of pipelines on site to connect the sensors, and it also affects the normal operation of the monitored structure.

[0004] It can be seen that the traditional method has great limitations in practical applications, and it is necessary to select a fixed reference point on the monitoring target to install the sensor. In many cases, such as bridges across rivers and valleys and some building structures located in rugged terrain, it is difficult to find a stable fixed point to install the measurement sensor, and it is difficult to meet the requirements of efficient, convenient and low-cost measurement of bridge displacement. In contrast, drones significantly shorten the detection cycle by virtue of their ability to quickly reach various parts of the bridge. Not only that, drones also have the ability to operate in various severe weather conditions, such as strong winds, rain and snow, under which contact measurements are often difficult to perform. In addition, drones are not restricted by complex terrain and can obtain more accurate data in areas with complex geographical environments such as mountainous areas and plateaus. Therefore, in order to make bridge detection adapt to more environments and be more convenient and efficient, it is urgent to invent a bridge displacement detection method using drones. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method for detecting bridge displacement using a drone based on target detection. The method uses a drone to take photos of bridges, extracts grid center positions, and performs drone image correction through a target detection method based on deep learning, so as to calculate bridge displacement with high precision.

[0006] Technical solution: The present invention provides a method for detecting bridge displacement using a drone based on target detection. The schematic diagram of the simulated site is as follows: Figure 1 As shown, the specific steps include:

[0007] S1: Design a Figure 2 Grid style shown.

[0008] S2: Take raster photos under different conditions, annotate key information, obtain a labeled dataset, and divide it into a training set and a test set.

[0009] S3: Use the YOLOv8 target detection network model as the base network and use the labeled training set to train the YOLOv8 target detection model. Perform inference on the test set to identify the target and obtain the labeled detection box information, and finally obtain accurate detection results.

[0010] S4: Taking the center point determined by the detection box as the range, perform color feature extraction on the image, extract the red area, and solve the area centroid.

[0011] S5: Adjust the center point position of the image based on the gradient information to improve the center point position to achieve sub-pixel level accuracy.

[0012] S6: Use the vector alignment method to initially calibrate the images of the drone before and after to reduce the impact of the drone's own vibration and airflow.

[0013] S7: For the raster area of ​​the image, image preprocessing operations such as grayscale and Gaussian blur are used to improve image quality and remove noise interference, and then the sampling moiré method is used to further compensate the image to achieve accurate image correction and precise alignment.

[0014] S8: Calculate bridge displacements with high accuracy based on coordinates.

[0015] In S1, Figure 2 As shown in the figure, the grid style shape used is composed of five rows and five columns of rectangular blocks, with a red square in the center and the rest of the blocks being black. The marking material is made of PVC plastic sheet. The target pattern is sprayed on the surface of the plastic sheet by a precision inkjet printer, and then covered with an anti-oxidation transparent protective layer to slow down the fading of the target. The grid marking consists of reference marks and measurement marks: those set at the two end supports of the bridge are called reference marks, and those set at the part to be measured on the bridge are called measurement marks.

[0016] In S2, the simulation scenario diagram is as follows Figure 1 The influence of different external complex interference environments was taken into consideration during the shooting, thus enhancing the application scope and applicability of the model under various conditions.

[0017] In S3, YOLOv8 is used because it has excellent ability in processing complex scenes. YOLOv8 inherits the consistent high efficiency of the YOLO series, and greatly speeds up the speed of target detection through the core idea of ​​single detection. YOLOv8 can complete the position regression and category prediction of the target through a single network forward pass. This method significantly improves the detection speed and is particularly suitable for application scenarios with high real-time requirements. At the same time, YOLOv8 adopts the latest convolutional layers, cross-layer connections and enhanced feature extraction modules, and uses deformation convolution technology to optimize traditional convolutional neural networks, which can better capture the detailed features of the image. In addition, YOLOv8 adds multi-scale feature fusion, taking into account details and global information at the same time, further improving the accuracy of detection. The training method of the YOLOv8 network model is: divide the data set into a training set and a validation set, set the training coefficient and the number of pictures input for each training, and save the trained weights after the training is completed. Then, the initial positioning center coordinate C0 (x, y) is determined according to the center of the detection frame.

[0018] In S4, the color feature extraction extracts the red area from the image based on the initially estimated center point position C0 (x, y) as the range. The area includes the color features of the target object. The extracted area is converted to the HSV color space. The HSV space consists of the color type (Hue), purity (Saturation) and brightness (Value). The description of color in the HSV color space is more in line with human perception, and the area of ​​the target color is marked by color thresholding. For each area contour, the center point C1 (x′, y′) of the grid is further located by calculating the pixel centroid through the spatial moment:

[0019] M ji =∑ x.y (array(x,y)·x j ·y j )

[0020] Where i, j represent the order in the x and y directions respectively, i, j∈N. I(x,y) is the pixel value of the binary image at the point (x,y).

[0021] From the above formula, we can get:

[0022]

[0023] In S5, in order to improve the center point position to achieve sub-pixel level accuracy, the center point position is further adjusted by image gradient information. The gradient represents the direction and rate of change of the image grayscale value in space, so the best position between pixel grids can be obtained by using this information. Therefore, based on the Sobel operator to calculate the horizontal and vertical gradient values ​​of the image, the gradient information can provide the direction and intensity information of local changes in the image. Wherein the Sobel operator:

[0024]

[0025] Therefore, the gradient in the x direction is:

[0026]

[0027] The gradient in the y direction is:

[0028]

[0029] Assume that the gradient map in the x direction near the point C1(x′,y′) obtained above (considering the window size is 3×3) is as follows:

[0030]

[0031] The gradient can be expressed as:

[0032]

[0033] The lateral displacement in the x direction near the center point C1 can be expressed as:

[0034]

[0035] By using the above formula, the adjustment amount dx in the x direction can be calculated as:

[0036]

[0037] Similarly, the adjustment amount dy in the y direction can be calculated. At this point, the marker center coordinates C2(x″,y″) after gradient optimization are as follows:

[0038]

[0039] Through the above adjustments, the new position better meets the gradient balance condition, that is, it is closer to the actual marking center.

[0040] In S6, a model for eliminating drone vibration based on the vector alignment method is established. Due to the movement of the drone, the images captured before and after the deformation will change between the two moments. The affine transformation matrix can be used for image correction. Assume that there is a point (x, y) in the plane, which can be changed to (x', y') through the affine transformation matrix:

[0041]

[0042] But the scaling factor s, rotation angle θ, and translation t in the matrix x and t y Unknown. Assume that the reference point in the initial image obtained by the grid center extraction method is C 2a (x a ,y a ), C 2b (x b ,y b ). Similarly, the reference point in the deformed image is C 2a ′(x a ',y a '), C 2b ′(x b ',y b '). Through point C 2a and C 2b Determinable vector C 2a ′ and C 2b ′ can determine the vector Then, the unknown quantity S,θ,t is obtained by the following formula: x and t y :

[0043]

[0044] Using the complete affine matrix, the deformed image is corrected. The corrected reference point coordinates C can be obtained. 2a ′' and C 2b ′', and the corrected measurement point coordinates i" (x i ”,y i ”). At this point, the measurement errors of the four degrees of freedom caused by the translational displacement in the x, y, and z directions and the angular displacement γ are eliminated.

[0045] In S7, after the image before and after deformation is corrected by the vector alignment method, the error is controlled within half the range of the grid. Therefore, in order to achieve accurate image correction, for the image area of ​​the reference grid, image preprocessing operations such as grayscale and Gaussian blur are first used to improve image quality and remove noise interference. Then the sampling moiré method is used to further compensate the image to achieve accurate alignment. In the sampling moiré method, multiple phase-shifted moiré fringes are obtained through downsampling and intensity interpolation techniques. Downsampling combined with intensity interpolation can collect multiple phase-shifted fringe patterns, represented as f m (x, y; k), expressed as follows:

[0046]

[0047] Subsequently, the phase distribution of the moiré fringe is accurately determined by using the discrete Fourier transform (DFT) algorithm. This phase analysis step ensures high accuracy in the calculation process. The phase shift method of the discrete Fourier transform (DFT) algorithm is shown below:

[0048]

[0049] In the sampling moiré method, the zero phase point is unnecessary because the displacement δ x , δ y It can be determined based on the phase difference before and after deformation. Finally, the in-plane displacement is calculated based on the phase difference of the moiré fringes captured before and after deformation, achieving an accuracy of 1 / 100 pixel or 1 / 1000 grating spacing. At this point, the corrected measurement point coordinates i”'(x i ”',y i ”'):

[0050] x i ″′=x i ″+δ x ,y i ″′=y i ″+δ y

[0051] In S8, according to the compensated coordinates (x i ”',y i ”'), and the coordinates of the initial image (x i ,y i ), the relative displacement value is calculated with high precision by the following formula:

[0052] h x =x i -x i ″′,h y =y i -y i ″′

[0053] Since the reference marker coordinates reflect the movement of the drone, the reference frame displacement h′ can be calculated based on the reference coordinates x ,h′ y :

[0054]

[0055] Finally, the relative displacement value is subtracted from the reference system displacement value to obtain the accurate absolute displacement value:

[0056] H′ x =h x -h′ x ,H′ y =h y -h′ y

[0057] The above displacement values ​​are calculated based on pixel displacement values. To calculate the actual displacement value of the bridge, it is necessary to adjust the scale. The actual distance between the corner points of the measurement mark in the horizontal and vertical directions is known to be L. x , L y The horizontal distance of the corner pixels calculated on the image is recorded as Δ x , similarly, the average vertical distance is recorded as Δ y , then the horizontal scale conversion factor s x and the vertical scale conversion factor s y They are:

[0058]

[0059] The actual displacement of the bridge is:

[0060] H x =s x ·H′ x ,H y =s y ·H′ y

[0061] The beneficial effects of the present invention are:

[0062] The present invention designs a method for detecting bridge displacement using a drone based on target detection. The drone is used to take grid photos, the grid area is located by target tracking technology based on YOLOv8 deep learning, and the center point is accurately determined by using grid color features and image gradient information. Subsequently, the influence of drone vibration is eliminated by the proposed vector alignment method and phase information fine compensation, and finally the bridge displacement is calculated according to the coordinates. The solution for detecting bridge displacement using a drone based on target detection can realize high-precision detection of bridge displacement under complex terrain background conditions, and has the characteristics of high precision, high robustness, non-contact, low cost, etc., and requires fewer operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a schematic diagram of the simulation scene designed by the present invention.

[0064] Figure 2 This is a grid marking schematic diagram designed by the present invention

[0065] Figure 3 It is the technical roadmap designed by the present invention DETAILED DESCRIPTION

[0066] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0067] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below.

[0068] The present invention provides a method for detecting bridge displacement using a drone based on target detection. Figure 3 As shown, the following steps are included:

[0069] S1: Design a grid pattern. The grid pattern is composed of five rows and five columns of rectangular blocks, with a red square in the center and black squares in the rest. The marking material is made of PVC plastic sheet. The target pattern is sprayed on the surface of the plastic sheet by a precision inkjet printer, and then covered with an anti-oxidation transparent protective layer to slow down the fading of the target. The marking consists of reference marks and measurement marks: the ones set at the two ends of the bridge are called reference marks, and the ones set at the parts to be measured on the bridge are called measurement marks.

[0070] S2: Take raster-style photos under different conditions, annotate key information, obtain labeled data sets, and divide them into training sets and test sets. First, use professional drones to fly under different complex background conditions to capture different visual environments. These pictures under different conditions provide rich visual information for the data set, which is extremely important for training and testing target tracking systems. Then use the open source graphic image annotation tool Labelimg for data annotation. This step uses Labelimg to create bounding boxes and polygon annotations, and it is written in Python and uses Qt as its graphical interface, so it can effectively extract and generate labeled data in YOLO format.

[0071] S3: Use the YOLOv8 target detection network model as the base network, train the YOLOv8 target detection model with the labeled training set, set the training coefficient and the number of images input for each training, and save the trained weights after the training. Perform inference on the trained model on the test set, identify the target and obtain the labeled detection frame information, and finally obtain accurate detection results, and determine the preliminary positioning center coordinates C0(x,y) based on the center of the detection frame.

[0072] S4: Taking the center point determined by the detection box as the range, perform color feature extraction on the image, extract the red area, and solve the area centroid. Color feature extraction extracts the red area from the image based on the initial estimated center point position C0(x,y) as the range. The area includes the color features of the target object. Convert the extracted area to the HSV color space. The HSV space consists of the color type (Hue), purity (Saturation) and brightness (Value). The description of color in the HSV color space is more in line with human perception, and the area of ​​the target color is marked by color thresholding. For each area outline, the center point C1(x′,y′) of the grid is further located by calculating the pixel centroid through the spatial moment:

[0073] M ji =∑ x.y (array(x,y)·x j ·y j )

[0074] Where i, j represent the order in the x and y directions respectively, i, j∈N. I(x,y) is the pixel value of the binary image at the point (x,y).

[0075] From the above formula, we can get:

[0076]

[0077] S5: Use the gradient information of the image to adjust the center point position, improve the center point position to sub-pixel level accuracy, and further adjust the center point position through the image gradient information. The gradient represents the direction and rate of change of the image grayscale value in space, so this information can be used to obtain the optimal position between pixel grids. Therefore, based on the Sobel operator, the horizontal and vertical gradient values ​​of the image are calculated, and the gradient information can provide the direction and intensity information of local changes in the image. The Sobel operator:

[0078]

[0079] Therefore, the gradient in the x direction is:

[0080]

[0081] The gradient in the y direction is:

[0082]

[0083] Assume that the gradient map in the x direction near the point C1(x′,y′) obtained above (considering the window size is 3×3) is as follows:

[0084]

[0085] The gradient can be expressed as:

[0086]

[0087] The lateral displacement in the x direction near the center point C1 can be expressed as:

[0088]

[0089] By using the above formula, the adjustment amount dx in the x direction can be calculated as:

[0090]

[0091] Similarly, the adjustment amount dy in the y direction can be calculated. At this point, the marker center coordinates C2 (x″, y″) after gradient optimization are as follows:

[0092]

[0093] Through the above adjustments, the new position better meets the gradient balance condition, that is, it is closer to the actual marking center.

[0094] S6: Use the affine transformation matrix to establish a model for drone vibration elimination based on the vector alignment method. Assume that there is a point (x, y) in the plane, which can be changed to (x', y') through the affine transformation matrix:

[0095]

[0096] But the scaling factor s, rotation angle θ, and translation t in the matrix x and t y Unknown. Assume that the reference point in the initial image obtained by the grid center extraction method is C 2a (x a ,y a ), C 2b (x b ,y b ). Similarly, the reference point in the deformed image is C 2a ′(x a ',y a '), C 2b ′(x b ',yb '). Through point C 2a and C 2b Determinable vector C 2a ′ and C 2b ′ can determine the vector Then, the unknown quantity S,θ,t is obtained by the following formula: x and t y :

[0097]

[0098] Using the complete affine matrix, the deformed image is corrected. The corrected reference point coordinates C can be obtained. 2a ′' and C 2b ′', and the corrected measurement point coordinates i" (x i ”,y i ”). At this point, the measurement errors of the four degrees of freedom caused by the translational displacement in the x, y, and z directions and the angular displacement γ are eliminated.

[0099] S7: After the deformation image is corrected by the vector alignment method, the error is controlled within half of the grid. Therefore, in order to achieve accurate image correction, for the image area of ​​the reference grid, image preprocessing operations such as grayscale and Gaussian blur are first used to improve image quality and remove noise interference. Then the image is further compensated using the sampling moiré method to achieve precise alignment. In the sampling moiré method, the intensity of the one-dimensional or two-dimensional grating f(x,y) when recorded with a digital camera, characterized by the spacing p (mm) and the initial phase φ0, can be described by the following formula:

[0100]

[0101] where a(x,y) represents the amplitude of the grating intensity, b(x,y) corresponds to the background intensity, P represents the grating pitch recorded in pixel units on the image plane, and φ represents the phase value of the grating. Since the same grating pitch is used in the x and y directions in typical experiments, Px and Py are described here as P. Then, downsampling using integer sampling spacing of pixels combined with intensity interpolation can collect multiple phase-shifted fringe patterns, represented as f m (x, y; k), expressed as follows:

[0102] Phase distribution of moiré fringes φ m (x,y) can be found using the phase shift method with a discrete Fourier transform (DFT) algorithm as shown below:

[0103]

[0104] Likewise, the phase distribution of the moiré fringes after deformation can be evaluated using the same equation. Subsequently, the in-plane displacement is derived directly from the phase difference distribution of the moiré fringes before and after deformation. In the sampled moiré method, the zero phase point is unnecessary because the displacement can be determined from the phase difference before and after deformation:

[0105]

[0106] At this point, the corrected measurement point coordinates i”’(x i ”',y i ”'):

[0107] x i ″′=x i ″+δ x ,y i ″′=y i ″+δ y

[0108] S8: According to the compensated coordinates (x i ”',y i ”'), and the coordinates of the initial image (x i ,y i ), the relative displacement value is calculated with high precision by the following formula:

[0109] h x =x i -x i ″′,h y =y i -y i ″′

[0110] Since the reference marker coordinates reflect the movement of the drone, the reference frame displacement h′ can be calculated based on the reference coordinates x ,h′ y :

[0111]

[0112] Finally, the relative displacement value is subtracted from the reference system displacement value to obtain the accurate absolute displacement value:

[0113] H′ x =h x -h′ x ,H′ y =h y -h′ y

[0114] The above displacement values ​​are calculated based on pixel displacement values. To calculate the actual displacement value of the bridge, it is necessary to adjust the scale. The actual distance between the corner points of the measurement mark in the horizontal and vertical directions is known to be L. x , L y , the horizontal distance of the corner pixels calculated on the image is recorded as Δ x , similarly, the average vertical distance is recorded as Δ y , then the horizontal scale conversion factor s x and the vertical scale conversion factor s y They are:

[0115]

[0116] The actual displacement of the bridge is:

[0117] H x =x s ·H′ x ,H y =s y ·H′ y

[0118] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A method for detecting bridge displacement using an unmanned aerial vehicle based on target detection, characterized in that: The method comprises the following steps: S1: Design a grid style; S2: Take raster photos under different conditions, annotate key information, obtain labeled data sets, and divide them into training sets and test sets; S3: Use the YOLOv8 target detection network model as the base network and use the labeled training set to train the YOLOv8 target detection model. Perform inference on the test set to identify the target and obtain the labeled detection box information to obtain accurate detection results. S4: Taking the center point determined by the detection box as the range, extract the color features of the image, extract the red area, and solve the area centroid; S5: adjusting the center point position of the image based on the gradient information so that the center point position reaches sub-pixel level accuracy; S6: Use the vector alignment method to initially calibrate the images of the drone before and after, taking into account the influence of the drone's own vibration and airflow; S7: For the grid area of ​​the image, image preprocessing operations such as grayscale and Gaussian blur are used to improve image quality and remove noise interference, and then the sampling moiré method is used to further compensate the image to achieve accurate image correction and precise alignment; S8: Calculate bridge displacements with high accuracy based on coordinates.

2. The method for detecting bridge displacement using an unmanned aerial vehicle based on target detection according to claim 1 is characterized in that: This method uses a deep learning-based target detection method to extract the grid center position and perform drone image correction to calculate the bridge displacement with high precision.

3. The method for detecting bridge displacement using an unmanned aerial vehicle based on target detection according to claim 1 is characterized in that: In S1, the grid style shape used is composed of five rows and five columns of rectangular blocks, the center of which is a red block and the rest are black blocks.

4. The method for detecting bridge displacement using an unmanned aerial vehicle based on target detection according to claim 1 is characterized in that: In S2, a labeled data set is constructed, taking into account the impact of different external complex interference environments to adapt to the application scope of the model under various conditions.

5. The method for detecting bridge displacement using an unmanned aerial vehicle based on target detection according to claim 1, characterized in that: In S3, the training method of the YOLOv8 network model is: dividing the data set into a training set and a validation set, setting the training coefficient and the number of images input for each training, and saving the trained weights after the training is completed.

6. The method for detecting bridge displacement using an unmanned aerial vehicle based on target detection according to claim 1 is characterized in that: In S4, according to the preliminarily estimated target range, the red area is extracted from the image by the color feature of the target object. The extracted area is converted to the HSV color space, and the area of ​​the target color is marked by color thresholding. For each area outline, the center point of the grid is further located by calculating the pixel centroid through the spatial matrix.

7. The method for detecting bridge displacement using an unmanned aerial vehicle based on target detection according to claim 1, characterized in that: In S5, the image is processed for gradient information to improve the center point position to achieve sub-pixel level accuracy, and the center point position is adjusted by calculating the horizontal and vertical gradient values ​​of the image based on the Sobel operator.

8. The method for detecting bridge displacement using an unmanned aerial vehicle based on target detection according to claim 1, characterized in that: In S6, a UAV vibration elimination method based on a vector alignment method is established to solve the affine transformation matrix for image correction.

9. The method for detecting bridge displacement using an unmanned aerial vehicle based on target detection according to claim 1, characterized in that: In S7, after the images before and after deformation are corrected by the vector alignment method, for the image area of ​​the reference grid, image preprocessing operations such as grayscale and Gaussian blur are first used to improve image quality and remove noise interference, and then the sampling moiré method is used to further compensate the image to achieve precise alignment.

10. The method for detecting bridge displacement using an unmanned aerial vehicle based on target detection according to claim 1, characterized in that: In S8, the relative displacement of the target is obtained according to the compensated coordinates and the coordinates of the initial image. Then, the absolute displacement of the measuring point is obtained by subtracting the reference system displacement from the relative displacement of the measuring point, and the actual bridge displacement is obtained by proportional conversion.

Images