Multi-spectrum based unmanned aerial vehicle bridge appearance inspection method and system

Abstract

The application discloses a kind of unmanned aerial vehicle bridge appearance inspection method and system based on multispectral, comprising: ground control point reflective sticker and bridge deck linear reflective sticker are set in bridge;Laser point cloud data is collected using basic route, and bridge three-dimensional model is constructed;Based on bridge three-dimensional model, plan fine route, select orthographic image from the bridge image shot, detect two kinds of reflective sticker marks, output reflective sticker center pixel coordinates and corresponding reflective sticker orthographic image;Read the metadata of reflective sticker orthographic image, obtain the three-dimensional coordinates and pixel coordinates of ground control point;The three-dimensional coordinates of reflective sticker center are obtained by solving;Bridge photo is detected using bridge surface disease identification model, and disease photo is screened, and disease boundary box center coordinates are used as disease position;The same disease in multiple photos is extracted, and the three-dimensional coordinates of disease boundary box center point are calculated;Based on disease three-dimensional coordinate set and the change condition of bridge deck reflective sticker position, plan key area reinspection route.

Description

Technical Field

[0001] This application relates to the field of bridge inspection technology, specifically to a method and system for unmanned aerial vehicle (UAV) bridge appearance inspection based on multispectral imaging. Background Technology

[0002] As of the end of 2023, China had 1.0793 million highway bridges, including 10,239 extra-large bridges and 177,700 large bridges. However, after bridges are built and put into operation, they are subjected to the combined effects of cyclical vehicle loads, adverse weather conditions, temperature fluctuations, material degradation, and accidents such as ship collisions and fires. This can easily lead to deviations in the technical condition of bridge structures from design standards, reduced structural load-bearing capacity, and significant risks to the safe operation and longevity of the structures. Therefore, it is essential to strengthen the inspection of bridges during the operational phase to detect risks as early as possible.

[0003] Traditional bridge inspection typically employs a two-person collaborative work model, with one person responsible for visual or instrumental inspection and the other for paper-based record-keeping. However, manual inspection is difficult to access areas such as towers and bridge bases, and it is infrequent, costly, and disruptive to traffic. In recent years, with advancements in drone flight control technology and the widespread availability of supporting equipment, the industry has experimented with drone-assisted inspection. However, most organizations still rely on manual drone operation for fixed-point photography, failing to achieve fully autonomous inspection and resulting in no substantial breakthrough in inspection efficiency. A few organizations have attempted to use oblique photogrammetry to construct 3D models of bridges. Technical personnel generally use visible light to create a rough model of the bridge, but visible light struggles to identify slender structures such as cables, lampposts, and high-voltage lines. These objects have similar colors and few feature points, leading to poor 3D modeling results and posing significant safety hazards to the planning of precise drone inspection routes. Furthermore, in terms of defect localization, existing research largely relies on BIM models to map defect images to 3D spatial coordinates. While this facilitates information management and visualization, the high cost, long lead time, and technical complexity of BIM modeling make it difficult to meet the needs of frontline bridge inspection operations.

[0004] Meanwhile, current research and engineering applications of unmanned aerial vehicles (UAVs) for bridge inspection mainly focus on the detection of bridge defects, with little attention paid to the measurement of bridge deck alignment. Bridge deck alignment measurement is one of the main tasks of bridge inspection, primarily used to assess the bridge's alignment status and its changing trends, and to promptly identify potential structural problems. Traditional bridge deck alignment measurement mainly relies on manual operation of precision levels or total stations. Measurement points are typically placed at the edges of the curbs on both sides of the roadway, and longitudinal measurement points need to be manually positioned according to specific proportions such as 1 / 4 or 1 / 8 of the bridge span. This method has significant limitations: low operational efficiency, the need to ensure unobstructed view of the instrument, susceptibility to traffic control measures during on-site measurements, the risk of subjective errors in manual reading and point placement, and cumbersome data post-processing. Therefore, constructing a systematic solution that integrates automated bridge defect inspection, intelligent identification, precise positioning, and automated alignment measurement has become a crucial breakthrough for improving bridge inspection efficiency. Summary of the Invention

[0005] To address the problems existing in the prior art, this application proposes a multispectral-based UAV bridge appearance inspection method and system, which deeply integrates bridge inspection, photogrammetry, computer vision, deep learning and other technologies to form a practical and intelligent bridge inspection closed-loop solution.

[0006] Firstly, a multispectral-based UAV bridge appearance inspection method includes the following:

[0007] S1, At least three ground control point reflective stickers and several bridge deck linear reflective stickers are set on the bridge. The ground control point reflective stickers and the bridge deck linear reflective stickers are all circular and different colors.

[0008] S2: Plan the basic flight path of the UAV, collect laser point cloud data of the target bridge, and build a three-dimensional model of the bridge;

[0009] S3, planning refined flight paths for UAVs based on 3D bridge models;

[0010] S4, based on a refined flight path, uses a visible light camera to capture images of the bridge;

[0011] S5: Select orthophotos from bridge images within a preset distance of the UAV from the ground control point. Based on the color saturation of the reflective stickers, detect two types of circular reflective stickers from the orthophotos and output the pixel coordinates of the center of each reflective sticker and the corresponding orthophoto image.

[0012] S6, read the metadata of the orthophoto image of the reflector to obtain the three-dimensional coordinates and pixel coordinates of the ground control points; the metadata of the orthophoto image of the reflector includes the three-dimensional coordinates of the visible light camera center, attitude angle, and camera intrinsic parameters;

[0013] S7. Construct the projection equation of each bridge deck linear reflective orthophoto image, and combine the projection equations of all bridge deck linear reflective orthophoto images to form an overdetermined set of equations; introduce ground control point constraints, solve the initial values ​​through SVD decomposition, and optimize through bundle adjustment to obtain the three-dimensional coordinates of the center of the bridge deck linear reflective image, and determine the bridge deck linearity based on the three-dimensional coordinates.

[0014] S8, Construct a bridge surface defect identification model;

[0015] S9 uses a bridge appearance defect recognition model to detect defects in bridge photos taken by drones, filters defect photos, and uses the pixel coordinates of the center point of the detected defect bounding box as the defect location.

[0016] S10: Extract the same disease from multiple photos, calculate the three-dimensional coordinates of the pixel coordinates of the center point of the disease bounding box, and realize the spatial positioning of the disease.

[0017] S11, based on the three-dimensional coordinate set of the located defects and the positional changes of the bridge deck reflective stickers, plans a re-inspection and patrol route.

[0018] In one possible implementation of the first aspect, step S2 uses a five-way flight path as the basic route.

[0019] In one possible implementation of the first aspect, step S5 converts the orthophoto image into a grayscale image, preprocesses the image, and uses an edge detection algorithm to obtain the image edges; based on the color saturation of the reflective sticker, an image processing algorithm is used to detect the circular reflective sticker mark, and outputs its center coordinates and the corresponding orthophoto image of the reflective sticker.

[0020] In one possible implementation of the first aspect, step S6 involves pre-measuring the three-dimensional coordinates of the ground control point using a total station.

[0021] In one possible implementation of the first aspect, step S7 specifically includes the following:

[0022] (1) Following the rotation order of Z-axis-Y-X-axis, the overall rotation matrix of the i-th orthophoto image of the bridge surface linear reflective texture is:

[0023]

[0024] Among them, R z (β) represents the rotation matrix around the Z-axis with a heading angle β. R y (ω) represents the rotation matrix around the Y-axis, where the pitch angle is ω. Indicates the roll angle about the X-axis. The rotation matrix, β, ω, These represent the heading angle, pitch angle, and roll angle, respectively.

[0025] (2) Define the projection matrix:

[0026] P i =K[R i |-R i ·C i ]

[0027] C i This represents the three-dimensional coordinates of the visible light camera center; K is the intrinsic parameter matrix. f represents the camera focal length, (c z ,c y () represents the coordinates of the principal point;

[0028] (3) For each orthophoto image of the bridge deck linear reflector, the coordinates of the reflector center (u) are calculated. i ,v i Establish the projection equation:

[0029]

[0030] λ i X represents the scale factor; X, Y, and Z represent the three-dimensional coordinates of the center of the bridge deck linear reflective patch.

[0031] Eliminate the scale factor λ i Expanded into a system of linear equations:

[0032]

[0033] By combining the equations of all orthophotos of the bridge deck linear reflectors, an overdetermined system of equations is formed:

[0034]

[0035] A represents the global matrix. N represents the number of target points, and M represents the number of images per target point;

[0036] (4) Introduce ground control point constraints:

[0037] The known coordinates (X) of the control points k ,Y k Z k ) and corresponding pixel coordinates (u k ,v k Substituting into the overdetermined system of equations, we obtain the control point projection error equation:

[0038]

[0039] (5) For the overdetermined system of equations Perform SVD decomposition, and the right singular vector corresponding to the minimum singular value is the initial 3D point X0;

[0040] (6) The target point X is calculated using the algorithm that minimizes the reprojection error. j Perform bundle adjustment iterative optimization:

[0041]

[0042] The precise three-dimensional coordinates of the center of the j-th reflective sticker are obtained through iterative optimization.

[0043] In the formula, (u ij ,v ij ) represents the pixel coordinates of the bridge surface linear reflective texture of the j-th target point on the i-th image;

[0044] (7) The above steps are used to calculate the three-dimensional coordinates of all bridge deck reflective stickers to realize the detection of bridge deck alignment.

[0045] In one possible implementation of the first aspect, step S8 involves collecting photos of bridge surface defects, constructing a dataset, and training a deep learning model based on this dataset to obtain a bridge surface defect identification model.

[0046] Furthermore, the UAV bridge appearance inspection method also includes:

[0047] S12 automatically generates a drone bridge inspection report based on a preset report template.

[0048] Secondly, this application provides a system for performing the aforementioned UAV bridge appearance inspection method, comprising:

[0049] Unmanned aerial vehicle (UAV) terminals, used to carry lidar and industrial cameras;

[0050] LiDAR is used to collect laser point cloud data of the target bridge.

[0051] Industrial cameras are used to capture images of target bridges.

[0052] The basic flight path planning module is used to plan the basic flight path of the drone.

[0053] The oblique photography modeling module is electrically / communicationally connected to the lidar to construct a 3D model of the bridge from the lidar point cloud;

[0054] The refined flight path planning module plans refined flight paths for UAVs based on a 3D model of a bridge.

[0055] The bridge deck alignment detection module includes a reflective sticker recognition module and a reflective sticker coordinate calculation module. The reflective sticker recognition module filters orthophotos from target bridge images within a preset distance of the UAV from ground control points, converts the orthophotos to grayscale images, preprocesses the images, and uses an edge detection algorithm to obtain image edges. It detects circular markers and outputs the coordinates of the reflective sticker center and the corresponding orthophoto image. The reflective sticker coordinate calculation module reads the metadata of the orthophoto image and obtains the three-dimensional coordinates and pixel coordinates of the ground control points. The module further calculates the three-dimensional coordinates of the reflective sticker center, thereby achieving bridge deck alignment detection.

[0056] The apparent disease identification module includes an intelligent disease detection module and a disease location and tracking module. The intelligent disease detection module collects photos of apparent diseases on the bridge, constructs a dataset, and trains a deep learning model based on this dataset to obtain a bridge apparent disease identification model. The disease location and tracking module uses the bridge apparent disease identification model to detect diseases in bridge photos taken by UAVs at fixed points, filters out photos containing diseases, and records the center coordinates of the detected disease boundary boxes as the disease location. By extracting the same disease from multiple photos, the three-dimensional coordinates of the pixel coordinates of the center point of the disease boundary box are calculated to achieve spatial location of the disease.

[0057] Furthermore, the system also includes:

[0058] The inspection report generation module is used to generate inspection reports from the drone inspection results according to a preset template.

[0059] Compared with existing methods, this application proposes a closed-loop solution covering the entire process from 3D modeling, autonomous inspection, alignment detection, intelligent disease diagnosis, disease location, and disease tracking, significantly improving the automation level and data application value of bridge inspection operations; specifically, it has the following beneficial effects:

[0060] 1. This application constructs a three-dimensional model of the bridge foundation using lidar point cloud data. It overcomes the bottleneck of visible light modeling in recognizing objects with low texture features by using active laser scanning, and accurately reconstructs the spatial morphology of slender structures such as tree branches, high-voltage cables, cables, and street light poles, providing a reliable spatial benchmark for UAVs to plan millimeter-level safety inspection paths.

[0061] 2. This application identifies reflective stickers in photographs based on digital image correlation methods, and directly calculates their three-dimensional spatial coordinates through multi-image triangulation combined with bundle adjustment algorithm, providing data support for periodic inspection of bridge deck alignment; this method solves the problems of low measurement efficiency and strong subjectivity of traditional total station measurement, and at the same time makes up for the lack of alignment detection capability of UAVs in bridge defect detection;

[0062] 3. Based on the YOLO deep convolutional neural network, this application constructs an intelligent detection model for bridge surface defects, which enables rapid qualitative identification of defect types and quantitative segmentation of morphology and size to be completed simultaneously within a single model;

[0063] 4. This application is based on multiple views Figure 3 The angle measurement method automatically extracts the same target from multiple images of the detected disease area, and accurately calculates the three-dimensional world coordinates of its bounding box center through the bundle adjustment optimization algorithm, so as to achieve precise spatial positioning of the disease. This method overcomes the bottleneck of existing intelligent disease detection technology that can only identify but not automatically locate, and solves the problems of low efficiency and error in manually finding the spatial location of bridges from images.

[0064] 5. This application combines the three-dimensional coordinates of the located defects with the location information of the reflective stickers on the bridge deck to plan efficient inspection routes for key areas. Through regular re-inspection and intelligent analysis, it realizes the temporal evolution analysis of defect characteristics and bridge deck alignment, providing data support for precise bridge maintenance decisions. Attached Figure Description

[0065] Figure 1 A flowchart of a UAV-based bridge appearance inspection method provided for embodiments of this application;

[0066] Figure 2 A three-dimensional model of a bridge is provided for embodiments of this application; wherein Figure 2 (a) is a schematic diagram of the three-dimensional model of the bridge taken by oblique photography. Figure 2 (b) is a schematic diagram of the key inspection areas (web) of the bridge planned based on the 3D model of the bridge by oblique photography;

[0067] Figure 3 A circular bridge deck linear reflective sticker sign provided in this application embodiment;

[0068] Figure 4 A set of orthophoto images of circular bridge deck linear reflective stickers and schematic diagrams for reflective sticker coordinate recognition are provided for embodiments of this application; wherein Figure 4 (a) is the first image of the bridge surface linear reflective strip taken by a drone. Figure 4 (b) is Figure 4 (a) Schematic diagram of reflective map coordinate recognition; Figure 4 (c) is the second image of the bridge surface linear reflective material taken by a drone. Figure 4 (d) is Figure 4 (c) Schematic diagram of reflective patch coordinate recognition; Figure 4 (e) is the third image of the bridge surface linear reflective material taken by a drone. Figure 4 (f) is Figure 4 (e) Schematic diagram of reflective map coordinate recognition;

[0069] Figure 5 This application provides an architecture diagram of an unmanned aerial vehicle (UAV) bridge appearance inspection system;

[0070] Figure 6 This application provides an example of a bridge defect (damage) detection result diagram.

[0071] Figure 7 An image showing the detection results of bridge defects (exposed reinforcement) provided in an embodiment of this application;

[0072] Figure 8 This image shows the detection results of bridge defects (honeycomb surface) provided in an embodiment of this application. Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the application will now be described in further detail with reference to the accompanying drawings.

[0074] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0075] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0076] Firstly, such as Figure 1 As shown, this application provides a multispectral-based UAV bridge appearance inspection method, including the following:

[0077] S1. At least three ground control points (GCPs) reflective stickers are set at the bridgehead, mid-span of the side span, mid-span of the middle span, quarter point, and three-quarter point. The ground control points are non-collinear control points, which can meet the requirements of multi-view calculation and ensure the accuracy of plane and elevation. At the same time, N bridge deck line reflective stickers are set on the target detection bridge, which is the number of all target points whose three-dimensional coordinates need to be calculated. In this application, both reflective stickers are circular and different in color, and are distinguished by color saturation. The ground control points are used to determine the pixel coordinates of the ground control points, and the bridge deck line reflective stickers are used to determine the bridge deck line. This application uses circular reflective stickers, which can significantly reduce the algorithm complexity compared with reflective stickers of other complex shapes.

[0078] S2. Plan the basic flight path of the UAV, set the lateral overlap rate to 60%, and use the UAV platform equipped with LiDAR to collect the laser point cloud data of the target bridge and build a three-dimensional model of the bridge based on the LiDAR point cloud.

[0079] For example, a DJI Matrice 350RTK equipped with a DJI Zenmuse L2 LiDAR can be used, with flight control and mission management software adapted to the drone platform configured on the remote control end. A five-directional flight path can be planned in DJIPilot 2 software to collect LiDAR point cloud data of the target bridge; DJI Terra can then be used to construct a 3D model of the bridge based on the LiDAR point cloud, such as... Figure 2 (a) shows a 3D model of a bridge taken by oblique photography.

[0080] S3, based on the 3D model of the bridge, plans a refined flight path for the UAV, with a 70% overlap rate in the forward direction and a 50% overlap rate in the lateral direction. This refined flight path covers key inspection areas of the bridge, including but not limited to the bridge web, bottom slab, bridge deck, and piers. Figure 2 (b) shows the situation in Figure 2 (a) Key inspection areas (webs) of bridges planned based on the existing infrastructure.

[0081] S4, based on a refined flight path, uses an unmanned aerial vehicle platform equipped with a visible light camera to capture images of the target bridge;

[0082] S5. Select orthophoto images from the target bridge imagery that are within a preset distance of the UAV from the ground control point. Convert the orthophoto images to grayscale images, preprocess the images, and use an edge detection algorithm to obtain the image edges. Based on the color saturation of the reflective stickers, use an image processing algorithm to detect two types of circular reflective stickers and output their center coordinates and the corresponding orthophoto images. Note that each orthophoto image contains only one reflective sticker. Figure 3 The image shown is a linear reflective sticker sign for a bridge surface provided in an embodiment of this application;

[0083] For example, in the preprocessing process, Gaussian blur and median filtering can be selected to denoise the image; and algorithms such as Canny edge detection algorithm and Sobel operator can be selected to obtain the edges of the denoised image.

[0084] S6, through refined flight path design, sets specific headings and lateral overlap rates to ensure multiple imaging of the same target, thus the same reflective sticker will appear in multiple orthophoto images; it reads metadata from three or more orthophoto images containing the same reflective sticker and obtains the precise three-dimensional coordinates (X) of the ground control point. k ,Y k Z k ) and the pixel coordinates of the ground control points determined based on the ground control point reflector (uk ,v k k∈[1,N], where N represents the number of ground control points; the metadata includes, but is not limited to, the three-dimensional coordinates of the visible light camera center, attitude angles (heading angle β, roll angle) Pitch angle ω), camera intrinsic parameters (focal length f, principal point coordinates (c)). z ,c y ));

[0085] S7. For the same bridge deck linear reflective patch, construct the projection equation for each orthophoto image of the bridge deck linear reflective patch, and combine the projection equations of all orthophoto images of the bridge deck linear reflective patch to form an overdetermined system of equations; introduce ground control point constraints, solve the initial values ​​through SVD decomposition, and optimize through bundle adjustment to obtain the precise three-dimensional coordinates of the center of the bridge deck linear reflective patch; this step specifically includes the following:

[0086] (1) Following the rotation order of Z-axis-Y-X-axis, the overall rotation matrix of the i-th orthophoto image of the bridge surface linear reflective texture is:

[0087]

[0088] Among them, R z (β) represents the rotation matrix around the Z-axis with a heading angle β. R y (ω) represents the rotation matrix around the Y-axis, where the pitch angle is ω. Indicates the roll angle about the X-axis. The rotation matrix, β, ω, These represent the heading angle, pitch angle, and roll angle, respectively.

[0089] (2) Define the projection matrix:

[0090] P i =K[R i |-R i ·C i ]

[0091] C i This represents the three-dimensional coordinates of the visible light camera center. This represents the longitude, latitude, and altitude of the i-th orthophoto image of the bridge deck linear reflective texture; K is the intrinsic parameter matrix. f represents the camera focal length, (c z ,c y ) represents the principal point coordinates; i∈[1,M], where M represents the number of images taken by the drone at different positions of the bridge surface linear reflective stickers;

[0092] (3) For the coordinates (u) of the center of the reflector in the i-th orthophoto image of the bridge surface linear reflector, i,v i Establish the projection equation:

[0093]

[0094] λ i X represents the scale factor; X, Y, and Z represent the three-dimensional coordinates of the center of the bridge deck linear reflective patch, namely longitude, latitude, and altitude.

[0095] Eliminate the scale factor λ i Expanded into a system of linear equations:

[0096]

[0097] By combining the linear equations of all orthophotos of the bridge deck linear reflectors, an overdetermined system of equations is constructed:

[0098]

[0099] A represents the global matrix. N represents the number of target points, and M represents the number of images per target point;

[0100] (4) Introduce ground control point constraints:

[0101] The known coordinates (X) of the control points k ,Y k Z k ) and corresponding pixel coordinates (u k ,v k Substituting into the overdetermined system of equations, we obtain the control point projection error equation:

[0102]

[0103] By incorporating the projection error of the control points into the optimization objective function using the aforementioned ground control point pixel coordinates, the constraint of the equations is enhanced, and the solution accuracy is improved.

[0104] P i (1,:) represents the projection matrix P. i The first row vector; P i (2,:) represents the projection matrix P. i The second row vector, P i (3,:) represents the projection matrix P. i The third row vector;

[0105] (5) For the overdetermined system of equations Performing SVD decomposition, the right singular vector corresponding to the minimum singular value is the initial 3D point X0. The initial 3D point X0 is the initial guess value of the bundle adjustment iterative optimization, which is then gradually corrected through nonlinear optimization (such as the Levenberg-Marquardt algorithm) until it converges to the exact solution.

[0106] (6) The target point X is calculated using the algorithm that minimizes the reprojection error. j Perform bundle adjustment iterative optimization:

[0107]

[0108] The precise three-dimensional coordinates of the center of the j-th reflective sticker are obtained through iterative optimization.

[0109] In the formula, N represents the number of target points whose three-dimensional coordinates need to be calculated, that is, the total number of bridge deck linear reflective stickers participating in the optimization; M represents the number of images of the bridge deck linear reflective stickers taken, that is, the number of photos taken by the drone at each bridge deck linear reflective sticker location; N and M are used to accumulate the reprojection error of all images and all target points to ensure global optimization; the accuracy of X0 depends on the number of reflective sticker images M, and must satisfy the observability of N target points;

[0110] (u ij ,v ij ) represents the pixel coordinates of the bridge surface linear reflective texture of the j-th target point on the i-th image; target point X j The theoretical pixel coordinates on the orthophoto image of the i-th bridge surface linear reflective patch are:

[0111]

[0112] (7) The above steps are used to calculate the three-dimensional coordinates of all bridge deck reflective stickers to realize the detection of bridge deck alignment.

[0113] Figure 4 This is an example of a method for calculating the coordinates of three photos of a circular bridge deck reflective sticker that an applicant affixed to the Liandong U Valley (International Enterprise Port, Shapingba District, Chongqing). The method involves using DJI M4E to take orthophotos of different locations on the bridge and then performing coordinate calculations on three photos of one of the bridge deck reflective stickers. Figure 4 The orthophoto images of the bridge deck's linear reflective surface and their center coordinates were identified.

[0114] Table 1. Test Photo Metadata

[0115]

[0116] The calculated three-dimensional geodetic coordinates of the center of the bridge deck reflective patch in the WGS84 coordinate system are as follows:

[0117] Latitude = 29.571815216°, Longitude = 106.289196487°, Altitude = 282.042m.

[0118] S8. Collect a certain number (preferably more than 2,000) of bridge surface defects photos, construct a dataset, and train a deep learning model based on this dataset to obtain a bridge surface defect identification model.

[0119] For example, the deep learning model employs the YOLO object detection model, such as YOLOv12.

[0120] S9 uses a bridge appearance defect identification model to detect defects in bridge photos taken by UAVs at fixed points, filters out photos containing defects, and records the center coordinates (u,v) of the detected defect boundary box as the defect location.

[0121] S10, based on multi-view Figure 3 The angle measurement method extracts the same disease target from multiple photos of the detected disease area, and calculates the three-dimensional coordinates (u,v) of the disease boundary box center coordinates through bundle adjustment optimization algorithm to realize the spatial positioning of the disease.

[0122] S11, based on the three-dimensional coordinate set of the located defects and the position of the reflective stickers on the bridge deck where significant displacement has occurred, plans inspection routes for key areas. Through regular inspections and intelligent analysis, it realizes the time-series evolution tracking of defect characteristics and bridge deck alignment, providing data support for bridge maintenance decisions.

[0123] Furthermore, in a preferred embodiment of the first aspect, step S2 uses a five-directional flight path as the basic flight path.

[0124] Furthermore, in a preferred embodiment of the first aspect, step S5 detects the reflective sticker mark based on the Hough circle transform algorithm and outputs its center coordinates.

[0125] Furthermore, in a preferred embodiment of the first aspect, step S6 can be performed by pre-measuring the three-dimensional coordinates (X, Y, X) of the ground control point using a total station. k ,Y k Z k ).

[0126] Furthermore, the multispectral-based UAV bridge appearance inspection method also includes:

[0127] S12 automatically generates a drone bridge inspection report based on a preset report template. The report content includes, but is not limited to, the location, size, and type of defects.

[0128] Thus, this application has constructed a complete intelligent closed-loop system for the entire process of bridge surface defects detection, which includes "multispectral data acquisition, intelligent identification, precise positioning, time-series tracking, and automatic reporting." This system can significantly improve engineering inspection efficiency and contribute to the realization of intelligent bridge maintenance.

[0129] Secondly, such as Figure 5 As shown, this application provides a system for performing the UAV bridge appearance inspection method, comprising:

[0130] Unmanned aerial vehicle (UAV) terminals, used to carry lidar,

[0131] LiDAR is used to collect laser point cloud data of the target bridge.

[0132] Industrial cameras are used to capture images of target bridges.

[0133] The basic flight path planning module is used to plan the basic flight path of the drone.

[0134] The oblique photography modeling module is electrically / communicationally connected to the lidar to construct a 3D model of the bridge from the lidar point cloud;

[0135] The refined route planning module, based on the 3D model of the bridge, plans refined routes for autonomous inspection of the bridge structure, safety zones under the bridge, etc.

[0136] The bridge deck alignment detection module includes a reflective sticker recognition module and a reflective sticker coordinate calculation module. The reflective sticker recognition module filters orthophotos within a preset distance from ground control points from the target bridge image, converts the orthophotos to grayscale, performs noise reduction, and uses an edge detection algorithm to obtain image edges. It detects circular markers and outputs the center coordinates and the corresponding reflective sticker orthophoto image. The reflective sticker coordinate calculation module reads the metadata of the reflective sticker orthophoto image and obtains the three-dimensional coordinates and two-dimensional pixel coordinates of the ground control points. The module further constructs a projection matrix, combines the equations of all bridge deck alignment reflective sticker orthophoto images to form an overdetermined system of equations, introduces ground control point constraints, solves the initial values ​​through SVD decomposition, and optimizes through bundle adjustment to obtain the three-dimensional coordinates of the reflective sticker center, thereby achieving bridge deck alignment detection.

[0137] The apparent disease identification module includes an intelligent disease detection module and a disease location and tracking module. It qualitatively and quantitatively identifies apparent diseases and enables regular tracking of diseases. The intelligent disease detection module collects photos of bridge apparent diseases, constructs a dataset, and trains a deep learning model based on this dataset to obtain a bridge apparent disease identification model. The disease location and tracking module uses the bridge apparent disease identification model to detect diseases in bridge photos taken by UAVs at fixed points, filters out photos containing diseases, and records the center coordinates of the detected disease boundary boxes as the disease location. By extracting the same disease from multiple photos, the three-dimensional coordinates of the pixel coordinates of the center point of the disease boundary box are calculated to achieve spatial location of the disease.

[0138] Furthermore, the system also includes:

[0139] The inspection report generation module is used to generate inspection reports from the drone inspection results according to a preset template.

[0140] The feasibility and effectiveness of this application are illustrated below with an application example. This embodiment uses YOLOv12 as the basic model for bridge defect detection.

[0141] (1) Dataset construction:

[0142] A dataset was constructed by collecting photos of bridge surface defects. All images in the bridge defect dataset were divided into training and validation sets according to an 80% and 20% ratio for training.

[0143] Based on the size, shape, and other characteristics of the target disease, it is divided into Model 1 and Model 2 for detection.

[0144] Model 1: Detects exposed rebar and damage in bridge defects; a total of 396 images were collected; training took 421 rounds and 0.699 hours.

[0145] Model 2: Detects honeycomb and pitted surfaces in bridge defects; a total of 148 images were collected; training took 272 rounds and 0.217 hours.

[0146] Use the calibration tool labelme to calibrate the model.

[0147] (2) Evaluation indicators:

[0148] Training time, single-image detection time, average precision (AP), and detection results were selected as evaluation metrics for the algorithm's performance. The AP value fully reflects the model's ability to detect diseases; a higher AP value indicates higher detection accuracy. mAP50% represents the average detection accuracy for all target categories when the IOU (Intersection over Union) threshold is 0.5. mAP50-95% represents the average detection accuracy for all target categories within the IOU threshold range of 0.5 to 0.95.

[0149] Table 1. Detection results of exposed reinforcement / damage in Model 1

[0150] Algorithm Name Training time Model memory usage Single image detection time mAP50% mAP 50-95% YOLOv12 0.699 hours 23M 7.3ms 62.5％ 41.4％

[0151] Table 2. Detection results of honeycomb surface defects in Model 2

[0152] Algorithm Name Training time Model memory usage Single image detection time mAP50% mAP 50-95% YOLOv12 0.217 hours 23M 6.3ms 74.5％ 50.5％

[0153] like Figure 6 , Figure 7 The detection results of Model 1 are for damage and exposed reinforcement. Figure 8 This is the detection result image for Model 2 targeting honeycomb-like pitted surfaces.

[0154] Compared to existing technologies, this embodiment does not require similarity calculation for every location in the feature map. By dividing the feature map into horizontal and vertical regions (4 by default in the experiment), the receptive field is reduced to 1 / 4 of its original size. Only the attention score within each region is calculated, significantly improving the computation speed with almost negligible impact on performance. Secondly, this embodiment introduces residual connections and feature aggregation optimization through the YOLO algorithm. Block-level residual design and bottleneck structures enhance gradient flow stability, effectively solving the problem of gradient hindrance and difficulty in convergence during large model training.

[0155] The above embodiments are only for illustrating the technical concept and features of this application, and are intended to enable those skilled in the art to understand the content of this application and implement it accordingly. They should not be used to limit the scope of protection of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for inspecting the appearance of bridges using unmanned aerial vehicles (UAVs) based on multispectral imaging, characterized in that, Includes the following: S1, At least three ground control point reflective stickers and several bridge deck linear reflective stickers are set on the bridge. The ground control point reflective stickers and the bridge deck linear reflective stickers are all circular and different colors. S2: Plan the basic flight path of the UAV, collect laser point cloud data of the target bridge, and build a three-dimensional model of the bridge; S3, planning refined flight paths for UAVs based on 3D bridge models; S4, based on a refined flight path, uses a visible light camera to capture images of the bridge; S5: Select orthophotos from bridge images within a preset distance of the UAV from the ground control point. Based on the color saturation of the reflective stickers, detect two types of circular reflective stickers from the orthophotos and output the pixel coordinates of the center of each reflective sticker and the corresponding orthophoto image. S6, read the metadata of the orthophoto image of the reflector and obtain the three-dimensional coordinates and pixel coordinates of the ground control points; S7. Construct the projection equation of each bridge deck linear reflective orthophoto image, and combine the projection equations of all bridge deck linear reflective orthophoto images to form an overdetermined set of equations; introduce ground control point constraints, solve the initial values ​​through SVD decomposition, and optimize through bundle adjustment to obtain the three-dimensional coordinates of the center of the bridge deck linear reflective image, and determine the bridge deck linearity based on the three-dimensional coordinates. S8, Construct a bridge surface defect identification model; S9 uses a bridge appearance defect recognition model to detect defects in bridge photos taken by drones, filters defect photos, and uses the pixel coordinates of the center point of the detected defect bounding box as the defect location. S10: Extract the same disease from multiple photos, calculate the three-dimensional coordinates of the pixel coordinates of the center point of the disease bounding box, and realize the spatial positioning of the disease. S11, based on the three-dimensional coordinate set of the located defects and the positional changes of the bridge deck reflective stickers, plans a re-inspection and patrol route.

2. The UAV-based bridge appearance inspection method according to claim 1, characterized in that, In step S2, the basic flight path adopts a five-directional flight path.

3. The UAV-based bridge appearance inspection method according to claim 1, characterized in that, In step S5, the orthophoto image is converted into a grayscale image, the image is preprocessed, and the image edge is obtained using an edge detection algorithm. Based on the color saturation of the reflective sticker, an image processing algorithm is used to detect the circular reflective sticker mark, and the coordinates of its center and the corresponding orthophoto image of the reflective sticker are output.

4. The UAV-based bridge appearance inspection method according to claim 1, characterized in that, In step S6, the metadata of the orthophoto image of the reflective patch includes the three-dimensional coordinates of the visible light camera center, the attitude angle, and the camera intrinsic parameters.

5. The UAV-based bridge appearance inspection method according to claim 1, characterized in that, In step S6, the three-dimensional coordinates of the ground control point are determined in advance by a total station.

6. The UAV-based bridge appearance inspection method according to claim 4, characterized in that, Step S7 specifically includes the following: (1) Following the rotation order of Z-axis-Y-X-axis, the overall rotation matrix of the i-th orthophoto image of the bridge surface linear reflective patch is: in, R z(β) represents the rotation matrix around the Z-axis with a heading angle β. R y (ω) represents the rotation matrix around the Y-axis to rotate the pitch angle ω. Indicates the roll angle about the X-axis. The rotation matrix, β, ω, These represent the heading angle, pitch angle, and roll angle, respectively. (2) Define the projection matrix: P i =K[R i |-R i ·C i ] C i This represents the three-dimensional coordinates of the visible light camera center; K is the intrinsic parameter matrix. f represents the camera focal length, (c z ,c y () represents the coordinates of the principal point; (3) For each orthophoto image of the bridge deck linear reflector, the coordinates of the reflector center (u) are calculated. i ,v i Establish the projection equation: λ i X represents the scale factor; X, Y, and Z represent the three-dimensional coordinates of the center of the bridge deck linear reflective patch. Eliminate the scaling factor λ i Expanded into a system of linear equations: By combining the equations of all orthophotos of the bridge deck linear reflectors, an overdetermined system of equations is formed: A represents the global matrix. N represents the number of target points, and M represents the number of images per target point; (4) Introduce ground control point constraints: The known coordinates (X) of the control points k ,Y k Z k ) and corresponding pixel coordinates (u k ,v k Substituting into the overdetermined system of equations, we obtain the control point projection error equation: (5) For the overdetermined system of equations Perform SVD decomposition, and the right singular vector corresponding to the minimum singular value is the initial 3D point X0; (6) The target point X is calculated using the algorithm that minimizes the reprojection error. j Perform bundle adjustment iterative optimization: The precise three-dimensional coordinates of the center of the j-th reflective sticker are obtained through iterative optimization. In the formula, (u ij ,v ij ) represents the pixel coordinates of the bridge surface linear reflective texture of the j-th target point on the i-th image; (7) The above steps are used to calculate the three-dimensional coordinates of all bridge deck reflective stickers to realize the detection of bridge deck alignment.

7. The UAV-based bridge appearance inspection method according to claim 1, characterized in that, In step S8, photos of bridge surface defects are collected, a dataset is constructed, and a deep learning model is trained based on this dataset to obtain a bridge surface defect identification model.

8. The UAV-based bridge appearance inspection method according to claim 1, characterized in that, Also includes: S12 automatically generates a drone bridge inspection report based on a preset report template.

9. A system for performing the method as described in any one of claims 1-8, characterized in that, include: Unmanned aerial vehicle (UAV) terminals, used to carry lidar and industrial cameras; LiDAR is used to collect laser point cloud data of the target bridge. Industrial cameras are used to capture images of target bridges; The basic flight path planning module is used to plan the basic flight paths of drones; The oblique photography modeling module is electrically / communicationally connected to the lidar to construct a 3D model of the bridge from the lidar point cloud; The refined flight path planning module plans refined flight paths for UAVs based on a 3D model of a bridge. The bridge deck alignment detection module includes a reflective sticker recognition module and a reflective sticker coordinate calculation module. The reflective sticker recognition module filters orthophotos from target bridge images within a preset distance of the UAV from ground control points, converts the orthophotos to grayscale images, preprocesses the images, and uses an edge detection algorithm to obtain image edges. It detects circular markers and outputs the coordinates of the reflective sticker center and the corresponding orthophoto image. The reflective sticker coordinate calculation module reads the metadata of the orthophoto image and obtains the three-dimensional coordinates and pixel coordinates of the ground control points. The module further calculates the three-dimensional coordinates of the reflective sticker center, thereby achieving bridge deck alignment detection. The apparent disease identification module includes an intelligent disease detection module and a disease location and tracking module. The intelligent disease detection module collects photos of apparent diseases on the bridge, constructs a dataset, and trains a deep learning model based on this dataset to obtain a bridge apparent disease identification model. The disease location and tracking module uses the bridge apparent disease identification model to detect diseases in bridge photos taken by UAVs at fixed points, filters out photos containing diseases, and records the center coordinates of the detected disease boundary boxes as the disease location. By extracting the same disease from multiple photos, the three-dimensional coordinates of the pixel coordinates of the center point of the disease boundary box are calculated to achieve spatial location of the disease.

10. The system according to claim 8, characterized in that, Also includes: The inspection report generation module is used to generate inspection reports from the drone inspection results according to a preset template.

Images