Unmanned aerial vehicle bridge appearance inspection method and system based on multispectrum

Abstract

The invention discloses a multi-spectrum-based unmanned aerial vehicle bridge appearance inspection method and system. The method comprises the steps that ground control point reflective stickers and bridge floor linear reflective stickers are arranged on a bridge; acquiring laser point cloud data by adopting a basic route, and constructing a bridge three-dimensional model; a refined route is planned based on the bridge three-dimensional model, an orthographic image is screened from shot bridge images, two reflective sticker marks are detected, and reflective sticker circle center pixel coordinates and corresponding reflective sticker orthographic images are output; reading metadata of an orthographic image of the reflective sticker, and obtaining three-dimensional coordinates and pixel coordinates of ground control points; calculating to obtain three-dimensional coordinates of the circle center of the reflective sticker; carrying out disease detection on the bridge photos by adopting a bridge apparent disease recognition model, screening disease photos, and taking central coordinates of a disease bounding box as disease positions; extracting the same disease in the plurality of photos, and calculating three-dimensional coordinates of a central point of a disease bounding box; and based on the disease three-dimensional coordinate set and the change condition of the position of the bridge floor reflective sticker, planning a reinspection route of the key area.

Description

Technical Field

[0001] The present application relates to the field of bridge inspection technology, and specifically to a multi-spectral drone-based bridge appearance inspection method and system. Background Art

[0002] By the end of 2023, China had 1,079,300 highway bridges, including 10,239 super-large bridges and 177,700 large bridges. However, after completion and commissioning, bridges are subject to the combined effects of cyclical vehicle loads, adverse weather conditions, temperature fluctuations, material degradation, and accidents such as ship collisions and fires. This can easily cause the technical condition of bridge structures to deviate from design standards, reducing their bearing capacity and posing significant risks to their safe operation and longevity. Therefore, enhanced inspections of bridges during operational stages are crucial to identify risks early.

[0003] Traditional bridge inspections typically rely on a two-person collaborative model: one person conducts visual or instrumental inspections while the other maintains paper records. However, manual inspections are difficult to reach, often resulting in low frequency, high costs, and traffic disruptions. In recent years, with the advancement of drone flight control technology and the widespread availability of supporting equipment, the industry has experimented with drone-assisted inspections. However, most organizations are still limited to manually piloting drones for fixed-point photography, and have yet to achieve fully autonomous inspections, resulting in no substantial improvement in inspection efficiency. A few organizations have attempted to use oblique photography to construct three-dimensional bridge models. Technicians typically use visible light to create a rough model of the bridge. However, visible light has difficulty recognizing slender structures such as cables, light poles, and high-voltage power lines. These objects have similar colors and few feature points, resulting in poor 3D modeling and significant safety risks for drone inspection route planning. Furthermore, existing research on defect location relies heavily on BIM models to map defect images to 3D spatial coordinates. While this facilitates information management and visualization, the high cost, timeliness, and technical complexity of BIM modeling make it difficult to meet the demands of frontline bridge inspections.

[0004] Existing research and engineering applications of drone-based bridge inspections primarily focus on detecting bridge defects, with little involvement in measuring bridge deck alignment. Bridge deck alignment measurement is a key component of bridge inspection, primarily used to assess the bridge's alignment status and its changing trends, enabling the timely identification of potential structural issues. Traditional bridge deck alignment measurement relies primarily on manually operated precision levels or total stations. Measurement points are typically placed at the curb edges on either side of the roadway, with longitudinal measurement points manually positioned at specific ratios such as 1 / 4 or 1 / 8 of the bridge span. This method has significant limitations: low operational efficiency, requiring clear line of sight between the instrument and the front and rear, and on-site measurements are susceptible to traffic control. Manual reading and point placement carry the risk of subjective error, and data post-processing is cumbersome. Therefore, developing a systematic solution that integrates automated bridge defect inspection, intelligent identification, precise positioning, and automated alignment measurement has become a key breakthrough in improving bridge inspection efficiency. Summary of the Invention

[0005] In response to the problems existing in the existing technology, this application proposes a multispectral drone-based bridge appearance inspection method and system, which deeply integrates bridge detection, photogrammetry, computer vision, deep learning and other technologies to form a feasible closed-loop solution for bridge intelligent inspection.

[0006] First, a multispectral UAV bridge appearance inspection method includes the following:

[0007] S1, installing at least three ground control point reflective stickers and several bridge deck linear reflective stickers on the bridge, wherein the ground control point reflective stickers and the bridge deck linear reflective stickers are all round and of different colors;

[0008] S2, plans the basic UAV route, collects laser point cloud data of the target bridge, and constructs a three-dimensional model of the bridge;

[0009] S3, plans the drone’s refined route based on the bridge’s 3D model;

[0010] S4, based on the refined route, uses a visible light camera to capture bridge images;

[0011] S5, selecting orthophoto images within a preset distance from the drone to the ground control point from the bridge image, detecting two types of circular reflective stickers from the orthophoto images based on the color saturation of the reflective stickers, and outputting the pixel coordinates of the center of each reflective sticker and the corresponding orthophoto image;

[0012] S6, reading metadata of the orthophoto image of the reflective tape to obtain the three-dimensional coordinates of the ground control points and the pixel coordinates of the ground control points; the metadata of the orthophoto image of the reflective tape includes the three-dimensional coordinates of the center of the visible light camera, the attitude angle, and the camera intrinsic parameters;

[0013] S7, constructing the projection equation of each bridge deck linear reflective sticker orthophoto image, combining the projection equations of all bridge deck linear reflective sticker orthophoto images to form an overdetermined system of equations; introducing ground control point constraints, solving the initial value through SVD decomposition, and optimizing through bundle adjustment to obtain the three-dimensional coordinates of the bridge deck linear reflective sticker center, and determining the bridge deck alignment based on the three-dimensional coordinates;

[0014] S8, building a bridge apparent disease identification model;

[0015] S9, using the bridge apparent defect recognition model to detect defects in bridge photos taken by the drone, screening defect photos, and taking the pixel coordinates of the center point of the detected defect bounding box as the defect location;

[0016] S10, extracting the same disease from multiple photos, calculating the three-dimensional coordinates of the pixel coordinates of the center point of the disease boundary box, and realizing spatial positioning of the disease;

[0017] S11, based on the three-dimensional coordinate set of the located defects and the position changes of the bridge deck reflective stickers, plan the re-inspection route.

[0018] In a possible implementation of the first aspect, step S2 uses a five-way flight route as the basic route.

[0019] In a possible implementation of the first aspect, step S5 converts the orthophoto image into a grayscale image, preprocesses the image, and uses an edge detection algorithm to obtain the image edge; based on the color saturation of the reflective tape, an image processing algorithm is used to detect the circular reflective tape mark, and the coordinates of the center of the circle and the corresponding orthophoto image of the reflective tape are output.

[0020] In a possible implementation of the first aspect, step S6 pre-measures the three-dimensional coordinates of the ground control points using a total station.

[0021] In a possible implementation of the first aspect, step S7 specifically includes the following content:

[0022] (1) According to the rotation order of Z axis-Y axis-X axis, the overall rotation matrix of the i-th bridge deck linear reflective sticker orthophoto image is:

[0023]

[0024] Among them, R z (β) represents the rotation matrix of the heading angle β around the Z axis, R y (ω) represents the rotation matrix of the pitch angle ω around the Y axis, Indicates the roll angle around the X axis The rotation matrix of β, ω, Represents heading angle, pitch angle, and roll angle respectively;

[0025] (2) Define the projection matrix:

[0026] P i =K[R i |-R i ·C i ]

[0027] C i Represents the three-dimensional coordinates of the visible light camera center; K is the internal parameter matrix, f represents the focal length of the camera, (c z ,c y ) represents the coordinates of the principal point;

[0028] (3) For each bridge deck linear reflective sticker orthophoto image, the coordinates of the reflective sticker center (u i ,v i ), establish the projection equation:

[0029]

[0030] λ i Indicates the scale factor; X, Y, and Z represent the three-dimensional coordinates of the center of the bridge deck linear reflective sticker;

[0031] Eliminate the scaling factor λ i , which is expanded into a system of linear equations:

[0032]

[0033] Combining the equations of all bridge deck linear reflective sticker orthophoto images, we can form an overdetermined system of equations:

[0034]

[0035] A represents the global matrix, N represents the number of target points, and M represents the number of images for each target point;

[0036] (4) Introducing ground control point constraints:

[0037] The known coordinates of the control point (X k ,Y k ,Z k ) and the corresponding pixel coordinates (u k ,v k ) into the overdetermined equations, and we get the control point projection error equation:

[0038]

[0039] (5) For overdetermined equations Perform SVD decomposition, and the right singular vector corresponding to the minimum singular value is the initial three-dimensional point X0;

[0040] (6) The target point X is minimized by the reprojection error algorithm j Perform bundle adjustment iterative optimization:

[0041]

[0042] Obtain the precise three-dimensional coordinates of the center of the j-th reflective sticker through iterative optimization;

[0043] Where, (u ij ,v ij ) represents the pixel coordinates of the bridge linear reflective sticker of the j-th target point on the i-th image;

[0044] (7) The above steps are used to solve the three-dimensional coordinates of all bridge deck linear reflective stickers to realize the detection of bridge deck linear shape.

[0045] In a possible implementation of the first aspect, step S8 collects photos of bridge apparent defects, constructs a data set, and trains a deep learning model based on the data set to obtain a bridge apparent defect recognition model.

[0046] Furthermore, the UAV bridge appearance inspection method also includes:

[0047] S12, automatically generates a drone bridge inspection report based on the preset report template.

[0048] In a second aspect, the present application provides a system for performing the drone bridge appearance inspection method, comprising:

[0049] UAV terminals, used to carry lidar and industrial cameras;

[0050] LiDAR, used to collect laser point cloud data of the target bridge;

[0051] Industrial camera, used to capture images of target bridges;

[0052] Basic route planning module, used to plan the basic route of the drone;

[0053] An oblique photography modeling module is electrically connected / communicationally connected to the laser radar to construct a three-dimensional model of the bridge from the laser point cloud;

[0054] Refined route planning module, which plans the UAV’s refined route based on the 3D model of the bridge;

[0055] The bridge deck alignment detection module includes a reflective sticker recognition module and a reflective sticker coordinate calculation module. The reflective sticker recognition module selects orthophoto images taken by the drone within a preset distance from the ground control point from the target bridge image, converts the orthophoto images into grayscale images, pre-processes the images, and uses an edge detection algorithm to obtain image edges. The module detects circular marks and outputs the coordinates of the reflective sticker center and the corresponding orthophoto image. The reflective sticker coordinate calculation module reads the metadata of the reflective sticker orthophoto image and obtains the three-dimensional coordinates and pixel coordinates of the ground control point. The reflective sticker coordinate calculation module further calculates the three-dimensional coordinates of the reflective sticker center to achieve bridge deck alignment detection.

[0056] The apparent defect recognition module includes an intelligent defect detection module and a defect positioning and tracking module. The intelligent defect detection module collects photos of bridge apparent defects, constructs a data set, and trains a deep learning model based on this data set to obtain a bridge apparent defect recognition model; the disease positioning and tracking module uses the bridge apparent defect recognition model to detect defects in bridge photos taken by a drone at a fixed point, screens out photos containing defects, and records the center coordinates of the detected defect boundary box as the defect position. By extracting the same defect in multiple photos, the three-dimensional coordinates of the pixel coordinates of the center point of the disease boundary box are calculated to achieve spatial positioning of the defect.

[0057] Furthermore, the system further comprises:

[0058] The inspection report generation module is used to generate an inspection report based on the drone inspection results according to the preset template.

[0059] Compared with existing methods, this application proposes a full-process closed-loop solution from 3D modeling, autonomous inspection, linear detection, intelligent disease diagnosis, disease location, and disease tracking, which greatly improves the automation level of bridge inspection operations and the value of data application. Specifically, it has the following beneficial effects:

[0060] 1. This application constructs a three-dimensional model of a bridge foundation using lidar point cloud data. Active laser scanning overcomes the bottleneck of visible light modeling in identifying objects with low texture features. This allows for precise reconstruction of the spatial form of slender structures such as tree branches, high-voltage cables, cables, and streetlight poles, providing a reliable spatial reference for drones to plan millimeter-level safety inspection paths.

[0061] 2. This application uses digital image correlation methods to identify reflective stickers in photos and directly calculates their three-dimensional spatial coordinates through multi-image triangulation combined with a bundle adjustment algorithm, providing data support for regular bridge deck alignment monitoring. This method addresses the low efficiency and high subjectivity of traditional total station measurements and compensates for the lack of alignment detection capabilities commonly found in drones for bridge defect inspection.

[0062] 3. This application builds an intelligent bridge surface defect detection model based on the YOLO deep convolutional neural network, achieving rapid qualitative identification of defect types and quantitative segmentation of morphological dimensions within a single model.

[0063] 4. This application is based on multi-view Figure 3 The angle measurement method automatically extracts the same target from multiple images of the detected defect area and uses a bundle adjustment optimization algorithm to accurately calculate the 3D world coordinates of the center of its bounding box, achieving precise spatial positioning of the defect. This method overcomes the bottleneck of existing intelligent defect detection technologies that can only identify but not automatically locate the defect, and solves the problem of inefficiency and error-proneness in manually searching for the spatial location of defective bridges in imagery.

[0064] 5. This application combines the three-dimensional coordinates of the located defects and the location information of the bridge deck reflective stickers to plan efficient inspection routes for key areas. Through regular re-inspections and intelligent analysis, it realizes the temporal evolution analysis of the defect characteristics and the bridge deck line shape, providing data support for accurate bridge maintenance decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flow chart of a multispectral drone-based bridge appearance inspection method provided in an embodiment of the present application;

[0066] Figure 2 A three-dimensional bridge model is provided in the embodiment of the present application; wherein Figure 2 (a) is a schematic diagram of the three-dimensional model of the bridge using oblique photography. Figure 2 (b) Schematic diagram of the key inspection parts (web) of the bridge planned based on the 3D bridge model obtained by oblique photography;

[0067] Figure 3 A circular bridge deck linear reflective sticker sign provided in an embodiment of the present application;

[0068] Figure 4 A set of orthophoto images of linear reflective stickers on a circular bridge surface and a schematic diagram of their coordinate identification are provided in the embodiment of the present application; Figure 4 (a) is the first image of the bridge surface linear reflective stickers taken by the drone. Figure 4 (b) Figure 4 (a) Schematic diagram of reflective sticker coordinate recognition; Figure 4 (c) is the second image of the bridge surface linear reflective stickers taken by the drone. Figure 4 (d) Figure 4 (c) Schematic diagram of reflective sticker coordinate recognition; Figure 4 (e) is the third image of the bridge surface linear reflective stickers taken by the drone. Figure 4 (f) Figure 4 (e) Schematic diagram of reflective sticker coordinate recognition;

[0069] Figure 5 A diagram of the architecture of a drone bridge appearance inspection system is provided for this application;

[0070] Figure 6 A diagram showing the detection results of a bridge disease (damage) provided in an embodiment of the present application;

[0071] Figure 7 A diagram showing the detection results of a bridge defect (exposed reinforcement) provided in an embodiment of the present application;

[0072] Figure 8 This is a diagram of the detection results of a bridge disease (honeycombed surface) provided in an embodiment of the present application. Implementation Method

[0073] In order to make the purpose, technical solutions and advantages of this application clearer, this application will be further described in detail below with reference to the accompanying drawings.

[0074] It should be understood that the embodiments described are only a portion of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0075] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0076] First, as Figure 1 As shown, this application provides a multispectral UAV bridge appearance inspection method, including the following contents:

[0077] S1. Set at least three ground control points (GCPs) reflective stickers at the bridge head, the middle of the side span, the middle of the middle span, the quarter point, the three-quarter point, etc. The ground control points are non-collinear control points. Non-collinear control points can meet the needs of multi-view solution and ensure plane and elevation accuracy; at the same time, set N bridge deck linear reflective stickers on the target detection bridge, that is, the number of all target points that need to calculate the three-dimensional coordinates; in this application, both reflective stickers are circular and have different colors, and are distinguished by color saturation. The ground control points are used to determine the pixel coordinates of the ground control points, and the bridge deck linear reflective stickers are used to determine the bridge deck line shape; this application uses circular reflective stickers, which can significantly reduce the algorithm complexity compared to other complex-shaped reflective stickers.

[0078] S2: Plan the basic UAV flight path, set the lateral overlap rate to 60%, use the UAV platform equipped with a LiDAR to collect laser point cloud data of the target bridge, and construct a 3D model of the bridge based on the LiDAR point cloud;

[0079] For example, a DJI Matrice 350RTK equipped with a DJI Zenmuse L2 LiDAR can be used. The remote controller is equipped with flight control and mission management software adapted to the drone platform. A five-way flight route can be planned in DJI Pilot 2 software to collect laser point cloud data of the target bridge. DJI Zhitu can be used to construct a 3D model of the bridge based on the LiDAR point cloud. Figure 2 (a) shows a three-dimensional model of a bridge using oblique photography.

[0080] S3, based on the three-dimensional model of the bridge, plan the drone's refined route, set the heading overlap rate to 70%, and the lateral overlap rate to 50%. The refined route covers the key inspection parts of the bridge, including but not limited to the bridge web, bottom plate, bridge deck, and bridge piers. Figure 2 (b) shows the Figure 2 (a) Key inspection areas (webs) of the bridge planned on the basis.

[0081] S4, based on the refined route, uses a drone platform equipped with a visible light camera to capture images of the target bridge;

[0082] S5, select the orthophoto images within the preset distance from the drone to the ground control point from the target bridge image, convert the orthophoto images into grayscale images, pre-process the images, and use the edge detection algorithm to obtain the image edges; according to the color saturation of the reflective tape, use the image processing algorithm to detect two circular reflective tape marks, and output their center coordinates and the corresponding reflective tape orthophoto images. It should be noted that each reflective tape orthophoto image only contains one reflective tape mark; Figure 3 The figure shows a bridge deck linear reflective sticker sign provided in an embodiment of the present application;

[0083] For example, in the preprocessing process, Gaussian blur and median filtering can be selected to perform noise reduction on the image; and algorithms such as the Canny edge detection algorithm and the Sobel operator can be selected to obtain the edge of the noise-reduced image.

[0084] S6, the drone sets a specific heading and lateral overlap rate through refined route design to ensure multiple imaging of the same target, so the same reflective sticker will appear in multiple orthophotos; read the metadata of more than three orthophotos containing the same reflective sticker, and obtain the precise three-dimensional coordinates of the ground control point (X k ,Y k ,Z k ) and the pixel coordinates of the ground control points determined based on the ground control point reflective stickers (uk ,v k ), k∈[1,N], N represents the number of ground control points; the metadata include but are not limited to the three-dimensional coordinates of the visible light camera center, attitude angle (heading angle β, roll angle Pitch angle ω), camera internal parameters (focal length f, image principal point coordinates (c z ,c y ));

[0085] S7: For the same bridge linear reflective sticker, construct the projection equation for each orthophoto image of the bridge linear reflective sticker, and combine the projection equations of all orthophoto images of the bridge linear reflective sticker to form an overdetermined system of equations; introduce ground control point constraints, solve the initial value through SVD decomposition, and optimize through bundle adjustment to obtain the precise three-dimensional coordinates of the center of the bridge linear reflective sticker; this step specifically includes the following:

[0086] (1) According to the rotation order of Z axis-Y axis-X axis, the overall rotation matrix of the i-th bridge deck linear reflective sticker orthophoto image is:

[0087]

[0088] Among them, R z (β) represents the rotation matrix of the heading angle β around the Z axis, R y (ω) represents the rotation matrix of the pitch angle ω around the Y axis, Indicates the roll angle around the X axis The rotation matrix of β, ω, Represents heading angle, pitch angle, and roll angle respectively;

[0089] (2) Define the projection matrix:

[0090] P i =K[R i |-R i ·C i ]

[0091] C i represents the three-dimensional coordinates of the center of the visible light camera, represents the longitude, latitude, and altitude of the i-th bridge deck linear reflective sticker orthophoto image; K is the internal parameter matrix, f represents the focal length of the camera, (c z ,c y ) represents the coordinates of the principal point of the image; i∈[1,M], M represents the number of images of different bridge surface linear reflective sticker positions taken by the UAV;

[0092] (3) For the coordinates of the center of the reflective sticker in the i-th bridge surface linear reflective sticker orthophoto image (u i,v i ), establish the projection equation:

[0093]

[0094] λ i Indicates the scale factor; X, Y, and Z represent the three-dimensional coordinates of the center of the bridge deck linear reflective sticker, namely longitude, latitude, and altitude;

[0095] Eliminate the scaling factor λ i , which is expanded into a system of linear equations:

[0096]

[0097] The linear equations of all bridge deck linear reflective tape orthophoto images are combined to form an overdetermined equation system:

[0098]

[0099] A represents the global matrix, N represents the number of target points, and M represents the number of images for each target point;

[0100] (4) Introducing ground control point constraints:

[0101] The known coordinates of the control point (X k ,Y k ,Z k ) and the corresponding pixel coordinates (u k ,v k ) into the overdetermined equations, and we get the control point projection error equation:

[0102]

[0103] Through the pixel coordinates of the above-mentioned ground control points, the projection error of the control points is added to the optimization objective function to enhance the constraints of the equation and improve the solution accuracy;

[0104] P i (1,:) represents the projection matrix P i The first row vector of P i (2,:) represents the projection matrix P i The second row vector, P i (3,:) represents the projection matrix P i The third row vector of ;

[0105] (5) For overdetermined equations Perform SVD decomposition, and the right singular vector corresponding to the minimum singular value is the initial three-dimensional point X0; the initial three-dimensional point X0 is the initial guess value of the bundle adjustment iterative optimization, which is then gradually corrected through nonlinear optimization (such as the Levenberg-Marquardt algorithm) and finally converges to the exact solution

[0106] (6) The target point X is minimized by the reprojection error algorithm j Perform bundle adjustment iterative optimization:

[0107]

[0108] Obtain the precise three-dimensional coordinates of the center of the j-th reflective sticker through iterative optimization;

[0109] Where N represents the number of target points whose three-dimensional coordinates need to be calculated, that is, the total number of bridge linear reflective stickers involved in the optimization; M represents the number of images of the bridge linear reflective stickers taken, that is, the number of photos of each bridge linear reflective sticker position taken by the drone; N and M are used to accumulate the reprojection errors of all images and all target points to ensure global optimization; the accuracy of X0 depends on the number of reflective sticker images M and must meet the observability of N target points;

[0110] (u ij ,v ij ) represents the pixel coordinates of the bridge linear reflective sticker of the jth target point on the i-th image; the target point X j The theoretical pixel coordinates on the i-th bridge deck linear reflective tape orthophoto are:

[0111]

[0112] (7) The above steps are used to solve the three-dimensional coordinates of all bridge deck linear reflective stickers to realize the detection of bridge deck linear shape.

[0113] Figure 4 An embodiment of the invention involves attaching circular linear reflective stickers to a bridge deck at Liandong U Valley (International Enterprise Port in Shapingba District, Chongqing), using a DJI M4E to take orthophotos of different locations on the bridge, and then performing coordinate calculations on three photos of one of the linear reflective stickers. Figure 4 This is the orthophoto image of the bridge deck linear reflective stickers and the identification image of their center coordinates.

[0114] Table 1 Test photo metadata

[0115]

[0116] The three-dimensional geodetic coordinates of the center of the bridge deck linear reflective sticker in the WGS84 coordinate system are calculated as follows:

[0117] Latitude = 29.571815216°, Longitude = 106.289196487°, Altitude = 282.042m.

[0118] S8, collecting a certain number (preferably more than 2000) of bridge surface defect photos, constructing a data set, and training a deep learning model based on the data set to obtain a bridge surface defect recognition model;

[0119] Exemplarily, the deep learning model adopts a YOLO target detection model, such as YOLOv12.

[0120] S9, using the bridge apparent defect recognition model to detect defects in the bridge photos taken by the drone, filter out photos containing defects, and record the center coordinates (u, v) of the detected defect bounding box as the defect location.

[0121] S10, based on multi-view Figure 3 The angle measurement method extracts the same disease target from multiple photos of the detected disease area, and calculates the three-dimensional coordinates of the center coordinates (u, v) of the disease boundary box through the bundle adjustment optimization algorithm to achieve spatial positioning of the disease.

[0122] S11, based on the three-dimensional coordinate set of located defects and the position of reflective stickers on bridge deck lines with large displacements, plans re-inspection routes for key areas. Through regular re-inspections and intelligent analysis, it can track the temporal evolution of defect characteristics and bridge deck lines, providing data support for bridge maintenance decisions.

[0123] Furthermore, in a preferred embodiment of the first aspect, step S2 adopts a five-way flight route as a basic route.

[0124] Furthermore, in a preferred embodiment of the first aspect, step S5 detects the reflective sticker mark based on a Hough circle transform algorithm and outputs the coordinates of its center.

[0125] Furthermore, in a preferred embodiment of the first aspect, step S6 can determine the three-dimensional coordinates (X k ,Y k ,Z k ).

[0126] Furthermore, the multispectral UAV bridge appearance inspection method also includes:

[0127] S12, automatically generates a drone bridge inspection report based on a preset report template, and the report content includes but is not limited to the location of the defect, the size of the defect, and the type of defect.

[0128] At this point, this application has built a complete "multi-spectral data acquisition - intelligent identification - precise positioning - time series tracking - automatic reporting" full-process intelligent detection closed-loop system for bridge surface defects, which can significantly improve engineering detection efficiency and provide support for the realization of intelligent bridge maintenance.

[0129] Second, as Figure 5 As shown, the present application provides a system for performing the UAV bridge appearance inspection method, comprising:

[0130] UAV terminal, used to carry laser radar,

[0131] LiDAR, used to collect laser point cloud data of the target bridge;

[0132] Industrial camera, used to capture images of target bridges;

[0133] Basic route planning module, used to plan the basic route of the drone;

[0134] An oblique photography modeling module is electrically connected / communicationally connected to the laser radar to construct a three-dimensional model of the bridge from the laser point cloud;

[0135] The refined route planning module plans refined routes for autonomous inspections of bridge structures, under-bridge safety zones, etc. based on the three-dimensional bridge model;

[0136] The bridge deck alignment detection module includes a reflective sticker recognition module and a reflective sticker coordinate calculation module; the reflective sticker recognition module selects orthophoto images of drones within a preset distance from ground control points from the target bridge image, converts the orthophoto images into grayscale images, performs noise reduction on the images, and uses an edge detection algorithm to obtain image edges; detects circular marks and outputs the coordinates of the circle center and the corresponding reflective sticker orthophoto image; the reflective sticker coordinate calculation module reads the metadata of the reflective sticker orthophoto image and obtains the three-dimensional coordinates of the ground control points and the two-dimensional pixel coordinates of the ground control points; the reflective sticker coordinate calculation module further constructs a projection matrix, combines the equations of all bridge deck alignment reflective sticker orthophoto images to form an overdetermined set of equations, introduces ground control point constraints, solves the initial values ​​through SVD decomposition, and obtains the three-dimensional coordinates of the bridge deck alignment reflective sticker center through bundle adjustment optimization, thereby realizing the detection of the bridge deck alignment;

[0137] The apparent defect recognition module, including an intelligent defect detection module and a defect location and tracking module, qualitatively and quantitatively identifies apparent defects and regularly tracks defects. The intelligent defect detection module collects photos of bridge apparent defects, constructs a data set, and trains a deep learning model based on this data set to obtain a bridge apparent defect recognition model. The defect location and tracking module uses the bridge apparent defect recognition model to detect defects in photos of bridges taken by drones at fixed points, filters out photos containing defects, and records the center coordinates of the detected defect bounding box as the defect location. By extracting the same defect from multiple photos, the three-dimensional coordinates of the pixel coordinates of the center point of the defect bounding box are calculated to achieve spatial positioning of the defect.

[0138] Furthermore, the system further comprises:

[0139] The inspection report generation module is used to generate an inspection report based on the drone inspection results according to the preset template.

[0140] The feasibility and effectiveness of this application are described below with reference to an application example. This embodiment uses YOLOv12 as the basic model for bridge disease detection.

[0141] (1) Dataset construction:

[0142] Photos of bridge surface defects were collected to construct a data set, and all images in the data set were divided into training set and validation set in a ratio of 80% and 20% for training.

[0143] Based on the size, shape and other characteristics of the target disease, it is divided into model one and model two for testing;

[0144] Model 1: Detecting exposed rebar and damage in bridge defects; 396 data images; 421 training rounds, 0.699 hours;

[0145] Model 2: Detect honeycomb and pitting in bridge defects; 148 data images; trained for 272 rounds, taking 0.217 hours;

[0146] Use the calibration tool labelme to calibrate the model.

[0147] (2) Evaluation indicators:

[0148] Training time, single-image detection time, average precision (AP), and detection results were selected as evaluation metrics for the algorithm. The AP value fully reflects the model's ability to detect disease defects; higher AP values ​​indicate higher detection accuracy. mAP50% represents the average detection accuracy of all target categories when the Intersection over Union (IOU) threshold is 0.5; mAP50-95% represents the average detection accuracy of all target categories when the IOU threshold is between 0.5 and 0.95.

[0149] Table 1. Model 1 exposed reinforcement / damage detection results

[0150] Algorithm Name Training time Model memory usage Single image detection time mAP 50% mAP50-95% YOLOv12 0.699 hours 23M 7.3ms 62.5％ 41.4％

[0151] Table 2 Test results of model two honeycomb surface

[0152] Algorithm Name Training time Model memory usage Single image detection time mAP 50% mAP50-95% YOLOv12 0.217 hours 23M 6.3ms 74.5％ 50.5％

[0153] like Figure 6 、 Figure 7 The detection results of model 1 for damage and exposed reinforcement respectively. Figure 8 This is the test result diagram of model 2 for honeycomb surface.

[0154] Compared to existing technologies, this embodiment does not require similarity calculations for every position in the feature map. By dividing the feature map into horizontal and vertical regions (the default setting is 4 in the experiment), the receptive field is reduced to 1 / 4 of the original, and only the attention score within each region is calculated, which significantly improves the calculation speed with almost negligible impact on performance. Secondly, this embodiment introduces residual connections and feature aggregation optimization through the YOLO algorithm, improves the stability of the gradient flow through block-level residual design and bottleneck structure, and effectively solves the problem of gradient obstruction and difficulty in convergence when training large models.

[0155] The above embodiments are intended only to illustrate the technical concepts and features of this application. Their purpose is to enable those skilled in the art to understand the content of this application and implement it accordingly. They are not intended to limit the scope of protection of this application. Those skilled in the art may make improvements and modifications without departing from the principles of this application, and such improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A multispectral UAV bridge appearance inspection method, characterized in that: Includes the following: S1, installing at least three ground control point reflective stickers and several bridge deck linear reflective stickers on the bridge, wherein the ground control point reflective stickers and the bridge deck linear reflective stickers are all round and of different colors; S2, plans the basic UAV route, collects laser point cloud data of the target bridge, and constructs a three-dimensional model of the bridge; S3, plans the drone’s refined route based on the bridge’s 3D model; S4, based on the refined route, uses a visible light camera to capture bridge images; S5, selecting orthophoto images within a preset distance from the drone to the ground control point from the bridge image, detecting two types of circular reflective stickers from the orthophoto images based on the color saturation of the reflective stickers, and outputting the pixel coordinates of the center of each reflective sticker and the corresponding orthophoto image; S6, reading metadata of the orthophoto image of the reflective tape to obtain the three-dimensional coordinates and pixel coordinates of the ground control points; S7, constructing the projection equation of each bridge deck linear reflective sticker orthophoto image, combining the projection equations of all bridge deck linear reflective sticker orthophoto images to form an overdetermined system of equations; introducing ground control point constraints, solving the initial value through SVD decomposition, and optimizing through bundle adjustment to obtain the three-dimensional coordinates of the bridge deck linear reflective sticker center, and determining the bridge deck alignment based on the three-dimensional coordinates; S8, building a bridge apparent disease identification model; S9, using the bridge apparent defect recognition model to detect defects in bridge photos taken by the drone, screening defect photos, and taking the pixel coordinates of the center point of the detected defect bounding box as the defect location; S10, extracting the same disease from multiple photos, calculating the three-dimensional coordinates of the pixel coordinates of the center point of the disease boundary box, and realizing spatial positioning of the disease; S11, based on the three-dimensional coordinate set of the located defects and the position changes of the bridge deck reflective stickers, plan the re-inspection route.

2. The multispectral UAV bridge appearance inspection method according to claim 1 is characterized in that: In step S2, the basic route adopts a five-direction flight route.

3. The multispectral UAV bridge appearance inspection method according to claim 1 is characterized in that: In step S5, the orthophoto image is converted into a grayscale image, the image is preprocessed, and the image edge is obtained using an edge detection algorithm; according to the color saturation of the reflective tape, an image processing algorithm is used to detect the circular reflective tape mark, and the coordinates of the circle center and the corresponding reflective tape orthophoto image are output.

4. The multispectral UAV bridge appearance inspection method according to claim 1 is characterized in that: In step S6, the metadata of the orthophoto image of the reflective tape includes the three-dimensional coordinates of the center of the visible light camera, the attitude angle, and the camera intrinsic parameters.

5. The multispectral UAV bridge appearance inspection method according to claim 1 is characterized in that: In step S6, the three-dimensional coordinates of the ground control points are determined in advance by total station measurement.

6. The multispectral UAV bridge appearance inspection method according to claim 4 is characterized in that: Step S7 specifically includes the following contents: (1) According to the rotation order of Z axis-Y axis-X axis, the overall rotation matrix of the i-th bridge deck linear reflective sticker orthophoto image is: in, R z(β) represents the rotation matrix of the heading angle β around the Z axis, R y (ω) represents the rotation matrix of the pitch angle ω around the Y axis, Indicates the roll angle around the X axis The rotation matrix of β, ω, Represents heading angle, pitch angle, and roll angle respectively; (2) Define the projection matrix: P i =K[R i |-R i ·C i ] C i Represents the three-dimensional coordinates of the visible light camera center; K is the internal parameter matrix, f represents the focal length of the camera, (c z ,c y ) represents the coordinates of the principal point; (3) For each bridge deck linear reflective sticker orthophoto image, the coordinates of the reflective sticker center (u i ,v i ), establish the projection equation: λ i Indicates the scale factor; X, Y, and Z represent the three-dimensional coordinates of the center of the bridge deck linear reflective sticker; Eliminate the scaling factor λ i , which is expanded into a system of linear equations: Combining the equations of all bridge deck linear reflective sticker orthophoto images, we can form an overdetermined system of equations: A represents the global matrix, N represents the number of target points, and M represents the number of images for each target point; (4) Introducing ground control point constraints: The known coordinates of the control point (X k ,Y k ,Z k ) and the corresponding pixel coordinates (u k ,v k ) into the overdetermined equations, and we get the control point projection error equation: (5) For overdetermined equations Perform SVD decomposition, and the right singular vector corresponding to the minimum singular value is the initial three-dimensional point X0; (6) The target point X is minimized by the reprojection error algorithm j Perform bundle adjustment iterative optimization: Obtain the precise three-dimensional coordinates of the center of the j-th reflective sticker through iterative optimization; Where, (u ij ,v ij ) represents the pixel coordinates of the bridge linear reflective sticker of the j-th target point on the i-th image; (7) The above steps are used to solve the three-dimensional coordinates of all bridge deck linear reflective stickers to realize the detection of bridge deck linear shape.

7. The multispectral UAV bridge appearance inspection method according to claim 1 is characterized in that: In step S8, photos of bridge surface defects are collected to construct a data set, and a deep learning model is trained based on the data set to obtain a bridge surface defect recognition model.

8. The multispectral UAV bridge appearance inspection method according to claim 1 is characterized in that: Also includes: S12, automatically generates a drone bridge inspection report based on the preset report template.

9. A system for executing the method according to any one of claims 1 to 8, characterized in that: include: UAV terminals, used to carry lidar and industrial cameras; LiDAR, used to collect laser point cloud data of the target bridge; Industrial camera, used to capture images of target bridges; Basic route planning module, used to plan the basic route of the drone; An oblique photography modeling module is electrically connected / communicationally connected to the laser radar to construct a three-dimensional model of the bridge from the laser point cloud; Refined route planning module, which plans the UAV’s refined route based on the 3D model of the bridge; The bridge deck alignment detection module includes a reflective sticker recognition module and a reflective sticker coordinate calculation module. The reflective sticker recognition module selects orthophoto images taken by the drone within a preset distance from the ground control point from the target bridge image, converts the orthophoto images into grayscale images, pre-processes the images, and uses an edge detection algorithm to obtain image edges. The module detects circular marks and outputs the coordinates of the reflective sticker center and the corresponding orthophoto image. The reflective sticker coordinate calculation module reads the metadata of the reflective sticker orthophoto image and obtains the three-dimensional coordinates and pixel coordinates of the ground control point. The reflective sticker coordinate calculation module further calculates the three-dimensional coordinates of the reflective sticker center to achieve bridge deck alignment detection. The apparent defect recognition module includes an intelligent defect detection module and a defect positioning and tracking module. The intelligent defect detection module collects photos of bridge apparent defects, constructs a data set, and trains a deep learning model based on this data set to obtain a bridge apparent defect recognition model; the disease positioning and tracking module uses the bridge apparent defect recognition model to detect defects in bridge photos taken by a drone at a fixed point, screens out photos containing defects, and records the center coordinates of the detected defect boundary box as the defect position. By extracting the same defect in multiple photos, the three-dimensional coordinates of the pixel coordinates of the center point of the disease boundary box are calculated to achieve spatial positioning of the defect.

10. The system according to claim 8, wherein: Also includes: The inspection report generation module is used to generate an inspection report based on the drone inspection results according to the preset template.

Images