Air-land integrated bridge disease monitoring method and application system thereof

Abstract

The invention relates to an air-ground integrated bridge disease monitoring method and an application system thereof, and the method comprises the steps: controlling an unmanned plane to take off from a nest, and transmitting a bridge initial inspection task to the unmanned plane; sending the bridge image data to a vehicle-mounted high-performance edge server for primary three-dimensional modeling; carrying out monomer splitting on the general three-dimensional model of the bridge, and carrying out close photogrammetry on the bridge by an unmanned aerial vehicle based on a close inspection route to obtain a monomer high-resolution image of each part of the bridge; the unmanned aerial vehicle automatic inspection and data processing platform executes disease target detection and disease semantic segmentation on the single high-resolution images; and associating and binding disease target detection and disease semantic segmentation results with the bridge refined three-dimensional model to generate a bridge disease detection report. A worker only needs to draw an inspection area, and the unmanned aerial vehicle automatic inspection and data processing platform automatically generates an unmanned aerial vehicle inspection route according to the drawn inspection area, so that the unmanned aerial vehicle autonomously and safely inspects a bridge.

Description

Technical Field

[0001] The present invention relates to the field of bridge monitoring, and particularly to an air-ground integrated bridge disease monitoring method and its application system. Background Art

[0002] How to more scientifically and efficiently manage, maintain, and conduct emergency rescue for bridges has also become the top priority of traffic work at the present stage.

[0003] There are mainly some problems with the manual detection method: Manual inspection consumes a large amount of manpower and material resources, obstructs traffic, has low work efficiency, and is prone to missed or incorrect inspections due to personal reasons. For areas such as high piers, bridge towers, or bridge components that cannot be reached by bridge inspection vehicles, conventional detection means and methods have limitations, and the operability is very difficult, resulting in detection blind spots. The data acquisition is incomplete, the image data quality is poor, and the positioning and measurement accuracy are low. The degree of digitization and integration is low, the results are not intuitive, and it is difficult to review the inspection results.

[0004] In recent years, with the booming development of technologies such as remote sensing, unmanned aerial vehicles (UAVs), and computer vision, the bridge inspection industry has actively carried out research on new construction and operation and maintenance inspection technologies. Due to its flexible and good passability characteristics, the UAV has become a new type of equipment that cannot be ignored in bridge operation and maintenance inspection. For bridge disease locations that are difficult to detect and reach manually, the UAV can take pictures of diseases through the pan-tilt camera carried, ensuring the safety of personnel, and is of practical significance for the inspection of structures such as stay cables, bridge towers, and high piers.

[0005] However, there are still some problems in using drones for bridge inspection: In the traditional drone inspection method, the inspection personnel or drone pilots manually control the aircraft to the designated position to take bridge photos. Due to the diverse types and complex structures of bridges and the complex on-site environment for inspection, it has high requirements for the drone flight experience of inspection personnel or drone pilots, which greatly limits the application and promotion of drones in the field of bridge inspection. Common bridge modeling methods usually use drone oblique photography technology to carry out real-scene three-dimensional modeling of bridges, use the real-scene three-dimensional model to label and correlate the positions of bridge diseases, and generate bridge disease detection reports. However, due to the single flight route and fixed flight height of drones in the oblique photography modeling method, it can often only obtain the main frame and top information of the bridge, and it is difficult to obtain side and bottom information when facing bridges with complex structures, resulting in problems such as missing, blurred, and streaky model textures, and unable to meet the requirements of bridge inspection accuracy. The existing methods for recording and archiving bridge diseases usually rely on inspection personnel to visually inspect the appearance of bridges, evaluate the health status of bridges by observing surface cracks, spalling, deformation, corrosion, etc., and record the disease conditions using text or photos, or mark the positions of diseases on the design drawings or schematic diagrams of bridges. Although this method is simple and direct, it is easily affected by the experience and subjective judgment of inspection personnel, and cannot visually review the disease conditions and quickly locate diseases. After the traditional drone inspection fieldwork is completed, it is usually necessary to upload the taken oblique photography photos to the user's local server or cloud server and use the high computing power of the server for three-dimensional modeling. However, the user's local server is usually deployed in the user's own data center, which is far from the inspected bridge. The in-house work cannot be completed quickly, and rapid modeling cannot be achieved, resulting in the inability of the staff to quickly and timely grasp the bridge disease conditions. And the method of using cloud modeling is usually limited by the network quality near the bridge. Some bridges may be located in remote mountainous areas or other environments without network connection or with extremely poor network quality. In addition, high-resolution oblique photography data files are usually very large, and transmitting these data through the network may be very time-consuming and limited by bandwidth, resulting in the inability to quickly generate three-dimensional models and timely generate bridge disease reports. Summary of the Invention

[0006] The present invention provides an air-ground integrated bridge disease monitoring method and its application system, aiming to solve at least one of the technical problems existing in the prior art.

[0007] The technical solution of the present invention is an air-ground integrated bridge disease monitoring method, which is applied to an air-ground integrated bridge disease monitoring system. The air-ground integrated bridge disease monitoring system includes a hangar, which is electrically connected to an unmanned aerial vehicle (UAV) automatic inspection and data processing platform; a UAV, which is arranged in the hangar and charged through the charging interface of the hangar. The UAV includes a visual obstacle avoidance module, an infrared obstacle avoidance module, a positioning module, a pan-tilt camera, a top-view camera and a control module; a UAV automatic inspection and data processing platform, which is used to plan the flight route of the UAV, issue route tasks, control the flight process and actions of the UAV, and perform data processing and data exchange on the data collected by the UAV; a vehicle-mounted high-performance edge server, which is used to perform data exchange with the UAV automatic inspection and data processing platform, and the vehicle-mounted high-performance edge server is electrically connected to the UAV automatic inspection and data processing platform; a user terminal device, which is used to run the UAV automatic inspection and data processing platform, and the user terminal device is electrically connected to the UAV automatic inspection and data processing platform. The air-ground integrated bridge disease monitoring method includes the following steps:

[0008] S100. The inspection vehicle drives to a suitable position of the bridge to be inspected. The UAV automatic inspection and data processing platform controls the UAV to take off from the hangar and sends a primary inspection task of the bridge to the UAV. The UAV obtains bridge image data through an oblique photography device. After completing the oblique photogrammetry task of the bridge, the UAV lands on the hangar and uploads the bridge image data to the UAV automatic inspection and data processing platform through the hangar;

[0009] S200. The UAV automatic inspection and data processing platform sends the bridge image data to the vehicle-mounted high-performance edge server for primary 3D modeling to obtain a rough 3D model of the bridge. The vehicle-mounted high-performance edge server sends the rough 3D model of the bridge to the UAV automatic inspection and data processing platform;

[0010] S300. The UAV automatic inspection and data processing platform performs monomerization splitting on the rough 3D model of the bridge to obtain multiple monomerized components. The UAV automatic inspection and data processing platform plans a close inspection route corresponding to each monomerized component. The UAV performs close photogrammetry on the bridge based on the close inspection route to obtain high-resolution images of each part of the bridge;

[0011] S400. The UAV sends the high-resolution monomer images to the UAV automatic inspection and data processing platform. The UAV automatic inspection and data processing platform performs disease target detection and disease semantic segmentation on the high-resolution monomer images to obtain the shape, size and spatial distribution of each bridge disease;

[0012] S500. Perform the second three-dimensional reconstruction based on close-range photogrammetry and conduct disease location to obtain a refined three-dimensional model of the bridge;

[0013] S600. The UAV automatic inspection and data processing platform associates and binds the disease target detection and disease semantic segmentation results with the refined three-dimensional model of the bridge to generate a bridge disease detection report.

[0014] Furthermore, the present invention also proposes an air-ground integrated bridge disease monitoring system for implementing the air-ground integrated bridge disease monitoring method. The air-ground integrated bridge disease monitoring system is carried on an inspection vehicle, and the air-ground integrated bridge disease monitoring system includes:

[0015] A machine nest, which is electrically connected to the UAV automatic inspection and data processing platform;

[0016] A UAV, which is arranged in the machine nest. The UAV is charged through the charging interface of the machine nest. The UAV includes a visual obstacle avoidance module, an infrared obstacle avoidance module, a positioning module, a gimbal camera, a top view camera, and a control module. The gimbal camera is an oblique photography device. The visual obstacle avoidance module is installed in the front, rear, sides, and above of the UAV, and the infrared obstacle avoidance module is installed at the bottom of the UAV;

[0017] A UAV automatic inspection and data processing platform, which is used to plan the flight route of the UAV, issue route tasks, control the flight process and actions of the UAV, and perform data processing and data exchange on the data collected by the UAV;

[0018] A vehicle-mounted high-performance edge server, which is used to perform data exchange with the UAV automatic inspection and data processing platform. The vehicle-mounted high-performance edge server is electrically connected to the UAV automatic inspection and data processing platform;

[0019] A user terminal device, which is used to run the UAV automatic inspection and data processing platform. The user terminal device is electrically connected to the UAV automatic inspection and data processing platform.

[0020] The beneficial effects of the present invention include: for the air-ground integrated bridge disease monitoring system and method, the staff only needs to draw the inspection area, and the UAV automatic inspection and data processing platform can automatically generate the UAV inspection route according to the drawn inspection area, realizing the autonomous and safe inspection of the bridge by the UAV.

[0021] In addition, some of the additional aspects and advantages of the present invention will be given in the following description, some will become obvious from the following description, or will be understood through the practice of the present invention. Description of the Drawings

[0022] Figure 1 It is a flowchart of the air-ground integrated bridge disease monitoring method according to an embodiment of the present invention.

[0023] Figure 2 It is a schematic flowchart of the air-ground integrated bridge disease monitoring method according to an embodiment of the present invention.

[0024] Figure 3 It is an "orthogonal flight" route map of oblique photography for the air-ground integrated bridge disease monitoring method according to an embodiment of the present invention.

[0025] Figure 4 It is a "circumferential flight" route map of oblique photography for the air-ground integrated bridge disease monitoring method according to an embodiment of the present invention.

[0026] Figure 5 It is a framework diagram of the YOLO v5 network structure for the air-ground integrated bridge disease monitoring method according to an embodiment of the present invention.

[0027] Figure 6 It is a block diagram of the main module structure of YOLO v5 for the air-ground integrated bridge disease monitoring method according to an embodiment of the present invention.

[0028] Figure 7 It is a network structure diagram of the DeeplabV3+ model for the air-ground integrated bridge disease monitoring method according to an embodiment of the present invention.

[0029] Figure 8 It is a schematic diagram of the structure of the air-ground integrated bridge disease monitoring system according to an embodiment of the present invention. Detailed implementation manners

[0030] The following will clearly and completely describe the concept, specific structure and technical effects generated by the present invention in combination with embodiments and drawings, so as to fully understand the purpose, solution and effects of the present invention. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0031] It should be noted that, unless otherwise specified, when a certain feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to another feature, or indirectly fixed or connected to another feature. In addition, the up, down, left, right, top, bottom, etc. used in the present invention are only relative to the mutual positional relationship of the various components of the present invention in the drawings.

[0032] In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this technology belongs. The terms used in the description of this specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0033] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, without departing from the scope of this disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element.

[0034] Referring to Figures 1 to 8 , in some embodiments, according to the air-ground integrated bridge disease monitoring method of the present invention, it is applied to an air-ground integrated bridge disease monitoring system. The air-ground integrated bridge disease monitoring system includes a hangar, and the hangar is electrically connected to an unmanned aerial vehicle (UAV) automatic inspection and data processing platform; a UAV, the UAV is arranged in the hangar, and the UAV is charged through the charging interface of the hangar. The UAV includes a vision obstacle avoidance module, an infrared obstacle avoidance module, a positioning module, a pan-tilt camera, a top view camera, and a control module; a UAV automatic inspection and data processing platform, which is used to plan the flight route of the UAV, issue route tasks, control the flight process and actions of the UAV, and perform data processing and data exchange on the data collected by the UAV; a vehicle-mounted high-performance edge server, which is used to perform data exchange with the UAV automatic inspection and data processing platform, and the vehicle-mounted high-performance edge server is electrically connected to the UAV automatic inspection and data processing platform; a user terminal device, which is used to run the UAV automatic inspection and data processing platform, and the user terminal device is electrically connected to the UAV automatic inspection and data processing platform. Referring to Figure 1 and Figure 2 , the air-ground integrated bridge disease monitoring method includes the following steps:

[0035] S100. The inspection vehicle travels to a suitable position of the bridge to be inspected. The UAV automatic inspection and data processing platform controls the UAV to take off from the hangar and sends the initial bridge inspection task to the UAV. The UAV obtains bridge image data through the oblique photography device. After completing the bridge oblique photogrammetry task, the UAV lands on the hangar and uploads the bridge image data to the UAV automatic inspection and data processing platform through the hangar.

[0036] S200. The UAV automatic inspection and data processing platform sends the bridge image data to the in-vehicle high-performance edge server for initial 3D modeling to obtain a rough 3D model of the bridge, and the in-vehicle high-performance edge server sends the rough 3D model of the bridge to the UAV automatic inspection and data processing platform;

[0037] S300. The UAV automatic inspection and data processing platform performs monomerization splitting on the rough 3D model of the bridge to obtain multiple monomerized components. The UAV automatic inspection and data processing platform plans the close inspection flight routes corresponding to each monomerized component, and the UAV conducts close-range photogrammetry on the bridge based on the close inspection flight routes to obtain high-resolution images of each part of the bridge;

[0038] S400. The UAV sends the high-resolution monomer images to the UAV automatic inspection and data processing platform, and the UAV automatic inspection and data processing platform performs disease target detection and disease semantic segmentation on the high-resolution monomer images to obtain the shape, size, and spatial distribution of each bridge disease;

[0039] S500. Perform the second 3D reconstruction based on close-range photogrammetry and conduct disease positioning to obtain a refined 3D model of the bridge;

[0040] S600. The UAV automatic inspection and data processing platform associates and binds the disease target detection and disease semantic segmentation results with the refined 3D model of the bridge to generate a bridge disease detection report.

[0041] The beneficial effects of the present invention include: for the above-mentioned air-ground integrated bridge disease monitoring system and method, the staff only needs to draw the inspection area, and the UAV automatic inspection and data processing platform can automatically generate the UAV inspection flight route according to the drawn inspection area, realizing the autonomous and safe inspection of the bridge by the UAV.

[0042] Specifically, in view of the problem that the existing UAV inspection method overly relies on manual operation and the UAV flight experience of the drone pilots, the present invention proposes a bridge automated inspection method and platform. The staff only needs to draw the inspection area, and the platform can automatically generate the UAV inspection route according to the drawn inspection area, enabling the UAV to autonomously and safely conduct inspections on the bridge. In view of the problems in the existing oblique photogrammetry technology, such as the single UAV flight route, fixed flight altitude, and the inability to accurately obtain the structural information and texture information of the side and bottom of the bridge when the UAV conducts shooting on a bridge with a complex structure, resulting in problems such as missing, blurred, and streaked textures in the 3D model, it is proposed to first use the basic oblique photogrammetry route to conduct the initial inspection of the bridge, use the bridge photos obtained from the initial inspection results for the initial 3D modeling to obtain a general bridge 3D model, monomerize and split the general bridge 3D model into components such as pylons, main cables, suspenders, stiffening girders, anchorages, bridge decks, and bridge piers. According to the structural characteristics of different components, plan the refined inspection routes for different components, design the route tasks, and conduct the second inspection, enabling the UAV to achieve fully autonomous flight on the refined routes, complete the close-range photogrammetry of the bridge, and obtain a refined 3D model of the bridge. With the help of the refined 3D model, the staff can quickly judge the bridge disease conditions and the locations of the diseases. The existing methods for bridge disease recording and archiving usually rely on the visual inspection of the bridge appearance by the inspectors, evaluate the health status of the bridge by observing the surface cracks, spalling, deformation, corrosion and other appearances, and record the disease conditions using text or photos, or mark the locations of the diseases on the design drawings or schematic diagrams of the bridge. Although this method is simple and direct, it is easily affected by the experience and subjective judgment of the inspectors, and cannot intuitively review the disease conditions and quickly locate the diseases. The present invention conducts object detection, semantic segmentation, and refined 3D modeling on the bridge diseases by means of the high-resolution photos obtained from the close-range photography of the bridge, quickly identifies the types, areas, or lengths of the bridge diseases, and obtains a refined 3D model of the bridge. The user can conduct all-round browsing and review of the bridge at any time and place, and understand the specific geographical coordinates of the diseases. By associating and binding information such as the disease types, areas, or lengths with the disease locations on the 3D model, the user only needs to click on the disease location in the 3D model to view all the information of the bridge diseases, and can also export the disease detection report, the information including the bridge name, bridge type, shooting time, disease type, disease size, and disease 3D geographical coordinates, realizing the electronic and information-based archiving of the bridge. The existing oblique photogrammetry modeling technology highly relies on the high computing power of the server in the user data center or the cloud server to reconstruct the image data captured by the UAV into a 3D model, and the accuracy of the oblique photogrammetry model often fails to meet the requirements of bridge disease detection.To address this issue, the present invention deploys edge servers with high computing power on inspection vehicles. By leveraging the flexibility of inspection vehicles and the high computing power of edge servers, it is possible to achieve rapid data processing and 3D modeling of multiple bridges with different types and complex structures in environments without network connection such as remote mountainous areas or environments with extremely poor network quality. The traditional simple in-house data processing process is transferred to the site to improve the modeling efficiency, facilitating the subsequent decomposition of the approximate bridge model obtained from oblique photography into different components, designing UAV close-range photography routes for each part in a targeted manner, collecting more detailed and rich bridge geometric structures and textures, performing disease target detection and semantic segmentation based on high-resolution bridge photos, and achieving refined 3D modeling of the bridge and quickly generating a bridge disease detection report.

[0043] The air-ground integrated bridge disease monitoring system mainly consists of two major parts. One is the UAV nest and the UAVs inside it, and the other is the ground station equipment. Among them, the UAV includes an obstacle avoidance module, a positioning module, a gimbal camera, a top view camera, and a control module. Each module coordinates and cooperates to achieve real-time control of the UAV's flight attitude, position, and speed, complete the route tasks issued by the UAV automatic inspection and data processing platform, and realize the inspection work of the bridge. The UAV nest is mainly used for the automatic takeoff and landing of the UAV. The UAV can also automatically charge through the charging interface of the nest, improving the UAV's endurance in the outdoors. The nest is equipped with data transmission and storage functions, and can transmit the data collected by the UAV back to the UAV automatic inspection and data processing platform.

[0044] The ground station equipment includes user terminal equipment and a high-performance edge server. Among them, the user terminal equipment and the high-performance edge server are fixedly installed on the inspection vehicle. The user terminal equipment is used to run the UAV automatic inspection and data processing platform. The platform can plan flight routes for the UAV, issue route tasks, control the flight process and actions of the UAV, and can also perform data exchange and task offloading with the high-performance edge server. For example, it offloads computationally intensive tasks such as 3D modeling to the high-performance edge server, and the server is responsible for reconstructing the bridge photos obtained from oblique photography into a real-scene 3D model of the bridge. In addition, the functions of the platform also include: monomerization and splitting of the 3D model, bridge disease target detection and semantic segmentation, 3D model viewing and annotation, disease information association and binding, disease information storage, and disease report generation.

[0045] The high-performance edge server in the ground station equipment is mainly used to process scenarios that strongly rely on computing power, such as 3D modeling. The UAV automatic inspection and data processing platform transmits photo data to the high-performance edge server, uses the high computing power of the server to quickly model and obtain a rough 3D model of the bridge. The server uploads the rough 3D model file back to the platform. The platform performs monomerization splitting on the overall bridge model, automatically designs corresponding planned flight routes for each monomerized component. The UAV conducts close-range photogrammetry of the bridge according to the planned flight routes. After the platform obtains the close-range photography photos, it uses the high-performance edge server for the second 3D modeling to obtain a refined real-scene 3D model of the bridge.

[0046] Further, the step S100 includes:

[0047] S110. After the inspection vehicle arrives at the target bridge, by operating the user terminal device, the UAV automatic inspection and data processing platform first controls the UAV to take off from the nest, and in the flight process, it real-time controls the flight attitude, position, and speed of the UAV;

[0048] S120. Send the initial inspection task of the bridge to the UAV through the UAV automatic inspection and data processing platform, and conduct oblique photogrammetry on the bridge. In the initial inspection task, the inspection area of the UAV, the fixed flight altitude of the UAV, the flight route, and the image overlap degree for photographing the bridge are set;

[0049] S130. The UAV automatic inspection and data processing platform generates a KML file that can be used for the UAV to execute according to the set parameters, connects to the UAV and sends the KML file, and controls the UAV to execute the initial inspection task;

[0050] S140. The UAV executes the initial inspection task, obtains the bridge image data. After completing the initial inspection task, the UAV returns and lands on the nest, and then sends the bridge image data to the UAV automatic inspection and data processing platform, and the UAV automatic inspection and data processing platform processes the bridge image data.

[0051] In a specific embodiment, after the inspection personnel drive the inspection vehicle to the target bridge, they only need to operate the user terminal on the inspection vehicle. The user terminal is installed with an unmanned aerial vehicle (UAV) automatic inspection and data processing platform. The platform first controls the UAV to take off from the hangar and real-time controls the flight attitude, position, and speed of the UAV during the flight. Then, the platform issues the initial inspection task of the bridge to the UAV, that is, to carry out oblique photogrammetry on the bridge. In the initial inspection task, the inspection area of the UAV, the fixed flight altitude of the UAV, the flight route, and the image overlap degree when taking pictures of the bridge are set. The platform generates a KML file that can be used for the UAV to execute according to the set parameters, and finally connects to the UAV and imports the KML file to let the UAV execute the inspection task. After the UAV completes the inspection task, it lands on the UAV hangar, and then uploads the bridge inspection data to the platform, and the platform is responsible for processing the data.

[0052] The air-ground integrated bridge disease monitoring system mainly includes two major parts. One is the UAV hangar and the UAV inside it, and the other is the ground station equipment. Among them, the UAV includes an obstacle avoidance module, a positioning module, a pan-tilt camera, a top view camera, and a control module. Each module coordinates and cooperates to achieve real-time control of the flight attitude, position, and speed of the UAV, complete the route task issued by the UAV automatic inspection and data processing platform, and realize the inspection work of the bridge. The UAV hangar is mainly used for the automatic takeoff and landing of the UAV. The UAV can also automatically charge through the charging interface of the hangar to improve the endurance of the UAV outdoors. The hangar is equipped with data transmission and storage functions and can transmit the data collected by the UAV back to the UAV automatic inspection and data processing platform.

[0053] The ground station equipment includes user terminal equipment and a high-performance edge server. Among them, the user terminal equipment and the high-performance edge server are fixedly installed on the inspection vehicle. The user terminal equipment is used to run the UAV automatic inspection and data processing platform. The platform can plan the flight route of the UAV, issue the route task, control the flight process and actions of the UAV, and can also perform data exchange and task offloading with the high-performance edge server. For example, it offloads computationally intensive tasks such as 3D modeling to the high-performance edge server, and the server reconstructs the bridge photos obtained by oblique photography into a 3D real scene model of the bridge. In addition, the functions of the platform also include: monomerization and splitting of the 3D model, bridge disease target detection and semantic segmentation, 3D model viewing and annotation,

[0054] association and binding of disease information, storage of disease information, and generation of disease reports.

[0055] The high-performance edge server in the ground station equipment is mainly used to process scenarios that strongly rely on computing power, such as 3D modeling. The UAV automatic inspection and data processing platform transmits photo data to the high-performance edge server, uses the high computing power of the server to perform rapid modeling to obtain a preliminary 3D model of the bridge, and the server uploads the preliminary 3D model file back to the platform. The platform performs monomerization splitting on the overall bridge model, automatically designs corresponding planned flight routes for each monomerized component, and the UAV conducts close-range photogrammetry of the bridge according to the planned flight routes. After the platform obtains the close-range photography photos, it uses the high-performance edge server to perform a second 3D modeling to obtain a refined real-scene 3D model of the bridge.

[0056] After the inspection personnel drive the inspection vehicle to the target bridge, they only need to operate the user terminal on the inspection vehicle. The user terminal is installed with the UAV automatic inspection and data processing platform. The platform first controls the UAV to take off from the nest and real-time controls the flight attitude, position, and speed of the UAV during the flight. Then, the platform issues the initial inspection task of the bridge to the UAV, that is, to conduct oblique photogrammetry on the bridge. In the initial inspection task, the inspection area of the UAV, the fixed flight altitude of the UAV, the flight route, the camera tilt angle, and the image overlap degree when shooting the bridge are set. The platform generates a KML file that can be used by the UAV to execute according to the set parameters, and finally connects to the UAV and imports the KML file to let the UAV execute the inspection task. After the UAV completes the inspection task, it uploads the bridge inspection data to the platform, and the platform is responsible for processing the data.

[0057] Refer to Figure 3 and Figure 4 , since oblique photogrammetry is only used to obtain a preliminary 3D model of the bridge, the fixed flight altitude is generally set to 50 - 100 meters to avoid potential flight safety problems (such as collision risks) that may be caused by too low altitude. The flight route generally adopts two flight modes: "cross flight" and "circular flight". In "cross flight", the bridge is photographed from five different oblique perspectives: front, back, left, right, and vertical to obtain high-resolution images of the ground from different perspectives. When photographing obliquely forward, backward, left, and right, the camera tilt angle is 45°. In "circular flight", the flight route is set as a circle with a certain radius, and 24 shooting stations are evenly distributed on the circle. The camera tilt angle is 45°. The hot task is designed for the UAV to fly around a certain area (a certain area on the bridge as the point of interest) with a specified flight radius. By setting parameters such as the coordinates of the point of interest, the flight altitude, and the flight radius of the UAV, the UAV is controlled to fly around a certain area. The range of the forward overlap rate in the two flight modes of "cross flight" and "circular flight" is set to 70 - 80%, and the range of the side overlap rate is set to 60 - 70%.

[0058] Furthermore, the step S200 includes:

[0059] S210. The UAV forwards the obtained bridge image data to the in-vehicle high-performance edge server through the UAV automatic inspection and data processing platform for 3D modeling.

[0060] S220. The in-vehicle high-performance edge server obtains a preliminary 3D model of the bridge through steps of preprocessing bridge image data, aerial triangulation, image dense matching, constructing a Triangulated Irregular Network (TIN) model, and automatic texture slicing and mapping for the bridge image data, and sends the preliminary 3D model of the bridge to the UAV automatic inspection and data processing platform.

[0061] Furthermore, in step S220, the Delaunay triangulation algorithm is used to construct the Triangulated Irregular Network (TIN) model. The Delaunay triangulation algorithm generates triangles by connecting the points in the bridge point cloud, maximizing the interior angles of the triangles, thereby avoiding long and thin triangles and improving the quality of the Triangulated Irregular Network (TIN) model. The specific process is as follows:

[0062] First, perform an initialization operation. Select a certain number of feature points from the point cloud. The feature points need to be evenly distributed on the surface of the bridge. Then determine the boundary of the point cloud and construct an initial outer convex polygon according to the shape of the bridge as the basis for Delaunay triangulation. Gradually add the feature points to the existing triangular mesh, ensuring that the Delaunay criterion is met, that is, no other points are contained within the circumcircle of any triangle. After triangulation, boundary adjustment is required. Adjust the triangles that are too long and thin or do not conform to the actual bridge features to increase the uniformity and stability of the mesh.

[0063] After constructing the initial Triangulated Irregular Network (TIN) model of the bridge, it is necessary to optimize the Triangulated Irregular Network (TIN) model to better adapt to the actual shape and details of the bridge. First, perform smoothing to reduce the high-frequency noise in the Triangulated Irregular Network (TIN) model and make the surface smoother, while maintaining the details of the important structures of the bridge. Then, according to the local geometric complexity of the bridge, locally refine the Triangulated Irregular Network (TIN) model. For complex parts such as bridge piers and beam bodies, perform mesh refinement to represent the details at a higher resolution. For flat areas, keep larger triangles to reduce the computational cost. Finally, remove the redundant points that do not significantly affect the model accuracy to simplify the mesh structure of the bridge and improve the computational efficiency.

[0064] In a specific embodiment, after the drone completes the bridge inspection task, that is, after completing the bridge oblique photogrammetry task, the data such as the obtained bridge image photos are uploaded to the drone automatic inspection and data processing platform, and the platform distributes the image photo data to the in-vehicle high-performance edge server for 3D modeling. The construction from photos to a 3D model is achieved through steps such as image preprocessing, aerial triangulation, image dense matching, constructing a TIN model, and automatic texture slicing mapping. All these steps are completed on the in-vehicle high-performance edge server. After the modeling is completed, the server uploads the obtained approximate 3D model of the bridge to the platform. The step of constructing the bridge 3D model from bridge photos needs to be completed on the ContextCapture software. After deploying the Context Capture software on the server, first, perform image preprocessing operations such as removing borders, distortion correction, image tiling, and image color homogenization to improve the efficiency and accuracy of subsequent processing. After completing the image preprocessing, import the images and POS data (positioning system data) for aerial triangulation. The POS information comes from the GNSS module and IMU sensor of the drone, recording the spatial position and attitude information of the camera at the moment of shooting. The SIFT feature matching algorithm is used to extract key feature points from each image. According to the extracted feature points, match the same points in adjacent images to generate the relative position relationship between images, and combine the POS data to convert the image coordinates into geographic coordinates. At this time, the position, size, and shape of the model are basically determined. However, the point cloud generated at this time is relatively sparse, and image dense matching is still required to obtain a dense point cloud. Based on the previously calculated exterior orientation elements of the images, use the MVS (Multi-View Stereo matching technology) to match each pixel in the image to generate a dense 3D point cloud. The MVS technology finds the same pixel points in different images and calculates the 3D spatial positions of these pixel points by combining camera parameters. Calculate the depth map (the distance from each pixel to the camera) for each image, and generate a dense 3D point cloud by merging multiple depth maps. After completing the image dense matching, it is necessary to convert the dense point cloud data into a triangular mesh and construct a TIN (Triangulated Irregular Network) model. After obtaining the dense 3D point cloud, noise filtering and point cloud simplification processing are usually performed to reduce redundant data and improve processing efficiency. Based on the simplified point cloud, use algorithms such as Delaunay triangulation to generate a TIN model. This process converts the point cloud data into a surface model composed of countless triangular patches, and the vertices of these triangular patches correspond to points in 3D space. To generate a smooth 3D surface, the mesh is usually optimized, including removing abnormal triangles and smoothing the mesh surface. The TIN model itself is just a mesh composed of geometry and does not contain texture information. Through automatic texture slicing mapping, the color information of the image can be mapped onto the mesh surface to generate a realistic 3D model. First, extract the color and detail information from the captured images and map it onto the surface of the triangular mesh.This process requires matching the pixel positions in the images with the three-dimensional coordinates of the grid vertices. Since the model is usually generated from multiple images, it is necessary to slice the texture information of each image, select the most suitable image area for splicing and covering, and avoid generating texture gaps or overlapping areas. Optimize the spliced texture to ensure texture consistency and eliminate color differences and seams between different images. The construction of the bridge three-dimensional model from bridge photos is realized through steps such as bridge image preprocessing, aerial triangulation, image dense matching, constructing a TIN model, and automatic texture slicing and mapping.

[0065] Furthermore, the step S300 includes:

[0066] S310. The UAV automatic inspection and data processing platform performs monomerization and splitting on the obtained general bridge three-dimensional model through a monomerization algorithm, and splits the entire bridge into at least towers, main cables, suspenders, stiffening girders, anchorages, bridge decks, and bridge piers;

[0067] S320. According to the structural characteristics of different components, plan the close inspection flight routes and photography parameters for different components. The photography parameters at least include shooting overlap rate, shooting spacing, and shooting angle to ensure obtaining high-quality image data;

[0068] S330. After completing the planning of the close inspection flight routes for each part of the bridge, the UAV conducts a second inspection task to carry out close-range photogrammetry of the bridge, obtain high-resolution images of each part of the bridge, and collect more refined and rich details of the bridge's geometric structure and surface texture, facilitating subsequent detection of diseases in each part of the bridge.

[0069] Specifically, after using the server to complete the initial three-dimensional modeling, the UAV automatic inspection and data processing platform performs monomerization and splitting on the obtained general bridge three-dimensional model. The platform automatically splits the entire bridge into components such as towers, main cables, suspenders, stiffening girders, anchorages, bridge decks, and bridge piers through a monomerization algorithm, and plans the close inspection flight routes for different components according to the structural characteristics of different components.

[0070] The monomerization algorithm used in the present invention is RANSAC (Random Sample Consensus algorithm), which performs automated morphological-based structure extraction on the bridge point cloud model. First, downsample and denoise the bridge point cloud, and then use the RANSAC algorithm to fit geometric forms such as planes, cylinders, and lines. Use the plane model to fit the surfaces of the stiffening girder, bridge deck, or bridge pier, use the cylinder model to fit structures with columnar forms such as towers and suspenders, and use the line model to fit slender structures such as the main cable. After extracting each component, the RANSAC algorithm can be continuously applied to the remaining point cloud to extract other components until all the main structures are extracted. Finally, the entire bridge is automatically split into components such as towers, main cables, suspenders, stiffening girders, anchorages, bridge decks, and bridge piers.

[0071] For components such as main cables, hangers, and anchors, image collection is carried out in a way that they are facing their facade structures. Symmetrical routes are set on both sides of the main cables, hangers, and anchors, and shooting points are set at the top and bottom of each part to obtain comprehensive information. For parts such as stiffening beams, symmetrical routes parallel to the stiffening beams are set at the top and bottom for image collection. For columnar parts such as cable towers and piers, a close-to-circling flight route is adopted, and spiral flights are performed around cable towers, piers, etc., to comprehensively shoot cable towers, piers, etc. For shooting of bridge decks, a route for the drone to fly in a straight line along the bridge deck can be planned, and at the same time, a certain angle of oblique shooting is set on both sides of the bridge deck to obtain the pavement and structural conditions of the bridge deck. At this point, the drone route planning for close-to-close photogrammetry of bridges is completed.

[0072] After completing the targeted route design for each part of the bridge, the drone conducts a second inspection, that is, close-up photogrammetry of the bridge to obtain high-resolution images of each part of the bridge, collect more detailed and richer bridge geometry and surface texture details, and facilitate the subsequent detection of defects in various parts of the bridge. In the design of the close-up photography route mission of the bridge, in addition to planning the route, it is also necessary to set some key photography parameters, such as shooting overlap rate, shooting spacing, shooting angle, etc., to ensure the acquisition of high-quality image data.

[0073] For the main cables, suspenders and other parts that are relatively slender and have unclear features, the overlap rate is generally set between 80% and 90% to ensure that there are enough overlapping areas between the images. The shooting interval needs to be determined according to the shooting angle of the drone and the length of the main cable or suspender. The drone is usually shot at a vertical angle to the main cable or suspender to ensure that the shape and details of the main cable can be clearly captured. For parts such as stiffening beams or bridge decks, since the stiffening beams are long and the bridge decks are wide, the overlap rate can be appropriately reduced to improve shooting efficiency. The overlap rate is generally set between 60% and 75%. The drone is usually shot at a vertical angle to the stiffening beams and bridge decks to show the full view and structural details of the stiffening beams. It is also necessary to ensure that details such as the bridge deck pavement and marking lines can be clearly seen. For important parts such as cable towers, anchors, and bridge piers with complex structures but obvious features, the overlap rate is generally set between 70% and 85%. Generally, the circumferential flight method is used to shoot the cable towers, anchors, bridge piers, etc. from different angles. At this point, the drone has completed the close-up photogrammetry of the bridge.

[0074] Further, refer to Figure 5 and Figure 6 In step S400, the UAV automatic inspection and data processing platform performs disease target detection and disease semantic segmentation on the single high-resolution image.

[0075] Use the YOLO V5 object detection model to perform bridge disease object detection and obtain disease images containing disease objects;

[0076] Among them, the YOLO V5 object detection model includes YOLO V5s, YOLO V5m, YOLO V5l, and YOLO V5x. The network structures of YOLOV5s, YOLO V5m, YOLO V5l, and YOLO V5x are the same, but the network depth factor width_multiple and the network width factor depth_multiple need to be changed according to the accuracy requirements and detection time requirements of the task. Since bridge diseases such as cracks, spalling, and corrosion usually have small targets and complex structures, the YOLO V5 object detection model needs to have high accuracy and the ability to identify small targets.

[0077] Specifically, the present invention uses bridge apparent disease images obtained from the National Public Science Data Center of Basic Disciplines for annotation to form a detection data set. The data set is divided into a training set, a validation set, and a test set. After grouping, basic parameters are set to train the data set, and evaluation indicators are obtained through the validation set. Using the trained model, the high-resolution images of each part of the bridge obtained after close-range photogrammetry of the bridge are used as a new validation set to identify and classify bridge diseases, and the whole process of the object detection algorithm to identify and locate bridge apparent diseases is completed.

[0078] Furthermore, a semantic segmentation algorithm is used to process the disease images. Since a disease image may have multiple types of diseases, and each type of disease may exist in multiple regions, the semantic segmentation algorithm is used to extract information for each disease in each region; for diseases such as spalling, exposed reinforcement, collapse, corrosion, or cracks, the semantic segmentation algorithm is used to identify the region of the disease and accurately locate the edge, shape, and size of the disease; through the semantic segmentation algorithm, diseases such as spalling, exposed reinforcement, collapse, corrosion of suspenders and anchors are segmented from the whole bridge, and then the area of the disease is extracted. For diseases such as cracks and fissures, the length of the crack needs to be calculated. Finally, the area or length of each disease is obtained, and the quantitative analysis of the disease is finally realized;

[0079] Refer to Figure 7, the semantic segmentation algorithm uses the DeepLabv3+ algorithm to segment bridge diseases. The DeepLabv3+ algorithm includes an encoder module with dilated convolutions, an Atrous Spatial Pyramid Pooling (ASPP), and a decoder module. In the figure, the input on the left is a picture of a bridge crack. The upper half is the Encoder, that is, the encoder, which is mainly responsible for extracting high-level features of the input image. The lower half is the Decoder, that is, the decoder. The decoder uses the high-level features and low-level features extracted by the encoder to generate the final semantic segmentation result (prediction map). Among them, the Encoder (encoder) contains a DCNN (Deep Convolutional Neural Network) module. Dilated convolutions are used in the DCNN (Deep Convolutional Neural Network), which can expand the receptive field without increasing the computational complexity and help capture a larger range of context information. Inside the DCNN, there are multiple parallel convolutional operations and image pooling operations, including 1×1Conv, that is, a 1x1 convolutional kernel for feature dimensionality reduction, and 3×3 convolutional kernels with dilation rates of 6, 12, and 18 respectively. Then, global Image Pooling is performed on the entire feature map to capture global context information. Finally, these multi-scale features are fused together through concatenation. After the multi-scale feature fusion, 1×1 convolutions are used for further dimensionality reduction to reduce the number of channels in the feature map.

[0080] Figure 7 The lower half of the Decoder (decoder) is responsible for combining the high-level features extracted by the encoder with the low-level features, gradually restoring the spatial details of the image, and generating the final prediction result. First, Low-LevelFeatures, that is, low-level features (mainly details such as edges and textures) are obtained from the encoder, and then these low-level features are further processed through 1×1Conv convolutions to reduce the number of channels and noise interference. Next, the high-level features output by the encoder are upsampled by 4, that is, the spatial resolution of the feature map is enlarged by 4 times. At this time, the resolution of the feature map gradually returns to close to the size of the input image. Then, the upsampled high-level features and the low-level features are concatenated (feature concatenation), and 3×3Conv (convolution) is used to further fuse the feature information to extract the final prediction feature map. Finally, Upsample by 4 (upsampling with a 4-fold increase in resolution) is performed again to restore the resolution of the feature map to the size of the input image, generating the final semantic segmentation result of the bridge disease.

[0081] The ASPP (Atrous Spatial Pyramid Pooling) captures multi-scale global and local context information in the disease image through multiple different dilation rates. For bridge disease segmentation, the Atrous Spatial Pyramid Pooling ASPP is used to understand the different shapes, sizes, and spatial distributions of the diseases;

[0082] The decoder module is used to gradually restore the deep feature information extracted by the encoder to the resolution of the original image and generate the final pixel-level segmentation result. The decoder module combines the low-level features and high-level features of the encoder module to restore the details of the disease image while maintaining the accuracy of disease segmentation.

[0083] In a specific embodiment, after the UAV completes the close-range photogrammetry work of the bridge, the high-resolution photos of each part of the bridge obtained have two uses. One is for the object detection and semantic segmentation of bridge diseases, and the other is for the construction of a refined 3D model of the bridge.

[0084] The common disease types of bridges can be divided into: apparent diseases such as cracks, spalling, exposed reinforcement, collapse, corrosion of suspenders and anchors, etc. The present invention uses the YOLO V5 object detection model to detect bridge diseases. YOLO V5 includes YOLO V5s, YOLOV5m, YOLO V5l, and YOLO V5x, and their network structures are the same, but the network depth factor width_multiple and the network width factor depth_multiple need to be changed according to the accuracy requirements of the task and the detection time requirements. The present invention uses the YOLO V5x model because bridge diseases such as cracks, spalling, and corrosion usually have small targets and complex structures. Therefore, the detection model needs to have high accuracy and the ability to identify small targets. The YOLO V5x model has the largest scale, the highest accuracy, and strong ability to identify small targets, and is most suitable for scenarios with high accuracy requirements such as bridge disease detection.

[0085] Refer to Figure 5, the structure of YOLO v5 is generally divided into three major parts: Backbone, Head, and Detect. The Backbone mainly extracts bridge disease features. The Head fuses bridge disease feature maps of different scales. The Detect outputs the final disease target detection results. The Backbone structure is mainly composed of the ConvBNSiLU module, the C3 Module, and the SPPF (Spatial Pyramid Pooling-Fast), that is, the fast spatial pyramid pooling module. Among them, the ConvBNSiLU module is composed of Conv (convolution), BN (batch normalization), and the SiLU activation function, and its main purpose is to extract the disease features in the image. Specific parameters are recorded below the ConvBNSiLU module. For example, K3, S1, P1, C256 represent: the convolution kernel size is 3×3, the stride is 1, the padding is 1, and the number of output channels is 256. The Backbone outputs three feature maps of different scales: P3, P4, and P5, which are the feature maps after 1 / 8 downsampling, the feature maps after 1 / 16 downsampling, and the feature maps after 1 / 32 downsampling respectively. These feature maps will be sent to the Head for further fusion.

[0086] The role of the Head is to upsample and fuse feature maps of different scales so that the detector can process targets of different sizes. First, perform the Upsample operation on the feature map, usually using the nearest neighbor interpolation or transposed convolution (deconvolution) technology to increase the resolution of the feature map. Then, concatenate the upsampled high-level semantic features with the low-level high-resolution features to combine the advantages of both. The concatenated features contain high-level semantic information (deep abstract features) and low-level spatial details (high-resolution features). After concatenation, the C3 Module further fuses the features to extract richer context information.

[0087] The Detect module is the last step of the YOLO regression detection part, which includes classification and regression. Both of them first perform convolution for the corresponding functions through a specific convolution, and finally complete the classification and regression tasks by predicting the loss between the object and the target. After the feature fusion is completed, the output feature map is sent to the detection head for the final target detection. The detection head will output the category, bounding box position, and confidence of the target. Since the network structure outputs three feature maps of different scales, the detector can process small targets, medium targets, and large targets, that is, the detector can detect bridge diseases of different sizes.

[0088] Refer toFigure 6 The C3 module is an important part of the YOLO v5 network. It draws on the design concept of the residual network and combines the feature fusion of the main path and the bypass path. The C3 module is mainly composed of multiple Bottleneck structures and bypass connections. The main path is used for efficient feature extraction, and the bypass connection helps with information transmission. Finally, the output feature map is fused through Concat (feature concatenation) and 1×1 convolution. Among them, the Bottleneck is used for efficient feature extraction. It usually first uses 1x1 convolution to reduce the channel dimension (dimensionality reduction), reducing the computational amount, then uses 3x3 convolution to extract feature information, and finally restores the number of channels through 1x1 convolution (dimensionality increase). The convolution operation in the C3 module uses a 3×3 convolution kernel, the activation function is SiLU, and at the same time, it is combined with the BN (batch normalization) layer to improve the stability and generalization performance of the model. The main role of the C3 module is to enhance the feature extraction ability of the network, which is particularly suitable for object detection tasks. Its main role is to increase the depth and receptive field of the network and improve the ability to extract bridge disease features.

[0089] Refer to Figure 6 The SPPF (Spatial Pyramid Pooling-Fast), that is, the fast spatial pyramid pooling module, realizes the feature fusion of the same feature map at different pooling scales by cascading three MaxPooling operations with a kernel size of 5×5, a stride of 1, and a padding of 2, and stitches together the multi-scale feature maps through the Concat (feature concatenation) operation. Finally, the stitched feature map passes through the ConvBNSiLU part, that is, the 1×1 Conv (convolution) layer + BN (batch normalization) layer + SiLU activation function. The SPPF (fast spatial pyramid pooling) module can achieve multi-scale feature fusion of bridge diseases, adapt to disease targets of different sizes, improve the detection effect of small and large targets, and show higher speed and performance when processing large-scale high-resolution disease data.

[0090] The preprocessed bridge disease image is input into the YOLO V5 network. The input image is usually an RGB image with 3 channels, and its shape is (3, 640, 640), that is, 3 represents the number of channels, and 640 represents the height and width of the image. The output of YOLO V5 is 3 feature tensors, containing prediction information about bounding boxes, classes, and confidence, in the form of (80, 80, 3, num_classes + 5), (40, 40, 3, num_classes + 5), and (20, 20, 3, num_classes + 5), which are used to detect targets of different scales respectively.

[0091] Cracks and minor diseases on bridges are usually relatively small with rich details. Therefore, the (80x80) feature map is more suitable for detecting this type of small targets. This is because the high-resolution feature map can better retain the details of the diseases and help the model detect tiny features. If the scale of the disease is slightly larger but still within the medium range, such as moderate spalling or corrosion, the (40x40) feature map can be used. This level of feature map can capture larger structural information while still maintaining sufficient details. For large-scale diseases, such as large-area concrete spalling or severe corrosion, the (20x20) feature map is more appropriate. This feature map has a lower resolution but a larger receptive field, which is suitable for detecting diseases in a larger range.

[0092] Before carrying out the disease target detection work, it is first necessary to annotate the bridge apparent disease images obtained from the National Center for Public Science Data of Basic Disciplines. However, due to the limited number of bridge apparent disease images, in order to further improve the accuracy of the target detection model, it is necessary to perform data augmentation on the bridge apparent disease images, and expand the dataset size by operating on the original images such as image rotation and random cropping.

[0093] After obtaining a sufficient number of disease image data, label the image disease types as cracks, spalling, exposed reinforcement, collapse, corrosion of suspenders and anchors into five categories to form a detection dataset. Randomly divide the dataset into a training set, a validation set, and a test set according to the ratio of 8:1:1. The training set is used to train the model to obtain weight parameters, the validation set is used to adjust the model parameters, and the test set is used to evaluate the performance of the model. After grouping, set the basic parameters to train the dataset and obtain the evaluation index through the validation set. Using the trained model, take the high-resolution images of each part of the bridge obtained by close-range photogrammetry of the unmanned aerial vehicle as a new validation set to identify and classify the bridge diseases, and complete the process of the target detection algorithm to identify and locate the bridge apparent diseases.

[0094] After completing the disease detection and classification of the bridge, it is also necessary to quantify the degree of the disease. During the bridge detection process, the semantic segmentation algorithm can effectively identify the target categories and accurately segment the diseases, thus achieving the purpose of quantifying the diseases. The present invention uses the DeepLabv3+ algorithm to segment the bridge diseases. DeepLabv3+ is a semantic segmentation network based on deep learning. It adds an additional network structure on the basis of the original DeepLab series, improving the segmentation accuracy and efficiency, and is more suitable for scenarios such as bridge disease segmentation that pursue high-precision segmentation. After segmenting the diseases using the DeepLabv3+ algorithm, it is necessary to extract information from it to achieve the purpose of quantifying the diseases. Since there may be multiple types of diseases in an image, and each type of disease may have multiple regions, it is necessary to extract information for each disease and each region. After segmenting diseases such as spalling, exposed reinforcement, collapse, corrosion of suspenders and anchors from the bridge image, then extract the area of the disease. For diseases such as cracks, it is necessary to calculate the length of the cracks. Finally, obtain the area or length of each disease to achieve the quantitative analysis of the diseases.

[0095] Further, in the step S500,

[0096] Convert the bridge photos into a 3D model through steps such as image preprocessing, aerial triangulation, image dense matching, constructing a triangulated irregular network model TIN, and automatic texture slice mapping;

[0097] The resolution of the single high-resolution image obtained by close-range photogrammetry is higher than that of the bridge image data, and it can collect more delicate and rich bridge geometric structures and surface texture details. Finally, the generated 3D model has higher geometric accuracy and clearer surface texture details.

[0098] Specifically, the high-resolution photos of each part of the bridge obtained by the drone's close-range photogrammetry are also used for the construction of the refined 3D model of the bridge. The second modeling process is the same as the method of the initial 3D modeling in S3, that is, convert the bridge photos into a 3D model through steps such as image preprocessing, aerial triangulation, image dense matching, constructing a TIN model, and automatic texture slice mapping. The difference is that the photos obtained by close-range photogrammetry have higher resolution, can collect more delicate and rich bridge geometric structures and surface texture details, and finally the generated 3D model has higher geometric accuracy and clearer surface texture details.

[0099] After the second three-dimensional modeling, a refined three-dimensional model of the bridge is generated. The refined three-dimensional model of the bridge constructed using bridge images containing POS information has world coordinates. Through coordinate transformation, each point on the three-dimensional model can be transformed into coordinates in geographic coordinate systems such as WGS84 and CGCS2000. In the present invention, each point on the refined three-dimensional model of the bridge is transformed into coordinates in the WGS84 coordinate system, and the specific location of each disease, such as longitude, latitude, and elevation, is recorded to facilitate the positioning and analysis of the diseases in the three-dimensional model.

[0100] Among them, the process of coordinate transformation is called photogrammetric coordinate recovery, which involves calculating the transformation from the camera coordinate system to the three-dimensional model coordinate system of the object. When taking photos, the POS data (position and attitude) of the drone or camera records the coordinates and orientation of the camera at the time of shooting. The POS data usually includes: the longitude, latitude, and elevation of the camera, as well as the orientation angles of the camera, including the pitch angle, yaw angle, and roll angle. Through multi-view photos, the photogrammetric algorithm can calculate the coordinates of the three-dimensional points of the object being photographed. This process is based on the principle of triangulation and relies on the projection positions of the same feature points in different photos under multiple views. The common feature points are extracted from multiple photos through the SIFT (Scale-Invariant Feature Transform) feature matching algorithm, and their corresponding relationships in different photos are found. According to the POS data of the camera and the matched feature points, the bundle adjustment method is used to solve the position and orientation of each photo and the coordinates of the three-dimensional points. This step uses the internal parameters (focal length, distortion parameters) and external parameters (the attitude and position of the camera) of the camera to adjust and optimize the accuracy of the three-dimensional reconstruction. Through the result of the bundle adjustment, each pixel point in each photo can be mapped onto the three-dimensional model of the object. After multi-view photogrammetry, the feature points of all images will correspond to the three-dimensional coordinate system of the object. Usually, the coordinate system of the model is a local coordinate system, but if POS data is used during the photogrammetric process, the model can be aligned to the actual global coordinate system (such as WGS84 or CGCS2000). Finally, the three-dimensional model (such as point cloud or mesh) obtained through photogrammetry can be output in a format containing geographic coordinates, and each vertex in the point cloud or model will have information such as longitude, latitude, and elevation.

[0101] Further, in the step S600,

[0102] The refined three-dimensional model of the bridge constructed using bridge images containing POS information contains world coordinates. Through coordinate transformation, each point on the refined three-dimensional model of the bridge is transformed into coordinates in geographic coordinate systems such as WGS84 and CGCS2000;

[0103] Among them, the coordinate transformation includes the transformation from the camera coordinate system of the drone to the image plane of the bridge, the transformation from the camera coordinate system to the world coordinate system, and the transformation from the world coordinate system to the WGS84 coordinate system. Specifically, it includes the following steps:

[0104] S610. The conversion from the UAV camera coordinate system to the bridge image plane, that is, the conversion from pixel coordinates to camera coordinates, from the 2D points on the pixel coordinate image plane to the 3D points in the camera coordinate system, is mainly based on the camera internal parameter matrix K. The camera internal parameters describe the focal length, principal point, and distortion information of the camera, etc., and are represented as a 3x3 matrix:

[0105]

[0106] where f x and f y are the focal lengths in the horizontal and vertical directions respectively, and c x and c y constitute the principal point of the bridge image, which is usually the center of the bridge image. Given a pixel coordinate [u, v] on the bridge image, it is converted into the coordinate (x c , y c ) in the normalized camera coordinate system through the internal parameter matrix:

[0107]

[0108] where K ―1 is the inverse matrix of the UAV camera internal parameter matrix. This conversion maps the points on the 2D image plane to the camera coordinate system. In the actual scenario, the depth Z c of the three-dimensional point cannot be directly obtained from a single image and needs to be calculated through multi-view photos;

[0109] S620. The conversion from the UAV camera coordinate system to the world coordinate system. In order to convert the points in the camera coordinate system to the world coordinate system, the camera external parameter matrix [R|T] is used. The camera external parameter matrix [R|T] describes the position and orientation of the camera relative to the world coordinate system. The external parameter matrix consists of a rotation matrix R and a translation vector T:

[0110]

[0111] where (X w , Y w , Z w ) are the coordinates of the point in the world coordinate system. R is a 3x3 rotation matrix representing the attitude of the UAV camera, that is, the yaw angle, pitch angle, and roll angle. T is a 3x1 translation vector representing the position of the camera, that is, the longitude, latitude, and elevation in the POS data, which converts the points in the camera coordinate system to the world coordinate system;

[0112] S630. Conversion from the world coordinate system of a 3D model to the WGS84 coordinate system. The world coordinate system is usually a local 3D coordinate system. To convert the coordinates to the WGS84 coordinate system on the Earth, a geodetic coordinate transformation is required. The WGS84 coordinate system uses longitude and latitude (φ, λ) and elevation (h) to represent. To convert the 3D coordinates of the 3D model in the world coordinate system to ECEF (Earth-Centered, Earth-Fixed) coordinates, the ECEF coordinate system is a 3D rectangular coordinate system with the Earth's centroid as the origin. The formula is as follows:

[0113] X = (N + h)·cos(φ)·cos(λ),

[0114] Y = (N + h)·cos(φ)·sin(λ),

[0115] Z = ((1 - e 2 )N + h)·sin(φ),

[0116] where N is the radius of curvature, a is the equatorial radius of the Earth, and e is the eccentricity of the Earth; based on the ECEF coordinates (X, Y, Z), through the inverse formula, it can be converted to the longitude, latitude, and elevation under WGS84. The inverse formula is as follows:

[0117] Calculate longitude λ:

[0118] λ = arctan 2(Y, X)

[0119] Obtained from the ratio of X and Y, using the arctangent function. Then calculate the initial latitude φ0:

[0120] φ0 = atan2(Z, r·(1 - e 2 ))

[0121] where, is the projection distance on the equatorial plane. Iteratively calculate the latitude φ:

[0122] φ = atan2(Z + e 2 ·N·sin(φ0), r)

[0123] where, is the radius of curvature. Finally, calculate the elevation h:

[0124]

[0125] The elevation h is the vertical distance relative to the ellipsoid surface; the 3D coordinates (X, Y, Z) of each point on the refined 3D model of the bridge are converted to the geodetic coordinates (λ, φ, h) under the WGS84 coordinate system, that is, longitude, latitude, and elevation.

[0126] In a specific embodiment, the disease identification and segmentation results are fused with the 3D model information. Based on the results of bridge disease identification, classification, and segmentation in S600, the types, areas, or lengths of bridge diseases can be obtained. Based on the refined 3D model of the bridge obtained through 3D modeling in S6, the bridge can be comprehensively viewed at any time and place through the UAV automatic inspection and data processing platform, and the specific geographical coordinates of the diseases can be understood. By mapping the disease images to the disease positions in the 3D model, the detected positions, boundary contours, and other information (such as types, areas, etc.) of each disease can be bound to the point cloud or mesh vertices of the 3D model, and disease information is added as vertex attributes to the vertex data structure of the 3D model. For example, each vertex can store information such as disease type, area, and length. In this way, when browsing or analyzing the 3D model, by clicking on the disease position on the 3D model, the user can query the disease-related data. The types, areas, lengths, and geographical coordinates of each disease of the bridge are stored in the background MySQL database of the UAV automatic inspection and data processing platform. The platform automatically generates a corresponding disease detection report for each disease. Combining the timestamp information of bridge photo shooting and model construction, the user can review and track the diseases of the bridge in the future and evaluate the development and changes of the diseases.

[0127] Specifically, based on the results of bridge disease identification, classification, and segmentation, the types, areas, or lengths of bridge diseases can be obtained. Based on the refined 3D model of the bridge obtained through 3D modeling, the bridge can be comprehensively viewed at any time and place through the UAV automatic inspection and data processing platform, and the specific geographical coordinates of the diseases can be understood. By mapping the disease images to the disease positions in the 3D model, the detected positions, boundary contours, and other information (such as types, areas, etc.) of each disease can be bound to the point cloud or mesh vertices of the 3D model, and disease information is added as vertex attributes to the vertex data structure of the 3D model. For example, each vertex can store information such as disease type, area, and length. In this way, when browsing or analyzing the 3D model, by clicking on the disease position on the 3D model, the user can query the disease-related data. The types, areas, lengths, and geographical coordinates of each disease of the bridge are stored in the background MySQL database of the UAV automatic inspection platform. The platform automatically generates a corresponding disease detection report for each disease. The report information includes the bridge name, bridge type, shooting time, disease type, disease size, and disease 3D geographical coordinates. By browsing the disease detection report, the diseases of the bridge can be reviewed and tracked in the future, and the development and changes of the diseases can be evaluated, realizing the electronic and information-based archiving and rapid review of the bridge.

[0128] Further, referring to Figure 8, the present invention also proposes an air-ground integrated bridge disease monitoring system for implementing the air-ground integrated bridge disease monitoring method. The air-ground integrated bridge disease monitoring system is carried on an inspection vehicle, and the air-ground integrated bridge disease monitoring system includes:

[0129] A machine nest, which is electrically connected to the unmanned aerial vehicle (UAV) automatic inspection and data processing platform;

[0130] A UAV, which is arranged in the machine nest. The UAV is charged through the charging interface of the machine nest. The UAV includes a visual obstacle avoidance module, an infrared obstacle avoidance module, a positioning module, a gimbal camera, a top view camera and a control module. The gimbal camera is an oblique photography device. The visual obstacle avoidance module is installed in the front, rear, side and above of the UAV, and the infrared obstacle avoidance module is installed at the bottom of the UAV;

[0131] The UAV automatic inspection and data processing platform, which is used to plan the flight route of the UAV, issue route tasks, control the flight process and actions of the UAV, and perform data processing and data exchange on the data collected by the UAV;

[0132] A vehicle-mounted high-performance edge server, which is used to perform data exchange with the UAV automatic inspection and data processing platform. The vehicle-mounted high-performance edge server is electrically connected to the UAV automatic inspection and data processing platform;

[0133] A user terminal device, which is used to run the UAV automatic inspection and data processing platform. The user terminal device is electrically connected to the UAV automatic inspection and data processing platform.

[0134] In a specific embodiment, the UAV is equipped with a visual obstacle avoidance module and a gimbal camera in the front, a visual obstacle avoidance module, an RTK positioning module and a top view camera above, visual obstacle avoidance modules on the rear and side, an infrared obstacle avoidance module at the bottom, and a conventional positioning module and a UAV flight control module inside. The vehicle for bridge inspection is equipped with a UAV machine nest, a high-performance edge server and a user terminal device. The user terminal device deploys the UAV automatic inspection and data processing platform. The high-performance edge server and the user terminal device that deploys the UAV automatic inspection and data processing platform are collectively referred to as ground station devices. The UAV automatic inspection and data processing platform and the high-performance edge server perform data transmission through a wired Ethernet connection. The UAV automatic inspection and data processing platform communicates with the UAV machine nest or the UAV through a 4G / 5G wireless network, and the UAV and the machine nest also communicate through a 4G / 5G wireless network.

[0135] The drone is equipped with visual obstacle avoidance modules in the front, rear, sides, and above, and an infrared obstacle avoidance module at the bottom. These modules work together to enable the drone to obtain all-round information, allowing the drone to identify and avoid obstacles during flight.

[0136] The positioning module of the drone includes Global Navigation Satellite System (GNSS) positioning, visual-aided positioning, and RTK (Real-Time Kinematic) module. When the drone is in an open and unobstructed environment, RTK and GNSS integrated positioning is usually adopted. When the drone is in places where signals are easily blocked, such as under a bridge, visual-aided positioning is used.

[0137] The drone uses a gimbal camera and a top-view camera to capture images of various parts of the bridge. The gimbal camera is installed in the front of the drone, with a controllable pitching range of -90° to 35° and a controllable rolling range of -45° to 45°. Therefore, usually the drone cannot capture the bottom of the bridge. The top-view camera is installed above the drone and can capture the details of the bottom of the bridge that the gimbal camera cannot reach.

[0138] The drone control module includes sensors such as IMU (Inertial Measurement Unit), Global Positioning System (GNSS), barometer, and magnetometer, which provide attitude information, position, and altitude information. The IMU includes a three-axis gyroscope and a three-axis accelerometer, which are used to monitor and respond to the attitude changes of the drone in real time. The GNSS provides the precise position information of the drone to assist the drone in positioning and hovering. The barometer is used to measure the flight altitude of the drone and assist the IMU to achieve precise hovering. The magnetometer provides information about the direction of the Earth's magnetic field to assist the drone in determining the direction.

[0139] Glossary:

[0140] TIN: Triangulated Irregular Network, which is a method for representing Digital Elevation Model (DEM). It fits the surface of the earth or other irregular surfaces through a series of connected triangles and is often used to construct digital terrain models, especially digital elevation models.

[0141] POS: Position and Orientation System, which is a system used to record the position and orientation information of a camera or sensor at a specific moment and is widely used in applications such as drones, photogrammetry, remote sensing, and navigation. POS data contains the position information and orientation information of the shooting device when taking pictures, and this information is important for the three-dimensional reconstruction, precise positioning, and navigation of multi-view images.

[0142] WGS84: World Geodetic System 1984, the most widely used global geodetic coordinate reference system at present, was developed by the U.S. Department of Defense and is mainly used for the Global Positioning System (GPS). WGS84 defines the shape of the Earth, the gravity field, and a globally unified coordinate reference system, and is widely used in fields such as global navigation, surveying and mapping, and Geographic Information System (GIS).

[0143] CGCS2000: China Geodetic Coordinate System 2000, is China's national geodetic reference system. It is the latest coordinate system released and implemented by the China National Administration of Surveying, Mapping and Geoinformation after the Beijing Geodetic Coordinate System 1954 (BJ54) and the Xi'an Geodetic Coordinate System 1980 (XIAN80), and is widely used in fields such as Geographic Information System (GIS), surveying and mapping, and navigation within China.

[0144] SIFT: Scale-Invariant Feature Transform, is a computer vision algorithm for image feature detection and description, proposed by David Lowe in 1999. The SIFT algorithm extracts local feature points (keypoints) and their descriptors in an image, enabling objects or scenes in the image to be stably recognized and matched under different scales, rotation angles, and lighting conditions. SIFT is widely used in fields such as image stitching, 3D reconstruction, object recognition, and robot navigation.

[0145] MVS: Multi-View Stereo, is a computer vision technology for reconstructing the 3D geometry of a scene or object surface from multiple 2D images taken from different viewpoints. MVS uses multiple images taken from different angles, and by matching feature points, pixels, and texture information between the images, generates a high-precision 3D point cloud or mesh model. This technology has wide applications in fields such as UAV photogrammetry, virtual reality, autonomous driving, and digitalization of cultural heritage.

[0146] Delaunay triangulation: is a triangulation method in computational geometry that constructs a triangular mesh for a set of points, satisfying specific geometric properties. Delaunay triangulation is widely used in fields such as computer graphics, terrain modeling, 3D reconstruction, and finite element analysis. This method is named after the algorithm proposed by the French mathematician Boris Delaunay in 1934.

[0147] RANSAC: Random Sample Consensus, is an iterative method for data fitting and model estimation, widely used in fields such as computer vision, robotics, image processing, and machine learning. RANSAC is particularly suitable for processing datasets containing a large number of outliers (noise), and can effectively extract the valid information in the data without affecting the model quality.

[0148] MySQL: A widely used open-source relational database management system (RDBMS), initially developed by the Swedish company MySQL AB and now maintained and supported by Oracle. It has become one of the most popular database systems globally due to its high performance, reliability, and ease of use, especially in web applications, web services, and cloud computing platforms.

[0149] As mentioned above, these are only the preferred embodiments of the present invention. The present invention is not limited to the above-described embodiments. As long as the same means are used to achieve the technical effects of the present invention, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure. They shall all fall within the scope of protection of the present invention. Within the scope of protection of the present invention, various different modifications and variations may be made to its technical solutions and / or implementation manners.

Claims

1. The air-ground integrated bridge disease monitoring method is applied to the air-ground integrated bridge disease monitoring system, and is characterized in that: The air-ground integrated bridge disease monitoring system includes a machine nest, which is electrically connected to an unmanned aerial vehicle automatic inspection and data processing platform; a unmanned aerial vehicle, which is arranged in the machine nest and charged through a charging interface of the machine nest, and includes a visual obstacle avoidance module, an infrared obstacle avoidance module, a positioning module, a pan-tilt camera, a top-view camera and a control module; an unmanned aerial vehicle automatic inspection and data processing platform, which is used to plan the flight route of the unmanned aerial vehicle, issue route tasks, control the flight process and actions of the unmanned aerial vehicle, and perform data processing and data exchange on the data collected by the unmanned aerial vehicle; The vehicle-mounted high-performance edge server is used to exchange data with the unmanned aerial vehicle automatic inspection and data processing platform, and the vehicle-mounted high-performance edge server is electrically connected to the unmanned aerial vehicle automatic inspection and data processing platform; the user terminal device is used to run the unmanned aerial vehicle automatic inspection and data processing platform, and the user terminal device is electrically connected to the unmanned aerial vehicle automatic inspection and data processing platform. The air-ground integrated bridge disease monitoring method includes the following steps: S100, the inspection vehicle drives to a suitable position of the bridge to be inspected, the UAV automatic inspection and data processing platform controls the UAV to take off from the machine nest, and sends the initial bridge inspection task to the UAV, the UAV obtains bridge image data through the oblique photography device, and after completing the bridge oblique photography measurement task, the UAV lands on the machine nest, and uploads the bridge image data to the UAV automatic inspection and data processing platform through the machine nest; S200, the UAV automatic inspection and data processing platform sends the bridge image data to the vehicle-mounted high-performance edge server for initial three-dimensional modeling to obtain a rough three-dimensional model of the bridge, and the vehicle-mounted high-performance edge server sends the rough three-dimensional model of the bridge to the UAV automatic inspection and data processing platform; S300, the UAV automatic inspection and data processing platform, separates the general three-dimensional model of the bridge into individual parts to obtain multiple individual parts. The UAV automatic inspection and data processing platform plans the close inspection route corresponding to each individual part. The UAV performs close photogrammetry of the bridge based on the close inspection route to obtain individual high-resolution images of each part of the bridge. S400, the UAV sends the single-unit high-resolution image to the UAV automatic inspection and data processing platform, which performs disease target detection and disease semantic segmentation on the single-unit high-resolution image to obtain the shape, size and spatial distribution of each bridge disease; S500, performing a second 3D reconstruction based on close-up photogrammetry and locating defects to obtain a refined 3D model of the bridge; S600, the UAV automatic inspection and data processing platform associates and binds the disease target detection and disease semantic segmentation results with the refined three-dimensional model of the bridge to generate a bridge disease detection report.

2. The air-ground integrated bridge disease monitoring method according to claim 1 is characterized in that: The step S100 includes: S110, after the inspection vehicle arrives at the target bridge, by operating the user terminal device, the UAV automatic inspection and data processing platform first controls the UAV to take off from the machine nest, and controls the flight attitude, position and speed of the UAV in real time during the flight; S120, sending the initial bridge inspection mission to the drone through the drone automatic inspection and data processing platform, carrying out oblique photogrammetry of the bridge, and setting the drone inspection area, the fixed flight altitude of the drone, the flight route, and the overlap of the images taken of the bridge in the initial inspection mission; S130, the UAV automatic inspection and data processing platform generates a KML file that can be used for the UAV to execute according to the set parameters, connects to the UAV and sends the KML file, and controls the UAV to perform the initial inspection task; S140, the UAV performs the initial inspection mission and obtains bridge image data. After completing the initial inspection mission, the UAV returns to land on the machine nest and then sends the bridge image data to the UAV automatic inspection and data processing platform, which processes the bridge image data.

3. The air-ground integrated bridge disease monitoring method according to claim 1 is characterized in that: The step S200 includes: S210, the drone forwards the obtained bridge image data to the vehicle-mounted high-performance edge server through the drone automatic inspection and data processing platform for three-dimensional modeling; S220, the vehicle-mounted high-performance edge server obtains a rough three-dimensional model of the bridge through the steps of bridge image data preprocessing, aerial triangulation, image dense matching, construction of a triangular irregular network model TIN and automatic texture slice mapping, and sends the rough three-dimensional model of the bridge to the drone automatic inspection and data processing platform.

4. The air-ground integrated bridge disease monitoring method according to claim 3 is characterized in that: In step S220, the Delaunay triangulation algorithm is used to construct the triangular irregular network model TIN. The Delaunay triangulation algorithm generates triangles by connecting points in the bridge point cloud to maximize the inner angle of the triangle, thereby avoiding long and thin triangles and improving the quality of the triangular irregular network model TIN. The specific process is as follows: First, an initialization operation is performed to select a certain number of feature points from the point cloud. The feature points must be evenly distributed on the surface of the bridge. Then, the boundary of the point cloud is determined, and an initial outer convex polygon is constructed according to the shape of the bridge as the basis for Delaunay triangulation. The feature points are gradually added to the existing triangular mesh to ensure that the Delaunay criterion is met, that is, the circumscribed circle of any triangle does not contain other points. After the triangulation is completed, boundary adjustment is required to adjust the triangles that are too slender or do not meet the actual bridge characteristics to increase the uniformity and stability of the mesh. After constructing the initial triangular irregular network model TIN of the bridge, the triangular irregular network model TIN needs to be optimized to better adapt to the actual shape and details of the bridge; first, smoothing is performed to reduce the high-frequency noise in the triangular irregular network model TIN to make the surface smoother, while maintaining the details of the important structures of the bridge; then, the triangular irregular network model TIN is locally refined according to the local geometric complexity of the bridge; for complex parts such as piers and beams, mesh refinement is performed to represent details with higher resolution; For flat areas, larger triangles are kept to reduce computational costs. Finally, redundant points that do not significantly affect the accuracy of the model are removed to simplify the mesh structure of the bridge and improve computational efficiency.

5. The air-ground integrated bridge disease monitoring method according to claim 1 is characterized in that: The step S300 includes: S310, the UAV automatic inspection and data processing platform performs monomer splitting on the obtained three-dimensional model of the bridge through a monomer splitting algorithm, and splits the entire bridge into at least towers, main cables, hangers, stiffening beams, anchors, bridge decks and bridge piers; S320, planning close inspection routes and photography parameters for different components according to the structural characteristics of different components, wherein the photography parameters at least include a photography overlap rate, a photography interval, and a photography angle, so as to ensure acquisition of high-quality image data; S330. After completing the close inspection route planning of various parts of the bridge, the UAV will carry out the second inspection mission, conduct close-up photogrammetry of the bridge, obtain high-resolution images of various parts of the bridge, and collect more detailed and richer bridge geometric structure and surface texture details to facilitate subsequent detection of diseases in various parts of the bridge.

6. The air-ground integrated bridge disease monitoring method according to claim 1 is characterized in that: In step S400, the UAV automatic inspection and data processing platform performs disease target detection and disease semantic segmentation on the single high-resolution image. The YOLO V5 target detection model is used to perform bridge damage target detection and obtain a damage image containing the damage target; Among them, the YOLO V5 target detection model includes YOLO V5s, YOLO V5m, YOLO V5l and YOLO V5x. The network structures of YOLOV5s, YOLO V5m, YOLO V5l and YOLO V5x are the same, but the network depth factor width_multiple and the network width factor depth_multiple need to be changed according to the accuracy and detection time requirements of the task. Since bridge diseases such as cracks, spalling, corrosion, etc. usually have smaller targets and complex structures, the YOLO V5 target detection model needs to have higher accuracy and the ability to recognize small targets.

7. The air-ground integrated bridge disease monitoring method according to claim 6 is characterized in that: The semantic segmentation algorithm is used to process the defect image. Since a defect image may contain multiple types of defects, and each type of defect may exist in multiple areas, the semantic segmentation algorithm is used to extract information from each defect in each area. For defects such as spalling, exposed reinforcement, collapse, corrosion or cracks, the semantic segmentation algorithm is used to identify the area of ​​the defect and accurately locate the edge, shape and size of the defect. The semantic segmentation algorithm is used to separate the defects such as spalling, exposed reinforcement, collapse, rust of hangers and anchors from the entire bridge, and then the area of ​​the defect is extracted. For defects such as cracks, the length of the cracks needs to be calculated, and finally the area or length of each defect is obtained, and finally the quantitative analysis of the defects is achieved. The semantic segmentation algorithm uses the DeepLabv3+ algorithm to segment bridge defects. The DeepLabv3+ algorithm includes an encoder module of dilated convolution, a dilated spatial pyramid pooling ASPP and a decoder module. The atrous spatial pyramid pooling ASPP captures multi-scale global and local contextual information in the defect image through multiple different atrous convolution rates. For bridge defect segmentation, the atrous spatial pyramid pooling ASPP is used to understand the different forms, sizes and spatial distributions of defects. The decoder module is used to gradually restore the deep feature information extracted by the encoder to the resolution of the original image and generate the final pixel-level segmentation result. The decoder module combines the low-level features and high-level features of the encoder module to restore the details of the diseased image while maintaining the accuracy of disease segmentation.

8. The air-ground integrated bridge disease monitoring method according to claim 1 is characterized in that: In step S500, The bridge photos are converted into 3D models through image preprocessing, aerial triangulation, dense image matching, construction of triangular irregular network model TIN, automatic texture slice mapping and other steps; The resolution of the single high-resolution image obtained by close photogrammetry is higher than that of the bridge image data, and it can collect finer and richer bridge geometric structures and surface texture details. The final generated three-dimensional model has higher geometric accuracy and clearer surface texture details.

9. The air-ground integrated bridge disease monitoring method according to claim 1 is characterized in that: In step S600, The refined 3D model of the bridge constructed using the bridge image containing POS information contains world coordinates. Each point on the refined 3D model of the bridge is converted into coordinates in geographic coordinate systems such as WGS84 and CGCS2000 through coordinate transformation. The coordinate transformation includes the transformation from the camera coordinate system of the drone to the image plane of the bridge, the transformation from the camera coordinate system to the world coordinate system, and the transformation from the world coordinate system to the WGS84 coordinate system, which specifically includes the following steps: S610, the conversion from the drone camera coordinate system to the bridge image plane, that is, the pixel coordinate to the camera coordinate, the conversion from the 2D point on the pixel coordinate image plane to the 3D point in the camera coordinate system, is mainly based on the camera intrinsic parameter matrix K. The camera intrinsic parameter describes the focal length, principal point, and distortion information of the camera, etc., and is expressed as a 3x3 matrix: Among them, f x and f y are the focal lengths in the horizontal and vertical directions, c x and c y The principal point of the bridge image is usually the center of the bridge image. Given a pixel coordinate [u, v] on the bridge image, it is converted to the coordinate (x) in the normalized camera coordinate system through the intrinsic matrix. c ,y c ): Among them, K ―1 It is the inverse matrix of the drone camera’s intrinsic matrix. This transformation maps points on the 2D image plane to the camera coordinate system. In actual scenes, the depth Z of the 3D point c It cannot be obtained directly from a single image, but needs to be calculated from photos from multiple perspectives; S620, conversion from drone camera coordinate system to world coordinate system. In order to convert points in the camera coordinate system to the world coordinate system, the camera extrinsic matrix [R|T] is used. The camera extrinsic matrix [R|T] describes the position and orientation of the camera relative to the world coordinate system. The extrinsic matrix consists of a rotation matrix R and a translation vector T: Among them, (X w ,Y w ,Z w ) is the coordinate of the point in the world coordinate system, R is a 3x3 rotation matrix, which represents the attitude of the drone camera, namely the yaw angle, pitch angle and roll angle, and T is a 3x1 translation vector, which represents the position of the camera, namely the latitude, longitude and elevation in the POS data, which converts the point in the camera coordinate system to the world coordinate system; S630, the conversion of the world coordinate system of the 3D model to the WGS84 coordinate system. The world coordinate system is usually a local 3D coordinate system. To convert the coordinates to the WGS84 coordinate system on the earth, geographic coordinate conversion is required. The WGS84 coordinate system is expressed in longitude and latitude (φ, λ) and elevation (h). The 3D coordinates of the 3D model in the world coordinate system are converted to ECEF (Earth-Centered, Earth-Fixed) coordinates. The ECEF coordinate system is a 3D rectangular coordinate system with the center of mass of the earth as the origin. The formula is as follows: X=(N+h)·cos(φ)·cos(λ), Y=(N+h)·cos(φ)·sin(λ), Z=((1―e 2 )N+h)·sin(φ), Where N is the radius of curvature, a is the equatorial radius of the earth, and e is the eccentricity of the earth; according to the ECEF coordinates (X, Y, Z), they can be converted into longitude, latitude and elevation under WGS84 through the inverse formula. The inverse formula is as follows: Calculate the longitude λ: λ=arctan 2(Y,X) Obtained by the ratio of X and Y, using the inverse tangent function, then calculating the initial latitude φ 0 : φ 0 Then2(Z,r·(1-e 2 )) in, is the projected distance on the equatorial plane, and the latitude φ is calculated iteratively: φ=atan2(Z+e 2 ·N·sin(φ 0 ),r) in, is the radius of curvature, and finally calculate the elevation h: The elevation h is the vertical distance relative to the ellipsoid; the three-dimensional coordinates (X, Y, Z) of each point on the refined three-dimensional model of the bridge are converted into geographic coordinates (λ, φ, h) in the WGS84 coordinate system, namely longitude, latitude and elevation.

10. An air-ground integrated bridge disease monitoring system, used to implement the air-ground integrated bridge disease monitoring method as claimed in any one of claims 1 to 9, wherein the air-ground integrated bridge disease monitoring system is mounted on a patrol vehicle, and is characterized in that: The air-ground integrated bridge disease monitoring system comprises: A machine nest, the machine nest being electrically connected to the UAV automatic inspection and data processing platform; A drone, wherein the drone is arranged in the nest, the drone is charged through a charging port of the nest, the drone comprises a visual obstacle avoidance module, an infrared obstacle avoidance module, a positioning module, a pan-tilt camera, a top-view camera and a control module, the pan-tilt camera is an oblique photography device, the visual obstacle avoidance module is installed in front, rear, side and top of the drone, and the infrared obstacle avoidance module is installed at the bottom of the drone; An automatic inspection and data processing platform for drones, which is used to plan the flight routes of drones, issue route tasks, control the flight process and actions of drones, and process and exchange data collected by drones; A vehicle-mounted high-performance edge server, used for exchanging data with the UAV automatic inspection and data processing platform, wherein the vehicle-mounted high-performance edge server is electrically connected to the UAV automatic inspection and data processing platform; The user terminal device is used to run the UAV automatic inspection and data processing platform, and the user terminal device is electrically connected to the UAV automatic inspection and data processing platform.

Images