Bridge appearance detection method and system based on cooperation of unmanned aerial vehicle and workstation

Abstract

The invention relates to the technical field of bridge detection and data processing, in particular to a bridge appearance detection method and system based on cooperation of an unmanned aerial vehicle and a workstation. The technical problems are as follows: the modern bridge detection technology is complex in bridge environment, most of the existing unmanned aerial vehicle detection schemes depend on manual control flight, the flight difficulty is high, and high-definition pictures are difficult to shoot, so that a large amount of rework is caused; according to the technical scheme, the bridge appearance detection system based on cooperation of the unmanned aerial vehicle and the work station comprises an unmanned aerial vehicle platform and a ground work station; real-time data acquisition is performed on the bridge and the surrounding environment thereof through the unmanned aerial vehicle, a bridge digital map is produced by integrating a multi-sensor fusion technology and a three-dimensional modeling and deep learning technology, and a fine acquisition path is planned; and automatic, high-precision and high-coverage data acquisition of the bridge appearance in a complex environment is completed, so that the precision and efficiency of disease detection are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge detection and data processing, and in particular to a bridge appearance detection method and system based on collaboration between an unmanned aerial vehicle and a workstation. Background Art

[0002] With the rapid development of intelligent detection technology, artificial intelligence, drones and deep learning algorithms have provided new solutions for bridge safety detection. As an important part of transportation infrastructure, bridges have complex structures and are located in a changing environment. The timely discovery and detection of bridge defects is crucial to ensuring traffic safety.

[0003] Traditional bridge inspection methods mainly rely on manual inspections or inspection equipment. Due to the limitations of the working environment and equipment performance, there are often problems such as low inspection efficiency, insufficient coverage, and strong subjectivity in the results. In addition, for bridges in complex environments, such as those crossing rivers, canyons, and other inaccessible areas, it is difficult to obtain accurate disease data using traditional methods.

[0004] Modern bridge inspection technology is becoming more and more automated and intelligent, especially modern tools and means such as bridge inspection vehicles, cable-climbing robots, and drones, which assist manual work and improve inspection efficiency. Among them, drones are popular because of their flexibility and wide application scenarios. Due to the complex bridge environment, most existing drone inspection solutions rely on manual flight, which is difficult to fly and difficult to capture high-definition images, resulting in a lot of rework, which has become a bottleneck for improving intelligent inspection efficiency.

[0005] Therefore, there is an urgent need for a bridge inspection system based on the collaboration of drones and workstations that can achieve comprehensive and efficient monitoring of the bridge appearance; through multi-sensor fusion, three-dimensional modeling and deep learning technology, the system can perform high-precision data collection and disease analysis in complex environments, reduce manual intervention, and improve inspection efficiency and accuracy. Summary of the invention

[0006] In order to overcome the limitations of modern bridge inspection technology due to the complex bridge environment, existing drone inspection solutions mostly rely on manual flight, which is difficult to fly and difficult to capture high-definition images, resulting in a large number of rework problems.

[0007] The technical solution of the present invention is: a bridge appearance detection method based on the collaboration of a drone and a workstation, comprising the following steps:

[0008] S1: Use drones to collect real-time data and locate and map the bridge and its surroundings. The drones are equipped with 3D laser radar and high-definition cameras to collect images and point cloud data of the bridge structure and its surroundings, perform real-time positioning and mapping, and transmit these data to the ground workstation;

[0009] S2: The workstation receives data from the drone, and based on the calculation of the spatial relationship of each sensor data, uses multi-sensor fusion technology to optimize and refine the collected images, point cloud data, and real-time positioning and mapping results to generate a three-dimensional model of the bridge structure. Then, the workstation uses the point cloud deep learning algorithm to perform semantic recognition on the point cloud, distinguish the bridge structure from the background point cloud, and produce a 3D digital map of the bridge through vectorization post-processing;

[0010] S3: The workstation plans the inspection path for the digital model of the bridge, and sends the inspection path and digital map to the drone. The drone flies along the planned route, traverses the preset collection positions, adjusts the camera posture to capture images of the bridge surface, and transmits the captured images to the ground workstation in real time;

[0011] S4: The workstation uses a deep learning model to analyze the transmitted images, detect and identify defects on the bridge surface, focus on detecting cracks and peeling defects, and generate an inspection report containing the location and type of the defect.

[0012] Preferably, in step S1, real-time data collection and positioning and mapping of the bridge and its surrounding environment are performed by a drone, and the drone is equipped with a 3D laser radar and a high-definition camera to collect images and point cloud data of the bridge structure and its surroundings, perform real-time positioning and mapping, and transmit these data to a ground workstation; the specific steps include:

[0013] S11: The operator formulates a flight route according to the range of the bridge, wherein the flight route includes GPS flight anchor points and covers the spatial range of the bridge surface;

[0014] S12: The drone automatically performs flight missions according to the flight route, and activates the lidar and camera to scan and collect data during the flight, and uses the data for real-time positioning and mapping;

[0015] S13: Data and positioning detection results are transmitted to the ground workstation in real time via wireless communication.

[0016] Preferably, in step S2, the workstation receives data from the drone, and based on the calculation of the spatial relationship of each sensor data, uses multi-sensor fusion technology to optimize and refine the collected images, point cloud data, and real-time positioning and mapping results to generate a three-dimensional model of the bridge structure. Then, the workstation uses a point cloud deep learning algorithm to perform semantic recognition on the point cloud, distinguish the bridge structure from the background point cloud, and produce a 3D digital map of the bridge through vectorization post-processing. The specific steps include:

[0017] S21: The workstation uses multi-sensor data fusion technology to fuse and optimize LiDAR point cloud data, image data, and inertial measurement unit (IMU) data to generate a high-quality 3D model of the bridge and its surrounding environment;

[0018] S22: The workstation uses a point cloud recognition algorithm to perform semantic segmentation on the 3D model and identify the types of bridge structure surfaces and background obstacles;

[0019] S23: The workstation generates a digital 3D model of the bridge with semantic information, marks the structural surfaces belonging to the bridge, forms a data collection task, and presents it to the operator for review;

[0020] S24: The operator confirms each work surface, deletes the area that does not need to be inspected, and then issues further work instructions.

[0021] Preferably, in step S3, the workstation plans a detection path for the digital model of the bridge, and sends the detection path and the digital map to the drone. The drone flies according to the planned route, traverses the preset collection positions, adjusts the camera posture to capture images of the bridge surface, and transmits the captured images to the ground workstation in real time. The specific steps include:

[0022] S31: The workstation samples the position points of the bridge digital model so that the working surface can be traversed and covered, and generates a smooth and continuous shooting path, thereby ensuring that the pictures taken at the sampling points can cover the bridge and meet the resolution requirements of the disease detection model for the pictures taken. The shooting path and digital map are sent to the drone;

[0023] S32: The UAV performs automatic flight and traverses all target locations in the flight route. During the flight, the UAV is relocated through real-time mapping and the flight location is located on the global map of the bridge.

[0024] S33: When the UAV reaches each target position, it dynamically adjusts its posture until the high-definition camera of the UAV reaches the predetermined posture;

[0025] S34: The drone starts the high-definition camera to take pictures and transmits the pictures back to the ground workstation.

[0026] Preferably, in step S4, the workstation uses a deep learning model to analyze the returned image, detects and identifies the defects on the bridge surface, focuses on detecting cracks, peeling and other minor defects, and generates a test report including the location and type of the defect; the specific steps include:

[0027] S41: After receiving the high-definition images of the bridge surface, the ground workstation uses a deep learning model to perform fine identification and segmentation of defects, including but not limited to cracks, spalling, and corrosion;

[0028] S42: The ground workstation maps the identified defect locations from the image onto the digital model of the bridge, stores the data in a database, and generates a detailed inspection report.

[0029] The bridge appearance inspection system based on the collaboration of drones and workstations includes:

[0030] UAV platform for real-time multi-sensor data collection, real-time positioning and local mapping of the bridge and surrounding environment;

[0031] The ground workstation is used to receive and process the point cloud and image data transmitted by the drone, and to perform three-dimensional reconstruction and disease detection on the entire bridge.

[0032] Preferably, the UAV platform is equipped with a laser radar and a high-definition camera for acquiring point cloud and image data around the bridge. The UAV platform is integrated with a real-time positioning and mapping module and a shooting module for fine path planning of the UAV and shooting of high-definition pictures.

[0033] Preferably, the real-time positioning and mapping module is used to process the data collected by the laser radar and high-definition camera carried by the drone in real time through the mapping module, generate a local three-dimensional point cloud map, and provide information on the surrounding environment of the bridge observed during the flight; the shooting module is used to plan the flight path, dynamically adjust the camera posture, use a high-definition camera to shoot the bridge surface, and send the image data back to the ground workstation for further analysis.

[0034] Preferably, the ground workstation is responsible for real-time interaction between the operator and the UAV, automatically processing various data sent back from the UAV, and analyzing and presenting the test results; the ground workstation includes:

[0035] The working space generation module is used to generate the working space of the UAV, including the three-dimensional semantic map of the structural surface of the bridge, obstacles in the surrounding environment, etc.

[0036] Multi-sensor fusion 3D reconstruction unit: Utilizes the images, point clouds, and real-time positioning and mapping results collected by drones to optimize the production of high-quality 3D models of bridge structures, and provide global environmental information and location information for optimizing detection paths and disease location.

[0037] Preferably, the ground workstation also includes:

[0038] Intelligent path planning module, which is used to generate optimized shooting paths based on 3D mapping results, so as to capture high-quality and clear images, thereby performing high-precision detection of diseased areas;

[0039] Intelligent disease analysis module, which is used to analyze the disease conditions on the bridge surface by combining the data collected by drones through deep learning and image processing technology, and provide disease identification, analysis and statistical results;

[0040] The human-computer interaction module is used for interaction between the staff and the drone, real-time monitoring of the drone status and shooting progress, and remote intervention or adjustment of the drone mission parameters when necessary;

[0041] Visualization module: Through the visualization module of the workstation terminal, staff can view the three-dimensional structure diagram and disease analysis results of the bridge in real time, and play back and review the data.

[0042] Beneficial effects of the present invention:

[0043] 1. Compared with the modern bridge inspection technology under the existing technology, due to the complex bridge environment, the existing UAV inspection solutions mostly rely on manual flight, which is difficult to fly and difficult to take high-definition pictures, resulting in a lot of rework; this invention integrates multi-sensor fusion technology, 3D modeling and deep learning technology to produce digital maps of bridges, plan fine collection paths, and complete the automatic, high-precision and high-coverage data collection of bridge appearance in complex environments through the autonomous flight control technology of UAVs, thereby greatly improving the accuracy and efficiency of disease detection;

[0044] 2. Use drones to collect real-time data on the bridge and its surrounding environment, collect images and point cloud data of the bridge structure and its surroundings, perform real-time positioning and mapping, and transmit these data to the ground workstation; after receiving the data, the workstation performs multi-sensor data fusion processing, and uses point cloud data, image information, and real-time positioning and mapping results to perform multi-sensor fusion optimization to achieve high-quality three-dimensional reconstruction of the bridge structure; the workstation then uses the point cloud deep learning algorithm to perform semantic recognition on the point cloud, distinguish the bridge structure from the background point cloud, and produce a digital map of the bridge through vectorization post-processing; the workstation plans the detection path for the bridge digital model, and sends the detection path and digital map to the drone; the drone The drone flies along the planned route, traverses the preset collection positions, and adjusts the camera posture to capture images of the bridge surface, and transmits the captured images to the ground workstation in real time; the workstation analyzes the returned images and uses a deep learning model to detect and identify defects on the bridge surface, focusing on detecting subtle defects such as cracks and peeling, and generates an inspection report containing the location and type of the defect; thus, by integrating multi-sensor fusion technology, 3D modeling and deep learning technology, a digital map of the bridge is produced, a precise collection path is planned, and the autonomous flight control technology of the drone is used to complete the automated, high-precision, and high-coverage data collection of the bridge surface in a complex environment, thereby greatly improving the accuracy and efficiency of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 What is shown is a schematic flow chart of the steps of the bridge appearance inspection method based on the collaboration between the drone and the workstation of the present invention;

[0046] Figure 2 What is shown is a schematic diagram of the bridge appearance inspection system architecture based on the collaboration between the drone and the workstation of the present invention;

[0047] Figure 3 What is shown is the bridge digital model and working surface schematic diagram of the bridge appearance detection method based on the collaboration of the drone and the workstation of the present invention;

[0048] Figure 4 Shown is a schematic diagram of the relative posture of the camera and the shooting surface of the bridge appearance detection method based on the cooperation of the drone and the workstation of the present invention;

[0049] Figure 5 What is shown is a schematic diagram of factor graph modeling of multi-sensor fusion of the bridge appearance detection method based on collaboration between drone and workstation of the present invention;

[0050] Figure 6 What is shown is a schematic diagram of multi-sensor fusion three-dimensional mapping optimization of the bridge appearance detection method based on collaboration between drones and workstations of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0052] See also Figure 1-6 The present invention provides an embodiment: a bridge appearance detection method based on the collaboration of a drone and a workstation, comprising the following steps:

[0053] S1: Use drones to collect real-time data and locate and map the bridge and its surroundings. The drones are equipped with 3D laser radar and high-definition cameras to collect images and point cloud data of the bridge structure and its surroundings, perform real-time positioning and mapping, and transmit these data to the ground workstation;

[0054] S2: The workstation receives data from the drone, and based on the calculation of the spatial relationship of each sensor data, uses multi-sensor fusion technology to optimize and refine the collected images, point cloud data, and real-time positioning and mapping results to generate a three-dimensional model of the bridge structure. Then, the workstation uses the point cloud deep learning algorithm to perform semantic recognition on the point cloud, distinguish the bridge structure from the background point cloud, and produce a 3D digital map of the bridge through vectorization post-processing;

[0055] S3: The workstation plans the inspection path for the digital model of the bridge, and sends the inspection path and digital map to the drone. The drone flies along the planned route, traverses the preset collection positions, adjusts the camera posture to capture images of the bridge surface, and transmits the captured images to the ground workstation in real time;

[0056] S4: The workstation uses a deep learning model to analyze the transmitted images, detect and identify defects on the bridge surface, focus on detecting cracks and peeling defects, and generate an inspection report containing the location and type of the defect.

[0057] Preferably, in step S1, real-time data collection and positioning and mapping of the bridge and its surrounding environment are performed by a drone, and the drone is equipped with a 3D laser radar and a high-definition camera to collect images and point cloud data of the bridge structure and its surroundings, perform real-time positioning and mapping, and transmit these data to a ground workstation; the specific steps include:

[0058] S11: The operator formulates a flight route according to the range of the bridge, wherein the flight route includes GPS flight anchor points and covers the spatial range of the bridge surface;

[0059] S12: The drone automatically performs flight missions according to the flight route, and activates the lidar and camera to scan and collect data during the flight, and uses the data for real-time positioning and mapping;

[0060] S13: Data and positioning detection results are transmitted to the ground workstation in real time via wireless communication.

[0061] Preferably, in step S2, the workstation receives data from the drone, and based on the calculation of the spatial relationship of each sensor data, uses multi-sensor fusion technology to optimize and refine the collected images, point cloud data, and real-time positioning and mapping results to generate a three-dimensional model of the bridge structure. Then, the workstation uses a point cloud deep learning algorithm to perform semantic recognition on the point cloud, distinguish the bridge structure from the background point cloud, and produce a 3D digital map of the bridge through vectorization post-processing. The specific steps include:

[0062] S21: The workstation uses multi-sensor data fusion technology to fuse and optimize LiDAR point cloud data, image data, and inertial measurement unit (IMU) data to generate a high-quality 3D model of the bridge and its surrounding environment;

[0063] S22: The workstation uses a point cloud recognition algorithm to perform semantic segmentation on the 3D model and identify the types of bridge structure surfaces and background obstacles;

[0064] S23: The workstation generates a digital 3D model of the bridge with semantic information, marks the structural surfaces belonging to the bridge, forms a data collection task, and presents it to the operator for review;

[0065] S24: The operator confirms each work surface, deletes the area that does not need to be inspected, and then issues further work instructions.

[0066] Preferably, in step S3, the workstation plans a detection path for the digital model of the bridge, and sends the detection path and the digital map to the drone. The drone flies according to the planned route, traverses the preset collection positions, adjusts the camera posture to capture images of the bridge surface, and transmits the captured images to the ground workstation in real time. The specific steps include:

[0067] S31: The workstation samples the position points of the bridge digital model so that the working surface can be traversed and covered, and generates a smooth and continuous shooting path, thereby ensuring that the pictures taken at the sampling points can cover the bridge and meet the resolution requirements of the disease detection model for the pictures taken. The shooting path and digital map are sent to the drone;

[0068] S32: The UAV performs automatic flight and traverses all target locations in the flight route. During the flight, the UAV is relocated through real-time mapping and the flight location is located on the global map of the bridge.

[0069] S33: When the UAV reaches each target position, it dynamically adjusts its posture until the high-definition camera of the UAV reaches the predetermined posture;

[0070] S34: The drone starts the high-definition camera to take pictures and transmits the pictures back to the ground workstation.

[0071] Preferably, in step S4, the workstation uses a deep learning model to analyze the returned image, detects and identifies the defects on the bridge surface, focuses on detecting cracks, peeling and other minor defects, and generates a test report including the location and type of the defect; the specific steps include:

[0072] S41: After receiving the high-definition images of the bridge surface, the ground workstation uses a deep learning model to perform fine identification and segmentation of defects, including but not limited to cracks, spalling, and corrosion;

[0073] S42: The ground workstation maps the identified defect locations from the image onto the digital model of the bridge, stores the data in a database, and generates a detailed inspection report.

[0074] The bridge appearance inspection system based on the collaboration of drones and workstations includes:

[0075] UAV platform for real-time multi-sensor data collection, real-time positioning and local mapping of the bridge and surrounding environment;

[0076] The ground workstation is used to receive and process the point cloud and image data transmitted by the drone, and to perform three-dimensional reconstruction and disease detection on the entire bridge.

[0077] Preferably, the UAV platform is equipped with a laser radar and a high-definition camera for acquiring point cloud and image data around the bridge. The UAV platform is integrated with a real-time positioning and mapping module and a shooting module for fine route planning of the UAV and shooting of high-definition pictures.

[0078] Preferably, the real-time positioning and mapping module is used to process the data collected by the laser radar and high-definition camera carried by the drone in real time through the mapping module, generate a local three-dimensional point cloud map, and provide information on the surrounding environment of the bridge observed during the flight; the shooting module is used to plan the flight path, dynamically adjust the camera posture, use a high-definition camera to shoot the bridge surface, and send the image data back to the ground workstation for further analysis.

[0079] Preferably, the ground workstation is responsible for real-time interaction between the operator and the UAV, automatically processing various data sent back from the UAV, and analyzing and presenting the test results; the ground workstation includes:

[0080] The working space generation module is used to generate the working space of the UAV, including the three-dimensional semantic map of the structural surface of the bridge, obstacles in the surrounding environment, etc.

[0081] Multi-sensor fusion 3D reconstruction unit: Utilizes the images, point clouds, and real-time positioning and mapping results collected by drones to optimize the production of high-quality 3D models of bridge structures, and provide global environmental information and location information for optimizing detection paths and disease location.

[0082] Preferably, the ground workstation also includes:

[0083] Intelligent path planning module, which is used to generate optimized shooting paths based on 3D mapping results, so as to capture high-quality and clear images, thereby performing high-precision detection of diseased areas;

[0084] Intelligent disease analysis module, which is used to analyze the disease conditions on the bridge surface by combining the data collected by drones through deep learning and image processing technology, and provide disease identification, analysis and statistical results;

[0085] The human-computer interaction module is used for interaction between the staff and the drone, real-time monitoring of the drone status and shooting progress, and remote intervention or adjustment of the drone mission parameters when necessary;

[0086] Visualization module: Through the visualization module of the workstation terminal, staff can view the three-dimensional structure diagram and disease analysis results of the bridge in real time, and play back and review the data.

[0087] Example 1

[0088] Optionally, the drone automatically performs flight missions according to the flight route, and starts the lidar and camera to scan and collect data during the flight, and uses the data for real-time positioning and mapping. The details are as follows:

[0089] Real-time positioning and mapping: In order to solve the real-time positioning problem of UAVs in different bridge scenarios, the observation adopts multi-sensor fusion positioning and mapping technology. This technology adds GNSS positioning observations, IMU pre-integration observations, relative posture observations obtained by camera-IMU odometers, and relative posture observations obtained by laser-IMU odometers as observation factors into the factor graph, and performs smoothing filtering optimization on the time axis to obtain the real-time posture of multi-sensor fusion, while taking into account the random walk noise of the IMU's gyroscope and accelerometer bias in time;

[0090] The specific factor diagram is as follows Figure 5 As shown, Offset is the offset of the repositioned pose relative to the GNSS or other local coordinate system, p is the pose of the drone, v is the velocity, and b is the angular velocity and acceleration bias of the IMU;

[0091] In the absence of a three-dimensional map, the above sensor fusion positioning technology does not require map repositioning related observation-state factors; when a three-dimensional positioning map exists, the repositioning factor on the map can also be added to the factor graph; at the same time, considering that the repositioning posture of the camera or laser on the map relative to the GNSS or other local coordinate system has the characteristics of random walk, the random walk factor is also added to the IMU bias b and the repositioning offset in the factor graph; after adding the repositioning factor, the positioning has the ability to locate on the map, and can perform fine positioning more efficiently;

[0092] The data and positioning detection results are transmitted to the ground workstation in real time via wireless communication. The multi-sensor positioning technology can obtain a robust and stable UAV posture, complete the coordinate transformation of the 3D points generated by the camera and laser relative sensors, and generate a real-time 3D map. GNSS, IMU and other positioning data, as well as positioning posture and 3D maps will be transmitted back to the workstation for further three-dimensional reconstruction and optimization.

[0093] Example 2

[0094] Optionally, after receiving the data from the drone, the workstation uses multi-sensor fusion technology to optimize and refine the collected images, point cloud data, and real-time positioning and mapping results based on the calculation of the spatial relationship of each sensor data, and generates a three-dimensional model of the bridge structure; then, the workstation uses the point cloud deep learning algorithm to perform semantic recognition on the point cloud, distinguish the bridge structure from the background point cloud, and produce a 3D digital map of the bridge through vectorization post-processing; among them, the ground workstation, as the core data processing platform, is responsible for receiving multi-source data from the drone and providing the drone with a three-dimensional digital map and shooting route planning; the workstation software includes a real-time visualization system, a real-time human-computer interaction system, and a back-end algorithm system; the algorithm system includes algorithm modules such as operation space generation, intelligent path planning, and intelligent disease analysis; the specific steps include:

[0095] A1: The workstation uses multi-sensor data fusion technology to fuse and optimize the LiDAR point cloud data, image data, and inertial measurement unit (IMU) data to generate a high-quality 3D model of the bridge and its surrounding environment. The specific steps are as follows:

[0096] a1: 3D mapping optimization: A fast multi-pass mapping solution is used to ensure the overall geometric consistency of the 3D mapping; multi-pass mapping solutions such as Figure 6 As shown, it is mainly divided into three steps:

[0097] The first step is to divide the multi-pass collected data into segments according to the curvature of the trajectory (L1, L2, L3 in the figure) ( Figure 3 The green and yellow routes in the figure can be understood as data of different trips);

[0098] In the second step, the three-dimensional map optimization of the camera, laser, and positioning data fusion is performed on each trip of data. During the optimization, the matching feature points of the image, the ICP matching feature points of the laser, as well as the IMU and GNSS are all used as optimization factors for adjustment optimization to obtain a more accurate three-dimensional pose and map.

[0099] The third step is to select the data with the widest coverage and the best quality from all the mapping data as the benchmark. The other data are relocated on this data, and the pose and map are aligned and optimized, and then merged to form a new benchmark map. This cycle is repeated until all the data are aligned.

[0100] a2: After the map is built, the feature points and 3D coordinates of the image are used to build a positioning layer as the positioning layer used for relocation in step S3; ensure that the position of step S3 is on the reconstructed map;

[0101] A2: The workstation uses the point cloud recognition algorithm to perform semantic segmentation on the 3D model and identify the types of bridge structure surfaces and background obstacles. The specific steps are as follows:

[0102] The first step is to build a deep learning network OneFormer3D model that can be used for point cloud panorama segmentation;

[0103] In the second step, the reconstructed colored point cloud of the bridge and its surroundings (x, y, z represent the geometric coordinates of each point, and r, g, b represent the color of each point fused from the image) is input into the network;

[0104] In the third step, the network encodes the geometric and color features to extract semantically relevant features. The output of the network predicts a mask for each foreground object. For example, all points belonging to pier A are labeled as (pier, instance A), all points belonging to pier B are labeled as (pier, instance B), all points belonging to vehicle C are predicted as (vehicle, instance C), and so on.

[0105] A3: The workstation generates a digital 3D model of the bridge with semantic information, marks the structural surfaces belonging to the bridge, forms a data collection task, and presents it to the operator for review; the digital 3D model of the bridge is fitted by the bridge point cloud and the types related to the bridge (main beam, pier, bridge deck) are marked; the surface direction of the bridge is determined by the relative position of the scanning point and the camera when it is scanned; the fitted 3D model marks the outer surface of the bridge and distinguishes it by type, and each instance generates a working surface, such as Figure 3 As shown;

[0106] A4: The operator confirms each working surface and deletes areas that do not need to be inspected, such as the driving surface of a bridge, and then issues further work instructions.

[0107] Example 3

[0108] Optionally, the workstation plans a detection path for the digital model of the bridge, and sends the detection path and the digital map to the drone; the drone flies along the planned route, traverses the preset collection positions, and adjusts the camera posture to capture images of the bridge surface, and transmits the captured images to the ground workstation in real time; the specific steps are as follows:

[0109] B1: The workstation samples the position points of the bridge digital model so that the working surface can be traversed and covered, and generates a smooth and continuous shooting path to ensure that the pictures taken at the sampling points can cover the bridge and meet the resolution requirements of the disease detection model for the pictures taken; the shooting path and digital map are sent to the drone;

[0110] The shooting path consists of a series of target tasks {Ti}; the specifications of each task Ti are: reach the target point Xi, adjust the camera to posture Yi, and take a picture Zi; the specific steps include:

[0111] First, the relative posture of the camera and the shooting surface is Figure 4 As shown in the figure, S_L is the sampling distance of the shooting surface, P_L is the pixel length, N is the number of pixels of the image, d is the vertical distance from the optical center of the camera to the shooting surface, and f is the focal length of the camera;

[0112] Second, according to the camera imaging principle, d=S_L*f / (N*P_L) is satisfied, where S_L / N is the image resolution (GSD), in mm / pixel. In bridge inspection, the GSD requirement is related to the type of disease. In this embodiment, for cracks, the GSD is required to be no more than 0.1 mm / pixel.

[0113] B2: The UAV performs automatic flight and traverses all target locations in the flight route. During the flight, the UAV repositions itself through real-time mapping and locates the flight location on the global map of the bridge.

[0114] B3: When the drone reaches each target location, it dynamically adjusts the camera to reach the predetermined posture;

[0115] B4: The drone starts taking pictures with a high-definition camera, and the pictures are transmitted back to the ground workstation via image transmission.

[0116] Example 4

[0117] Optionally, the workstation analyzes the returned images and uses a deep learning model to detect and identify defects on the bridge surface, focusing on detecting minor defects such as cracks and spalling, and generates a test report containing the location and type of the defect; the specific steps are as follows:

[0118] C1: After receiving high-definition images of the bridge surface, the ground workstation uses a deep learning model to perform fine identification and segmentation of defects, including cracks, spalling, corrosion, etc.

[0119] Among them, the image semantic segmentation model is used to detect the apparent cracks of the bridge;

[0120] Build a SAM segmentation model, input the high-definition picture sent back by the drone, and add the prompt word "crack";

[0121] The deep network encodes the pixels of the image, encodes the prompt word, and outputs a mask image that marks the pixels in the image that belong to the crack;

[0122] C2: The ground workstation maps the identified defect locations from the image to the digital model of the bridge, stores the data in the database, and generates a detailed inspection report; the specific steps are as follows:

[0123] Through the camera posture when the image was taken, the pixel position of the crack in the image was reversely mapped to the three-dimensional space to obtain the direction of the incident light of the pixel. The first intersection of this ray and the three-dimensional model was solved, which was the crack position on the bridge corresponding to the pixel point.

[0124] C3: The ground workstation marks all crack locations in the digital model of the bridge for subsequent analysis.

[0125] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of those skilled in the art without departing from the spirit of the present invention.

Claims

1. A bridge appearance inspection method based on the collaboration between drones and workstations; it is characterized by: The following steps are included: S1: Use drones to collect real-time data and locate and map the bridge and its surroundings. The drones are equipped with 3D laser radar and high-definition cameras to collect images and point cloud data of the bridge structure and its surroundings, perform real-time positioning and mapping, and transmit these data to the ground workstation; S2: The workstation receives data from the drone, and based on the calculation of the spatial relationship of each sensor data, uses multi-sensor fusion technology to optimize and refine the collected images, point cloud data, and real-time positioning and mapping results to generate a three-dimensional model of the bridge structure. Then, the workstation uses the point cloud deep learning algorithm to perform semantic recognition on the point cloud, distinguish the bridge structure from the background point cloud, and produce a 3D digital map of the bridge through vectorization post-processing; S3: The workstation plans the inspection path for the digital model of the bridge, and sends the inspection path and digital map to the drone. The drone flies along the planned route, traverses the preset collection positions, adjusts the camera posture to capture images of the bridge surface, and transmits the captured images to the ground workstation in real time; S4: The workstation uses a deep learning model to analyze the transmitted images, detect and identify defects on the bridge surface, focus on detecting cracks and peeling defects, and generate an inspection report containing the location and type of the defect.

2. The bridge appearance inspection method based on the collaboration between drone and workstation according to claim 1 is characterized by: In step S1, real-time data collection and positioning and mapping of the bridge and its surrounding environment are performed by using a drone. The drone is equipped with a 3D laser radar and a high-definition camera to collect images and point cloud data of the bridge structure and its surroundings, perform real-time positioning and mapping, and transmit these data to a ground workstation. The specific steps include: S11: The operator formulates a flight route according to the range of the bridge, wherein the flight route includes GPS flight anchor points and covers the spatial range of the bridge surface; S12: The drone automatically performs flight missions according to the flight route, and activates the lidar and camera to scan and collect data during the flight, and uses the data for real-time positioning and mapping; S13: Data and positioning detection results are transmitted to the ground workstation in real time via wireless communication.

3. The bridge appearance inspection method based on the collaboration between drone and workstation according to claim 1 is characterized by: In step S2, the workstation receives data from the drone, and based on the calculation of the spatial relationship of each sensor data, uses multi-sensor fusion technology to optimize and refine the collected images, point cloud data, and real-time positioning and mapping results to generate a three-dimensional model of the bridge structure. Then, the workstation uses a point cloud deep learning algorithm to perform semantic recognition on the point cloud, distinguish the bridge structure from the background point cloud, and produce a 3D digital map of the bridge through vectorization post-processing; The specific steps include: S21: The workstation uses multi-sensor data fusion technology to fuse and optimize LiDAR point cloud data, image data, and inertial measurement unit (IMU) data to generate a high-quality 3D model of the bridge and its surrounding environment; S22: The workstation uses a point cloud recognition algorithm to perform semantic segmentation on the 3D model and identify the types of bridge structure surfaces and background obstacles; S23: The workstation generates a digital 3D model of the bridge with semantic information, marks the structural surfaces belonging to the bridge, forms a data collection task, and presents it to the operator for review; S24: The operator confirms each work surface, deletes the area that does not need to be inspected, and then issues further work instructions.

4. The bridge appearance inspection method based on the collaboration between drone and workstation according to claim 1 is characterized by: In step S3, the workstation plans a detection path for the digital model of the bridge, and sends the detection path and the digital map to the drone. The drone flies along the planned route, traverses the preset collection positions, adjusts the camera posture to capture images of the bridge surface, and transmits the captured images to the ground workstation in real time; The specific steps include: S31: The workstation samples the position points of the bridge digital model so that the working surface can be traversed and covered, and generates a smooth and continuous shooting path, thereby ensuring that the pictures taken at the sampling points can cover the bridge and meet the resolution requirements of the disease detection model for the pictures taken. The shooting path and digital map are sent to the drone; S32: The UAV performs automatic flight and traverses all target locations in the flight route. During the flight, the UAV is relocated through real-time mapping and the flight location is located on the global map of the bridge. S33: When the UAV reaches each target position, it dynamically adjusts its posture until the high-definition camera of the UAV reaches the predetermined posture; S34: The drone starts the high-definition camera to take pictures and transmits the pictures back to the ground workstation.

5. The bridge appearance inspection method based on the collaboration between drone and workstation according to claim 1 is characterized by: In step S4, the workstation uses a deep learning model to analyze the returned images, detect and identify the defects on the bridge surface, focus on detecting cracks, peeling and other minor defects, and generate a test report containing the location and type of the defect; The specific steps include: S41: After receiving the high-definition images of the bridge surface, the ground workstation uses a deep learning model to perform fine identification and segmentation of defects, including but not limited to cracks, spalling, and corrosion; S42: The ground workstation maps the identified defect locations from the image onto the digital model of the bridge, stores the data in a database, and generates a detailed inspection report.

6. The bridge appearance inspection system based on the collaboration of UAV and workstation is characterized by: Included are: UAV platform for real-time multi-sensor data collection, real-time positioning and local mapping of the bridge and surrounding environment; The ground workstation is used to receive and process the point cloud and image data transmitted by the drone, and to perform three-dimensional reconstruction and disease detection on the entire bridge.

7. The bridge appearance inspection system based on the collaboration of unmanned aerial vehicle and workstation according to claim 6 is characterized by: The drone platform is equipped with a laser radar and a high-definition camera for acquiring point cloud and image data around the bridge. The drone platform is integrated with a real-time positioning and mapping module and a shooting module for fine path planning of the drone and shooting of high-definition pictures.

8. The bridge appearance inspection system based on the collaboration between drone and workstation according to claim 7 is characterized by: The real-time positioning and mapping module is used to process the data collected by the laser radar and high-definition camera carried by the drone in real time through the mapping module, generate a local three-dimensional point cloud map, and provide information about the surrounding environment of the bridge observed during the flight; the shooting module is used to plan the flight path, dynamically adjust the camera posture, use a high-definition camera to shoot the bridge surface, and send the image data back to the ground workstation for further analysis.

9. The bridge appearance inspection system based on the collaboration of unmanned aerial vehicles and workstations according to claim 6 is characterized by: The ground workstation is responsible for real-time interaction between the operator and the drone, automatically processing various data sent back from the drone, and analyzing and presenting the test results; the ground workstation includes: The working space generation module is used to generate the working space of the UAV, including the three-dimensional semantic map of the structural surface of the bridge, obstacles in the surrounding environment, etc. Multi-sensor fusion 3D reconstruction unit: Utilizes the images, point clouds, and real-time positioning and mapping results collected by drones to optimize the production of high-quality 3D models of bridge structures, and provide global environmental information and location information for optimizing detection paths and disease location.

10. The bridge appearance inspection system based on the collaboration of unmanned aerial vehicles and workstations according to claim 9 is characterized in that: The ground workstation also includes: Intelligent path planning module, which is used to generate optimized shooting paths based on 3D mapping results, so as to capture high-quality and clear images, thereby performing high-precision detection of diseased areas; Intelligent disease analysis module, which is used to analyze the disease conditions on the bridge surface by combining the data collected by drones through deep learning and image processing technology, and provide disease identification, analysis and statistical results; The human-computer interaction module is used for interaction between the staff and the drone, real-time monitoring of the drone status and shooting progress, and remote intervention or adjustment of the drone mission parameters when necessary; Visualization module: Through the visualization module of the workstation terminal, staff can view the three-dimensional structure diagram and disease analysis results of the bridge in real time, and play back and review the data.

Images