Unmanned aerial vehicle laser and vision fusion inspection method and system for bridge bottom disease detection

Abstract

The invention discloses an unmanned aerial vehicle laser and vision fusion inspection method and system for bridge bottom disease detection, and the method comprises the steps: carrying out the synchronous data collection through employing a calibrated laser radar, a camera and an IMU, and obtaining a three-dimensional laser point cloud and a two-dimensional visual image of the appearance of a bridge; sharpening the image containing the motion blur and completing brightness self-adaption of the image; stable feature points are extracted, multi-frame matching is carried out, the corresponding poses of the images are estimated, and bridge dense point cloud reconstruction is completed; performing geometric component segmentation on the point cloud to generate a geometric prior region; component segmentation is carried out on a support area in the image, and a continuous and accurate component segmentation result is obtained in combination with a geometric prior area; screening the image, calling a targeted disease detection model in a corresponding component area, and generating a segmentation mask for the disease; obtaining a real disease three-dimensional point cloud, and carrying out quantitative calculation on the physical size of the disease; and displaying the real disease three-dimensional point cloud data and the physical size of the disease. The method is high in efficiency and precision.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of bridge detection, and specifically relates to a UAV laser and vision fusion inspection method and system for bridge bottom disease detection. BACKGROUND

[0002] During the long-term service of a bridge, the surface of the bridge is affected by various factors and may have diseases such as concrete cracks, spalling and exposed reinforcement, which seriously threaten the safety of the bridge. Detecting and positioning the diseases on the surface of the bridge helps to develop targeted maintenance measures in a timely manner and prolong the service life. The traditional bridge detection method mainly relies on manual inspection, which has problems such as low detection efficiency, poor operation safety and strong subjectivity of information recording, and manual detection is more difficult for small and medium span bridges with narrow space under the bridge and limited traffic conditions. With the rapid development of UAV remote sensing, computer vision and artificial intelligence technology, bridge disease detection based on UAV has gradually become the mainstream bridge disease detection method.

[0003] However, the current bridge disease detection method based on UAV still has certain limitations. For example, patent documents CN115797350B and CN117710810B both use visual images for recognition and three-dimensional reconstruction. In this method, factors such as image overlap rate, field of view size and feature point number will affect the recognition and positioning accuracy. When this technology is applied to small and medium span bridge detection, due to the narrow space under the bridge, insufficient lighting conditions and frequent gusts, the images taken often have problems such as low brightness, motion blur and sparse texture features on the surface of the components, which significantly reduces the feature extraction and matching effect; at the same time, the GNSS signal in the bridge bottom environment is easily blocked, and cannot provide reliable initial pose information for image three-dimensional reconstruction, further reducing the accuracy and stability of recognition and spatial positioning.

[0004] At present, some researches have fused laser point cloud and visual image to make up for the limitations of images with the spatial geometric information of point cloud, for example, patent documents CN119810084B and CN119246518B both use the technology route of fusion of laser point cloud and visual image to realize automatic identification and spatial positioning of diseases. However, this method has certain limitations in disease identification. Firstly, it does not fully consider the morphological differences of the same type of disease on different bridge components. Taking crack disease as an example, in small and medium span bridges, beam slab cracks are usually caused by bending or shearing action, and usually present a vertical or oblique elongated regular form; while pier cracks are mainly affected by temperature changes, material shrinkage or foundation settlement, and usually present a vertical, horizontal or irregular radial distribution, with more complex morphological characteristics. Secondly, it does not fully consider the identification and visualization of disease types specific to bearing components.

[0005] Existing bridge surface defect detection technologies based on the fusion of laser point clouds and visual images, while capable of two-dimensional image defect identification and three-dimensional point cloud localization, still have several shortcomings. This technology does not address motion blur and uneven illumination issues in images from UAV inspections or low-light environments under bridges, leading to unstable image features and affecting defect identification accuracy. Point cloud screening relies solely on two-dimensional segmentation masks and ray tracing, lacking component geometric constraints, making it prone to false positives or false negatives. Furthermore, the system's overall adaptability is low, making it difficult to meet the needs of UAV inspections in complex environments under bridges.

[0006] In summary, existing disease identification methods suffer from drawbacks such as low image quality leading to unstable spatial positioning, insufficient consideration of the differences in disease morphology among different components, inadequate 3D reconstruction accuracy, and inaccurate disease visualization. The accuracy and visualization effects of disease detection need to be further improved. Summary of the Invention

[0007] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide a high-precision and high-efficiency UAV laser and vision fusion inspection method for detecting bridge under-bridge defects. Another purpose of this invention is to provide a UAV laser and vision fusion inspection method for detecting bridge under-bridge defects that can accurately quantify defects.

[0008] Technical Solution: The present invention provides a UAV laser and vision fusion inspection method for detecting bridge under-bridge defects, comprising the following steps:

[0009] (1) Jointly calibrate the lidar, camera and IMU and complete time synchronization, and use the calibrated lidar and camera to collect synchronous data to obtain the three-dimensional laser point cloud and two-dimensional visual image of the bridge surface.

[0010] (2) Using the attitude and motion information of the IMU, a motion point spread function is constructed and input into the deep deblurring network DeblurGAN-v2 algorithm for spatiotemporal characteristic correction and sharpness enhancement to obtain the deblurred image;

[0011] (3) The pre-set image enhancement network EnlightenGAN model is used to process the deblurred image, and adaptive enhancement is performed on the local dark areas and highlight areas to obtain an image with clear features and uniform illumination;

[0012] (4) Extract stable feature points from images with clear features and uniform illumination and perform multi-frame matching. Estimate the corresponding pose of the image through the lidar-vision-IMU fusion method and perform three-dimensional reconstruction of the dense point cloud image of the bridge.

[0013] (5) Establish a mapping model between three-dimensional laser point cloud and two-dimensional visual image. Use random sampling consistency algorithm combined with region growing algorithm based on normal similarity and spatial adjacency to obtain preliminary geometric region. Cluster optimization is performed on beam and slab and pier components. Generate prior geometric regions of beam and slab and pier in image plane through mapping model.

[0014] (6) The YOLOv11-Seg model is used as the segmentation backbone network to perform component-level instance segmentation of the bridge supports in the image. The segmentation results are re-filtered by combining the geometric prior region and post-processed by the conditional random field to obtain continuous and accurate component segmentation results.

[0015] (7) Filter the images by component semantic tags, identify defects in the corresponding beam, slab and pier concrete component areas, and generate segmentation masks;

[0016] (8) Project the three-dimensional laser point cloud of the bridge onto the two-dimensional image plane, filter and segment the points in the mask and transform them to the world coordinate system. After filtering the error by ray tracing, obtain the real three-dimensional point cloud of the disease. At the same time, generate a depth map, calculate the spatial coordinates and physical size of the disease based on the depth map, and visualize it in the three-dimensional point cloud.

[0017] Furthermore, step (2) specifically includes:

[0018] (2.1) The camera rotation trajectory is calculated by integrating the IMU angular velocity during the exposure period, and the translation is estimated by integrating the acceleration, providing motion estimation priors for image processing;

[0019] (2.2) Calculate the trajectory of each pixel during the exposure based on the IMU motion estimation results, construct the motion point spread function for deblurred image enhancement, and consider the line time offset of the rolling shutter;

[0020] (2.3) The point spread function is input into the generator of the DeblurGAN-v2 network. The network corrects the blur based on the known motion information. The generator generates a clear image based on the prior. The discriminator evaluates the generated image and optimizes the generator's output. Finally, the deblurred image is output.

[0021] Furthermore, step (3) specifically includes:

[0022] (3.1) Input the image after deblurring network processing into the preset image enhancement network EnlightenGAN model, and perform size normalization and pixel value normalization preprocessing on the input image;

[0023] (3.2) The generator in the EnlightenGAN model is used to extract multi-scale illumination and texture features of the input image, and the hierarchical representation of local illumination information and spatial feature preservation are realized based on the encoder-decoder structure;

[0024] (3.3) The brightness distribution of the input image is adaptively adjusted by a pre-trained illumination mapping function, which performs brightness enhancement and detail enhancement on dark areas and brightness suppression and contrast balancing on bright areas to achieve regional illumination balance.

[0025] (3.4) Perform gamma correction and color balance processing on the enhancement results, and output the image after illumination adaptive enhancement to obtain image results with clear features, uniform brightness and natural fidelity.

[0026] Furthermore, step (4) specifically includes:

[0027] (4.1) Feature extraction is performed on the processed image sequence. The scale-invariant feature transformation algorithm is used to extract local key points and their descriptors. Simultaneously, the lidar point cloud and IMU inertial measurement data are read to form a spatiotemporally aligned multi-sensor dataset.

[0028] (4.2) Based on the extrinsic parameter calibration results, the image features and point cloud features are spatially registered. The correspondence between the vision and the point cloud is established by using the nearest neighbor feature matching algorithm and feature consistency constraints. The relative pose of the camera and the lidar is estimated in advance.

[0029] (4.3) Construct a joint factor graph model of vision-lidar-IMU, with pose as the node, visual feature observation, point cloud matching residual and IMU integral result as the constraint edge, and use the sliding window optimization algorithm to realize multi-source information fusion positioning to obtain high-precision camera motion trajectory;

[0030] (4.4) After obtaining the camera pose optimized by multi-source fusion, the camera pose is used to generate a sparse point cloud through triangulation. A multi-view stereo vision method is used to calculate the three-dimensional information of each pixel in the image according to the correspondence to realize the three-dimensional reconstruction of the dense point cloud image of the bridge.

[0031] Furthermore, step (5) specifically includes:

[0032] (5.1) Establish a mapping model between three-dimensional laser point clouds and two-dimensional visual images;

[0033] (5.2) The random sampling consensus algorithm is used to perform global plane fitting and outlier removal on the point cloud. The normal vector and curvature features of the fitting residual points are calculated. The region growing algorithm based on normal similarity and spatial adjacency is used for local segmentation to achieve the preliminary geometric division of bridge beams, piers and curved components.

[0034] (5.3) Cluster analysis and boundary optimization are performed on the point cloud region after geometric segmentation. A density-based spatial clustering algorithm is used to automatically generate connected point cloud clusters based on the neighborhood density and spatial reachability of the point cloud, so as to realize the automatic clustering and division of beams, slabs and piers.

[0035] (5.4) Project each cluster of points onto the two-dimensional image plane, calculate the pixel coordinates of each three-dimensional point using the mapping model established in step (5.1), and generate the corresponding geometric prior region mask based on the projection distribution of the cluster.

[0036] Furthermore, step (6) specifically includes:

[0037] (6.1) A pre-trained YOLOv11-Seg network is used as the instance segmentation backbone network to perform pixel-level segmentation of the support components in the image and obtain the segmentation mask of the support region.

[0038] (6.2) Using the prior geometric regions of beams and piers generated in step (5) as geometric constraints, the support segmentation results are assisted in screening. In the region adjacent to the boundary of the prior regions of beams and piers, the support pixels with strong spatial correlation are retained based on the consistency of point cloud projection density and depth. If outside the prior region, the support pixels with confidence levels lower than the preset threshold are downsampled and filtered out to suppress false detection and boundary diffusion.

[0039] (6.3) The re-selected mask is used as the initial energy term to input the fully connected conditional random field to construct a similarity term based on pixel RGB color, depth value and spatial position. The mean field approximation algorithm is used for iterative optimization to obtain the final support segmentation result with continuous boundaries and accurate details.

[0040] Furthermore, step (7) specifically includes:

[0041] (7.1) Based on the differences in structural characteristics and disease morphology of beams, slabs, piers and bearings, datasets were constructed and targeted disease identification models were trained.

[0042] (7.2) For the selected concrete component areas such as beams, slabs and piers, the YOLOv8 network is called to detect defects such as cracks, spalling and exposed reinforcement, and the Deeplabv3+ network is used for defect segmentation. The YOLOv11 detection network is used to quickly locate candidate areas of typical concrete surface defects such as cracks, spalling and exposed reinforcement in the target area, and the Deeplabv3+ segmentation network is used to perform fine segmentation of the defect boundaries in the candidate areas to obtain pixel-level defect masks.

[0043] (7.3) For the selected rubber support component area, YOLOv11 based on attention mechanism enhancement is called to detect and classify rubber aging, shear deformation, cracking and other defects, and Deeplabv3+ network is called to segment crack defects.

[0044] Furthermore, the YOLOv11 backbone network introduces the attention module CBAM, which includes channel attention module and spatial attention module, to enhance the model's feature response to micro-cracks, aging discoloration and voiding defects on the rubber bearing surface. The Deeplabv3+ segmentation network is used to perform fine segmentation of defect boundaries in candidate regions to obtain pixel-level defect masks.

[0045] Furthermore, step (8) is followed by the following step:

[0046] (9) Based on the dense point cloud image model of the bridge in step (4), generate the spatial distribution information of each component and the defects. Project the three-dimensional coordinates of the defects obtained in step (8) into the existing BIM model coordinate system to realize the automatic calibration and structural update of the BIM model. In this way, the identified defect information is embedded into the BIM model in the form of additional attributes to form a digital bridge management model that can be dynamically updated.

[0047] The present invention discloses an unmanned aerial vehicle (UAV) laser and vision fusion inspection system for detecting bridge under-bridge defects, comprising:

[0048] The multi-source data synchronous acquisition and registration module is used to jointly calibrate the lidar, camera and IMU, and after calibration, synchronously acquire the 3D point cloud, 2D image and inertial navigation data of the bridge. At the same time, it establishes a mapping model between the point cloud and the image to realize the projection of the 3D point cloud onto the 2D image plane and the generation of the front view.

[0049] The image enhancement module is used to deblur the acquired image with motion blur using a deblurring network, and to adaptively adjust the brightness and sharpness of the image using an image enhancement network.

[0050] The component segmentation module is used to identify components in a two-dimensional visual image using a preset image component segmentation model and point cloud clustering algorithm, and generate a pixel-level segmentation mask for each component.

[0051] The disease identification module is used to identify diseases in two-dimensional visual images using a preset target detection model, and to generate a pixel-level segmentation mask for each disease.

[0052] The bridge model generation module is used to reconstruct the bridge structure with high precision by combining multi-view image data with laser point clouds, and to complete the model through point cloud registration.

[0053] The disease model generation module is used to filter out points located within the segmentation mask from the point cloud front view, and use ray tracing to filter out error points caused by local occlusion in the potential disease 3D point cloud to obtain the disease 3D point cloud; the depth information of the real disease 3D point cloud is assigned to the corresponding pixels in the 2D visual image, wherein, for pixels lacking depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the disease segmentation mask area; based on the depth map, the physical parameters of the disease are calculated.

[0054] The defect display module is used to display the actual 3D point cloud data of defects and the physical dimensions of defects in the 3D laser point cloud data of the bridge.

[0055] Beneficial effects: Compared with the prior art, the present invention has the following significant features:

[0056] 1. By introducing image enhancement algorithms into visual image processing to improve image quality, and by fusing laser point cloud data with inertial measurement unit (IMU) motion information, high-precision registration and fusion of multi-source information is achieved, which improves the detail representation and spatial positioning stability of visual images.

[0057] 2. By combining the geometric features of component point clouds with image information, a preliminary component-level classification of three types of components in bridge structures—beams, slabs, piers, and supports—is carried out. In addition, based on the structural features and disease morphology differences of different components, datasets are constructed and targeted disease identification models are trained to achieve accurate disease identification and pixel-level segmentation, thereby improving the accuracy and visualization effect of disease detection and expanding the range of disease identification types.

[0058] 3. High-quality visual images enhanced by image enhancement were used for 3D reconstruction, and geometric completion was performed using laser point clouds to construct a high-precision 3D model of the bridge. Based on this, the actual physical dimensions of various identified defects were calculated, realizing the accurate visualization of defects on the complete bridge model. This provides reliable data support and technical foundation for the dynamic updating and maintenance of the bridge BIM model. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the UAV laser and vision fusion inspection method for detecting bridge under-bridge defects according to the present invention.

[0060] Figure 2 This is the coordinate system involved in the mapping model between the three-dimensional laser point cloud and the two-dimensional visual image of this invention;

[0061] Figure 3 This is a demonstration image of the motion blur removal algorithm of this invention;

[0062] Figure 4These are the detection results of four types of defects according to the present invention, where a is support crack, b is concrete crack, c is exposed rebar, and d is spalling.

[0063] Figure 5 This is a visualization of the crack defects of the present invention on a bridge model. Detailed Implementation

[0064] Example 1

[0065] This invention provides a UAV laser and vision fusion inspection method for detecting bridge under-bridge defects, such as... Figure 1 As shown, it includes the following steps:

[0066] (1) Jointly calibrate the lidar, camera and inertial measurement unit (IMU), and use the calibrated lidar and camera to collect data synchronously to obtain the three-dimensional laser point cloud and two-dimensional visual image of the bridge surface.

[0067] The 3D laser point cloud data is obtained through LiDAR scanning, which can be either mechanical or solid-state. Two-dimensional visual images are captured by a camera. The joint calibration method for LiDAR, camera, and IMU first synchronizes the acquisition systems of the radar, camera, and IMU using hardware triggering to eliminate time delays between different sensors. Then, a checkerboard calibration board is set up. With the relative installation positions of the radar and camera fixed, the calibration board is moved or rotated to different positions and orientations, acquiring multiple sets of synchronized data including radar point clouds, camera images, and corresponding IMU measurements for the corresponding time periods. Camera images are used to extract the sub-pixel coordinates of the checkerboard corner points. The radar point cloud intersection coordinates are extracted by establishing a checkerboard model and fitting the data. The IMU data undergoes pre-integration processing to obtain the relative attitude changes during the acquisition period, providing inertial constraints for subsequent optimization. For the LiDAR and camera, a spatial correspondence is established between extracted image corner points and radar corner points. The PnP algorithm is used to calculate the initial rotation and translation relationships between the radar and camera. Optimization methods are then used to jointly optimize multiple sets of data to obtain a more accurate radar-camera extrinsic parameter matrix. Furthermore, the rotation and translation information obtained through short-time integration of the IMU, along with the pose changes of the camera or radar between adjacent frames, constitute hand-eye calibration constraints to solve for the initial extrinsic parameter matrices from the IMU to the camera and from the IMU to the radar.

[0068] A semi-solid-state lidar was selected to acquire point cloud data of the bridge. Its laser field of view is 70° × 77.2°. This device can obtain a point cloud model of the bridge from all directions. A focal length (f...) was selected... x f y An industrial camera with coordinates (393.556, 393.62) has an optical center (c). x cy The coordinates are (322.908, 240.29), and the image resolution is 640×480 pixels. The two-dimensional visual image is obtained by the camera at 1-second intervals during the LiDAR scanning process. A single image cannot cover the occluded bridge structure data obtained by the LiDAR scanning at other locations due to structural occlusion.

[0069] The relative extrinsic parameter matrix obtained from the joint calibration of IMU and lidar is:

[0070]

[0071] The relative extrinsic parameter matrix obtained from the joint calibration of the lidar and camera is:

[0072]

[0073] The camera's intrinsic parameter matrix is:

[0074]

[0075] (2) The motion point spread function (PSF) is constructed using the attitude and motion information of the IMU and input into the deep deblurring network DeblurGAN-v2 algorithm for spatiotemporal characteristic correction and sharpness enhancement.

[0076] This step specifically includes:

[0077] (2.1) The camera rotation trajectory is calculated by integrating the IMU angular velocity during the exposure period, and the translation is estimated by integrating the acceleration, providing motion estimation priors for image processing;

[0078] (2.2) Calculate the trajectory of each pixel during exposure based on the IMU motion estimation results, construct the motion point spread function (PSF) for deblurred image enhancement, and consider the line time offset of the rolling shutter;

[0079]

[0080] In the formula, Let be the spread function of the moving point; Exposure time; For two-dimensional Dirac functions; These are the imaging points of the original image; For ideal imaging points; The projection of camera motion onto the image plane; This is the camera intrinsic parameter matrix; The three-dimensional coordinates of a point in the scene; For camera rotation matrix; Let be the camera displacement vector.

[0081] (2.3) The point spread function is input into the generator of the DeblurGAN-v2 deblurring network. The network corrects the blur based on the known motion information. The generator generates a clear image based on the prior information, and the discriminator evaluates the generated image and optimizes the generator's output. Finally, the deblurred image is output. The discriminator loss function adopts RaGAN-LS. In this embodiment, the deblurring process and results of a certain image are as follows: Figure 3 As shown. In the formula, For the RaGAN-LS discriminator loss; The expected value output by the real image discriminator; To generate the desired output value of the image discriminator; The output of the discriminator represents the degree of truthfulness. Output to the generator; The generator's total loss function; For pixel loss; Loss of "content"; To combat the losses.

[0082] (3) The image after deblurring is processed by the pre-set image enhancement network EnlightenGAN model, and the local dark areas and highlights are adaptively enhanced to obtain an image with clear features and uniform illumination.

[0083] This step specifically includes:

[0084] (3.1) Input the image after deblurring network processing into the preset image enhancement network EnlightenGAN model, and perform size normalization and pixel value normalization preprocessing on the input image;

[0085] (3.2) The generator in the EnlightenGAN model is used to extract multi-scale illumination and texture features of the input image, and the hierarchical representation of local illumination information and spatial feature preservation are realized based on the encoder-decoder structure;

[0086]

[0087]

[0088]

[0089] In the formula, The overall loss function; The loss function is used to preserve global self-features. The loss function is used to preserve local self-features. The global generator loss function; The loss function is the local generator function. The loss function is used to preserve the self-features. Input a low-light image; The dimension of the extracted feature map; This is the i-th max-pooling layer; It is the j-th convolutional layer following the i-th max pooling layer; These are feature maps extracted from a pre-trained model. Enhanced output for the generator; The desired output of the image generator; The expected output of the realistic image generator; The output of the relative discriminator is the probability difference between the generated image and the real image. The desired output is the local distribution of the cropped image.

[0090] (3.3) The brightness distribution of the input image is adaptively adjusted by a pre-trained illumination mapping function, which performs brightness enhancement and detail enhancement on dark areas and brightness suppression and contrast balancing on bright areas to achieve regional illumination balance.

[0091] (3.4) Perform gamma correction and color balance processing on the enhancement results, and output the image after illumination adaptive enhancement to obtain image results with clear features, uniform brightness and natural fidelity.

[0092] (4) Extract stable feature points from the processed image and perform multi-frame matching. Estimate the corresponding pose of the image using the lidar-vision-IMU fusion method and perform three-dimensional reconstruction of the image.

[0093] This step specifically includes:

[0094] (4.1) Feature extraction is performed on the processed image sequence. The scale-invariant feature transform (SIFT) algorithm is used to extract local key points and their descriptors. Simultaneously, lidar point cloud and IMU inertial measurement data are read to form a spatiotemporally aligned multi-sensor dataset.

[0095] (4.2) Based on the extrinsic calibration results, the image features and point cloud features are spatially registered. The nearest neighbor feature matching algorithm (NDT) and feature consistency constraints are used to establish the correspondence between the vision and the point cloud, and the relative pose of the camera and the lidar is initially estimated.

[0096] (4.3) Construct a joint factor graph model of vision-lidar-IMU, with pose as the node, visual feature observation, point cloud matching residual and IMU integral result as the constraint edge, and use the sliding window optimization algorithm to realize multi-source information fusion positioning to obtain high-precision camera motion trajectory;

[0097] (4.4) After obtaining the camera pose optimized by multi-source fusion, the camera pose is used to generate a sparse point cloud through triangulation. The multi-view stereo vision (MVS) method is used to calculate the three-dimensional information of each pixel in the image according to the correspondence to realize the reconstruction of the dense point cloud of the bridge.

[0098] (5) Establish a mapping model between three-dimensional laser point cloud and two-dimensional visual image, use RANSAC combined with RegionGrowing to obtain preliminary geometric region, use DBSCAN algorithm to perform cluster optimization on beam and slab and pier components, and generate geometric prior region in image plane through mapping model.

[0099] This step specifically includes:

[0100] (5.1) The mapping model between the three-dimensional laser point cloud and the two-dimensional visual image is as follows:

[0101]

[0102]

[0103] In the formula, These are the pixel coordinates in the pixel coordinate system. , This refers to the extrinsic parameters of the lidar in the world coordinate system. , These are roll angle, pitch angle, and yaw angle, respectively. The coordinates of the lidar in the world coordinate system during image acquisition; , The relative extrinsic parameters obtained from the joint calibration of the lidar and camera. , Let be the focal length of the camera along the x-axis and y-axis in the pixel coordinate system, respectively. , The coordinates of the camera's optical center in the pixel coordinate system. These are the coordinates of a point in the world coordinate system.

[0104] The extrinsic parameter matrix and camera intrinsic parameter matrix obtained by the joint calibration of the lidar and vision camera are shown in step (1).

[0105] (5.2) The Random Sample Consensus (RANSAC) algorithm is used to perform global plane fitting and outlier removal on the point cloud. The normal vector and curvature features of the fitting residual points are used to perform local segmentation using the region growing algorithm based on normal similarity and spatial adjacency relationship, so as to realize the preliminary geometric division of bridge beams, piers and curved components.

[0106] (5.3) Cluster analysis and boundary optimization are performed on the point cloud region after geometric segmentation. The density-based spatial clustering algorithm (DBSCAN) is used to automatically generate connected point cloud clusters based on the neighborhood density and spatial reachability of the point cloud, so as to realize the automatic clustering and division of beams, slabs and piers.

[0107] (5.4) Project each cluster of points onto the two-dimensional image plane, calculate the pixel coordinates of each three-dimensional point using the mapping model established in (5.1), and generate the corresponding geometric region mask according to the projection distribution of the cluster.

[0108] (6) The YOLOv11-Seg model is used as the segmentation backbone network to perform component-level instance segmentation of the bridge supports in the image. The segmentation results are re-filtered by combining the geometric prior region and post-processed by the conditional random field to obtain continuous and accurate component segmentation results.

[0109] This step specifically includes:

[0110] (6.1) A pre-trained YOLOv11-Seg network is used as the instance segmentation backbone network to perform pixel-level segmentation of the support components in the image and obtain the segmentation mask of the support region.

[0111] (6.2) Using the prior regions of beams and piers generated in step (5.4) as geometric constraints, the support segmentation results are assisted in screening. In the region adjacent to the boundary of the prior regions of beams and piers, the support pixels with strong spatial correlation are retained based on the consistency of point cloud projection density and depth. If outside the prior region, the support pixels with confidence levels lower than the preset threshold are downsampled and filtered out to suppress false detection and boundary diffusion.

[0112] (6.3) The re-selected mask is used as the initial energy term to input the fully connected conditional random field (CRF) to construct a similarity term based on pixel RGB color, depth value and spatial position. The mean field approximation algorithm is used for iterative optimization to obtain the final support segmentation result with continuous boundaries and accurate details.

[0113]

[0114]

[0115] In the formula, The capability function of a fully connected conditional random field; A collection of pixel tags; This represents the energy at a single point; a higher value indicates a higher probability that the pixel belongs to a support. It uses two-point energy, combining position, color, and depth, to enhance boundary continuity and detail precision; These are pixel coordinates; RGB values ​​of pixels; For pixel depth; when , Otherwise, it is 0; These are the weights and Gaussian kernel parameters.

[0116] (7) The image is filtered by the semantic tags of the components. YOLOv11 combined with Deeplabv3+ is called to identify the defects in the corresponding beam, slab and pier concrete component areas. YOLOv11 based on attention mechanism is called to identify and classify the defects in the corresponding rubber bearing component areas. Deeplabv3+ is called to generate a segmentation mask for the bearing.

[0117] In this embodiment, the identification effect of various diseases is as follows: Figure 4 As shown.

[0118] This step specifically includes:

[0119] (7.1) In view of the differences in structural characteristics and disease morphology of beams, slabs, piers and bearings, datasets were constructed and targeted disease identification models were trained. A total of 3751 bridge disease images were selected in this example dataset. Among them, the apparent diseases of beams and slabs include four types of diseases: cracks, spalling, exposed reinforcement and water seepage; the apparent diseases of piers include three types of diseases: cracks, spalling and exposed reinforcement; and the diseases of bearings include three types of diseases: aging, shear deformation and cracking.

[0120] (7.2) For the selected concrete component areas such as beams, slabs and piers, the YOLOv8 network is used to detect defects such as cracks, spalling, and exposed reinforcement, and the Deeplabv3+ network is used for defect segmentation. The YOLOv11 detection network is used to quickly locate candidate areas of typical concrete surface defects such as cracks, spalling, and exposed reinforcement in the target area, and the Deeplabv3+ segmentation network is used to perform fine segmentation of the defect boundaries in the candidate areas to obtain pixel-level defect masks. In this embodiment, the preset threshold of YOLOv11 is set to 70%.

[0121] (7.3) For the selected rubber bearing component area, YOLOv11 based on attention mechanism enhancement is called to detect and classify rubber aging, shear deformation, cracking and other defects, and Deeplabv3+ network is called for segmentation of crack defects; Among them, the YOLOv11 backbone network introduces the attention module CBAM (Convolutional Block AttentionModule), which includes channel attention module and spatial attention module, to enhance the model’s feature response to small cracks, aging discoloration and void defects on the rubber bearing surface. The Deeplabv3+ segmentation network is used to perform fine segmentation of the defect boundary in the candidate area to obtain pixel-level defect mask. In this example, the preset threshold of YOLOv11 is set to 70%.

[0122]

[0123] In the formula, This is the output of the channel attention module; This is the output of the spatial attention module; Input feature map; Use the Sigmoid activation function; It is a multilayer perceptron; For global average pooling; This is a max pooling operation; It is a 7×7 convolution; The average pooling and max pooling results are spliced ​​along the channel axis.

[0124] (8) Project the three-dimensional laser point cloud of the bridge onto the two-dimensional image plane, filter and segment the points in the mask and transform them to the world coordinate system. After filtering the error by ray tracing, obtain the real three-dimensional point cloud of the disease. At the same time, generate a depth map, calculate the spatial coordinates and physical size of the disease based on the depth map, and visualize it in the three-dimensional point cloud.

[0125] This step specifically includes:

[0126] (8.1) Using the mapping model constructed in step (5.1), the points in the disease segmentation mask are transformed to the world coordinate system to obtain the three-dimensional point cloud of potential diseases, and ray tracing is used to filter error points to obtain the real three-dimensional point cloud of diseases.

[0127] In the formula, These are the pixel coordinates in the pixel coordinate system. , This refers to the extrinsic parameters of the lidar in the world coordinate system. , The relative extrinsic parameters obtained from the joint calibration of the lidar and camera. , Let be the focal length of the camera along the x-axis and y-axis in the pixel coordinate system, respectively. , The coordinates of the camera's optical center in the pixel coordinate system. These are the coordinates of a point in the world coordinate system.

[0128] (8.2) Assign the depth information of the real disease 3D point cloud to the corresponding pixel in the 2D visual image. For the pixel that lacks depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the disease segmentation mask area.

[0129] (8.3) Based on the depth map, the true coordinates and physical dimensions of the disease are quantitatively calculated, and the physical area of ​​the disease is calculated according to the following formula:

[0130] In the formula, This refers to the physical area of ​​the disease. For the set of pixels of the segmentation mask, This indicates the depth of the corresponding point in the disease point cloud depth map. f represents the pixel coordinates in the pixel coordinate system. x f y Let be the focal length of the camera along the x-axis and y-axis in the pixel coordinate system, respectively;

[0131] The physical length of the disease is calculated using the following formula:

[0132] In the formula, The physical length of the disease. To segment the pixel length of the defect in the mask, Indicates the arc length along the skeleton line;

[0133] The physical width of the disease is calculated using the following formula:

[0134] In the formula, The physical width of the disease. To segment the pixel width of the defects in the mask, This indicates the direction perpendicular to the skeleton line.

[0135] (9) Generate spatial distribution information of each component and defect based on the dense point cloud model, project the obtained three-dimensional coordinates of the defect onto the existing BIM model coordinate system, realize automatic calibration and structural update of the BIM model, and embed the identified defect information into the BIM model in the form of additional attributes to form a dynamically updated digital bridge management model, providing accurate spatiotemporal information support for subsequent bridge inspection, evaluation and maintenance.

[0136] In this embodiment, the distribution of defects on the bridge underside is as follows: Figure 5 Taking crack-5 as an example, its actual physical length is 268.93 cm, its average physical width is 2.96 cm, and its physical area is 849.59 cm². 2 .

[0137] Example 2

[0138] This invention provides an unmanned aerial vehicle (UAV) laser and vision fusion inspection system for detecting bridge under-bridge defects, comprising:

[0139] The multi-source data synchronous acquisition and registration module is used to jointly calibrate and synchronize the lidar, camera and IMU, and use the calibrated equipment to acquire synchronous data to obtain the three-dimensional laser point cloud, two-dimensional visual image and inertial navigation data of the bridge appearance; at the same time, a mapping model between the three-dimensional laser point cloud and the two-dimensional visual image is established, and the three-dimensional laser point cloud of the bridge appearance is projected onto the two-dimensional visual image plane to generate a front view of the point cloud through the mapping model.

[0140] The image enhancement module is used to deblur the acquired image with motion blur using a deblurring network, and to adaptively adjust the brightness and sharpness of the image using an image enhancement network.

[0141] The component segmentation module is used to identify components in a two-dimensional visual image using a preset image component segmentation model and point cloud clustering algorithm, and generate a pixel-level segmentation mask for each component.

[0142] The disease identification module is used to identify diseases in two-dimensional visual images using a preset target detection model, and to generate a pixel-level segmentation mask for each disease.

[0143] The bridge model generation module is used to reconstruct the bridge structure with high precision by combining multi-view image data with laser point clouds, and to complete the model through point cloud registration.

[0144] The disease model generation module is used to filter out points located within the segmentation mask from the front view of the point cloud, and use ray tracing to filter out error points caused by local occlusion in the potential disease 3D point cloud to obtain the disease 3D point cloud; the depth information of the real disease 3D point cloud is assigned to the corresponding pixels in the 2D visual image, wherein, for pixels lacking depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the disease segmentation mask region; and the physical parameters of the disease are calculated based on the depth map.

[0145] The defect display module is used to display the actual 3D point cloud data of defects and the physical dimensions of defects in the 3D laser point cloud data of the bridge.

[0146] The multi-source data synchronous acquisition and registration module specifically includes:

[0147] The sensor registration module is used to perform joint calibration and time synchronization of lidar, camera and IMU, and to build a mapping model between 3D lidar point cloud and 2D visual image;

[0148]

[0149]

[0150] In the formula, These are the pixel coordinates in the pixel coordinate system. , This refers to the extrinsic parameters of the lidar in the world coordinate system. , These are roll angle, pitch angle, and yaw angle, respectively. The coordinates of the lidar in the world coordinate system during image acquisition; , The relative extrinsic parameters obtained from the joint calibration of the lidar and camera. , Let be the focal length of the camera along the x-axis and y-axis in the pixel coordinate system, respectively. , The coordinates of the camera's optical center in the pixel coordinate system. These are the coordinates of a point in the world coordinate system.

[0151] The data synchronization acquisition module is used to perform synchronous acquisition using calibrated and registered sensor devices.

[0152] The image enhancement module specifically includes:

[0153] The motion blur removal module is used to estimate the camera's motion trajectory during exposure by using the angular velocity and acceleration information of the IMU sensor during image acquisition, and then uses the DeblurGAN-v2 model to achieve deblurred image enhancement based on this.

[0154] The adaptive illumination module is used to achieve adaptive enhancement of local and global illumination based on the deblurred image through the EnlightenGAN image enhancement network model.

[0155] Specifically, the component identification module includes:

[0156] The point cloud component segmentation module is used to obtain the preliminary geometric region based on the corresponding mapping model by using RANSAC combined with RegionGrowing, and to perform clustering optimization on beam, slab and pier components using the DBSCAN algorithm to form the preliminary point cloud component segmentation effect.

[0157] The image component segmentation module uses the YOLOv11-Seg network as an instance to segment the backbone network, performs pixel-level segmentation of the supporting components in the image, and optimizes it by combining prior geometric regions.

[0158] Specifically, the bridge model generation module includes:

[0159] The camera pose acquisition module is used to extract stable feature points from the processed image and perform multi-frame matching, and to estimate the corresponding pose of the image through the LiDAR-Vision-IMU fusion method.

[0160] The bridge 3D model reconstruction module is used to calculate the 3D information of each pixel in the image based on the correspondence relationship using a multi-view stereo vision method to achieve dense point cloud reconstruction of the bridge.

[0161] The disease model generation module specifically includes:

[0162] The disease 3D point cloud extraction module is used to transform the points in the disease segmentation mask to the world coordinate system to obtain the potential disease 3D point cloud, and use ray tracing to filter error points to obtain the real disease 3D point cloud;

[0163]

[0164] The disease depth map generation module is used to assign the depth value of each real disease 3D point to the corresponding pixel in the 2D visual image. For pixels that lack depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the disease segmentation mask area.

[0165] The disease physical parameter quantification module is used to quantify and calculate the true coordinates and physical dimensions of the disease based on the depth map, and calculate the physical area of ​​the disease according to the following formula:

[0166] In the formula, This refers to the physical area of ​​the disease. For the set of pixels of the segmentation mask, This indicates the depth of the corresponding point in the disease point cloud depth map. f represents the pixel coordinates in the pixel coordinate system. x f y Let be the focal length of the camera along the x-axis and y-axis in the pixel coordinate system, respectively;

[0167] The physical length of the disease is calculated using the following formula:

[0168] In the formula, The physical length of the disease. To segment the pixel length of the defect in the mask, Indicates the arc length along the skeleton line;

[0169] The physical width of the disease is calculated using the following formula:

[0170] In the formula, The physical width of the disease. To segment the pixel width of the defects in the mask, This indicates the direction perpendicular to the skeleton line.

Claims

1. A UAV laser and vision fusion inspection method for detecting bridge under-bridge defects, characterized in that, Includes the following steps: (1) Jointly calibrate the lidar, camera and IMU and complete time synchronization, and use the calibrated lidar and camera to collect synchronous data to obtain the three-dimensional laser point cloud and two-dimensional visual image of the bridge surface. (2) Using the attitude and motion information of the IMU, a motion point spread function is constructed and input into the deep deblurring network DeblurGAN-v2 algorithm for spatiotemporal characteristic correction and sharpness enhancement to obtain the deblurred image; (3) The pre-set image enhancement network EnlightenGAN model is used to process the deblurred image, and adaptive enhancement is performed on the local dark areas and highlight areas to obtain an image with clear features and uniform illumination; (4) Extract stable feature points from images with clear features and uniform illumination and perform multi-frame matching. Estimate the corresponding pose of the image through the lidar-vision-IMU fusion method and perform three-dimensional reconstruction of the dense point cloud image of the bridge. (5) Establish a mapping model between three-dimensional laser point cloud and two-dimensional visual image. Use random sampling consistency algorithm combined with region growing algorithm based on normal similarity and spatial adjacency to obtain preliminary geometric region. Cluster optimization is performed on beam and slab and pier components. Generate prior geometric regions of beam and slab and pier in image plane through mapping model. (6) The YOLOv11-Seg model is used as the segmentation backbone network to perform component-level instance segmentation of the bridge supports in the image. The segmentation results are re-filtered by combining the geometric prior region and post-processed by the conditional random field to obtain continuous and accurate component segmentation results. (7) Filter the images by component semantic tags, identify defects in the corresponding beam, slab and pier concrete component areas, and generate segmentation masks; (8) Project the three-dimensional laser point cloud of the bridge onto the two-dimensional image plane, filter and segment the points in the mask and transform them to the world coordinate system. After filtering the error by ray tracing, obtain the real three-dimensional point cloud of the disease. At the same time, generate a depth map, calculate the spatial coordinates and physical size of the disease based on the depth map, and visualize it in the three-dimensional point cloud.

2. The UAV laser and vision fusion inspection method for detecting bridge under-bridge defects according to claim 1, characterized in that: Step (2) specifically includes: (2.1) The camera rotation trajectory is calculated by integrating the IMU angular velocity during the exposure period, and the translation is estimated by integrating the acceleration, providing motion estimation priors for image processing; (2.2) Calculate the trajectory of each pixel during the exposure based on the IMU motion estimation results, construct the motion point spread function for deblurred image enhancement, and consider the line time offset of the rolling shutter; (2.3) The point spread function is input into the generator of the DeblurGAN-v2 network. The network corrects the blur based on the known motion information. The generator generates a clear image based on the prior. The discriminator evaluates the generated image and optimizes the generator's output. Finally, the deblurred image is output.

3. The UAV laser and vision fusion inspection method for detecting bridge under-bridge defects according to claim 1, characterized in that: Step (3) specifically includes: (3.1) Input the image after deblurring network processing into the preset image enhancement network EnlightenGAN model, and perform size normalization and pixel value normalization preprocessing on the input image; (3.2) The generator in the EnlightenGAN model is used to extract multi-scale illumination and texture features of the input image, and the hierarchical representation of local illumination information and spatial feature preservation are realized based on the encoder-decoder structure; (3.3) The brightness distribution of the input image is adaptively adjusted by a pre-trained illumination mapping function, which performs brightness enhancement and detail enhancement on dark areas and brightness suppression and contrast balancing on bright areas to achieve regional illumination balance. (3.4) Perform gamma correction and color balance processing on the enhancement results, and output the image after illumination adaptive enhancement to obtain image results with clear features, uniform brightness and natural fidelity.

4. The UAV laser and vision fusion inspection method for detecting bridge under-bridge defects according to claim 1, characterized in that: Step (4) specifically includes: (4.1) Feature extraction is performed on the processed image sequence. The scale-invariant feature transformation algorithm is used to extract local key points and their descriptors. Simultaneously, the lidar point cloud and IMU inertial measurement data are read to form a spatiotemporally aligned multi-sensor dataset. (4.2) Based on the extrinsic parameter calibration results, the image features and point cloud features are spatially registered. The correspondence between the vision and the point cloud is established by using the nearest neighbor feature matching algorithm and feature consistency constraints. The relative pose of the camera and the lidar is estimated in advance. (4.3) Construct a joint factor graph model of vision-lidar-IMU, with pose as the node, visual feature observation, point cloud matching residual and IMU integral result as the constraint edge, and use the sliding window optimization algorithm to realize multi-source information fusion positioning to obtain high-precision camera motion trajectory; (4.4) After obtaining the camera pose optimized by multi-source fusion, the camera pose is used to generate a sparse point cloud through triangulation. A multi-view stereo vision method is used to calculate the three-dimensional information of each pixel in the image according to the correspondence to realize the three-dimensional reconstruction of the dense point cloud image of the bridge.

5. The UAV laser and vision fusion inspection method for detecting bridge under-bridge defects according to claim 1, characterized in that: Step (5) specifically includes: (5.1) Establish a mapping model between three-dimensional laser point clouds and two-dimensional visual images; (5.2) The random sampling consensus algorithm is used to perform global plane fitting and outlier removal on the point cloud. The normal vector and curvature features of the fitting residual points are calculated. The region growing algorithm based on normal similarity and spatial adjacency is used for local segmentation to achieve the preliminary geometric division of bridge beams, piers and curved components. (5.3) Cluster analysis and boundary optimization are performed on the point cloud region after geometric segmentation. A density-based spatial clustering algorithm is used to automatically generate connected point cloud clusters based on the neighborhood density and spatial reachability of the point cloud, so as to realize the automatic clustering and division of beams, slabs and piers. (5.4) Project each cluster of points onto the two-dimensional image plane, calculate the pixel coordinates of each three-dimensional point using the mapping model established in step (5.1), and generate the corresponding geometric prior region mask based on the projection distribution of the cluster.

6. The UAV laser and vision fusion inspection method for detecting bridge under-bridge defects according to claim 5, characterized in that: Step (6) specifically includes: (6.1) A pre-trained YOLOv11-Seg network is used as the instance segmentation backbone network to perform pixel-level segmentation of the support components in the image and obtain the segmentation mask of the support region. (6.2) Using the prior geometric regions of beams and piers generated in step (5) as geometric constraints, the support segmentation results are assisted in screening. In the region adjacent to the boundary of the prior regions of beams and piers, the support pixels with strong spatial correlation are retained based on the consistency of point cloud projection density and depth. If outside the prior region, the support pixels with confidence levels lower than the preset threshold are downsampled and filtered out to suppress false detection and boundary diffusion. (6.3) The re-selected mask is used as the initial energy term to input the fully connected conditional random field to construct a similarity term based on pixel RGB color, depth value and spatial position. The mean field approximation algorithm is used for iterative optimization to obtain the final support segmentation result with continuous boundaries and accurate details.

7. The UAV laser and vision fusion inspection method for detecting bridge under-bridge defects according to claim 1, characterized in that: Step (7) specifically includes: (7.1) Based on the differences in structural characteristics and disease morphology of beams, slabs, piers and bearings, datasets were constructed and targeted disease identification models were trained. (7.2) For the selected concrete component areas such as beams, slabs and piers, the YOLOv8 network is called to detect defects such as cracks, spalling and exposed reinforcement, and the Deeplabv3+ network is used for defect segmentation. The YOLOv11 detection network is used to quickly locate candidate areas of typical concrete surface defects such as cracks, spalling and exposed reinforcement in the target area, and the Deeplabv3+ segmentation network is used to perform fine segmentation of the defect boundaries in the candidate areas to obtain pixel-level defect masks. (7.3) For the selected rubber support component area, YOLOv11 based on attention mechanism enhancement is called to detect and classify rubber aging, shear deformation, cracking and other defects, and Deeplabv3+ network is called to segment crack defects.

8. The UAV laser and vision fusion inspection method for detecting bridge under-bridge defects according to claim 7, characterized in that: The YOLOv11 backbone network introduces an attention module CBAM, which includes a channel attention module and a spatial attention module, to enhance the model's response to the features of micro-cracks, aging discoloration, and voids on the rubber bearing surface. The Deeplabv3+ segmentation network is used to perform fine segmentation of the disease boundaries in the candidate region to obtain a pixel-level disease mask.

9. The UAV laser and vision fusion inspection method for detecting bridge under-bridge defects according to claim 1, characterized in that: The step (8) is followed by the following steps: (9) Based on the dense point cloud image model of the bridge in step (4), generate the spatial distribution information of each component and the defects. Project the three-dimensional coordinates of the defects obtained in step (8) into the existing BIM model coordinate system to realize the automatic calibration and structural update of the BIM model. In this way, the identified defect information is embedded into the BIM model in the form of additional attributes to form a digital bridge management model that can be dynamically updated.

10. A UAV laser and vision fusion inspection system for detecting bridge under-bridge defects, characterized in that, include: The multi-source data synchronous acquisition and registration module is used to jointly calibrate the lidar, camera and IMU, and after calibration, synchronously acquire the 3D point cloud, 2D image and inertial navigation data of the bridge. At the same time, it establishes a mapping model between the point cloud and the image to realize the projection of the 3D point cloud onto the 2D image plane and the generation of the front view. The image enhancement module is used to deblur the acquired image with motion blur using a deblurring network, and to adaptively adjust the brightness and sharpness of the image using an image enhancement network. The component segmentation module is used to identify components in a two-dimensional visual image using a preset image component segmentation model and point cloud clustering algorithm, and generate a pixel-level segmentation mask for each component. The disease identification module is used to identify diseases in two-dimensional visual images using a preset target detection model, and to generate a pixel-level segmentation mask for each disease. The bridge model generation module is used to reconstruct the bridge structure with high precision by combining multi-view image data with laser point clouds, and to complete the model through point cloud registration. The disease model generation module is used to filter out points located within the segmentation mask from the front view of the point cloud, and use ray tracing to filter out error points caused by local occlusion in the potential disease 3D point cloud to obtain the disease 3D point cloud; the depth information of the real disease 3D point cloud is assigned to the corresponding pixels in the 2D visual image, wherein, for pixels lacking depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the disease segmentation mask region; and the physical parameters of the disease are calculated based on the depth map. The defect display module is used to display the actual 3D point cloud data of defects and the physical dimensions of defects in the 3D laser point cloud data of the bridge.

Images