Bridge apparent disease detection method and system based on laser point cloud and visual image fusion

Abstract

The application discloses a bridge apparent disease detection method and system based on laser point cloud and visual image fusion, and the method comprises the following steps: synchronous data acquisition is carried out by using a calibrated laser radar and a camera to obtain three-dimensional laser point cloud and two-dimensional visual image of the bridge appearance; three-dimensional laser point cloud is projected to the two-dimensional visual image plane to generate a point cloud front view through a mapping model; a preset target detection model is used to identify diseases in the two-dimensional visual image, and a segmentation mask is generated for each disease; points located in the segmentation mask are screened out from the point cloud front view to obtain potential disease three-dimensional point cloud; light ray tracing is used to filter error points in the potential disease three-dimensional point cloud due to local occlusion to obtain real disease three-dimensional point cloud; the depth information of the real disease three-dimensional point cloud is distributed to the corresponding pixel points to generate a depth map of the disease segmentation mask area, and then the physical size of the disease is quantitatively calculated; and the real disease three-dimensional point cloud data and the physical size of the disease are displayed. The application has high efficiency and high precision.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a method and system for detecting apparent bridge defects based on the fusion of laser point cloud and visual images. Background Art

[0002] During the long service life of a bridge, various factors can cause surface defects such as concrete cracks, spalling, and exposed rebar, posing a serious threat to its safety. Detecting and locating these surface defects facilitates timely implementation of targeted maintenance measures, extending its service life. Traditional inspection methods rely on costly and inefficient manual inspections, with information primarily recorded in text form, which lacks intuitiveness. With technological advancements, computer vision is becoming a mainstream method for bridge defect detection.

[0003] However, current computer vision-based defect detection methods rely on a single type of data, either laser point clouds or visual images, which presents certain limitations. Laser scanners can quickly reconstruct high-precision point cloud models of bridges, but due to the limited resolution of the instrument, they struggle to depict tiny cracks or damage on the surface of the structure. Furthermore, point cloud-based defect detection methods are still immature. In contrast, visual images can capture high-resolution texture features of the surface, and visual image-based defect detection techniques have high recognition rates. However, localizing defects identified in visual images to the entire bridge model requires the use of 3D reconstruction algorithms, which places high demands on image acquisition. For example, patents CN115797350B and CN117710810A both use visual images for identification and 3D reconstruction. However, factors such as image overlap, field of view size, and the number of feature points in these methods affect the accuracy of identification and localization, resulting in relatively low computational efficiency. Furthermore, both of these documents directly use visual images for 3D reconstruction, but because visual images lack depth information, the identified defects also lack depth information, making accurate quantification of the defects impossible. Summary of the Invention

[0004] In view of the problems existing in the prior art, the purpose of the present invention is to provide a bridge surface defect detection method and system based on the fusion of laser point cloud and visual image, which has high precision, high efficiency and can accurately quantify the defects.

[0005] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0006] A bridge surface defect detection method based on laser point cloud and visual image fusion includes the following steps:

[0007] (1) Jointly calibrate the LiDAR and camera, and use the calibrated LiDAR and camera for synchronous data acquisition to obtain a three-dimensional laser point cloud and a two-dimensional visual image of the bridge surface;

[0008] (2) Establish a mapping model between the 3D laser point cloud and the 2D visual image, and project the 3D laser point cloud of the bridge onto the 2D visual image plane through the mapping model to generate a point cloud front view;

[0009] (3) using a preset target detection model to identify defects in the two-dimensional visual image and generating a pixel-level segmentation mask for each defect, wherein the preset target detection model is a trained neural network model;

[0010] (4) selecting points within the segmentation mask from the front view of the point cloud, converting the selected points into a world coordinate system, and obtaining a three-dimensional point cloud of potential defects;

[0011] (5) Using ray tracing to filter out the error points caused by local occlusion in the potential defect 3D point cloud to obtain the real defect 3D point cloud;

[0012] (6) Assigning the depth information of the real disease 3D point cloud to the corresponding pixel points in the 2D visual image. For the pixel points that lack depth information, the depth of the point is estimated by weighted linear interpolation to generate a depth map of the disease segmentation mask area;

[0013] (7) Quantify the physical size of the disease based on the depth map;

[0014] (8) Display the actual three-dimensional point cloud data of the defect and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

[0015] Furthermore, the mapping model between the three-dimensional laser point cloud and the two-dimensional visual image in step (2) is:

[0016]

[0017] Where (u, v) is the pixel coordinate in the pixel coordinate system; R l 、T l is the external parameter of the laser radar in the world coordinate system, α, β, and γ are the roll angle, pitch angle, and yaw angle respectively; C X 、C Y 、C Z is the coordinate of the laser radar in the world coordinate system during image acquisition; R lc 、T lc is the relative external parameter obtained by joint calibration of lidar and camera, f x 、f y is the focal length of the camera in the x-axis and y-axis directions in the pixel coordinate system, cx 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

[0018] Furthermore, step (3) specifically includes:

[0019] (3.1) using a preset target detection model to identify the disease in the two-dimensional visual image and obtain a bounding box and confidence score of the disease; wherein the preset target detection model is a trained neural network model;

[0020] (3.2) Deleting the boundary boxes of the diseases whose confidence level is less than the preset threshold;

[0021] (3.3) Using a preset instance segmentation model, the pixels within the bounding box processed in step (3.2) are segmented to distinguish the diseased area from the non-diseased area, and a segmentation mask of the diseased area is generated, wherein the preset instance segmentation model is a trained neural network model.

[0022] Furthermore, step (4) specifically includes:

[0023] (4.1) Filter out points within the segmentation mask from the front view of the point cloud as defect points I = {(u, v)∈A} in the pixel coordinate system, where A represents the segmentation mask and (u, v) represents the pixel coordinates in the pixel coordinate system;

[0024] (4.2) The defect points are converted from the pixel coordinate system to the world coordinate system according to the following formula to obtain the potential defect 3D point cloud;

[0025]

[0026] Where (u, v) is the pixel coordinate in the pixel coordinate system, R l 、T l is the external parameter of the laser radar in the world coordinate system, R lc 、T lc is the relative external parameter obtained by joint calibration of lidar and camera, f x 、f y is the focal length of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

[0027] Furthermore, step (5) specifically includes:

[0028] (5.1) In the world coordinate system, calculate the straight line between the camera optical center and each point in the potential defect 3D point cloud;

[0029] (5.2) For each line, calculate the spatial angle between it and all remaining lines, and select the points whose spatial angle is less than the angle threshold as candidate disease points, where the angle threshold is determined by the spatial resolution of the lidar;

[0030] (5.3) Calculate the depth of all candidate damage points based on the elevation information of the 3D laser point cloud, and calculate the average value of the depth of all candidate damage points;

[0031] (5.4) The candidate defect points with depth less than or equal to the average value are regarded as real defect points, and all real defect points are formed into a real defect three-dimensional point cloud.

[0032] Furthermore, step (6) specifically includes:

[0033] (6.1) Assigning the depth value of each real 3D defect point to the corresponding pixel point in the 2D visual image;

[0034] (6.2) If there are pixels without depth information in the segmentation mask of the 2D visual image, then the coverage area of ​​the segmentation mask on the image is expanded by n pixels in each of the four directions (up, down, left, and right). The depth of the pixels with known depth information in the new coverage area is estimated using the weighted linear interpolation formula below, where the value of n is determined by the image resolution:

[0035]

[0036] Where h represents the estimated depth, j U 、h D 、h L 、h R Respectively represents the depth of the pixel closest to the point in the four directions of up, down, left, and right, w U 、w D 、w L 、w R Respectively represents the distance between the pixel closest to the point in the four directions of up, down, left, and right;

[0037] (6.3) Extract the depth information of each point in the segmentation mask to generate a depth map of the disease segmentation mask area.

[0038] Furthermore, step (7) specifically includes:

[0039] (7.1) Based on the defect point cloud depth map and segmentation mask, the physical area of ​​the defect is calculated according to the following formula:

[0040]

[0041] Where A w is the physical area of ​​the disease, A p is the pixel point set of the segmentation mask, h represents the depth of the corresponding point in the depth map of the disease point cloud, (u, v) is the pixel coordinate in the pixel coordinate system, and f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system;

[0042] (7.2) Based on the defect point cloud depth map and segmentation mask, the physical length of the defect is calculated according to the following formula:

[0043]

[0044] Where, l w is the physical length of the lesion, l p is the pixel length of the disease in the segmentation mask, l represents the arc length along the skeleton line;

[0045] (7.3) Based on the defect point cloud depth map and segmentation mask, the physical width of the defect is calculated according to the following formula:

[0046]

[0047] Where w w is the physical width of the lesion, w p is the pixel width of the disease in the segmentation mask, w represents the direction perpendicular to the skeleton line;

[0048] (7.4) The calculated physical area, length, and width of the disease are used as the quantified physical dimensions of the disease.

[0049] Furthermore, step (8) further includes the following steps:

[0050] (9) Map the pixel points in the bridge structure area in the two-dimensional visual image to the world coordinate system to obtain the partial bridge structure point cloud that has been inspected, and calculate its overlap rate with the complete bridge point cloud as the detection coverage rate of this inspection.

[0051] Furthermore, step (9) further includes the following steps:

[0052] (10) Obtain the BIM model of the bridge, align the three-dimensional laser point cloud of the bridge with the BIM model, solve the three-dimensional coordinates of the apparent defects of the bridge in the BIM model coordinate system, attach the segmentation mask to the corresponding surface of the BIM model, integrate the defects into the BIM model as additional information, and mark the parts of the bridge structure that have been inspected as a reference for subsequent bridge inspection and management and maintenance.

[0053] A bridge surface defect detection system based on laser point cloud and visual image fusion, including:

[0054] The data acquisition module is used to jointly calibrate the LiDAR and camera, and use the calibrated LiDAR and camera to synchronously collect data to obtain a 3D laser point cloud and 2D visual image of the bridge surface;

[0055] A mapping module is used to establish a mapping model between the 3D laser point cloud and the 2D visual image, and project the 3D laser point cloud of the bridge surface onto the 2D visual image plane through the mapping model to generate a point cloud front view;

[0056] A disease recognition module is used to identify diseases in the two-dimensional visual image using a preset target detection model, which is a trained neural network model, and generate a pixel-level segmentation mask for each disease;

[0057] A potential defect 3D point cloud generation module is used to filter out points within the segmentation mask from the point cloud front view, and convert the filtered points into a world coordinate system to obtain a potential defect 3D point cloud;

[0058] The real disease 3D point cloud generation module is used to use ray tracing to filter out the error points caused by local occlusion in the potential disease 3D point cloud to obtain the real disease 3D point cloud;

[0059] The defect depth map generation module is used to assign the depth information of the real defect 3D point cloud to the corresponding pixel points in the 2D visual image. For the pixel points that lack depth information, the depth of the point is estimated by weighted linear interpolation to generate the depth map of the defect segmentation mask area;

[0060] The disease size quantification module is used to quantify the physical size of the disease based on the depth map;

[0061] The defect display module is used to display the real defect three-dimensional point cloud data and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) The present invention identifies and locates defects through the steps of point cloud and image data registration and fusion, image target recognition, etc., focusing on the apparent defect characteristics of multiple types of concrete, thereby improving the recognition accuracy and precision;

[0064] (2) The application directly obtains the extrinsic information of the camera and the spatial coordinates of the image pixels by using the laser point cloud data, without using a three-dimensional reconstruction algorithm, has high efficiency, is less affected by factors such as the field of view size, the overlap rate size and the number of feature points of the image, has higher stability in disease identification and positioning, and has a wider application range.

[0065] (3) The physical size of each disease is calculated, accurate quantitative analysis of the disease on the curved surface is realized, and the application range is expanded. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is a process schematic diagram of a bridge apparent disease detection method based on laser point cloud and visual image fusion provided by the embodiment of the application;

[0067] Figure 2 is a coordinate system involved in the mapping model between the three-dimensional laser point cloud and the two-dimensional visual image according to the embodiment of the application;

[0068] Figure 3 is a principle schematic diagram of filtering error points according to the embodiment of the application;

[0069] Figure 4 is a detection result diagram of a crack disease provided by the embodiment of the application;

[0070] Figure 5 is a detection result diagram of a peeling disease provided by the embodiment of the application. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0072] Embodiment one

[0073] The embodiment of the application provides a bridge apparent disease detection method based on laser point cloud and visual image fusion, as shown in the figure, comprising the following steps: Figure 1

[0074] (1) jointly calibrate the laser radar and the camera, and synchronously collect data by using the calibrated laser radar and the camera to obtain three-dimensional laser point cloud and two-dimensional visual image of the bridge appearance.

[0075] The three-dimensional laser point cloud data is obtained by laser radar scanning, and the laser radar can be a mechanical laser radar or a solid-state laser radar. The two-dimensional visual image is obtained by camera shooting. Since the method of the application does not use an image-based three-dimensional reconstruction algorithm, the overlap rate of the image to be collected is not required, therefore, the camera used can be a wide-angle camera with a large field of view, or a long-focus camera with a small field of view.

[0076] ​For example, in this embodiment, a mechanical laser radar is used to collect bridge point cloud data. Its laser field of view is 270°×360°. Using this device, a full range of point cloud models of the bridge can be obtained. The focal length (f x 、f y ) is an industrial camera with (1015.9557, 1014.3076), the camera optical center (c x 、c y ) is (2125.3699, 1056.7953), and the image resolution is 2592 × 1944 pixels. The 2D visual image is acquired by the camera at 1-second intervals during the lidar scanning process. Due to structural occlusion, a single image cannot capture obscured bridge structure data obtained by the lidar scan at other locations.

[0077] (2) A mapping model between the three-dimensional laser point cloud and the two-dimensional visual image is established, and the three-dimensional laser point cloud of the bridge surface is projected onto the two-dimensional visual image plane through the mapping model to generate a point cloud front view.

[0078] Among them, such as Figure 2 As shown in the figure, the mapping model between the three-dimensional laser point cloud and the two-dimensional visual image is:

[0079]

[0080]

[0081] Where (u, v) is the pixel coordinate in the pixel coordinate system; R l 、T l is the external parameter of the laser radar in the world coordinate system, α, β, and γ are the roll angle, pitch angle, and yaw angle respectively; C X 、C Y 、C z is the coordinate of the laser radar in the world coordinate system during image acquisition; R lc 、T lc is the relative external parameter obtained by joint calibration of lidar and camera, f x 、f y is the focal length of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

[0082] For example, in this embodiment, the relative extrinsic parameter matrix obtained by joint calibration of the laser radar and the camera is:

[0083]

[0084] The intrinsic parameter matrix of the camera is:

[0085]

[0086] (3) A preset target detection model is used to identify defects in two-dimensional visual images and generate pixel-level segmentation masks for each defect.

[0087] This step specifically includes:

[0088] (3.1) Using a preset object detection model to identify defects in the two-dimensional visual image and obtain a bounding box and confidence score for the defect; the preset object detection model is a trained neural network model, such as a YOLO series model. In this embodiment, YOLO v8 is used; the defects include concrete cracks, spalling, and exposed rebar;

[0089] (3.2) Deleting the bounding boxes of the disease whose confidence is less than a preset threshold; for example, in this example, the preset threshold is set to 70%;

[0090] (3.3) Using a preset instance segmentation model, the pixels within the bounding box processed in step (3.2) are segmented to distinguish the diseased area from the non-diseased area, and a segmentation mask of the diseased area is generated, wherein the preset instance segmentation model is a trained neural network model, such as a Mask R-CNN model.

[0091] (4) Filter out points within the segmentation mask from the front view of the point cloud, convert the filtered points into the world coordinate system, and obtain a three-dimensional point cloud of potential defects.

[0092] This step specifically includes:

[0093] (4.1) Filter out points within the segmentation mask from the front view of the point cloud as defect points I = {(u, v)∈A} in the pixel coordinate system, where A represents the segmentation mask and (u, v) represents the pixel coordinates in the pixel coordinate system;

[0094] (4.2) The defect points are converted from the pixel coordinate system to the world coordinate system according to the following formula to obtain the potential defect three-dimensional point cloud data;

[0095]

[0096] (5) Use ray tracing to filter out the error points caused by local occlusion in the potential defect three-dimensional point cloud to obtain the real defect three-dimensional point cloud.

[0097] When projecting the point cloud, there is a problem of projecting non-diseased points into the area of ​​the damaged pixels. Therefore, filtering is performed, such as Figure 3 As shown, specifically including:

[0098] (5.1) In the world coordinate system, calculate the straight line between the camera optical center and each point in the potential defect 3D point cloud;

[0099] (5.2) For each line, calculate the spatial angle between it and all remaining lines, and select points whose spatial angle is less than an angle threshold as candidate disease points. The angle threshold is determined based on the spatial resolution of the lidar. For example, in this embodiment, the spatial resolution of the lidar is 0.38°, so the angle threshold is set to 0.4°.

[0100] (5.3) Calculate the depth of all candidate damage points based on the elevation information of the 3D laser point cloud, and calculate the average value of the depth of all candidate damage points;

[0101] (5.4) The candidate defect points with depth less than or equal to the average value are regarded as real defect points, and all real defect points are formed into a real defect three-dimensional point cloud.

[0102] (6) The depth information of the real defect three-dimensional point cloud is assigned to the corresponding pixel points in the two-dimensional visual image. For the pixel points that lack depth information, the depth of the point is estimated by weighted linear interpolation to generate a depth map of the defect segmentation mask area.

[0103] This step specifically includes:

[0104] (6.1) Assigning the depth value of each real 3D defect point to the corresponding pixel point in the 2D visual image;

[0105] (6.2) If there are pixels without depth information within the segmentation mask of the 2D visual image, the coverage area of ​​the segmentation mask on the image is expanded by n pixels in each of the four directions: upward, downward, left, and right. The depth of the pixels with known depth information within the new coverage area is estimated using weighted linear interpolation as shown in the following formula, where the value of n is determined by the image resolution. For example, in this embodiment, the image resolution is 2592×1944, and the unit is pixels, so the value of n is set to 5:

[0106]

[0107] Where h represents the estimated depth, h U 、h D 、h L 、h R Respectively represents the depth of the pixel closest to the point in the four directions of up, down, left, and right, w U 、w D 、w L 、w R Respectively represents the distance between the pixel closest to the point in the four directions of up, down, left, and right;

[0108] (6.3) Extract the depth information of each point in the segmentation mask to generate a depth map of the disease segmentation mask area.

[0109] (7) Based on the depth map, the physical size of the disease is quantitatively calculated.

[0110] This step specifically includes:

[0111] (7.1) Based on the defect point cloud depth map and segmentation mask, the physical area of ​​the defect is calculated according to the following formula:

[0112]

[0113] Where A w is the physical area of ​​the disease, A p is the pixel point set of the segmentation mask, h represents the depth of the corresponding point in the depth map of the disease point cloud, (u, v) is the pixel coordinate in the pixel coordinate system, and f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system;

[0114] (7.2) Based on the defect point cloud depth map and segmentation mask, the physical length of the defect is calculated according to the following formula:

[0115]

[0116] Where, l w is the physical length of the lesion, l p is the pixel length of the disease in the segmentation mask, l represents the arc length along the skeleton line;

[0117] (7.3) Based on the defect point cloud depth map and segmentation mask, the physical width of the defect is calculated according to the following formula:

[0118]

[0119] Where w w is the physical width of the lesion, w p is the pixel width of the disease in the segmentation mask, w represents the direction perpendicular to the skeleton line;

[0120] (7.4) The calculated physical area, length, and width of the disease are used as the quantified physical dimensions of the disease.

[0121] The pixel length and width of concrete cracks are calculated based on the segmentation mask. A skeleton extraction algorithm is used to refine the crack region's segmentation mask into a single-pixel-wide skeleton. The total crack length is calculated by traversing the skeleton pixel by pixel. For each pixel on the skeleton, the distance to the crack edge is measured along its normal direction to calculate the maximum, minimum, and average widths. The pixel area of ​​concrete cracks, spalling, and exposed rebar is calculated based on the defect segmentation mask. The pixel area of ​​the defect is calculated by summing the number of pixels within the defect area.

[0122] In this embodiment, the crack and spalling diseases identified are as follows: Figure 4 and Figure 5 As shown, the pixel length of the crack disease is 212 pixels, the pixel area is 978 pixels, the maximum width is 4.39 pixels, the minimum width is 2.00 pixels, and the average width is 3.57 pixels; the physical length is 32.82 cm, and the physical area is 23.44 cm 2 The maximum width is 0.67 cm, the minimum width is 0.31 cm, and the average width is 0.55 cm. The pixel area of ​​the spalling disease is 353,037 pixels; the physical area is 9,617.23 cm 2 .

[0123] (8) Display the actual three-dimensional point cloud data of the defect and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

[0124] Specifically, the real disease 3D point cloud can be displayed in red, and the others can be displayed in normal non-red colors.

[0125] (9) Map the pixel points in the bridge structure area in the two-dimensional visual image to the world coordinate system to obtain the partial bridge structure point cloud that has been inspected, and calculate its overlap rate with the complete bridge point cloud as the detection coverage rate of this inspection.

[0126] (10) Obtain the BIM model of the bridge, align the three-dimensional laser point cloud of the bridge with the BIM model, solve the three-dimensional coordinates of the apparent defects of the bridge in the BIM model coordinate system, attach the segmentation mask to the corresponding surface of the BIM model, integrate the defects into the BIM model as additional information, and mark the parts of the bridge structure that have been inspected as a reference for subsequent bridge inspection and management and maintenance.

[0127] Example 2

[0128] The embodiment of the present invention provides a bridge surface defect detection system based on the fusion of laser point cloud and visual image, comprising:

[0129] The data acquisition module is used to jointly calibrate the LiDAR and camera, and use the calibrated LiDAR and camera to synchronously collect data to obtain a 3D laser point cloud and 2D visual image of the bridge surface;

[0130] A mapping module is used to establish a mapping model between the 3D laser point cloud and the 2D visual image, and project the 3D laser point cloud of the bridge surface onto the 2D visual image plane through the mapping model to generate a point cloud front view;

[0131] A disease recognition module is used to identify diseases in the two-dimensional visual image using a preset target detection model, which is a trained neural network model, and generate a pixel-level segmentation mask for each disease;

[0132] A potential defect 3D point cloud generation module is used to filter out points within the segmentation mask from the point cloud front view, and convert the filtered points into a world coordinate system to obtain a potential defect 3D point cloud;

[0133] The real disease 3D point cloud generation module is used to use ray tracing to filter out the error points caused by local occlusion in the potential disease 3D point cloud to obtain the real disease 3D point cloud;

[0134] The defect depth map generation module is used to assign the depth information of the real defect 3D point cloud to the corresponding pixel points in the 2D visual image. For the pixel points that lack depth information, the depth of the point is estimated by weighted linear interpolation to generate the depth map of the defect segmentation mask area;

[0135] The disease size quantification module is used to quantify the physical size of the disease based on the depth map;

[0136] The defect display module is used to display the real defect three-dimensional point cloud data and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

[0137] The mapping model between the three-dimensional laser point cloud and the two-dimensional visual image is:

[0138]

[0139] Where (u, v) is the pixel coordinate in the pixel coordinate system; R l 、T l is the external parameter of the laser radar in the world coordinate system, α, β, and γ are the roll angle, pitch angle, and yaw angle respectively; C X 、C Y 、C Z is the coordinate of the laser radar in the world coordinate system during image acquisition; R lc 、T lc is the relative external parameter obtained by joint calibration of lidar and camera, fx 、f y is the focal length of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

[0140] The disease identification module specifically includes:

[0141] A preliminary defect recognition unit is configured to identify defects in a two-dimensional visual image using a preset target detection model, and obtain a bounding box and confidence score for the defect. The preset target detection model is a trained neural network model, such as a YOLO series model. In this embodiment, YOLO v8 is used. Defects include concrete cracks, spalling, and exposed rebar.

[0142] A disease screening unit is used to delete the bounding boxes of diseases whose confidence level is less than a preset threshold. For example, in this example, the preset threshold is set to 70%.

[0143] The segmentation unit is used to segment the pixels within the bounding box processed by the disease screening unit using a preset instance segmentation model, distinguish between diseased areas and non-diseased areas, and generate a segmentation mask for the diseased area, wherein the preset instance segmentation model is a trained neural network model, such as a Mask R-CNN model.

[0144] The potential disease 3D point cloud generation module specifically includes:

[0145] a defect point screening unit, configured to screen out points within the segmentation mask from the front view of the point cloud as defect points I={(u,v)∈A} in a pixel coordinate system, where A represents the segmentation mask and (u,v) represents the pixel coordinates in the pixel coordinate system;

[0146] A coordinate conversion unit is used to convert the defect points from the pixel coordinate system to the world coordinate system according to the following formula to obtain the three-dimensional point cloud data of potential defects;

[0147]

[0148] The real disease 3D point cloud generation module specifically includes:

[0149] A straight line calculation unit is used to calculate the straight line between the camera optical center and each point in the potential defect three-dimensional point cloud in the world coordinate system;

[0150] The candidate defect point calculation unit is used to calculate the spatial angle between each line and all remaining lines, and select points with spatial angles less than an angle threshold as candidate defect points. The angle threshold is determined based on the spatial resolution of the lidar. For example, in this embodiment, the spatial resolution of the lidar is 0.38°, so the angle threshold is set to 0.4°.

[0151] A depth average calculation unit is used to calculate the depth of all candidate disease points based on the elevation information of the three-dimensional laser point cloud, and calculate the average value of the depth of all candidate disease points;

[0152] The real defect point extraction unit is used to take candidate defect points with depths less than or equal to the average value as real defect points, and form a real defect three-dimensional point cloud from all real defect points.

[0153] The disease depth map generation module specifically includes:

[0154] A depth assignment unit, used to assign the depth value of each real disease 3D point to the corresponding pixel point in the 2D visual image;

[0155] The depth estimation unit is configured to, if there are pixels lacking depth information within the segmentation mask of the two-dimensional visual image, expand the coverage area of ​​the segmentation mask on the image by n pixels in each of the four directions of upward, downward, left and right, and estimate the depth of the pixels with known depth information within the new coverage area using weighted linear interpolation in the following formula, where the value of n is determined by the image resolution. For example, in this embodiment, the image resolution is 2592×1944, and the unit is pixels, so the value of n is set to 5:

[0156]

[0157] Where h represents the estimated depth, h U 、h D 、h L 、h R Respectively represents the depth of the pixel closest to the point in the four directions of up, down, left, and right, w U 、w D 、w L 、w R Respectively represents the distance between the pixel closest to the point in the four directions of up, down, left, and right;

[0158] The depth map generating unit is used to extract the depth information of each point in the segmentation mask to generate a depth map of the defect segmentation mask area.

[0159] The disease size quantification module specifically includes:

[0160] The damage area calculation unit is used to calculate the physical area of ​​the damage according to the damage point cloud depth map and segmentation mask according to the following formula:

[0161]

[0162] Where A w is the physical area of ​​the disease, A p is the pixel point set of the segmentation mask, h represents the depth of the corresponding point in the depth map of the disease point cloud, (u, v) is the pixel coordinate in the pixel coordinate system, and f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system;

[0163] The defect length calculation unit is used to calculate the physical length of the defect according to the defect point cloud depth map and segmentation mask according to the following formula:

[0164]

[0165] Where, l w is the physical length of the lesion, l p is the pixel length of the disease in the segmentation mask, l represents the arc length along the skeleton line;

[0166] The defect width calculation unit is used to calculate the physical width of the defect according to the defect point cloud depth map and segmentation mask according to the following formula:

[0167]

[0168] Where w w is the physical width of the lesion, w p is the pixel width of the disease in the segmentation mask, w represents the direction perpendicular to the skeleton line;

[0169] The disease physical size acquisition unit is used to use the calculated physical area, length and width of the disease as the quantified physical size of the disease.

[0170] In addition, the system can also include a coverage calculation module, which is used to map the pixel points in the bridge structure area in the two-dimensional visual image to the world coordinate system, obtain the partial bridge structure point cloud that has been inspected, and calculate its overlap rate with the complete bridge point cloud as the detection coverage of this inspection.

[0171] In addition, the system can further comprise a BIM model labeling module, which is configured to acquire a BIM model of the bridge, register the three-dimensional laser point cloud of the bridge with the BIM model, solve the three-dimensional coordinates of the apparent diseases of the bridge in the coordinate system of the BIM model, attach the segmentation mask to the corresponding surface of the BIM model, integrate the diseases as additional information into the BIM model, and label the detected part of the bridge structure, serving as a reference basis for subsequent bridge detection and management and maintenance.

[0172] The system corresponds to the method of embodiment one, and the details are described in the method of embodiment one, which will not be described herein.

[0173] It should be understood that the above embodiments and descriptions in the specification are only the principles, main features and advantages of the present application, and various changes and improvements can be made without departing from the spirit and scope of the present application, and these changes and improvements all fall within the protection scope of the present application.

Claims

1. A bridge surface defect detection method based on the fusion of laser point cloud and visual image, characterized in that: The steps include: (1) Jointly calibrate the LiDAR and camera, and use the calibrated LiDAR and camera for synchronous data acquisition to obtain a three-dimensional laser point cloud and a two-dimensional visual image of the bridge surface; (2) Establish a mapping model between the 3D laser point cloud and the 2D visual image, and project the 3D laser point cloud of the bridge onto the 2D visual image plane through the mapping model to generate a point cloud front view; (3) using a preset target detection model to identify defects in the two-dimensional visual image and generating a pixel-level segmentation mask for each defect, wherein the preset target detection model is a trained neural network model; (4) selecting points within the segmentation mask from the front view of the point cloud, converting the selected points into a world coordinate system, and obtaining a three-dimensional point cloud of potential defects; (5) Using ray tracing to filter out the error points caused by local occlusion in the potential defect 3D point cloud to obtain the real defect 3D point cloud; (6) Assigning the depth information of the real disease 3D point cloud to the corresponding pixel points in the 2D visual image. For the pixel points that lack depth information, the depth of the point is estimated by weighted linear interpolation to generate a depth map of the disease segmentation mask area; (7) Quantify the physical size of the disease based on the depth map; (8) Displaying the actual three-dimensional point cloud data of the defect and the physical size of the defect in the three-dimensional laser point cloud data of the bridge; Wherein, step (5) specifically includes: (5.1) In the world coordinate system, calculate the straight line between the camera optical center and each point in the potential defect 3D point cloud; (5.2) For each line, calculate the spatial angle between it and all remaining lines, and select the points whose spatial angle is less than the angle threshold as candidate disease points, where the angle threshold is determined by the spatial resolution of the lidar; (5.3) Calculate the depth of all candidate damage points based on the elevation information of the 3D laser point cloud, and calculate the average value of the depth of all candidate damage points; (5.4) The candidate defect points with depth less than or equal to the average value are regarded as real defect points, and all real defect points are formed into a real defect three-dimensional point cloud.

2. The bridge apparent defect detection method based on laser point cloud and visual image fusion according to claim 1 is characterized in that: The mapping model between the three-dimensional laser point cloud and the two-dimensional visual image in step (2) is: Where (u, v) is the pixel coordinate in the pixel coordinate system; R l 、T l is the external parameter of the laser radar in the world coordinate system, α, β, and γ are the roll angle, pitch angle, and yaw angle respectively; C X 、C Y 、C Z is the coordinate of the laser radar in the world coordinate system during image acquisition; R lc 、T lc is the relative external parameter obtained by joint calibration of lidar and camera, f x 、f y is the focal length of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

3. The bridge apparent defect detection method based on laser point cloud and visual image fusion according to claim 1 is characterized in that: Step (3) specifically includes: (3.1) using a preset target detection model to identify the disease in the two-dimensional visual image and obtain a bounding box and confidence score of the disease; wherein the preset target detection model is a trained neural network model; (3.2) Deleting the boundary boxes of the diseases whose confidence level is less than the preset threshold; (3.3) Using a preset instance segmentation model, the pixels within the bounding box processed in step (3.2) are segmented to distinguish the diseased area from the non-diseased area, and a segmentation mask of the diseased area is generated, wherein the preset instance segmentation model is a trained neural network model.

4. The bridge apparent defect detection method based on laser point cloud and visual image fusion according to claim 1 is characterized in that: Step (4) specifically includes: (4.1) Filter out points within the segmentation mask from the front view of the point cloud as defect points I = {(u, v)∈A} in the pixel coordinate system, where A represents the segmentation mask and (u, v) represents the pixel coordinates in the pixel coordinate system; (4.2) The defect points are converted from the pixel coordinate system to the world coordinate system according to the following formula to obtain the potential defect 3D point cloud; Where (u, v) is the pixel coordinate in the pixel coordinate system, R l 、T l is the external parameter of the laser radar in the world coordinate system, R lc 、T lc is the relative external parameter obtained by joint calibration of lidar and camera, f x 、f y is the focal length of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

5. The bridge apparent defect detection method based on laser point cloud and visual image fusion according to claim 1 is characterized in that: Step (6) specifically includes: (6.1) Assigning the depth value of each real 3D defect point to the corresponding pixel point in the 2D visual image; (6.2) If there are pixels without depth information in the segmentation mask of the 2D visual image, then the coverage area of ​​the segmentation mask on the image is expanded by n pixels in each of the four directions (up, down, left, and right). The depth of the pixels with known depth information in the new coverage area is estimated using the weighted linear interpolation formula below, where the value of n is determined by the image resolution: Where h represents the estimated depth, h U 、h D 、h L 、h R Respectively represents the depth of the pixel closest to the point in the four directions of up, down, left, and right, w U 、w D 、w L 、w R Respectively represents the distance between the pixel closest to the point in the four directions of up, down, left, and right; (6.3) Extract the depth information of each point in the segmentation mask to generate a depth map of the disease segmentation mask area.

6. The bridge apparent defect detection method based on laser point cloud and visual image fusion according to claim 1 is characterized in that: Step (7) specifically includes: (7.1) Based on the defect point cloud depth map and segmentation mask, the physical area of ​​the defect is calculated according to the following formula: Where A w is the physical area of ​​the disease, A p is the pixel point set of the segmentation mask, h represents the depth of the corresponding point in the depth map of the disease point cloud, (u, v) is the pixel coordinate in the pixel coordinate system, and f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system; (7.2) Based on the defect point cloud depth map and segmentation mask, the physical length of the defect is calculated according to the following formula: Where, l w is the physical length of the lesion, l p is the pixel length of the disease in the segmentation mask, l represents the arc length along the skeleton line; (7.3) Based on the defect point cloud depth map and segmentation mask, the physical width of the defect is calculated according to the following formula: Where w w is the physical width of the lesion, w p is the pixel width of the disease in the segmentation mask, w represents the direction perpendicular to the skeleton line; (7.4) The calculated physical area, length, and width of the disease are used as the quantified physical dimensions of the disease.

7. The bridge apparent defect detection method based on laser point cloud and visual image fusion according to claim 1 is characterized in that: Step (8) is followed by the following steps: (9) Map the pixel points in the bridge structure area in the two-dimensional visual image to the world coordinate system to obtain the partial bridge structure point cloud that has been inspected, and calculate its overlap rate with the complete bridge point cloud as the detection coverage rate of this inspection.

8. The bridge apparent defect detection method based on laser point cloud and visual image fusion according to claim 7 is characterized in that: After step (9), the following steps are also included: (10) Obtain the BIM model of the bridge, align the three-dimensional laser point cloud of the bridge with the BIM model, solve the three-dimensional coordinates of the apparent defects of the bridge in the BIM model coordinate system, attach the segmentation mask to the corresponding surface of the BIM model, integrate the defects into the BIM model as additional information, and mark the parts of the bridge structure that have been inspected as a reference for subsequent bridge inspection and management and maintenance.

9. A bridge surface defect detection system based on the fusion of laser point cloud and visual image, characterized in that: include: The data acquisition module is used to jointly calibrate the LiDAR and camera, and use the calibrated LiDAR and camera to synchronously collect data to obtain a 3D laser point cloud and 2D visual image of the bridge surface; A mapping module is used to establish a mapping model between the 3D laser point cloud and the 2D visual image, and project the 3D laser point cloud of the bridge surface onto the 2D visual image plane through the mapping model to generate a point cloud front view; A disease recognition module is used to identify diseases in the two-dimensional visual image using a preset target detection model, which is a trained neural network model, and generate a pixel-level segmentation mask for each disease; A potential defect 3D point cloud generation module is used to filter out points within the segmentation mask from the point cloud front view, and convert the filtered points into a world coordinate system to obtain a potential defect 3D point cloud; The real disease 3D point cloud generation module is used to use ray tracing to filter out the error points caused by local occlusion in the potential disease 3D point cloud to obtain the real disease 3D point cloud; The defect depth map generation module is used to assign the depth information of the real defect 3D point cloud to the corresponding pixel points in the 2D visual image. For the pixel points that lack depth information, the depth of the point is estimated by weighted linear interpolation to generate the depth map of the defect segmentation mask area; The disease size quantification module is used to quantify the physical size of the disease based on the depth map; The defect display module is used to display the real defect 3D point cloud data and the physical size of the defect in the 3D laser point cloud data of the bridge; The real disease 3D point cloud generation module specifically includes: A straight line calculation unit is used to calculate the straight line between the camera optical center and each point in the potential defect three-dimensional point cloud in the world coordinate system; The candidate defect point calculation unit is used to calculate the spatial angle between each line and all remaining lines, and select the points with spatial angles less than the angle threshold as candidate defect points; A depth average calculation unit is used to calculate the depth of all candidate disease points based on the elevation information of the three-dimensional laser point cloud, and calculate the average value of the depth of all candidate disease points; The real defect point extraction unit is used to take candidate defect points with depths less than or equal to the average value as real defect points, and form a real defect three-dimensional point cloud from all real defect points.

Images