Bridge apparent disease detection method and system based on laser point cloud and visual image fusion

Abstract

The invention discloses a bridge appearance disease detection method and system based on laser point cloud and visual image fusion, and the method comprises the steps: carrying out the synchronous data collection through employing a calibrated laser radar and a camera, and obtaining a three-dimensional laser point cloud and a two-dimensional visual image of a bridge appearance; projecting the three-dimensional laser point cloud to a two-dimensional visual image plane through a mapping model to generate a point cloud front view; identifying diseases in the two-dimensional visual image by adopting a preset target detection model, and generating a segmentation mask for each disease; screening out points in the segmentation mask from the point cloud front view to obtain a potential disease three-dimensional point cloud; error points generated by local shielding in the potential disease three-dimensional point cloud are filtered through ray tracing, and a real disease three-dimensional point cloud is obtained; distributing the depth information of the real disease three-dimensional point cloud to corresponding pixel points, generating a depth map of a disease segmentation mask region, and performing quantitative calculation on the physical size of the disease; and displaying the real disease three-dimensional point cloud data and the disease physical size. The method is high in efficiency and precision.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a bridge surface defect detection method and system based on the fusion of laser point cloud and visual image. Background Art

[0002] During the long-term service of bridges, the surface of bridges will be affected by various factors and will have concrete cracks, spalling, exposed steel bars and other defects, which seriously threaten the safety of bridges. By detecting and locating bridge surface defects, it is helpful to formulate targeted maintenance measures in a timely manner and extend the service life. Traditional detection methods are mainly based on manual inspections with high costs and low efficiency, and information records are mainly based on text materials with poor intuitiveness. With the development of technology, computer vision has gradually become the mainstream method for bridge defect detection.

[0003] However, the current computer vision-based defect detection methods use a single type of data, either laser point cloud or visual image, which has certain limitations. Laser scanners can quickly reconstruct high-precision bridge point cloud models, but due to the limitation of instrument resolution, it is difficult to describe tiny cracks or damage on the surface of the structure, and the defect detection method based on point cloud is not yet mature. In contrast, visual images can capture high-resolution texture features on the surface of the structure, and the defect detection technology based on visual images has a high recognition rate. However, in order to locate the defects identified in the visual image to the whole bridge model, a three-dimensional reconstruction algorithm is required, and the image acquisition requirements are high. For example, patent documents CN115797350B and CN117710810A both use visual images for recognition and three-dimensional reconstruction. In this method, factors such as image overlap rate, field of view size, and number of feature points will affect the recognition and positioning accuracy, and the computational efficiency is also relatively low. In addition, both documents directly use visual images for three-dimensional reconstruction, because visual images do not contain the depth information of the scene, resulting in the identified defects also not having depth information, so the defects cannot be accurately quantified. Summary of the invention

[0004] In view of the problems existing in the prior art, the purpose of the present invention is to provide a method and system for detecting bridge surface defects based on the fusion of laser point cloud and visual image, which has high precision, high efficiency and can accurately quantify the defects.

[0005] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0006] A bridge surface defect detection method based on laser point cloud and visual image fusion includes the following steps:

[0007] (1) Jointly calibrate the laser radar and camera, and use the calibrated laser radar and camera for synchronous data acquisition to obtain the three-dimensional laser point cloud and two-dimensional visual image of the bridge surface;

[0008] (2) Establish a mapping model between the 3D laser point cloud and the 2D visual image, and project the 3D laser point cloud of the bridge surface onto the 2D visual image plane through the mapping model to generate a point cloud front view;

[0009] (3) using a preset target detection model to identify defects in the two-dimensional visual image and generating a pixel-level segmentation mask for each defect, wherein the preset target detection model is a trained neural network model;

[0010] (4) selecting points within the segmentation mask from the front view of the point cloud, converting the selected points into a world coordinate system, and obtaining a three-dimensional point cloud of potential defects;

[0011] (5) Using ray tracing to filter out the error points caused by local occlusion in the potential defect 3D point cloud to obtain the real defect 3D point cloud;

[0012] (6) Assigning the depth information of the real defect three-dimensional point cloud to the corresponding pixel points in the two-dimensional visual image. For the pixel points that lack depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the defect segmentation mask area;

[0013] (7) Quantify the physical size of the disease based on the depth map;

[0014] (8) Display the actual three-dimensional point cloud data of the defect and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

[0015] Furthermore, the mapping model between the three-dimensional laser point cloud and the two-dimensional visual image in step (2) is:

[0016]

[0017] Where (u, v) is the pixel coordinate in the pixel coordinate system; R l 、T l is the external parameter of the laser radar in the world coordinate system, α, β, γ are the roll angle, pitch angle and yaw angle respectively; C X , C Y , C Z is the coordinate of the laser radar in the world coordinate system during image acquisition; R lc 、T lc is the relative external parameter obtained by joint calibration of LiDAR and camera, f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system, cx 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

[0018] Furthermore, step (3) specifically includes:

[0019] (3.1) using a preset target detection model to identify the disease in the two-dimensional visual image and obtain a bounding box and a confidence score of the disease; wherein the preset target detection model is a trained neural network model;

[0020] (3.2) Deleting the boundary boxes of the diseases whose confidence is less than the preset threshold;

[0021] (3.3) Using a preset instance segmentation model, the pixels within the bounding box processed in step (3.2) are segmented to distinguish the diseased area from the non-diseased area, and a segmentation mask of the diseased area is generated, wherein the preset instance segmentation model is a trained neural network model.

[0022] Furthermore, step (4) specifically includes:

[0023] (4.1) Filter out points within the segmentation mask from the front view of the point cloud as defect points I = {(u, v) ∈ A} in the pixel coordinate system, where A represents the segmentation mask and (u, v) represents the pixel coordinates in the pixel coordinate system;

[0024] (4.2) The defect points are converted from the pixel coordinate system to the world coordinate system according to the following formula to obtain the potential defect three-dimensional point cloud;

[0025]

[0026] Where (u, v) is the pixel coordinate in the pixel coordinate system, R l 、T l is the external parameter of the laser radar in the world coordinate system, R lc 、T lc is the relative external parameter obtained by joint calibration of LiDAR and camera, f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

[0027] Furthermore, step (5) specifically includes:

[0028] (5.1) In the world coordinate system, calculate the straight line between the camera optical center and each point in the potential defect three-dimensional point cloud;

[0029] (5.2) For each straight line, calculate the spatial angle between it and all remaining straight lines, and select the points whose spatial angle is less than the angle threshold as candidate disease points, where the angle threshold is determined according to the spatial resolution of the lidar;

[0030] (5.3) Calculate the depth of all candidate disease points based on the elevation information of the three-dimensional laser point cloud, and calculate the average value of the depth of all candidate disease points;

[0031] (5.4) The candidate defect points whose depth is less than or equal to the average value are regarded as the real defect points, and all the real defect points are formed into a real defect three-dimensional point cloud.

[0032] Furthermore, step (6) specifically includes:

[0033] (6.1) Assigning the depth value of each real 3D defect point to the corresponding pixel point in the 2D visual image;

[0034] (6.2) If there are pixels lacking depth information in the segmentation mask of the two-dimensional visual image, the coverage area of ​​the segmentation mask on the image is expanded by n pixels in the four directions of upward, downward, left and right. The depth of the pixel with known depth information in the new coverage area is estimated using the weighted linear interpolation in the following formula, where the value of n is determined according to the image resolution:

[0035]

[0036] Where h represents the estimated depth, h U 、h D 、h L 、h R Respectively represent the depth of the pixel closest to the point in the four directions of up, down, left, and right, w U 、w D 、w L 、w R Respectively represent the distance between the pixel closest to the point in the four directions of up, down, left, and right;

[0037] (6.3) Extract the depth information of each point in the segmentation mask to generate a depth map of the diseased segmentation mask area.

[0038] Furthermore, step (7) specifically includes:

[0039] (7.1) According to the defect point cloud depth map and segmentation mask, the physical area of ​​the defect is calculated according to the following formula:

[0040]

[0041] In the formula, A w is the physical area of ​​the disease, A p is the pixel set of the segmentation mask, h represents the depth of the corresponding point in the defect point cloud depth map, (u, v) is the pixel coordinate in the pixel coordinate system, and f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system;

[0042] (7.2) According to the defect point cloud depth map and segmentation mask, the physical length of the defect is calculated according to the following formula:

[0043]

[0044] In the formula, l w is the physical length of the lesion, l p is the pixel length of the defect in the segmentation mask, l represents the arc length along the skeleton line;

[0045] (7.3) According to the defect point cloud depth map and segmentation mask, the physical width of the defect is calculated according to the following formula:

[0046]

[0047] In the formula, w w is the physical width of the lesion, w p is the pixel width of the defect in the segmentation mask, and w represents the vertical direction along the skeleton line;

[0048] (7.4) The calculated physical area, length and width of the disease are used as the quantified physical size of the disease.

[0049] Furthermore, step (8) further includes the following steps:

[0050] (9) Map the pixel points in the bridge structure area in the two-dimensional visual image to the world coordinate system to obtain the point cloud of the partial bridge structure that has been inspected, and calculate its overlap rate with the complete bridge point cloud as the detection coverage rate of this inspection.

[0051] Furthermore, step (9) further includes the following steps:

[0052] (10) Obtain the BIM model of the bridge, align the three-dimensional laser point cloud of the bridge with the BIM model, solve the three-dimensional coordinates of the bridge's apparent defects in the BIM model coordinate system, attach the segmentation mask to the corresponding surface of the BIM model, integrate the defects into the BIM model as additional information, and mark the parts of the bridge structure that have been inspected as a reference for subsequent bridge inspection and management and maintenance.

[0053] A bridge surface defect detection system based on laser point cloud and visual image fusion, including:

[0054] The data acquisition module is used to jointly calibrate the laser radar and camera, and use the calibrated laser radar and camera to perform synchronous data acquisition to obtain the three-dimensional laser point cloud and two-dimensional visual image of the bridge surface;

[0055] A mapping module is used to establish a mapping model between a three-dimensional laser point cloud and a two-dimensional visual image, and to project the three-dimensional laser point cloud of the bridge surface onto a two-dimensional visual image plane through the mapping model to generate a point cloud front view;

[0056] A disease recognition module, used to identify diseases in a two-dimensional visual image using a preset target detection model and generate a pixel-level segmentation mask for each disease, wherein the preset target detection model is a trained neural network model;

[0057] A potential defect three-dimensional point cloud generation module is used to filter out points located within the segmentation mask from the point cloud front view, and convert the filtered points into a world coordinate system to obtain a potential defect three-dimensional point cloud;

[0058] A real disease 3D point cloud generation module is used to use ray tracing to filter the error points caused by local occlusion in the potential disease 3D point cloud to obtain the real disease 3D point cloud;

[0059] The defect depth map generation module is used to assign the depth information of the real defect three-dimensional point cloud to the corresponding pixel points in the two-dimensional visual image. For the pixel points lacking depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the defect segmentation mask area;

[0060] The disease size quantification module is used to quantify the physical size of the disease according to the depth map;

[0061] The defect display module is used to display the real defect three-dimensional point cloud data and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] (1) The present invention performs defect identification and positioning through steps such as point cloud and image data registration and fusion, image target recognition, etc., focusing on the surface defect characteristics of multiple types of concrete, thereby improving the recognition accuracy and precision;

[0064] (2) The present invention directly obtains the camera's external parameter information and the spatial coordinates of the image pixels by using laser point cloud data, without using a three-dimensional reconstruction algorithm. It has high efficiency and is less affected by factors such as the image's field of view, overlap rate, and number of feature points. It has higher stability in disease identification and positioning and a wider range of applications.

[0065] (3) The physical dimensions of each defect are calculated, which enables accurate quantitative analysis of defects on the surface and expands the scope of application. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of a bridge surface defect detection method based on the fusion of laser point cloud and visual image provided by an embodiment of the present invention;

[0067] Figure 2 It is the coordinate system involved in the mapping model between the three-dimensional laser point cloud and the two-dimensional visual image in the embodiment of the present invention;

[0068] Figure 3 is a schematic diagram of the principle of filtering error points according to an embodiment of the present invention;

[0069] Figure 4 is a diagram of crack disease detection results provided by an embodiment of the present invention;

[0070] Figure 5 This is a graph showing the detection results of the spalling disease provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0072] Embodiment 1

[0073] The embodiment of the present invention provides a bridge surface defect detection method based on the fusion of laser point cloud and visual image, such as Figure 1 As shown, the following steps are included:

[0074] (1) The laser radar and camera are jointly calibrated, and the calibrated laser radar and camera are used for synchronous data acquisition to obtain the three-dimensional laser point cloud and two-dimensional visual image of the bridge surface.

[0075] The three-dimensional laser point cloud data is obtained by scanning a laser radar, and the laser radar can be a mechanical laser radar or a solid-state laser radar. The two-dimensional visual image is obtained by shooting with a camera. Since the method of the present invention does not require the use of an image-based three-dimensional reconstruction algorithm, there is no requirement for the overlap rate of the images to be collected. Therefore, the camera used can be a wide-angle camera with a large field of view or a telephoto camera with a small field of view.

[0076] For example, in this embodiment, a mechanical laser radar is used to collect bridge point cloud data. Its laser field of view is 270°×360°. This device can be used to obtain a full range of point cloud models of the bridge. The focal length (f x 、f y ) is (1015.9557, 1014.3076) of the industrial camera, the camera optical center (c x 、c y ) is (2125.3699, 1056.7953), and the image resolution is 2592×1944, in pixels. The two-dimensional visual image is acquired by the camera at intervals of 1 second during the laser radar scanning process. A single image cannot cover the obscured bridge structure data obtained by the laser radar scanning at other locations due to structural occlusion.

[0077] (2) A mapping model between the three-dimensional laser point cloud and the two-dimensional visual image is established, and the three-dimensional laser point cloud of the bridge surface is projected onto the two-dimensional visual image plane through the mapping model to generate a point cloud front view.

[0078] Among them, Figure 2 As shown, the mapping model between the three-dimensional laser point cloud and the two-dimensional visual image is:

[0079]

[0080]

[0081] Where (u, v) is the pixel coordinate in the pixel coordinate system; R l 、T l is the external parameter of the laser radar in the world coordinate system, α, β, γ are the roll angle, pitch angle and yaw angle respectively; C X , C Y , C z is the coordinate of the laser radar in the world coordinate system during image acquisition; R lc 、T lc is the relative external parameter obtained by joint calibration of LiDAR and camera, f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

[0082] For example, in this embodiment, the relative external parameter matrix obtained by joint calibration of the laser radar and the camera is:

[0083]

[0084] The intrinsic parameter matrix of the camera is:

[0085]

[0086] (3) A preset object detection model is used to identify defects in the two-dimensional visual image and generate a pixel-level segmentation mask for each defect.

[0087] This step specifically includes:

[0088] (3.1) Using a preset target detection model to identify defects in a two-dimensional visual image, and obtaining a bounding box and a confidence score of the defect; wherein the preset target detection model is a trained neural network model, such as a YOLO series model, and YOLO v8 is used in this embodiment; the defects include concrete cracks, spalling, and exposed reinforcement;

[0089] (3.2) deleting the boundary boxes of the diseases whose confidence is less than a preset threshold; for example, in this example, the preset threshold is set to 70%;

[0090] (3.3) Using a preset instance segmentation model, the pixels within the bounding box processed in step (3.2) are segmented to distinguish the diseased area from the non-diseased area, and a segmentation mask of the diseased area is generated, wherein the preset instance segmentation model is a trained neural network model, such as a Mask R-CNN model.

[0091] (4) Filter out points within the segmentation mask from the point cloud front view, convert the filtered points into a world coordinate system, and obtain a three-dimensional point cloud of potential defects.

[0092] This step specifically includes:

[0093] (4.1) Filter out points within the segmentation mask from the front view of the point cloud as defect points I = {(u, v) ∈ A} in the pixel coordinate system, where A represents the segmentation mask and (u, v) represents the pixel coordinates in the pixel coordinate system;

[0094] (4.2) The defect points are converted from the pixel coordinate system to the world coordinate system according to the following formula to obtain the potential defect three-dimensional point cloud data;

[0095]

[0096] (5) Use ray tracing to filter out the error points caused by local occlusion in the potential defect 3D point cloud to obtain the real defect 3D point cloud.

[0097] When projecting the point cloud, there is a problem of projecting non-diseased points into the damaged pixel area. Therefore, filtering is performed, such as Figure 3 As shown, specifically including:

[0098] (5.1) In the world coordinate system, calculate the straight line between the camera optical center and each point in the potential defect three-dimensional point cloud;

[0099] (5.2) For each straight line, calculate the spatial angle between it and all the remaining straight lines, and select the points whose spatial angle is less than the angle threshold as candidate defect points; wherein the angle threshold is determined according to the spatial resolution of the laser radar. For example, in this embodiment, the spatial resolution of the laser radar is 0.38°, so the angle threshold is set to 0.4°;

[0100] (5.3) Calculate the depth of all candidate disease points based on the elevation information of the three-dimensional laser point cloud, and calculate the average value of the depth of all candidate disease points;

[0101] (5.4) The candidate defect points whose depth is less than or equal to the average value are regarded as the real defect points, and all the real defect points are formed into a real defect three-dimensional point cloud.

[0102] (6) The depth information of the real defect three-dimensional point cloud is assigned to the corresponding pixel points in the two-dimensional visual image. For the pixel points that lack depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the defect segmentation mask area.

[0103] This step specifically includes:

[0104] (6.1) Assigning the depth value of each real 3D defect point to the corresponding pixel point in the 2D visual image;

[0105] (6.2) If there are pixels lacking depth information in the segmentation mask of the two-dimensional visual image, the coverage area of ​​the segmentation mask on the image is expanded by n pixels in the four directions of upward, downward, left and right. The depth of the pixel with known depth information in the new coverage area is estimated by using the weighted linear interpolation in the following formula, where the value of n is determined according to the image resolution. For example, the image resolution of this embodiment is 2592×1944, and the unit is pixel, so the value of n is set to 5:

[0106]

[0107] Where h represents the estimated depth, h U 、h D 、h L 、h R Respectively represent the depth of the pixel closest to the point in the four directions of up, down, left, and right, w U 、w D 、w L 、w R Respectively represent the distance between the pixel closest to the point in the four directions of up, down, left, and right;

[0108] (6.3) Extract the depth information of each point in the segmentation mask to generate a depth map of the diseased segmentation mask area.

[0109] (7) Based on the depth map, the physical size of the disease is quantified.

[0110] This step specifically includes:

[0111] (7.1) According to the defect point cloud depth map and segmentation mask, the physical area of ​​the defect is calculated according to the following formula:

[0112]

[0113] In the formula, A w is the physical area of ​​the disease, A p is the pixel set of the segmentation mask, h represents the depth of the corresponding point in the defect point cloud depth map, (u, v) is the pixel coordinate in the pixel coordinate system, and f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system;

[0114] (7.2) According to the defect point cloud depth map and segmentation mask, the physical length of the defect is calculated according to the following formula:

[0115]

[0116] In the formula, l w is the physical length of the lesion, l p is the pixel length of the defect in the segmentation mask, l represents the arc length along the skeleton line;

[0117] (7.3) According to the defect point cloud depth map and segmentation mask, the physical width of the defect is calculated according to the following formula:

[0118]

[0119] In the formula, w w is the physical width of the lesion, w p is the pixel width of the defect in the segmentation mask, and w represents the vertical direction along the skeleton line;

[0120] (7.4) The calculated physical area, length and width of the disease are used as the quantified physical size of the disease.

[0121] The pixel length and width of concrete cracks are calculated based on the segmentation mask, and the segmentation mask of the crack area is refined into a single-pixel-wide skeleton using a skeleton extraction algorithm. The total length of the crack is calculated by traversing the path of the skeleton pixel by pixel; for each pixel point on the skeleton, the distance to the crack edge is measured along its normal direction, and the maximum, minimum and average widths are calculated. The pixel area of ​​concrete cracks, spalling and exposed reinforcement is calculated based on the defect segmentation mask. The pixel area of ​​the defect can be calculated by accumulating the number of pixels in the defect area.

[0122] In this embodiment, the crack diseases and spalling diseases identified are as follows: Figure 4 and Figure 5 As shown, the pixel length of the crack disease is 212 pixels, the pixel area is 978 pixels, the maximum width is 4.39 pixels, the minimum width is 2.00 pixels, and the average width is 3.57 pixels; the physical length is 32.82 cm, and the physical area is 23.44 cm 2 , the maximum width is 0.67cm, the minimum width is 0.31cm, and the average width is 0.55cm. The pixel area of ​​the peeling disease is: 353037 pixels; the physical area is 9617.23cm 2 .

[0123] (8) Display the actual three-dimensional point cloud data of the defect and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

[0124] Specifically, the real disease 3D point cloud can be displayed in red, and the others can be displayed in normal non-red colors.

[0125] (9) Map the pixel points in the bridge structure area in the two-dimensional visual image to the world coordinate system to obtain the point cloud of the partial bridge structure that has been inspected, and calculate its overlap rate with the complete bridge point cloud as the detection coverage rate of this inspection.

[0126] (10) Obtain the BIM model of the bridge, align the three-dimensional laser point cloud of the bridge with the BIM model, solve the three-dimensional coordinates of the bridge's apparent defects in the BIM model coordinate system, attach the segmentation mask to the corresponding surface of the BIM model, integrate the defects into the BIM model as additional information, and mark the parts of the bridge structure that have been inspected as a reference for subsequent bridge inspection and management and maintenance.

[0127] Embodiment 2

[0128] The embodiment of the present invention provides a bridge surface defect detection system based on the fusion of laser point cloud and visual image, comprising:

[0129] The data acquisition module is used to jointly calibrate the laser radar and camera, and use the calibrated laser radar and camera to perform synchronous data acquisition to obtain the three-dimensional laser point cloud and two-dimensional visual image of the bridge surface;

[0130] A mapping module is used to establish a mapping model between a three-dimensional laser point cloud and a two-dimensional visual image, and to project the three-dimensional laser point cloud of the bridge surface onto a two-dimensional visual image plane through the mapping model to generate a point cloud front view;

[0131] A disease recognition module, used to identify diseases in a two-dimensional visual image using a preset target detection model and generate a pixel-level segmentation mask for each disease, wherein the preset target detection model is a trained neural network model;

[0132] A potential defect three-dimensional point cloud generation module is used to filter out points located within the segmentation mask from the point cloud front view, and convert the filtered points into a world coordinate system to obtain a potential defect three-dimensional point cloud;

[0133] A real disease 3D point cloud generation module is used to use ray tracing to filter the error points caused by local occlusion in the potential disease 3D point cloud to obtain the real disease 3D point cloud;

[0134] The defect depth map generation module is used to assign the depth information of the real defect three-dimensional point cloud to the corresponding pixel points in the two-dimensional visual image. For the pixel points lacking depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the defect segmentation mask area;

[0135] The disease size quantification module is used to quantify the physical size of the disease according to the depth map;

[0136] The defect display module is used to display the real defect three-dimensional point cloud data and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

[0137] The mapping model between the three-dimensional laser point cloud and the two-dimensional visual image is:

[0138]

[0139] Where (u, v) is the pixel coordinate in the pixel coordinate system; R l 、T l is the external parameter of the laser radar in the world coordinate system, α, β, γ are the roll angle, pitch angle and yaw angle respectively; C X , C Y , C Z is the coordinate of the laser radar in the world coordinate system during image acquisition; R lc 、T lc is the relative external parameter obtained by joint calibration of LiDAR and camera, fx 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

[0140] The disease identification module specifically includes:

[0141] A defect preliminary identification unit is used to identify defects in a two-dimensional visual image using a preset target detection model, and obtain a boundary box and a confidence score of the defect; wherein the preset target detection model is a trained neural network model, such as a YOLO series model, and YOLO v8 is used in this embodiment; the defects include concrete cracks, spalling, exposed reinforcement, etc.;

[0142] A disease screening unit is used to delete the boundary boxes of diseases whose confidence is less than a preset threshold; for example, in this example, the preset threshold is set to 70%;

[0143] The segmentation unit is used to segment the pixels within the boundary box processed by the disease screening unit using a preset instance segmentation model, distinguish the diseased area from the non-diseased area, and generate a segmentation mask of the diseased area, wherein the preset instance segmentation model is a trained neural network model, such as a Mask R-CNN model.

[0144] The potential damage three-dimensional point cloud generation module specifically includes:

[0145] A defect point screening unit is used to screen out points located within the segmentation mask from the point cloud front view as defect points I={(u,v)∈A} in a pixel coordinate system, where A represents the segmentation mask and (u,v) represents the pixel point coordinates in the pixel coordinate system;

[0146] A coordinate conversion unit is used to convert the defect point from the pixel coordinate system to the world coordinate system according to the following formula to obtain the potential defect three-dimensional point cloud data;

[0147]

[0148] The real disease three-dimensional point cloud generation module specifically includes:

[0149] A straight line calculation unit, used to calculate the straight line between the camera optical center and each point in the potential defect three-dimensional point cloud in the world coordinate system;

[0150] A candidate defect point calculation unit is used to calculate the spatial angle between each straight line and all the remaining straight lines, and select points whose spatial angles are less than an angle threshold as candidate defect points; wherein the angle threshold is determined according to the spatial resolution of the laser radar. For example, in this embodiment, the spatial resolution of the laser radar is 0.38°, so the angle threshold is set to 0.4°;

[0151] A depth average value calculation unit is used to calculate the depth of all candidate disease points according to the elevation information of the three-dimensional laser point cloud, and calculate the average value of the depth of all candidate disease points;

[0152] The real defect point extraction unit is used to take the candidate defect points whose depth is less than or equal to the average value as the real defect points, and form a real defect three-dimensional point cloud from all the real defect points.

[0153] The disease depth map generation module specifically includes:

[0154] A depth assignment unit, used to assign the depth value of each real three-dimensional disease point to the corresponding pixel point in the two-dimensional visual image;

[0155] The depth estimation unit is used to expand the coverage area of ​​the segmentation mask on the image by n pixels in the four directions of upward, downward, left and right if there are pixels lacking depth information in the segmentation mask of the two-dimensional visual image, and estimate the depth of the point by using the weighted linear interpolation in the following formula through the pixels with known depth information in the new coverage area, wherein the value of n is determined according to the image resolution. For example, the image resolution of this embodiment is 2592×1944, and the unit is pixel, so the value of n is set to 5:

[0156]

[0157] Where h represents the estimated depth, h U 、h D 、h L 、h R Respectively represent the depth of the pixel closest to the point in the four directions of up, down, left, and right, w U 、w D 、w L 、w R Respectively represent the distance between the pixel closest to the point in the four directions of up, down, left, and right;

[0158] The depth map generating unit is used to extract the depth information of each point in the segmentation mask to generate a depth map of the defect segmentation mask area.

[0159] The disease size quantification module specifically includes:

[0160] The damage area calculation unit is used to calculate the physical area of ​​the damage according to the damage point cloud depth map and segmentation mask according to the following formula:

[0161]

[0162] In the formula, A w is the physical area of ​​the disease, A p is the pixel set of the segmentation mask, h represents the depth of the corresponding point in the defect point cloud depth map, (u, v) is the pixel coordinate in the pixel coordinate system, and f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system;

[0163] The defect length calculation unit is used to calculate the physical length of the defect according to the defect point cloud depth map and the segmentation mask according to the following formula:

[0164]

[0165] In the formula, l w is the physical length of the lesion, l p is the pixel length of the defect in the segmentation mask, l represents the arc length along the skeleton line;

[0166] The defect width calculation unit is used to calculate the physical width of the defect according to the defect point cloud depth map and the segmentation mask according to the following formula:

[0167]

[0168] In the formula, w w is the physical width of the lesion, w p is the pixel width of the defect in the segmentation mask, and w represents the vertical direction along the skeleton line;

[0169] The disease physical size acquisition unit is used to use the calculated physical area, length and width of the disease as the quantified physical size of the disease.

[0170] In addition, the system can also include a coverage calculation module, which is used to map the pixel points in the bridge structure area in the two-dimensional visual image to the world coordinate system, obtain the partial bridge structure point cloud that has been inspected, and calculate its overlap rate with the complete bridge point cloud as the detection coverage of this inspection.

[0171] In addition, the system can also include a BIM model annotation module, which is used to obtain the BIM model of the bridge, align the three-dimensional laser point cloud of the bridge with the BIM model, solve the three-dimensional coordinates of the bridge's apparent defects in the BIM model coordinate system, attach the segmentation mask to the corresponding surface of the BIM model, integrate the defects into the BIM model as additional information, and mark out some bridge structures that have been inspected as a reference for subsequent bridge inspection and management and maintenance.

[0172] This system corresponds to the method of the first embodiment one by one. For details not described in detail, please refer to the description of the method of the first embodiment and will not be repeated here.

[0173] It should be understood that the above embodiments and descriptions only describe the principles, main features and advantages of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, and these changes and improvements all fall within the scope of protection of the present invention.

Claims

1. A bridge surface defect detection method based on the fusion of laser point cloud and visual image, characterized in that: The steps include: (1) Jointly calibrate the laser radar and camera, and use the calibrated laser radar and camera for synchronous data acquisition to obtain the three-dimensional laser point cloud and two-dimensional visual image of the bridge surface; (2) Establish a mapping model between the 3D laser point cloud and the 2D visual image, and project the 3D laser point cloud of the bridge surface onto the 2D visual image plane through the mapping model to generate a point cloud front view; (3) using a preset target detection model to identify the defects in the two-dimensional visual image and generating a pixel-level segmentation mask for each defect, wherein the preset target detection model is a trained neural network model; (4) selecting points within the segmentation mask from the front view of the point cloud, converting the selected points into a world coordinate system, and obtaining a three-dimensional point cloud of potential defects; (5) Using ray tracing to filter out the error points caused by local occlusion in the potential defect 3D point cloud to obtain the real defect 3D point cloud; (6) Assigning the depth information of the real defect three-dimensional point cloud to the corresponding pixel points in the two-dimensional visual image. For the pixel points that lack depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the defect segmentation mask area; (7) Quantify the physical size of the disease based on the depth map; (8) Display the actual three-dimensional point cloud data of the defect and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

2. The bridge apparent disease detection method based on the fusion of laser point cloud and visual image according to claim 1 is characterized in that: The mapping model between the three-dimensional laser point cloud and the two-dimensional visual image in step (2) is: Where (u, v) is the pixel coordinate in the pixel coordinate system; R l , T l is the external parameter of the laser radar in the world coordinate system, α, β, γ are the roll angle, pitch angle and yaw angle respectively; C X , C Y , C Z is the coordinate of the laser radar in the world coordinate system during image acquisition; R lc , T lc is the relative external parameter obtained by joint calibration of LiDAR and camera, f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

3. The bridge apparent disease detection method based on the fusion of laser point cloud and visual image according to claim 1 is characterized in that: Step (3) specifically includes: (3.1) using a preset target detection model to identify the disease in the two-dimensional visual image and obtain a bounding box and a confidence score of the disease; wherein the preset target detection model is a trained neural network model; (3.2) Deleting the boundary boxes of the diseases whose confidence is less than the preset threshold; (3.3) Using a preset instance segmentation model, the pixels within the bounding box processed in step (3.2) are segmented to distinguish the diseased area from the non-diseased area, and a segmentation mask of the diseased area is generated, wherein the preset instance segmentation model is a trained neural network model.

4. The bridge apparent disease detection method based on the fusion of laser point cloud and visual image according to claim 1 is characterized in that: Step (4) specifically includes: (4.1) Filter out points within the segmentation mask from the front view of the point cloud as defect points I = {(u, v) ∈ A} in the pixel coordinate system, where A represents the segmentation mask and (u, v) represents the pixel coordinates in the pixel coordinate system; (4.2) The defect points are converted from the pixel coordinate system to the world coordinate system according to the following formula to obtain the potential defect three-dimensional point cloud; Where (u, v) is the pixel coordinate in the pixel coordinate system, R l , T l is the external parameter of the laser radar in the world coordinate system, R lc , T lc is the relative external parameter obtained by joint calibration of LiDAR and camera, f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system, c x 、c y is the coordinate of the camera optical center in the pixel coordinate system, (x w ,y w , z w ) are the point coordinates in the world coordinate system.

5. The bridge apparent disease detection method based on the fusion of laser point cloud and visual image according to claim 1 is characterized in that: Step (5) specifically includes: (5.1) In the world coordinate system, calculate the straight line between the camera optical center and each point in the potential defect three-dimensional point cloud; (5.2) For each straight line, calculate the spatial angle between it and all remaining straight lines, and select the points whose spatial angle is less than the angle threshold as candidate disease points, where the angle threshold is determined according to the spatial resolution of the lidar; (5.3) Calculate the depth of all candidate disease points based on the elevation information of the three-dimensional laser point cloud, and calculate the average value of the depth of all candidate disease points; (5.4) The candidate defect points whose depth is less than or equal to the average value are regarded as the real defect points, and all the real defect points are formed into a real defect three-dimensional point cloud.

6. The bridge apparent disease detection method based on the fusion of laser point cloud and visual image according to claim 1 is characterized in that: Step (6) specifically includes: (6.1) Assigning the depth value of each real 3D defect point to the corresponding pixel point in the 2D visual image; (6.2) If there are pixels lacking depth information in the segmentation mask of the two-dimensional visual image, the coverage area of ​​the segmentation mask on the image is expanded by n pixels in the four directions of upward, downward, left and right. The depth of the pixel with known depth information in the new coverage area is estimated using the weighted linear interpolation in the following formula, where the value of n is determined according to the image resolution: Where h represents the estimated depth, h U 、h D 、h L 、h R Respectively represent the depth of the pixel closest to the point in the four directions of up, down, left, and right, w U 、w D 、w L 、w R Respectively represent the distance between the pixel closest to the point in the four directions of up, down, left, and right; (6.3) Extract the depth information of each point in the segmentation mask to generate a depth map of the diseased segmentation mask area.

7. The bridge apparent disease detection method based on the fusion of laser point cloud and visual image according to claim 1 is characterized in that: Step (7) specifically includes: (7.1) According to the defect point cloud depth map and segmentation mask, the physical area of ​​the defect is calculated according to the following formula: In the formula, A w is the physical area of ​​the disease, A p is the pixel set of the segmentation mask, h represents the depth of the corresponding point in the defect point cloud depth map, (u, v) is the pixel coordinate in the pixel coordinate system, and f x 、f y are the focal lengths of the camera in the x-axis and y-axis directions in the pixel coordinate system; (7.2) According to the defect point cloud depth map and segmentation mask, the physical length of the defect is calculated according to the following formula: In the formula, l w is the physical length of the lesion, l p is the pixel length of the defect in the segmentation mask, l represents the arc length along the skeleton line; (7.3) According to the defect point cloud depth map and segmentation mask, the physical width of the defect is calculated according to the following formula: In the formula, w w is the physical width of the lesion, w p is the pixel width of the defect in the segmentation mask, and w represents the vertical direction along the skeleton line; (7.4) The calculated physical area, length and width of the disease are used as the quantified physical size of the disease.

8. The bridge apparent disease detection method based on the fusion of laser point cloud and visual image according to claim 1 is characterized in that: After step (8), the following steps are also included: (9) Map the pixel points in the bridge structure area in the two-dimensional visual image to the world coordinate system to obtain the point cloud of the partial bridge structure that has been inspected, and calculate its overlap rate with the complete bridge point cloud as the detection coverage rate of this inspection.

9. The bridge apparent disease detection method based on the fusion of laser point cloud and visual image according to claim 8 is characterized in that: After step (9), the method further comprises the following steps: (10) Obtain the BIM model of the bridge, align the three-dimensional laser point cloud of the bridge with the BIM model, solve the three-dimensional coordinates of the bridge's apparent defects in the BIM model coordinate system, attach the segmentation mask to the corresponding surface of the BIM model, integrate the defects into the BIM model as additional information, and mark the parts of the bridge structure that have been inspected as a reference for subsequent bridge inspection and management and maintenance.

10. A bridge surface defect detection system based on the fusion of laser point cloud and visual image, characterized in that: include: The data acquisition module is used to jointly calibrate the laser radar and camera, and use the calibrated laser radar and camera to perform synchronous data acquisition to obtain the three-dimensional laser point cloud and two-dimensional visual image of the bridge surface; A mapping module is used to establish a mapping model between a three-dimensional laser point cloud and a two-dimensional visual image, and to project the three-dimensional laser point cloud of the bridge surface onto a two-dimensional visual image plane through the mapping model to generate a point cloud front view; A disease recognition module, used to identify diseases in a two-dimensional visual image using a preset target detection model and generate a pixel-level segmentation mask for each disease, wherein the preset target detection model is a trained neural network model; A potential defect three-dimensional point cloud generation module is used to filter out points located within the segmentation mask from the point cloud front view, and convert the filtered points into a world coordinate system to obtain a potential defect three-dimensional point cloud; A real disease 3D point cloud generation module is used to use ray tracing to filter the error points caused by local occlusion in the potential disease 3D point cloud to obtain the real disease 3D point cloud; The defect depth map generation module is used to assign the depth information of the real defect three-dimensional point cloud to the corresponding pixel points in the two-dimensional visual image. For the pixel points lacking depth information, the depth of the point is estimated by weighted linear interpolation, thereby generating a depth map of the defect segmentation mask area; The disease size quantification module is used to quantify the physical size of the disease according to the depth map; The defect display module is used to display the real defect three-dimensional point cloud data and the physical size of the defect in the three-dimensional laser point cloud data of the bridge.

Images