A bridge defect detection and positioning method and device based on 2D-3D data fusion

Abstract

A method and device for detecting and locating bridge defects based on 2D-3D data fusion. This invention utilizes the homogeneity between images and point clouds to identify bridge defects under complex backgrounds and unbalanced data conditions. The specific implementation steps are: ① Collect multi-view images and defect images of the bridge structure; ② Reconstruct the point cloud model of the bridge structure using the multi-view images and output a depth image; ③ Build and train an ROI extraction model, and use the depth image to output the defect image after removing the complex background; ④ Build and train an improved DeepLabv3+ defect segmentation model, detect bridge defects in the defect image after ROI extraction, and output a bridge defect segmentation mask; ⑤ Realize three-dimensional visualization of the defect based on the bridge defect segmentation mask and the point cloud model. The method disclosed in the present invention can achieve high-precision detection and positioning of bridge defects, providing a powerful tool for routine inspection of bridge structures.

Description

Technical Field

[0001] The present invention relates to a bridge structure defect detection method and device, specifically a bridge structure defect identification and three-dimensional positioning technology based on 2D-3D data fusion and computer vision technology, belonging to the field of structural engineering. Background Art

[0002] Bridges are a vital component of transportation infrastructure. During daily operation, they are inevitably subject to various adverse environmental conditions and applied loads, which can accelerate structural damage and shorten their service life. Various types of surface defects, such as cracks, concrete spalling, and voids, are outward manifestations of bridge condition. Their presence indicates a decline in the bridge's durability and even its load-bearing capacity, seriously compromising its structural safety. According to incomplete statistics, approximately 40% of the bridges currently in service on my country's highway network are over 20 years old, 30% are affected by defects, and over 100,000 bridges are considered dangerous. Therefore, routine inspections to ensure the safety of bridge structures are crucial.

[0003] Traditional bridge inspection typically relies on manual visual assessment and large-scale inspection equipment, which is costly, inefficient, and poses significant safety risks. With the rapid development of technologies such as deep learning and artificial intelligence, applying computer vision methods to images captured by drone surveys has become an emerging approach for bridge defect detection. However, since most bridges are located in natural environments, drone-captured images often contain irrelevant and complex background elements such as rivers, trees, hills, and the sky, which significantly affects the accuracy of defect identification. Furthermore, deep learning models typically require large, high-quality datasets for training. Existing bridge defect datasets suffer from severe inter-class imbalance due to the small size of the defects, which can lead to poor model training results. Finally, the actual location of bridge defects in three-dimensional space is difficult to determine, which is crucial for subsequent in-depth analysis. Therefore, it is necessary to develop a highly accurate and robust defect detection and localization method to identify bridge defects in complex backgrounds. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this paper proposes a bridge defect detection and location method and device based on 2D-3D data fusion to improve the feasibility of bridge defect detection based on computer vision technology in practical applications. The specific contents include:

[0005] A first aspect of the present invention relates to a bridge defect detection and location method based on 2D-3D data fusion, comprising the following steps:

[0006] A. Use drones to collect multi-view image data I1 and close-range image data I2 of bridge structures;

[0007] B. Reconstructing a three-dimensional point cloud model of the bridge using the multi-view image data I1, and generating a depth image D corresponding to the multi-view image through coordinate system conversion;

[0008] The process of generating the depth image D by coordinate transformation includes: using the camera pose to transform the point cloud coordinates (U, V, W) into the pixel coordinate system (u, v), which can be expressed as

[0009]

[0010] Where K1 and K2 represent the external parameters of the camera, f c Represents the focal length of the camera, dx and dy represent the ratio of the pixel to the physical size in the x and y directions respectively, (u0, v0) represents the coordinates of the pixel origin in the image coordinate system, and Z represents the depth value; the corresponding depth information Z is used as the pixel value of the projected image and saved as a grayscale image;

[0011] C. Build and train a ROI (Region of Interest) extraction model based on RGB-D images to output a bridge image with complex background removed.

[0012] C1. Build a MobileSal model based on RGB-D images for image ROI extraction. The model consists of three parts: an encoder, a feature fusion module, and a decoder.

[0013] C2. Annotate the multi-view bridge structure image I1, where the annotated objects are the pixels of the bridge structure in the image, and generate a corresponding segmentation mask for the multi-view bridge structure image I1; integrate the corresponding multi-view bridge structure image I1, the depth image D, and the segmentation mask to form an RGB-D dataset for training the image ROI extraction model;

[0014] C3. Split the RGB-D dataset into a training set and a test set, the training dataset is used to train the ROI extraction model, and the test dataset is used to evaluate the performance of the ROI extraction model; use the trained ROI extraction model to segment the image data I2, and output the background-removed image data I2;

[0015] D. Build and train an improved DeepLabv3+ disease segmentation model to output disease segmentation masks;

[0016] D1. Build an improved DeepLabv3+ disease segmentation model, consisting of an encoder and decoder. The encoder uses ResNet101 as the backbone network and combines it with Atrous Spatial Pyramid Pooling (ASPP) to extract image features with different receptive fields, thereby generating image features that integrate multi-scale information. The decoder processes low-level feature maps from the backbone network's intermediate layers and the output of the ASPP module, and generates predicted masks through multiple convolutions and interpolation upsampling operations.

[0017] D2. To improve image segmentation accuracy, a focal loss function is introduced into the DeepLabv3+ model to replace the original cross-entropy loss function.

[0018] D3. Collect image data of bridge structure defects and annotate the images. The annotated objects are the pixels in the images that represent bridge defects, including four types of defects: cracks, voids, spalling, and water erosion. Copy-paste data augmentation technology is introduced to enhance the defect images and their annotated information, increasing the proportion of defect pixels and alleviating the inter-class imbalance in the data. This will form a defect dataset for improving the training of the DeepLabv3+ defect segmentation model.

[0019] D4. Split the enhanced disease dataset into a training set and a test set. The training set is used to train the improved DeepLabv3+ disease segmentation model, and the test set is used to evaluate model performance. Use the trained improved DeepLabv3+ disease segmentation model to detect the background-removed image data I2 and output the corresponding disease segmentation mask.

[0020] E. 3D visualization of bridge defects based on defect segmentation mask and reconstructed bridge point cloud;

[0021] E1. To prevent hidden points behind the bridge target point from being incorrectly displayed in the point cloud from the current image perspective, a hidden point removal algorithm is used to calculate the visible points of the point cloud from the current perspective based on the shooting pose of the current defect image.

[0022] E2. Project the point cloud containing only visible points onto a 2D image based on the camera's intrinsic and extrinsic parameters, and establish a correspondence between the image pixels and the points in the 3D point cloud.

[0023] E3. Combine the damage segmentation mask with the projected 2D image to determine whether a pixel in the 2D image is damaged. For pixels identified as damaged, index the corresponding point in the bridge point cloud, change the color of the point to the preset damage color, and attach damage information.

[0024] E4. Traverse all the defect segmentation masks and display the defect information detected from the 2D image on the 3D point cloud to achieve 3D visualization of bridge defects.

[0025] Furthermore, the method for collecting the multi-perspective image data I1 and the close-range image data I2 of the bridge structure in step A is as follows: using drone equipment to perform 360° surround shooting of the entire bridge, the overlapping part between adjacent images should not be less than 75%, and the shooting angle between adjacent images should not exceed 15°, and finally the multi-perspective image data I1 is collected; based on the drone equipment, close-range shooting is performed close to the surface of the bridge to ensure that the clarity of the image meets the requirements for disease detection, and finally the close-range image data I2 is collected.

[0026] Furthermore, the method used to reconstruct the three-dimensional point cloud model of the bridge in step B is COLMAP modeling software based on Structure from Motion (SfM).

[0027] Furthermore, the encoder in step C1 includes two branches: RGB stream and depth stream. Both use the MobileNet v2 network as the base network, and the number of network layers of the depth stream is much smaller than that of the RGB stream, so as to keep the model lightweight.

[0028] Furthermore, the RGB-D dataset in step C2 is composed of three images: a color image in RGB format, a depth image in grayscale format, and a binary segmentation mask, and there is a strict one-to-one correspondence between the three images.

[0029] Furthermore, the formula of focal loss in step D2 is:

[0030] FL(p)=-β(1-p) γ log(p) (2)

[0031] Where β represents the pixel weights of different disease categories in the model, p represents the probability that the model predicts that the pixel belongs to the foreground, and γ is a parameter preset to 2; when p is close to 1, it means that the pixel sample is easy to distinguish, (1-p) γ Close to 0, thereby reducing the loss caused by these easily distinguishable pixels; through the focal loss function, the impact of the loss values ​​of a large number of background pixels in the data can be effectively reduced.

[0032] Furthermore, the copy-paste data enhancement technology in step D3 includes two parts: random transformation and defect pasting. The random transformation part is specifically: randomly selecting a source image P1 and a target image P2, and performing random translation, flipping and resizing operations on the source image, the target image and the corresponding annotation information; the defect pasting part is specifically: using the annotation information to extract pixels of the defect area from the transformed source image, and randomly pasting the extracted pixels to the transformed target image, and finally generating an enhanced image P aug It can be expressed as

[0033] P aug =αP1+(1-α)P2 (3)

[0034] where α represents the annotation mask of the source image.

[0035] Furthermore, the pre-set disease colors in step E3 are specifically: cracks - yellow, voids - green, peeling - purple, and water erosion - red.

[0036] The second aspect of the present invention relates to a bridge defect detection and positioning device based on 2D-3D data fusion, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a bridge defect detection and positioning method based on 2D-3D data fusion according to the present invention.

[0037] Compared with the existing methods, the present invention has the following advantages:

[0038] (1) The proposed ROI extraction model combines 2D RGB images and 3D depth information to accurately extract bridge pixels from complex backgrounds, avoiding the computational burden of large-scale point cloud processing.

[0039] (2) Compared with the original DeepLabv3 model, the improved DeepLabv3+ model integrates copy-paste data enhancement and focal loss technology, solves the serious inter-class imbalance problem in the dataset, and significantly improves the segmentation accuracy;

[0040] (3) A damage localization method based on camera pose is proposed. The hidden point removal algorithm is used to calculate the visible points in the point cloud, realizing the three-dimensional visualization of bridge damage. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method of the present invention;

[0042] Figure 2 is a reconstructed three-dimensional point cloud image of the bridge structure of the present invention;

[0043] Figure 3(a) to Figure 3(b) 3(a) is the original image and FIG3(b) is the corresponding depth image.

[0044] Figure 4(a) to Figure 4(b) 4(a) is the original image, and FIG4(b) is the corresponding image after ROI extraction;

[0045] Figure 5 This is the improved DeepLabv3+ model structure diagram of the present invention;

[0046] FIG6(a) is a close-up original image of the bridge structure of the present invention, and FIG6(b) is a predicted defect segmentation mask of FIG6(a); FIG6(c) is a close-up original image of the bridge structure after ROI extraction of the present invention, and FIG6(d) is a predicted defect segmentation mask of FIG6(c);

[0047] Figure 7 It is a three-dimensional visualization effect diagram of the disease of the present invention. DETAILED DESCRIPTION

[0048] The following is a further detailed description of a bridge defect detection and positioning method based on 2D-3D data fusion according to the present invention with reference to the accompanying drawings.

[0049] Example 1

[0050] The present invention provides a bridge disease detection and positioning method based on 2D-3D data fusion, such as Figure 1 As shown, it specifically includes the following steps:

[0051] A. This example involves the piers of a multi-span simply supported beam bridge. First, a drone was used on-site to collect image data. The drone captured the entire bridge in 360°, ensuring that the overlap between adjacent images should not be less than 75%, and the angle between adjacent images should not exceed 15°. This ultimately yielded multi-view image data I1. The drone then flew close to the bridge surface, capturing close-up images to ensure that the image clarity met defect detection requirements. Finally, close-up image data I2 was collected.

[0052] B. Reconstruct the three-dimensional point cloud model of the bridge pier using the multi-view image data I1, such as Figure 2 As shown, the depth image D corresponding to the multi-view image is generated by coordinate system conversion and saved as a grayscale image. Some depth images are as follows Figure 3(a) to Figure 3(b) As shown;

[0053] C. Build and train a ROI extraction model based on RGB-D images to output a bridge image with complex background removed.

[0054] C1. Build a MobileSal model based on RGB-D images for image ROI extraction. The model consists of three parts: an encoder, a feature fusion module, and a decoder.

[0055] C2. Annotate the multi-view image I1 of the bridge structure, with the annotated objects being the pixels of the bridge structure in the image, and generate a corresponding segmentation mask for the multi-view image I1 of the bridge structure; integrate the corresponding multi-view image I1 of the bridge structure, the depth image D, and the segmentation mask to form an RGB-D dataset for training the image ROI extraction model; the RGB-D dataset in this example contains a total of 519 images, of which 415 are used for model training and the remaining 104 are used for model testing;

[0056] C3. Use the training data to train the ROI extraction model, and use the test set to evaluate the performance of the ROI extraction model. The results are shown in Table 1. The model using the depth map has a significant improvement in segmentation accuracy compared to the original model, and the model parameters only increase by 22%, which can also meet the real-time requirements. Use the trained ROI extraction model to segment the image data I2 and output the image data I2 after removing the background. The data after some ROI extraction is shown in Table 1. Figure 4(a) to Figure 4(b) As shown;

[0057] Table 1 ROI extraction model accuracy

[0058]

[0059] D. Build and train an improved DeepLabv3+ disease segmentation model to output disease segmentation masks;

[0060] D1. Build and improve DeepLabv3+ disease segmentation model, such as Figure 5 As shown in the figure, the model consists of an encoder and a decoder: the encoder uses ResNet101 as the backbone network and combines the Atrous Spatial Pyramid Pooling (ASPP) technology to extract image features of different receptive fields, thereby generating image features that integrate multi-scale information; the decoder processes the low-level feature maps from the middle layer of the backbone network and the output of the ASPP module, and generates a predicted mask through multiple convolution and interpolation upsampling operations; to improve the image segmentation accuracy, the focal loss function is introduced in the DeepLabv3+ model to replace the original cross entropy loss function;

[0061] D2. Collect image data of bridge structure defects and annotate the images. The annotation targets are the pixels in the images that represent bridge defects, including four types of defects: cracks, voids, spalling, and water erosion. Copy-paste data augmentation technology is introduced to enhance the defect images and their annotations, increasing the proportion of defect pixels and alleviating the inter-class imbalance in the data. This creates a defect dataset for improving the DeepLabv3+ defect segmentation model training. In this example, a total of 4,571 images of defective bridges were selected, 80% of which were used for model training and the remaining 20% ​​for testing.

[0062] D3. The improved DeepLabv3+ disease segmentation model was trained using the training dataset, and the model performance was evaluated using the test dataset. The results are shown in Table 2. The improved DeepLabv3+ disease segmentation model significantly improves segmentation accuracy compared to the original model, significantly alleviating the inter-class imbalance problem in the original model.

[0063] Table 2 Disease segmentation model accuracy

[0064]

[0065] D4. Use the trained improved DeepLabv3+ disease segmentation model to detect the background-removed image data I2 and output the corresponding disease segmentation mask. The disease segmentation mask effect before and after removing the ROI is as follows: Figure 6(a) to Figure 6(d) As shown in the figure, ROI extraction effectively improves the accuracy of disease segmentation;

[0066] E. 3D visualization of bridge defects based on defect segmentation mask and reconstructed bridge point cloud;

[0067] E1. To prevent hidden points behind the bridge target point from being incorrectly displayed in the point cloud from the current image perspective, a hidden point removal algorithm is used to calculate the visible points of the point cloud from the current perspective based on the shooting pose of the current defect image.

[0068] E2. Project the point cloud containing only visible points onto a 2D image based on the camera's intrinsic and extrinsic parameters, and establish a correspondence between the image pixels and the points in the 3D point cloud.

[0069] E3. Combine the damage segmentation mask with the projected 2D image to determine whether a pixel in the 2D image is damaged. For pixels identified as damaged, index the corresponding point in the bridge point cloud, change the color of the point to the preset damage color, and attach damage information.

[0070] E4. Traverse all the disease segmentation masks and display the disease information detected from the two-dimensional image on the three-dimensional point cloud to achieve three-dimensional visualization of bridge diseases. The three-dimensional visualization effect is as follows: Figure 7As shown, it can be seen that the bridge has obvious defects in the cap beam.

[0071] Example 2

[0072] This embodiment relates to a bridge defect detection and positioning device based on 2D-3D data fusion, including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement a bridge defect detection and positioning method based on 2D-3D data fusion according to Example 1.

[0073] The contents described in the implementation cases of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the implementation cases. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A bridge defect detection and location method based on 2D-3D data fusion, characterized in that: The following steps are involved: A. Use drones to collect multi-view image data I1 and close-range image data I2 of bridge structures; B. Reconstructing a three-dimensional point cloud model of the bridge using the multi-view image data I1 of the bridge structure, generating a depth image D corresponding to the multi-view image through coordinate system conversion, and saving it as a grayscale image; C. Build and train a ROI extraction model based on RGB-D images to output a bridge image with complex background removed. C1. Build a MobileSal model based on RGB-D images for image ROI extraction. The model consists of three parts: an encoder, a feature fusion module, and a decoder. C2. Annotating the bridge structure multi-view image data I1, where the annotated objects are the pixels of the bridge structure portion of the image, and generating a corresponding segmentation mask for the bridge structure multi-view image data I1; integrating the corresponding bridge structure multi-view image data I1, the depth image D, and the segmentation mask to form an RGB-D dataset for training the image ROI extraction model; C3. Split the RGB-D dataset into a training set and a test set, where the training set is used to train the ROI extraction model and the test set is used to evaluate the performance of the ROI extraction model; use the trained ROI extraction model to segment the image data I2 and output the background-removed image data I2; D. Build and train an improved DeepLabv3+ disease segmentation model to output disease segmentation masks; D1. Build an improved DeepLabv3+ disease segmentation model, consisting of an encoder and decoder. The encoder uses ResNet101 as the backbone network and combines it with Atrous Spatial Pyramid Pooling to extract image features with different receptive fields, thereby generating image features that integrate multi-scale information. The decoder processes low-level feature maps from the backbone network's intermediate layers and the output of the ASPP module, and generates predicted masks through multiple convolutions and interpolation upsampling operations. D2. To improve image segmentation accuracy, a focal loss function is introduced into the DeepLabv3+ disease segmentation model to replace the original cross-entropy loss function. D3. Collect image data of bridge structure defects and annotate the images. The annotated objects are the pixels in the images that represent bridge defects, including four types of defects: cracks, voids, spalling, and water erosion. Copy-paste data augmentation technology is introduced to enhance the defect images and their annotated information, increasing the proportion of defect pixels and alleviating the inter-class imbalance in the data. This will form a defect dataset for improving the training of the DeepLabv3+ defect segmentation model. D4. Split the enhanced disease dataset into a training set and a test set. The training set is used to train the improved DeepLabv3+ disease segmentation model, and the test set is used to evaluate model performance. Use the trained improved DeepLabv3+ disease segmentation model to detect the background-removed image data I2 and output the corresponding disease segmentation mask. E. 3D visualization of bridge defects based on defect segmentation mask and reconstructed bridge point cloud; E1. To prevent hidden points behind the bridge target point from being incorrectly displayed in the point cloud from the current image perspective, a hidden point removal algorithm is used to calculate the visible points of the point cloud from the current perspective based on the shooting pose of the current defect image. E2. Project the point cloud containing only visible points onto a 2D image based on the camera's intrinsic and extrinsic parameters, and establish a correspondence between the image pixels and the points in the 3D point cloud. E3. Combine the damage segmentation mask with the projected 2D image to determine whether a pixel in the 2D image is damaged. For pixels identified as damaged, index the corresponding point in the bridge point cloud, change the color of the point to the preset damage color, and attach damage information. E4. Traverse all the defect segmentation masks and display the defect information detected from the 2D image on the 3D point cloud to achieve 3D visualization of bridge defects.

2. The bridge defect detection and location method based on 2D-3D data fusion according to claim 1 is characterized by: The method for collecting the multi-perspective image data I1 and the close-range image data I2 of the bridge structure in step A is as follows: using a drone device to perform 360° surround shooting of the entire bridge, the overlapping portion between adjacent images should not be less than 75%, and the shooting angle between adjacent images should not exceed 15°, and finally the multi-perspective image data I1 of the bridge structure is collected; using a drone device to perform close-range shooting close to the bridge surface to ensure that the image clarity meets the requirements for disease detection, and finally the close-range image data I2 is collected.

3. The bridge defect detection and location method based on 2D-3D data fusion according to claim 1 is characterized by: The method used for reconstructing the three-dimensional point cloud model of the bridge in step B is the COLMAP modeling software based on Structure from Motion.

4. The bridge defect detection and location method based on 2D-3D data fusion according to claim 1 is characterized by: The encoder in step C1 includes two branches: RGB stream and depth stream. Both use the MobileNet v2 network as the base network, and the number of network layers of the depth stream is much smaller than that of the RGB stream, so as to keep the model lightweight.

5. The bridge defect detection and location method based on 2D-3D data fusion according to claim 1 is characterized by: The RGB-D dataset in step C2 consists of three images: a color image in RGB format, a depth image in grayscale format, and a binary segmentation mask. There is a strict one-to-one correspondence between the three images.

6. The bridge defect detection and location method based on 2D-3D data fusion according to claim 1 is characterized by: The formula of focal loss in step D2 is: FL(p)=-β(1-p) γ log(p) (2) Where β represents the pixel weights of different disease categories in the model, p represents the probability that the model predicts that the pixel belongs to the foreground, and γ is a parameter preset to 2; when p is close to 1, it means that the pixel is easy to distinguish, (1-p) γ Close to 0, thus reducing the loss caused by these easily distinguishable pixels; The focal loss function can effectively reduce the impact of a large number of background pixel loss values ​​in the data.

7. The bridge defect detection and location method based on 2D-3D data fusion according to claim 1 is characterized by: The copy-paste data enhancement technology in step D3 includes two parts: random transformation and defect pasting. The random transformation part specifically includes randomly selecting a source image P1 and a target image P2, and performing random translation, flipping, and resizing operations on the source image, target image, and corresponding annotation information. The defect pasting part specifically includes extracting pixels of the defect area from the transformed source image using the annotation information, and randomly pasting the extracted pixels onto the transformed target image to finally generate an enhanced image.

8. The bridge defect detection and location method based on 2D-3D data fusion according to claim 1 is characterized by: The pre-set disease colors in step E3 are specifically: cracks - yellow, voids - green, peeling - purple, and water erosion - red.

9. A bridge defect detection and positioning device based on 2D-3D data fusion, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method is used to implement a bridge defect detection and positioning method based on 2D-3D data fusion according to any one of claims 1 to 8.

Images