Bridge disease detection and positioning method and device based on 2D-3D data fusion

Abstract

The invention discloses a bridge disease detection and positioning method and device based on 2D-3D data fusion, and the method can achieve the recognition of bridge diseases under the conditions of a complex background and unbalanced data through the homogeneity between an image and a point cloud. The method comprises the following specific implementation steps: (1) collecting a multi-view image and a disease image of a bridge structure; (2) reconstructing a point cloud model of the bridge structure by using the multi-view image and outputting a depth image; (3) building and training an ROI (Region of Interest) extraction model, and outputting a disease image of which the complex background is removed by utilizing the depth image; (4) building and training an improved DeepLabv3 + disease segmentation model, detecting a bridge disease in the disease image after ROI extraction, and outputting a bridge disease segmentation mask; and (5) realizing three-dimensional visualization of the disease based on the bridge disease segmentation mask and the point cloud model. According to the method disclosed by the invention, high-precision detection and positioning of bridge diseases can be realized, and a powerful tool is provided for daily detection of a bridge structure.

Description

Technical Field

[0001] The present invention relates to a bridge structure defect detection method and device, specifically to a bridge structure defect identification and three-dimensional positioning technology based on 2D-3D data fusion and computer vision technology, and belongs to the field of structural engineering. Background Art

[0002] Bridges are an important part of transportation infrastructure. In daily operation, they are inevitably affected by various adverse environments and external loads, which accelerate structural damage and shorten their service life. Various types of apparent defects such as cracks, concrete spalling and voids are external manifestations of bridge conditions. Their appearance indicates that the durability and even the bearing capacity of the bridge have declined, seriously affecting the structural safety. According to incomplete statistics, about 40% of the bridges in service in my country's highway network have been in service for more than 20 years, 30% of the bridges are eroded by diseases, and more than 100,000 bridges are dangerous bridges. Therefore, it is crucial to carry out routine inspections to ensure the safety of bridge structures.

[0003] Traditional bridge inspection usually uses manual visual assessment and large-scale inspection equipment, which is costly, inefficient, and has major safety hazards. With the rapid development of technologies such as deep learning and artificial intelligence, applying computer vision methods to images obtained from drone survey activities has gradually become one of the emerging means of bridge disease detection. However, since most bridges are located in natural environments, images captured by drones often contain irrelevant complex background elements such as rivers, trees, hills, and the sky, which greatly affects the accuracy of disease identification; in addition, deep learning models usually require a large number of high-quality data sets for training, and the existing bridge disease data sets have serious inter-class imbalance problems due to the small size of the disease, which may lead to poor model training results; finally, the actual location of bridge diseases in three-dimensional space is also difficult to determine, which plays a very important role in the subsequent in-depth analysis of the disease. Therefore, it is necessary to develop a high-precision and high-robust disease detection and localization method to identify bridge diseases in complex backgrounds. Summary of the invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a bridge defect detection and positioning method and device based on 2D-3D data fusion to improve the feasibility of bridge defect detection based on computer vision technology in practical application. The specific contents include:

[0005] The first aspect of the present invention relates to a bridge disease detection and positioning method based on 2D-3D data fusion, comprising the following steps:

[0006] A. Use drones to collect multi-view image data I1 of bridge structures and close-range image data I2 of bridge structures;

[0007] B. Reconstructing a three-dimensional point cloud model of the bridge using the multi-view image data I1, and generating a depth image D corresponding to the multi-view image by coordinate system conversion;

[0008] The process of generating the depth image D by coordinate transformation includes: using the camera pose to transform the point cloud coordinates (U, V, W) into the pixel coordinate system (u, v), which can be expressed as:

[0009]

[0010] Where K1 and K2 represent the external parameters of the camera, f c represents the focal length of the camera, dx and dy represent the ratio of the pixel to the physical size in the x and y directions respectively, (u0, v0) represents the coordinates of the pixel origin in the image coordinate system, and Z represents the depth value; the corresponding depth information Z is used as the pixel value of the projected image and saved as a grayscale image;

[0011] C. Build and train a ROI (Region of Interest) extraction model based on RGB-D images, and output the bridge image after removing the complex background;

[0012] C1. Build a MobileSal model based on RGB-D images for image ROI extraction. The model consists of three parts: encoder, feature fusion module and decoder;

[0013] C2. Annotate the bridge structure multi-view image I1, where the annotated object is the pixels of the bridge structure part in the image, and generate a corresponding segmentation mask of the bridge structure multi-view image I1; integrate the corresponding bridge structure multi-view image I1, the depth image D and the segmentation mask to form an RGB-D data set for image ROI extraction model training;

[0014] C3. Segment the RGB-D data set into a training set and a test set, the training data set is used to train the ROI extraction model, and the test data set is used to evaluate the performance of the ROI extraction model; use the trained ROI extraction model to segment the image data I2, and output the image data I2 with the background removed;

[0015] D. Build and train the improved DeepLabv3+ disease segmentation model and output the disease segmentation mask;

[0016] D1. Build an improved DeepLabv3+ disease segmentation model, which consists of an encoder and a decoder: the encoder uses ResNet101 as the backbone network and combines the Atrous Spatial Pyramid Pooling (ASPP) technology to extract image features of different receptive fields, thereby generating image features that integrate multi-scale information; the decoder processes the low-level feature maps from the middle layer of the backbone network and the output of the ASPP module, and generates predicted masks through multiple convolution and interpolation upsampling operations;

[0017] D2. In order to improve the image segmentation accuracy, the focal loss function is introduced into the DeepLabv3+ model to replace the original cross entropy loss function;

[0018] D3. Collect image data of bridge structure damage and annotate the collected images. The annotated objects are the pixels of bridge damage in the images, including four types of damage: cracks, voids, spalling, and water erosion. Introduce copy-paste data enhancement technology to enhance the damage images and their annotation information, increase the proportion of damage pixels, alleviate the problem of inter-class imbalance in the data, and then form a damage dataset for improving the training of DeepLabv3+ damage segmentation model.

[0019] D4. The enhanced disease dataset is divided into a training set and a test set, the training dataset is used to train the improved DeepLabv3+ disease segmentation model, and the test dataset is used to evaluate the model performance; the trained improved DeepLabv3+ disease segmentation model is used to detect the background-removed image data I2, and the corresponding disease segmentation mask is output;

[0020] E. Realize 3D visualization of bridge defects based on defect segmentation mask and reconstructed bridge point cloud;

[0021] E1. In order to avoid the hidden points behind the bridge target points from being displayed incorrectly in the point cloud seen from the current image perspective, a hidden point removal algorithm is used to calculate the visible points of the point cloud from the current perspective according to the shooting pose of the current defect image;

[0022] E2. Project the point cloud containing only visible points onto the two-dimensional image according to the camera internal and external parameters, and establish the correspondence between the image pixels and the points in the three-dimensional point cloud;

[0023] E3. Combine the damage segmentation mask with the projected two-dimensional image to determine whether the pixel point on the two-dimensional image is damaged; according to the pixel point confirmed as damaged, index the corresponding point in the bridge point cloud, change the color of the point to the preset damage color, and attach the damage information;

[0024] E4. Traverse all the defect segmentation masks and display the defect information detected from the two-dimensional image on the three-dimensional point cloud to achieve three-dimensional visualization of bridge defects.

[0025] Furthermore, the method for collecting the multi-perspective image data I1 and the close-range image data I2 of the bridge structure in step A is as follows: based on the drone equipment, a 360° surround shot of the entire bridge is performed, the overlapping part between adjacent images should not be less than 75%, and the shooting angle between adjacent images should not exceed 15°, and finally the multi-perspective image data I1 is collected; based on the drone equipment, close-range shooting is performed close to the bridge surface to ensure that the clarity of the image meets the requirements of disease detection, and finally the close-range image data I2 is collected.

[0026] Furthermore, the method used to reconstruct the three-dimensional point cloud model of the bridge in step B is COLMAP modeling software based on Structure from Motion (SfM).

[0027] Furthermore, the encoder in step C1 includes two branches, an RGB stream and a depth stream, both of which use the MobileNet v2 network as the basic network, and the number of network layers of the depth stream is much smaller than that of the RGB stream, so as to keep the model lightweight.

[0028] Furthermore, the RGB-D data set in step C2 is composed of three images: a color image in RGB format, a depth image in grayscale format, and a binary segmentation mask, and there is a strict one-to-one correspondence between the three images.

[0029] Furthermore, the formula of focal loss in step D2 is:

[0030] FL(p)=-β(1-p) γ log(p) (2)

[0031] Where β represents the pixel weights of different types of diseases in the model, p represents the probability that the model predicts that the pixel belongs to the foreground, and γ is a parameter preset to 2; when p is close to 1, it means that the pixel sample is easy to distinguish, (1-p) γ Close to 0, thereby reducing the loss caused by these easily distinguishable pixels; through the focal loss function, the impact of a large number of background pixel loss values ​​in the data can be effectively reduced.

[0032] Furthermore, the copy-paste data enhancement technology in step D3 includes two parts: random transformation and defect pasting. The random transformation part is specifically: randomly selecting a source image P1 and a target image P2, and performing random translation, flipping and resizing operations on the source image, the target image and the corresponding annotation information; the defect pasting part is specifically: using the annotation information to extract pixels of the defect area from the transformed source image, and randomly pasting the extracted pixels to the transformed target image, and finally generating an enhanced image P aug It can be expressed as

[0033] P aug =αP1+(1-α)P2 (3)

[0034] Where α represents the annotation mask of the source image.

[0035] Furthermore, the disease colors pre-set in step E3 are specifically: cracks-yellow, voids-green, peeling-purple, water erosion-red.

[0036] The second aspect of the present invention relates to a bridge defect detection and positioning device based on 2D-3D data fusion, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a bridge defect detection and positioning method based on 2D-3D data fusion of the present invention.

[0037] Compared with the existing methods, the present invention has the following advantages:

[0038] (1) The proposed ROI extraction model combines 2D RGB images and 3D depth information to accurately extract bridge pixels from complex backgrounds, avoiding the computational burden of large-scale point cloud processing.

[0039] (2) Compared with the original DeepLabv3 model, the improved DeepLabv3+ model integrates copy-paste data enhancement and focal loss technology, solves the serious inter-class imbalance problem in the dataset, and significantly improves the segmentation accuracy;

[0040] (3) A damage localization method based on camera pose is proposed. The hidden point removal algorithm is used to calculate the visible points in the point cloud, thus realizing the three-dimensional visualization of bridge damage. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method of the present invention;

[0042] Figure 2 is a reconstructed three-dimensional point cloud image of the bridge structure of the present invention;

[0043] Figure 3(a) to Figure 3(b) 3(a) is the original image and FIG3(b) is the corresponding depth image;

[0044] Figure 4(a) to Figure 4(b) 4(a) is the original image, and FIG4(b) is the corresponding image after ROI extraction;

[0045] Figure 5 It is the structure diagram of the improved DeepLabv3+ model of the present invention;

[0046] FIG6(a) is a close-up original image of the bridge structure of the present invention, and FIG6(b) is a predicted disease segmentation mask of FIG6(a); FIG6(c) is a close-up original image of the bridge structure after ROI extraction of the present invention, and FIG6(d) is a predicted disease segmentation mask of FIG6(c);

[0047] Figure 7 It is a three-dimensional visualization effect diagram of the disease of the present invention. DETAILED DESCRIPTION

[0048] The following is a further detailed description of a bridge defect detection and positioning method based on 2D-3D data fusion according to the present invention in conjunction with the accompanying drawings.

[0049] Example 1

[0050] The present invention provides a bridge disease detection and positioning method based on 2D-3D data fusion, such as Figure 1 As shown, the specific steps include:

[0051] A. The object of this example is the pier part of a multi-span simply supported beam bridge. First, the image data is collected on site using a drone: the entire bridge is photographed 360° around the drone, the overlap between adjacent images should not be less than 75%, and the shooting angle between adjacent images should not exceed 15°, and finally multi-view image data I1 is collected; the drone is used to take close-up photos close to the bridge surface to ensure that the image clarity meets the requirements for disease detection, and finally close-up image data I2 is collected;

[0052] B. Reconstruct the three-dimensional point cloud model of the bridge pier using the multi-view image data I1, such as Figure 2 As shown, the depth image D corresponding to the multi-view image is generated by coordinate system transformation and saved as a grayscale image. Some depth images are shown in Figure 3(a) to Figure 3(b) As shown;

[0053] C. Build and train a ROI extraction model based on RGB-D images, and output the bridge image after removing the complex background;

[0054] C1. Build a MobileSal model based on RGB-D images for image ROI extraction. The model consists of three parts: encoder, feature fusion module and decoder;

[0055] C2. Annotate the multi-view image I1 of the bridge structure, where the annotated object is the pixels of the bridge structure part in the image, and generate a corresponding segmentation mask of the multi-view image I1 of the bridge structure; integrate the corresponding multi-view image I1 of the bridge structure, the depth image D and the segmentation mask, so as to form an RGB-D data set for image ROI extraction model training; the RGB-D data set of this example contains a total of 519 groups of images, of which 415 are used for model training and the remaining 104 are used for model testing;

[0056] C3. Use the training data to train the ROI extraction model, and use the test set to evaluate the performance of the ROI extraction model. The results are shown in Table 1. The model using the depth map has a significant improvement in segmentation accuracy compared to the original model, and the model parameters are only increased by 22%, which can also meet the real-time requirements. Use the trained ROI extraction model to segment the image data I2 and output the image data I2 after removing the background. The data after some ROI extraction is shown in Table 1. Figure 4(a) to Figure 4(b) As shown;

[0057] Table 1 ROI extraction model accuracy

[0058]

[0059] D. Build and train the improved DeepLabv3+ disease segmentation model and output the disease segmentation mask;

[0060] D1. Build and improve DeepLabv3+ disease segmentation model, such as Figure 5 As shown in the figure, the model consists of an encoder and a decoder: the encoder uses ResNet101 as the backbone network and combines the Atrous Spatial Pyramid Pooling (ASPP) technology to extract image features of different receptive fields, thereby generating image features that integrate multi-scale information; the decoder processes the low-level feature maps from the middle layer of the backbone network and the output of the ASPP module, and generates a predicted mask through multiple convolution and interpolation upsampling operations; in order to improve the image segmentation accuracy, the focal loss function is introduced in the DeepLabv3+ model to replace the original cross entropy loss function;

[0061] D2. Collect image data of bridge structure damage and annotate the collected images. The annotated objects are the pixels of bridge damage in the images, including four types of damage: cracks, voids, spalling, and water erosion. Introduce copy-paste data enhancement technology to enhance the damage images and their annotation information, increase the proportion of damage pixels, alleviate the problem of inter-class imbalance in the data, and then form a damage dataset for improving the training of DeepLabv3+ damage segmentation model. In this example, a total of 4571 images of damage bridges were selected, of which 80% were used for model training and the remaining 20% ​​were used for testing.

[0062] D3. The improved DeepLabv3+ disease segmentation model was trained using the training data set, and the model performance was evaluated using the test data set. The results are shown in Table 2. The improved DeepLabv3+ disease segmentation model has significantly improved the segmentation accuracy compared with the original model, and significantly alleviated the inter-class imbalance problem in the original model.

[0063] Table 2 Disease segmentation model accuracy

[0064]

[0065] D4. Use the trained improved DeepLabv3+ disease segmentation model to detect the image data I2 with the background removed, and output the corresponding disease segmentation mask. The disease segmentation mask effect before and after removing the ROI is as follows: Figure 6(a) to Figure 6(d) As shown, ROI extraction effectively improves the accuracy of disease segmentation;

[0066] E. Realize 3D visualization of bridge defects based on defect segmentation mask and reconstructed bridge point cloud;

[0067] E1. In order to avoid the hidden points behind the bridge target points from being displayed incorrectly in the point cloud seen from the current image perspective, a hidden point removal algorithm is used to calculate the visible points of the point cloud from the current perspective according to the shooting pose of the current defect image;

[0068] E2. Project the point cloud containing only visible points onto the two-dimensional image according to the camera internal and external parameters, and establish the correspondence between the image pixels and the points in the three-dimensional point cloud;

[0069] E3. Combine the damage segmentation mask with the projected two-dimensional image to determine whether the pixel point on the two-dimensional image is damaged; according to the pixel point confirmed as damaged, index the corresponding point in the bridge point cloud, change the color of the point to the preset damage color, and attach the damage information;

[0070] E4. Traverse all the disease segmentation masks and display the disease information detected from the two-dimensional image on the three-dimensional point cloud to achieve three-dimensional visualization of bridge diseases. The three-dimensional visualization effect is as follows: Figure 7As shown, it can be seen that the bridge has obvious defects in the cap beam.

[0071] Example 2

[0072] The present embodiment relates to a bridge defect detection and positioning device based on 2D-3D data fusion, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a bridge defect detection and positioning method based on 2D-3D data fusion of Example 1.

[0073] The contents described in the implementation cases of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be deemed to be limited to the specific forms described in the implementation cases. The protection scope of the present invention also extends to the equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A bridge disease detection and location method based on 2D-3D data fusion, characterized in that: The following steps are involved: A. Use drones to collect multi-view image data I1 of bridge structures and close-range image data I2 of bridge structures; B. Reconstructing a three-dimensional point cloud model of the bridge using the multi-view image data I1, and generating a depth image D corresponding to the multi-view image by coordinate system conversion; The process of generating the depth image D by coordinate transformation includes: using the camera pose to transform the point cloud coordinates (U, V, W) into the pixel coordinate system (u, v), which can be expressed as: Where K1 and K2 represent the external parameters of the camera, f c represents the focal length of the camera, dx and dy represent the ratio of the pixel to the physical size in the x and y directions respectively, (u0, v0) represents the coordinates of the pixel origin in the image coordinate system, and Z represents the depth value; the corresponding depth information Z is used as the pixel value of the projected image and saved as a grayscale image; C. Build and train a ROI (Region of Interest) extraction model based on RGB-D images, and output the bridge image after removing the complex background; C1. Build a MobileSal model based on RGB-D images for image ROI extraction. The model consists of three parts: encoder, feature fusion module and decoder; C2. Annotate the bridge structure multi-view image I1, where the annotated object is the pixels of the bridge structure part in the image, and generate a corresponding segmentation mask of the bridge structure multi-view image I1; integrate the corresponding bridge structure multi-view image I1, the depth image D and the segmentation mask to form an RGB-D data set for image ROI extraction model training; C3. Segment the RGB-D data set into a training set and a test set, the training data set is used to train the ROI extraction model, and the test data set is used to evaluate the performance of the ROI extraction model; use the trained ROI extraction model to segment the image data I2, and output the image data I2 with the background removed; D. Build and train the improved DeepLabv3+ disease segmentation model and output the disease segmentation mask; D1. Build an improved DeepLabv3+ disease segmentation model, which consists of an encoder and a decoder: the encoder uses ResNet101 as the backbone network and combines the Atrous Spatial Pyramid Pooling (ASPP) technology to extract image features of different receptive fields, thereby generating image features that integrate multi-scale information; the decoder processes the low-level feature maps from the middle layer of the backbone network and the output of the ASPP module, and generates predicted masks through multiple convolution and interpolation upsampling operations; D2. In order to improve the image segmentation accuracy, the focal loss function is introduced into the DeepLabv3+ model to replace the original cross entropy loss function; D3. Collect image data of bridge structure damage and annotate the collected images. The annotated objects are the pixels of bridge damage in the images, including four types of damage: cracks, voids, spalling, and water erosion. Introduce copy-paste data enhancement technology to enhance the damage images and their annotation information, increase the proportion of damage pixels, alleviate the problem of inter-class imbalance in the data, and then form a damage dataset for improving the training of DeepLabv3+ damage segmentation model. D4. The enhanced disease dataset is divided into a training set and a test set, the training dataset is used to train the improved DeepLabv3+ disease segmentation model, and the test dataset is used to evaluate the model performance; the trained improved DeepLabv3+ disease segmentation model is used to detect the background-removed image data I2, and the corresponding disease segmentation mask is output; E. Realize 3D visualization of bridge defects based on defect segmentation mask and reconstructed bridge point cloud; E1. In order to avoid the hidden points behind the bridge target points from being displayed incorrectly in the point cloud seen from the current image perspective, a hidden point removal algorithm is used to calculate the visible points of the point cloud from the current perspective according to the shooting pose of the current defect image; E2. Project the point cloud containing only visible points onto the two-dimensional image according to the camera internal and external parameters, and establish the correspondence between the image pixels and the points in the three-dimensional point cloud; E3. Combine the damage segmentation mask with the projected two-dimensional image to determine whether the pixel point on the two-dimensional image is damaged; according to the pixel point confirmed as damaged, index the corresponding point in the bridge point cloud, change the color of the point to the preset damage color, and attach the damage information; E4. Traverse all the defect segmentation masks and display the defect information detected from the two-dimensional image on the three-dimensional point cloud to achieve three-dimensional visualization of bridge defects.

2. The bridge defect detection and positioning method based on 2D-3D data fusion according to claim 1 is characterized by: The method for collecting the multi-perspective image data I1 and the close-range image data I2 of the bridge structure in step A is as follows: using drone equipment to perform 360° surround shooting of the entire bridge, the overlapping portion between adjacent images should not be less than 75%, and the shooting angle between adjacent images should not exceed 15°, and finally the multi-perspective image data I1 is collected; using drone equipment to perform close-range shooting close to the surface of the bridge to ensure that the clarity of the image meets the requirements for disease detection, and finally the close-range image data I2 is collected.

3. The bridge defect detection and positioning method based on 2D-3D data fusion according to claim 1 is characterized by: The method used for reconstructing the three-dimensional point cloud model of the bridge in step B is COLMAP modeling software based on Structure from Motion (SfM).

4. The bridge defect detection and positioning method based on 2D-3D data fusion according to claim 1 is characterized by: The encoder in step C1 includes two branches, an RGB stream and a depth stream, both of which use the MobileNet v2 network as the basic network, and the number of network layers of the depth stream is much smaller than that of the RGB stream, so as to keep the model lightweight.

5. The bridge defect detection and positioning method based on 2D-3D data fusion according to claim 1 is characterized by: The RGB-D data set in step C2 consists of three images: a color image in RGB format, a depth image in grayscale format, and a binary segmentation mask. There is a strict one-to-one correspondence between the three images.

6. The bridge defect detection and positioning method based on 2D-3D data fusion according to claim 1 is characterized by: The formula of focal loss in step D2 is: FL(p)=-β(1-p) γ log(p) (2) Where β represents the pixel weights of different types of diseases in the model, p represents the probability that the model predicts that the pixel belongs to the foreground, and γ is a parameter preset to 2; when p is close to 1, it means that the pixel sample is easy to distinguish, (1-p) γ Close to 0, thus reducing the loss caused by these easily distinguishable pixels; Through the focal loss function, the influence of a large number of background pixel loss values ​​in the data can be effectively reduced.

7. The bridge defect detection and positioning method based on 2D-3D data fusion according to claim 1 is characterized by: The copy-paste data enhancement technology in step D3 includes two parts: random transformation and defect pasting. The random transformation part is specifically: randomly selecting a source image P1 and a target image P2, and performing random translation, flipping and resizing operations on the source image, the target image and the corresponding annotation information; the defect pasting part is specifically: using the annotation information to extract pixels of the defect area from the transformed source image, and randomly pasting the extracted pixels to the transformed target image, and finally generating an enhanced image P aug It can be expressed as P aug =αP1+(1-α)P2 (3) Where α represents the annotation mask of the source image.

8. The bridge defect detection and positioning method based on 2D-3D data fusion according to claim 1 is characterized by: The disease colors pre-set in step E3 are specifically: cracks-yellow, voids-green, peeling-purple, and water erosion-red.

9. A bridge disease detection and positioning device based on 2D-3D data fusion, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement a bridge defect detection and positioning method based on 2D-3D data fusion as described in any one of claims 1 to 8.

Images