Multi-type bridge substructure crack depth detection system

Abstract

The invention relates to the technical field of bridge detection, in particular to a multi-type bridge substructure crack depth detection system. The system comprises a data acquisition module, a three-dimensional reconstruction module and a crack detection module. The data acquisition module acquires a multi-view image of a crack area at the lower part of the bridge through a camera and a checkerboard calibration plate, and generates a multi-view crack feature image set; the three-dimensional reconstruction module generates a high-precision three-dimensional point cloud by using the multi-view crack image, and constructs a depth map of a crack region; and the crack detection module realizes classification of crack types and regression prediction of crack depths based on a deep learning model. According to the method, accurate calculation of the crack depth and automatic identification of multiple types of cracks can be efficiently realized, danger early warning is given out in combination with the depth threshold value, and reliable data support is provided for bridge maintenance.

Description

Technical Field

[0001] The invention relates to the technical field of bridge detection, and in particular to a crack depth detection system for substructures of multiple types of bridges. Background Art

[0002] As an important part of modern transportation infrastructure, bridges play a vital role in transportation. However, as the service life of bridges increases, the aging and damage of bridges are becoming increasingly prominent, especially cracks in the bridge substructure. Cracks are a common manifestation of bridge structural damage, and their occurrence is usually closely related to factors such as the load borne by the bridge, environmental factors, material fatigue, and construction quality. If cracks are not detected and repaired in time, they may lead to further damage to the bridge structure, and in severe cases may even cause structural failure and safety accidents.

[0003] Traditional bridge crack detection methods usually rely on manual inspections and simple visual inspections. Although these methods can detect obvious cracks in some cases, they still have many limitations. First, manual inspections require a lot of manpower and time, and the experience level of the inspectors directly affects the accuracy of the detection results. Secondly, manual inspections are difficult to fully cover all parts of the bridge structure, especially some difficult-to-reach areas, which are prone to miss subtle cracks. In addition, traditional visual inspection methods are relatively rough in judging information such as crack depth and morphology, and cannot provide accurate crack parameters.

[0004] In order to improve the efficiency and accuracy of crack detection, automated detection methods based on image processing and computer vision have been widely used in recent years. These methods usually use cameras to capture images of the bridge surface and combine image processing algorithms to identify the location and type of cracks. However, such methods still face certain challenges, especially in terms of crack detection accuracy and depth estimation. Most existing image processing methods rely on single-view images for analysis, resulting in the inability to fully capture the three-dimensional features and details of the cracks. In addition, single-view images often cannot effectively distinguish different types of cracks, especially when the crack morphology is complex or the crack spacing is small, which may lead to misjudgment. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-type bridge substructure crack depth detection system, which combines multi-view image acquisition, three-dimensional reconstruction and deep learning technology to efficiently realize accurate detection of bridge cracks. The system first collects multi-view images of the crack area under the bridge through a camera and a checkerboard calibration plate to ensure that the image overlap rate is moderate and the crack details are fully captured. Then, a high-precision three-dimensional point cloud is generated using a three-dimensional reconstruction module to further construct a depth map of the crack area. Finally, combined with a deep learning model, the system can automatically identify the crack type and regress its depth value, providing effective crack analysis and evaluation.

[0006] To this end, the technical solution adopted by the present invention is as follows: a multi-type bridge substructure crack depth detection system, the system comprising: M1: Data acquisition module. This module uses a camera and a checkerboard calibration plate to collect multi-view images of the crack area under the bridge to generate a multi-view crack feature image set. The multi-view crack feature image of a single crack is represented as ,in represents the crack feature image at the kth viewing angle; The chessboard calibration plate is The matrix contains W rows, H columns and grids with side length d.

[0007] M2: 3D reconstruction module, which converts the multi-view crack feature images collected by M1 into 3D point clouds; the 3D point clouds are used to accurately describe the geometric morphology of the cracks. The implementation steps of this module include S21 camera calibration, S22 3D reconstruction and S23 depth map generation; M3, crack detection module, this module uses a preset deep learning model to identify the depth map D(x, y) output by M2. The detection tasks of this module are divided into two types, namely, the classification task of detecting crack types and the regression task of detecting crack depths.

[0008] Furthermore, the camera calibration of the 3D reconstruction module is specifically as follows: S21, the camera calibration is performed by obtaining the intrinsic parameter matrix of the camera and external reference , used for geometric calculation of multi-view crack feature images; The intrinsic parameter matrix represents the imaging parameters of the camera, including focal length and principal point position, in the following form: , in, and Indicates the focal length of the camera in the x and y directions, in pixels; and They respectively represent the position of the camera principal point in the image coordinate system, that is, the intersection of the optical axis and the image plane; The external parameter matrix Describe the rotation and translation of the camera in the real world coordinate system respectively, as follows: , Among them, the rotation matrix R represents the rotation relationship between the camera coordinate system and the real world coordinate system. Translation Matrix Represents the translation relationship between the camera coordinate system and the real world coordinate system; Establish the real-world coordinates of the chessboard through the chessboard calibration method The projection error relationship between the pixel coordinates of the crack feature image , of the following form: , in, and Respectively represent the horizontal and vertical coordinates of the i-th pixel of the crack feature image; is the scale factor used for coordinate normalization; by optimizing Problem, get the best external parameter matrix .

[0009] Furthermore, the three-dimensional reconstruction is to generate a three-dimensional point cloud of the crack area by combining the multi-view crack feature image with the camera calibration internal parameter matrix and external parameter matrix using feature point matching and triangulation methods; the feature point matching refers to finding the corresponding pixel points of the same real-world physical point in the crack feature images of different viewpoints; the specific steps are: S22, use SIFT feature point detection algorithm to extract feature points of multi-view crack feature images, and then establish matching relationships between feature points. The form is as follows: , Where k and l are the view numbers, represents the i-th pixel coordinate of the feature point in the crack feature image at the k-th viewing angle, Represents the i-th pixel coordinate of the feature point in the crack feature image at the l-th viewing angle; After the matching relationship is generated, the matched feature point pairs are triangulated, and the real position of the feature points in the three-dimensional space is calculated by combining the intrinsic and extrinsic parameters of the camera to generate a three-dimensional point cloud. The triangulation formula is as follows, , in, , and Represents the kth camera extrinsic rotation matrix The row vector of Represents the coordinate relationship matrix of feature points of multi-view crack feature images; The three-dimensional point cloud can be defined as the following problem: , Solved by singular value decomposition method , and finally obtain complete three-dimensional point cloud data, which is expressed as follows: , in, represents a single 3D point in a 3D point cloud, and M represents the number of 3D points.

[0010] Furthermore, the depth map generation is to calculate the depth value corresponding to each three-dimensional point from the three-dimensional point cloud data; the specific steps are as follows: 1) Initialize all pixel values ​​of the depth map D(x,y) to infinity , indicating that there is no point coverage in the initial state, , 2) Traverse all 3D points , using the camera's intrinsic parameter matrix Project it to the pixel coordinates on the depth map , the formula is as follows: , Expand to get: , in, For point Corresponding pixel coordinates The depth value of If the coordinates Depth value is less than the depth value of the corresponding pixel coordinate in the depth map, then the depth value of the pixel coordinate is updated to ; 3) If there is a pixel coordinate without a depth value in the depth map, that is, the depth value of the pixel coordinate is , the nearest neighbor interpolation algorithm is used to complete the pixel coordinates without depth values; 4) The final generated size is The depth map D(x,y) is a graph in which each pixel value is the depth value.

[0011] Furthermore, the crack types include longitudinal cracks, transverse cracks, oblique cracks and network cracks.

[0012] Furthermore, the deep learning model is built based on a convolutional neural network, and its structure is as follows: The first part is the input layer, the input is a single-channel depth map , size is ; The second part is a shared convolutional layer, which is used to extract the features of the crack area. Each convolutional layer is connected to a pooling layer. The third part is the branched fully connected layer, which includes the crack type classification branch and the crack depth regression branch; The crack type classification branch flattens the vector output by the pooling layer into a one-dimensional vector through the flattening layer, and outputs it to the fully connected layer for classification prediction of the crack type; the classification prediction uses the softmax classification function for classification, and outputs the probability of the crack type; The crack depth regression branch flattens the vector output by the pooling layer into a one-dimensional vector through a flattening layer, and outputs it to the fully connected layer for regression prediction of the crack depth; the regression prediction uses a linear activation function to output the crack depth; The loss function of the crack type classification branch is a cross entropy loss function.

[0013] The loss function of the crack depth regression branch is a mean square error function.

[0014] Compared with the prior art, the advantages of the present invention are: (1) The present invention combines a data acquisition module with a three-dimensional reconstruction module, adopts multi-view image acquisition and three-dimensional point cloud reconstruction technology, and can achieve high-precision geometric description of the crack area, accurately calculate the crack depth, and provide reliable data support for bridge maintenance.

[0015] (2) Through the classification branch of the deep learning model, the system can efficiently distinguish different types of cracks (such as longitudinal cracks, transverse cracks, oblique cracks and network cracks), realize automatic classification of crack types, and make up for the shortcomings of traditional detection methods in scenes with complex crack morphology.

[0016] (3) The present invention combines the regression branch of the deep learning model to accurately predict the crack depth and issue a danger warning when the depth exceeds the set threshold, promptly notifying bridge maintenance personnel to take targeted repair measures, thereby improving the efficiency and safety of bridge inspection and maintenance work. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a schematic diagram of the system module of the present invention; Figure 2 A schematic diagram of generating a three-dimensional point cloud of a crack according to the present invention; Figure 3 This is a diagram of the deep learning structure for crack detection of the present invention. DETAILED DESCRIPTION

[0019] To achieve the above objectives, the present invention is implemented through the following technical solutions: the present invention provides a multi-type bridge substructure crack depth detection system, combined with Figure 1 The system comprises: M1: Data acquisition module. This module uses a camera and a checkerboard calibration plate to collect multi-view images of the crack area under the bridge and provide raw data for subsequent processing. To ensure the complete acquisition of the crack features under the bridge, the overlap rate between multiple views should be between 60% and 80%. And to ensure the details of the target crack area are clear, the shooting distance needs to be adjusted according to the size of the crack. This module finally generates a set of multi-view crack feature images. The multi-view crack feature image of a single crack is represented as ,in Represents the crack feature image at the kth viewing angle.

[0020] The chessboard calibration plate is The matrix contains W rows, H columns and grids with side length d.

[0021] M2: 3D reconstruction module, which converts the multi-view crack feature images collected by M1 into high-precision 3D point clouds. The 3D point clouds are used to accurately describe the geometric morphology of the cracks and provide data support for subsequent depth calculation and deep learning recognition. The implementation steps of this module include camera calibration, 3D reconstruction and depth map generation, such as Figure 2 shown.

[0022] S21. The purpose of the camera calibration is to obtain the intrinsic parameter matrix of the camera. and external reference , used for geometric calculation of multi-view crack feature images.

[0023] The intrinsic parameter matrix represents the imaging parameters of the camera, including focal length and principal point position, in the following form: , in, and Indicates the focal length of the camera in the x and y directions, in pixels; and They respectively represent the position of the camera principal point in the image coordinate system, that is, the intersection of the optical axis and the image plane.

[0024] The external parameter matrix Describe the rotation and translation of the camera in the real world coordinate system respectively, as follows: , Among them, the rotation matrix R represents the rotation relationship between the camera coordinate system and the real world coordinate system. Translation Matrix Represents the translation relationship between the camera coordinate system and the real world coordinate system.

[0025] Establish the real-world coordinates of the chessboard through the chessboard calibration method The projection error relationship between the pixel coordinates of the crack feature image , of the following form: , in, and Respectively represent the horizontal and vertical coordinates of the i-th pixel of the crack feature image; is the scale factor used for coordinate normalization; by optimizing Problem, get the best external parameter matrix .

[0026] S22. The goal of the three-dimensional reconstruction is to generate a high-precision three-dimensional point cloud of the crack area by using multi-view crack feature images, combined with the internal and external parameters of camera calibration, and using feature point matching and triangulation methods.

[0027] The feature point matching refers to finding corresponding pixel points of the same real-world physical point in crack feature images at different viewing angles.

[0028] The SIFT feature point detection algorithm is used to extract the feature points of the multi-view crack feature image, and then the matching relationship of the feature points is established. The form is as follows: , Where k and l are the view numbers, represents the i-th pixel coordinate of the feature point in the crack feature image at the k-th viewing angle, Represents the i-th pixel coordinate of the feature point in the crack feature image at the l-th viewing angle.

[0029] After the matching relationship is generated, the matched feature point pairs are triangulated, and the real position of the feature points in the three-dimensional space is calculated by combining the intrinsic and extrinsic parameters of the camera to generate a three-dimensional point cloud. The triangulation formula is as follows: , in, , and Represents the kth camera extrinsic rotation matrix The row vector of Represents the feature point coordinate relationship matrix of the multi-view crack feature image.

[0030] The three-dimensional point cloud can be defined as the following problem: , Solved by singular value decomposition method , and finally obtain complete three-dimensional point cloud data, which is expressed as follows: , in, represents a single 3D point in a 3D point cloud, and M represents the number of all points.

[0031] S23, the depth map is generated to calculate the depth value corresponding to each 3D point from the 3D point cloud data. The specific steps are as follows: 1) Initialize all pixel values ​​of the depth map D(x,y) to infinity , indicating that there is no point coverage in the initial state, , 2) Traverse all 3D points , using the camera's intrinsic parameter matrix Project it to the pixel coordinates on the depth map , the formula is as follows: , Expand to get: , in, For point Corresponding pixel coordinates The depth value of If the coordinates Depth value is less than the depth value of the corresponding pixel coordinate in the depth map, then the depth value of the pixel coordinate is updated to .

[0032] 3) If there is a pixel coordinate without a depth value in the depth map, that is, the depth value of the pixel coordinate is , the nearest neighbor interpolation algorithm is used to complete the pixel coordinates without depth values.

[0033] The final size is The depth map D(x,y) is a graph in which each pixel value is the depth value.

[0034] M3, crack detection module, this module uses a preset deep learning model to identify the depth map D(x,y) output by M2. The detection tasks of this module are divided into two types, namely, the classification task of detecting crack types and the regression task of detecting crack depths, such as Figure 3 shown.

[0035] The crack types include longitudinal cracks, transverse cracks, oblique cracks and reticular cracks.

[0036] The deep learning model is built based on a convolutional neural network, and its structure is as follows: The first part is the input layer, the input is a single-channel depth map , size is ; The second part is a shared convolutional layer, which is used to extract the features of the crack area. Each convolutional layer is connected to a pooling layer. The third part is the branched fully connected layer, which includes the crack type classification branch and the crack depth regression branch.

[0037] The crack type classification branch flattens the vector output by the pooling layer into a one-dimensional vector through the flattening layer, and outputs it to the fully connected layer for classification prediction of the crack type; the classification prediction uses the softmax classification function for classification and outputs the type probability of the crack.

[0038] The crack depth regression branch flattens the vector output by the pooling layer into a one-dimensional vector through a flattening layer, and outputs it to the fully connected layer for regression prediction of the crack depth; the classification prediction uses a linear activation function to output the crack depth.

[0039] The loss function of the crack type classification branch is a cross entropy loss function.

[0040] The loss function of the crack depth regression branch is a mean square error function.

[0041] The specific structure of the deep learning model is as follows: First convolutional layer: the input is W×H×1; the convolution kernel size is 3×3, the number is 16, the activation function is ReLU; the output is W / 2×H / 2×16.

[0042] Second pooling layer: maximum pooling is 2×2; output is W / 4×H / 4×16.

[0043] Second convolutional layer: The convolution kernel size is 3×3, the number is 32, the activation function is ReLU; the output is W / 8×H / 8×32.

[0044] Second pooling layer: maximum pooling is 2×2; output is W / 16×H / 16×32.

[0045] The third convolutional layer: the convolution kernel size is 3×3, the number is 64, the activation function is ReLU; the output is W / 32×H / 32×64.

[0046] Flatten layer: Flattens the output of the third convolutional layer into a one-dimensional vector.

[0047] The fully connected layer of the crack type classification branch includes four layers: the first layer has 128 hidden units and the activation function is ReLU; the second layer has 64 hidden units and the activation function is ReLU; the third layer has 32 hidden units and the activation function is ReLU; the fourth layer has 1 hidden unit and the activation function is ReLU.

[0048] The fully connected layer of the crack depth regression branch includes four layers: the first layer has 128 hidden units and the activation function is ReLU; the second layer has 64 hidden units and the activation function is ReLU; the third layer has 32 hidden units and the activation function is ReLU; the fourth layer has 4 hidden units and uses the softmax function for classification.

[0049] If the crack depth exceeds the set threshold, a danger warning will be issued to notify bridge maintenance personnel to perform maintenance based on the crack type.

[0050] The present invention combines a data acquisition module with a three-dimensional reconstruction module, adopts multi-view image acquisition and three-dimensional point cloud reconstruction technology, realizes high-precision geometric description of the crack area, can accurately calculate the crack depth, and provide reliable data support for bridge maintenance.

[0051] Through the classification branches of the deep learning model, the system can efficiently distinguish different types of cracks (such as longitudinal cracks, transverse cracks, oblique cracks and network cracks), realize automatic classification of crack types, and make up for the shortcomings of traditional detection methods in scenes with complex crack morphology.

[0052] The present invention combines the regression branch of the deep learning model to accurately predict the crack depth, and issue a danger warning when the depth exceeds the set threshold, promptly notifying bridge maintenance personnel to take targeted repair measures, thereby improving the efficiency and safety of bridge inspection and maintenance work.

[0053] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-type bridge substructure crack depth detection system, characterized in that: The system includes: M1: Data acquisition module. This module uses a camera and a checkerboard calibration plate to collect multi-view images of the crack area under the bridge to generate a multi-view crack feature image set. The multi-view crack feature image of a single crack is represented as ,in represents the crack feature image at the kth viewing angle; The chessboard calibration plate is The matrix contains W rows, H columns and grids with side length d; M2: 3D reconstruction module, which converts the multi-view crack feature images collected by M1 into 3D point clouds; the 3D point clouds are used to accurately describe the geometric morphology of the cracks. The implementation steps of this module include S21 camera calibration, S22 3D reconstruction and S23 depth map generation; M3, crack detection module, this module uses a preset deep learning model to identify the depth map D(x, y) output by M2. The detection tasks of this module are divided into two types, namely, the classification task of detecting crack types and the regression task of detecting crack depths.

2. A multi-type bridge substructure crack depth detection system according to claim 1, characterized in that: The camera calibration of the 3D reconstruction module is specifically as follows: S21, the camera calibration is performed by obtaining the intrinsic parameter matrix of the camera and external reference , used for geometric calculation of multi-view crack feature images; The intrinsic parameter matrix represents the imaging parameters of the camera, including focal length and principal point position, in the following form: ， in, and Indicates the focal length of the camera in the x and y directions, in pixels; and They respectively represent the position of the camera principal point in the image coordinate system, that is, the intersection of the optical axis and the image plane; The external parameter matrix Describe the rotation and translation of the camera in the real world coordinate system respectively, as follows: ， Among them, the rotation matrix R represents the rotation relationship between the camera coordinate system and the real world coordinate system, and the translation matrix Represents the translation relationship between the camera coordinate system and the real world coordinate system; Establish the real-world coordinates of the chessboard through the chessboard calibration method The projection error relationship between the pixel coordinates of the crack feature image , of the following form: ， in, and Respectively represent the horizontal and vertical coordinates of the i-th pixel of the crack feature image; is the scale factor used for coordinate normalization; by optimizing Problem, get the best external parameter matrix .

3. A multi-type bridge substructure crack depth detection system according to claim 2, characterized in that: The three-dimensional reconstruction is to generate a three-dimensional point cloud of the crack area by combining the multi-view crack feature image with the camera calibration internal parameter matrix and external parameter matrix using feature point matching and triangulation method; the feature point matching refers to finding the corresponding pixel points of the same real-world physical point in the crack feature images of different viewpoints; the specific steps are: S22, use SIFT feature point detection algorithm to extract feature points of multi-view crack feature images, and then establish matching relationships between feature points. The form is as follows: ， Where k and l are the view numbers, represents the i-th pixel coordinate of the feature point in the crack feature image at the k-th viewing angle, Represents the i-th pixel coordinate of the feature point in the crack feature image at the l-th viewing angle; After the matching relationship is generated, the matched feature point pairs are triangulated, and the real position of the feature points in the three-dimensional space is calculated by combining the intrinsic and extrinsic parameters of the camera to generate a three-dimensional point cloud. The triangulation formula is as follows, ， in, , and Represents the kth camera extrinsic rotation matrix The row vector of Represents the coordinate relationship matrix of feature points of multi-view crack feature images; The three-dimensional point cloud can be defined as the following problem: ， Solved by singular value decomposition method , and finally obtain complete three-dimensional point cloud data, which is expressed as follows: ， in, represents a single 3D point in a 3D point cloud, and M represents the number of 3D points.

4. A multi-type bridge substructure crack depth detection system according to claim 3, characterized in that: The depth map generation is to calculate the depth value corresponding to each three-dimensional point from the three-dimensional point cloud data; the specific steps are as follows: 1) Initialize all pixel values ​​of the depth map D(x,y) to infinity , indicating that there is no point coverage in the initial state, ， 2) Traverse all 3D points , using the camera's intrinsic parameter matrix Project it to the pixel coordinates on the depth map , the formula is as follows: ， Expand to get: ， in, For point Corresponding pixel coordinates The depth value of If the coordinates Depth value is less than the depth value of the corresponding pixel coordinate in the depth map, then the depth value of the pixel coordinate is updated to ; 3) If there is a pixel coordinate without a depth value in the depth map, that is, the depth value of the pixel coordinate is , the nearest neighbor interpolation algorithm is used to complete the pixel coordinates without depth values; 4) The final generated size is The depth map D(x,y) is a graph in which each pixel value is the depth value.

5. The multi-type bridge substructure crack depth detection system according to claim 1 is characterized in that: The crack types include longitudinal cracks, transverse cracks, oblique cracks and reticular cracks.

6. A multi-type bridge substructure crack depth detection system according to claim 5, characterized in that: The deep learning model is built based on a convolutional neural network, and its structure is as follows: The first part is the input layer, the input is a single-channel depth map , size is ; The second part is a shared convolutional layer, which is used to extract the features of the crack area. Each convolutional layer is connected to a pooling layer. The third part is the branched fully connected layer, which includes the crack type classification branch and the crack depth regression branch; The crack type classification branch flattens the vector output by the pooling layer into a one-dimensional vector through the flattening layer, and outputs it to the fully connected layer for classification prediction of the crack type; the classification prediction uses the softmax classification function for classification, and outputs the probability of the crack type; The crack depth regression branch flattens the vector output by the pooling layer into a one-dimensional vector through a flattening layer, and outputs it to the fully connected layer for regression prediction of the crack depth; the regression prediction uses a linear activation function to output the crack depth; The loss function of the crack type classification branch is a cross entropy loss function; The loss function of the crack depth regression branch is a mean square error function.

Images