A bridge structure crack position identification method of deep learning and computer vision

Abstract

The application provides a bridge structure crack position identification method bridging deep learning and computer vision, comprising: acquiring multi-view visible light images and depth images, fusing the visible light images and the depth images according to registration parameters to generate a fusion image containing texture and depth information of a bridge; for the fusion image, a self-adaptive threshold segmentation algorithm is used to extract a crack region, a threshold size is dynamically adjusted, and a crack binary image is obtained; according to the crack binary image, a three-dimensional coordinate of a crack pixel point is calculated through a three-dimensional reconstruction algorithm, and three-dimensional point cloud data containing spatial position and morphological information of the crack are reconstructed; based on the three-dimensional point cloud data, a three-dimensional local feature descriptor is used to describe the geometric properties of the crack surface, crack segment parameters are calculated, and a three-dimensional crack model is constructed; based on a deep learning model, a transfer learning strategy is used to form an adaptive and transferable three-dimensional bridge crack detection method, and model parameters are updated to adapt to characteristics of a target bridge.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and in particular to a bridge structure crack position identification method combining deep learning and computer vision. BACKGROUND

[0002] The core technical challenge of bridge crack detection lies in accurately identifying and locating cracks in complex and variable environments. First, due to the presence of stains and paint peeling on the bridge surface, a single visible light image cannot accurately capture crack features. Second, cracks are often small in size and shallow in depth, making it difficult to accurately segment them in complex backgrounds. Third, three-dimensional spatial information of cracks is crucial for assessing their severity, but reconstructing the three-dimensional structure of cracks from two-dimensional images has insufficient accuracy. In addition, the morphology of cracks is diverse, and how to effectively describe and match crack features is also a major difficulty. Finally, the generalization ability of crack detection models is insufficient, making it difficult to adapt to the characteristics of different bridge structures. These problems are interrelated, forming a complex chain of technical difficulties. How to comprehensively utilize multi-source image information and build a self-adaptive and high-precision crack detection and evaluation system is a key problem that needs to be solved. SUMMARY

[0003] The present application provides a bridge structure crack position identification method combining deep learning and computer vision, mainly comprising:

[0004] Obtain multi-view visible light images and depth images, fuse visible light and depth images according to registration parameters to generate a fusion image containing texture and depth information of the bridge;

[0005] For the fusion image, an adaptive threshold segmentation algorithm is used to extract the crack region, and the threshold size is dynamically adjusted to obtain a crack binary image;

[0006] According to the crack binary image, calculate the three-dimensional coordinates of the crack pixel points through a three-dimensional reconstruction algorithm, and reconstruct to obtain three-dimensional point cloud data containing the spatial position and morphological information of the crack;

[0007] Based on the three-dimensional point cloud data, use a three-dimensional local feature descriptor to describe the geometric properties of the crack surface, calculate the crack segment parameters, and construct a three-dimensional crack model;

[0008] According to the spatial position, morphological information and geometric properties of the three-dimensional crack model, select a non-contact excitation scheme, collect images before and after excitation, and extract crack dynamic response features through image differencing;

[0009] Take the crack features as input, re-establish the crack model, match the reconstructed crack model with the three-dimensional models in the crack feature database, form a deep learning model after matching, and realize crack classification and severity evaluation;

[0010] Based on the deep learning model, a transfer learning strategy is adopted to form a self-adaptive transferable bridge crack three-dimensional detection method, and model parameters are updated to adapt to the characteristics of the target bridge.

[0011] The technical scheme provided by the embodiment of the application can include the following beneficial effects:

[0012] The application discloses a bridge structure crack position recognition method based on deep learning and computer vision. Firstly, a registration fusion algorithm is used to generate a fusion image containing rich texture and depth information, and a self-adaptive threshold segmentation algorithm is used to extract a crack region, so that a crack binary image is obtained by dynamically adjusting the threshold size. Then, according to the crack binary image, a three-dimensional reconstruction algorithm is used to calculate three-dimensional coordinates of crack pixel points, and three-dimensional point cloud data containing crack spatial position and shape information are reconstructed. Based on the three-dimensional point cloud data, a three-dimensional local feature descriptor is used to describe the geometric properties of the crack surface, crack segment parameters are calculated, and a three-dimensional crack model is constructed. According to the spatial position, shape information and geometric properties of the three-dimensional crack model, a non-contact excitation scheme is selected, images before and after excitation are collected, and crack dynamic response features are extracted through image difference. The crack features are taken as input, a crack model is re-established, the reconstructed crack model is matched with three-dimensional models in a crack feature database, a deep learning model is formed, and crack classification and degree evaluation are realized. Based on the deep learning model, a transfer learning strategy is adopted to form a self-adaptive transferable bridge crack three-dimensional detection method, and model parameters are updated to adapt to the characteristics of the target bridge. The application realizes accurate detection and evaluation of bridge cracks by combining multi-view image fusion, three-dimensional reconstruction, feature extraction and deep learning. BRIEF DESCRIPTION OF DRAWINGS

[0013] Fig. 1 A flowchart of the bridge structure crack position recognition method based on deep learning and computer vision.

[0014] Fig. 2 A schematic diagram of the bridge structure crack position recognition method based on deep learning and computer vision.

[0015] Fig. 3 Another schematic diagram of the bridge structure crack position recognition method based on deep learning and computer vision. DETAILED DESCRIPTION

[0016] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described in the specification below in conjunction with the drawings in the specification. Obviously, the described embodiments are only some of the embodiments of the specification, not all. Based on the embodiments in the specification, all other embodiments obtained by those of ordinary skill in the art without creative labor should belong to the scope of protection of the specification.

[0017] As Figs. 1-3 The bridge structure crack position identification method of deep learning and computer vision in the embodiment can specifically include:

[0018] S101, obtain multi-view visible light images and depth images, fuse visible light and depth images according to registration parameters, and generate a fusion image containing texture and depth information of the bridge.

[0019] Obtain multi-view visible light images and depth images, extract feature points for each pair of visible light images and depth images using the SIFT algorithm, and calculate the feature point descriptors; according to the feature point descriptors, perform feature point matching using the KNN algorithm, and filter the feature point matching through the RANSAC algorithm to obtain image registration parameters; perform geometric transformation on the depth images according to the image registration parameters, perform multi-scale decomposition on the geometrically transformed images using the discrete wavelet transform to obtain low-frequency subbands and high-frequency subbands; for the low-frequency subbands, perform image fusion using the weighted average method, and for the high-frequency subbands, calculate the energy values of the corresponding regions in each subband, and select the regions with energy values higher than a preset threshold as the fusion results; perform inverse wavelet transform on the fusion results to obtain a fusion image; perform post-processing on the fusion image, calculate the peak signal-to-noise ratio and structural similarity index; if the peak signal-to-noise ratio and structural similarity index do not meet a preset quality threshold, perform multi-scale decomposition again, adjust the fusion weight, and perform image fusion again until the final fusion image meeting the quality requirements is obtained.

[0020] Specifically, the multi-view visible light images and depth images are acquired, the SIFT algorithm is used to extract feature points for each pair of visible light image and depth image, the feature point descriptors are calculated, the KNN algorithm is used for feature point matching, and the RANSAC algorithm is used to screen the matching points to obtain image registration parameters. According to the image registration parameters, the depth image is geometrically transformed, and the registered image is subjected to multi-scale decomposition by using discrete wavelet transform, and the image is decomposed into a low-frequency subband and a high-frequency subband by using a Haar wavelet basis function. For the low-frequency subband obtained by decomposition, a weighted average method is used for image fusion, and for the high-frequency subband, the energy values of the corresponding regions in each subband are calculated, and the regions with larger energy values are selected as the fusion results. The fused frequency band coefficients are subjected to inverse wavelet transform to obtain a fused image containing rich texture and depth information. The fused image is post-processed, a bilateral filter is used to remove noise and retain edge information, and then adaptive histogram equalization is used to enhance the image contrast and improve the visual quality of the fused image. Finally, the enhanced fused image is quality evaluated, the peak signal-to-noise ratio and structural similarity index are calculated, it is judged whether the fused image meets the preset quality threshold, if not, the multi-scale decomposition step is returned, the fusion weight is adjusted, and the image fusion is performed again until the final fused image meeting the quality requirements is obtained. First, a plurality of visible light images and depth images of different views are acquired, for example, 10 groups of image data of different views are collected by using a depth camera such as Kinect. Then, for each pair of visible light image and depth image, the SIFT algorithm is used to extract feature points, and a 128-dimensional descriptor vector of the feature points is calculated. Then, according to the feature point descriptors, the KNN algorithm is used for feature point matching, the K value is set to 2, the nearest neighbor and the second nearest neighbor distance ratio are compared to screen the matching points, and the ratio threshold is set to 8. Then, the RANSAC algorithm is used to further screen the matching points, assuming that the proportion of inliers is 5 and the iteration is 1000 times, to obtain the homography matrix parameters of image registration. According to the matrix, the depth image is subjected to perspective transformation and aligned with the visible light image. The registered image is subjected to 3-layer wavelet decomposition, db4 wavelet basis is selected, and 1 low-frequency subband and 9 high-frequency subbands are obtained. For the low-frequency subband, a weighted average fusion rule is used, and the weight proportion is adaptively determined according to the image clarity. For the high-frequency subband, the energy values in a 3*3 neighborhood of each subband are calculated, the energy threshold is set to 6 times the square sum of the subband coefficients, and the region with larger energy is selected as the fusion coefficient. Finally, the fused low-frequency and high-frequency subbands are subjected to inverse wavelet transform, and a 5*5 Gaussian filter is used for smoothing to obtain the final fused image. To evaluate the fusion quality, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the fused image and the original image are calculated. If the PSNR is lower than 35 dB or the SSIM is lower than 85, the wavelet transform layer number and the weighting coefficient are adjusted, and the fusion is performed again until the quality meets the requirements.

[0021] S102. For the fused image, an adaptive threshold segmentation algorithm is used to extract the crack region, and the threshold size is dynamically adjusted to obtain a binary image of the crack.

[0022] A Gaussian filter is used to remove noise from the fused image, and histogram equalization is used to enhance contrast, resulting in a preprocessed fused image. Based on the preprocessed fused image, the Otsu algorithm is used to calculate an initial global threshold to divide the foreground and background regions. For the divided foreground region, the Niblack adaptive local thresholding algorithm is used, setting a sliding window to calculate the local mean and standard deviation, and dynamically adjusting the local threshold according to the Niblack formula. The gray-level variance within the window is calculated based on the local region's gray-level distribution characteristics. If the variance is greater than a preset threshold, the local threshold is increased by a preset percentage; if the variance is less than a preset threshold, the local threshold is decreased by a preset percentage. Morphological operations are performed on the binarized image: opening operations using rectangular structuring elements are used to remove noise points, closing operations using rectangular structuring elements are used to connect and disconnect crack regions, and connected component analysis is used to remove regions with areas smaller than a preset threshold, resulting in a crack binary image.

[0023] Specifically, the fused image undergoes preprocessing. A 3x3 Gaussian filter is used to remove noise while preserving image edge information. Histogram equalization is then applied to enhance image contrast, resulting in a preprocessed fused image. For the preprocessed fused image, the Otsu algorithm is used to calculate an initial global threshold, dividing the image into foreground and background regions. For the foreground region, the Niblack adaptive local thresholding algorithm is used, setting a 15x15 pixel sliding window. The local mean and standard deviation of pixels within the window are calculated, and the local threshold is dynamically adjusted according to the Niblack formula. Each sub-region is then binarized. Based on the grayscale distribution characteristics of the local image regions, the grayscale variance within each 15x15 pixel window is calculated. When the variance is greater than a preset threshold, the local threshold is increased by 10%; when the variance is less than the preset threshold, the local threshold is decreased by 10%, achieving dynamic threshold adjustment. Morphological operations are performed on the binarized image. A 3x3 rectangular structuring element is used for opening operations to remove small noise points, and a 5x5 rectangular structuring element is used for closing operations to connect broken crack regions. Finally, connected component analysis is performed on the processed image to remove regions with an area smaller than 20 square pixels, resulting in the final binary image of the cracks. During preprocessing of the fused image, a 3x3 Gaussian filter with a standard deviation of 0.5 is used to effectively remove noise while preserving edge information. Histogram equalization is then applied to expand the image grayscale level to the 0-255 range, enhancing contrast. The Otsu algorithm is used to calculate the initial global threshold; assuming a threshold of 127 is obtained. The Niblack adaptive local thresholding algorithm is applied to the foreground region, using a 15x15 pixel sliding window that moves 5 pixels at a time. Within each window, the pixel mean m and standard deviation s are calculated using the formula... The local threshold is calculated, wherein k takes the value of -0.2. At the same time, the gray scale variance in the window is calculated, and the variance threshold is set to 100. When the variance is greater than 100, the local threshold is increased by 10%; when the variance is less than 100, the local threshold is decreased by 10%, to achieve dynamic adjustment. Morphological operations are performed on the binary image, using a 3x3 rectangular structure element for opening operation, with 2 iterations; and then using a 5x5 rectangular structure element for closing operation, with 1 iteration. Finally, connected region analysis is performed, all connected regions are labeled, the number of pixels in each region is calculated, and regions with an area less than 20 square pixels are removed. Through this series of processing, the final crack binary image is obtained, in which the crack region is white (255) and the background is black (0).

[0024] S103, according to the crack binary image, the three-dimensional coordinates of the crack pixel points are calculated by a three-dimensional reconstruction algorithm, and three-dimensional point cloud data containing spatial position and shape information of the cracks are reconstructed.

[0025] The Harris corner point detection algorithm is used to identify the crack edges and corner points, and the feature point two-dimensional coordinates are obtained according to the crack binary image. The camera is calibrated using a calibration board to obtain the camera intrinsic matrix and distortion coefficient, and the camera extrinsic matrix is determined by an external reference. The Semi-Global Block Matching algorithm is used to calculate the feature point disparity, and the feature point two-dimensional coordinates and disparity information are converted into feature point three-dimensional coordinates by combining the camera intrinsic matrix, distortion coefficient and extrinsic matrix. If the feature point three-dimensional coordinates are obtained, dense reconstruction is performed based on the sparse reconstruction result, the points on the skeleton structure are selected as seed points, and the region growing algorithm is used to fill the crack region. The filled crack region is subjected to point cloud registration using the Iterative Closest Point algorithm, and the Moving Least Squares algorithm is used to smooth the registered point cloud to obtain the final crack three-dimensional point cloud model.

[0026] Specifically, crack feature points are extracted from the crack binary image, and the Harris corner detection algorithm is used to identify crack edges and corner points, with a corner response threshold of 0.01 to obtain two-dimensional coordinates of feature points. The camera is calibrated using a calibration board to obtain camera intrinsic matrix and distortion coefficients, and the camera extrinsic matrix is determined by an external reference. The Semi-Global Block Matching algorithm is used to calculate the feature point disparity, and the two-dimensional coordinates and disparity information are converted into three-dimensional coordinates by combining the camera parameters obtained by calibration. The obtained three-dimensional feature points are sparsely reconstructed, and the spatial relationship between the feature points is calculated by the triangulation method using the camera parameters and the corresponding relationship between the feature points to construct an initial three-dimensional skeleton structure. Based on the sparse reconstruction result, dense reconstruction is performed, and the points on the skeleton structure are selected as seed points to fill the crack area using the region growing algorithm, with a depth value similarity threshold of 0.05 mm to obtain complete three-dimensional point cloud data. K-nearest neighbor statistical filtering is used to remove noise points, with a K value of 50 and a standard deviation multiple of 1.0 to obtain a preliminary three-dimensional crack point cloud model. The point cloud is registered using the Iterative Closest Point algorithm, with a maximum iteration count of 100 and a convergence threshold of 0.0001 m to improve the reconstruction accuracy. Finally, the Moving Least Squares algorithm is applied to smooth the point cloud, with a search radius of 0.03 m and a polynomial order of 2 to obtain the final three-dimensional crack point cloud model. The Harris corner detection algorithm is applied to the crack binary image, with a corner response threshold of 0.01 and a window size of 3x3 pixels to obtain about 1000 feature points. Camera calibration is performed using a 9x6 checkerboard calibration board, and 20 calibration images at different angles are collected to calculate the camera intrinsic matrix with a focal length of 1000 pixels and a principal point coordinate of (320, 240), as well as radial distortion coefficients of [-0.1, 0.01, 0.001, -0.0005]. The camera extrinsic matrix is determined using a known-size cube as an external reference. For each feature point, a 11x11 pixel matching window is used, and the Semi-Global Block Matching algorithm is used to calculate the disparity with a disparity search range of 0-64 pixels. The two-dimensional coordinates (u, v) and the disparity d of the feature points are converted into three-dimensional coordinates (X, Y, Z) by combining the camera parameters. In three-dimensional space, the K-D tree data structure is used to quickly find nearby points, and for each feature point, 10 nearest neighbors are selected to fit a local plane by the least squares method to construct an initial three-dimensional skeleton structure. The region growing algorithm is used to fill the crack area by selecting 100 points on the skeleton structure as seed points and setting a depth value similarity threshold of 0.05 mm to obtain a dense point cloud of about 100,000 points. K-nearest neighbor statistical filtering is performed on the point cloud, with a K value of 50 and a standard deviation multiple of 1.0 to filter out about 5% of the noise points.The point cloud registration is performed using an iterative closest point algorithm, the maximum number of iterations is set to 100, and the convergence threshold is 0.0001 m, so that sub-millimeter accuracy is achieved. Finally, the Moving Least Squares algorithm is applied, the search radius is set to 0.03 m, the polynomial order is 2, the point cloud is smoothed, and the final three-dimensional crack point cloud model with a uniform and smooth surface is obtained, containing about 95,000 points.

[0027] In S104, based on the three-dimensional point cloud data, the three-dimensional local feature descriptor is used to describe the geometric properties of the crack surface, the crack segment parameters are calculated, and the three-dimensional crack model is constructed.

[0028] The three-dimensional point cloud data is preprocessed, and an adaptive sampling method based on curvature is used to obtain the sampled point cloud data; according to the sampled point cloud data, a fast point feature histogram algorithm is used to extract local geometric features and obtain feature descriptors of each point; based on the local geometric features, a region growing algorithm is used to segment the point cloud and determine the parameters of each crack segment; for the crack segment, a least squares plane fitting method is used for modeling and calculating the plane equation coefficients; if the fitting residual of the plane equation coefficients exceeds a preset threshold, a quadratic surface fitting is performed on the crack segment; the cross-validation of the modeled three-dimensional crack model is performed to determine whether the root mean square error of the three-dimensional crack model is less than a preset error threshold; if the root mean square error is less than the preset error threshold, the three-dimensional crack model is determined to be effective; according to the effective three-dimensional crack model, the coordinates and angles of the crack intersection points are calculated, the crack connectivity is determined, and a complete three-dimensional crack network model is obtained.

[0029] Specifically, the three-dimensional point cloud data is preprocessed, an adaptive sampling method based on curvature is adopted, the curvature threshold is set to 0.05, the crack edges and corners are retained, and the normal vector and curvature of the sampled point cloud are calculated. The local geometric features are extracted using the fast point feature histogram algorithm, the search radius is set to 2 mm, the feature descriptors of each point are calculated, including normal vector estimation, adjacent point angle difference statistics and spatial distribution of normal vector in spherical coordinate system, and the geometric attribute representation of the crack surface is obtained. Based on the local geometric features, the region growing algorithm is used to segment the point cloud, the point with the maximum curvature is selected as the seed point, the normal vector angle threshold is set to 15 degrees, the curvature difference threshold is set to 0.02, the points with similar features are classified into a segment, and the length, width, depth and direction of each segment are calculated. The least square plane fitting method is used to model each crack segment, the plane equation coefficients are calculated, the fitting residual threshold is set to 0.1 mm, and for the complex shape cracks with residual error exceeding the threshold, quadratic surface fitting is used. The three-dimensional crack model is constructed by combining the segment parameters, and all segment models are combined to form a complete crack network structure. The cross-validation of the constructed three-dimensional crack model is performed, 20% of the points are randomly selected for verification, the root mean square error of the model prediction value and the actual value is calculated, and the error threshold is set to 0.5 mm. Finally, the crack network topology analysis is performed, the crack intersection point coordinates and angle are calculated, the crack connectivity is judged, and the complete three-dimensional crack network model is obtained. The three-dimensional point cloud data containing 100 million points is processed, first, the adaptive sampling method based on curvature is adopted, the curvature threshold is set to 0.05, the number of point clouds is reduced to 50 million, and the crack edge and corner features are retained. For the sampled point cloud, the KNN algorithm is used to calculate the 20 nearest neighbors of each point, the normal vector and curvature are estimated. The fast point feature histogram algorithm is applied, the search radius is set to 2 mm, the spherical space is divided into 11x11x11 grids, the normal vector angle and point pair distance are counted, and a 352-dimensional feature descriptor is generated. Based on the feature descriptor, the region growing algorithm is executed, 1000 points with the maximum curvature are selected as the initial seed points, the normal vector angle threshold is set to 15 degrees, the curvature difference threshold is set to 0.02, and finally about 5000 crack segments are obtained. For each crack segment, the principal component analysis method is used to calculate the principal direction, the length, width and depth are obtained, the typical crack segment length is 50-200 mm, the width is 0.5-5 mm, and the depth is 1-10 mm. The least square method is used for plane fitting, the coefficients of the plane equation Ax+By+Cz+D=0 are calculated, the fitting residual threshold is set to 0.1 mm, and for about 10% of the complex shape cracks, quadratic surface fitting is used. All crack segment models are combined to form a complete crack network structure. 100,000 points are randomly selected for cross-validation, the root mean square error is 0.3 mm, which meets the error threshold requirement of 0.5 mm.Finally, topology analysis is performed to identify 500 fracture intersection points, and the intersection angle distribution is calculated to be between 30-90 degrees. It is determined that 85% of the fractures have connectivity, and a three-dimensional fracture network model containing comprehensive geometric and topological information is finally generated.

[0030] S105, according to the spatial position, shape information and geometric attribute of the three-dimensional fracture model, a non-contact excitation scheme is selected, images before and after excitation are collected, and a dynamic response feature of the fracture is extracted through image difference.

[0031] According to the spatial position, shape information and geometric attribute of the three-dimensional fracture model, laser excitation is selected as a non-contact excitation scheme, and laser power and pulse duration are set according to the fracture depth; the laser power and pulse duration are obtained, a high-resolution camera is used to collect a reference image before excitation; laser excitation is performed according to the laser power and pulse duration, and a dynamic image sequence during excitation is collected by a high-speed camera; the reference image before excitation and the image sequence after excitation are processed by image difference, a cross-correlation algorithm at a sub-pixel level is used to calculate a displacement field, and a dynamic response feature of the fracture is obtained; wavelet transform is performed on the dynamic response feature of the fracture to extract time-frequency features, main frequency components and energy distribution are calculated; if the main frequency components and energy distribution meet preset conditions, the dynamic response feature of the fracture is associated with the three-dimensional fracture model, and a mapping relationship between the geometric parameters of the fracture and the dynamic response feature is established.

[0032] Specifically, according to the spatial position, shape information and geometric properties of the three-dimensional crack model, laser excitation is selected as the non-contact excitation scheme. For cracks with a depth less than 5 mm, the laser power is set to 5-10 W, and the pulse duration is 10-20 ms. For cracks with a depth of 5-10 mm, the laser power is set to 10-15 W, and the pulse duration is 20-30 ms. For cracks with a depth greater than 10 mm, the laser power is set to 15-20 W, and the pulse duration is 30-50 ms. A high-resolution reference image is collected before excitation, and the camera parameters are set as follows: exposure time 1 / 1000 s, aperture value f / 8, ISO sensitivity 400, focal length 50 mm, and shooting distance 0.5 m. The laser excitation is performed, and a dynamic image sequence during the excitation process is collected simultaneously. A high-speed camera is used, with a frame rate of 1000 fps and a resolution of 1280x1024 pixels. Precise synchronization of excitation and image acquisition is achieved through an external trigger signal, and the image changes during the excitation duration are recorded. The reference image before excitation and the image sequence after excitation are subjected to image difference processing, and a cross-correlation algorithm at the sub-pixel level is used to calculate the displacement field. The correlation window size is set to 32x32 pixels, and the step size is 8 pixels. The bicubic interpolation method is used to improve the accuracy, and Gaussian filtering is used to reduce noise effects, obtaining the crack dynamic response characteristics. The obtained dynamic response characteristics are subjected to wavelet transform, and the time-frequency characteristics are extracted. The main frequency components and energy distribution are calculated. The dynamic response characteristics are associated with the three-dimensional crack model, and the mapping relationship between the crack geometric parameters and the dynamic response characteristics is established. The state and development trend of the crack are comprehensively evaluated. For a typical concrete structure crack, the three-dimensional crack model shows that its depth is 7.5 mm, width is 0.8 mm, and length is 150 mm. According to these parameters, the laser excitation power is selected as 12 W, and the pulse duration is 25 ms. A high-resolution camera is used to collect the reference image, and the parameters are set as follows: exposure time 1 / 1000 s, aperture value f / 8, ISO sensitivity 400, focal length 50 mm, shooting distance 0.5 m, and reference image size 4000x3000 pixels. During laser excitation, a high-speed camera is used to synchronously collect a dynamic image sequence, with a frame rate of 1000 fps and a resolution of 1280x1024 pixels. Precise synchronization is achieved through an external trigger, and a total of 25 images are collected to cover the entire excitation process. The collected image sequence is preprocessed, including denoising and distortion correction, and then subjected to difference processing with the reference image. A sub-pixel level cross-correlation algorithm is used to calculate the displacement field, with a correlation window size of 32x32 pixels and a step size of 8 pixels. The bicubic interpolation method is used to improve the accuracy, and a 5x5 Gaussian kernel is used for smoothing filtering, obtaining a sub-micron level displacement field. Wavelet transform is performed on the displacement field data, using a Morlet wavelet with a scale range of 1-64, obtaining the time-frequency characteristics. The analysis results show that the main response frequency of the crack region is 85 Hz, and the energy is concentrated in the range of 20-150 Hz.The dynamic response characteristics are associated with the three-dimensional fracture model to establish a mapping relationship, and it is found that the fracture depth is negatively correlated with the response frequency, with a correlation coefficient of -0.85, and the fracture width is positively correlated with the response amplitude, with a correlation coefficient of 0.72. Based on these correlations, it is evaluated that the fracture is in a stable state, but needs to be continuously monitored.

[0033] S106, the fracture characteristics are taken as input to re-establish the fracture model, and the reconstructed fracture model is matched with the three-dimensional model in the fracture feature database to form a deep learning model, realizing the classification and degree evaluation of the fracture.

[0034] The input data containing fracture feature information is received, and the fracture feature information includes spatial position and geometric attribute. According to the fracture feature information, data preprocessing is carried out, the feature selection method based on variance is adopted to screen out the highest contribution degree feature, and the principal component analysis method is used to reduce the high-dimensional feature to the preset dimension. If the fracture feature width after preprocessing is greater than the preset threshold, the first preset voxel size is adopted; if the fracture feature width after preprocessing is less than the preset threshold, the second preset voxel size is adopted, and the voxel grid containing spatial position and geometric attribute is generated. The three-dimensional model in the fracture feature database is obtained, and the iterative closest point algorithm is used to calculate the similarity between the voxel grid and the three-dimensional model. It is judged whether the similarity is greater than the preset matching threshold, if it is greater than the preset matching threshold, the preset number of models with the highest similarity are selected as initial training samples. The three-dimensional convolutional neural network is constructed by using the initial training samples, and the three-dimensional convolutional neural network includes a preset number of three-dimensional convolutional layers and fully connected layers. The gradient weighted class activation mapping technology is used for explainability analysis of the three-dimensional convolutional neural network, and the fracture feature area concerned by the model is obtained.

[0035] Specifically, data preprocessing is performed on the input crack features, including feature normalization and dimension reduction. The feature selection method based on variance is used to select the features with the highest contribution. Then, the principal component analysis method is used to reduce the high-dimensional features to 64 dimensions, retaining 95% of the information. Based on the preprocessed crack features, the adaptive voxelization method is used to reconstruct the three-dimensional crack model. For cracks with a width greater than 1 mm, the voxel size is set to 0.1 mm. For cracks with a width less than 1 mm, the voxel size is set to 0.01 mm. A voxel grid containing spatial location and geometric attributes is generated. The reconstructed crack model is matched with the three-dimensional models in the crack feature database. The iterative closest point algorithm is used to calculate the similarity between the models. The matching threshold is set to 0.85. The top 50 most similar models are selected as the initial training samples. The samples are expanded to 500 by rotation and scaling. The expanded training samples are used to construct a three-dimensional convolutional neural network. The input layer is set to a three-dimensional voxel grid of 64x64x64. It contains four three-dimensional convolutional layers (convolution kernel size of 3x3x3, step size of 1, using 32, 64, 128, and 256 convolution kernels, respectively) and three fully connected layers (neuron numbers of 1024, 512, and 256, respectively). Each layer is followed by a batch normalization and a ReLU activation function. The last layer uses a Softmax function to output the crack type and severity assessment results. The model is trained using a cross-entropy loss function. The model performance is evaluated using 5-fold cross-validation. The average accuracy and F1 score are calculated. The trained model is analyzed for interpretability. The gradient-weighted class activation mapping technique is used to visualize the crack feature areas that the model focuses on. The features that have the most impact on the classification and assessment results are identified. In practical applications, for a typical concrete crack detection task, 1000 features are collected from the original crack data. Through variance-based feature selection, 200 features with a variance contribution rate of more than 0.1% are retained. Subsequently, principal component analysis is used to reduce them to 64 dimensions, with a cumulative explained variance ratio of 96.5%. For a crack with a length of 150 mm, a maximum width of 2.5 mm, and a depth of 15 mm, the adaptive voxelization method is used to reconstruct a three-dimensional model. The main part of the crack uses 0.1 mm voxels, while the small branches use 0.01 mm voxels. A voxel grid of 62x48x150 is generated. This model is matched with 10000 pre-stored models in the database. The iterative closest point algorithm is used to calculate the similarity. 53 models with a similarity greater than 0.87 are selected as the initial samples. The samples are expanded to 530 through ±15° rotation and ±10% scaling. The three-dimensional convolutional neural network is trained on an NVIDIA Tesla V100 GPU. The batch size is set to 32. The Adam optimizer is used with an initial learning rate of 0.001. After 100 epochs, the validation set accuracy reaches 93.7%, and the F1 score is 0.915.Using gradient-weighted class activation map analysis, it is found that the model mainly focuses on the intersection of cracks, edge contours and areas of depth changes, which are key features for crack type judgment and degree assessment. Finally, the model judges the input crack as a stress crack, the risk degree is evaluated as medium (level 3, a total of 5 levels), and the confidence is 0.89.

[0036] S107, based on the deep learning model, a self-adaptive transferable bridge crack three-dimensional detection method is formed by using a transfer learning strategy, and model parameters are updated to adapt to the characteristics of the target bridge.

[0037] The feature extraction layer and the classification layer are obtained from the source domain deep learning model, and a progressive unfreezing strategy is used according to the feature extraction layer and the classification layer. All layers except the last two fully connected layers are initially frozen. Obtain the target bridge crack data, including crack images and three-dimensional point cloud data. The crack images are preprocessed and enhanced, and the three-dimensional point cloud data is voxelized to obtain a three-dimensional grid. The model is fine-tuned using target domain data, and a fine-tuned model is obtained through fine-tuning. According to the fine-tuned model, an adversarial domain adaptation network is introduced, a gradient reversal layer and a domain classifier are added after the feature extraction layer, and the difference between the source domain and the target domain feature distribution is reduced through adversarial training. An active learning strategy is used to select samples with the highest prediction uncertainty from the target domain data for manual labeling, and the manually labeled samples are added to the training set. The fine-tuned model is updated according to the training set to obtain an updated fine-tuned model. Model ensemble technology is used to combine multiple different random seed training of the updated fine-tuned model, and the final prediction result is obtained by voting.

[0038] Specifically, the feature extraction layer and classification layer are extracted from the source domain deep learning model, and a progressive unfreezing strategy is adopted. All layers except the last two fully connected layers are initially frozen, and then one layer is unfrozen every 10 epochs until the fifth-to-last layer is unfrozen. Crack data of the target bridge is collected, including crack images and three-dimensional point cloud data. The image data is preprocessed and enhanced, and the three-dimensional point cloud data is voxelized with a voxel size of 0.01 m to generate a 64x64x64 three-dimensional grid, ensuring consistency with the input format of the source domain model. The model is fine-tuned using target domain data with a small learning rate of 0.0001, a batch size of 16, and 50 training rounds. The early stopping strategy is used, and training is stopped when the validation set loss does not decrease for 5 consecutive epochs. An adversarial domain adaptation network is introduced, with a gradient reversal layer and a domain classifier added after the feature extraction layer to reduce the difference in feature distribution between the source and target domains. An active learning strategy is used to select the 20% samples with the highest prediction uncertainty for manual annotation, and these high-information-content samples are added to the training set to update the model. Finally, model ensemble techniques are used to combine five fine-tuned models trained with different random seeds to obtain the final prediction result, forming an adaptive and transferable bridge crack three-dimensional detection method. In the crack detection task of a sea-crossing bridge, transfer learning is performed based on a pre-trained deep learning model. First, 12 convolutional layers and 3 fully connected layers are extracted from the source model, and the first 10 convolutional layers are initially frozen. 1000 bridge crack images and 500 point cloud datasets are collected, and the images are processed by cropping, rotating, and contrast enhancement, and the point cloud data is voxelized into a 64x64x64 three-dimensional grid. In the fine-tuning training, the learning rate is set to 0.0001, one layer is unfrozen every 10 epochs, and a total of 50 rounds are trained. The validation set loss does not decrease for 5 consecutive rounds, triggering the early stopping mechanism at the 37th round. An adversarial domain adaptation network is introduced, with a gradient reversal layer and a 2-layer fully connected domain classifier added after the 10th convolution, and the adversarial loss weight is 0.3. The uncertainty is calculated by the prediction entropy, and the top 20% (300 samples) are selected for manual annotation and added to the training set, resulting in a 3.5% improvement in model accuracy. Finally, five models are integrated, three of which use different random seeds and two use different learning rates (0.0005 and 0.00005), and the final prediction is determined by majority voting. On the test set, the single model F1 score is 0.86, and after integration it improves to 0.91, successfully adapting to the special structure and environmental factors of the target bridge.

[0039] The above only describes the preferred embodiments of the present application, and it should be noted that those skilled in the art can make several improvements and supplements without departing from the principles of the present application, and these improvements and supplements should also be considered within the scope of protection of the present application.

Claims

1. A method for identifying the location of cracks in bridge structures using deep learning and computer vision, characterized in that, The method comprises: acquiring a multi-view visible light image and a depth image, fusing the visible light image and the depth image according to registration parameters to generate a fused image containing texture and depth information of the bridge; for the fused image, a self-adaptive threshold segmentation algorithm is used to extract a crack region, a threshold size is dynamically adjusted, and a crack binary image is obtained; according to the crack binary image, a three-dimensional coordinate of a crack pixel point is calculated through a three-dimensional reconstruction algorithm, and three-dimensional point cloud data containing spatial position and morphological information of the crack are reconstructed; based on the three-dimensional point cloud data, a three-dimensional local feature descriptor is used to describe the geometric properties of the crack surface, crack segment parameters are calculated, and a three-dimensional crack model is constructed, including: the three-dimensional point cloud data is preprocessed, an adaptive sampling method based on curvature is used to obtain sampled point cloud data; according to the sampled point cloud data, a fast point feature histogram algorithm is used to extract local geometric features, and a feature descriptor of each point is obtained; based on the local geometric features, a region growing algorithm is used to segment the point cloud, and parameters of each crack segment are determined; for the crack segment, a least square plane fitting method is used for modeling, and a plane equation coefficient is calculated; if a fitting residual of the plane equation coefficient exceeds a preset threshold, a quadratic surface fitting is performed on the crack segment; the three-dimensional crack model after modeling is cross-validated, and whether a root mean square error of the three-dimensional crack model is less than a preset error threshold is judged; if the root mean square error is less than the preset error threshold, the three-dimensional crack model is determined to be effective; according to the effective three-dimensional crack model, a crack intersection point coordinate and an included angle are calculated, crack connectivity is judged, and a complete three-dimensional crack network model is obtained; according to the spatial position, morphological information and geometric properties of the three-dimensional crack model, a non-contact excitation scheme is selected, pre-excitation and post-excitation images are collected, and crack dynamic response features are extracted through image differencing, including: according to the spatial position, morphological information and geometric properties of the three-dimensional crack model, laser excitation is selected as a non-contact excitation scheme, and laser power and pulse duration are set according to crack depth; laser power and pulse duration are obtained, a high-resolution camera is used to collect a pre-excitation reference image; laser excitation is performed according to laser power and pulse duration, and a dynamic image sequence in the excitation process is collected through a high-speed camera; the pre-excitation reference image and the post-excitation image sequence are subjected to image differencing processing, a cross-correlation algorithm at a sub-pixel level is used to calculate a displacement field, and crack dynamic response features are obtained; the crack dynamic response features are subjected to wavelet transform, time-frequency features are extracted, main frequency components and energy distributions are calculated; if the main frequency components and the energy distributions meet preset conditions, the crack dynamic response features are associated with the three-dimensional crack model, a mapping relationship between crack geometric parameters and dynamic response features is established; crack features are taken as input, a crack model is re-established, the re-established crack model is matched with three-dimensional models in a crack feature database, a good match forms a deep learning model, and crack classification and degree evaluation are realized.Based on deep learning model, the transfer learning strategy is adopted to form a self-adaptive transferable bridge crack three-dimensional detection method, and the model parameters are updated to adapt to the characteristics of the target bridge.

2. The method of claim 1, wherein, The method comprises the following steps: acquiring multi-view visible light images and depth images, extracting feature points by using a SIFT algorithm for each pair of visible light image and depth image, and calculating feature point descriptors; performing feature point matching by using a KNN algorithm according to the feature point descriptors, and screening the feature point matching by using a RANSAC algorithm to obtain image registration parameters; performing geometric transformation on the depth image according to the image registration parameters, performing multi-scale decomposition on the image after the geometric transformation by using a discrete wavelet transform to obtain low-frequency subbands and high-frequency subbands; performing image fusion on the low-frequency subbands by using a weighted average method, and calculating energy values of corresponding regions in each subband for the high-frequency subbands, and selecting regions with energy values higher than a preset threshold value as fusion results; and performing inverse wavelet transform on the fusion results to obtain a fusion image. The fusion image is post-processed to calculate a peak signal-to-noise ratio and a structural similarity index; if the peak signal-to-noise ratio and the structural similarity index do not satisfy a preset quality threshold value, then multi-scale decomposition is performed again, the fusion weight is adjusted, and image fusion is performed again until a final fusion image that satisfies the quality requirement is obtained.

3. The method of claim 1, wherein, The method comprises the following steps: removing noise from the fusion image by using a Gaussian filter, and enhancing contrast by using histogram equalization to obtain a preprocessed fusion image; calculating an initial global threshold value by using an Otsu algorithm according to the preprocessed fusion image, and dividing foreground and background regions; The method comprises the following steps: for the divided foreground region, a Niblack adaptive local threshold algorithm is used, a sliding window is set to calculate a local mean value and a standard deviation, and a local threshold value is dynamically adjusted according to a Niblack formula; the gray variance in the window is calculated according to the gray distribution characteristics of the local region, if the variance is greater than a preset threshold value, the local threshold value is increased by a preset proportion, and if the variance is less than the preset threshold value, the local threshold value is decreased by a preset proportion; morphological operations are performed on the binary processed image, a rectangular structure element is used to perform an opening operation to remove noise points, a rectangular structure element is used to perform a closing operation to connect disconnected crack regions, regions with an area less than a preset threshold value are removed through connected region analysis, and a crack binary image is obtained.

4. The method of claim 1, wherein, The method comprises the following steps: a Harris corner point detection algorithm is used to identify crack edges and corner points, and two-dimensional coordinates of feature points are obtained according to the crack binary image; a camera is calibrated by using a calibration board to obtain an intrinsic matrix and distortion coefficients of the camera, and an extrinsic matrix of the camera is determined through an external reference; The Semi-Global Block Matching algorithm is used to calculate the parallax of the feature points, and the two-dimensional coordinates and parallax information of the feature points are converted into three-dimensional coordinates of the feature points by combining the camera intrinsic matrix, distortion coefficients and the extrinsic matrix; if the three-dimensional coordinates of the feature points are obtained, dense reconstruction is performed based on the sparse reconstruction result, points on the skeleton structure are selected as seed points, and a region growing algorithm is used to fill the crack area; the point cloud registration is performed on the filled crack area by using the Iterative Closest Point algorithm, the Moving Least Squares algorithm is used to smooth the registered point cloud, and finally the three-dimensional point cloud model of the crack is obtained.

5. The method of claim 1, wherein, The crack features are taken as inputs, the crack model is re-established, the reconstructed crack model is matched with the three-dimensional model in the crack feature database, a deep learning model is formed after the matching, and the classification and degree evaluation of the crack are realized, including: receiving input data containing crack feature information, the crack feature information including spatial position and geometric attribute; performing data preprocessing according to the crack feature information, screening out the highest contribution features by using a feature selection method based on variance, and reducing high-dimensional features to a preset dimension by using a principal component analysis method; If the width of the preprocessed crack feature is greater than a preset threshold, a first preset voxel size is used; if the width of the preprocessed crack feature is less than the preset threshold, a second preset voxel size is used, and a voxel grid containing spatial position and geometric attribute is generated; a three-dimensional model in the crack feature database is obtained, the similarity between the voxel grid and the three-dimensional model is calculated by using an iterative closest point algorithm; it is judged whether the similarity is greater than a preset matching threshold, if the similarity is greater than the preset matching threshold, a preset number of models with the highest similarity are selected as initial training samples; a three-dimensional convolutional neural network is constructed by using the initial training samples, the three-dimensional convolutional neural network includes a preset number of three-dimensional convolutional layers and fully connected layers; the three-dimensional convolutional neural network is subjected to explainability analysis by using a gradient-weighted class activation mapping technology, and a crack feature area focused on by the model is obtained.

6. The method of claim 1, wherein, Based on the deep learning model, a self-adaptive transferable bridge crack three-dimensional detection method is formed by using a transfer learning strategy, model parameters are updated to adapt to the characteristics of the target bridge, including: feature extraction layers and classification layers are obtained from a source domain deep learning model, an incremental unfreezing strategy is used according to the feature extraction layers and the classification layers, and all layers except the last two fully connected layers are initially frozen; crack data of a target bridge are obtained, including crack images and three-dimensional point cloud data; the crack images are preprocessed and enhanced, and the three-dimensional point cloud data are subjected to voxelization to obtain a three-dimensional grid; the model is fine-tuned by using the target domain data, a fine-tuned model is obtained through fine-tuning, an adversarial domain adaptation network is introduced according to the fine-tuned model, a gradient reversal layer and a domain classifier are added after the feature extraction layer, and the feature distribution difference between the source domain and the target domain is reduced through adversarial training. Adopting an active learning strategy, samples with the highest prediction uncertainty are selected from the target domain data for manual labeling, and the manually labeled samples are added to the training set; the fine-tuned model is updated according to the training set to obtain an updated fine-tuned model; model ensemble technology is adopted to train the updated fine-tuned model in combination with multiple different random seeds, and a final prediction result is obtained by voting.

Images