Bridge structure crack position identification method based on deep learning and computer vision

Abstract

The invention provides a bridge structure crack position identification method based on deep learning and computer vision, and the method comprises the steps: obtaining a multi-view visible light image and a depth image, fusing the visible light image and the depth image according to registration parameters, and generating a fusion image containing the texture and depth information of a bridge; for the fused image, extracting a crack region by adopting an adaptive threshold segmentation algorithm, and dynamically adjusting the threshold to obtain a crack binary image; according to the crack binary image, calculating three-dimensional coordinates of crack pixel points through a three-dimensional reconstruction algorithm, and performing reconstruction to obtain three-dimensional point cloud data containing crack space position and form information; on the basis of the three-dimensional point cloud data, three-dimensional local feature descriptors are adopted to describe geometric attributes of the surface of the crack, crack fragment parameters are calculated, and a three-dimensional crack model is constructed; based on a deep learning model, a transfer learning strategy is adopted, a self-adaptive transferable bridge crack three-dimensional detection method is formed, and model parameters are updated so that the model parameters can adapt to the characteristics of a target bridge.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method for identifying crack positions in bridge structures using deep learning and computer vision. Background Art

[0002] The core technical challenge facing bridge crack detection is how to accurately identify and locate cracks in a complex and changing environment. First, due to the interference factors of stains and paint peeling on the bridge surface, it is difficult for a single visible light image to accurately capture the crack characteristics. Secondly, cracks are often small in size and shallow in depth, making it difficult to accurately segment them against a complex background. Furthermore, the three-dimensional spatial information of the cracks is crucial for assessing their severity, but there is a problem of insufficient accuracy in reconstructing the three-dimensional structure of the cracks from two-dimensional images. In addition, the morphology of cracks is diverse, and how to effectively describe and match crack characteristics is also a major difficulty. Finally, the generalization ability of the crack detection model is insufficient, making it difficult to adapt to the characteristics of different bridge structures. These problems are interrelated, forming a complex chain of technical problems. How to comprehensively utilize multi-source image information to build an adaptive, high-precision crack detection and assessment system is a key issue that needs to be solved urgently. Summary of the invention

[0003] The present invention provides a method for identifying crack positions in bridge structures based on deep learning and computer vision, which mainly includes:

[0004] Acquire multi-view visible light images and depth images, fuse the visible light images and the depth images according to the registration parameters, and generate a fused image containing the texture and depth information of the bridge;

[0005] For the fused image, an adaptive threshold segmentation algorithm is used to extract the crack area, and the threshold value is dynamically adjusted to obtain a crack binary image.

[0006] According to the crack binary image, the three-dimensional coordinates of the crack pixel points are calculated by a three-dimensional reconstruction algorithm, and three-dimensional point cloud data containing the crack spatial position and morphological information is reconstructed;

[0007] Based on the 3D point cloud data, the 3D local feature descriptor is used to characterize the geometric properties of the crack surface, calculate the crack segment parameters, and construct a 3D crack model.

[0008] According to the spatial position, morphological information and geometric properties of the 3D crack model, a non-contact excitation scheme is selected, images before and after excitation are collected, and the dynamic response characteristics of the crack are extracted by image difference.

[0009] Taking the crack features as input, re-establish the crack model, match the reconstructed crack model with the three-dimensional model in the crack feature database, and form a deep learning model to achieve crack classification and degree assessment;

[0010] Based on the deep learning model, a transfer learning strategy is adopted to form an adaptive and transferable three-dimensional detection method for bridge cracks, and the model parameters are updated to adapt to the characteristics of the target bridge.

[0011] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0012] The invention discloses a method for identifying crack positions of bridge structures by deep learning and computer vision. First, a fusion image containing rich texture and depth information is generated by a registration fusion algorithm, and an adaptive threshold segmentation algorithm is used to extract the crack area, and the threshold value is dynamically adjusted to obtain a crack binary image. Then, according to the crack binary image, the three-dimensional coordinates of the crack pixel points are calculated by a three-dimensional reconstruction algorithm, and three-dimensional point cloud data containing the spatial position and morphological information of the crack is reconstructed. Based on the three-dimensional point cloud data, a three-dimensional local feature descriptor is used to characterize the geometric properties of the crack surface, the crack fragment parameters are calculated, and a three-dimensional crack model is constructed. According to the spatial position, morphological information, and geometric properties of the three-dimensional crack model, a non-contact excitation scheme is selected, and images before and after the excitation are collected, and the dynamic response characteristics of the crack are extracted by image difference. The crack features are used as input to re-establish the crack model, and the reconstructed crack model is matched with the three-dimensional model in the crack feature database to form a deep learning model to realize the classification and degree evaluation of the crack. Based on the deep learning model, a transfer learning strategy is adopted to form an adaptive and transferable three-dimensional detection method for bridge cracks, and the model parameters are updated to adapt to the characteristics of the target bridge. The present invention realizes accurate detection and assessment of bridge cracks by combining multi-perspective image fusion, three-dimensional reconstruction, feature extraction and deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The present invention is a flow chart of a method for identifying crack positions in bridge structures using deep learning and computer vision.

[0014] Figure 2 A schematic diagram of a method for identifying crack positions in bridge structures using deep learning and computer vision according to the present invention.

[0015] Figure 3 This is another schematic diagram of a method for identifying crack positions in bridge structures using deep learning and computer vision according to the present invention. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0017] like Figure 1-3 In this embodiment, a method for identifying crack positions in bridge structures using deep learning and computer vision may specifically include:

[0018] S101, acquiring a multi-view visible light image and a depth image, fusing the visible light image and the depth image according to a registration parameter, and generating a fused image including texture and depth information of a bridge.

[0019] Obtain multi-view visible light images and depth images, use SI FT algorithm to extract feature points for each pair of visible light images and depth images, and calculate the feature point descriptors; use KNN algorithm to match feature points according to the feature point descriptors, and use RANSAC algorithm to filter the feature point matches to obtain image registration parameters; perform geometric transformation on the depth image according to the image registration parameters, and use discrete wavelet transform to perform multi-scale decomposition on the geometrically transformed image to obtain low-frequency sub-bands and high-frequency sub-bands; for the low-frequency sub-bands, use weighted average method to perform image fusion, for the high-frequency sub-bands, calculate the energy value of the corresponding area in each sub-band, and select the area with energy value higher than the preset threshold as the fusion result; perform inverse wavelet transform on the fusion result to obtain a fused image; post-process the fused image to calculate the peak signal-to-noise ratio and the structural similarity index; if the peak signal-to-noise ratio and the structural similarity index do not meet the preset quality threshold, re-perform multi-scale decomposition, adjust the fusion weights, and re-perform image fusion until a final fused image that meets the quality requirements is obtained.

[0020] Specifically, obtain multi-view visible light images and depth images, use SI FT algorithm to extract feature points for each pair of visible light images and depth images, calculate feature point descriptors, use KNN algorithm to match feature points, and then use RANSAC algorithm to screen matching point pairs to obtain image registration parameters. Perform geometric transformation on the depth image according to the image registration parameters, perform multi-scale decomposition on the registered image using discrete wavelet transform, and use Haar wavelet basis function to decompose the image into low-frequency sub-bands and high-frequency sub-bands. For the low-frequency sub-bands obtained by decomposition, use weighted average method to perform image fusion. For the high-frequency sub-bands, calculate the energy value of the corresponding area in each sub-band, select the area with larger energy value as the fusion result, perform inverse wavelet transform on the fused frequency band coefficients, and obtain a fused image containing rich texture and depth information. Post-process the fused image, use bilateral filter to remove noise, retain edge information, and then enhance image contrast through adaptive histogram equalization to improve the visual quality of the fused image. Finally, the enhanced fused image is evaluated for quality, and the peak signal-to-noise ratio and structural similarity index are calculated to determine whether the fused image meets the preset quality threshold. If not, return to the multi-scale decomposition step, adjust the fusion weights, and re-fuse the images until the final fused image that meets the quality requirements is obtained. First, obtain visible light images and depth images from multiple perspectives. For example, use a depth camera such as Kinect to collect 10 sets of image data from different perspectives. Then, for each pair of visible light images and depth images, use the SIFT algorithm to extract feature points and calculate the 128-dimensional descriptor vector of the feature points. Then, based on the feature point descriptors, the KNN algorithm is used to match the feature points. The K value is set to 2, and the matching points are screened by comparing the nearest neighbor and next nearest neighbor distance ratio. The ratio threshold is set to 8. The matching points are further screened using the RANSAC algorithm. Assuming that the internal point ratio is 5, it is iterated 1000 times to obtain the homography matrix parameters for image registration. According to the matrix, the depth image is perspective transformed and aligned with the visible light image. The registered image is decomposed by 3 layers of wavelet, and the db4 wavelet basis is selected to obtain 1 low-frequency subband and 9 high-frequency subbands. For the low-frequency subbands, the weighted average fusion rule is adopted, and the weight ratio is adaptively determined according to the image clarity. For the high-frequency subbands, the energy value in the 3×3 neighborhood of each subband is calculated, and the energy threshold is set to 6 times the sum of the squares of the subband coefficients. The area with larger energy is selected as the fusion coefficient. Finally, the fused high and low frequency subbands are inversely transformed by wavelet, and smoothed with a 5×5 Gaussian filter to obtain the final fused image. In order to evaluate the fusion quality, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the fused image and the original image are calculated. If the PSNR is lower than 35dB or the SSIM is lower than 85, return to adjust the number of wavelet transform layers and weighting coefficients, and re-fuse until the quality meets the requirements.

[0021] S102, for the fused image, using an adaptive threshold segmentation algorithm to extract the crack area, dynamically adjusting the threshold value, and obtaining a crack binary image.

[0022] A Gaussian filter is used to remove noise from the fused image, and the contrast is enhanced by histogram equalization to obtain the preprocessed fused image; the initial global threshold is calculated using the Otsu algorithm based on the preprocessed fused image to divide the foreground and background areas; the Niblack adaptive local threshold algorithm is used for the divided foreground area, a sliding window is set to calculate the local mean and standard deviation, and the local threshold is dynamically adjusted according to the Niblack formula; the grayscale variance in the window is calculated according to the grayscale distribution characteristics of the local area, if the variance is greater than the preset threshold, the local threshold is increased by a preset proportion, if the variance is less than the preset threshold, the local threshold is reduced by a preset proportion; morphological operations are performed on the binary image, and rectangular structure elements are used for opening operations to remove noise points, and rectangular structure elements are used for closing operations to connect and disconnect crack areas, and connected area analysis is used to remove areas with an area less than the preset threshold to obtain a crack binary image.

[0023] Specifically, the fused image is preprocessed, a 3x3 Gaussian filter is used to remove noise, the image edge information is retained, and the image contrast is enhanced by histogram equalization to obtain the preprocessed fused image. For the preprocessed fused image, the Otsu algorithm is used to calculate the initial global threshold, and the image is divided into two areas: foreground and background. For the divided foreground area, the Niblack adaptive local threshold algorithm is used, a sliding window of 15x15 pixels is set, the local mean and standard deviation of the pixels in the window are calculated, the local threshold is dynamically adjusted according to the Niblack formula, and each sub-area is binarized. According to the grayscale distribution characteristics of the local area of ​​the image, the grayscale variance in each 15x15 pixel window is calculated. When the variance is greater than the preset threshold, the local threshold is increased by 10%, and when the variance is less than the preset threshold, the local threshold is reduced by 10%, so as to realize the dynamic adjustment of the threshold. Morphological operations are performed on the binarized image, and a 3x3 rectangular structure element is used to perform an open operation to remove small noise points, and then a 5x5 rectangular structure element is used to perform a closed operation to connect the disconnected crack area. Finally, the processed image is subjected to connected region analysis, and the area with an area less than 20 square pixels is removed to obtain the final crack binary image. When preprocessing the fused image, a 3x3 Gaussian filter is used with a standard deviation of 0.5 to effectively remove noise while retaining edge information. Subsequently, histogram equalization is applied to expand the image grayscale to the range of 0-255 to enhance the contrast. The initial global threshold is calculated using the Otsu algorithm, and the threshold is assumed to be 127. The Niblack adaptive local threshold algorithm is applied to the foreground area, and a sliding window of 15x15 pixels is set, and the window moves 5 pixels each time. In each window, the pixel mean m and standard deviation s are calculated, and the local threshold is calculated using the formula T=m+k*s, where k is -0.2. At the same time, the grayscale variance in the window is calculated, and the variance threshold is set to 100. When the variance is greater than 100, the local threshold is increased by 10%; when the variance is less than 100, the local threshold is reduced by 10% to achieve dynamic adjustment. Morphological operations were performed on the binary image. The 3x3 rectangular structure element was used for opening operations and the operation was repeated twice. The 5x5 rectangular structure element was used for closing operations and the operation was repeated once. Finally, a connected region analysis was performed to mark all connected regions, calculate the number of pixels in each region, and remove regions with an area less than 20 square pixels. Through this series of processing, the final crack binary image was obtained, in which the crack region was white (255) and the background was black (0).

[0024] S103, calculating the three-dimensional coordinates of the crack pixels according to the crack binary image by a three-dimensional reconstruction algorithm, and reconstructing three-dimensional point cloud data containing the crack spatial position and morphological information.

[0025] Harris corner detection algorithm is used to identify crack edges and corner points, and the two-dimensional coordinates of feature points are obtained according to the crack binary image. The camera is calibrated using a calibration plate to obtain the camera intrinsic parameter matrix and distortion coefficient, and the camera extrinsic parameter matrix is ​​determined by an external reference. The Semi-Globa lBlock Matching algorithm is used to calculate the parallax of the feature points, and the two-dimensional coordinates and parallax information of the feature points are converted into three-dimensional coordinates of the feature points by combining the camera intrinsic parameter matrix, distortion coefficient and extrinsic parameter matrix. If the three-dimensional coordinates of the feature points are obtained, dense reconstruction is performed based on the sparse reconstruction results, and points on the skeleton structure are selected as seed points, and the crack area is filled using the regional growing algorithm. The filled crack area is aligned with the point cloud using the iterative nearest point algorithm, and the aligned point cloud is smoothed using the MovingLeastSquares algorithm to obtain the final three-dimensional point cloud model of the crack.

[0026] Specifically, crack feature points are extracted from the crack binary image, and the Harris corner detection algorithm is used to identify the crack edge and corner point. The corner response threshold is set to 0.01 to obtain the two-dimensional coordinates of the feature points. The camera is calibrated using a calibration plate to obtain the camera intrinsic parameter matrix and distortion coefficient, and then the camera extrinsic parameter matrix is ​​determined by an external reference. The Semi-Global Block Matching algorithm is used to calculate the parallax of the feature points, and the two-dimensional coordinates and parallax information are converted into three-dimensional coordinates in combination with the camera parameters obtained by calibration. The obtained three-dimensional feature points are sparsely reconstructed, and the spatial relationship between the feature points is calculated by triangulation using the corresponding relationship between the camera parameters and the feature points to construct the initial three-dimensional skeleton structure. Dense reconstruction is performed based on the sparse reconstruction results, and points on the skeleton structure are selected as seed points. The crack area is filled using the regional growing algorithm, and the growth criterion is set to the depth value similarity threshold of 0.05 mm to obtain complete three-dimensional point cloud data. The K nearest neighbor statistical filter is used to remove noise points, and the K value is set to 50 and the standard deviation multiple is 1.0 to obtain a preliminary three-dimensional point cloud model of the crack. The point cloud was registered using the iterative closest point algorithm, with the maximum number of iterations set to 100 and the convergence threshold set to 0.0001m to improve the reconstruction accuracy. Finally, the MovingLeastSquares algorithm was used to smooth the point cloud, with the search radius set to 0.03m and the polynomial order set to 2, to obtain the final three-dimensional point cloud model of the crack. The Harris corner detection algorithm was applied to the crack binary image, with the corner response threshold set to 0.01 and the window size set to 3x3 pixels, to obtain about 1000 feature points. The camera was calibrated using a 9x6 chessboard calibration plate, and 20 calibration images at different angles were collected to calculate the camera intrinsic parameter matrix with a focal length of 1000 pixels and a principal point coordinate of (320,240), as well as the radial distortion coefficient of [-0.1, 0.01, 0.001, -0.0005]. A cube of known size was used as an external reference to determine the camera extrinsic parameter matrix. For each feature point, a matching window of 11x11 pixels is used to calculate the disparity using the Semi-Global Block Matching algorithm, and the disparity search range is set to 0-64 pixels. Combined with the camera parameters, the two-dimensional coordinates (u, v) and disparity d of the feature point are converted into three-dimensional coordinates (X, Y, Z). In three-dimensional space, the KD tree data structure is used to quickly find neighboring points. For each feature point, 10 nearest neighbor points are selected, and the local plane is fitted by the least squares method to construct the initial three-dimensional skeleton structure. 100 points on the skeleton structure are selected as seed points, the depth value similarity threshold is set to 0.05mm, and the regional growing algorithm is used to fill the dense point cloud of about 100,000 points. K nearest neighbor statistical filtering is performed on the point cloud, with the K value set to 50 and the standard deviation multiple being 1.0, filtering out about 5% of noise points.The iterative closest point algorithm was used for point cloud registration, with the maximum number of iterations set to 100 and the convergence threshold set to 0.0001m to achieve sub-millimeter accuracy. Finally, the MovingLeastSquares algorithm was applied, with the search radius set to 0.03m and the polynomial order set to 2, to smooth the point cloud and obtain the final crack 3D point cloud model with a uniform and smooth surface, containing about 95,000 points.

[0027] S104. Based on the three-dimensional point cloud data, a three-dimensional local feature descriptor is used to characterize the geometric properties of the crack surface, calculate the crack segment parameters, and construct a three-dimensional crack model.

[0028] The three-dimensional point cloud data is preprocessed, and a curvature-based adaptive sampling method is used to obtain sampled point cloud data; based on the sampled point cloud data, local geometric features are extracted using a fast point feature histogram algorithm to obtain a feature descriptor for each point; based on the local geometric features, the point cloud is segmented using a regional growing algorithm to determine the parameters of each crack fragment; for the crack fragment, a least squares plane fitting method is used to model the crack fragment and calculate the plane equation coefficient; if the fitting residual of the plane equation coefficient exceeds a preset threshold, a quadratic surface fitting is used for the crack fragment; the modeled three-dimensional crack model is cross-validated to determine whether the root mean square error of the three-dimensional crack model is less than a preset error threshold; if the root mean square error is less than the preset error threshold, the three-dimensional crack model is determined to be valid; based on the valid three-dimensional crack model, the coordinates and angles of the crack intersections are calculated to determine the crack connectivity and obtain a complete three-dimensional crack network model.

[0029] Specifically, the three-dimensional point cloud data was preprocessed, and the curvature-based adaptive sampling method was used. The curvature threshold was set to 0.05, the crack edges and corners were retained, and the normal vector and curvature of the sampled point cloud were calculated. The local geometric features were extracted using the fast point feature histogram algorithm, and the search radius was set to 2 mm. The feature descriptors of each point were calculated, including normal vector estimation, angle difference statistics of adjacent points, and spatial distribution of normal vectors in the spherical coordinate system, to obtain the geometric attribute representation of the crack surface. Based on the local geometric features, the point cloud was segmented using the regional growing algorithm, and the point with the largest curvature was selected as the seed point. The normal vector angle threshold was set to 15 degrees, and the curvature difference threshold was set to 0.02. Points with similar features were classified as a fragment, and the length, width, depth, and direction of each fragment were calculated. The least squares plane fitting method was used to model each crack fragment, calculate the plane equation coefficient, set the fitting residual threshold to 0.1 mm, and use quadratic surface fitting for complex shape cracks with residuals exceeding the threshold. The three-dimensional crack model was constructed by combining the fragment parameters, and all fragment models were combined to form a complete crack network structure. The constructed three-dimensional crack model was cross-validated, and 20% of the points were randomly selected for validation. The root mean square error between the model prediction value and the actual value was calculated, and the error threshold was set to 0.5mm. Finally, the crack network topology analysis was performed, the coordinates and angles of the crack intersection points were calculated, the crack connectivity was determined, and a complete three-dimensional crack network model was obtained. The three-dimensional point cloud data containing 1 million points was processed. First, the curvature-based adaptive sampling method was used, and the curvature threshold was set to 0.05 to reduce the number of point clouds to 500,000, while retaining the crack edge and corner features. For the sampled point cloud, the KNN algorithm was used to calculate the 20 nearest neighbors of each point to estimate the normal vector and curvature. The fast point feature histogram algorithm was applied, the search radius was set to 2mm, the sphere space was divided into 11x11x11 grids, the normal vector angle and point pair distance were statistically analyzed, and a 352-dimensional feature descriptor was generated. Based on the feature descriptor, the regional growing algorithm is executed, and the 1000 points with the largest curvature are selected as the initial seed points. The normal vector angle threshold is set to 15 degrees, and the curvature difference threshold is set to 0.02. Finally, about 5000 crack fragments are obtained. For each crack fragment, the principal component analysis method is used to calculate the main direction to obtain the length, width and depth. The typical crack fragment length is 50-200mm, the width is 0.5-5mm, and the depth is 1-10mm. The least squares method is used for plane fitting, and the coefficients of the plane equation Ax+By+Cz+D=0 are calculated. The fitting residual threshold is set to 0.1mm, and quadratic surface fitting is used for about 10% of complex shape cracks. All crack fragment models are combined to form a complete crack network structure. 100,000 points are randomly selected for cross-validation, and the root mean square error is calculated to be 0.3mm, which meets the error threshold requirement of 0.5mm.Finally, a topological analysis was performed to identify 500 crack intersections, calculate the intersection angle distribution between 30 and 90 degrees, and determine that 85% of the cracks are connected, ultimately generating a three-dimensional crack network model containing comprehensive geometric and topological information.

[0030] S105. According to the spatial position, morphological information, and geometric properties of the three-dimensional crack model, a non-contact excitation scheme is selected, images before and after excitation are collected, and dynamic response characteristics of the crack are extracted by image difference.

[0031] According to the spatial position, morphological information and geometric properties of the three-dimensional crack model, laser excitation is selected as the non-contact excitation scheme, and the laser power and pulse duration are set according to the crack depth; the laser power and pulse duration are obtained, and a high-resolution camera is used to collect a reference image before excitation; laser excitation is performed according to the laser power and pulse duration, and a dynamic image sequence during the excitation process is collected by a high-speed camera; image difference processing is performed on the reference image before excitation and the image sequence after excitation, and the displacement field is calculated using a sub-pixel level cross-correlation algorithm to obtain the dynamic response characteristics of the crack; wavelet transform is performed on the dynamic response characteristics of the crack, the time-frequency characteristics are extracted, and the main frequency components and energy distribution are calculated; if the main frequency components and energy distribution meet the preset conditions, the dynamic response characteristics of the crack are associated with the three-dimensional crack model, and a mapping relationship between the crack geometric parameters and the dynamic response characteristics is established.

[0032] Specifically, according to the spatial position, morphological information and geometric properties of the three-dimensional crack model, laser excitation is selected as a non-contact excitation scheme. When the crack depth is less than 5mm, the laser power is set to 5-10W and the pulse duration is 10-20ms; when the crack depth is 5-10mm, the laser power is set to 10-15W and the pulse duration is 20-30ms; when the crack depth is greater than 10mm, the laser power is set to 15-20W and the pulse duration is 30-50ms. Before excitation, a high-resolution reference image is collected, and the camera parameters are set: exposure time 1 / 1000 second, aperture value f / 8, ISO sensitivity 400, focal length 50mm, shooting distance 0.5m. Laser excitation is performed, and a dynamic image sequence during the excitation process is collected at the same time. A high-speed camera is used, the frame rate is set to 1000fps, and the resolution is 1280x1024 pixels. The excitation and image acquisition are precisely synchronized through an external trigger signal, and the image changes during the excitation duration are recorded. The image difference processing was performed on the pre-excitation reference image and the post-excitation image sequence. The displacement field was calculated using the sub-pixel level cross-correlation algorithm. The correlation window size was set to 32x32 pixels and the step size was 8 pixels. The bicubic interpolation method was used to improve the accuracy. The Gaussian filter was used to reduce the noise influence and obtain the dynamic response characteristics of the crack. The obtained dynamic response characteristics were subjected to wavelet transform, the time-frequency characteristics were extracted, and the main frequency components and energy distribution were calculated. The dynamic response characteristics were associated with the three-dimensional crack model, and the mapping relationship between the crack geometric parameters and the dynamic response characteristics was established to comprehensively evaluate the state and development trend of the crack. For a typical concrete structure crack, the three-dimensional crack model shows that its depth is 7.5mm, the width is 0.8mm, and the length is 150mm. According to these parameters, the laser excitation power is selected to be 12W and the pulse duration is 25ms. The reference image is acquired using a high-resolution camera. The parameters are set as follows: exposure time 1 / 1000 seconds, aperture value f / 8, ISO sensitivity 400, focal length 50mm, shooting distance 0.5m, and a 4000x3000 pixel reference image is obtained. When performing laser excitation, a high-speed camera is used to synchronously collect dynamic image sequences, with a frame rate of 1000fps and a resolution of 1280x1024 pixels. Accurate synchronization is achieved through an external trigger, and a total of 25 frames of images are collected to cover the entire excitation process. The collected image sequence is preprocessed, including denoising and distortion correction, and then differentiated from the reference image. The displacement field is calculated using a sub-pixel level cross-correlation algorithm, a 32x32 pixel correlation window is set, the step size is 8 pixels, bicubic interpolation is used to improve the accuracy, and a 5x5 Gaussian kernel is used for smoothing filtering to obtain a sub-micron displacement field. The displacement field data is subjected to wavelet transform, using Mor l et wavelet, with a scale range of 1-64, to obtain time-frequency characteristics. The analysis results show that the main response frequency of the crack area is 85Hz, and the energy is concentrated in the range of 20-150Hz.By correlating these dynamic response characteristics with the three-dimensional crack model and establishing a mapping relationship, it was found that the crack depth was negatively correlated with the response frequency, with a correlation coefficient of -0.85, while the crack width was positively correlated with the response amplitude, with a correlation coefficient of 0.72. Based on these correlations, it was assessed that the crack was in a stable state, but continued monitoring was required.

[0033] S106. Use the crack features as input to re-establish the crack model, match the reconstructed crack model with the three-dimensional model in the crack feature database, and form a deep learning model after matching to achieve crack classification and degree assessment.

[0034] Receive input data containing crack feature information, wherein the crack feature information includes spatial position and geometric attributes. Perform data preprocessing according to the crack feature information, use a variance-based feature selection method to screen out the features with the highest contribution, and reduce the high-dimensional features to a preset dimension through principal component analysis. If the width of the crack feature after preprocessing is greater than a preset threshold, use a first preset voxel size; if the width of the crack feature after preprocessing is less than the preset threshold, use a second preset voxel size to generate a voxel grid containing spatial position and geometric attributes. Obtain a three-dimensional model in a crack feature database, and use an iterative closest point algorithm to calculate the similarity between the voxel grid and the three-dimensional model. Determine whether the similarity is greater than a preset matching threshold. If it is greater than the preset matching threshold, select a preset number of models with the highest similarity as initial training samples. Use the initial training samples to construct a three-dimensional convolutional neural network, which includes a preset number of three-dimensional convolutional layers and a fully connected layer. Use a gradient-weighted class activation mapping technique to perform interpretability analysis on the three-dimensional convolutional neural network to obtain the crack feature area of ​​concern to the model.

[0035] Specifically, the input crack features were preprocessed, including feature normalization and dimensionality reduction. The variance-based feature selection method was used to screen out the features with the highest contribution, and then the principal component analysis method was used to reduce the high-dimensional features to 64 dimensions, retaining 95% of the information. Based on the preprocessed crack features, the three-dimensional crack model was reconstructed using an adaptive voxelization method. For cracks with a width greater than 1 mm, the voxel size was set to 0.1 mm, and for cracks with a width less than 1 mm, the voxel size was set to 0.01 mm, generating a voxel grid containing spatial position and geometric attributes. The reconstructed crack model was matched with the three-dimensional model in the crack feature database, and the iterative closest point algorithm was used to calculate the similarity between the models. The matching threshold was set to 0.85, and the top 50 most similar models were selected as initial training samples, and the samples were expanded to 500 by rotation and scaling. The 3D convolutional neural network was constructed using the expanded training samples. The input layer was set to a 64x64x64 3D voxel grid, including 4 3D convolutional layers (the convolution kernel size was 3x3x3, the step size was 1, and 32, 64, 128, and 256 convolution kernels were used respectively) and 3 fully connected layers (the number of neurons was 1024, 512, and 256 respectively). Each layer was followed by batch normalization and ReLU activation function. The last layer used the Softmax function to output the crack type and degree evaluation results. The model was trained using the cross entropy loss function. The model performance was evaluated using 5-fold cross validation, and the average accuracy and F1 score were calculated. The trained model was analyzed for interpretability, and the gradient weighted class activation mapping technique was used to visualize the crack feature areas that the model focused on, and identify the features that had the greatest impact on the classification and evaluation results. In practical applications, for a typical concrete crack detection task, the original crack data containing 1000 features was first collected. Through variance-based feature selection, 200 features with a variance contribution rate of more than 0.1% were retained, and then principal component analysis was used to reduce them to 64 dimensions, with a cumulative explained variance ratio of 96.5%. For a crack with a length of 150mm, a maximum width of 2.5mm, and a depth of 15mm, an adaptive voxelization method was used to reconstruct the three-dimensional model. The main part of the crack used 0.1mm voxels, while the small branches used 0.01mm voxels, and finally generated a 62x48x150 voxel grid. This model was matched with 10,000 pre-stored models in the database, and the iterative closest point algorithm was used to calculate the similarity. 53 models with a similarity greater than 0.87 were selected as initial samples. The samples were expanded to 530 by ±15° rotation and ±10% scaling. The constructed three-dimensional convolutional neural network was trained on NVID IATes l aV100 GPU with a batch size of 32 and the Adam optimizer. The initial learning rate was 0.001. After 100 epochs, the validation set accuracy reached 93.7% and the F1 score was 0.915.Using gradient weighted class activation mapping analysis, it was found that the model mainly focused on the intersections, edge contours and depth change areas of the cracks, which are key to the judgment of crack types and assessment of their severity. Ultimately, the model determined the input cracks to be stress-type cracks, with a hazard level of medium (level 3 out of 5), and a confidence level of 0.89.

[0036] S107. Based on the deep learning model, a transfer learning strategy is adopted to form an adaptive and transferable three-dimensional detection method for bridge cracks, and the model parameters are updated to adapt to the characteristics of the target bridge.

[0037] The feature extraction layer and the classification layer are obtained from the source domain deep learning model. A progressive unfreezing strategy is adopted according to the feature extraction layer and the classification layer. All layers except the last two fully connected layers are initially frozen. The crack data of the target bridge is obtained, including crack images and three-dimensional point cloud data. The crack images are preprocessed and enhanced, and the three-dimensional point cloud data is voxelized to obtain a three-dimensional grid. The model is fine-tuned and trained using the target domain data, and a fine-tuned model is obtained through fine-tuning training. According to the fine-tuned model, an adversarial domain adaptation network is introduced, and a gradient reversal layer and a domain classifier are added after the feature extraction layer. The difference in feature distribution between the source domain and the target domain is reduced through adversarial training. An active learning strategy is adopted to select samples with the highest prediction uncertainty from the target domain data for manual annotation, and the manually annotated samples are added to the training set. The fine-tuned model is updated according to the training set to obtain an updated fine-tuned model. The updated fine-tuned model is trained by combining multiple different random seeds using model integration technology, and the final prediction result is obtained by voting method.

[0038] Specifically, the feature extraction layer and classification layer are extracted from the source domain deep learning model. A progressive unfreezing strategy is adopted. All layers except the last two fully connected layers are initially frozen, and then one layer is unfrozen every 10 epochs of training until the fifth to last layer is unfrozen. The crack data of the target bridge is collected, including crack images and 3D point cloud data. The image data is preprocessed and enhanced. The 3D point cloud data is voxelized with a voxel size of 0.01m to generate a 64x64x64 3D grid to ensure consistency with the input format of the source domain model. The model is fine-tuned using the target domain data, with a small learning rate of 0.0001, a batch size of 16, and 50 training rounds. The early stopping strategy is used to stop training when the validation set loss does not decrease for 5 consecutive epochs. The adversarial domain adaptation network is introduced, and a gradient reversal layer and a domain classifier are added after the feature extraction layer. The difference in feature distribution between the source domain and the target domain is reduced through adversarial training. An active learning strategy was adopted to select 20% of the samples with the highest prediction uncertainty for manual annotation, and these high-information samples were added to the training set for model update. Finally, the model ensemble technique was used to combine the fine-tuning models trained with 5 different random seeds, and the final prediction results were obtained by voting method to form an adaptive and transferable three-dimensional detection method for bridge cracks. In the crack detection task of a cross-sea bridge, transfer learning was performed based on the pre-trained deep learning model. First, 12 convolutional layers and 3 fully connected layers were extracted from the source model, and the first 10 convolutional layers were initially frozen. 1000 bridge crack images and 500 point cloud data sets were collected, and the images were cropped, rotated, and contrast enhanced. The point cloud data was voxelized into a 64x64x64 three-dimensional grid. In the fine-tuning training, the learning rate was set to 0.0001, and one layer was unfrozen every 10 epochs, for a total of 50 rounds. The validation set loss did not decrease for 5 consecutive rounds, and the early stopping mechanism was triggered in the 37th round. The adversarial domain adaptation network was introduced, and a gradient reversal layer and a 2-layer fully connected domain classifier were added after the 10th convolution layer, with an adversarial loss weight of 0.3. The uncertainty was calculated by predicting entropy, and the top 20% (300 samples) of entropy values ​​were selected for manual annotation. After adding them to the training set, the model accuracy increased by 3.5%. Finally, 5 models were integrated, of which 3 models used different random seeds and 2 used different learning rates (0.0005 and 0.00005), and the final prediction was determined by majority voting. On the test set, the single model F1 score was 0.86, which increased to 0.91 after integration, successfully adapting to the special structure and environmental factors of the target bridge.

[0039] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the principle of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying crack positions in bridge structures using deep learning and computer vision, characterized in that: The method comprises: Acquire multi-view visible light images and depth images, fuse the visible light images and the depth images according to the registration parameters, and generate a fused image containing the texture and depth information of the bridge; For the fused image, an adaptive threshold segmentation algorithm is used to extract the crack area, and the threshold value is dynamically adjusted to obtain a crack binary image. According to the crack binary image, the three-dimensional coordinates of the crack pixel points are calculated by a three-dimensional reconstruction algorithm, and three-dimensional point cloud data containing the crack spatial position and morphological information is reconstructed; Based on the 3D point cloud data, the 3D local feature descriptor is used to characterize the geometric properties of the crack surface, calculate the crack segment parameters, and construct a 3D crack model. According to the spatial position, morphological information and geometric properties of the 3D crack model, a non-contact excitation scheme is selected, images before and after excitation are collected, and the dynamic response characteristics of the crack are extracted by image difference. Taking the crack features as input, re-establish the crack model, match the reconstructed crack model with the three-dimensional model in the crack feature database, and form a deep learning model to achieve crack classification and degree assessment; Based on the deep learning model, a transfer learning strategy is adopted to form an adaptive and transferable three-dimensional detection method for bridge cracks, and the model parameters are updated to adapt to the characteristics of the target bridge.

2. The method according to claim 1, wherein: The method of acquiring a multi-view visible light image and a depth image, fusing the visible light image and the depth image according to a registration parameter, and generating a fused image containing texture and depth information of the bridge includes: Acquire multi-view visible light images and depth images, extract feature points using the SIFT algorithm for each pair of visible light images and depth images, and calculate the feature point descriptors; According to the feature point descriptors, feature point matching is performed using a KNN algorithm, and the feature point matching is screened using a RANSAC algorithm to obtain image registration parameters; Performing geometric transformation on the depth image according to the image registration parameters, and performing multi-scale decomposition on the geometrically transformed image using discrete wavelet transform to obtain low-frequency sub-bands and high-frequency sub-bands; For the low-frequency sub-bands, a weighted average method is used to perform image fusion. For the high-frequency sub-bands, the energy value of the corresponding area in each sub-band is calculated, and the area with an energy value higher than a preset threshold is selected as the fusion result; Performing inverse wavelet transform on the fusion result to obtain a fused image; Post-processing the fused image to calculate a peak signal-to-noise ratio and a structural similarity index; If the peak signal-to-noise ratio and the structural similarity index do not meet the preset quality threshold, the multi-scale decomposition is performed again, and the fusion weight is adjusted and the image fusion is performed again until a final fused image that meets the quality requirements is obtained.

3. The method according to claim 1, wherein: The method uses an adaptive threshold segmentation algorithm to extract the crack area from the fused image, dynamically adjusts the threshold value, and obtains a crack binary image, including: The fused image is subjected to a Gaussian filter to remove noise, and the contrast is enhanced by histogram equalization to obtain a preprocessed fused image. The initial global threshold is calculated using the Otsu algorithm based on the preprocessed fusion image to divide the foreground and background areas; The Niblack adaptive local threshold algorithm is used for the foreground area after division, and a sliding window is set to calculate the local mean and standard deviation, and the local threshold is dynamically adjusted according to the Niblack formula; The grayscale variance in the window is calculated according to the grayscale distribution characteristics of the local area. If the variance is greater than a preset threshold, the local threshold is increased by a preset ratio. If the variance is less than the preset threshold, the local threshold is reduced by a preset ratio. Morphological operations are performed on the binary image. Rectangular structure elements are used for opening operations to remove noise points, and rectangular structure elements are used for closing operations to connect and disconnect crack areas. Connected area analysis is used to remove areas with an area smaller than a preset threshold to obtain a crack binary image.

4. The method according to claim 1, wherein: The method of calculating the three-dimensional coordinates of the crack pixel points by a three-dimensional reconstruction algorithm based on the crack binary image, and reconstructing three-dimensional point cloud data containing the spatial position and morphological information of the crack, includes: Harris corner detection algorithm is used to identify crack edges and corner points, and the two-dimensional coordinates of feature points are obtained based on the crack binary image; Use the calibration board to calibrate the camera, obtain the camera's intrinsic parameter matrix and distortion coefficient, and determine the camera's extrinsic parameter matrix through external reference objects; The Semi-GlobalBlockMatching algorithm is used to calculate the parallax of the feature points, and the two-dimensional coordinates and parallax information of the feature points are converted into three-dimensional coordinates of the feature points by combining the camera intrinsic parameter matrix, distortion coefficient and extrinsic parameter matrix; If the three-dimensional coordinates of the feature points are obtained, dense reconstruction is performed based on the sparse reconstruction results, points on the skeleton structure are selected as seed points, and the crack area is filled using the region growing algorithm; The iterative nearest point algorithm is used to perform point cloud registration on the filled crack area, and the Moving Least Squares algorithm is used to smooth the registered point cloud to obtain the final crack three-dimensional point cloud model.

5. The method according to claim 1, wherein: The method uses a three-dimensional local feature descriptor to describe the geometric properties of the crack surface based on the three-dimensional point cloud data, calculates the crack segment parameters, and constructs a three-dimensional crack model, including: Preprocess the 3D point cloud data and use a curvature-based adaptive sampling method to obtain sampled point cloud data; According to the sampled point cloud data, the fast point feature histogram algorithm is used to extract local geometric features and obtain the feature descriptor of each point; Based on the local geometric features, the point cloud is segmented using a region growing algorithm to determine the parameters of each crack segment; For the crack fragment, a least square plane fitting method is used to build a model and calculate the plane equation coefficients; If the fitting residual of the plane equation coefficient exceeds a preset threshold, a quadratic surface fitting is adopted for the crack segment; Cross-validate the modeled three-dimensional crack model to determine whether the root mean square error of the three-dimensional crack model is less than a preset error threshold; If the root mean square error is less than a preset error threshold, it is determined that the three-dimensional crack model is valid; According to the effective three-dimensional fracture model, the coordinates and angles of fracture intersections are calculated, the fracture connectivity is determined, and a complete three-dimensional fracture network model is obtained.

6. The method according to claim 1, wherein: The method comprises: selecting a non-contact excitation scheme according to the spatial position, morphological information, and geometric properties of the three-dimensional crack model, collecting images before and after the excitation, and extracting the dynamic response characteristics of the crack by image difference, including: According to the spatial position, morphological information and geometric properties of the three-dimensional crack model, laser excitation is selected as the non-contact excitation scheme, and the laser power and pulse duration are set according to the crack depth; Obtain laser power and pulse duration, and use a high-resolution camera to capture a pre-excitation reference image; Perform laser excitation according to laser power and pulse duration, and collect dynamic image sequences during the excitation process by a high-speed camera; Perform image difference processing on the pre-excitation reference image and the post-excitation image sequence, and use the sub-pixel level cross-correlation algorithm to calculate the displacement field to obtain the dynamic response characteristics of the crack; Perform wavelet transform on the dynamic response characteristics of the crack, extract the time-frequency characteristics, and calculate the main frequency components and energy distribution; If the main frequency components and energy distribution meet the preset conditions, the dynamic response characteristics of the crack are associated with the three-dimensional crack model to establish a mapping relationship between the crack geometric parameters and the dynamic response characteristics.

7. The method according to claim 1, wherein: The method uses the crack features as input, re-establishes the crack model, matches the reconstructed crack model with the three-dimensional model in the crack feature database, forms a deep learning model after matching, and realizes the classification and degree assessment of cracks, including: receiving input data including fracture characteristic information, the fracture characteristic information including spatial location and geometric attributes; Performing data preprocessing according to the crack feature information, using a variance-based feature selection method to screen out the features with the highest contribution, and reducing the high-dimensional features to a preset dimension through a principal component analysis method; If the preprocessed crack feature width is greater than a preset threshold, a first preset voxel size is used; If the preprocessed crack feature width is less than a preset threshold, a second preset voxel size is used to generate a voxel grid including spatial positions and geometric attributes; Obtain the 3D model in the crack feature database, and use the iterative closest point algorithm to calculate the similarity between the voxel grid and the 3D model; Determine whether the similarity is greater than a preset matching threshold, and if so, select a preset number of models with the highest similarity as initial training samples; Constructing a three-dimensional convolutional neural network using the initial training samples, wherein the three-dimensional convolutional neural network includes a preset number of three-dimensional convolutional layers and fully connected layers; The gradient weighted class activation mapping technique is used to perform interpretability analysis on the three-dimensional convolutional neural network to obtain the crack feature areas that the model focuses on.

8. The method according to claim 1, wherein: The method is based on a deep learning model and adopts a transfer learning strategy to form an adaptive and transferable three-dimensional bridge crack detection method, and updates the model parameters to adapt to the characteristics of the target bridge, including: Obtain feature extraction layers and classification layers from the source domain deep learning model, adopt a progressive unfreezing strategy based on the feature extraction layers and classification layers, and initially freeze all layers except the last two fully connected layers; Obtain target bridge crack data, including crack images and 3D point cloud data; Preprocessing and enhancing the crack image, and obtaining a three-dimensional grid by voxelizing the three-dimensional point cloud data; The target domain data is used to fine-tune the model, and a fine-tuned model is obtained through fine-tuning training; According to the fine-tuning model, an adversarial domain adaptation network is introduced, a gradient reversal layer and a domain classifier are added after the feature extraction layer, and the difference in feature distribution between the source domain and the target domain is reduced through adversarial training; Using an active learning strategy, we select samples with the highest prediction uncertainty from the target domain data for manual labeling, and add the manually labeled samples to the training set; Updating the fine-tuning model according to the training set to obtain an updated fine-tuning model; The updated fine-tuning model is trained by combining multiple different random seeds using model ensemble technology, and the final prediction result is obtained by voting method.

Images