A concrete bridge crack detection method based on computer vision

Abstract

The present application discloses a method for detecting cracks in concrete bridges based on computer vision, which involves bridge detection, including: collecting crack images, GPS data and depth images of concrete bridges; establishing a mapping relationship between pixel coordinates and physical space coordinates of cracks based on GPS data; obtaining pixel-level alignment of crack images and depth images based on feature matching of ORB features and registration optimization algorithm based on mutual information, and obtaining color crack images and depth crack images; extracting pixel-level crack segmentation masks of crack areas using MaskR‑CNN model; estimating depth information of crack areas based on crack segmentation masks and registered depth crack images through depth completion algorithm based on conditional generative adversarial network (CGAN), and obtaining crack depth distribution map; extracting geometric parameters of cracks based on crack depth distribution map through morphological algorithm and medial axis transformation algorithm. In view of the low detection accuracy of cracks in concrete bridges in the prior art, the present application improves the detection accuracy.

Description

Technical Field

[0001] The present application relates to the field of bridge detection, and in particular to a concrete bridge crack detection method based on computer vision. Background Art

[0002] At present, crack detection in concrete bridges mainly adopts manual inspection and traditional image processing methods. Manual inspection relies on the experience and professional knowledge of the inspectors, and has problems such as low detection efficiency, strong subjectivity, and high missed detection rate. Although traditional image processing methods, such as threshold segmentation and edge detection, have realized the automation of the detection process to a certain extent, they face limitations such as low detection accuracy and poor robustness. These methods are mainly based on RGB images to extract and analyze crack areas, which are easily affected by factors such as lighting changes and noise interference, resulting in inaccurate detection results. In addition, due to the complex texture and irregular shape of the crack area, it is difficult for traditional methods to accurately depict the detailed characteristics of the crack.

[0003] In recent years, artificial intelligence technologies represented by computer vision and deep learning have made great progress, providing new ideas and methods for concrete bridge crack detection. Computer vision technology can collect bridge surface images through cameras and other devices, and automatically extract and analyze crack information using image processing and pattern recognition algorithms. It has the advantages of high detection efficiency and strong data processing capabilities. Deep learning methods, especially convolutional neural networks, can automatically learn and mine the essential features of crack images by building multi-level feature extraction and classification models, and realize end-to-end crack area detection and segmentation.

[0004] However, existing concrete bridge crack detection methods based on computer vision and deep learning still have some shortcomings. Most methods only use RGB image information for crack detection, ignoring the depth and 3D geometric features of the crack area, resulting in low detection accuracy and reliability. Summary of the invention

[0005] In response to the problem of low accuracy in concrete bridge crack detection in the prior art, the present application provides a concrete bridge crack detection method based on computer vision, which improves the crack detection accuracy by establishing a mapping between image coordinates and physical space coordinates, performing pixel-level alignment on the crack image and depth image, and using a generative adversarial network to complete the crack depth information.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] The present application provides a method for detecting cracks in concrete bridges based on computer vision, comprising: collecting crack images and GPS data of concrete bridges, collecting depth images of the bridge surface through a laser radar, wherein each pixel value in the depth image represents the depth value of the bridge surface position corresponding to the pixel point relative to the laser radar; establishing a mapping relationship between the pixel coordinates of the crack image and the physical space coordinates of the crack according to the GPS data, and calculating the conversion coefficient from the pixel coordinates of the crack image to the physical space coordinates of the crack; registering the collected crack image and the depth image, and obtaining the pixel-level correlation between the crack image and the depth image through feature matching based on ORB features and a registration optimization algorithm based on mutual information. Align to obtain the aligned color crack image and depth crack image; take the color crack image as input, and use the MaskR-CNN model to extract the pixel-level crack segmentation mask of the crack area; according to the crack segmentation mask and the aligned depth crack image, the depth information of the crack area is estimated by the depth completion algorithm based on the conditional generative adversarial network (CGAN) to obtain the crack depth distribution map; according to the crack depth distribution map, the geometric parameters of the crack are extracted by the morphological algorithm and the medial axis transformation algorithm. The geometric parameters include the length, width and depth of the crack. According to the conversion coefficient from pixel coordinates to physical coordinates, the pixel size of the crack geometric parameters is converted into physical size and output as the detection result.

[0008] Furthermore, the conversion coefficients from the pixel coordinates of the crack image to the physical space coordinates of the crack are calculated, including: extracting the GPS coordinates of the boundary points of the crack area according to the GPS data, the GPS coordinates including longitude, latitude and elevation; according to the GPS coordinates of the boundary points of the crack area, converting the GPS coordinates into physical space coordinates in the local tangent plane coordinate system with the location of the bridge as the origin through a conversion algorithm of the GPS coordinates to the local tangent plane coordinate system, and obtaining physical space coordinate data representing the physical position of the crack area; according to the physical space coordinate data of the crack area, calculating the physical space geometric parameters of the crack area through three-dimensional space distance calculation, the physical space geometric parameters including the length and width of the crack area; extracting the crack area image from the crack image using an image segmentation algorithm; extracting the resolution and pixel number parameters of the crack area image, calculating the physical space size corresponding to each pixel in the crack area image through pixel size calculation, and obtaining the pixel size parameters of the crack area; according to the physical space geometric parameters of the crack area and the pixel size parameters of the crack area, calculating the conversion coefficients from the pixel coordinates of the crack image to the physical space coordinates of the crack through a coordinate conversion matrix calculation formula, and obtaining a conversion matrix that maps the pixel coordinates of the crack image to the physical space coordinates of the crack, and outputting it as the conversion coefficient.

[0009] Furthermore, the coordinate transformation matrix calculation formula is as follows:

[0010]

[0011] Among them, (x p ,y p ,w p ) represents the homogeneous coordinates of the crack physical space coordinates, (u, v) represents the pixel coordinates of the crack image; a 11 , a 22 Respectively represent the scaling factors in the x and y directions; a 12 , a 21 Respectively represent the shear factors in the x and y directions; a 13 , a 23 They respectively represent the translation transformation in the control affine transformation, and represent the translation amount of the origin of the crack image pixel coordinate system relative to the origin of the crack physical space coordinate system; a 31 , a 32 They respectively represent the perspective factors in controlling the perspective transformation. The introduction of perspective transformation can better deal with the perspective distortion problem in the crack image.

[0012] Further calculation of the parameters in the transformation matrix includes: selecting 4 pairs of corresponding control points (u i ,v i ) and (x pi ,y pi ), where i = 1, 2, 3, 4; using the selected control points, construct a linear equation system:

[0013]

[0014] The constructed linear equation is solved by the least square method to obtain 8 parameters a 11 , a 12 , a 13 , a 21 , a 22 , a 23 , a 31 , a 32 .

[0015] Furthermore, the aligned color crack image and depth crack image are obtained, including: performing multi-scale ORB feature extraction on the collected crack image and depth image respectively, detecting ORB feature points at different scales by constructing an image pyramid, extracting corresponding position coordinates and multi-scale feature descriptors, and obtaining two sets of multi-scale ORB feature point sets; obtaining potential matching point pairs according to the two sets of multi-scale ORB feature point sets; according to the obtained potential matching points, eliminating wrong matching point pairs by an external point elimination algorithm to obtain a matching point pair set, each point pair in the matching point pair set contains corresponding pixel coordinates in the crack image and the depth image; according to the matching point pair set, estimating the initial distance from the crack image to the depth image by the least squares method. An initial transformation matrix is ​​obtained, which includes a rotation matrix and a translation vector; the initial transformation matrix is ​​used as the initial value, and the mutual information-based registration optimization algorithm is used to iteratively optimize the transformation matrix, and the optimal transformation matrix is ​​obtained by maximizing the mutual information between the crack image and the depth image; the crack image is geometrically transformed and interpolated using the optimal transformation matrix, and the crack image is registered to the coordinate space of the depth image to obtain an aligned color crack image; the aligned color crack image is pixel-wise fused with the original depth image to generate an aligned depth crack image, and the value of each pixel in the depth crack image is a combination of the RGB value of the corresponding color crack image pixel and the depth value of the original depth image pixel, which is output as the registration result.

[0016] Furthermore, according to the two sets of ORB feature point sets, potential matching point pairs are obtained, including: constructing a KD tree index structure of the ORB feature point set of the depth image, the KD tree recursively divides the feature point set into K-dimensional subspaces, each subspace contains a part of the feature points, and forms a tree index structure; for each ORB feature point of the crack image, a nearest neighbor search is performed in the constructed KD tree, and the coordinate values ​​of the feature point in K dimensions are recursively compared to obtain the candidate matching point closest to it in the feature space, and obtain the candidate matching point pair; for each obtained candidate matching point pair, the relationship between the crack image feature point and the depth image candidate is calculated. The feature descriptor distance between matching points is measured by the Hamming distance, which represents the number of different bits between two binary feature descriptors. The calculated feature descriptor distance is tested by the nearest neighbor ratio, and the ratio between the nearest neighbor distance and the second nearest neighbor distance in the candidate matching point is compared. If the ratio is less than a given threshold, the nearest neighbor point is accepted as the matching point. The obtained matching point pairs are cross-consistency checked, with the crack image feature points and the depth image feature points as the benchmarks, and the above steps are repeated. Only the consistent matching point pairs that meet the nearest neighbor ratio test in both directions are retained as the final potential matching point pair output.

[0017] Specifically, "directions" refer to: searching for potential matching point pairs in the KD tree of the deep image feature point set based on the crack image feature points. Searching for potential matching point pairs in the KD tree of the crack image feature point set based on the deep image feature points. Specifically, the steps of the cross consistency check are as follows: First, based on the crack image feature points, for each crack image feature point, a nearest neighbor search is performed in the KD tree of the deep image feature point set to obtain candidate matching point pairs, and matching point pairs that meet the conditions are screened out through the nearest neighbor ratio test. Then, based on the deep image feature points, for each deep image feature point, a nearest neighbor search is performed in the KD tree of the crack image feature point set to obtain candidate matching point pairs, and matching point pairs that meet the conditions are screened out through the nearest neighbor ratio test. Finally, only matching point pairs whose search results are consistent in both directions are retained, that is, the matching point pairs obtained by searching based on the crack image feature points are the same as the matching point pairs obtained by searching based on the deep image feature points, and they are considered to be reliable potential matching point pairs. In this way, through the two-way cross-validation method, the possible false matches caused by one-way search can be effectively eliminated, and the reliability and accuracy of the matching point pairs can be improved.

[0018] Furthermore, the color crack image is taken as input, and the MaskR-CNN model is used to extract the pixel-level crack segmentation mask of the crack area, including: taking the aligned color crack image as input, and using the backbone network of the MaskR-CNN model to extract the convolutional feature map of the image at different scales and semantic levels; the backbone network adopts the residual network ResNet-FPN structure, and generates a multi-scale convolutional feature map pyramid through bottom-up feature extraction and top-down feature fusion; on the convolutional feature map, the region proposal network RPN is used to generate the bounding box of the crack candidate area, and the bounding box indicates the position and size of the potential crack area; according to the bounding box of the crack candidate area, the region of interest pooling operation RoIAlign is used to extract the fixed-length feature vector of each candidate area from the convolutional feature map, and the candidate areas of different sizes are uniformly pooled into a fixed-size feature representation; the fixed-length feature vector is taken as input, and the fully connected classification of the MaskR-CNN model is used. The network classifies the features of the candidate regions and predicts the probability score of whether the candidate regions belong to cracks or non-cracks through the softmax activation function. The extracted fixed-length feature vector is used as input, and the fully convolutional segmentation network of the MaskR-CNN model is used to predict the probability mask of each pixel in the candidate region belonging to a crack or non-crack through upsampling and convolution operations. The predicted crack probability score is compared with the preset threshold, and the candidate region with a probability score greater than the threshold is determined to be a real crack region. The fixed-length feature vector corresponding to the real crack region is input into the bounding box regression network of the MaskR-CNN model to predict the corrected offset of the corresponding candidate bounding box, and the corresponding bounding box is fine-tuned according to the corrected offset to obtain the final crack detection frame. The final crack detection frame is matched with the predicted crack probability mask. For each detected crack region, the pixels in the crack region are binarized according to the corresponding probability mask to obtain a pixel-level crack segmentation mask.

[0019] Furthermore, a crack depth distribution map is obtained, including: taking the crack segmentation mask as the conditional input of the CGAN network, and the aligned deep crack image as the real sample input of the CGAN network; using the generator network of CGAN to extract features and estimate the depth of the crack segmentation mask and the deep crack image, the generator network includes an encoder sub-network and a decoder sub-network, the encoder sub-network respectively encodes the features of the crack segmentation mask and the deep crack image, fuses the encoded feature vectors, and the decoder decodes the fused feature vectors to generate a depth prediction map of the same size as the input deep crack image; using the discriminator network of CGAN to discriminate the authenticity of the depth prediction map and the real sample, and output a true probability score; constructing the loss function of the CGAN network, the loss function includes the generator network The reconstruction loss and adversarial loss of the network are adopted; the reconstruction loss uses the L1 norm distance between the depth prediction map and the real sample at the pixel level as the loss metric, and the adversarial loss uses the binary cross entropy loss of the discriminator network as the loss metric. The true probability score is improved by jointly optimizing the reconstruction loss and the adversarial loss. The batch gradient descent algorithm is used to train the CGAN network, and the model parameters of the generator network and the discriminator network are updated alternately. S56, the trained CGAN model is used to perform depth completion on the crack segmentation mask, and a depth prediction map of the crack area is generated through the forward reasoning of the generator network. The pixel depth values ​​of the corresponding crack area in the depth prediction map are extracted, and the missing depth values ​​in the aligned deep crack image are replaced by the extracted depth values. Through pixel-level replacement and fusion, a complete crack depth distribution map is obtained.

[0020] Furthermore, the detection result output is obtained, including: taking the obtained crack depth distribution map as input, using morphological opening operation to corrode the input image and then performing dilation operation to remove noise points in the image; using morphological closing operation to dilate the denoised image and then perform dilation operation to fill the holes in the image; according to the crack depth distribution map after denoising and filling, using the medial axis transformation algorithm to extract the refined skeleton image of the crack area; obtaining the coordinate distribution of all skeleton pixel points on the refined skeleton image of the crack area, using the skeleton pixel point coordinate sequence as the input of principal component analysis, performing covariance matrix decomposition on the skeleton pixel point coordinates, extracting the eigenvector corresponding to the maximum eigenvalue of the coordinate distribution as the main direction of the skeleton, and calculating the angle between the main direction and the horizontal direction as the crack strike angle, and the crack strike angle represents The crack extension direction; taking the crack depth distribution map as input, extracting the depth values ​​of all pixels on the image, calculating the global maximum and global minimum values ​​of the depth values ​​of all pixels, and using them as the maximum depth and minimum depth of the crack area respectively; taking the refined skeleton image as input, extracting the coordinates of all adjacent skeleton pixels on the refined skeleton, calculating the distance between adjacent skeleton pixels by Euclidean distance, and accumulating the Euclidean distances between all adjacent skeleton pixels to obtain the total length of the crack skeleton; according to the crack depth distribution map after denoising and filling, obtaining the pixel-level width of the crack area by morphological algorithm; converting the total length of the crack skeleton, the pixel-level width of the crack area, and the maximum and minimum values ​​of the crack depth into the physical space size parameters of the crack according to the conversion coefficient from pixel coordinates to physical coordinates, and outputting them as the crack detection result.

[0021] Further, according to the crack depth distribution map after denoising and filling, a medial axis transformation algorithm is used to extract a refined skeleton image of the crack area, including: taking the crack depth distribution map after denoising and filling as input, binarizing the input image, setting non-zero pixels as foreground images with pixel values ​​of 1, and setting zero-value pixels as background pixels with pixel values ​​of 0, to obtain a binary crack area image; taking the binary crack area image as input, using a distance transformation algorithm to calculate the distance from each foreground pixel in the image to the nearest background pixel, and generating a distance map, in which the value of each pixel in the distance map is equal to the distance from the corresponding pixel to the nearest background pixel; taking the distance map as input, using a regional growing method to extract local maximum pixel points on the distance map, taking the local maximum pixel points as seed points, taking the 8-neighborhood as a growth criterion, marking the 8-neighborhood pixel points with the same distance value as the seed point as new seed points, iteratively calculating until no new pixel points are marked as seed points, setting all marked pixel points as skeleton pixels, and generating a medial axis skeleton image of the crack area; taking the medial axis skeleton image as input, using a refinement algorithm to extract a refined skeleton image of the crack area.

[0022] Compared with the prior art, the advantages of this application are:

[0023] The laser radar is introduced to collect the depth image of the bridge surface. By performing pixel-level registration between the crack image and the depth image, the accurate correspondence and fusion of RGB information and depth information are achieved. Compared with the traditional method, this method adds the depth dimension on the basis of two-dimensional image detection and obtains the complete three-dimensional geometric information of the crack area.

[0024] The MaskR-CNN deep learning model is used to perform pixel-level semantic segmentation on crack images, and deep features are introduced as prior knowledge to improve the accuracy of crack area segmentation. Traditional threshold segmentation and edge detection methods are easily affected by light and noise, often leading to problems such as false detection and missed detection. However, this application uses multi-scale feature extraction and end-to-end semantic segmentation in deep learning methods to adaptively extract significant features of crack areas, and can also achieve accurate segmentation of crack areas with irregular shapes and complex textures, thereby improving the accuracy and robustness of crack detection.

[0025] In order to solve the problem of missing depth information in crack areas, this application innovatively introduces a depth completion strategy based on conditional generative adversarial network (CGAN). Using the crack segmentation mask as a conditional constraint and the powerful generative modeling capability of the generative adversarial network, the context-aware completion of the depth information in the crack area is achieved through the encoder-decoder network. Compared with traditional depth completion methods, this method makes full use of the structural prior of the crack area, and the generated depth information is more realistic and natural in terms of crack edges and internal texture details.

[0026] After obtaining a complete crack depth distribution map, image processing techniques such as morphological processing and central axis extraction are used to achieve automatic and refined extraction of crack area parameters. Through preprocessing such as image denoising and skeleton extraction, a refined skeleton that can accurately reflect the crack morphology and direction is obtained. On this basis, a crack strike angle calculation method based on principal component analysis, a pixel accumulation method for the total length of the crack skeleton, and a crack depth extreme value statistical method are proposed, which can comprehensively and finely characterize the geometric characteristic parameters of the crack area. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0028] Figure 1 is an exemplary flow chart of a concrete bridge crack detection method based on computer vision according to some embodiments of the present application;

[0029] Figure 2 is an exemplary flow chart of obtaining conversion coefficients according to some embodiments of the present application;

[0030] Figure 3 is an exemplary flow chart of obtaining a color crack image and a depth crack image according to some embodiments of the present application;

[0031] Figure 4 is an exemplary flow chart of the final potential matching point pairs shown in some embodiments of the present application;

[0032] Figure 5 is an exemplary flow chart of obtaining a crack segmentation mask according to some embodiments of the present application;

[0033] Figure 6 This is an exemplary flow chart for obtaining crack detection results according to some embodiments of the present application. DETAILED DESCRIPTION

[0034] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0035] Figure 1This is an exemplary flow chart of a computer vision-based concrete bridge crack detection method shown in some embodiments of the present application, including: collecting crack images and GPS data of concrete bridges, collecting depth images of the bridge surface through lidar, and each pixel value in the depth image represents the depth value of the bridge surface position corresponding to the corresponding pixel point relative to the lidar; specifically, collecting color crack images of the concrete bridge surface through a visible light camera to obtain color crack image data reflecting visual information such as crack texture and color; collecting geographic coordinate information data of the crack area of ​​the concrete bridge through the global positioning system (GPS) to obtain GPS data reflecting the crack location; collecting depth images of the concrete bridge surface through lidar to obtain depth image data reflecting the three-dimensional morphological characteristics of the bridge surface , each pixel value in the depth image represents the depth value of the bridge surface position corresponding to the corresponding pixel point relative to the laser radar; according to the GPS data, a mapping relationship between the pixel coordinates of the crack image and the physical space coordinates of the crack is established, and the conversion coefficient from the pixel coordinates of the crack image to the physical space coordinates of the crack is calculated; the collected crack image and the depth image are registered, and the alignment of the crack image and the depth image at the pixel level is obtained through feature matching based on ORB features and registration optimization algorithm based on mutual information, so as to obtain the aligned color crack image and depth crack image; wherein, the color crack image refers to the collected color image of the concrete bridge crack, which is usually an RGB image taken by a visible light camera or a smart phone and other devices, and contains visual information such as texture and color of the crack area. The depth crack image refers to the depth image of the bridge crack collected by the laser radar, and each pixel value in the depth image represents the distance of the pixel point from the camera or laser radar, and contains the three-dimensional shape and depth information of the crack area. The color crack image is taken as input, and the MaskR-CNN model is used to extract the pixel-level crack segmentation mask of the crack area. According to the crack segmentation mask and the aligned depth crack image, the depth information of the crack area is estimated by the depth completion algorithm based on the conditional generative adversarial network (CGAN) to obtain the crack depth distribution map. According to the crack depth distribution map, the geometric parameters of the crack are extracted by the morphological algorithm and the medial axis transformation algorithm. The geometric parameters include the length, width and depth of the crack. According to the conversion coefficient from pixel coordinates to physical coordinates, the pixel size of the crack geometric parameters is converted into physical size and output as the detection result.

[0036] Figure 2According to the exemplary flow chart of obtaining conversion coefficients shown in some embodiments of the present application, the conversion coefficients of the pixel coordinates of the crack image to the physical space coordinates of the crack are calculated, including: collecting GPS coordinate data of the boundary points of the crack area through a GPS device. For each boundary point of the crack area, the corresponding GPS coordinates are recorded, including longitude, latitude and elevation information. GPS coordinates usually adopt global reference coordinate systems such as WGS84. The accuracy of GPS coordinate data can be improved by measuring multiple times and taking the average value. Finally, a set of GPS coordinate data representing the boundary of the crack area is obtained, which is used for subsequent coordinate conversion and geometric parameter calculation.

[0037] The GPS coordinates of the obtained boundary points of the crack area are converted into physical space coordinates in the local tangent plane coordinate system with the location of the bridge as the origin. Since the GPS coordinates are defined in the geocentric coordinate system, and the local tangent plane coordinate system is a plane coordinate system in the local area of ​​the bridge, it is necessary to consider the influence of the earth's curvature and elevation on the coordinate conversion. Usually, the GPS coordinates are projected to the local tangent plane coordinate system using methods such as UTM projection or Gauss-Krüger projection. The location of the bridge is selected as the origin of the local tangent plane coordinate system, usually the center point of the bridge or a feature point. Using the GPS coordinates of the origin, the conversion relationship between the GPS coordinate system and the local tangent plane coordinate system is established. According to the orientation and shape of the bridge, the X-axis, Y-axis and Z-axis directions of the local tangent plane coordinate system are defined. Usually, the longitudinal direction of the bridge is set as the positive direction of the X-axis, the transverse direction of the bridge is set as the positive direction of the Y-axis, and the elevation direction is set as the positive direction of the Z-axis.

[0038] Specifically, the GPS coordinates of the boundary points of the crack area are converted into physical space coordinates in the local tangent plane coordinate system. Suppose the GPS coordinates of a boundary point of the crack area are (φ, λ, h), where φ is latitude, λ is longitude, and h is elevation. Suppose the GPS coordinates of the origin of the local tangent plane coordinate system are (φ0, λ0, h0). Calculate the longitude and latitude difference of the boundary point of the crack area relative to the origin of the local tangent plane coordinate system. Δφ=φ-φ0; Δλ=λ-λ0. Convert the longitude and latitude difference into radians. Δφ_rad=Δφ×π / 180; Δλ_rad=Δλ×π / 180. Calculate the X-coordinate and Y-coordinate of the boundary point of the crack area in the local tangent plane coordinate system. X=R×Δλ_rad×cos(φ0_rad);Y=R×Δφ_rad;wherein, R is the average radius of the earth, which is about 6371000 meters;φ0_rad is the latitude value of the origin of the local tangent plane coordinate system, in radians, and φ0_rad=φ0×π / 180. Calculate the Z coordinate of the boundary point of the crack area in the local tangent plane coordinate system, that is, the elevation difference. Z=h-h0. In summary, the conversion formula of the GPS coordinates (φ, λ, h) of the boundary point of the crack area to the physical space coordinates (X, Y, Z) in the local tangent plane coordinate system is: X=R×(λ-λ0)×π / 180×cos(φ0×π / 180);Y=R×(φ-φ0)×π / 180;Z=h-h0.

[0039] The crack region image is extracted from the crack image using an image segmentation algorithm; commonly used image segmentation algorithms include threshold segmentation, edge detection, region growing, watershed algorithm, etc. According to the image features of the crack region, such as grayscale value, texture, edge, etc., an appropriate image segmentation algorithm is selected to segment the crack image. For example, the threshold segmentation algorithm can be used to set a suitable threshold to separate the crack region from the background according to the grayscale difference between the crack region and the background. The width and height of the crack region image, that is, the number of columns and rows of the image, are obtained, which represents the resolution of the image. The total number of pixels in the crack region image is counted, that is, the pixel area of ​​the crack region. According to the actual physical size of the crack region (such as measured by a ruler) and the image resolution, the physical space size corresponding to each pixel is calculated. Assuming that the actual physical length of the crack region is L, the width is W, and the image resolution is H×V, the physical space size corresponding to each pixel is: pixel physical length = L / H; pixel physical width = W / V; pixel physical area = L×W / (H×V). According to the physical space geometric parameters of the crack area and the pixel size parameters of the crack area, the conversion coefficients from the crack image pixel coordinates to the crack physical space coordinates are calculated through the coordinate transformation matrix calculation formula, and the transformation matrix that maps the crack image pixel coordinates to the crack physical space coordinates is obtained as the conversion coefficient output.

[0040] Among them, the coordinate transformation matrix calculation formula is as follows:

[0041] Among them, (x p ,y p ,w p ) represents the homogeneous coordinates of the crack physical space coordinates, (u, v) represents the pixel coordinates of the crack image; a 11 , a 22 Respectively represent the scaling factors in the x and y directions; a 12 , a 21 Respectively represent the shear factors in the x and y directions; a 13 , a 23 They respectively represent the translation transformation in the control affine transformation, and represent the translation amount of the origin of the crack image pixel coordinate system relative to the origin of the crack physical space coordinate system; a 31 , a 32 They respectively represent the perspective factors in controlling the perspective transformation. The introduction of perspective transformation can better deal with the perspective distortion problem in the crack image.

[0042] In the crack area image and the crack area physical space coordinates, select 4 pairs of corresponding control points (u i ,v i ) and (x pi ,y pi ), where i = 1, 2, 3, 4; using the selected control points, construct a linear equation system:

[0043]

[0044] The constructed linear equation is solved by the least square method to obtain 8 parameters a 11 , a 12 , a 13 , a 21 , a 22 , a 23 , a 31 , a 32 .

[0045] Figure 3It is an exemplary flow chart for obtaining a color crack image and a depth crack image according to some embodiments of the present application, including: performing multi-scale ORB feature extraction on the collected crack image and depth image respectively, detecting ORB feature points at different scales by constructing an image pyramid, extracting corresponding position coordinates and multi-scale feature descriptors, and obtaining two sets of multi-scale ORB feature point sets; specifically, for the crack image and the depth image, an image pyramid is constructed respectively. An image pyramid is a multi-scale representation, which generates a series of images with different resolutions by repeatedly downsampling and smoothing the original image. Each layer of the pyramid represents a different scale, and the scale decreases from bottom to top, and the image resolution also decreases accordingly. Common image pyramids include Gaussian pyramids and Laplacian pyramids, and Gaussian pyramids are used here. The construction process of the Gaussian pyramid is as follows: the original image is used as the bottom layer (0th layer) of the pyramid. Gaussian smoothing is performed on the current layer image, and then the smoothed image is downsampled (usually 2 times) to obtain the previous layer image. Repeat the above smoothing and downsampling operations until the predetermined number of pyramid layers is reached or the image resolution is less than a certain threshold. By constructing an image pyramid, features of images can be extracted and analyzed at different scales.

[0046] Detect ORB feature points at different scales, and perform ORB feature point detection for each layer of the image pyramid. ORB (Oriented FAST and Rotated BRIEF) is a fast feature point detection algorithm with rotation invariance. The steps of ORB feature point detection are as follows: Use the FAST (Features from Accelerated Segment Test) algorithm to detect candidate feature points at the current scale. Perform Harris corner point responsivity calculation on the candidate feature points and select feature points with higher responsivity. Perform non-maximum suppression on the selected feature points to obtain the final ORB feature points. By detecting ORB feature points at different scales of the image pyramid, a set of feature points with a certain degree of invariance to scale changes can be obtained.

[0047] Extract the position coordinates and feature descriptors of the ORB feature points. For each detected ORB feature point, extract its position coordinates and feature descriptors. The position coordinates represent the pixel coordinates (x, y) of the feature point in the image. The feature descriptor is a binary vector of fixed length that describes the features of the image patch around the feature point. The calculation steps of the ORB feature descriptor are as follows: Select an image patch of fixed size with the feature point as the center. Rotate and scale normalize the image patch to make it rotation invariant. Use the BRIEF (Binary Robust Independent Elementary Features) algorithm to calculate the binary feature descriptor. The BRIEF algorithm generates a binary descriptor vector by comparing the brightness values ​​of randomly selected pixel pairs in the image patch. Save the position coordinates and feature descriptors of each ORB feature point to form a feature point set.

[0048] According to the acquired potential matching points, the wrong matching point pairs are eliminated by the outlier elimination algorithm to obtain a matching point pair set, each point pair in the matching point pair set contains the corresponding pixel coordinates in the crack image and the depth image; specifically, according to the potential matching point set obtained by feature matching in S32, each matching point pair contains the position coordinates of the feature points in the crack image and the depth image. The potential matching point set may contain some wrong matching point pairs, which need to be further eliminated.

[0049] The outlier removal algorithm is used to remove wrong matching point pairs from the potential matching point set to obtain a more reliable matching point pair set. Common outlier removal algorithms include RANSAC (Random Sample Consensus) and LMedS (Least Median of Squares). Here, the RANSAC algorithm is used as an example to illustrate the process of outlier removal: randomly select a minimum sample set (usually 2 or 3 pairs of matching points), and estimate a transformation model (such as affine transformation or homography transformation) based on the sample set. Use the estimated transformation model to transform all potential matching point pairs, and calculate the distance between the transformed points and the corresponding points in the other image. Matching point pairs with distances less than a preset threshold are regarded as inliers, and the rest are regarded as outliers. Count the number of inliers. If the number of inliers is greater than the preset threshold, the current transformation model is used as a candidate model. Repeat the above steps, randomly select different sample sets, estimate different transformation models, and count the number of inliers. Select the transformation model with the largest number of inliers as the final transformation model, and the corresponding inlier set as the final set of matching point pairs. Through the RANSAC algorithm, incorrect matching point pairs can be effectively eliminated to obtain a more reliable set of matching point pairs.

[0050] The set of matching point pairs is obtained, and after being processed by the external point elimination algorithm, the final set of matching point pairs is obtained. Each point pair in the matching point pair set contains the corresponding pixel coordinates in the crack image and the depth image. Each matching point pair can be expressed in the form of (x1, y1, x2, y2), where (x1, y1) is the pixel coordinate of the feature point in the crack image, and (x2, y2) is the pixel coordinate of the corresponding feature point in the depth image. The matching point pair set establishes the pixel-level correspondence between the crack image and the depth image, providing a basis for subsequent image registration and three-dimensional reconstruction.

[0051] According to the set of matching point pairs, the initial transformation matrix from the crack image to the depth image is estimated by the least squares method. The transformation matrix includes a rotation matrix and a translation vector. Specifically, the coordinates of the feature points on the crack RGB image are (x_i, y_i), and the coordinates of the feature points on the corresponding matching depth image are (X_i, Y_i, Z_i), i = 1, 2, ..., N, where N is the number of matching point pairs. We assume that the transformation from the RGB image to the depth image consists of a rotation matrix R and a translation vector t, that is: [X_i, Y_i, Z_i, 1]^T = [R|t]*[x_i, y_i, 1]^T, where R is a 3x3 rotation matrix and t is a 3x1 translation vector. Expanding the above formula, we can get: Xi = r11x + r12y + t1; Yi = r21x + r22y + t2; Zi = r31x + r32y + t3; where rjk is the element of the rotation matrix R. To estimate the parameters of the transformation matrix, the least squares objective function is constructed: minsum [(Xi-(r11x + r12y + t1))^2 + (Yi-(r21x + r22y + t2))^2 + (Zi-(r31x + r32y + t3))^2], where the transformation matrix describes the spatial transformation relationship from the RGB image to the depth image.

[0052] Taking the initial transformation matrix as the initial value, the mutual information-based registration optimization algorithm is used to iteratively optimize the transformation matrix. By maximizing the mutual information between the crack image and the depth image, the optimal transformation matrix is ​​obtained; specifically, let the crack RGB image be I_rgb and the depth image be I_depth. The transformation relationship between them can be represented by the transformation matrix T, that is, I_depth = T(I_rgb), where T is a 4x4 transformation matrix composed of the rotation matrix R and the translation vector t. Mutual Information (MI) measures the statistical dependence between the two images. The larger the mutual information, the better the registration of the two images. Therefore, the objective function based on mutual information can be constructed: maxMI(I_depth, T(I_rgb)), where MI(I_depth, T(I_rgb)) represents the mutual information between the depth image I_depth and the transformed RGB image T(I_rgb). In order to solve the objective function, an iterative optimization strategy is adopted, and the specific steps are as follows: the estimated initial transformation matrix T_init is used as the starting point of the optimization iteration, that is, T_0 = T_init; at the kth iteration, the current transformation matrix T_k is perturbed to obtain a series of candidate transformation matrices T_k^i, i = 1, 2, ..., m, where m is the number of candidate transformation matrices; the mutual information MI(I_depth, T_k^i(I_rgb)) between the depth image I_depth and the RGB image transformed by each candidate transformation matrix T_k^i is calculated respectively; the candidate transformation matrix with the largest mutual information is selected as the optimal transformation matrix for the kth iteration, that is, T_k+1 = argmax_iMI(I_depth, T_k^i(I_rgb)); it is determined whether the iteration termination condition is met (such as the increase in mutual information is less than the threshold, the number of iterations reaches the upper limit, etc.), if so, the iteration is terminated and the final optimal transformation matrix T_opt is output; otherwise, k = k+1 is set and the iteration is continued.

[0053] Using the optimal transformation matrix, the crack image is geometrically transformed and interpolated, and the crack image is aligned to the coordinate space of the depth image to obtain the aligned color crack image; specifically, let the optimal transformation matrix be T_opt, the crack RGB image be I_rgb, the depth image be I_depth, and the aligned color crack image be I_rgb_aligned. The process of geometrically transforming and interpolating I_rgb using the optimal transformation matrix T_opt is as follows: For each pixel position (x, y) in I_rgb_aligned, calculate its corresponding position (x', y') in the original RGB image I_rgb, that is: [x', y', 1]^T = T_opt^(-1)*[x, y, 1]^T; where T_opt^(-1) is the inverse matrix of the optimal transformation matrix T_opt. Since (x', y') are real coordinates and the image pixel positions are integer coordinates, (x', y') needs to be interpolated to obtain its pixel value. Common interpolation methods include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. Taking bilinear interpolation as an example, suppose the four nearest integer coordinates around (x', y') are (x1, y1), (x1, y2), (x2, y1), (x2, y2), then the pixel value at (x', y') can be calculated by the bilinear interpolation formula: I_rgb_aligned(x, y) = (1-dx)(1-dy)I_rgb(x1, y1) + (1-dx)dyI_rgb(x1, y2) + dx(1-dy)I_rgb(x2, y1) + dxdy*I_rgb(x2, y2); where dx = x'-x1, dy = y'-y1, which is the offset of (x', y') relative to the upper left corner position (x1, y1). Repeat until all pixel positions in I_rgb_aligned obtain corresponding pixel values, that is, the registration and interpolation process of the crack RGB image to the depth image coordinate space is completed, and the aligned color crack image I_rgb_aligned is obtained.

[0054] The aligned color crack image is fused with the original depth image at the pixel level to generate an aligned depth crack image. The value of each pixel in the depth crack image is a combination of the RGB value of the corresponding color crack image pixel and the depth value of the original depth image pixel, which is output as the registration result. Specifically, let the aligned color crack image be I_rgb_aligned, the original depth image be I_depth, and the generated depth crack image be I_crack_depth.

[0055] I_crack_depth has the same resolution and corresponding pixel position as I_rgb_aligned and I_depth. For each pixel position (x, y) in I_crack_depth, its pixel value is calculated as follows: I_crack_depth(x, y) = [I_rgb_aligned(x, y), I_depth(x, y)]; where I_rgb_aligned(x, y) is the RGB value of the aligned color crack image at position (x, y), and I_depth(x, y) is the depth value of the original depth image at position (x, y). In this way, the aligned color crack image and the original depth image are fused at each pixel position, and the obtained deep crack image contains not only the color information of the crack, but also the depth information of the crack area. Each pixel of the deep crack image is composed of [RGB value, depth value], where the RGB value comes from the aligned color crack image, which represents the color information of the crack; the depth value comes from the original depth image, which represents the depth information of the crack area. This integrated representation of RGB values ​​and depth values ​​not only retains the visual appearance characteristics of the cracks, but also includes the three-dimensional structural information of the crack area, providing a data basis for subsequent quantitative analysis of cracks based on depth information.

[0056] Figure 4 This is an exemplary flow chart of the final potential matching point pairs shown in some embodiments of the present application. According to the two sets of ORB feature point sets of the extracted crack image and depth image, the potential matching point pairs between them are obtained. In order to efficiently find potential matching points for each feature point of the crack image in the feature point set of the depth image, a KD tree index structure of the ORB feature point set of the depth image is first constructed. The KD tree recursively divides the feature point set into subspaces of K dimensions (usually K=2 or 3), each subspace containing a part of the feature points, forming a tree index structure. Then, for each ORB feature point of the crack image, a nearest neighbor search is performed in the constructed KD tree. By recursively comparing the coordinate values ​​of the feature point in K dimensions with the boundaries of the subspace division in the KD tree, one or several candidate matching points closest to it in the feature space can be quickly located, thereby obtaining a set of candidate matching point pairs.

[0057] For each candidate matching point pair obtained, it is necessary to further evaluate whether they are true matching points. A commonly used evaluation method is to calculate the distance between the feature descriptors of the two feature points in the candidate matching point pair. The feature descriptors generated by the ORB algorithm are binary strings, so the Hamming distance can be used as a measure of the distance between feature descriptors. Assume that a feature point in the crack image is P1, and its corresponding ORB feature descriptor is D1; ​​a candidate matching point obtained by KD tree search in the depth image is P2, and its corresponding ORB feature descriptor is D2. D1 and D2 are both binary strings of the same length, for example: D1: 1011010...1001; D2: 1001110...1011; The calculation method of the Hamming distance is as follows: perform bitwise exclusive OR (XOR) operation on two binary feature descriptors of equal length D1 and D2 to obtain a new binary string D_xor. The rule of the exclusive OR operation is: the same is 0, and the different is 1. For example: D1: 1011010...1001; D2: 1001110...1011; D_xor: 0010100...0010; the number of bits with a value of 1 in the XOR result D_xor is the Hamming distance between the two feature descriptors D1 and D2. In the above example, there are 3 bits with a value of 1 in D_xor, so the Hamming distance between D1 and D2 is 3.

[0058] After obtaining the candidate matching point pairs and calculating their feature descriptor Hamming distances, in order to further improve the reliability of feature point matching, the candidate matching point pairs can be subjected to a nearest neighbor ratio test. The nearest neighbor ratio test is a commonly used feature point matching optimization strategy that can effectively eliminate incorrect matching point pairs. For each feature point P1 in the crack image, the set of candidate matching points found in the depth image by the KD tree search is {Q1, Q2, ..., Qn}, which are sorted from small to large according to their Hamming distances with P1. The nearest neighbor point Q1 is the candidate matching point with the smallest Hamming distance, and the second nearest neighbor point Q2 is the candidate matching point with the second smallest Hamming distance. The steps of the nearest neighbor ratio test are as follows: calculate the nearest neighbor distance d_near, that is, the Hamming distance between P1 and Q1; calculate the next nearest neighbor distance d_next, that is, the Hamming distance between P1 and Q2; calculate the nearest neighbor ratio r = d_near / d_next; compare the nearest neighbor ratio r with the preset threshold τ (usually τ = 0.7 or 0.8): if r < τ, then accept the nearest neighbor point Q1 as the matching point of the feature point P1; if r ≥ τ, it is considered that the feature point P1 has no reliable matching point, and its matching point pair is eliminated. After the nearest neighbor ratio test, some unreliable matching point pairs can be eliminated, and the nearest neighbor matching point pairs with significantly higher similarity than other candidate points are retained, thereby improving the matching accuracy.

[0059] In order to further improve the stability and consistency of the matching point pairs, the matching point pairs obtained by the nearest neighbor ratio test can be cross-consistency checked. The steps of the cross-consistency check are as follows: Taking the crack image feature points as the benchmark, using KD tree search, Hamming distance calculation and nearest neighbor ratio test, find its nearest neighbor matching points in the depth image to obtain a set of matching point pairs; Taking the depth image feature points as the benchmark, using KD tree search, Hamming distance calculation and nearest neighbor ratio test, find its nearest neighbor matching points in the crack image to obtain another set of matching point pairs; Only the consistent matching point pairs that meet the nearest neighbor ratio test in both directions are retained. The cross-consistency check can filter out inconsistent matching point pairs where the one-way nearest neighbor matching is established but the reverse matching is not established, further improving the reliability of the matching results. Only the point pairs that are consistent in both directions can be considered as stable and high-confidence matches. Among them, the ORB feature matching method introduces the KD tree index structure, quickly locates the candidate matching points through the nearest neighbor search, and improves the efficiency of feature matching; At the same time, the nearest neighbor ratio test and cross-consistency check are used to effectively eliminate the wrong matching points and improve the accuracy of feature matching.

[0060] Figure 5 It is an exemplary flow chart for obtaining crack segmentation masks according to some embodiments of the present application, taking a color crack image as input, and using the MaskR-CNN model to extract pixel-level crack segmentation masks of crack areas, including: taking the aligned color crack image as input of the MaskR-CNN model, and adjusting and normalizing the image size according to the model requirements. The residual network ResNet-FPN is used as the backbone network, ResNet is responsible for extracting deep semantic features of the image, and FPN (Feature Pyramid Network) is responsible for generating multi-scale feature map pyramids. The input image is convolved and downsampled layer by layer using different residual blocks of ResNet (such as ResBlock-2, ResBlock-3, ResBlock-4, and ResBlock-5) to extract convolution feature maps of the image at different scales and semantic levels. As the number of network layers increases, the spatial resolution of the feature map gradually decreases, but the semantic information becomes richer and richer. Through the FPN structure, the convolution feature maps of different scales extracted by ResNet are upsampled and fused to generate a series of feature map pyramids with different spatial resolutions but similar semantic information. Each layer of feature maps in the feature map pyramid contains image information of different scales and semantic levels.

[0061] One or more feature maps are selected from the multi-scale convolutional feature map pyramid as the input of RPN. Usually, feature layers with rich semantic information and moderate spatial resolution (such as the feature map corresponding to ResBlock-4) are selected. On the selected feature map, a fixed-size sliding window (such as 3x3) is used to densely scan the feature map, and each sliding window corresponds to an area on the feature map. For each sliding window area, RPN uses a set of predefined anchor boxes (AnchorBox) with different scales and aspect ratios as references to generate multiple candidate bounding boxes. The candidate bounding box represents the position and size of the potential crack area, which is obtained by translation, scaling and adjustment of the anchor box. For each candidate bounding box, RPN uses a regressor to predict its position offset and scale adjustment relative to the corresponding anchor box, and corrects the position and size of the candidate bounding box through bounding box regression to make it more accurately match the actual crack area. For each candidate bounding box, RPN uses a classifier to predict whether it contains a crack area and outputs a confidence score for the crack area. The higher the confidence score, the greater the possibility that the candidate bounding box contains a crack. According to the confidence score and overlap threshold of the bounding box, the candidate bounding boxes are screened and non-maximum suppression (NMS) is performed, and the candidate bounding boxes with high confidence and low overlap are retained as the final crack candidate areas.

[0062] The bounding box of the crack candidate region generated by RPN is mapped to the convolutional feature map for feature extraction. Since the candidate bounding box is defined at the original image scale, and the convolutional feature map is downsampled multiple times, the scale and resolution are different from the original image, so the coordinates of the candidate bounding box need to be adjusted according to the spatial scale of the feature map. For each mapped candidate region, the RoIAlign operation is used to extract its corresponding feature representation from the convolutional feature map. RoIAlign samples a fixed number of feature points on the feature map in the candidate region through bilinear interpolation, and performs a pooling operation to generate a fixed-length feature vector.

[0063] The fixed-length feature vector obtained by RoIAlign is input into a fully connected classification network, and a series of fully connected layers (such as FC layers) are usually used to perform nonlinear transformation and feature extraction on the feature vector. The fully connected layer transforms the input feature vector through matrix multiplication and nonlinear activation functions (such as ReLU), extracts higher-level semantic features, and gradually reduces the dimension of the feature vector. After the fully connected layer, the softmax activation function is used to perform classification prediction on the transformed feature vector. The Softmax function maps the feature vector to a set of probability scores, indicating the probability that the candidate region belongs to different categories (such as crack or non-crack).

[0064] The fixed-length feature vector obtained by RoIAlign is input into the fully convolutional segmentation network. A series of convolutional layers and upsampling layers are usually used to extract features and restore the spatial scale of the feature vector. The spatial scale of the feature vector is gradually restored to the same size as the candidate region through upsampling operations (such as deconvolution or bilinear interpolation) to enable pixel-level segmentation prediction. On the upsampled feature map, a series of convolutional layers are used to extract the spatial and semantic features required for segmentation. Small-sized convolution kernels (such as 3x3) are usually used for dense feature extraction. After the last convolutional layer, a sigmoid activation function is used to perform a binary classification prediction on each pixel to obtain a probability mask of whether each pixel in the candidate region belongs to a crack or not. The probability mask is the same size as the candidate region, and each pixel value represents the probability that the pixel belongs to a crack.

[0065] The predicted crack probability score of the candidate area is compared with the preset threshold (such as 0.5), and the candidate area with a probability score greater than the threshold is determined as a real crack area, and the candidate area with a probability score less than or equal to the threshold is regarded as a non-crack area. According to the threshold comparison result, the candidate area determined as the real crack area is screened out, and its corresponding fixed-length feature vector and candidate bounding box information are extracted. The fixed-length feature vector of the real crack area is input into the bounding box regression network of the MaskR-CNN model, and the feature vector is transformed through a series of fully connected layers to predict the corrected offset of the candidate bounding box. The corrected offset includes the center coordinate offset and width and height adjustment of the bounding box. According to the predicted corrected offset, the candidate bounding box of the real crack area is fine-tuned, and the position and size of the bounding box are adjusted by translation and scaling operations to more accurately frame the crack area. The fine-tuned bounding box is output as the final crack detection box. Each detection box corresponds to a real crack area, including the position, size and confidence information of the crack area.

[0066] According to the position and size information of the crack detection frame, the crack probability mask area corresponding to each detection frame is found in the predicted probability mask. The detection frame is aligned with the probability mask through coordinate mapping and scale adjustment. For each crack detection frame, the probability value of the crack area is extracted from the corresponding probability mask according to its position and size. The extracted probability value forms a probability mask block with the same size as the detection frame. The extracted crack area probability mask block is binarized. By setting a probability threshold (such as 0.5), the pixel points with probability values ​​greater than the threshold are set to 1, indicating that the pixel belongs to the crack; the pixel points with probability values ​​less than or equal to the threshold are set to 0, indicating that the pixel belongs to the non-crack area. Through binarization, the pixel-level crack segmentation mask in each crack detection frame is obtained. The segmentation mask is the same size as the detection frame, and each pixel value is 0 or 1, indicating whether the pixel belongs to the crack area.

[0067] The crack segmentation mask is used as the conditional input of CGAN. The segmentation mask has the same size as the deep crack image, and the pixel value is 0 or 1, indicating whether the pixel belongs to the crack area. The segmentation mask provides the location and shape information of the crack area as a priori condition for generating the depth prediction map. The aligned deep crack image is used as the real sample input of CGAN. The deep crack image has the same size as the segmentation mask, and the pixel value represents the depth value of the position. The deep crack image provides the real crack depth distribution as the target of the generator network learning. The crack segmentation mask and deep crack image are respectively input into the generator network and discriminator network of CGAN as the input data for network training and inference.

[0068] The encoder subnetwork of the generator network encodes the features of the crack segmentation mask and the deep crack image respectively. The encoder subnetwork usually adopts the structure of a convolutional neural network (CNN), extracts high-level semantic features of the input data through a series of convolutional layers and downsampling layers, and compresses the feature map to a lower spatial resolution. The crack segmentation mask features extracted by the encoder subnetwork and the deep crack image features are fused, usually by operations such as concatenation or element-wise addition, and the two feature vectors are concatenated or element-wise added in the channel dimension to obtain the fused feature vector. The fused feature vector is input into the decoder subnetwork of the generator network. The decoder subnetwork decodes the feature vector through a series of upsampling layers and convolutional layers, gradually restores the spatial resolution of the feature map, and generates a depth prediction map of the same size as the input deep crack image. The output of the decoder subnetwork is the generated depth prediction map, which has the same size as the input deep crack image, and the pixel value represents the predicted depth value. The generator network generates a depth prediction map similar to the true depth distribution by learning the feature representation of the crack segmentation mask and the deep crack image.

[0069] The depth prediction map and the real sample (aligned depth crack image) generated by the generator network are input into the discriminator network respectively. The discriminator network usually adopts the structure of convolutional neural network (CNN), and extracts and discriminates the input data through a series of convolutional layers and downsampling layers. The discriminator network extracts features from the input depth prediction map and real sample, extracts local and global features of the depth image through convolutional layers and activation functions, and gradually reduces the spatial resolution of the feature map. In the last layer of the discriminator network, the extracted features are judged for authenticity through a fully connected layer and a sigmoid activation function, and a true probability score between 0 and 1 is output. The closer the true probability score is to 1, the closer the discriminator network thinks the input depth image is to the real sample; the closer the score is to 0, the closer the discriminator network thinks the input depth image is to the fake sample generated by the generator network. The output of the discriminator network is the true probability score, which indicates the authenticity of the input depth image. During the training process of CGAN, the goal of the discriminator network is to maximize the true probability score of real samples and minimize the true probability score of the deep prediction map generated by the generator network to distinguish between real samples and generated samples.

[0070] The reconstruction loss measures the pixel-level difference between the depth prediction map generated by the generator network and the real sample, and usually uses the L1 norm distance as the loss metric. The L1 loss calculates the sum of the absolute differences between the depth prediction map and the real sample at each pixel position, encouraging the generator network to generate a depth prediction map that is closer to the real sample pixel value. The adversarial loss measures the authenticity evaluation of the depth prediction map generated by the generator network in the discriminator network, and usually uses the binary cross entropy loss as the loss metric. The adversarial loss calculates the probability that the depth prediction map generated by the generator network is discriminated as a real sample in the discriminator network, encouraging the generator network to generate more realistic and indistinguishable depth prediction maps to deceive the discriminator network. The reconstruction loss and the adversarial loss are combined to construct the loss function of CGAN. The two losses are usually combined in a weighted summation manner, and the weight coefficient controls the relative importance of the two losses. The goal of the loss function is to minimize the reconstruction loss and the adversarial loss, so that the depth prediction map generated by the generator network is close to the real sample at the pixel level and has a high authenticity evaluation score. During the training process of CGAN, the parameters of the generator network and the discriminator network are optimized by minimizing the loss function. The goal of the generator network is to minimize the reconstruction loss and adversarial loss to generate more accurate and realistic depth prediction maps; the goal of the discriminator network is to maximize the authenticity evaluation score of the real samples and minimize the authenticity evaluation score of the samples generated by the generator network to distinguish between real samples and generated samples.

[0071] The training dataset is divided into several batches, each batch contains a certain number of crack segmentation masks and corresponding deep crack images. Batching can improve training efficiency and reduce memory usage. The parameters of the discriminator network are fixed, and the generator network is trained using the crack segmentation masks and deep crack images of the current batch. The output and loss function of the generator network are calculated by forward propagation, and then the gradient of the loss function to the parameters of the generator network is calculated by back propagation. The parameters of the generator network are updated using the gradient descent algorithm to minimize the reconstruction loss and adversarial loss. The parameters of the generator network are fixed, and the discriminator network is trained using the deep crack images of the current batch and the deep prediction images generated by the generator network. The authenticity evaluation scores of the discriminator network for real samples and generated samples are calculated by forward propagation, and then the gradient of the loss function to the parameters of the discriminator network is calculated by back propagation. The parameters of the discriminator network are updated using the gradient descent algorithm to maximize the authenticity evaluation scores of real samples and minimize the authenticity evaluation scores of generated samples. In each training iteration, the generator network and the discriminator network are trained alternately, usually the generator network is trained first, and then the discriminator network is trained. Alternating training allows the generator network and the discriminator network to continuously compete and improve in the confrontation process until a dynamic balance is reached. After each training iteration, the parameters of the generator network and the discriminator network are updated using an optimization algorithm (such as Adam). The optimization algorithm adjusts the network parameters according to the calculated gradient to minimize the loss function. Repeat multiple training iterations of the CGAN network until the preset number of iterations is reached or the early stopping condition is met. During the training process, the generator network and the discriminator network continue to compete and improve, and the generator network gradually learns to generate more realistic and accurate depth prediction maps.

[0072] Figure 6 This is an exemplary flow chart for obtaining crack detection results according to some embodiments of the present application, using a morphological opening operation to perform an erosion operation on the crack depth distribution map to remove noise points in the image. The erosion operation uses a structural element to convolve with the image, removes the pixel area where the structural element does not completely match, and eliminates small noise points. Then the eroded image is expanded to restore the shape and size of the crack area. The morphological closing operation is used to expand the denoised image to fill the holes in the image. The expansion operation uses a structural element to convolve with the image, expands the pixel area completely covered by the structural element, and fills the small holes in the crack area. Then the expanded image is eroded to restore the shape and size of the crack area.

[0073] According to the crack depth distribution map after denoising and filling, the medial axis transformation algorithm is used to extract the refined skeleton image of the crack area. According to the obtained crack depth distribution map after denoising and filling, the medial axis transformation algorithm is used to extract the refined skeleton image of the crack area. The medial axis transformation algorithm generates a distance map by calculating the distance from each foreground pixel to the nearest background pixel in the binary image, and extracts the local maximum pixel points as skeleton pixels on the distance map, and finally obtains the refined skeleton image of the crack area. Specifically, the crack depth distribution map after denoising and filling is used as input, and the input image is binarized. The non-zero pixels (i.e., the crack area) in the input image are set as foreground pixels with a pixel value of 1; the zero-value pixels (i.e., the background area) are set as background pixels with a pixel value of 0. A binary crack area image is obtained, in which the crack area is the foreground (pixel value is 1) and the background area is the background (pixel value is 0).

[0074] Taking the binary crack region image as input, the distance transform algorithm is used to calculate the distance from each foreground pixel in the image to the nearest background pixel. Create a distance map of the same size as the input binary crack region image to store the distance from each pixel to the nearest background pixel. Initialize the values ​​of all pixels in the distance map to infinity (or a sufficiently large value), indicating that the distance has not been calculated yet. Scan the binary crack region image row by row and column by column. For each pixel: if the current pixel is a background pixel (pixel value is 0), set the value of the corresponding position in the distance map to 0, indicating that the distance to the nearest background pixel is 0. If the current pixel is a foreground pixel (pixel value is 1), proceed to the next step of distance calculation.

[0075] For each foreground pixel, search for the background pixels around it and calculate the Euclidean distance from the foreground pixel to the nearest background pixel. Search range, 8 neighborhoods: search the eight adjacent pixels above, below, left, right, upper left, upper right, lower left, and lower right of the current pixel. For each adjacent background pixel, calculate its Euclidean distance from the current foreground pixel: Euclidean distance formula: distance = sqrt{(x_1-x_2)^2+(y_1-y_2)^2}; where (x_1, y_1) is the coordinate of the current foreground pixel, and (x_2, y_2) is the coordinate of the adjacent background pixel. Among all the calculated Euclidean distances, select the smallest distance as the distance from the current foreground pixel to the nearest background pixel. Assign the calculated minimum distance to the pixel at the corresponding position in the distance map. Repeat steps 2 and 3 until the entire binary crack area image is scanned and the distance from all foreground pixels to the nearest background pixel is calculated. A complete distance map is obtained, and the value of each pixel in the distance map is equal to the distance from the corresponding pixel to the nearest background pixel. With the distance map as input, the region growing method is used to extract the local maximum pixel points on the distance map as the skeleton pixel points, and the skeleton is further refined by the thinning algorithm to obtain the thinned skeleton image of the crack area. Scan the distance map, and for each pixel point, check the pixels in its 8-neighborhood. If the distance value of the current pixel point is greater than the distance values ​​of all pixels in its 8-neighborhood, the current pixel point is marked as a local maximum pixel point as a seed point. The local maximum pixel point refers to the pixel point with the maximum distance value in its 8-neighborhood, which is usually located on the center line of the crack area. Take the local maximum pixel point as the initial seed point and start the region growing process.

[0076] For each seed point, check the pixels in its 8-neighborhood: if the pixel in the 8-neighborhood has the same distance value as the seed point, mark it as a new seed point. Add all the 8-neighborhood pixels with the same distance value as the seed point to the seed point set. Iterate the region growing process and continuously mark the 8-neighborhood pixels of the new seed point as seed points. When no new pixels are marked as seed points, the region growing process ends. Set all marked pixels as skeleton pixels to generate the medial skeleton image of the crack area. Take the medial skeleton image as input and use the thinning algorithm to further thin the skeleton. The thinning algorithm iteratively deletes the boundary pixels of the skeleton pixels until the width of the skeleton pixels is reduced to a single pixel. The thinning process needs to meet the following conditions: Maintain the connectivity of the skeleton and do not split the skeleton. Maintain the topological structure of the skeleton and do not change the number of branches and endpoints of the skeleton. Make the skeleton more refined and smooth, and reduce the burrs and noise of the skeleton. Common thinning algorithms are: Zhang-Suen thinning algorithm: alternately delete the skeleton boundary pixels through two sub-iteration steps until no more pixels can be deleted. Hilditch thinning algorithm: Through a series of conditional judgments and iterative steps, the skeleton boundary pixels are gradually deleted until the skeleton width is reduced to a single pixel. After thinning, a thin skeleton image of the crack area is obtained, in which the crack area is represented as a skeleton line with a single pixel width.

[0077] Get the coordinate distribution of all skeleton pixels on the refined skeleton image, traverse all pixels on the refined skeleton image, and extract the coordinates of the skeleton pixels. Store the x coordinates of the skeleton pixels in an array x_coords, and store the y coordinates in an array y_coords. Form a skeleton pixel coordinate sequence, where x_coords and y_coords represent the coordinate distribution of the skeleton pixels on the x-axis and y-axis, respectively. Use the skeleton pixel coordinate sequences x_coords and y_coords as input data for principal component analysis. Principal component analysis, calculate the mean vector mean_vector of the skeleton pixel coordinates, where mean_vector[0] represents the mean of the x coordinate, and mean_vector[1] represents the mean of the y coordinate. Calculate the covariance matrix cov_matrix of the skeleton pixel coordinates, where cov_matrix[0,0] represents the variance of the x coordinate, cov_matrix[1,1] represents the variance of the y coordinate, and cov_matrix[0,1] and cov_matrix[1,0] represent the covariance of the x coordinate and the y coordinate. Perform eigenvalue decomposition on the covariance matrix cov_matrix to obtain eigenvalues ​​and eigenvectors. Select the eigenvector principal_direction corresponding to the maximum eigenvalue, which represents the main direction of the skeleton pixel distribution, that is, the main direction of the crack extension.

[0078] Calculate the crack strike angle. The angle between the main direction eigenvector principal_direction and the horizontal direction (positive direction of the x-axis) is the crack strike angle. To calculate the angle angle between principal_direction and the positive direction of the x-axis, you can use the inverse tangent function atan2(y, x), where y represents the y component of principal_direction and x represents the x component of principal_direction. Convert the angle angle to degrees, ranging from 0° to 180°. You can use the formula angle_degrees=angle*180 / pi for conversion, where pi is pi. Get the crack strike angle angle_degrees, which represents the degree of deviation of the crack extension direction from the horizontal direction.

[0079] Traverse all pixels on the crack depth distribution map and extract the depth value of each pixel. Store the extracted depth values ​​in an array depth_values. Calculate the maximum depth and initialize a variable max_depth to store the maximum depth value. The initial value is set to negative infinity. Traverse each element in the depth value array depth_values: If the current depth value is greater than max_depth, update max_depth to the current depth value. Continue to traverse the next depth value until the entire array is traversed. After the traversal is completed, max_depth is the maximum depth of the crack area, indicating the maximum extension distance of the crack in the depth direction. Calculate the minimum depth and initialize a variable min_depth to store the minimum depth value. The initial value is set to positive infinity. Traverse each element in the depth value array depth_values: If the current depth value is less than min_depth, update min_depth to the current depth value. Continue to traverse the next depth value until the entire array is traversed. After the traversal is completed, min_depth is the minimum depth of the crack area, indicating the minimum extension distance of the crack in the depth direction.

[0080] Extract the coordinates of all adjacent skeleton pixels on the refined skeleton image and store the coordinate pairs of adjacent skeleton pixels in an array. Traverse all skeleton pixels on the refined skeleton image. For each skeleton pixel, check whether there are other skeleton pixels in its 8-neighborhood. If there are adjacent skeleton pixels, combine the coordinates of the current pixel and the coordinates of the adjacent pixel into a coordinate pair and store them in an array skeleton_coords. Continue traversing the next skeleton pixel until the entire refined skeleton image is traversed. Calculate the Euclidean distance between adjacent skeleton pixels. Initialize a variable total_length to store the total length of the crack skeleton, with the initial value set to 0. Traverse each pair of adjacent skeleton pixel coordinates in the coordinate pair array skeleton_coords: Extract the coordinates of the two pixels in the current coordinate pair (x1, y1) and (x2, y2). Use the Euclidean distance formula to calculate the distance between two pixels: distance = sqrt((x2-x1)^2+(y2-y1)^2), and add the calculated distance to total_length. Continue to traverse the coordinates of the next pair of adjacent skeleton pixels until the entire array is traversed. After the traversal is completed, total_length is the total length of the crack skeleton, indicating the extension distance of the crack on the plane.

[0081] Binarize the crack depth distribution map after denoising and filling, and extract the pixel mask of the crack area. Binarize the crack depth distribution map after denoising and filling, set the pixel points with depth values ​​greater than 0 to 1, indicating the crack area; set the pixel points with depth values ​​equal to 0 to 0, indicating the non-crack area. Get the binary pixel mask of the crack area, where 1 represents crack pixels and 0 represents non-crack pixels. Define a horizontal structural element, such as a rectangular structural element with a width of 3 pixels and a height of 1 pixel. Initialize a variable erosion_count to record the number of morphological erosion operations, and set the initial value to 0. Perform a morphological erosion operation on the pixel mask of the crack area: use the horizontal structural element to erode the pixel mask, set the pixel point corresponding to the center position of the structural element to 1, if and only if all the pixels in the area completely covered by the structural element are 1. The erosion operation will gradually erode the edge pixels of the crack area, shrinking the crack area. Record the number of erosion operations and add 1 to erosion_count. Repeat the erosion operation until the crack area is completely eroded to empty, that is, all pixels in the pixel mask are 0. After the erosion operation is completed, erosion_count represents the number of times the crack area is completely eroded in the horizontal direction. Multiply erosion_count by the width of the structure element to get the pixel-level width of the crack area.

[0082] pixel_width=erosion_count×structural element width; pixel-level width represents the horizontal extension distance of the crack, in pixels.

[0083] The total length of the crack skeleton, the pixel-level width of the crack area, the maximum and minimum values ​​of the crack depth are converted into the physical space size parameters of the crack according to the conversion coefficient from pixel coordinates to physical coordinates, and output as the crack detection result. Specifically, the total length of the crack skeleton: total_length (pixel); the pixel-level width of the crack area: pixel_width (pixel); the maximum value of the crack depth: max_depth (pixel); the minimum value of the crack depth: min_depth (pixel). According to the known correspondence relationship of the control points, estimate the parameters a_11, a_12, a_13, a_21, a_22, a_23, a_31, a_32 of the coordinate transformation matrix. The correspondence relationship of the control points can be established by selecting corresponding points in the crack image and the physical space, usually requiring at least 4 pairs of control points. Use the least squares method or other optimization algorithms to solve the parameters of the coordinate transformation matrix to minimize the error between the pixel coordinates and the physical coordinates of the control points. For the total length of the crack skeleton, total_length: Substitute the coordinates (u, v) of each pixel point on the skeleton into the coordinate conversion formula to obtain the corresponding physical space coordinates (x_p, y_p). Calculate the Euclidean distance between adjacent physical space coordinate points and add them up to get the total physical space length of the crack skeleton, length_physical.

[0084] For the pixel-level width pixel_width of the crack area: Substitute the pixel-level width pixel_width into the coordinate conversion formula to obtain the corresponding physical space width width_physical. Physical space width width_physical = pixel_width × a_11 (assuming that the width direction is parallel to the x-axis). For the maximum value max_depth and minimum value min_depth of the crack depth: Substitute the pixel depth value into the coordinate conversion formula to obtain the corresponding physical space depth value. The maximum physical space depth max_depth_physical = max_depth × a_22 (assuming that the depth direction is parallel to the y-axis). The minimum physical space depth min_depth_physical = min_depth × a_22 (assuming that the depth direction is parallel to the y-axis). Output the physical space size parameters of the crack, the total physical space length of the crack skeleton: length_physical; the physical space width of the crack area: width_physical; the physical space maximum value of the crack depth: max_depth_physical; the physical space minimum value of the crack depth: min_depth_physical.

Claims

1. A concrete bridge crack detection method based on computer vision, comprising: Collect crack images and GPS data of concrete bridges, and use lidar to collect depth images of the bridge surface. Each pixel value in the depth image represents the depth value of the bridge surface position corresponding to the pixel relative to the lidar. According to GPS data, a mapping relationship between the pixel coordinates of the crack image and the physical space coordinates of the crack is established, and the conversion coefficient from the pixel coordinates of the crack image to the physical space coordinates of the crack is calculated; The collected crack image and depth image are registered, and the crack image and the depth image are aligned at the pixel level through feature matching based on ORB features and a registration optimization algorithm based on mutual information, so as to obtain the registered color crack image and depth crack image; The color crack image is used as input, and the Mask R-CNN model is used to extract the pixel-level crack segmentation mask of the crack area; According to the crack segmentation mask and the registered deep crack image, the depth information of the crack area is estimated by the depth completion algorithm based on the conditional generative adversarial network (CGAN) to obtain the crack depth distribution map. According to the crack depth distribution map, the geometric parameters of the crack are extracted through the morphological algorithm and the medial axis transformation algorithm. The geometric parameters include the length, width and depth of the crack. According to the conversion coefficient from pixel coordinates to physical coordinates, the pixel size of the crack geometric parameters is converted into the physical size and output as the detection result. The aligned color crack image and depth crack image are obtained, including: Multi-scale ORB feature extraction is performed on the collected crack images and depth images respectively. By constructing an image pyramid, ORB feature points are detected at different scales, and the corresponding position coordinates and multi-scale feature descriptors are extracted to obtain two sets of multi-scale ORB feature point sets. Obtain potential matching point pairs based on two sets of multi-scale ORB feature point sets; According to the acquired potential matching points, the wrong matching point pairs are eliminated by the external point elimination algorithm to obtain a matching point pair set, each point pair in the matching point pair set contains the corresponding pixel coordinates in the crack image and the depth image; According to the set of matching point pairs, the initial transformation matrix from the crack image to the depth image is estimated by the least square method, and the transformation matrix includes a rotation matrix and a translation vector; Taking the initial transformation matrix as the initial value, the mutual information-based registration optimization algorithm is used to iteratively optimize the transformation matrix, and the optimal transformation matrix is ​​obtained by maximizing the mutual information between the crack image and the depth image. Using the optimal transformation matrix, the crack image is geometrically transformed and interpolated, and the crack image is registered to the coordinate space of the depth image to obtain an aligned color crack image; The aligned color crack image is fused with the original depth image at pixel level to generate an aligned depth crack image. The value of each pixel in the depth crack image is a combination of the RGB value of the corresponding color crack image pixel and the depth value of the original depth image pixel, which is output as the alignment result.

2. The method for detecting cracks in concrete bridges based on computer vision according to claim 1, characterized in that: Calculate the conversion coefficients from the pixel coordinates of the crack image to the physical space coordinates of the crack, including: According to GPS data, the GPS coordinates of the boundary points of the crack area are extracted. The GPS coordinates include longitude, latitude and elevation. According to the GPS coordinates of the boundary points of the crack area, the GPS coordinates are converted into physical space coordinates in the local tangent plane coordinate system with the location of the bridge as the origin through a conversion algorithm of the GPS coordinates to the local tangent plane coordinate system, so as to obtain physical space coordinate data representing the physical location of the crack area; According to the physical space coordinate data of the crack area, the physical space geometric parameters of the crack area are calculated by three-dimensional space distance calculation, and the physical space geometric parameters include the length and width of the crack area; The image segmentation algorithm is used to extract the crack area image from the crack image; The resolution and pixel number parameters of the crack area image are extracted, and the physical space size corresponding to each pixel in the crack area image is calculated through pixel size calculation to obtain the pixel size parameters of the crack area; According to the physical space geometric parameters of the crack area and the pixel size parameters of the crack area, the conversion coefficients from the crack image pixel coordinates to the crack physical space coordinates are calculated through the coordinate transformation matrix calculation formula, and the transformation matrix that maps the crack image pixel coordinates to the crack physical space coordinates is obtained as the conversion coefficient output.

3. The method for detecting cracks in concrete bridges based on computer vision according to claim 2 is characterized in that: The coordinate transformation matrix calculation formula is as follows: Among them, (x p ,y p ,w p ) represents the homogeneous coordinates of the crack physical space coordinates, (u, v) represents the pixel coordinates of the crack image; a 11 , a 22 Respectively represent the scaling factors in the x and y directions; a 12 , a 21 Respectively represent the shear factors in the x and y directions; a 13 , a 23 They respectively represent the translation transformation in the control affine transformation, and represent the translation amount of the origin of the crack image pixel coordinate system relative to the origin of the crack physical space coordinate system; a 31 , a 32 They respectively represent the perspective factors in controlling the perspective transformation.

4. The method for detecting cracks in concrete bridges based on computer vision according to claim 3 is characterized in that: Calculate the parameters in the transformation matrix, including: In the crack area image and the crack area physical space coordinates, select 4 pairs of control points (u i ,v i ) and (x pi ,y pi ), where i=1, 2, 3, 4; Using the selected control points, construct a linear system of equations: The constructed linear equation is solved by the least squares method to obtain 8 parameters a 11 , a 12 , a 13 , a 21 , a 22 , a 23 , a 31 , a 32 .

5. The method for detecting cracks in concrete bridges based on computer vision according to claim 1, characterized in that: According to two sets of multi-scale ORB feature point sets, potential matching point pairs are obtained, including: Construct a KD tree index structure of the ORB feature point set of the depth image. The KD tree recursively divides the feature point set into K-dimensional subspaces, each of which contains a part of the feature points, forming a tree index structure. For each ORB feature point in the crack image, a nearest neighbor search is performed in the constructed KD tree, and the closest candidate matching point in the feature space is obtained by recursively comparing the coordinate values ​​of the feature point in K dimensions, thereby obtaining a candidate matching point pair; For each candidate matching point pair obtained, the feature descriptor distance between the crack image feature point and the depth image candidate matching point is calculated. The feature descriptor distance is measured using the Hamming distance, which represents the number of different bits between two binary feature descriptors. Perform a nearest neighbor ratio test on the calculated feature descriptor distance, compare the ratio between the nearest neighbor distance and the second nearest neighbor distance in the candidate matching point, and accept the nearest neighbor point as the matching point if the ratio is less than a given threshold; The obtained matching point pairs are cross-consistency checked, and the above steps are repeated with the crack image feature points and the depth image feature points as the benchmarks. Only the consistent matching point pairs that meet the nearest neighbor ratio test in both directions are retained as the final potential matching point pair output.

6. The method for detecting cracks in concrete bridges based on computer vision according to claim 5 is characterized in that: The color crack image is used as input, and the MaskR-CNN model is used to extract the pixel-level crack segmentation mask of the crack area, including: The aligned color crack image is used as input, and the backbone network of the Mask R-CNN model is used to extract the convolutional feature maps of the image at different scales and semantic levels. The backbone network adopts the residual network ResNet-FPN structure, and generates a multi-scale convolutional feature map pyramid through bottom-up feature extraction and top-down feature fusion. On the convolutional feature map, the region proposal network RPN is used to generate the bounding box of the crack candidate area. The bounding box indicates the location and size of the potential crack area. According to the bounding box of the crack candidate region, the region of interest pooling operation RoIAlign is used to extract the fixed-length feature vector of each candidate region from the convolutional feature map, and the candidate regions of different sizes are uniformly pooled into a fixed-size feature representation; The fixed-length feature vector is used as input, and the fully connected classification network of the Mask R-CNN model is used to classify the candidate region features. The probability score of whether the candidate region belongs to a crack or not is predicted by the softmax activation function. The extracted fixed-length feature vector is used as input, and the fully convolutional segmentation network of the Mask R-CNN model is used to predict the probability mask of each pixel in the candidate region belonging to a crack or non-crack through upsampling and convolution operations; The predicted crack probability score is compared with the preset threshold, and the candidate area with a probability score greater than the threshold is determined as the real crack area; the fixed-length feature vector corresponding to the real crack area is input into the bounding box regression network of the Mask R-CNN model, and the correction offset of the corresponding candidate bounding box is predicted. The corresponding bounding box is fine-tuned according to the correction offset to obtain the final crack detection box; The final crack detection box is matched with the predicted crack probability mask. For each detected crack area, the pixels in the crack area are binarized according to the corresponding probability mask to obtain a pixel-level crack segmentation mask.

7. The method for detecting cracks in concrete bridges based on computer vision according to claim 6, characterized in that: Get the crack depth distribution map, including: The crack segmentation mask is used as the conditional input of the CGAN network, and the aligned deep crack image is used as the real sample input of the CGAN network; The generator network of CGAN is used to extract features and estimate depth of crack segmentation masks and deep crack images. The generator network includes an encoder subnetwork and a decoder subnetwork. The encoder subnetwork encodes features of crack segmentation masks and deep crack images respectively, fuses the encoded feature vectors, and the decoder decodes the fused feature vectors to generate a depth prediction map of the same size as the input deep crack image. Use the CGAN discriminator network to distinguish the authenticity of the depth prediction map and the real sample, and output the real probability score; Construct the loss function of the CGAN network. The loss function includes the reconstruction loss and adversarial loss of the generator network. The reconstruction loss uses the L1 norm distance between the depth prediction map and the real sample at the pixel level as the loss metric, and the adversarial loss uses the binary cross entropy loss of the discriminator network as the loss metric. By jointly optimizing the reconstruction loss and adversarial loss, the true probability score is improved. The batch gradient descent algorithm is used to train the CGAN network, and the model parameters of the generator network and the discriminator network are updated alternately; The trained CGAN model is used to perform depth completion on the crack segmentation mask. The depth prediction map of the crack area is generated through forward reasoning of the generator network. The pixel depth values ​​corresponding to the crack area in the depth prediction map are extracted, and the missing depth values ​​in the aligned deep crack image are replaced by the extracted depth values. Through pixel-level replacement and fusion, a complete crack depth distribution map is obtained.

8. The method for detecting cracks in concrete bridges based on computer vision according to claim 7, characterized in that: Get the test result output, including: The obtained crack depth distribution map is used as input, and the input image is corroded by morphological opening operation and then dilated to remove the noise points in the image; the denoised image is dilated by morphological closing operation and then corroded to fill the holes in the image; According to the crack depth distribution map after denoising and filling, the medial axis transformation algorithm is used to extract the refined skeleton image of the crack area; Obtain the coordinate distribution of all skeleton pixels on the refined skeleton image of the crack area, use the skeleton pixel coordinate sequence as the input of principal component analysis, perform covariance matrix decomposition on the skeleton pixel coordinates, extract the eigenvector corresponding to the maximum eigenvalue of the coordinate distribution as the main direction of the skeleton, and calculate the angle between the main direction and the horizontal direction as the crack strike angle, which represents the crack extension direction; Taking the crack depth distribution map as input, extract the depth values ​​of all pixels on the image, calculate the global maximum and global minimum of all pixel depth values, and use them as the maximum depth and minimum depth of the crack area respectively; Taking the refined skeleton image as input, the coordinates of all adjacent skeleton pixels on the refined skeleton are extracted, the distance between adjacent skeleton pixels is calculated by Euclidean distance, and the Euclidean distances between all adjacent skeleton pixels are accumulated to obtain the total length of the crack skeleton; According to the crack depth distribution map after denoising and filling, the pixel-level width of the crack area is obtained through morphological algorithm; The total length of the crack skeleton, the pixel-level width of the crack area, and the maximum and minimum values ​​of the crack depth are converted into the physical space size parameters of the crack according to the conversion coefficient from pixel coordinates to physical coordinates, and output as the crack detection result.

9. The method for detecting cracks in concrete bridges based on computer vision according to claim 8, characterized in that: According to the crack depth distribution map after denoising and filling, the medial axis transformation algorithm is used to extract the refined skeleton image of the crack area, including: The crack depth distribution map after denoising and filling is used as input, and the input image is binarized, and non-zero pixels are set as foreground images with a pixel value of 1, and zero-value pixels are set as background pixels with a pixel value of 0, so as to obtain a binary crack area image; Taking the binary crack area image as input, the distance transform algorithm is used to calculate the distance from each foreground pixel to the nearest background pixel in the image, and generate a distance map. The value of each pixel in the distance map is equal to the distance from the corresponding pixel to the nearest background pixel. With the distance map as input, the region growing method is used to extract the local maximum pixel points on the distance map. The local maximum pixel points are used as seed points. The 8-neighborhood is used as the growth criterion. The 8-neighborhood pixels with the same distance value as the seed point are marked as new seed points. The calculation is iterated until no new pixel points are marked as seed points. All marked pixel points are set as skeleton pixels to generate the median skeleton image of the crack area. Taking the medial axis skeleton image as input, a thinning algorithm is used to extract the thinned skeleton image of the crack area.

Images