Intelligent detection method for fishway cracks based on depth and grayscale information

Abstract

The application provides a fishway crack intelligent detection method based on depth and grayscale information, comprising the following steps: step one, data preprocessing: collecting two-dimensional images and three-dimensional point cloud data and performing preprocessing; step two, crack rough extraction: using an improved maximum entropy method to extract cracks in two-dimensional grayscale images and depth images; step three, image registration and fusion judgment: performing image registration based on feature points, judging crack pixels, and judging as true crack points only when the pixel points are cracks in both two-dimensional grayscale images and depth images, and otherwise as background points; step four, crack fine extraction: using a connected domain analysis method to remove larger crack noise points, and then using a morphological processing method to remove crack burrs and connect broken cracks; step five, crack classification and quantitative calculation: classifying cracks and quantitatively calculating the length and width of cracks. The detection method can more accurately and efficiently quantitatively detect cracks.

Description

Technical Field

[0001] This invention relates to the field of engineering inspection technology, and in particular to an intelligent detection method for fishway cracks based on depth and grayscale information. Background Technology

[0002] Concrete is a widely used building material in engineering projects such as dams, bridges, and roads. However, due to various complex environmental influences, concrete structures can develop defects such as cracks, like those found in fish passages. Cracks significantly impact the safety of fish passages, necessitating accurate identification and quantification to support structural monitoring and maintenance. Traditional concrete crack detection methods rely on visual inspection or the use of crack measuring rulers. This method is subjective, inefficient, and lacks guaranteed accuracy. With the development of digital image processing technology, crack detection methods based on two-dimensional image grayscale have become a research hotspot. For example, CN114839192A discloses a concrete bridge crack detection system that uses image recognition for crack patterns. However, two-dimensional images are susceptible to damage from stains, shadows, and other factors, resulting in low detection accuracy. Therefore, a high-precision crack detection method capable of intelligently identifying cracks and automatically measuring their length and width is urgently needed. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a smart detection method for fishway cracks based on depth and grayscale information, which can more accurately and efficiently detect cracks quantitatively.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an intelligent detection method for fishway cracks based on depth and grayscale information, comprising the following steps:

[0005] Step 1: Data Preprocessing: Acquire 2D imagery and 3D point cloud data respectively, and preprocess the two types of data to obtain high-quality 2D grayscale images and depth images;

[0006] Step 2, Crack Coarse Extraction: Using the improved maximum entropy method, cracks are extracted from the two-dimensional grayscale image and the depth image respectively;

[0007] Step 3: Image registration and fusion judgment: Image registration is performed based on feature points. The crack pixels extracted in Step 2 are judged. If and only if the pixel is a crack in both the two-dimensional grayscale image and the depth image, it is judged as a real crack point and the pixel is retained. Otherwise, it is a background point and the crack pixel is deleted.

[0008] Step 4, Crack Refinement: First, use connected component analysis to remove larger crack noise points, and then use morphological processing methods to remove crack burrs and broken connections.

[0009] Step 5: Crack Classification and Quantitative Calculation: Cracks are classified using the minimum bounding rectangle method, and then the length and width of the cracks are quantitatively calculated.

[0010] In a preferred embodiment, step one, the preprocessing of the acquired two-dimensional image includes image grayscale conversion, image denoising, and image enhancement. Image grayscale conversion includes component method, maximum value method, average value method, or weighted average value method. Image denoising includes mean filtering, median filtering, or adaptive median filtering. Image enhancement includes histogram equalization.

[0011] In the preferred embodiment, step one, the preprocessing of the acquired 3D point cloud data includes point cloud denoising, point cloud coordinate transformation, and depth image acquisition. Point cloud denoising includes using point cloud processing software to filter out drift points, outliers, and redundant points.

[0012] In the preferred embodiment, the point cloud coordinate transformation first removes cracked point clouds by using a depth threshold, then performs plane fitting on the remaining non-cracked point clouds to transform the point cloud from the original coordinate system to a new coordinate system with the fitting plane of the non-cracked concrete points as the XOY plane. The point cloud coordinate transformation is achieved by performing plane fitting on the point cloud and solving the rotation transformation matrix.

[0013] In a preferred embodiment, the depth image acquisition includes establishing a raster image, assigning raster values ​​to the image, and normalizing the image.

[0014] The size of the raster image is m×n, and the side length of the image pixels is P. s The relationship between image size and pixel side length is shown in the following formula.

[0015]

[0016]

[0017] Among them, P s The value of y is taken from the average point distance of the point cloud; max The maximum value of the y-coordinate in the depth image. min The minimum value of the y-coordinate of the depth image. The maximum value of the X-coordinate of the depth image. The minimum X-coordinate of the depth image;

[0018] The image raster is assigned the value of the number of points falling into the raster, and the average Z-coordinate value of the points falling into the raster is taken as the depth value, as shown in the following formula.

[0019]

[0020] Among them, z i,jis the depth value of the raster cell in row i and column j of the image, and n is the number of points falling into that raster cell. The depth average of the grid cells in row i and column j of the image;

[0021] The image normalization involves normalizing the depth values ​​to between 0 and 1 and outputting a depth image, as shown in the following formula.

[0022]

[0023] in, Let i be the average depth of the raster in row i and column j of the image. The maximum average depth value in the image. G is the minimum average depth value in the image. i,j This represents the depth value, ranging from 0 to 1.

[0024] In the preferred embodiment, in step two, the fused image is first divided into blocks, and then multiple constraints are introduced to classify the image sub-blocks. The constraints are local constraints related to the image sub-blocks and global constraints related to the entire image. The local constraints include pixel ratio p(r,c), local gray mean μ(r,c), gray mean difference d(r,c), and gray mean difference. The global constraints include global gray mean μ and global gray standard deviation σ. Blocks that meet the constraints are identified as crack sub-blocks and maximum entropy threshold segmentation is performed. Otherwise, they are identified as background sub-blocks and their pixel values ​​are directly set to 0. Finally, all the segmented sub-block images are merged to complete the initial crack extraction of the two-dimensional image and depth image.

[0025] In the preferred embodiment, when the pixel ratio p(r,c) < 0.1, it is a crack sub-block; when 0.3 < p(r,c) ≤ 0.4, μ(r,c) < μ + σ / 2, and d(r,c) ≥ 2σ are simultaneously satisfied, it is a crack sub-block; when 0.4 < p(r,c) ≤ 0.5, μ(r,c) < μ - σ / 2, and d(r,c) ≥ 3σ are simultaneously satisfied, it is a crack sub-block; when 0.5 < p(r,c) ≤ 0.6, μ(r,c) < μ + σ, and d(r,c) ≥ 4σ are simultaneously satisfied, it is a crack sub-block; otherwise, it is a background sub-block, and the pixel value is set to 0.

[0026] In the preferred embodiment, in step three, the image registration first requires reading a two-dimensional grayscale image and a depth image, then calculating the coordinates of the registration points using a target, which is a high-reflectivity patch pre-placed on a concrete surface, then calculating the affine transformation parameters using the registration points, and finally performing the transformation to complete the registration of the two images and output the image.

[0027] In a preferred embodiment, in step five, the crack classification divides the cracks into transverse cracks, longitudinal cracks, and inclined cracks. By calculating the slope of the minimum bounding rectangle of the crack, when the slope is less than 0.5, it is a transverse crack; when the slope is between 0.5 and 4, it is an inclined crack; and when the slope is greater than 4, it is a longitudinal crack.

[0028] In the preferred embodiment, in step five, the quantitative calculation of the crack first requires pixel calibration, i.e., calculating the true crack size corresponding to a single pixel, then extracting the crack skeleton, and finally performing quantitative calculation of the crack length and width. The quantitative calculation of the crack length and width first involves summing the pixels of the crack skeleton, which is the pixel length of the crack. This value is then multiplied by the true crack length corresponding to a single pixel obtained through pixel calibration to obtain the true crack length. The quantitative calculation of the crack width involves dividing the area of ​​the crack pixel by the crack pixel length to obtain the pixel width of the crack. Finally, this is multiplied by the true crack width corresponding to a single pixel obtained after calibration to obtain the true crack width.

[0029] This invention provides an intelligent fishway crack detection method based on depth and grayscale information. It proposes a crack detection method that integrates two-dimensional grayscale and three-dimensional depth information. When there are false cracks caused by stains on the surface of grayscale images, the depth information of the three-dimensional point cloud is not affected by this factor. Therefore, the influence of stains can be eliminated by using its depth information, thereby improving the crack detection accuracy. Attached Figure Description

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0031] Figure 1 This is a roadmap for intelligent fishway crack detection technology based on depth and grayscale information;

[0032] Figure 2 It contains grayscale image data of concrete cracks and a schematic diagram of the target.

[0033] Figure 3 It is a three-dimensional point cloud data of concrete cracks and a schematic diagram of the target;

[0034] Figure 4 This is a roadmap for point cloud to depth image technology.

[0035] Figure 5 This is a schematic diagram of a point cloud raster image;

[0036] Figure 6 This is a schematic diagram of image separation;

[0037] Figure 7 This is a roadmap for improving the maximum entropy crack detection method;

[0038] Figure 8 This is a schematic diagram of image sub-block discrimination and segmentation;

[0039] Figure 9 This is a diagram of image registration.

[0040] Figure 10 This is a schematic diagram for determining cracks through fusion;

[0041] Figure 11 This is a schematic diagram before noise reduction using connected component analysis.

[0042] Figure 12 This is a schematic diagram after noise reduction using connected component analysis.

[0043] Figure 13 This is a schematic diagram before morphological processing for noise reduction and treatment of connection cracks;

[0044] Figure 14 This is a schematic diagram of morphological processing for noise reduction and connection of cracks;

[0045] Figure 15 This is a schematic diagram of crack classification based on the minimum bounding rectangle method;

[0046] Figure 16 This is a schematic diagram of the crack skeleton extraction;

[0047] Figure 17 This is a schematic diagram of crack and burr removal; Detailed Implementation

[0048] like Figure 1 As shown, a smart detection method for fishway cracks based on depth and grayscale information includes the following steps:

[0049] Step 1: Data Preprocessing: Grayscale images and 3D point cloud data of the concrete structure are acquired using a camera and a 3D laser scanner. The camera and 3D laser scanner can be integrated into one device or used separately. To ensure data acquisition quality, the three cameras and the 3D laser scanner need to be as perpendicular as possible to the concrete surface. To facilitate subsequent image and point cloud registration and pixel calibration, three or more targets need to be placed in non-crack areas, and these three targets should not be in a straight line. Figure 2 and Figure 3 As shown.

[0050] The two types of data are preprocessed to obtain high-quality 2D grayscale images and depth images. For the acquired image and point cloud data, regions of interest are cropped through human-computer interaction.

[0051] The acquired two-dimensional images are preprocessed, including image grayscale conversion, image denoising, and image enhancement. Image grayscale conversion includes component method, maximum value method, average value method, or weighted average value method. Image denoising includes mean filtering, median filtering, or adaptive median filtering. The image enhancement includes histogram equalization.

[0052] In this embodiment, the two-dimensional grayscale image is subjected to weighted average grayscale conversion, adaptive median filtering for noise reduction, and histogram equalization to obtain image data with fewer noise points.

[0053] Preprocessing of the acquired 3D point cloud data includes point cloud denoising, point cloud coordinate transformation, and depth image acquisition. Point cloud denoising includes using point cloud processing software to filter out drift points, outliers, and redundant points.

[0054] like Figure 4 As shown, the point cloud coordinate transformation first removes cracked point clouds using a depth threshold, then performs plane fitting on the remaining non-cracked point clouds to transform the point cloud from the original coordinate system to a new coordinate system with the fitted plane of the non-cracked concrete points as the XOY plane. This point cloud coordinate transformation is achieved by performing plane fitting on the point cloud and solving for the rotation transformation matrix. The point cloud is then transformed into a coordinate system with the cracked point cloud depth direction as the Z direction.

[0055] The depth image acquisition includes creating a raster image, assigning raster values ​​to the image, and normalizing the image.

[0056] The creation of the raster image is based on data obtained from point cloud coordinate transformation. It can be obtained using CloudCompare software by setting parameters such as image size and Z-coordinate interpolation method.

[0057] The size of the raster image is m×n, and the side length of the image pixels is P. S The relationship between image size and pixel side length is shown in the following formula.

[0058]

[0059]

[0060] Among them, P s The value of y is taken from the average point distance of the point cloud; max The maximum value of the y-coordinate in the depth image. min The minimum value of the y-coordinate of the depth image. The maximum value of the X-coordinate of the depth image. This represents the minimum X-coordinate of the depth image.

[0061] The image raster is assigned the value of the number of points falling into the raster, and the average Z-coordinate value of the points falling into the raster is taken as the depth value, as shown in the following formula.

[0062]

[0063] Among them, z i,jis the depth value of the raster cell in row i and column j of the image, and n is the number of points falling into that raster cell. The depth average of the grid cells in row i and column j of the image;

[0064] The image normalization involves normalizing the depth values ​​to between 0 and 1 and outputting a depth image, as shown in the following formula.

[0065]

[0066] in, Let i be the average depth of the raster in row i and column j of the image. The maximum average depth value in the image. G is the minimum average depth value in the image. i,j This represents the depth value, ranging from 0 to 1.

[0067] Finally, the depth image is normalized and converted into a grayscale image, such as... Figure 5 As shown.

[0068] Step 2, Crack Coarse Extraction: Using the improved maximum entropy method, cracks are extracted from the two-dimensional grayscale image and the depth image respectively.

[0069] In step two, as Figure 6 As shown, the fused image is first divided into blocks, such as... Figure 7 As shown, the maximum entropy threshold of all image sub-blocks is calculated, and preliminary segmentation is performed. The segmented image sub-blocks include crack sub-blocks and background sub-blocks. Figure 8 As shown, multiple constraints are introduced to classify image sub-blocks. These constraints include local constraints related to the image sub-block and global constraints related to the entire image. Local constraints include pixel ratio p(r,c), local gray-level mean μ(r,c), gray-level mean difference d(r,c), and gray-level mean difference. Global constraints include global gray-level mean μ and global gray-level standard deviation σ. When pixel ratio p(r,c) < 0.1, it is a cracked sub-block; when both 0.3 < p(r,c) ≤ 0. 4. If μ(r,c) < μ + σ / 2 and d(r,c) ≥ 2σ, it is a crack sub-block; if 0.4 < p(r,c) ≤ 0.5, μ(r,c) < μ - σ / 2, and d(r,c) ≥ 3σ are simultaneously satisfied, it is a crack sub-block; if 0.5 < p(r,c) ≤ 0.6, μ(r,c) < μ + σ, and d(r,c) ≥ 4σ are simultaneously satisfied, it is a crack sub-block; otherwise, it is a background sub-block. Set the pixel value to 0, merge all the segmented sub-block images, and complete the initial crack extraction of the two-dimensional image and depth image.

[0070] Step 3: Image Registration and Fusion Judgment: Image registration is performed based on feature points. The registration of the 2D grayscale image and the depth image is achieved by using targets pre-placed on the concrete surface. The targets are high-reflectivity patches placed on the concrete surface beforehand. First, the 2D grayscale image and the depth image need to be read. Then, the coordinates of the registration points are calculated using the targets. Next, the affine transformation parameters are calculated using the registration points. Finally, the transformation is performed to complete the registration of the two images and the image is output.

[0071] Computers can obtain the coordinates of a target on grayscale and depth images. These calculated coordinates can then be used to register the two images using algorithms. Next, a human-computer interaction method is used to crop out the completely overlapping regions of the registered images, thus obtaining images from different data sources with perfectly corresponding coordinates, such as... Figure 9 As shown.

[0072] To eliminate image contamination and other defects in grayscale images, and leveraging the high precision and resolution of image data, the crack pixels extracted in step two from the 2D grayscale image and depth image are evaluated. Grayscale information fusion determines that a pixel is a true crack if and only if it is a crack in both the 2D grayscale image and the depth image; otherwise, it is considered background and deleted. Figure 10 As shown.

[0073] Step 4, Crack Refinement: First, use connected component analysis to remove larger crack noise points, and then use morphological processing methods to remove crack burrs and broken connections.

[0074] Improving the fusion of maximum entropy and depth / grayscale information reveals that detected cracks may contain burrs and fractures, which can affect crack detection accuracy. Therefore, connected component analysis and morphological processing are needed to remove burrs and connect fractured cracks. In connected component analysis, setting reasonable noise area and aspect ratio of the minimum bounding rectangle of the noise points can effectively remove speckled noise, such as... Figure 11 and Figure 12 As shown.

[0075] Morphological opening and closing operations can effectively remove cracks and burrs and repair cracks, such as... Figure 13 and Figure 14 As shown.

[0076] Step 5: Crack Classification and Quantitative Calculation: Cracks are classified using the minimum bounding rectangle method, and then the length and width of the cracks are quantitatively calculated.

[0077] The crack classification divides cracks into transverse cracks, longitudinal cracks, and inclined cracks. By calculating the slope of the minimum bounding rectangle of the crack, when the slope is less than 0.5, it is a transverse crack; when the slope is between 0.5 and 4, it is an inclined crack; and when the slope is greater than 4, it is a longitudinal crack.

[0078] Crack quantitative calculation is accomplished through pixel calibration and crack skeleton extraction. First, pixel calibration is required, which involves calculating the true crack size corresponding to a single pixel. Then, the crack skeleton is extracted, and finally, the crack length and width are quantitatively calculated.

[0079] Using the Zhang or Hilditch thinning algorithm, a wide crack can be thinned into a crack only one pixel wide, i.e., the crack skeleton. The length of this skeleton is the crack length. The presence of burrs in the crack skeleton affects the accuracy of the crack length, so a burr removal algorithm is needed to remove them. The pixel length of the crack can be calculated by summing the skeleton points. Multiplying this by the calibrated true crack length corresponding to a single pixel yields the true crack length. Figure 16 and Figure 17 As shown.

[0080] The quantitative calculation of crack length and width first involves summing the pixels of the crack skeleton; this value represents the pixel length of the crack. Then, this summation is multiplied by the actual crack length per unit pixel obtained from pixel calibration to calculate the true crack length. The quantitative calculation of crack width involves dividing the area of ​​the crack pixels by the crack pixel length. In this embodiment, the area of ​​the crack pixels is the sum of all crack pixels, which is then multiplied by the actual crack width per unit pixel obtained from calibration to calculate the true crack width.

Claims

1. A smart detection method for fishway cracks based on depth and grayscale information, characterized in that, Includes the following steps: Step 1: Data Preprocessing: Acquire 2D imagery and 3D point cloud data respectively, and preprocess the two types of data to obtain high-quality 2D grayscale images and depth images; Step 2, Crack Coarse Extraction: Using the improved maximum entropy method, cracks are extracted from the two-dimensional grayscale image and the depth image respectively; First, the fused image is divided into blocks. Then, multiple constraints are introduced to classify the image sub-blocks. The constraints are local constraints related to the image sub-blocks and global constraints related to the whole image. The local constraints include pixel ratio p(r,c), local gray mean μ(r,c), gray mean difference d(r,c), and gray mean difference. The global constraints include global gray mean μ and global gray standard deviation σ. Blocks that meet the constraints are identified as crack sub-blocks and maximum entropy thresholding is performed. Otherwise, they are identified as background sub-blocks and their pixel values ​​are set to 0. Finally, all the segmented sub-block images are merged to complete the initial crack extraction of the two-dimensional image and depth image. When the pixel ratio p(r,c) < 0.1, it is a crack sub-block; when 0.3 < p(r,c) ≤ 0.4, μ(r,c) < μ + σ / 2, and d(r,c) ≥ 2σ are simultaneously satisfied, it is a crack sub-block; when 0.4 < p(r,c) ≤ 0.5, μ(r,c) < μ - σ, and d(r,c) ≥ 3σ are simultaneously satisfied, it is a crack sub-block; when 0.5 < p(r,c) ≤ 0.6, μ(r,c) < μ + σ, and d(r,c) ≥ 4σ are simultaneously satisfied, it is a crack sub-block; otherwise, it is a background sub-block, and the pixel value is set to 0. Step 3: Image registration and fusion judgment: Image registration is performed based on feature points. The crack pixels extracted in Step 2 are judged. If and only if the pixel is a crack in both the two-dimensional grayscale image and the depth image, it is judged as a real crack point and the pixel point is retained. Otherwise, it is a background point and the crack pixel point is deleted. Step 4, Crack Refinement: First, use connected component analysis to remove larger crack noise points, and then use morphological processing methods to remove crack burrs and broken connections. Step 5: Crack Classification and Quantitative Calculation: Cracks are classified using the minimum bounding rectangle method, and then the length and width of the cracks are quantitatively calculated.

2. The intelligent detection method for fishway cracks based on depth and grayscale information according to claim 1, characterized in that, In step one, the preprocessing of the acquired two-dimensional image includes image grayscale conversion, image denoising, and image enhancement. Image grayscale conversion includes component method, maximum value method, average value method, or weighted average value method. Image denoising includes mean filtering, median filtering, or adaptive median filtering. Image enhancement includes histogram equalization.

3. The intelligent detection method for fishway cracks based on depth and grayscale information according to claim 1, characterized in that, In step one, the preprocessing of the acquired 3D point cloud data includes point cloud denoising, point cloud coordinate transformation, and depth image acquisition. Point cloud denoising includes using point cloud processing software to filter out drift points, outliers, and redundant points.

4. The intelligent detection method for fishway cracks based on depth and grayscale information according to claim 3, characterized in that, The point cloud coordinate transformation first removes cracked point clouds by using a depth threshold, then performs plane fitting on the remaining non-cracked point clouds to transform the point cloud from the original coordinate system to a new coordinate system with the fitted plane of the non-cracked concrete points as the XOY plane. The point cloud coordinate transformation is achieved by performing plane fitting on the point cloud and solving the rotation transformation matrix.

5. The intelligent detection method for fishway cracks based on depth and grayscale information according to claim 3, characterized in that, The depth image acquisition includes creating a raster image, assigning image raster values, and image normalization; The size of the raster image is: The side length of the image pixels is P. s The relationship between image size and pixel side length is shown in the following formula. （1）； （2）； in, The value of y is taken from the average point distance of the point cloud; max The maximum value of the y-coordinate in the depth image. min The minimum value of the y-coordinate of the depth image. X max The maximum value of the X-coordinate of the depth image. X min The minimum X-coordinate of the depth image; The image raster is assigned the value of the number of points falling into the raster, and the average Z-coordinate value of the points falling into the raster is taken as the depth value, as shown in the following formula. （3）； in, It is an image OK The depth value of the column grid, where n is the number of points falling into that grid. For image OK The average depth of the column grid; The image normalization involves normalizing the depth values ​​to between 0 and 1 and outputting a depth image, as shown in the following formula. （4）； in, For image OK The average depth of the column grid. The maximum average depth value in the image. The minimum average depth value in the image. This represents the depth value, ranging from 0 to 1.

6. The intelligent detection method for fishway cracks based on depth and grayscale information according to claim 1, characterized in that, In step three, the image registration first requires reading a two-dimensional grayscale image and a depth image, then calculating the coordinates of the registration points using a target, which is a high-reflectivity patch pre-placed on a concrete surface. Next, the affine transformation parameters are calculated using the registration points, and finally, the transformation is performed to complete the registration of the two images and output the image.

7. The intelligent detection method for fishway cracks based on depth and grayscale information according to claim 1, characterized in that, In step five, the crack classification divides cracks into transverse cracks, longitudinal cracks, and inclined cracks. By calculating the slope of the minimum bounding rectangle of the crack, when the slope is less than 0.5, it is a transverse crack; when the slope is between 0.5 and 4, it is an inclined crack; and when the slope is greater than 4, it is a longitudinal crack.

8. The intelligent detection method for fishway cracks based on depth and grayscale information according to claim 1, characterized in that, In step five, the quantitative calculation of the crack first requires pixel calibration, that is, solving the true size of the crack corresponding to a single pixel, then extracting the crack skeleton, and finally performing quantitative calculation of the crack length and width. The quantitative calculation of the crack length and width first sums the pixels of the crack skeleton, which is the pixel length of the crack, and then multiplies it by the true length of the crack corresponding to a unit pixel solved by pixel calibration to obtain the true length of the crack. The crack width is quantitatively calculated by dividing the area of ​​the crack pixel by the crack pixel length. Finally, multiplying by the actual crack width corresponding to a single pixel obtained after calibration gives the true crack width.

Images