A Pipeline Crack Detection Method Based on Point-Connected Deep Convolutional Networks

Abstract

This invention discloses a pipeline crack detection method based on a point-connected deep convolutional network. The method includes the following steps: sampling and fusing texture features from a pipeline crack image to obtain a pipeline crack feature map; performing pooling on the feature map to obtain the coordinates of the top-left and bottom-right corner points of the crack; then, placing the top-left and bottom-right corner points into Heatmaps, Embedding, and Offset branches respectively; the Heatmaps branch matching different types of crack feature maps with the crack feature map to which the corner point belongs to obtains the crack type; the Embedding branch comparing the distances between each corner point and retaining the vector value with the smallest distance; the Offset branch restoring the corner point coordinates according to the convolution ratio to obtain the position of the corner point on the original image; finally, combining the results of the three branches to obtain the corner point pair to which the crack belongs and to obtain the crack type and location. This invention utilizes point connections to achieve pipeline type and location determination, ensuring the accuracy and reliability of pipeline crack detection and location results.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology, specifically relating to a method for detecting pipeline cracks based on a point-connected deep convolutional network. Background Technology

[0002] Urban sewer pipes develop numerous problems after prolonged use, with cracks being particularly prevalent. These cracks not only accelerate pipe erosion but also cause environmental pollution. However, manually locating cracks is time-consuming and labor-intensive, and traditional target detection networks are inaccurate in detecting small cracks. Therefore, targeted detection methods for small cracks in pipes are urgently needed. Summary of the Invention

[0003] To address the aforementioned shortcomings, this invention provides a pipeline crack detection method based on a point-connected deep convolutional network, which improves the accuracy and reliability of determining the location of pipeline cracks.

[0004] The objective of this invention is achieved as follows: a method for detecting pipe cracks based on point-connected deep convolutional networks, characterized in that: the method includes:

[0005] Step 1: Sample and fuse texture features from the pipe crack image to obtain a pipe crack feature map;

[0006] Step 2: Perform corner pooling on the pipeline crack feature map obtained in Step 1 to obtain the upper left corner point and its coordinates, and the lower right corner point and its coordinates.

[0007] Step 3: Obtain the pipe crack type and category loss of the top left and bottom right corner points obtained in Step 2 by passing them through the Heatmaps branch;

[0008] Step 4: Obtain the distance vectors and vector losses of the top left and bottom right corner points obtained in Step 2 by passing them through the Embedding branch;

[0009] Step 5: Obtain the offset information and position offset loss of the two corner points by passing the offset branch through the top left and bottom right corner points obtained in Step 2;

[0010] Step 6: Combine the results obtained from Steps 3, 4, and 5 to obtain the pipe crack type and crack location, and calculate the total loss proportionally.

[0011] Preferably, the method for texture feature fusion in step 1 is as follows: the texture of the pipe crack image is convolved using a 7*7 128-channel convolution kernel, and then input into a stacked hourglass network for feature downsampling with four layers of convolutional pooling and feature upsampling with four layers of deconvolution to obtain a fused 128*128*512 pipe crack feature map.

[0012] Preferably, the method for corner pooling of the pipeline crack feature map in step 2 is as follows:

[0013] The corner pooling method includes two feature layers of size height (H) * width (W). When pooling the top left corner, the first feature layer determines the top boundary of the pipe crack through max pooling, and the second feature layer determines the left boundary of the pipe crack image through max pooling. The boundary determination formula is as follows:

[0014]

[0015]

[0016] Where f t f l For the top boundary prediction feature map and the left boundary prediction feature map, t ij The location of the top boundary point, l ij This indicates the location of the left boundary point;

[0017] The intersection of the sum of the topmost boundary and the leftmost boundary is the location of the top left corner of the target crack.

[0018] The max pooling formula for the bottom right corner is as follows:

[0019]

[0020]

[0021] Where f b f r b is the predicted feature map for the bottom boundary and the predicted feature map for the right boundary. ij r represents the location of the bottom boundary point. ij The position of the right boundary point is obtained by adding the bottom and rightmost boundaries. The intersection of the two boundaries gives the position of the bottom right corner.

[0022] Preferably, the Heatmaps branching method in step 3 is as follows:

[0023] The size of each Heatmap is determined to be height (H) * width (W) * number of crack categories (C). The crack category texture feature map stored in the Heatmap branch is matched with the crack region to which the corner point belongs on the feature map to obtain the confidence score of each category in the region. The crack type to which the corner point in the crack region belongs is the one with the highest confidence score.

[0024] During matching, for each corner point, there is one positive sample position, and the remaining positions are negative samples; the corner points of bounding boxes with an intersection-union ratio (IOU) greater than 0.7 with the original image's bounding box are transitioned using a Gaussian kernel, with the Gaussian kernel parameters set as follows:

[0025]

[0026] In the parameters, σ is set to one-third of the radius of the correct frame in the original crack image, and (x,y) are the coordinates of the corner point.

[0027] The class probability loss of pixels in the Heatmaps is summed by iterating through them. The probability of the c-th class corresponding to the pixel at position (i,j) is set as p. cij y cij The real data is obtained by applying Gaussian kernel enhancement to this location, where N is the number of cracks detected in the feature image. The crack region category loss is obtained according to the following type loss function:

[0028]

[0029] Preferably, the Embedding branch method in step 4 is as follows:

[0030] The distance between two corner points is determined using the `pull` function to obtain the distance vector between each corner point. The distance vector between the nearest corner point remains unchanged; otherwise, the `max` function in `pull` takes the maximum value of the distance vector between the corner points. The loss function is as follows:

[0031]

[0032]

[0033] Where etk is the top-left vector, ebk is the bottom-right vector, and ek is the average of etk and ebk. The pull and push operations keep the two points either together or separate them.

[0034] Preferably, the Offset branching method in step 5 is as follows:

[0035] Map the coordinates of each image downsampling stage in the stacked hourglass network. z represents the scaling factor; the offset information is calculated using the following formula to restore the image to its original position:

[0036]

[0037] The corner offset loss is obtained through the offset loss function:

[0038]

[0039] Preferably, the method for combining and calculating the loss in step 6 is as follows:

[0040] Corner points are classified according to the type obtained from heatmaps. Embedding is used to combine the top-left and bottom-right corner points with the closest distance vector values ​​within the same class into a corner point pair to determine the crack's location on the feature map. Finally, offset is used to restore the corner point pair to the original crack image, obtaining the crack's specific location and category information. The overall loss function is as follows:

[0041] Loss = Loss det +0.1 Loss pull +0.1 Loss push +Loss off

[0042] Among them, Loss off The offset loss function; Loss det Loss is the loss function for the crack region category; pull Loss is the loss function for the nearest corner point; push For corners that are not the closest, the loss function is used.

[0043] The beneficial effects of this invention are as follows: 1. By using channel convolution kernels to convolve the crack image and using stacked hourglass networks for feature downsampling and feature upsampling, a comprehensive pipeline crack feature map is obtained, which enables accurate location of the pipeline crack when performing corner pooling on the pipeline crack feature map.

[0044] 2. By using Heatmaps branches, Embedding branches, and Offset branches, the loss of the pipe crack type and category between two corner points, the distance vector and vector loss between two corner points, and the offset information and position offset loss between two corner points are realized. This further accurately determines the information between corner points, improves the accuracy of judging the location and size of pipe cracks, and ensures the reliability of the judgment and location results of small cracks in the pipeline. Attached Figure Description

[0045] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0046] The invention will be further summarized below with reference to the accompanying drawings.

[0047] A method for detecting pipe cracks based on point-connected deep convolutional networks, the method comprising:

[0048] Step 1: Sample and fuse texture features from the pipe crack image to obtain a pipe crack feature map;

[0049] The texture of the pipe crack image is convolved by using a 7*7 128-channel convolution kernel, and then input into a stacked hourglass network for feature downsampling with four layers of convolutional pooling and feature upsampling with four layers of deconvolution, resulting in a fused 128*128*512 pipe crack feature map.

[0050] The image is first sampled by a 7x7 128-channel convolution kernel, followed by max pooling. Convolutions of the pooled feature image are performed using 1x1, 3x3, and 1x1 kernel groups and a single 1x1 kernel, respectively, to obtain a 56x56 128-channel feature map and a 128-channel residual map. Then, convolutions of the same size kernel group are used to obtain 28x28, 14x14, and 7x7 feature maps and their residual maps. Transposed convolutions are used to restore the image to 14x14, 28x28, 56x56, and 128x128 sizes. The residual maps obtained from each transposed layer are summed to finally obtain a 128x128 512-channel feature map that contains both detailed locations and features.

[0051] Step 2: Perform corner pooling on the pipeline crack feature map obtained in Step 1 to obtain the upper left corner point and its coordinates, and the lower right corner point and its coordinates.

[0052] Corner pooling involves two feature layers of size height (H) * width (W). When pooling the top left corner, the first feature layer determines the top boundary of the pipe crack through max pooling, and the second feature layer determines the left boundary of the pipe crack image through max pooling. The boundary determination formula is as follows:

[0053]

[0054]

[0055] Where f t f l For the top boundary prediction feature map and the left boundary prediction feature map, t ij The location of the top boundary point, l ij This indicates the location of the left boundary point;

[0056] The intersection of the sum of the topmost boundary and the leftmost boundary is the location of the top left corner of the target crack.

[0057] The max pooling formula for the bottom right corner is as follows:

[0058]

[0059]

[0060] Where fb f r b is the predicted feature map for the bottom boundary and the predicted feature map for the right boundary. ij r represents the location of the bottom boundary point. ij The position of the right boundary point is obtained by adding the bottom and rightmost boundaries. The intersection of the two boundaries gives the position of the bottom right corner.

[0061] Step 3: Obtain the pipe crack type and category loss of the top left and bottom right corner points obtained in Step 2 by passing them through the Heatmaps branch;

[0062] The Heatmaps branching methods are as follows:

[0063] The size of each Heatmap is determined to be height (H) * width (W) * number of crack categories (C). The crack category texture feature map stored in the Heatmap branch is matched with the crack region to which the corner point belongs on the feature map to obtain the confidence score of each category in the region. The crack type to which the corner point in the crack region belongs is the one with the highest confidence score.

[0064] During matching, for each corner point, there is one positive sample position, and the remaining positions are negative samples; the corner points of bounding boxes with an intersection-union ratio (IOU) greater than 0.7 with the original image's bounding box are transitioned using a Gaussian kernel, with the Gaussian kernel parameters set as follows:

[0065]

[0066] In the parameters, σ is set to one-third of the radius of the correct frame in the original crack image, and (x,y) are the coordinates of the corner point.

[0067] The class probability loss of pixels in the Heatmaps is summed by iterating through them. The probability of the c-th class corresponding to the pixel at position (i,j) is set as p. cij y cij The real data is obtained by applying Gaussian kernel enhancement to this location, where N is the number of cracks detected in the feature image. The crack region category loss is obtained according to the following type loss function:

[0068]

[0069] Step 4: Obtain the distance vectors and vector losses of the top left and bottom right corner points obtained in Step 2 by passing them through the Embedding branch;

[0070] The Embedding branch method is as follows:

[0071] The distance between two corner points is determined using the `pull` function to obtain the distance vector between each corner point. The distance vector between the nearest corner point remains unchanged; otherwise, the `max` function in `pull` takes the maximum value of the distance vector between the corner points. The loss function is as follows:

[0072]

[0073]

[0074] Where etk is the top-left vector, ebk is the bottom-right vector, and ek is the average of etk and ebk. The pull and push operations keep the two points either together or separate them.

[0075] Step 5: Obtain the offset information and position offset loss of the two corner points by passing the offset branch through the top left and bottom right corner points obtained in Step 2;

[0076] The Offset branch method is as follows:

[0077] Map the coordinates of each image downsampling stage in the stacked hourglass network. (z is the scaling factor) The offset information is calculated using the following formula to restore the image to its original position:

[0078]

[0079] The corner offset loss is obtained through the offset loss function:

[0080]

[0081] Step 6: Combine the results obtained from steps 3, 4, and 5 to determine the pipe crack type and location, and calculate the total loss proportionally.

[0082] The method for combining and calculating the loss is as follows:

[0083] Corner points are classified according to the type obtained from heatmaps. Embedding is used to combine the top-left and bottom-right corner points with the closest distance vector values ​​within the same class into a corner point pair to determine the crack's location on the feature map. Finally, offset is used to restore the corner point pair to the original crack image, obtaining the crack's specific location and category information. The overall loss function is as follows:

[0084] Loss = Loss det +0.1 Loss pull +0.1 Loss push +Loss off

[0085] Among them, Loss off The offset loss function; Loss det Loss is the loss function for the crack region category; pull Loss is the loss function for the nearest corner point; push For corners that are not the closest, the loss function is used.

[0086] Working Principle: This invention presents a pipeline crack detection method based on a point-connected deep convolutional network. First, crack and type discrimination are performed on the pipeline crack feature map. Then, crack corner point pairs are obtained through pooling of the feature map. The final crack location is obtained through corner point pair matching. Finally, the crack's location, size, and type are determined by combining the corner point pairs with the crack type. For small targets with limited samples, this method removes the anchor box localization module frequently used in other target detection methods, avoiding the loss and interference of anchor boxes for small targets. Furthermore, corner point connectivity reduces the amount of data required for learning small targets, improving the accuracy and reliability of small crack target identification and localization.

[0087] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A pipeline crack detection method based on point connection deep convolutional network, characterized in that: The method comprises: Step 1: sampling and texture feature fusion of the pipeline crack image to obtain a pipeline crack feature map; Step 2: performing angle point pooling on the pipeline crack feature map obtained in step 1 to obtain a top-left corner point of the pipeline crack and a position coordinate of the top-left corner point, a bottom-right corner point and a position coordinate of the bottom-right corner point; Step 3: obtaining the pipeline crack type and the class loss of the two corner points through the Heatmaps branch respectively; Step 4: obtaining the distance vector and the vector loss of the two corner points through the Embedding branch respectively; Step 5: obtaining the offset information and the position offset loss of the two corner points through the Offset branch respectively; Step 6: combining the results obtained in steps 3, 4 and 5 to obtain the pipeline crack type and the crack position, and calculating the overall loss in proportion. The method for angle point pooling of the pipeline crack feature map in step 2 is as follows:

2. The method of claim 1, wherein: The angle point pooling includes two feature map layers with a height (H) and a width (W), when the top-left corner point is pooled, the first feature map layer determines the top boundary of the pipeline crack through maximum pooling, and the second feature map layer determines the left boundary of the pipeline crack image through maximum pooling, and the boundary determination formula is as follows: After calculating the topmost boundary and the left boundary, the intersection point of the sum is the position of the top-left corner point of the target crack; ； ； wherein , is a top boundary prediction feature map and a left boundary prediction feature map, is a top boundary point position, is a left boundary point position; The bottom-right corner point maximum pooling formula is as follows: The method of the Heatmaps branch in step 3 is as follows: ； ； wherein , are a bottom boundary prediction feature map and a right boundary prediction feature map, is a bottom boundary point position, is a right boundary point position, the intersection of the added boundaries of the bottom and right boundaries is a bottom-right corner point position.

3. The method of claim 1, wherein: Step 3-1: determining that the size of each group of Heatmaps is height (H) * width (W) * number of crack categories (C), matching the crack category texture feature map stored in the Heatmaps branch with the crack region to which the corner point belongs in the feature map to obtain the confidence score of each category in the region, and the highest confidence score is the crack type to which the corner point belongs in the crack region; Step 3-2: when matching, there is a positive sample position for each corner point, and the remaining positions are negative samples; the corner points of the bounding box with an intersection over union (IOU) greater than 0.7 with the original image are transitioned using a Gaussian kernel, and the Gaussian kernel parameters are set as follows: The method of the Embedding branch in step 4 is as follows: ； Parameters in Set to one third of the correct frame radius of the crack map, (x, y) is the corner point coordinate; Step 3-3: The class probability loss of the pixels on the Heatmap is traversed and added up, and the pixel at the (i, j) position corresponds to the c-th class probability, which is set as , is the real data obtained after the position is enhanced using the Gaussian kernel, N is the number of cracks detected in the feature image, and the crack region class loss is obtained according to the following type loss function: = 。 4. The method of claim 1, wherein: The distance between the two position corner point coordinates is judged, and the distance vector between each corner point is obtained using the pull function; for the distance vector of the closest corner point, the distance vector is kept unchanged, and for the distance that is not the closest, the distance vector between the corner points is taken as the maximum value through the max function in the push; the loss function is as follows: Where etk is the top-left corner vector, ebk is the bottom-right corner vector, and ek is the average of etk and ebk, and the two points are kept or pulled away through the pull and push operations. ； ； The method of the Offset branch in step 5 is as follows:

5. The method of claim 1, wherein: The corner point offset loss is obtained through the offset loss function: Mapping the coordinate positions of each layer image down-sampling stage in a stacked hourglass network z is the scale; the position on the original image is recovered by the following offset information formula: ； ​ 。 6. The method of claim 1, wherein: The method of combining and calculating the loss in step 6 is as follows: According to the type obtained by Heatmap, the corner points are classified, the left upper corner point and the right lower corner point in the same class whose distance vector values are closest are combined into a corner point pair by Embedding to determine the position of the crack in the feature map, and finally the specific position and category information of the crack are obtained by restoring the corner point pair to the original crack diagram through Offset, and the overall loss function is as follows: ； wherein, is an offset loss function; is a crack region class loss function; is a nearest corner distance loss function; is a non-nearest corner distance loss function.

Images