An automatic splicing and fusion method for concrete cracks

Abstract

The present invention discloses a method for automatically splicing and fusing concrete cracks. First, this method identifies different crack image sets from the input dataset through image matching and the union-find algorithm; then, the pictures in different crack sets are iteratively spliced and fused in sequence. The steps of splicing and fusing are as follows: first, calculate the global homography matrix and canvas size of the source image and the target image; then divide the canvas into different grids and calculate the local homography matrix of each grid vertex; next, transform the source image into the canvas with the target image as the reference system; finally, through the distance transformation method and the damage merging method, image fusion of the source image and target image and crack fusion are respectively achieved. The present invention can perform fully automatic, accurate splicing, fusion, and visualization on the detected and recognized crack dataset, so that we can more intuitively observe the complete situation of different cracks and better count the number of cracks.

Description

Technical Field

[0001] The present invention relates to the technical fields of intelligent bridge detection and image processing, and particularly relates to a method for automatically splicing and fusing concrete cracks. Background Art

[0002] Cracks are one of the most common diseases of concrete bridges. Comprehensively grasping the actual distribution of cracks is crucial for analyzing the safety of bridges. With the development of intelligent bridge detection equipment such as unmanned aerial vehicles and robotic arms, the automatic acquisition of bridge cracks has become increasingly common. When collecting these crack images, we generally need to take multiple pictures, and then detect, quantify, and visualize the collected pictures through deep learning or digital image processing methods to obtain data such as the length and width of each crack in each picture.

[0003] In the detected bridge crack dataset, there are usually many different crack images, and a complete crack is usually distributed in multiple pictures. Therefore, in order to perform image splicing automatically, we need to first identify and match these different crack images to find out which pictures belong to the same crack, and then splice the different pictures of the same crack to form a crack splicing image with a larger perspective.

[0004] However, the existing crack splicing method based on the global homography array generally realizes the splicing of a single crack image through the global homography matrix for a set of unvisualized crack image sets collected. The patent document "A Crack Splicing Method in Bridge Quality Detection" (Patent Application No.: 201910062706.3, Publication No.: CN109754368A) proposes a crack splicing method in bridge quality detection. This method first installs equipment, collects crack images, then performs crack image splicing, and finally preprocesses the spliced image by graying, gray-level transformation, and image filtering. The problems of this method are as follows:

[0005] First, the efficiency is low. This method first collects pictures of a certain area and then performs splicing. After splicing is completed, it collects pictures of another area for splicing, with low efficiency.

[0006] Second, the use of the global homography matrix is only applicable to the case where the target scene rotates along the perspective or the scene is a plane, and the splicing effect is poor, with phenomena such as ghosting or misalignment occurring.

[0007] Third, there is no quantification and visualization processing of cracks in the spliced image, that is, the crack position and crack damage data are not highlighted in the spliced image. Summary of the Invention

[0008] In view of this, the present invention provides a method for automatically splicing and fusing concrete cracks, which can perform fully automatic and accurate splicing and fusing on the detected and recognized crack data set, so that the complete situation of different cracks and crack statistical data can be observed more intuitively.

[0009] To solve the problems existing in the prior art, the technical solution adopted by the present invention is: a method for automatically splicing and fusing concrete cracks, comprising the following steps:

[0010] Step 1: Crack image matching

[0011] Take pictures to obtain a set of crack images of bridge structures, calculate the confidence of each image in the data set with the rest of the images through feature point matching, obtain the set of images that meet the threshold conditions between each image and the rest of the images, and then use the union-find method to merge the sets with common images in different image sets together to obtain a complete image set of different cracks;

[0012] Step 2: Crack image splicing

[0013] Iteratively splice the complete image sets of different cracks in sequence. Assume that any two images to be spliced are the source image and the target image respectively, and then splice the two images;

[0014] Step 3: Crack image fusion:

[0015] Solve the problem of unnatural transition between the overlapping area and the surrounding area in the spliced image through the linear weighted fusion method;

[0016] Step 4: Crack damage fusion:

[0017] During the actual detection and visualization of cracks, for the cracks in the overlapping area of the source image and the target image that are not connected together, perform fusion and reconnection processing on this part of the cracks in the overlapping area in the spliced image.

[0018] The specific steps of the splicing in Step 2 are as follows:

[0019] (1) Read the source image and the target image in the set of images of the same crack;

[0020] (2) Calculate the global homography matrix of the source image and the target image;

[0021] (3) Calculate the splicing canvas size of the source image and the target image;

[0022] (4) Divide the canvas into different grids and calculate the local homography matrix of the vertices of each grid;

[0023] (5) Image transformation, transform the source image into the canvas image with the target image as the reference system according to the local homography matrix.

[0024] The specific method of step (iii) is as follows:

[0025] (1) Calculate the distance transformation maps img1 and img2 of the source image and the target image mapped onto the canvas respectively;

[0026] (2) Calculate the fusion coefficient α of the source image in the canvas, α = img1 / (img1 + img2), and the fusion coefficient of the target image is 1 - α.

[0027] (3) Multiply the source image img1 and the target image img2 by their respective fusion coefficients for weighted fusion.

[0028] The specific method of step (iv) is as follows:

[0029] (1) Obtain the vertex coordinates of each crack in the source image and the vertex coordinates of the corresponding bounding box in the stitching canvas;

[0030] (2) Obtain the vertex coordinates of each crack in the target image and the corresponding bounding box coordinates in the stitching canvas;

[0031] (3) In the stitching canvas, if the bounding boxes of the cracks in the source image and the cracks in the target image overlap, merge these two cracks and the corresponding bounding boxes. After the merging is completed, delete these two cracks from the crack sets of the source image and the target image respectively;

[0032] Otherwise, starting from the first crack in the source image, calculate the distance between the lower vertex coordinate of the current crack and the upper vertex coordinates of each crack in the target image, and find the crack with the smallest distance that meets the distance threshold as the crack to be connected. If the target crack is found, connect the lower vertex coordinate of the current crack and the upper vertex coordinate of the target crack together with a solid red line. If the target crack is not found, select the second crack in the source image and continue the above process until all the cracks in the source image are traversed.

[0033] In step one, when taking images, take them in a "one - shaped" or "Z - shaped" pattern. There is an overlapping area between adjacent pictures of the same crack, and there is no overlapping area between different cracks. Extract the similar features between the overlapping areas of the images to identify and match multiple pictures belonging to the same crack.

[0034] In Step 2, if there are inconsistent visualization results in the overlapping areas of different images of the same crack, after splicing is completed, first calculate the minimum value ymin of the vertical coordinates of all crack vertices in the target image with respect to the canvas reference system based on the offset of the target image on the canvas and the vertex coordinate information of all cracks in the target image. Then, during the process of mapping the pixels in the source image and the target image to the canvas, only the crack pixel part above ymin in the source image is transformed to the canvas, and the cracks below ymin in the target image are spliced to the canvas.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention is divided into crack matching, splicing, and fusion according to functions, and each function is modular, which is convenient for optimization, update, and use.

[0037] 2. The present invention has strong adaptability and robustness, and can splice crack images on the surface of concrete structures collected in different ways.

[0038] 3. The visualization effect of the images of the present invention is better. Image fusion makes the overlapping areas in the spliced images transition more naturally and beautifully. Damage fusion can not only observe the complete damage situation of cracks in the spliced images, but also count data such as the number, length, and width of cracks.

[0039] 4. The present invention has high splicing efficiency, can achieve simultaneous splicing of multiple crack data sets, has high accuracy in image splicing alignment, and uses the local homography matrix for splicing, which not only optimizes the error generated by the global homography matrix, but also reduces the ghosting phenomenon. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is the flowchart of the method of the present invention;

[0041] Figure 2 is the flowchart of crack image splicing;

[0042] Figure 3 is an example diagram of image matching;

[0043] Figure 4 is the splicing and fusion result diagram of crack images;

[0044] Figure 5 is the fusion result diagram of crack images. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] This embodiment provides a method for automatically splicing and fusing concrete cracks, including the following steps:

[0047] Step 1: Crack image matching

[0048] Take images of a certain component of the bridge in the shape of "one - character" or "Z - shape". Among them, 9 images with cracks are used as the dataset of this example. As Figure 3 shown, where I 1 、I 2 、I 3 ,I 4 、I 5 、I 6 ,I 7 、I 8 、I 9 respectively belong to three different crack sets. Each crack set represents a complete crack. The index of each image can be seen in the white label at its bottom;

[0049] First, select I 1 as the source image. Calculate the confidence C between I 1 and I 2 through feature point matching and screening. If C > C th (C th is the confidence threshold. In this invention, C th takes the value of 1.5), then it is considered that these two images are correctly matched images. Save the indexes of these two images into an array. At the same time, delete the indexes of these images from the image dataset. And so on, continue to make I 1 match with I 3 、I 4 …I 9 ,I 2 match with I 3 …I 9 ,I 3 match with I 4 …I 9 ,…,I 9 …I 10 perform image matching. After executing this process, the matching result set of each image is obtained. The matching results for the three different cracks are [I 1 ,I 2 、[I 2 ,I 3 、[I 3 、[I 4 ,I 5 、[I 5 ,I 6 、[I 6 、[I 7 ,I 8 、[I 8,I 9 , [I 9 . [I 1 ,I 4 represents I 1 ,I 4 are a set of mutually matching images, and the same applies to the rest;

[0050] Then traverse the set of matching results corresponding to each image, and recursively query whether the elements in this set and the elements in the set merged with this set exist in the remaining sets of matching results. If they exist, merge these two sets; otherwise, do not merge. Thus, three complete sets of crack images are obtained. [I 1 ,I 2 ,I 3 , [I 4 ,I 5 ,I 6 , [I 7 ,I 8 ,I 9 ;

[0051] Step 2: Read the set of crack images

[0052] For the above image matching results, first read the set of crack images [I 1 ,I 2 ,I 3 . Concatenate and fuse image I 1 with I 2 , and denote the result as I 12 ; then concatenate and fuse I 12 with I 3 , and the result is I 13 ; thus, the concatenation and fusion of multiple images in this crack set are completed. Then, perform iterative concatenation and fusion on another set of crack images.

[0053] The concatenation and fusion of multiple images are carried out on the basis of the concatenation and fusion of two images. This embodiment specifically describes the process of concatenating and fusing two images.

[0054] Step 3: Crack image concatenation

[0055] Suppose the two images to be concatenated are the source image and the target image respectively. The steps for concatenating the two images are as Figure 2 shown.

[0056] (1) Read the source image and the target image in the set of images of the same crack.

[0057] (2) Calculate the global homography matrix of the source image and the target image.

[0058] Let x = [x, y] Tis the homogeneous coordinate of the feature point on the source image, x' = [x′, y′] T is the homogeneous coordinate of the corresponding feature point on the target image, then there is:

[0059]

[0060] In Equation 1, H is the global homography matrix for the transformation between the source image and the target image. Equation 1 is transformed into matrix form as shown in Equation 2

[0061]

[0062] Let h 33 = 1, then the global homography matrix H contains 8 unknowns after being converted to homogeneous coordinates. Solving for H requires at least 4 sets of non - collinear matching feature point pairs. Suppose there are N sets of matching feature point pairs between the source image and the target image, then 2N linear equations can be established. Let A represent the coefficient matrix and h represent the homography matrix to be solved, there is:

[0063] A 2N×9 h 9×1 = 0 2N×1 (Equation 3)

[0064] Ideally, a i h = 0, solve for the minimum value of Equation 3, that is, Equation 4, and solve the global homography matrix through the singular value decomposition algorithm

[0065]

[0066] (3) Calculate the size of the stitched canvas according to the global homography matrix.

[0067] Suppose A, B, C, D are the four vertices of the source image. By obtaining the global homography matrix map the four vertices A, B, C, D of the source image to the positions of A', B', C', D' in the coordinate system with the target image as the reference. Then, according to the size relationship of the coordinates of each vertex before and after mapping, the maximum value max_x of the abscissa, the minimum value min_x of the abscissa, the maximum value max_y of the ordinate, and the minimum value min_y of the ordinate in the coordinate system can be obtained. Further calculate that the width w of the stitched canvas = max_x - min_x, and the height h = max_y - min_y;

[0068] (4) Divide the stitched canvas into different grids and calculate the local homography matrix of each grid vertex.

[0069] Traverse each vertex in the grid and calculate the weight and local homography matrix of each grid vertex according to Equation 5 and Equation 6

[0070] w* = exp(-||x* - x|| 2 / σ 2 ) (Equation 5)

[0071]

[0072] In (Equation 5), w * is the weight matrix corresponding to the grid vertices, with a matrix size of 2N×1; x is the matrix composed of the coordinates of the feature points in the source image, with a matrix size of N×2; x * is the matrix composed of the grid vertex coordinates, with the same matrix size as x; σ is the scale parameter, which is a constant;

[0073] In (Equation 6), h ＊ is the homography matrix corresponding to the grid vertices, A is the coefficient matrix, h is the global homography matrix, and N is the number of feature point pairs in the source image;

[0074] (5) Image transformation

[0075] First, transform the source image into the canvas image with the target image as the reference system according to the local homography matrix. Let the local homography matrix of the grid vertices be H * , the pixel point of the source image is X * , X * The corresponding pixel point in the target image is, then there is X′ *

[0076] X′ * ~H * X * (Equation 7)

[0077] Multiply both sides of Equation 7 by on the left to get

[0078]

[0079] Traverse each pixel point in the canvas in a loop. According to the grid vertex to which the current pixel point belongs, map the current pixel point through Equation 8. If the mapped pixel point is in the source image area, first judge whether the HSV value of this pixel is the crack color; if it is the crack color, then judge whether the ordinate y of this pixel mapped to the canvas satisfies 0≤y≤ymin, where ymin is the minimum value of the ordinates of all crack vertices in the target image. If the condition is met, assign this pixel in the source image to the current pixel point, otherwise do not process. If it is not a crack color pixel, directly assign this pixel in the source image to the current pixel point. If the mapped pixel point is not in the source image area, no pixel transformation is performed. Until the traversal ends, the transformation of the source image to the canvas with the target image as the reference system is realized.

[0080] Finally, transform the target image to the corresponding position on the canvas, thus completing the stitching of the two images.

[0081] In this step, the present invention stitches the visualized crack images. Since there may be problems with inconsistent visualization results in the overlapping areas of different pictures of the same crack, after stitching, there may be a problem of repeated display of the same crack in the overlapping area of the stitching canvas.

[0082] The method of the present invention to solve this problem is as follows: First, according to the offset of the target image on the canvas and the vertex coordinate information of all cracks in the target image, calculate the minimum value ymin of the vertical coordinates of all crack vertices in the target image with the canvas as the reference system. Secondly, in the process of mapping the pixels in the source image and the target image to the canvas, convert the RGB value of the current pixel to the HSV value, and determine whether the HSV value of the current pixel is the crack color; if it is a pixel of the crack color, further determine whether the vertical coordinate y of this color pixel mapped to the canvas satisfies 0 ≤ y ≤ ymin. If the condition is met, map the color pixel to the canvas, otherwise do not perform the transformation. If it is not a pixel of this color, directly map it to the canvas. In this way, the crack pixel part above ymin in the source image is stitched to the canvas, and all cracks below ymin in the target image are stitched to the canvas, thus solving the above problem.

[0083] Step Four: Crack Image Fusion

[0084] (1) Calculate the distance transformation graphs image1 and image2 of the source image and the target image in the stitching canvas.

[0085] (2) Calculate the fusion coefficient α of the source image and the fusion coefficient 1 - α of the target image in the stitching canvas, where α = image1 / (image1 + image2).

[0086] (3) Linear weighted fusion

[0087] Traverse each pixel point in the source image and the target image. If the current pixel point is of the crack color, no fusion is performed; if it is not of the crack color, multiply the source image and the target image by their respective fusion coefficients for weighted fusion, and the fusion result is result = image1 × α + image2 × (1 - α).

[0088] This image fusion method performs weighted fusion on the source image and the target image according to different fusion coefficients, making the pixels in the overlapping area of the source image in the canvas gradually transition to the overlapping area of the target image, so that the pixels in the overlapping area change more naturally with the surrounding pixels.

[0089] It should be noted that when performing image fusion, the present invention only fuses the pixels of non-crack colors in the source image and the target image, and keeps the original pixel values for the crack colors. In this way, after image fusion, the crack color in the canvas can be kept consistent with the original crack visualization result.

[0090] Step Five: Crack Damage Fusion

[0091] In the actual detection and visualization of cracks, there may be situations such as false crack detection or incomplete crack recognition. These situations may cause the cracks in the overlapping area of the source image and the target image in the stitched image not to be connected together. At this time, we need to perform fusion and reconnection processing on these cracks in the overlapping area of the stitched image. The specific process is as follows:

[0092] (1) Obtain the vertex coordinates of each crack damage in the source image and the vertex coordinates of the corresponding bounding box in the stitched canvas.

[0093] (2) Obtain the vertex coordinates of each crack damage in the target image and the corresponding bounding box coordinates in the stitched canvas.

[0094] (3) In the stitched canvas, initialize and select the lower vertex coordinates of the first crack in the source image, traverse the cracks in the target image, and determine whether the upper vertex coordinates of the cracks in the target image are within the bounding box of the first crack in the source image. In the target image, if there is such a crack, it means that these two cracks belong to the same crack, then merge these two cracks and the corresponding bounding boxes. After the merge is completed, delete these two cracks from the crack sets of the source image and the target image respectively.

[0095] In the target image, if there is no such crack, then calculate the distance between the lower vertex coordinates of the first crack in the source image and the upper vertex coordinates of each crack in the target image, and find the crack with the smallest distance that meets the distance threshold as the crack to be connected. If the target crack is found, connect the lower vertex coordinates of the current crack and the upper vertex coordinates of the target crack together with a red solid line. If the target crack is not found, select the second crack in the source image and continue the above process until all the cracks in the source image are traversed.

[0096] For the crack image set [I 1 、I 2 、I 3 , its stitched fusion result is as Figure 4 shown, and the crack fusion result is as Figure 5 shown.

[0097] The image fusion method adopted by the present invention is to perform weighted fusion on the source image and the target image according to different fusion coefficients, so that the pixels in the overlapping area of the source image in the canvas gradually transition to the overlapping area of the target image, making the change of pixels in the overlapping area more natural with the pixels in the surrounding area.

[0098] When performing image fusion, the present invention only fuses the pixels of non-crack colors in the source image and the target image, and keeps the original pixel values for the crack colors. In this way, after image fusion, the crack colors in the canvas can be kept consistent with the original crack visualization results.

[0099] The present invention proposes a method for concrete crack matching, splicing and fusion. By using image matching and union-find algorithm, different crack image sets are identified from the visualized crack data set, and then the pictures in different crack sets are iteratively spliced and fused in sequence, enabling fully automatic and accurate splicing and fusion of the detected and identified crack data set, so that we can more intuitively observe the complete situation of different cracks and crack statistical data.

[0100] The content of the present invention is not limited to the examples listed. Any equivalent transformation of the technical solution of the present invention adopted by those of ordinary skill in the art by reading the specification of the present invention is covered by the claims of the present invention.

Claims

1. An automatic splicing and fusion method for concrete cracks, characterized in that: It includes the following steps: Step 1: Crack image matching Take pictures to obtain a set of bridge structure crack images, calculate the confidence of each image in the dataset with the rest of the images through feature point matching, obtain the set of images that meet the threshold conditions between each image and the rest of the images, and then use the union-find method to merge the sets with common images in different image sets together to obtain a complete image set of different cracks; Step 2: Crack image splicing Iteratively splice the complete image sets of different cracks in sequence. Assume that any two images to be spliced are the source image and the target image respectively, and then splice the two images; Step 3: Crack image fusion: Solve the problem of unnatural transition between the overlapping area and the surrounding area in the spliced image through the linear weighted fusion method; Step 4: Crack damage fusion: During the actual detection and visualization of cracks, for the cracks in the overlapping area of the source image and the target image that are not connected together, perform fusion and reconnection processing on this part of the cracks in the overlapping area in the spliced image; The specific steps of the splicing in Step 2 are: (1) Read the source image and the target image in the set of images of the same crack; (2) Calculate the global homography matrix of the source image and the target image; (3) Calculate the splicing canvas size of the source image and the target image; (4) Divide the canvas into different grids and calculate the local homography matrix of each grid vertex; (5) Image transformation, transform the source image into the canvas image with the target image as the reference system according to the local homography matrix; The specific method of Step 3 is: (1) Calculate the distance transformation maps img1 and img2 of the source image and the target image mapped into the canvas respectively; (2) Calculate the fusion coefficient α of the source image in the canvas, α = img1 / (img1 + img2), and the fusion coefficient of the target image is 1 - α; (3) Multiply the source image img1 and the target image img2 by their respective fusion coefficients for weighted fusion; The specific method of Step 4 is: (1) Obtain the vertex coordinates of each crack in the source image and the vertex coordinates of the corresponding bounding box in the splicing canvas; (2) Obtain the vertex coordinates of each crack in the target image and the corresponding bounding box coordinates in the splicing canvas; (3) In the splicing canvas, if the bounding boxes of the cracks in the source image and the cracks in the target image overlap, merge these two cracks and the corresponding bounding boxes. After the merger, delete these two cracks from the crack sets of the source image and the target image respectively; Otherwise, starting from the first crack in the source image, calculate the distance between the lower vertex coordinate of the current crack and the upper vertex coordinates of each crack in the target image, and find the crack with the smallest distance that meets the distance threshold as the crack to be connected. If the target crack is found, connect the lower vertex coordinate of the current crack and the upper vertex coordinate of the target crack together with a red solid line. If the target crack is not found, select the second crack in the source image and continue the above process until all the cracks in the source image are traversed.

2. An automatic splicing and fusion method for concrete cracks according to claim 1, It is characterized in that: In step one, when taking images, take them in a "one - shaped" or "Z - shaped" pattern. There is an overlapping area between adjacent pictures of the same crack, and there is no overlapping area between different cracks. Extract the similar features between the overlapping areas of the images to identify and match multiple pictures belonging to the same crack.

3. A method for automatic splicing and fusion of concrete cracks according to claim 1 or 2, It is characterized in that: In step two, if there is a situation where the visualization results are inconsistent in the overlapping area of different pictures of the same crack, after splicing is completed, first, according to the offset of the target image on the canvas and the vertex coordinate information of all cracks in the target image, calculate the minimum value ymin of the vertical coordinates of all crack vertices in the target image with the canvas as the reference system. Then, in the process of mapping the pixels in the source image and the target image to the canvas, only transform the crack pixel part above ymin in the source image to the canvas, and splice the crack below ymin in the target image to the canvas.

Images