A Video-Based Automatic Detection Method for Multiple-Image Cracks

Abstract

The present invention discloses a multi-image crack automatic detection method based on video. First, an initial path for video shooting is formulated; it is determined whether there are cracks in each frame of the video image. If there are cracks, the cracks are identified to the pixel level, and the initial crack image is determined; the crack pixels in the initial crack image are clustered to determine the cracks to be detected; based on the cracks to be detected, a PCA algorithm model based on spatial position information is established, and the vector corresponding to the maximum eigenvalue is selected as the overall trend of the cracks to be detected, and the shooting path is modified in real time; and by using the image recognition method of the video inter-frame, continuous cracks to be detected are obtained through multiple spatial clustering and PCA models. Finally, image matching and stitching fusion are realized by using the image recognition results. The present invention automatically tracks and shoots cracks by using the trend characteristics of the cracks, and then uses deep learning and image processing technologies to extract the overall information of the cracks, improving the automatic detection efficiency of cracks in the case of multiple images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and particularly relates to a method for automatically detecting multi-image cracks based on video. Background Art

[0002] There are many crack image recognition technologies, including traditional image processing methods and deep learning technologies represented by convolutional neural networks that have developed rapidly in recent years. When the crack is within one image, the existing methods can better recognize the crack and extract the information of the crack. However, when the crack is long enough and cannot be completely and clearly included in one captured image, a multi-image fusion processing method needs to be adopted. Currently, this method uses manual shooting of video images and judging the crack direction or automatic shooting of a large range of video images, and then performs image registration, splicing, and fusion, and finally forms a large image containing the entire crack to achieve the overall detection of the crack. However, the first method above is too dependent on manual work, and the second method has problems such as low image fusion efficiency due to excessive captured images and slow image loading and analysis speed due to the large size of the large image. Therefore, replacing manual work with an automatic method for judging the overall direction of the crack can effectively improve the detection efficiency of the crack and achieve the goal of being fast and practical. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for automatically detecting multi-image cracks based on video. This method uses the trend characteristics of the cracks to automatically track and shoot the cracks, and then adopts deep learning and image processing technologies to extract the overall information of the cracks, thereby improving the automatic detection efficiency in the case of cracks in multiple images.

[0004] To achieve the above technical purpose, the present invention adopts the following technical solutions:

[0005] A method for automatically detecting multi-image cracks based on video, comprising the following steps:

[0006] S01: Determine the initial path for video shooting and shoot the video along the initial path;

[0007] S02: Obtain each frame of the video image and use an image recognition method to judge whether there is a crack. If there is a crack, identify the crack to the pixel level, and use the video image initially identified as a crack as the initial crack image;

[0008] S03: According to the positions of the crack pixels in the initial crack image, perform spatial clustering, and select a type of crack whose length is significantly greater than other categories and extends to the image edge as the crack to be detected; if there are no such features, that is, there is no type of crack whose length is significantly greater than other categories and extends to the image edge, it is regarded as a single-image crack and no multi-image detection is required, and continue to execute step S02 along the original shooting path;

[0009] S04: Based on the cracks to be detected in the current crack image, establish a principal component analysis (PCA) algorithm model based on spatial position information. Use the vector corresponding to the maximum eigenvalue of the model as the overall trend of the cracks to be detected, and modify the shooting path in real time to follow the overall trend of the cracks to be detected, and continue to shoot the video along the modified shooting path;

[0010] S05: Obtain the video images at a specified interval frame away from the current crack image, and use image recognition methods to determine whether there are cracks: If there are no cracks, the crack detection is completed; If there are cracks, identify the cracks to the pixel level and continue with step S06;

[0011] S06: Perform spatial clustering on the positions of the pixel cracks in the images at the specified interval frame, and select the category that contains the cracks to be detected in the previous interval frame as the new continuous cracks to be detected. If the above category is not included, the crack detection is completed;

[0012] S07: Loop through steps S04 to S06 until the crack detection is completed;

[0013] S08: Register and splice and fuse multiple crack images including the cracks to be detected, and finally save the cracks to be detected as detected cracks in the multi-image crack database.

[0014] Furthermore, use the convolutional neural network algorithm (CNN) to classify the images for the presence or absence of cracks.

[0015] Furthermore, use the fully convolutional neural network algorithm (FCN) to perform pixel-level classification and recognition on the crack images.

[0016] Furthermore, use the DBSCAN algorithm to perform spatial clustering processing on the crack pixels.

[0017] Furthermore, use the long side of the minimum bounding rectangle of all the pixels of the crack as the length of the crack.

[0018] Furthermore, use the PCA algorithm to judge the overall trend of the crack. Specifically:

[0019] For the spatial column vector p of the spatial points of the crack pixels i , the corresponding covariance matrix C is as follows:

[0020]

[0021] C·v k =λ k ·v k ,k=1,2

[0022] Where: represents the average value of the spatial column vectors of all the pixel points of the crack, n represents the number of pixel points included in the crack; λk is the k-th eigenvalue of the covariance matrix, and v k is the k-th eigenvector; the eigenvector corresponding to the largest eigenvalue is taken as the overall trend of the crack.

[0023] Furthermore, in step S08, the SURF algorithm is used to extract feature points from the crack image. The Euclidean distance between the feature points is used as the similarity measure for rough matching of the feature points, and the RANSAC algorithm is used to eliminate the mismatched point pairs. Then, the homography matrix is solved using the correct matching point pairs, and the solved homography matrix is used to splice and fuse the images.

[0024] Furthermore, the method of eliminating the mismatched point pairs by the RANSAC algorithm and solving the homography matrix using the correct matching point pairs includes the following steps:

[0025] (1) Select k point pairs from the feature point pairs to establish a system of equations and find the homography matrix H;

[0026] (2) Calculate the Euclidean distance t between all the matched points after being transformed by the matrix H and their corresponding matched points;

[0027] (3) If the Euclidean distance t is less than the set threshold, then the matched point is taken as an inlier, otherwise it is an outlier;

[0028] (4) Count the number of inliers under the homography matrix H;

[0029] (5) Repeat the above steps (1) to (4) for iteration until the iteration stop condition is met. Then, select the set of points with the largest number of inliers as the inliers, and use these inliers to calculate the optimal homography matrix H by the least squares method;

[0030] The iteration stop condition includes: after a crack image is transformed by the homography matrix, the intersection with the corresponding another crack image in space reaches the maximum.

[0031] The maximum intersection of the crack pixels is the registration target. Therefore, in this scheme, by adding the iteration stop condition, the calculation time can be saved; due to the unobvious feature points of the weak texture, a large number of matching errors are likely to occur. By setting a clear registration target, the risk of image matching errors can be reduced, thereby improving the matching accuracy under weak texture conditions.

[0032] Furthermore, when splicing and fusing the crack recognition images, when the frequency of cracks at the same fusion position reaches more than half, the pixel value of the pixel point is marked as a crack according to the given value, otherwise the pixel value takes the average value of the non-crack pixels.

[0033] Beneficial effects

[0034] The present invention first formulates an initial path for video shooting; determines whether there are cracks in each frame of the video image. If cracks exist, it identifies the cracks to the pixel level and determines the initial crack image; clusters the crack pixels of the initial crack image to determine the cracks to be detected; establishes a principal component analysis (PCA) algorithm model based on spatial position information according to the cracks to be detected, selects the vector corresponding to the maximum eigenvalue as the overall trend of the cracks to be detected, and modifies the shooting path in real time, thereby realizing the automatic planning of the video shooting path. At the same time, the present invention uses the image recognition method of video inter-frame, obtains continuous cracks to be detected by adopting multiple spatial clusterings and PCA models, and finally realizes image matching and stitching fusion by using the image recognition result, realizing the overall fast and automatic detection of multi-image cracks. The method of the present invention is particularly applicable to the scenarios such as detecting main long cracks and weak textures in multi-image scenarios at close range. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0036] Figure 1 is a schematic flowchart of the multi-image crack image recognition method of the present invention;

[0037] Figure 2 is a field detection result diagram of the multi-image crack image recognition method of the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] The present invention provides a video-based multi-image crack automatic detection method, which is used to quickly realize the overall recognition of multi-image crack images. In the embodiments of the present invention, the concrete crack image recognition is taken as an example for illustration. It should be understood that the present invention is not limited to the concrete application field.

[0039] The following will further illustrate the present invention in combination with embodiments.

[0040] A video-based multi-image crack automatic detection method provided in this embodiment includes:

[0041] S01: Formulate an initial path for video shooting and shoot videos along the initial path;

[0042] Wherein: The video shooting path can be formulated manually or an automatic shooting path can be set, such as from left to right, from top to bottom, etc. In this embodiment, the form of shooting path from left to right and from top to bottom is adopted. Video shooting needs to be mounted on an unmanned aerial vehicle or robot device with positioning ability. In this embodiment, an unmanned aerial vehicle is used for shooting.

[0043] S02: Obtain each frame of video image and use image recognition method to judge whether there is a crack. If there is a crack, identify the crack to the pixel level and use the video image initially identified as a crack as the initial crack image.

[0044] Among them: Use the Convolutional Neural Network algorithm CNN to classify the image for the presence or absence of cracks. For images with low resolution, the whole picture recognition method can be used. For high-resolution images, the recognition can be carried out by means of scaling or local cropping. The pixels of the image in this embodiment are 960×544, so the whole picture recognition method is adopted. At the same time, for images with cracks, the Fully Convolutional Neural Network algorithm FCN is used in this embodiment to classify the crack image to the pixel level.

[0045] S03: According to the position of the crack pixels in the initial crack image, perform spatial clustering, and select the crack with a length significantly greater than other categories and extending to the edge of the image as the crack to be detected. If there are no such features, that is, there is no category of cracks with a length significantly greater than other categories and extending to the edge of the image, it is regarded as a single-image crack and no multi-image detection is required, and continue to execute step S02 along the original shooting path.

[0046] Among them: In this embodiment, the DBSCAN algorithm is used for the spatial clustering algorithm, and the minimum number of neighborhood points for a given point to become a core object in the DBSCAN algorithm is taken as 2, and the neighborhood radius is taken as 5. In this embodiment, considering the spatial continuity characteristics of the cracks, the DBSCAN spatial clustering method is preferably used. It should be understood that in other feasible embodiments, other parameters can be set or other spatial clustering algorithms can be selected. In this embodiment, the long side of the minimum circumscribed rectangle of all pixels of the crack is used as the length of the crack, and at the same time, a threshold for judging whether the crack extends to the edge is set. When the minimum value of the crack pixel position from the edge of the image is less than the threshold, it is considered a crack that requires multi-image detection.

[0047] S04: Establish a Principal Component Analysis PCA algorithm model based on spatial position information according to the crack to be detected. The vector corresponding to the maximum eigenvalue of the model is the overall trend of the crack to be detected. Specify the shooting path of the subsequent video to be along the overall trend of the crack to be detected, and continue to shoot the video along the current shooting path.

[0048] Among them: In the process of the PCA algorithm in this embodiment, that is, for the spatial p of the spatial points of the crack pixels i , the corresponding covariance matrix C is as follows:

[0049]

[0050] C·v k =λ k ·v k ,k=1,2

[0051] Wherein: represents the average value of the spatial positions of all crack pixels, and λ k is the k-th eigenvalue of the covariance matrix, and v k is the k-th eigenvector. The overall trend of the final crack is the eigenvector corresponding to the largest eigenvalue.

[0052] S05: Obtain a video image that is a specified number of frames apart from the current crack image, and use an image recognition method to determine whether there are cracks. If there are no cracks, the crack detection is completed. If there are cracks, identify the cracks to the pixel level and continue with step S06;

[0053] Wherein: In this embodiment, the video frame rate is 30 frames per second. Since the time interval between video image frames is too short, if the next frame of the image is directly obtained, a large number of overlapping regions are likely to be generated. At the same time, considering the time-consuming cost of image analysis, therefore, this video obtains images in the form of an interval of 50 frames, which can ensure that there is no excessive overlap and can also ensure the efficiency of image fusion.

[0054] S06: Perform spatial clustering on the positions of the crack pixels in the images of the specified interval frames, and select the category that contains the crack to be detected in the previous interval frame as the new continuous crack to be detected. If the above category is not included, the crack detection is completed.

[0055] Wherein: In this embodiment, the DBSCAN algorithm is used for the spatial clustering algorithm. The minimum number of neighborhood points for a given point to become a core object in the DBSCAN algorithm is taken as 2, and the neighborhood radius is taken as 5. When determining whether the image contains the category of the crack to be detected in the previous interval frame, it is necessary to locate the crack annotation of the previous interval frame in the current image.

[0056] S07: Loop and execute steps S04 to S06 until the crack detection is completed;

[0057] S08: Perform registration and stitching fusion on multiple crack images including the crack to be detected, and finally save the crack to be detected as a detected crack in the multi-image crack database.

[0058] This embodiment uses feature point extraction based on the SURF algorithm, uses the Euclidean distance as the similarity measure for rough matching of feature points, and eliminates the mismatched point pairs through the RANSAC algorithm, and finally realizes image registration.

[0059] Among them: feature extraction based on SURF algorithm and image registration of RANSAC random sampling consensus algorithm are adopted. In this embodiment, the threshold of Hessian matrix is ​​set to 200, and the algorithm directly calls OpenCV software. For the results of two images identified as cracks, an iteration condition of RANSAC algorithm is added, that is, after one of the pixels marked as cracks is transformed by homography matrix, the iteration stops when the intersection of the pixels marked as cracks in the other corresponding image is the largest in space. This iteration condition can speed up the iteration speed of the algorithm and improve the matching accuracy, especially under weak texture conditions. The preservation of crack database can provide judgment basis for subsequent crack detection.

[0060] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A video-based automatic multi-image crack detection method, characterized in that: It includes the following steps: S01: Define the initial path for video shooting and shoot videos along the initial path; S02: Obtain each frame of video image and use image recognition method to judge whether there are cracks. If there are cracks, identify the cracks to the pixel level, and use the video image initially identified as cracks as the initial crack image; S03: According to the positions of the crack pixels in the initial crack image, perform spatial clustering, and select a type of crack whose length is significantly greater than other categories and extends to the image edge as the crack to be detected; if there are no such features, that is, there is no type of crack whose length is significantly greater than other categories and extends to the image edge, it is regarded as a single-image crack and no multi-image detection is required, and continue to execute step S02 along the original shooting path; Use the DBSCAN algorithm to perform spatial clustering processing on the crack pixels; Use the long side of the minimum circumscribed rectangle of all pixels of the crack as the length of the crack; S04: According to the crack to be detected in the current crack image, establish a principal component analysis PCA algorithm model based on spatial position information, use the vector corresponding to the maximum eigenvalue of the model as the overall trend of the crack to be detected, and modify the shooting path in real time to follow the overall trend of the crack to be detected, and continue to shoot videos along the modified shooting path; Use the PCA algorithm to judge the overall trend of the crack, specifically: For the spatial column vector p of the crack pixel spatial points i , the corresponding covariance matrix C is as follows: C·v k = λ k ·v k , k = 1, 2 Wherein: represents the average value of the spatial column vectors of all pixel points of the crack, and n represents the number of pixel points included in the crack; λ k is the k-th eigenvalue of the covariance matrix, and v k is the k-th eigenvector; the eigenvector corresponding to the largest eigenvalue is used as the overall trend of the crack; S05: Obtain the video image at a specified interval frame away from the current crack image, and use the image recognition method to judge whether there are cracks: if there are no cracks, the crack detection is completed; if there are cracks, identify the cracks to the pixel level, and continue to step S06; S06: Perform spatial clustering on the positions of the crack pixels in the image at the specified interval frame, and select the category that contains the crack to be detected in the previous interval frame as the new continuous crack to be detected. If the above category is not included, the crack detection is completed; S07: Loop and execute steps S04 to S06 until the crack detection is completed; S08: Register and splice and fuse multiple crack images including the crack to be detected, and finally save the crack to be detected as a detected crack in the multi-image crack database; In step S08, use the SURF algorithm to extract feature points from the crack images, use the Euclidean distance between the feature points as the similarity measure for rough matching of feature points, and use the RANSAC algorithm to eliminate the mismatched point pairs, and then use the correct matching point pairs to solve the homography matrix, and use the solved homography matrix to splice and fuse the images.

2. The method according to claim 1, characterized in that: Use the convolutional neural network algorithm CNN to classify whether there are cracks in the image.

3. The method according to claim 1, characterized in that: Use the fully convolutional neural network algorithm FCN to perform pixel-level classification and recognition on the crack images.

4. The method according to claim 1, characterized in that: The method of eliminating the mismatched point pairs by the RANSAC algorithm and using the correct matching point pairs to solve the homography matrix includes the following steps: (1) Select k point pairs from the feature point pairs to establish an equation system and solve the homography matrix H; (2) Calculate the Euclidean distance t between all the matched points after being transformed by the matrix H and their corresponding matched points; (3) If the Euclidean distance t is less than the set threshold, then this matched point is regarded as an inlier, otherwise it is an outlier; (4) Count the number of inliers under the homography matrix H; (5) Repeat the above steps (1) to (4) for iteration until the iteration stop condition is met. Then select the set of points with the largest number of inliers as the inliers, and use these inliers to calculate the optimal homography matrix H by the least squares method; The iteration stop condition includes: after a crack image is transformed by the homography matrix, the intersection with the corresponding another crack image in space reaches the maximum.

5. According to the method described in claim 1, it is characterized in that: When stitching and fusing the crack recognition images, if the frequency of cracks at the same fusion position reaches more than half, then the pixel value of the pixel point is marked as a crack according to the given value, otherwise the pixel value takes the mean value of the non-crack pixels.

Images