Method and system for efficient dataset collection for concrete crack detection with spatial-adaptive data augmentation

Abstract

An efficient data set collection method and system for detecting concrete cracks through spatial adaptive data augmentation are presented. The efficient data set collection method for detecting concrete cracks through spatial adaptive data augmentation proposed in the present invention comprises the steps of: extracting skeletal data through a skeletal data extraction unit to preserve connectivity data of crack line segments in a crack image; extracting a corner, which is the intersection of two edges having a slope change greater than a predetermined standard, from the extracted skeletal data through a corner extraction unit; adjusting the weights of the standard deviation and deformation strength for the extracted skeletal data and the corner through a crack direction learning unit, and performing feature learning using an artificial neural network for direction-based elastic distortion to model the crack image based on the applied weights; and analyzing the shape of the crack and measuring its thickness through a crack thickness learning unit to perform thickness learning for the crack, and integrating the learning result for the direction-based elastic distortion and the thickness learning result for the crack.

Description

Technology Field

[0001] The present invention relates to an efficient data set collection method and system for concrete crack detection through spatially adaptive data augmentation. Background Technology

[0002] Technologies have been proposed from a computer science perspective to provide means to prevent frequently occurring building collapse accidents. When workers search for concrete cracks on-site, there is a risk of exposure to hazardous situations, and the process is time-consuming and costly. Furthermore, the accuracy of crack detection is limited by conditions such as line of sight and weather during the work.

[0003] One of the most common problems that arises when using concrete, the most widely used material for building construction, is cracking. As buildings age, the number and size of cracks increase, which adversely affects the safety and durability of the structure; therefore, it is important to regularly detect cracks in concrete structures and devise maintenance measures accordingly. Generally, structural cracks are detected by experts equipped with specialized equipment inspecting the site directly; however, this method requires significant labor and time, and in sites with large-scale apartment complexes, the accuracy of crack detection decreases, which can ultimately pose a threat to people's safety.

[0004] Many studies utilize image processing techniques to efficiently and reliably detect cracks and overcome the limitations of manual detection. In the field of computer vision, various studies—including image thresholding, edge detection, wavelet transforms, and machine learning—have been conducted for the automatic detection of cracks in images. Image thresholding methods classify cracks at the pixel level based on various pixel values, facilitating image simplification and post-processing. Edge detection methods efficiently detect cracks by applying differential operators to images. The fundamental concept of wavelet transforms is that it functions like the image signal, and it detects cracks using wavelet functions or a set of basis functions based on this signal. Machine learning extracts feature vectors of cracks from training data and infers results through learning. Although these methods exist, active research is underway to improve crack non-uniformity, surface texture diversity, and background complexity to enable efficient crack detection.

[0005] Deep learning techniques can learn deep non-linear network structures and model abstract representations. In the prior art, a deep learning-based crack detection technique was proposed as one of the early studies applying CNNs to road crack detection. Furthermore, the effectiveness of the deep learning technique was demonstrated by comparing it with existing machine learning classifiers such as Support Vector Machines (SVM) and boost methods. Another prior art investigated the influence of CNN depth and positional changes between training and test data on the accuracy of pavement crack detection techniques. It was found that increasing the network depth improved accuracy, while changing the image position significantly reduced detection accuracy. Yet another prior art generated a large-scale road damage dataset and marked the location and type of road damage in each image. Subsequently, a damage detection model was trained using a deep learning-based end-to-end object detection method. By applying this method to a mobile app, road damage could be easily detected even in areas lacking experts and labor.

[0006] Another prior art automatically detected bridge cracks by constructing an end-to-end detection model. It was demonstrated that the number of parameters can be effectively reduced by using depth-separable convolutions. Another prior art proposed the YOLOv3-Lite method to improve speed without compromising detection accuracy. Yet another prior art proposed a method for efficiently detecting damage at low cost by using CNNs to automatically detect and measure surface transmission images and reconstruct hidden cracks in 3D. Yet another prior art proposed a technique based on a Fully Convolutional Network (FCN) to detect cracks at the pixel level. The FCN consists of upsampling and downsampling layers capable of detecting objects of various scales, and the crack detection accuracy was measured at 97.96%. Yet another prior art improved VGG16 using transfer learning and modeled the accurate classification of surface defects in concrete bridges. In another conventional technology, region-based deformation modules were added to Faster R-CNN, R-FCN, and FPN-based Faster RCNN to improve the accuracy of crack detection. As mentioned earlier, most studies have modeled surface crack detection in concrete using various transfer learning techniques to enhance accuracy. While using transfer learning makes model training easier and faster, this is a characteristic of network architectures. There is no advantage to preprocessing or data augmentation methods for crack data. Prior art literature

[0007] [1] Kenney, Charles S and Zuliani, Marco and Manjunath, BS, "An axiomatic approach to corner detection," IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 1, pp. 191-197, 2005. The problem to be solved

[0008] The technical problem to be solved by the present invention is to provide an efficient data set collection method and system for concrete crack detection through spatially adaptive data augmentation, which utilizes a neural network to efficiently augment concrete crack data based on direction and thickness with the goal of improving the accuracy of crack detection. Considering the limitations of using transfer learning, the present invention proposes a new network training model for crack data augmentation. means of solving the problem

[0009] In one aspect, the method for collecting an efficient data set for detecting concrete cracks through spatially adaptive data augmentation proposed in the present invention comprises: a step of extracting skeletal data to preserve connectivity data of crack line segments in a crack image through a skeletal data extraction unit; a step of extracting a corner, which is the intersection of two edges having a slope change greater than a predetermined standard, from the extracted skeletal data through a corner extraction unit; a step of adjusting the weights of the standard deviation and deformation strength for the extracted skeletal data and the corner through a crack direction learning unit, and performing feature learning using an artificial neural network for direction-based elastic distortion to model the crack image based on the applied weights; and a step of analyzing the shape of the crack and measuring the thickness through a crack thickness learning unit to perform thickness learning for the crack, and integrating the learning result for direction-based elastic distortion and the thickness learning result for the crack.

[0010] The step of extracting a corner, which is the intersection of two edges having a slope change greater than a predetermined standard in the extracted skeletal data, calculates the rate of change between two edges to extract the corner, extracts a location where the bidirectional slope change is greater than a predetermined standard as a corner, moves an arbitrary pixel (x, y) as the center, moves the moved pixel (u, v) within a window to calculate the sum of squares of the difference in pixel values ​​within the mask window using the weight at the location of the arbitrary pixel (x, y), and if the calculation result is greater than a predetermined standard, determines the intersection of the two edges as a corner.

[0011] The step of adjusting the weights of the standard deviation and deformation strength for the extracted skeletal data and corners through the crack direction learning unit, and performing feature learning using an artificial neural network for direction-based elastic distortion to model the crack image based on the applied weights, involves performing upsampling using a direction-based learning image and a residual image as input through a residual-based CNN, and performing feature learning using the residual-based CNN by using an image in which direction-based elastic distortion is applied to the line segment representing the crack to model the crack image as input.

[0012] The step of analyzing the shape of the crack and measuring its thickness through the crack thickness learning unit to perform thickness learning for the crack, and integrating the learning result for the direction-based elastic distortion and the thickness learning result for the crack, uses contour approximation to perform thickness learning for the crack, and the contour approximation identifies boundaries formed by regions having the same color or pixel value to perform thickness learning for the crack, thereby integrating the learning result for the direction-based elastic distortion and the thickness learning result for the crack.

[0013] In another aspect, the efficient data set collection system for detecting concrete cracks through spatially adaptive data augmentation proposed in the present invention comprises: a skeletal data extraction unit that extracts skeletal data to preserve connectivity data of crack line segments in a crack image; a corner extraction unit that extracts a corner, which is the intersection of two edges having a slope change greater than a predetermined standard in the extracted skeletal data; a crack direction learning unit that adjusts the weights of the standard deviation and deformation strength for the extracted skeletal data and the corner, and performs feature learning using an artificial neural network for direction-based elastic distortion to model the crack image based on the applied weights; and a crack thickness learning unit that analyzes the shape of the crack and measures the thickness to perform thickness learning for the crack, and integrates the learning result for direction-based elastic distortion and the thickness learning result for the crack. Effects of the invention

[0014] According to embodiments of the present invention, the difficulty of dataset collection is efficiently resolved by using a data augmentation technique based on learning the direction and thickness of cracks, which is cost-effective and time-efficient. Furthermore, efficiency can be improved by proposing a method to adaptively process crack data by constructing a quadtree based on the presence of cracks. To verify the degree of accuracy improvement, experiments were conducted applying the crack detection algorithm to various scenes, and accuracy improved in all scenes when measured by Intersection over Union (IoU) accuracy. Brief explanation of the drawing

[0015] FIG. 1 is an exemplary diagram showing an image after data augmentation according to one embodiment of the present invention. FIG. 2 is an illustrative diagram for explaining a data set collection method for concrete crack detection according to an embodiment of the present invention. FIG. 3 is a diagram showing the configuration of a data set collection system for concrete crack detection according to one embodiment of the present invention. FIG. 4 is a drawing showing a skeleton extracted from a crack image according to one embodiment of the present invention. FIG. 5 is a diagram showing a crack vector with a skeleton and a corner set according to one embodiment of the present invention. FIG. 6 is a diagram illustrating elastic distortion based on the direction of the crack direction according to one embodiment of the present invention. FIG. 7 is a diagram illustrating crack data enhancement according to one embodiment of the present invention. FIG. 8 is a diagram illustrating the crack generation process through CNN according to one embodiment of the present invention. FIG. 9 is a diagram showing the architecture of a residual-based CNN according to one embodiment of the present invention. FIG. 10 is a drawing for explaining a contour approximation according to one embodiment of the present invention. FIG. 11 is a diagram showing the architecture of a CNN that uses the thickness of a crack extracted based on contour information according to one embodiment of the present invention. FIG. 12 is a diagram illustrating the process of constructing a quadtree according to one embodiment of the present invention. FIG. 13 is a diagram illustrating the process of classifying into FD and ED based on the presence or absence of density according to one embodiment of the present invention. FIG. 14 is a diagram illustrating a quadtree generation process including path state values ​​according to an embodiment of the present invention. FIG. 15 is a diagram showing an example of a quadtree node for collecting crack patches according to one embodiment of the present invention. Specific details for implementing the invention

[0016] The present invention proposes a data augmentation technique based on Convolutional Neural Network (CNN or ConvNet) training to efficiently obtain a dataset of images containing concrete cracks. Concrete cracks typically lack standardized shapes and possess complex patterns, making it difficult to obtain images and posing a risk of exposure to dangerous situations during data acquisition. Therefore, the present invention efficiently resolves the difficulties of dataset collection by using a data augmentation technique based on learning the direction and thickness of cracks, which is both cost-effective and time-efficient. Furthermore, to enhance efficiency, a method for adaptively processing crack data by constructing a quadtree based on the presence of cracks is proposed. To verify the degree of accuracy improvement, experiments were conducted applying the crack detection algorithm to various scenes, and accuracy improved in all scenes when measured by Intersection over Union (IoU) accuracy. When the algorithm was executed without augmenting crack data, the false positive rate was approximately 25%. In contrast, when data was augmented using the proposed method, the false positive rate decreased significantly to 3%. The embodiments of the present invention will be described in detail below with reference to the attached drawings.

[0018] In a crack detection network according to an embodiment of the present invention, handling issues such as non-uniform lighting, blotchy areas, blurring, and noise is crucial in image-based crack detection research. One method for detecting cracks is to define features that clearly distinguish between noise and cracks, such as the iterative clipping method, which assumes that cracks are generally darker than their surroundings. In the prior art, the Neighboring Difference Histogram (NDHM) method was proposed to segment crack images. However, this method may malfunction when there are many dark areas, such as shadows, in the test image. To improve the accuracy and completeness of crack detection, wavelet transform-based methods were proposed to enhance the contrast of crack regions. However, due to the anisotropic nature of wavelets, they often fail to process cracks with high curvature or low continuity. To extend NDH with more robust noise reduction capabilities, the F* seed-growing approach (FSA) method was proposed. The CrackTree method is designed to remove shadows cast on cracks, but it takes a long time to compute. Approaches that measure image anisotropy, such as Conditional Texture Anisotropy (CTA) and Free-Form Anisotropy (FFA), which consider both brightness and connectivity, have shown good results in crack detection. However, they are often sensitive to edge regions and can introduce noise. A method has been proposed to address these issues using saliency.

[0019] Another conventional technology utilized the Local Binary Pattern (LBP) method to achieve good crack detection results through local features, but required parameter adjustment for each image. Yet another conventional technology proposed Black Top-Hat (BTH) transformation and threshold segmentation methods to detect cracks on concrete tunnel surfaces, but these methods do not function well under uneven lighting conditions. Recently, Regions of Aggregation (ROA) have been modeled to consider multiple data such as the spatial distribution, intensity, and geometric features of cracks, while Regions of Belief (ROB) have been modeled to expand the crack area. This method yields good results for thin cracks.

[0020] Generally, data augmentation is a technique that applies various transformations to source data to increase the volume of data used in learning-based research. While data augmentation is typically used during the training phase, it can also be used during the testing phase, which is known as Test-Time Augmentation (TTA). This method augments a single test image with multiple images and performs inference on them. The output generated from this process is subsequently ensembled. This technique is commonly used in competitions such as Kaggle.

[0021] In the image manipulation-based approach according to an embodiment of the present invention, there are pixel-level transformation techniques and spatial-level transformation techniques. Pixel-level transformation is a technique that transforms an image at the pixel level by applying blur, jitter, noise, etc., to the image. Various techniques such as Gaussian blur, motion blur, brightness jitter, contrast jitter, saturation jitter, ISO noise, and JPEG compression are used for pixel-level transformation. Spatial-level transformation is a technique that transforms the image space. Spatial-level transformation is a technique that deforms the image space; representative methods include flipping and rotation, and cropping, which cuts out a part of the image, is also widely used.

[0022] Conventional techniques proposed a method of randomly shuffling feature values ​​within non-overlapping sliding windows. While applying this method improves performance, it is significantly affected by the window size. Another conventional technique proposed a method of randomly selecting two images from a training set, randomly cropping them to a size of 224×224, and applying random horizontal flipping. The two resulting patches are averaged to generate a blended patch. Although the two images are blended, accuracy may be degraded because only one label is used. Yet another conventional technique proposed an improved method for blending two images, moving away from the simple average blending approach. Instead, eight new blending methods were introduced to enhance the process. Yet another conventional technique proposed a method of interpolating two images using weighted linear interpolation with lambda values ​​between 0 and 1. Although simple, this method can improve the model's generalization performance, prevent the memory of corrupted labels, and make the model more sensitive to adversarial examples.

[0023] Another prior art proposed the RICAP (Random Image Cropping Patching) technique, which generates a single image by combining randomly cropped patches from four images. Additionally, similar to the Mixup technique, soft labels for training were generated by blending the labels of the four images based on the area ratio of the patches. However, if the background, rather than the main area of ​​the image, is cropped during the cropping process, incorrect labels may be assigned. Another prior art proposed a method to process Mixup at the level of hidden representations or feature maps instead of input image labels. Using this method allows for smoothing decision boundaries while preserving the advantages of Mixup. Yet another prior art adopted an approach that involves erasing image regions, unlike the previously mentioned method of image blending. This method involves creating bounding boxes of random size in the input images and filling them with random noise, ImageNet averages, 0, 255, or other values ​​before using them for training.

[0024] Another prior art proposed a method of dividing images into a grid and randomly erasing patches in each iteration during training. This approach enables more comprehensive predictions by allowing the network to consider various parts of the image rather than focusing only on specific parts of an object. Another prior art proposed a method called CutMix, which combines Mixup and Cutout techniques. This technique involves selecting and erasing boxed regions in an image and then filling the empty areas with patches extracted from other images. Labels are blended in proportion to the area of ​​the patches. Yet another prior art proposed a method called Augmix, which applies multiple augmentation techniques to an image serially or in parallel and then blends them with the original image. This method was proposed to improve performance on datasets such as ImageNet-C and ImageNet-P, which are designed to measure robustness rather than improve general test accuracy. To mitigate the strong edge issues that occur during the patch cutting and pasting process in CutMix, prior art proposed a method called SmoothMix, which smooths boundary regions to create a more gradual transition. Another prior art proposed a method called PuzzzleMix, which blends images while preserving their saliency. This approach aims to generate a blended output while maintaining the prominent regions of the original images. This enables the preservation of local statistics in each image and demonstrates better generalization performance compared to existing blending techniques. It also enhances robustness against adversarial attacks.

[0026] FIG. 1 is an exemplary diagram showing an image after data augmentation according to one embodiment of the present invention.

[0027] In the dataset and data augmentation for crack detection according to an embodiment of the present invention, the dataset collected according to the prior art is widely used in crack detection research. The dataset consists of images captured using a smartphone on the surface of the main tower and anchor room of a bridge in Dalian, Liaoning Province, China. Although the resolution of the original images is 4160x3120, they were manually subdivided into smaller images with a resolution of 256x256 and classified into images with cracks and images without cracks. The SDNET2018 dataset provides images of concrete bridges that include both images with cracks and images without cracks. The performance of an artificial neural network model is related to the size of the training dataset. However, it can be difficult to obtain a large and diverse dataset of actual crack images. Techniques such as data augmentation are often used to compensate for the limited availability of actual crack images in areas where it is difficult to collect a sufficient amount of real data. Data augmentation techniques can effectively resolve the limitations of small training datasets, such as overfitting, by artificially expanding with synthetically generated data.

[0028] One simple data augmentation technique is to use the ImageDataGenerator interface provided by TensorFlow 2.0, which is modeled based on spatial-level transformations. By applying operations such as image flipping, rotation, shifting, and other transformations, the number of data samples can be effectively increased. However, as illustrated in Fig. 1, since most of these are simple linear transformations, they may not be sufficient to represent the diversity of crack patterns. Fig. 1(a) shows the original image, Fig. 1(b) shows the image after horizontal flipping, Fig. 1(c) shows the image after rotation, and Fig. 1(d) shows the image after horizontal shifting and data augmentation, respectively.

[0029] According to an embodiment of the present invention, a CNN for crack pattern extraction and elastic distortion for crack shape deformation are utilized. Since cracks do not have standardized shapes and can vary depending on the surrounding environment, it is necessary to train and learn the characteristics of the cracks in order to generate metadata that considers the characteristics of actual crack data. Below, a learning-based data augmentation method utilizing the direction and thickness of cracks is described.

[0031] FIG. 2 is an illustrative diagram for explaining a data set collection method for concrete crack detection according to an embodiment of the present invention.

[0032] A method for collecting a data set for detecting concrete cracks according to an embodiment of the present invention comprises: a step (210) of extracting skeletal data to preserve connectivity data of crack line segments in a crack image through a skeletal data extraction unit; a step (220) of extracting a corner, which is the intersection point of two edges having a slope change greater than a predetermined standard, from the extracted skeletal data through a corner extraction unit; a step (230) of adjusting the weights of the standard deviation and deformation strength for the extracted skeletal data and the corner through a crack direction learning unit, and performing feature learning using an artificial neural network for direction-based elastic distortion to model the crack image based on the applied weights; and a step (240) of analyzing the shape of the crack and measuring the thickness to perform thickness learning for the crack through a crack thickness learning unit, and integrating the learning result for direction-based elastic distortion and the thickness learning result for the crack.

[0033] In step (210), skeletal data is extracted through a skeletal data extraction unit to preserve connectivity data of crack line segments in a crack image.

[0034] In step (220), a corner is extracted from the extracted skeletal data through a corner extraction unit, which is the intersection of two edges having a slope change greater than a predetermined standard.

[0035] According to an embodiment of the present invention, the rate of change between two edges for extracting a corner is calculated, and a location where the bidirectional slope change is greater than or equal to a predetermined standard is extracted as a corner.

[0036] According to an embodiment of the present invention, an arbitrary pixel (x, y) is moved around, and the moved pixel (u, v) is moved within a window to calculate the sum of squares of the difference in pixel values ​​within the mask window using the weight at the position of the arbitrary pixel (x, y), and if the calculation result is greater than or equal to a predetermined standard, the intersection point of the two edges is determined as a corner.

[0037] In step (230), the weights of the standard deviation and deformation strength for the extracted skeletal data and corners are adjusted through the crack direction learning unit, and feature learning using an artificial neural network is performed for the direction-based elastic distortion to model the crack image based on the applied weights.

[0038] According to an embodiment of the present invention, upsampling is performed using a direction-based learning image and a residual image as inputs through a residual-based CNN, and feature learning is performed using the residual-based CNN by using an image in which direction-based elastic distortion is applied to a line segment representing a crack as input to model a crack image.

[0039] In step (240), the shape of the crack is analyzed and the thickness is measured through the crack thickness learning unit to perform thickness learning for the crack, and the learning result for the direction-based elastic distortion and the thickness learning result for the crack are integrated.

[0040] According to an embodiment of the present invention, a contour approximation is used to perform thickness learning for the crack, and the contour approximation identifies boundaries formed by regions having the same color or pixel value to perform thickness learning for the crack, thereby integrating the learning result for the direction-based elastic distortion and the thickness learning result for the crack.

[0042] FIG. 3 is a diagram showing the configuration of a data set collection system for concrete crack detection according to one embodiment of the present invention.

[0043] The data set collection system (300) according to the present embodiment may include a processor (310), a bus (320), a network interface (330), a memory (340), and a database (350). The memory (340) may include an operating system (341) and an efficient data set collection routine (342) for detecting concrete cracks through spatially adaptive data augmentation. The processor (310) may include a skeletal data extraction unit (311), a corner extraction unit (312), a crack direction learning unit (313), and a crack thickness learning unit (314). In other embodiments, the data set collection system (300) may include more components than those of FIG. 3. However, it is not necessary to clearly illustrate most of the prior art components. For example, the data set collection system (300) may include other components such as a display or a transceiver.

[0044] Memory (340) is a computer-readable recording medium and may include a non-perishable permanent mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Additionally, program code for an operating system (341) and an efficient data set collection routine (342) for detecting concrete cracks through spatial adaptive data augmentation may be stored in memory (340). These software components may be loaded from a computer-readable recording medium separate from memory (340) using a drive mechanism (not shown). This separate computer-readable recording medium may include computer-readable recording media (not shown), such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, software components may be loaded into memory (340) via a network interface (330) rather than a computer-readable recording medium.

[0045] The bus (320) can enable communication and data transmission between components of the data set collection system (300). The bus (320) can be configured using a high-speed serial bus, a parallel bus, a Storage Area Network (SAN), and / or other suitable communication technology.

[0046] The network interface (330) may be a computer hardware component for connecting the data set collection system (300) to a computer network. The network interface (330) may connect the data set collection system (300) to a computer network via a wireless or wired connection.

[0047] The database (350) can serve to store and maintain all information necessary for efficient data set collection for concrete crack detection through spatial adaptive data augmentation. Although FIG. 3 illustrates the database (350) being built and included within the data set collection system (300), it is not limited thereto and may be omitted depending on the system implementation method or environment, or it is also possible for all or part of the database to exist as an external database built on a separate system.

[0048] The processor (310) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations of the data set collection system (300). Instructions may be provided to the processor (310) via memory (340) or a network interface (330) and via a bus (320). The processor (310) may be configured to execute program code for a skeletal data extraction unit (311), a corner extraction unit (312), a crack direction learning unit (313), and a crack thickness learning unit (314). Such program code may be stored in a recording device such as memory (340).

[0049] The skeletal data extraction unit (311), corner extraction unit (312), crack direction learning unit (313), and crack thickness learning unit (314) may be configured to perform the steps (210–240) of FIG. 2.

[0050] The data set collection system (300) may include a skeletal data extraction unit (311), a corner extraction unit (312), a crack direction learning unit (313), and a crack thickness learning unit (314).

[0051] The skeletal data extraction unit (311) according to an embodiment of the present invention extracts skeletal data to preserve connectivity data of crack line segments in a crack image.

[0052] A corner extraction unit (312) according to an embodiment of the present invention extracts a corner that is the intersection of two edges having a slope change greater than a predetermined standard from the extracted skeletal data.

[0053] A corner extraction unit (312) according to an embodiment of the present invention calculates the rate of change between two edges for extracting a corner and extracts a location where the bidirectional slope change is greater than or equal to a predetermined standard as a corner.

[0054] A corner extraction unit (312) according to an embodiment of the present invention moves an arbitrary pixel (x, y) around itself and moves the moved pixel (u, v) within a window to calculate the sum of squares of the difference in pixel values ​​within the mask window using weights at the position of the arbitrary pixel (x, y), and if the calculation result is greater than or equal to a predetermined standard, the intersection of the two edges is determined as a corner.

[0055] A crack direction learning unit (313) according to an embodiment of the present invention adjusts the weights of the standard deviation and deformation strength for the extracted skeletal data and corners, and performs feature learning using an artificial neural network for direction-based elastic distortion to model the crack image based on the applied weights.

[0056] A crack direction learning unit (313) according to an embodiment of the present invention performs upsampling by utilizing a direction-based learning image and a residual image as inputs through a residual-based CNN, and performs feature learning using the residual-based CNN by using an image in which direction-based elastic distortion is applied to a line segment representing a crack to model the crack image as input.

[0057] A crack thickness learning unit (314) according to an embodiment of the present invention analyzes the shape of the crack and measures the thickness to perform thickness learning for the crack, and integrates the learning result for the direction-based elastic distortion and the thickness learning result for the crack.

[0058] A crack thickness learning unit (314) according to an embodiment of the present invention uses a contour approximation to perform thickness learning for the crack, and the contour approximation identifies a boundary formed by an area having the same color or pixel value to perform thickness learning for the crack, thereby integrating the learning result for the direction-based elastic distortion and the thickness learning result for the crack.

[0060] FIG. 4 is a drawing showing a skeleton extracted from a crack image according to one embodiment of the present invention.

[0061] A method for extracting a skeleton from a crack image, that is, crack data, according to an embodiment of the present invention will be described. FIG. 4(a) shows an input crack image, and FIG. 4(b) shows an extracted skeleton.

[0062] Upon examining the actual crack data, it was found that the area where cracks exist is very small. Therefore, as shown in Fig. 4(b), when applying data transformation to crack line segments, the cracks tend to fragment irregularly. To preserve connectivity information of the crack line segments, the present invention first extracts a skeleton from the crack image. To this end, the present invention applies a skeletonization process to convert the crack line segments from a pixel-based representation to a thin vector-based representation.

[0064] FIG. 5 is a diagram showing a crack vector with a skeleton and a corner set according to one embodiment of the present invention.

[0065] A method for detecting a corner, which is the intersection of two edges with a large change in slope, in crack data according to an embodiment of the present invention is described. The present invention utilizes a Shi-Tomasi corner detection technique that improves upon the Harris corner detection method. This technique calculates the rate of change between edges and considers a location with a large change in slope in both directions as a corner. A single pixel is shifted as the center, and the x-axis pixel u and the y-axis pixel v are shifted within the window to calculate the sum of squares of the difference in pixel values ​​within the mask window. In Equation (1), W(x, y) represents the weight at position (x, y), I(x, y) represents the pixel value at the original position, and I(x + u, y + v) represents the pixel value at the shifted position.

[0066] (1)

[0067] Applying Taylor expansion to the above equation yields equation (2).

[0068] (2)

[0069] Equation (3) is the final equation obtained by rearranging Equation (2) into a matrix form, and if the eigenvalue of the matrix is ​​greater than a certain threshold, it is considered a corner. This process is calculated using the Singular Value Decomposition (SVD) technique.

[0070] (3)

[0071] If the user specifies the maximum number of corners, the slope change threshold, and the minimum distance between corners, N corners are extracted.

[0072] Figure 5(a) shows a crack vector with a skeleton and a corner set, Figure 5(a) shows the input data, Figure 5(b) shows the skeleton extraction, Figure 5(c) shows the corner extraction, and Figure 5(d) shows the corner direction generated by connecting the corners.

[0073] Figure 5(d) shows the result of connecting the detected corners with line segments. This data will be used for direction-based augmentation techniques. Figure 4 shows the stable extraction of the crack skeleton without disconnected segments, which is clear when compared to Figure 4.

[0075] FIG. 6 is a diagram illustrating elastic distortion based on the direction of the crack direction according to one embodiment of the present invention.

[0076] A direction-based elastic distortion technique modeled from both microscopic and macroscopic perspectives according to an embodiment of the present invention is described. At the microscopic level, the deformation for a crack is calculated by adjusting the weights of the standard deviation σ and the strain strength α.

[0077] Figure 4(a) shows the input data, and Figure 4(b) is an enlarged view of one of the line segments formed by connecting the extracted corners as previously mentioned. As can be seen in Figure 4(b), this allows the degree of deformation to be easily observed based on the applied weights. Figure 4(c) shows the result of applying direction-based elastic distortion to a line segment representing a crack. From a macroscopic perspective, elastic distortion involves adjusting the weights of the rotation angle and affine transformation to deform the surface shape of the crack.

[0079] FIG. 7 is a diagram illustrating crack data enhancement according to one embodiment of the present invention.

[0080] Figure 7 shows the result of applying micro-macro elastic distortion using direction-based segments. Unlike conventional pixel-based data augmentation techniques where controlling such directionality was difficult, the present invention can capture the direction of cracks through more diverse variations rather than simple straight cracks. Of course, introducing randomness can temporarily alleviate this problem, but transformations based on randomization can lead to the loss of the original crack patterns and characteristics.

[0082] FIG. 8 is a diagram illustrating the crack generation process through CNN according to one embodiment of the present invention.

[0083] A CNN-based network capable of effectively modeling crack data is proposed by training a direction-based data augmentation technique according to an embodiment of the present invention. For training, 500 skeletal crack images and 400 line segment images were utilized in the embodiment of the present invention. Additionally, additional training data was collected using a direction-based data augmentation technique. In the embodiment of the present invention, a feature learning process based on acquired crack images is described to obtain patterns similar to actual cracks. When training the network using crack images obtained through the skeletalization technique, convergence problems sometimes occurred during the learning process due to the complexity of the features. However, this problem was solved by utilizing direction data obtained by extracting corners from the cracks and connecting them to line segments.

[0084] Figure 8(a) shows skeletal data extracted from a crack image, and Figure 8(b) displays the results generated through CNN training, showing a pattern very similar to the actual crack. When the weight of elastic distortion is large, it can generate noise that does not sufficiently preserve the characteristics of the original crack. However, the method according to an embodiment of the present invention has mitigated this problem.

[0086] FIG. 9 is a diagram showing the architecture of a residual-based CNN according to one embodiment of the present invention.

[0087] The architecture of the residual-based CNN used in the experiment according to the embodiment of the present invention utilizes direction-based training images, a network consisting of 26 layers, three upsampling steps, and residual images as input to obtain optimal results during the network training process. As the number of upsampling steps increased, the results improved, and color-related issues in 3-channel training were resolved by using residual images. Finally, to obtain results that consider details of crack patterns, direction-based elastic distortion data was used as input, and results were obtained by training the network using this data. However, this approach considers only the direction of the cracks and not the thickness of the cracks, so it may not sufficiently capture changes in crack thickness during training. Therefore, in the embodiment of the present invention, the framework was extended to incorporate a crack thickness learning method.

[0089] FIG. 10 is a drawing for explaining a contour approximation according to one embodiment of the present invention.

[0090] In an embodiment of the present invention, contour approximation is utilized to analyze the shape of a crack and measure its thickness. A contour refers to a boundary formed by regions having the same color or pixel value. It is commonly used to identify the outline or shape of an object. In this invention, the Douglas-Peucker algorithm embedded in the OpenCV library was used. Fig. 10(a) shows the input data, and Fig. 10(b) shows the contour. The image with ≥0.005, FIG. 10 (c) is Each shows a contour image with a value of 0.005.

[0091] To obtain the result of Fig. 10(b), the parameters for the contour approximation It was set to 0.005. This value is determined empirically and is generally used to obtain a tight contour. If the value is small, a result very similar to the original contour is obtained, but If the value is large, a difference from the original contour may occur.

[0093] FIG. 11 is a diagram showing the architecture of a CNN that uses the thickness of a crack extracted based on contour information according to one embodiment of the present invention.

[0094] The final data extracted in the present invention captured more detailed features in terms of thickness compared to existing experimental results that considered only directionality. Figure 11 shows an artificial neural network trained using crack thickness extracted based on contour information. The network model follows the approach described above, and the input data consists of crack thickness and direction information.

[0096] FIG. 12 is a diagram illustrating the process of constructing a quadtree according to one embodiment of the present invention.

[0097] A quadtree according to an embodiment of the present invention is a tree data structure in which each internal node has four children, and is adaptively subdivided based on the existence of important data in each node. Depending on the application, the "minimum unit of interest" in terms of information is generally stored in the leaf cells of the quadtree.

[0099] FIG. 13 is a diagram illustrating the process of classifying into FD and ED based on the presence or absence of density according to one embodiment of the present invention.

[0100] In an embodiment of the present invention, a quadtree is used to reduce the data size because the proportion of cracks in the total space is small compared to the entire data set.

[0101] Referring to Fig. 13, the data is classified into FD (Full Density) and ED (Empty Density) based on the presence or absence of density. Fig. 13(a) shows the input data, Fig. 13(b) shows the patch segmentation, and Fig. 13(c) shows FD (red) and ED (other), respectively.

[0102] In an embodiment of the present invention, binary data representing cracks as white areas is used as input (see FIG. 13(a)). As shown in FIG. 13(b) and FIG. 13(c), the space is partitioned based on the crack areas as a criterion for quadtree segmentation. The reason for partitioning the crack data using a quadtree is not to obtain pixel information, but to extract meaningful information in the spatial domain. Network training is performed using image patches represented as leaf nodes of the quadtree, and C p It is represented as such. The lowest level nodes used to construct a quadtree are created using the method described above, and the nodes are merged upwards to form a tree structure.

[0104] FIG. 14 is a diagram illustrating a quadtree generation process including path state values ​​according to an embodiment of the present invention.

[0105] Before combining the nodes generated according to an embodiment of the present invention into a quadtree, the density value of each node is compared with a threshold value to determine whether a crack exists within the node. Based on this comparison, the node is classified into an FD or ED state (see FIG. 14(a)). The nodes at the lowest level have a state value classified by themselves, and the state value of the nodes at the upper level is determined by combining the state values ​​of the lower nodes (see FIG. 14(b)). Each node in the tree has a state value, density data, and an associated key value, and the key includes X and Y coordinates representing the location of the node and the depth of the tree, and the results are combined using the depth and location after the network process is completed. The state is indicated as FD, ED, or MIX, and in the present invention, the density of each patch is used as data for the nodes at the lowest level (see FIG. 14(a)). The depth of the nodes at the lowest level is calculated using the following equation (4).

[0106] (4)

[0107] Here, d represents the depth of the current node, and D width and N width represents the width of the total input data and the width of the current node, respectively. The parent node is created upwards from the four child nodes. The data of the parent node is the sum of the data from the child nodes. The depth is decreased by 1, and the position is determined when the child nodes are merged. Finally, the state of the child nodes is aggregated to determine the state value of the parent node. If all child nodes have the same state value, that value is assigned to the parent node. If child nodes have both FD and ED states, the state value of the parent node is assigned as MIX.

[0109] FIG. 15 is a diagram showing an example of a quadtree node for collecting crack patches according to one embodiment of the present invention.

[0110] According to an embodiment of the present invention, if all child nodes have the same state value, the parent node stores that value and the child nodes are removed. Nodes with a state of ED indicate that there are no cracks, so network training is not required. In the case of FD, instead of applying network training all at once, the embodiment of the present invention follows a more efficient approach by selectively applying it to child nodes containing the necessary data. This process is repeated until the root node is reached to construct a quadtree in which state values ​​are assigned to all nodes. Once the tree is complete, the data and key values ​​of the FD nodes are collected (see FIG. 15), and this dataset is used for network training. Experimental results according to an embodiment of the present invention showed that memory efficiency was improved, as approximately 37.5% of the data could be compressed overall.

[0112] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0113] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0114] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0115] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0116] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A step of extracting skeletal data through a skeletal data extraction unit to preserve connectivity data of crack line segments in a crack image; a step of extracting corners, which are intersection points of two edges having a slope change greater than a predetermined standard, from the extracted skeletal data through a corner extraction unit; a step of adjusting the weights of the standard deviation and deformation strength for the extracted skeletal data and corners through a crack direction learning unit, and performing feature learning using an artificial neural network for direction-based elastic distortion to model the crack image based on the adjusted weights; A step of generating augmented crack data by analyzing the shape of the crack and measuring its thickness through a crack thickness learning unit to perform thickness learning for the crack, and integrating the learning result for the direction-based elastic distortion and the thickness learning result for the crack; a step of converting the augmented crack data into binary data in which the crack region is represented as a white region, and constructing a quadtree by dividing the entire space of the binary data; a step of classifying each node into a Full Density (FD) state where a crack exists or an Empty Density (ED) state where no crack exists by comparing the density value of each node with a threshold value for the lowest level node of the constructed quadtree, and performing a process of determining the state value of the upper parent node by aggregating the state values ​​of the lower child nodes, wherein if the child nodes include both FD and ED states, the parent node is assigned to a MIX state, and if the child nodes all have the same state value, the corresponding state value is assigned to the parent node and the child nodes are removed, and repeating the merging process up to the root node.A method for collecting a dataset for concrete crack detection, comprising the step of excluding nodes in the ED state from the quadtree in which the above merging process is completed, and selecting key values ​​including data and location information for nodes in the FD state to collect as a training dataset, wherein the step of extracting a corner comprises calculating the rate of change between two edges for extracting a corner, and extracting a location where the bidirectional gradient change is greater than or equal to a predetermined criterion as a corner, wherein for an arbitrary pixel (x, y) and a pixel (x + u, y + v) at a moved location obtained by moving the pixel (x, y) by (u, v) within a mask window centered on the pixel (x, y), the eigenvalue of a matrix derived from a calculation result calculated using the sum of squares of the difference in pixel values ​​between the pixel (x, y) and the pixel at the moved location (x + u, y + v) and the weight at the pixel (x, y) location is greater than or equal to a certain threshold value. Claim 2 delete Claim 3 A method for collecting a dataset for concrete crack detection according to claim 1, wherein the step of performing feature learning comprises performing upsampling using a direction-based learning image and a residual image as inputs through a residual-based CNN, and performing feature learning using the residual-based CNN by using an image in which direction-based elastic distortion is applied to a line segment representing a crack to model the crack image. Claim 4 A method for collecting a dataset for concrete crack detection according to claim 1, wherein the step of integrating the thickness learning results integrates the learning results for the direction-based elastic distortion and the thickness learning results for the cracks by using a contour approximation to perform thickness learning for the cracks, and the contour approximation performs thickness learning for the cracks by identifying boundaries formed by regions having the same color or pixel value. Claim 5 A data set collection system for concrete crack detection comprises: a skeletal data extraction unit that extracts skeletal data to preserve connectivity data of crack line segments in a crack image; a corner extraction unit that extracts corners, which are intersection points of two edges having a slope change greater than a predetermined standard, from the extracted skeletal data; and a crack direction learning unit that adjusts weights of standard deviation and deformation strength for the extracted skeletal data and corners, and performs feature learning using an artificial neural network for direction-based elastic distortion to model the crack image based on the adjusted weights. The system includes a crack thickness learning unit that generates augmented crack data by analyzing the shape of the crack and measuring its thickness to perform thickness learning for the crack, and integrating the learning result for the direction-based elastic distortion and the thickness learning result for the crack. The skeletal data extraction unit, the corner extraction unit, the crack direction learning unit, and the crack thickness learning unit are implemented by a processor of the data set collection system. The processor converts the augmented crack data into binary data in which the crack area is represented as a white area, divides the entire space of the binary data to construct a quadtree, compares the density value of each node with a threshold value for the lowest level node of the constructed quadtree to classify each node into a Full Density (FD) state where a crack exists or an Empty Density (ED) state where a crack does not exist, and performs a process of determining the state value of the parent node by aggregating the state values ​​of the child nodes below, wherein if the child nodes include both FD and ED states, the parent node is assigned a MIX state, and if the child nodes all have the same state value, the corresponding state value is assigned to the parent node and the child The merging process of removing nodes is repeated up to the root node, and nodes in the ED state are excluded from the quadtree after the merge process is completed,A data set collection system for concrete crack detection, wherein a key value containing data and location information for a node in an FD state is selected and collected as a training data set, and the corner extraction unit calculates the rate of change between two edges for extracting a corner, and extracts a location where the bidirectional gradient change is greater than or equal to a predetermined standard as a corner, wherein for an arbitrary pixel (x, y) and a pixel (x + u, y + v) at a moved location obtained by moving the pixel (x, y) by (u, v) within a mask window centered on the pixel (x, y), the pixel is extracted as a corner if the eigenvalue of a matrix derived from a calculation result calculated using the sum of squares of the difference in pixel values ​​between the pixel (x, y) and the pixel at the moved location (x + u, y + v) and the weight at the pixel (x, y) location is greater than or equal to a certain threshold. Claim 6 delete Claim 7 A data set collection system for concrete crack detection according to claim 5, wherein the crack direction learning unit performs upsampling by utilizing a direction-based learning image and a residual image as inputs through a residual-based CNN, and performs feature learning using the residual-based CNN by utilizing an image in which direction-based elastic distortion is applied to a line segment representing a crack to model the crack image. Claim 8 A data set collection system for concrete crack detection according to claim 5, wherein the crack thickness learning unit uses contour approximation to perform thickness learning for the crack, and the contour approximation performs thickness learning for the crack by identifying boundaries formed by regions having the same color or pixel value, thereby integrating the learning result for the direction-based elastic distortion and the thickness learning result for the crack.

Images