Patch-based neural network method for crack detection and segmentation of construction structures

Abstract

A crack detection method comprises the steps of: a device acquiring an image of a structure; the crack detection device dividing the acquired image of the structure into a plurality of patches; the crack detection device regionally segmenting cracks in each of the plurality of patches; the crack detection device merging each of the patches in which the cracks were regionally segmented to generate a merged image; and the crack detection device global segmenting cracks in the merged image.

Description

Technology Field

[0001] The technology described below pertains to construction management. Background Technology

[0002] Over time, cracks develop in construction structures. These cracks significantly impact the stability and durability of structures. Therefore, early detection of cracks is a critical challenge in construction and infrastructure maintenance. To address this, image-based methods for detecting structural cracks have been developed. Conventional technologies utilized edge detection and threshold-based image processing techniques. However, these technologies suffered from a problem where detection accuracy decreased as the crack shape became more complex or irregular. Furthermore, these technologies had the limitation of failing to detect minute cracks.

[0003] Recently, deep learning-based image analysis technology has begun to advance. Deep learning-based image analysis technology has demonstrated high performance in various fields, such as medical imaging and satellite imagery. Deep learning-based image analysis technology has effectively analyzed complex patterns. For example, models such as U-Net are utilized to effectively segment complex patterns through an encoder-decoder structure.

[0004] However, deep learning-based image analysis technology processed the entire image area at once. In this process, there was a possibility that small cracks or minute damages might be missed. In particular, when complex surface textures of structures or background noise are combined, the performance of crack detection may be degraded. Prior art literature

[0005] Korean Registered Patent Publication 10-2624378 The problem to be solved

[0006] The technology described below aims to disclose a method for effectively detecting cracks by performing local and global partitioning through a patch-based approach. means of solving the problem

[0007] The technology described below aims to disclose a crack detection method.

[0008] In one embodiment, a crack detection method comprises: a step in which a device acquires an image of a structure; a step in which the crack detection device divides the acquired image of the structure into a plurality of patches; a step in which the crack detection device regionally divides a crack in each of the plurality of patches; a step in which the crack detection device merges each of the patches in which the crack was regionally divided to generate a merged image; and a step in which the crack detection device globally divides a crack in the merged image. Effects of the invention

[0009] The technology described below allows for the segmentation of cracks in structural images. The technology described below enables the segmentation of cracks in structural images without requiring significant manpower or cost. The technology described below allows for the segmentation of cracks by considering both the local and global characteristics of the structural image. The technology described below facilitates building management. Brief explanation of the drawing

[0010] FIG. 1 is one of the embodiments in which a crack detection device (100) performs a crack detection method. FIG. 2 is a flowchart (200) of one embodiment of a crack detection method. FIG. 3 is a configuration of one of the embodiments of a crack detection device (300). Figure 4 shows the overall process of splitting a crack using the method of the embodiment. Figure 5 shows the result of comparing the case using a simple U-Net-based model with the case using the method of the embodiment. Figure 6 shows the results of comparing the case using a simple U-Net-based model with the case using the method of the embodiment. Specific details for implementing the invention

[0011] The technology described below may be subject to various modifications and may have various embodiments. Specific embodiments of the technology described below may be described in the drawings of the specification. However, this is for the purpose of explaining the technology described below and is not intended to limit the technology described below to specific embodiments. Accordingly, it should be understood that all modifications, equivalents, and substitutions that fall within the spirit and scope of the technology described below are included in the technology described below.

[0012] Terms such as first, second, A, B, etc., may be used to describe various components. However, the above terms are used merely to distinguish one component from other components and are not intended to limit the components to the said terms. For example, without departing from the scope of rights of the technology described below, the first component may be named the second component, and similarly, the second component may be named the first component. The term “and / or” includes a combination of multiple related described items or any of the multiple related described items.

[0013] In the terms used below, singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "includes" should be understood to mean that the described features, number, steps, actions, components, parts, or combinations thereof exist, and not to exclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0014] Before providing a detailed description of the drawings, it is to clarify that the classification of components in this specification is merely based on the primary function each component is responsible for. That is, two or more components described below may be combined into a single component, or a single component may be divided into two or more components based on more subdivided functions. Furthermore, each component described below may additionally perform some or all of the functions of other components in addition to its own primary function, and it is obvious that some of the primary functions of each component may be exclusively performed by other components.

[0015] Furthermore, in performing the method or operation method, each process constituting the method may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, each process may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0017] FIG. 1 is one of the embodiments in which a crack detection device (100) performs a crack detection method.

[0018] The crack detection device (100) can be physically implemented in various forms. For example, the crack detection device (100) can take the form of a PC, laptop, smart device, server, or a chipset dedicated to data processing.

[0019] There may be at least one crack detection device (100). That is, the crack detection method may be performed by a single crack detection device or divided and performed by at least one device.

[0020] The crack detection device (100) may be a device that performs a crack detection method. The crack detection device (100) may acquire an image of a structure. The crack detection device (100) may divide cracks in the image of a structure. The crack detection device (100) may output the result of dividing the cracks. The crack detection device (100) may perform local division and global division. The crack detection device (100) may perform division using a first model and a second model.

[0022] FIG. 2 is a flowchart (200) of one embodiment of a crack detection method.

[0023] The crack detection device can acquire an image of the structure (210).

[0024] A structure may refer to a component that has a physical form to perform a certain function or purpose. Structures may include buildings, bridges, tunnels, dams, towers, etc.

[0025] The structure image may include an image of the structure. The structure image may include an image of the exterior walls of the structure. The structure image may include an image of the entire structure or a part thereof.

[0026] The crack detection device can divide the acquired structural image into multiple patches (220).

[0027] The size of the patch may vary depending on the characteristics of the crack and the resolution of the image.

[0028] The crack detection device can divide the crack into regions in each of the multiple patches (230).

[0029] Region segmentation can be the task of segmenting cracks within a single patch of a structure image. It can also be the task of segmenting cracks within only a specific area of ​​the structure image, rather than the entire image. Through region segmentation, even minute details of the cracks can be accurately detected.

[0030] Regional partitioning can be performed through the first model.

[0031] The first model may be a model that performs segmentation based on an image. The first model may be a model that classifies an image at the pixel level. The first model may be a model that assigns each pixel of an image to a desired class or category. The first model may be a model that classifies each pixel of an image into either a cracked class or a normal class.

[0032] The first model may be a model that takes one of a plurality of patches as input and performs partitioning.

[0033] The first model may be a machine learning-based model. In one embodiment, the machine learning model may include a supervised learning-based model, an unsupervised learning-based model, a semisupervised learning-based model, and a reinforcement learning-based model. In one embodiment, the machine learning model may include a decision tree, a random forest (RF), a K-nearest neighbor (KNN), Naive Bayes, a support vector machine (SVM), an artificial neural network (ANN), etc. In one embodiment, the ANN may be a Deep Neural Network (DNN), which may include a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Generative Adversarial Network (GAN), and Relation Networks (RL).

[0034] The first model may be a model that extracts features from an image and then reconstructs the image based on the extracted features. The first model may be a model having an encoder and a decoder structure. The encoder of the first model can extract features from the image. The decoder of the second model can reconstruct the image from the extracted features. The first model may be a U-Net-based model.

[0035] The first model may be a model that takes one of a plurality of patches as input and performs partitioning.

[0036] A crack detection device can generate a merged image by merging each patch that has been regionally divided into cracks (240).

[0037] The merged image may be an image that integrates the results of region partitioning from each of multiple patches. The merged image may be an image that has the same size as the original structure image obtained by merging multiple patches.

[0038] The crack detection device can input the merged image into the second model to perform global division (250).

[0039] The second model may be a model that performs segmentation based on images, identical or similar to the first model.

[0040] The second model may be a machine learning-based model identical or similar to the first model.

[0041] The second model may be a model having an encoder and decoder structure identical or similar to the first model.

[0042] The second model may be a model that takes a merged image as input and performs splitting.

[0043] The second model may be a model that takes a merged image as input and performs segmentation. Global segmentation may be the task of segmenting cracks within the merged image. Global segmentation may be the task of segmenting cracks within a merged image that has the same size as the entire initially acquired structure image, rather than at the patch level. Through global segmentation, the results of local segmentation can be corrected overall, and the consistency of the segmentation can be improved. Through local and global segmentation, global information and local features of the structure image are utilized simultaneously.

[0045] FIG. 3 is a configuration of one of the embodiments of a crack detection device (300).

[0046] The crack detection device (300) may correspond to the crack detection device (100) described above in FIG. 1. That is, the crack detection device (300) may be a device that performs the crack detection method described above.

[0047] The crack detection device (300) may include at least one input device (310), a storage device (320), a computing device (330), an output device (340), an interface device (350), and a communication device (360).

[0048] The input device (310) may receive data, information, or models necessary for performing the aforementioned crack detection method. The input device (310) may receive structural images, patch images, and merged images. The input device (310) may receive a first model and a second model. The input device (310) may receive training data necessary for training the first model and the second model. The input device (310) may include a device for inputting certain commands or data (keyboard, mouse and touchscreen, joystick, trackball, touchpad, scanner, webcam, etc.). The input device (310) may include a configuration for receiving data through a separate storage device (USB, CD, hard disk, etc.). The input device (310) may receive data through a separate measuring device or a separate database. The input device (310) may receive data via a wired or wireless connection through a communication device (360). The input device (310) may receive a control signal to control the crack detection device (300).

[0049] The storage device (320) can store data, information, or models necessary for performing the aforementioned crack detection method. The storage device (320) can store structural images, patch images, and merged images. The storage device (320) can store a first model and a second model. The storage device (320) can store training data necessary for training the first model and the second model. The storage device (320) may be a device that stores certain data, information, or models. The storage device (320) can store data, information, and models received through the input device (310). The storage device (320) can store commands that cause the computing device (330) to perform operations necessary for the crack detection method. The storage device (320) can store information generated during the process of computing by the computing device (330). That is, the storage device (320) may include memory. For example, storage devices may include HDD (Hard Disk Drive), SSD (Solid State Drive), ROM, RAM, and CD-ROM magnetic tape or floppy disk, etc.

[0050] The computing device (330) can perform calculations necessary to perform the aforementioned crack detection method. The computing device (330) can perform calculations necessary for the operation of the device acquiring a structure image. The computing device (330) performs the step of dividing the structure image acquired by the crack detection device into a plurality of patches; the computing device (330) can perform calculations necessary for the operation of the crack detection device regionally dividing cracks in each of the plurality of patches. The computing device (330) can perform calculations necessary for the operation of the crack detection device merging each patch in which cracks have been regionally divided to generate a merged image. The computing device (330) can perform calculations necessary for the operation of the crack detection device globally dividing cracks in the merged image. The computing device (330) may be a device such as a processor, an AP (Application Processor), or a chip with an embedded program that processes data and performs certain calculations. For example, the computing device (330) may include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit), etc. The computing device (330) may generate a control signal to control the crack detection device (300). The computing device (330) may generate a control signal to control the input device (310), storage device (320), output device (340), interface device (350), and communication device (360) included in the crack detection device (300).

[0051] The output device (340) may be a device that outputs certain data, information, and models. The output device (340) may be a device that outputs certain data, information, and models outside the crack detection device (300). The output device (340) may output interfaces, input data, analysis results, etc., necessary for the data processing process. The output device (340) may include a device that outputs data, etc., through tactile, visual, auditory, gustatory, and olfactory methods. The output device (340) may be implemented in various physical forms, such as a display, speaker, vibration motor, or document output device. The output device (340) may output data, information, or models, etc., stored in the storage device (320). The output device (340) may output data, information, and models, etc., generated during the process of calculation by the computing device (330). The output device (340) may output the results calculated by the computing device (330).

[0052] The interface device (350) may be a device that receives certain commands and data from the outside. The interface device (350) may receive control signals for controlling the crack detection device (300). The interface device (350) may output the results analyzed by the crack detection device (300). The interface device (350) may receive information necessary to perform the aforementioned crack detection method from a physically connected input device or an external storage device.

[0053] The communication device (360) can receive information necessary for performing the aforementioned crack detection method. The communication device (360) can receive a model necessary for performing the aforementioned crack detection method. The communication device (360) can transmit and receive structure images, patch images, and merged images. The communication device (360) can transmit and receive a first model and a second model. The communication device (360) can receive control signals necessary for controlling the crack detection device (300). The communication device (360) can transmit the results analyzed by the crack detection device (300). The communication device (360) may refer to a configuration that receives and transmits certain data, information, and models, etc., through a wired or wireless network. The communication device (360) can perform network communication such as Wi-Fi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, UWB (Ultra-Wide Band) or NFC (Near Field Communication), USB (Universal Serial Bus), or HDMI (High Definition Multimedia Interface), LAN (Local Area Network), etc.

[0055] The aforementioned crack detection method can be implemented as a program (or application) including an executable algorithm that can be executed on a computer.

[0056] The above program may be provided by storing it on a transitory or non-transitory computer-readable medium.

[0057] The above-mentioned temporary readable medium refers to various types of RAM such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synclink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0058] The above-mentioned non-transient readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short moment, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided on a non-transient readable medium such as a CD, DVD, hard disk, Blu-ray disc, USB, memory card, ROM (read-only memory), PROM (programmable read-only memory), EPROM (Erasable PROM, EPROM), EEPROM (Electrically EPROM), or flash memory.

[0060] Below, one of the embodiments of the crack detection method is implemented, and the results of evaluating the performance of the implemented crack detection method are examined.

[0061] Figure 4 shows the overall process of splitting a crack using the method of the embodiment.

[0062] In the embodiment, the first model and the second model are U-Net-based models.

[0063] Pixel accuracy, mean squared error (MSE), intersection over union (IoU), and dice score were used as metrics to evaluate the performance of crack detection methods.

[0064] Pixel accuracy represents the ratio of correctly classified pixels out of the total number of pixels. Pixel accuracy can be calculated using Equation 1.

[0065]

[0066] In mathematical formula 1 is the (i, j)-th pixel of the correct answer value. In Equation 1 is the (i, j)-th pixel of the predicted value. In Equation 1, H represents the height of the image and W represents the width of the image. In Equation 1 is an indicator function that returns 1 if the correct answer value matches the predicted value, and 0 otherwise.

[0067] MSE is calculated by squaring the difference between the predicted value and the correct value and finding the average. MSE can be calculated using Equation 2.

[0068]

[0069] IoU is an indicator of how much the predicted region and the correct region overlap in 2D data. IoU can be calculated using Equation 3.

[0070]

[0072] In mathematical formula 3 is the intersection between the correct answer value and the predicted value, that is, the overlapping pixels. In Equation 3 is the union between the correct answer value and the predicted value.

[0073] The Dice Score is an indicator that evaluates overlapping regions similar to IoU. The Dice Score is primarily used to measure the similarity between two sets in binary partitioning problems. The Dice score can be calculated using Equation 4.

[0074]

[0075] In the example, the dataset used is the Concrete Crack Segmentation Dataset released by Kaggle. The total dataset consists of 257 items. The entire dataset was divided into training, validation, and evaluation datasets in a ratio of 7:2:1.

[0076] Figure 5 shows the result of comparing the case using a simple U-Net-based model with the case using the method of the embodiment.

[0077] It can be confirmed that the method of the example shows improved performance when compared to U-Net in terms of image segmentation performance. While the pixel accuracy of U-Net was 0.9971, the method of the example was 0.9976, an improvement of 0.02%. Accordingly, it can be confirmed that the proportion of correctly classified pixels among the total pixels has increased. The MSE decreased by 7.69% from 0.0026 to 0.0024, indicating a significant reduction in the error between the predicted value and the actual value. The IoU (Intersection over Union) improved by 2.83% from 0.7533 to 0.7746, indicating an increased overlap ratio between the predicted crack area and the actual crack area. The Dice Score increased by 1.60% from 0.8508 to 0.8644, confirming that the similarity between the predicted area and the actual area in crack detection has been strengthened.

[0078] Figure 6 shows the results of comparing the case using a simple U-Net-based model with the case using the method of the embodiment.

[0079] It can be confirmed that the method of the embodiment detects cracks more accurately and continuously compared to U-Net. Furthermore, it can be confirmed that the method of the embodiment finely subdivides cracks with less error compared to U-Net. Through this, utilizing the method of the embodiment can provide high reliability and accuracy in crack detection and the maintenance of construction structures.

[0081] The embodiments and drawings attached to this specification merely clearly illustrate a part of the technical ideas included in the aforementioned technology, and it is self-evident that variations and specific embodiments that can be easily inferred by a person skilled in the art within the scope of the technical ideas included in the specification and drawings of the aforementioned technology are all included within the scope of the rights of the aforementioned technology.

Claims

Claim 1 A crack detection method comprising: a step in which a device acquires an image of a structure; a step in which the crack detection device divides the acquired image of the structure into a plurality of patches; a step in which the crack detection device regionally divides a crack in each of the plurality of patches; a step in which the crack detection device merges each of the patches in which the crack was regionally divided to generate a merged image; and a step in which the crack detection device globally divides a crack in the merged image. Claim 2 A crack detection method according to claim 1, wherein the region division is performed through a first model, and the first model is a model that receives one of the plurality of patches as input and performs the division. Claim 3 A crack detection method according to paragraph 2, wherein the first model is a model having an encoder and a decoder structure, the encoder of the first model is a model that extracts features from an image, and the decoder of the first model is a model that reconstructs an image based on the features extracted by the encoder. Claim 4 In paragraph 3, the crack detection method, wherein the first model is a U-Net-based model. Claim 5 A crack detection method according to claim 1, wherein the global segmentation is performed through a second model, and the second model is a model that receives the merged image as input and performs the segmentation. Claim 6 A crack detection method according to claim 5, wherein the second model is a model having an encoder and a decoder structure, the encoder of the second model is a model that extracts features from an image, and the decoder of the second model is a model that reconstructs an image based on the features extracted by the encoder. Claim 7 In paragraph 6, the above-mentioned second model is a crack detection method that is a U-Net-based model. Claim 8 A crack detection method according to claim 1, wherein the merged image is an image having the same size as the structure image. Claim 9 A crack detection device comprising: a storage device including a computing device and instructions that cause a crack detection device to perform operations when executed by the computing device; wherein the operations include: an operation in which the device acquires a structural image; an operation in which the crack detection device divides the acquired structural image into a plurality of patches; an operation in which the crack detection device localizes cracks in each of the plurality of patches; an operation in which the crack detection device merges each of the patches in which the cracks have localized cracks to generate a merged image; and an operation in which the crack detection device globalizes cracks in the merged image. Claim 10 A computer-readable recording medium storing a program for executing the crack detection method described in paragraph 1.

Images