Method and System for Classifying Cracks in Concrete Bridge Components Based on Deep Learning Models

Abstract

The present invention relates to a method and system for classifying cracks in concrete bridge components based on a deep learning model. More specifically, the method for classifying cracks in concrete bridge components based on a deep learning model according to the present invention may include the steps of: inputting a concrete bridge image into an encoder; obtaining a multi-scale feature map for the concrete bridge image from the encoder; processing the obtained multi-scale feature map as input to a first module and a second module, respectively; obtaining a first feature map corresponding to the concrete bridge image using the multi-scale feature map in the first module; obtaining a second feature map corresponding to the concrete bridge image using the multi-scale feature map in the second module; processing the first feature and the second feature as inputs to a decoder, respectively; generating a segment mask for the concrete bridge image using the first feature and the second feature in the decoder; and determining the severity of the concrete bridge image using the segment mask.

Description

Technology Field

[0001] The present invention relates to a method and system for classifying cracks in concrete bridge components based on a deep learning model. More specifically, the present invention relates to a method and system for classifying cracks in concrete bridge components based on a deep learning model that can accurately classify the type of crack in bridge components from concrete bridge images. Background Technology

[0002] Recently, as the aging of infrastructure accelerates and social demands for ensuring the safety of facilities increase, the importance of regular maintenance and condition assessment for large structures, such as concrete bridges, is being significantly highlighted. Concrete bridges have been widely used due to their ease of construction and excellent cost-effectiveness; however, they possess structural characteristics that make them vulnerable to aging, climatic stress (temperature changes, humidity, freeze-thaw cycles, etc.), and wear caused by repetitive loading and vibration. These factors have a direct or indirect impact on the structural integrity of bridges and pose a risk that can lead to collapse or loss of function in the long term.

[0003] One of the representative indicators of such structural deterioration is cracking. Cracks occur not only in the bridge superstructure but also in various components such as decks, pavements, and walls; these can shorten the lifespan of the bridge by causing moisture penetration into the structure, rebar corrosion, and fatigue accumulation. Therefore, the technology to detect cracks early and accurately assess their severity has become an essential element in the overall bridge maintenance system.

[0004] Until now, manual visual inspections or traditional image processing-based analysis techniques have been primarily used. However, these methods have limitations, including significant variation in judgment depending on the operator's skill level, time-consuming procedures, and, most importantly, the difficulty in accurately recognizing fine and complex cracks. In particular, since cracks in bridge structures possess diverse directions, widths, lengths, and patterns, and their visibility varies greatly due to lighting changes or background noise, precise classification and quantitative analysis are difficult using existing methods.

[0005] Against this backdrop, crack detection and classification technologies utilizing artificial intelligence, particularly deep learning-based image recognition technology, have recently been actively researched. Various structures such as CNN (Convolutional Neural Network), U-Net, ResNet, and Attention U-Net have been proposed, achieving a certain level of classification accuracy. Deep learning-based technology is attracting attention as an alternative capable of overcoming the limitations of existing methods, as it eliminates subjective human judgment and enables automated processing.

[0006] However, existing deep learning models also have the following structural limitations. First, since most CNN-based models extract only features of a single scale, it is difficult to achieve consistent performance regarding cracks appearing in various sizes and directions. Furthermore, due to the nature of CNNs, they respond only to information from adjacent regions, making it difficult to reflect the relationships between distant cracks or contextual information from the image as a whole. This leads to problems such as fragmentarily recognizing structurally connected cracks or failing to distinguish between the background and the cracks, resulting in a high false positive rate.

[0007] Accordingly, there is a need for deep learning-based crack detection technology that improves learning efficiency and classification precision. The problem to be solved

[0008] The present invention relates to a method and system for classifying cracks in concrete bridge components based on a deep learning model to improve the accuracy of concrete bridge crack classification.

[0009] In addition, the present invention relates to a method and system for classifying cracks in concrete bridge components based on a deep learning model that can overcome the limitations of manual-based bridge inspection.

[0010] In addition, the present invention relates to a method and system for classifying cracks in concrete bridge components based on a deep learning model capable of automating bridge crack classification. means of solving the problem

[0011] To solve the problem described above, the method according to the present invention may include the steps of: inputting a concrete bridge image into an encoder; obtaining a multi-scale feature map for the concrete bridge image from the encoder; processing the obtained multi-scale feature map as input to each of a first module and a second module; obtaining a first feature map corresponding to the concrete bridge image using the multi-scale feature map in the first module; obtaining a second feature map corresponding to the concrete bridge image using the multi-scale feature map in the second module; processing the first feature and the second feature as inputs to a decoder, respectively; generating a segment mask for the concrete bridge image using the first feature and the second feature in the decoder; and determining the severity of the concrete bridge image using the segment mask.

[0012] Furthermore, the step of acquiring the multi-scale feature map may include the step of performing preprocessing on the concrete bridge image, the step of inputting the preprocessed concrete bridge image into a specific block of the encoder, and the step of acquiring a multi-scale feature map for the concrete bridge image from the specific block.

[0013] Furthermore, the specific block can generate a multi-scale feature map for the concrete bridge image by performing a plurality of void coefficient calculations on the concrete bridge image.

[0014] Furthermore, the first module can generate a first feature map corresponding to the concrete bridge image by performing self-attention on the multi-scale feature map to determine the severity of the concrete bridge image.

[0015] Furthermore, the first feature map may include information for distinguishing at least one of a cracked area and a non-cracked area of ​​the concrete bridge image.

[0016] Furthermore, the second module can generate a second feature map corresponding to the concrete bridge image by performing global self-attention on the multi-scale feature map to determine the severity of the concrete bridge image.

[0017] Furthermore, the second feature map may include correlations between cracks in the concrete bridge image.

[0018] Furthermore, the step of determining the severity may include the step of extracting pixels classified as cracks among the segment masks, the step of calculating the crack length using the pixels, and the step of determining the severity of the crack according to the crack length.

[0019] Meanwhile, the method and system for classifying cracks in concrete bridge components based on a deep learning model according to the present invention inputs a concrete bridge image into an encoder, obtains a multi-scale feature map for the concrete bridge image from the encoder, processes the obtained multi-scale feature map as input to a first module and a second module, respectively, obtains a first feature map corresponding to the concrete bridge image using the multi-scale feature map in the first module, obtains a second feature map corresponding to the concrete bridge image using the multi-scale feature map in the second module, processes the first feature and the second feature as inputs to a decoder, respectively, generates a segment mask for the concrete bridge image using the first feature and the second feature in the decoder, and can determine the severity of the concrete bridge image using the segment mask.

[0020] Meanwhile, the program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored on a computer-readable recording medium, and may include instructions for performing the steps of: inputting a concrete bridge image into an encoder; obtaining a multi-scale feature map for the concrete bridge image from the encoder; processing the obtained multi-scale feature map as input to each of a first module and a second module; obtaining a first feature map corresponding to the concrete bridge image using the multi-scale feature map in the first module; obtaining a second feature map corresponding to the concrete bridge image using the multi-scale feature map in the second module; processing the first feature and the second feature as input to a decoder, respectively; generating a segment mask for the concrete bridge image using the first feature and the second feature in the decoder; and determining the severity of the concrete bridge image using the segment mask. Effects of the invention

[0021] As described above, the concrete bridge crack classification method according to the present invention can automatically classify the location and shape of concrete bridge cracks using a deep learning model. Through this, cracks can be consistently diagnosed for various components such as bridge decks, walls, and pavements without relying on human visual inspection.

[0022] In addition, the concrete bridge crack classification method according to the present invention can quantitatively measure the length of cracks by utilizing deep learning-based segmentation results. Through this, maintenance priorities based on the severity of cracks can be determined, and proactive management strategies can be established.

[0023] In addition, the concrete bridge crack classification method according to the present invention can efficiently process multiple concrete images by analyzing concrete bridge cracks using a deep learning model. Through this, the condition of bridge cracks can be evaluated quickly and consistently, thereby ensuring the stability and objectivity of maintenance work. Brief explanation of the drawing

[0024] FIGS. 1, FIGS. 2, FIGS. 3, FIGS. 4 and FIGS. 5 are conceptual diagrams illustrating a method and system for classifying cracks in concrete bridge components based on a deep learning model according to the present invention. FIG. 6 is a flowchart illustrating a method and system for classifying cracks in concrete bridge components based on a deep learning model according to the present invention. FIGS. 7 and 8 are conceptual diagrams illustrating a method and system for classifying cracks in concrete bridge components based on a deep learning model according to the present invention. FIGS. 9, FIGS. 10, FIGS. 11 and FIGS. 12 are formulas related to a deep learning model-based concrete bridge component crack classification method and system according to the present invention. Specific details for implementing the invention

[0025] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not have distinct meanings or roles in themselves. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0026] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0027] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0028] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0029] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0030] The present invention relates to a method and system for classifying cracks in concrete bridge components based on a deep learning model. The method and system for classifying cracks in concrete bridge components based on a deep learning model according to the present invention may be a system that performs precise crack classification using a first module and a second module.

[0031] Furthermore, the deep learning model-based method and system for classifying cracks in concrete bridge components according to the present invention may include a crack classification model. In particular, the present invention aims to provide a crack classification model capable of generating accurate crack classification results by combining a first module and a second module so as to accurately classify crack areas in bridge images.

[0032] In particular, the crack classification model aims to perform accurate classification by recognizing crack patterns in bridge images at various scales and resolutions by including a SAC encoder.

[0033] Hereinafter, the present invention will be examined in more detail with reference to the attached drawings. FIGS. 1, 2, 3, 4, and 5 are conceptual diagrams illustrating a method and system for classifying cracks in concrete bridge components based on a deep learning model according to the present invention. FIG. 6 is a flowchart illustrating a method and system for classifying cracks in concrete bridge components based on a deep learning model according to the present invention. FIGS. 7 and 8 are conceptual diagrams illustrating a method and system for classifying cracks in concrete bridge components based on a deep learning model according to the present invention. FIGS. 9, 10, 11, and 12 are formulas related to a method and system for classifying cracks in concrete bridge components based on a deep learning model according to the present invention.

[0034] As illustrated in FIG. 2, the deep learning model-based concrete bridge component crack classification method and system according to the present invention (hereinafter referred to as the “crack classification system”) may include at least one of a communication unit (210), a storage unit (220), a control unit (230), an input unit (240), a first module (250), a second module (260), and a post-processing unit (270).

[0035] Although not illustrated, the crack classification system according to the present invention may include one or more processors, and such processors may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), neural network processing units (NPUs), application-specific integrated circuits, application-specific semiconductors (ASICs), etc.). One or more processors may be configured to execute instructions, computer-readable instructions, and / or other instructions described herein that are stored (or included) in the storage unit (220) (120). The deep learning model-based concrete bridge component crack classification method and system according to the present invention may perform the data processing described below in cooperation with memory and at least one processor. The processor may perform a series of operations and data processing using data and information stored in memory. In this case, memory may be a component of the storage unit (220).

[0036] Meanwhile, the input unit (240) can be configured in various ways as a means of data input. For example, the input unit (240) can be configured to receive user input. The input unit (240) can be configured to receive user input from a user terminal. Here, “receiving input” may mean receiving an input signal (or selection signal) corresponding to the user’s input based on the input being made by the user through the configuration of the input unit (240) provided in the user terminal. In the present invention, the input unit (240) does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user.

[0037] The input unit (240) may also be referred to as a user interface module. The input unit (240) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of input unit (240).

[0038] Here, user input may include documents, text, images (or videos), voice, etc. In this case, the crack classification system may further include a module that converts voice into text.

[0039] Meanwhile, the communication unit (210) may be connected via a wireless or wired network to a user terminal, device (or device), server (e.g., external server, central server, etc.) and at least one network, and configured to receive or transmit overall data and information necessary for the operation of the crack classification system.

[0040] For example, the communication unit (210) can receive at least one of a bridge image and metadata related to the bridge image (e.g., location (or coordinate) information where the bridge image was taken, shooting time information, etc.) from at least one user terminal (mobile, tablet, laptop, etc.) and an external device (drone, CCTV, etc.).

[0041] The communication unit (210) may include one or more communication modules that enable wireless and / or wired communication between the crack classification system and the communication target. Additionally, the communication unit (210) may include one or more communication modules that connect the crack classification system to one or more networks.

[0042] Furthermore, the communication unit (210) can support various communication methods according to the communication standards of the communicating device.

[0043] For example, the communication unit (210) may be configured to communicate with a communication target using at least one of the following technologies: WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™ RFID (Radio Frequency Identification), Infrared Communication (Infrared Data Association; IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus).

[0044] Next, the storage unit (220) serves to store various data related to the present invention and may include one or more non-transient computer-readable storage media that can be read and / or accessed by at least one of one or more processors.

[0045] One or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage unit (220) may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage device), whereas in other examples, the storage unit (220) may be implemented using two or more physical devices.

[0046] The storage unit (220) may include computer-readable instructions and additional data. The storage unit (220) may include a storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functions of the device and network.

[0047] Furthermore, at least a portion of the storage unit (220) may be a cloud storage or a cloud server. At least a portion of the data corresponding to user input and training data received from the input unit (240) may be stored in the storage unit (220).

[0048] That is, the storage unit (220) is sufficient as a space where information necessary for the operation of the crack classification system (100) according to the present invention is stored, and it can be understood that there are no restrictions on the physical space.

[0049] Next, the first module (250) is for generating an optimized feature representation that includes local information in features (feature or feature information) extracted from a bridge image (110), and may include at least one of Transformer Self Attention (TSA), Cross Attention (CA), Sparse Attention (SA), or Swin Attention (SWA). In this specification, TSA is described as an example of the first module (250), but the invention is not limited thereto. The first module (250) can distinguish between cracked and non-cracked regions of the bridge image (110) by separating a multi-scale feature map (121) generated from an encoder (120) into token vectors and calculating attention scores.

[0050] The second module (260) is for generating an optimized feature representation that reflects global information to features extracted from a bridge image (110), and may include at least one of Global Self Attention (GSA), Cross Attention (CA), Dual Attention (DA), or Spatial Attention (SA). In this specification, GSA is described as an example of the second module (260), but the invention is not limited thereto.

[0051] Next, the control unit (230) can perform the role of controlling the overall operation of the crack classification system related to the present invention. The control unit (230) can process signals, data, information, etc. that are input or output through the components of the crack classification system (100) described above, or perform a series of data processing to provide or process appropriate information and functions to the user. The control unit (230) can be physically implemented by the processor described above.

[0052] The control unit (230) can detect and classify cracks in the bridge image (110) by using (or utilizing) a deep learning model-based concrete bridge component crack classification system for precise detection of the bridge image (110).

[0053] Furthermore, the crack classification system (200) according to the present invention can be implemented through a computing device described below and can perform data processing related to the crack classification method described above.

[0054] Meanwhile, FIG. 3 illustrates an example of a block diagram of a computing system in which the present invention can be implemented.

[0055] Referring to FIG. 3, a computing system (10000) for classifying cracks and measuring the severity of cracks according to one embodiment of the present invention may include at least one computing device. At this time, the at least one computing device may be a single processor or a multiprocessor computing device.

[0056] The components of at least one computing device of the present invention may include various hardware components such as one or more processors, memory, other hardware, and a system bus (not shown) that connects various system components so that they can transmit and receive data to and from each other (e.g., telecommutatively connected, physically connected, electrically connected), and the components of at least one computing device are not limited thereto and may be very diverse.

[0057] Meanwhile, at least one computing device included in the computing system (10000) for classifying cracks and measuring the severity of cracks may be connected to communicate via a network (1070). For example, at least one computing device included in the computing system (10000) may be clustered or part of a local area network (LAN). Additionally, at least one computing device may be part of a wide area network (WAN) or connected to at least one of a client-server network and a peer-to-peer network within the cloud.

[0058] Meanwhile, when at least one computing device is used in at least one of a network environment and a cloud computing environment, the at least one computing device may be connected to at least one of a public and private network through a network interface or adapter. In one embodiment, other communication connection devices, such as a modem, may be used to establish communication through the network. The modem may be at least one of an internal modem and an external modem, and may be connected to a system bus through a network interface or a specific mechanism, etc. A wireless network component consisting of an interface and an antenna may be coupled to the network through a device such as an access point, a peer computer, etc. In the present invention, the method of connecting at least one computing device to communicate through the network (1070) is not limited, and it may be connected to communicate in a manner different from the described example.

[0059] Furthermore, other computer-type devices and / or systems not shown in FIG. 3 may also interact technically with at least one computing device or other system through one or more connections to the network (1070) via a network interface. Here, the network interface may include network interface equipment such as a physical network interface controller (NIC) or a virtual network interface (VIF).

[0060] The network (1070) of the present invention may include various forms such as the Internet, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™ RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), etc., and in the present invention, data transmission may be performed based on standard communication protocols such as TCP / IP, HTTP, SSL, etc.

[0061] A computing system (10000) for classifying cracks and measuring the severity of cracks according to the present invention may include at least one of a user computing device (1010), a training computing system (1050), and a server computing system (1030).

[0062] A user computing device (1010) according to the present invention may be understood as a computing device comprising at least one processor (1011) and memory (1012) for classifying cracks and measuring the severity of cracks. For example, the user computing device (1010) may include at least one computing device among a smartphone, a smart TV, a laptop computer, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, and a head-mounted display).

[0063] At least one processor (1011) constituting the user computing device (1010) may include one or more general-purpose processors and / or one or more special-purpose processors. For example, at least one processor (1011) constituting the user computing device (1010) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), an application integrated circuit, an application semiconductor (ASIC), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or electrical units for performing other functions, or a plurality of electrically connected processors.

[0064] Furthermore, at least one processor (1011) may be configured to execute computer-readable instructions contained in memory (1012) and / or other instructions described herein.

[0065] The memory (1012) constituting the user computing system (1010) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media and / or other types of physically durable storage media.

[0066] For example, memory (1012) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, and combinations thereof, and may include web storage of a server that performs the storage function of memory on the internet. Such memory (1012) may store data and instructions necessary for the at least one processor (1011) to perform the operation of an application for crack classification and measurement of crack severity.

[0067] A user computing device (1010) may include one or more user input components (1021) that detect user input. For example, the user input component (1021) may also be referred to as a user interface module. The user input component (1021) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of user input component (1021). In this case, the user input component (1021) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user. Meanwhile, the user of the present invention may refer to an automated agent, script, playback software, etc., that operates on behalf of one or more people.

[0068] A user can interact with a computing system (10000) including at least one computing device through input text, touch, voice, movement, computer vision, gestures and / or other forms of input / output using a user input component (1021). For example, the user input component (1021) may include one or more of a command line interface (CLI), a graphical user interface (GUI), a natural user interface (NUI), a voice command interface and / or other user interface (UI) representations.

[0069] Between the user input component (1021) and the user computing device (1010), one or more application programming interface (API) calls may be made based on user input received from the user interface and / or network.

[0070] Here, the expression "based on" may be interpreted to include cases where it is based on the use of a specific configuration, modified from, derived from, influenced by, dependent on, or otherwise derived from a specific configuration. In some embodiments, an API call may be configured for a specific API, which may be interpreted or converted into an API call configured for another API. Here, an API may refer to a defined interface or connection between computers or between computer programs.

[0071] In one embodiment, the user computing device (1010) may store at least one machine learning model (1020). For example, the user computing device (1010) may be various machine learning models, such as a plurality of neural networks (e.g., deep neural networks) that perform crack classification and crack severity measurement using concrete bridge images, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0072] According to an embodiment of the present invention, a user computing device (1010) may perform crack classification and measure the severity of cracks using a local or / and external machine learning model (1020). Alternatively, the user computing device (1010) may perform crack classification and measure the severity of cracks using a machine learning model (1040) provided by a server.

[0073] In addition, according to another embodiment of the present invention, a server computing system (1030) communicating with a user computing device (1010) may provide the type and severity of a crack to the user computing device (1010) via an application or / and the web in accordance with a request from a user received through the user computing device (1010).

[0074] In addition, according to another embodiment of the present invention, at least a part of the user computing device (1010) and the server computing system (1030) are interconnected to perform crack classification and crack severity measurement, thereby providing the type of crack and crack severity to the user.

[0075] Additionally, according to various embodiments of the present invention, a user computing device (1010) and / or a server computing system (1030) may learn machine learning models (1020, 1040) performed in a crack classification and crack severity measurement method through interaction with a training computing system (1050) that is communicatedly connected via a network (1070). In this case, the training computing system (1050) may be a computing system separate from the server computing system (1030). Alternatively, in some embodiments, the training computing system (1050) may be part of the server computing system (1030) or part of the user computing device (1010).

[0076] Meanwhile, the server computing system (1030) may include at least one processor (1031) and memory (1032). Here, the processor (1031) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an application integrated circuit, an application semiconductor (ASIC), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions. For example, at least one processor (1031) may include a circuit and a transistor configured to execute instructions from memory (1032).

[0077] The memory (1032) constituting the server computing system (1030) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media, and / or other types of physically durable storage media. For example, the memory (1032) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and combinations thereof, and may include web storage of a server that performs the storage function of memory over the internet. Additionally, the server computing system (1030) may further include a data storage (data store). For example, the data storage may be composed of at least one of a relational database, a NoSQL database, a data warehouse, and a local file system.

[0078] In the memory (1032) constituting the server computing system (1030) according to the present invention, data and instructions necessary for the at least one processor (1031) to perform the operation of an application for crack classification and crack severity measurement may be stored.

[0079] In one embodiment, the server computing system (1030) may be composed of a single device or a plurality of computing devices, and may be configured to operate according to a sequential or parallel computing architecture. Additionally, a distributed processing system may be configured with a plurality of networked devices.

[0080] Meanwhile, the training computing system (1050) may include at least one processor (1051) and memory (1052). The model trainer (1060) is a logical component that executes the training of at least one machine learning model (1020, 1040) and may be implemented in the form of hardware, firmware, or software. For example, the model trainer (1060) may be executed by the processor (1051) after loading training data (1061) stored in a storage device into memory (1052). For example, the model trainer (1060) may be configured to execute one or more operations (e.g., model training, model reconstruction, model validation, model testing) on ​​at least one machine learning model.

[0081] The machine learning model of the present invention may include at least one of a statistical model, an algorithm, a neural network (NN), a convolutional neural network (CNN), a generative neural network (GNN), a Word2Vec model, a Bag of Words model, a TF-IDF (document frequency-inverse document frequency) model, a GPT (Generative Pre-trained Transformer) model (or other autoregressive models), a PPO (Proximal Policy Optimization) model, a nearest neighbor model (e.g., a k-nearest neighbor model), a linear regression model, a K-means clustering model, a Q-learning model, a TD (Temporal Difference) model, a Deep Adversarial Network model, and all other types of models further described herein.

[0082] Specifically, the model trainer (1060) may execute operations to train a machine learning model, and said operations may include at least one of adding, removing, and modifying model parameters. At this time, the training of the machine learning model may be at least one of supervised learning, semi-supervised learning, and unsupervised learning. In one embodiment, the training of the machine learning model may include the step of repeatedly inputting training data (1061) based on epochs and repeatedly performing the machine learning model training process configured in this way. Here, an epoch may refer to a unit in which the entire set of training data (1061) undergoes forward and backpropagation processing once. In some implementations, different levels of training methods (e.g., supervised learning, semi-supervised learning, unsupervised learning) may be used for different epochs.

[0083] The training data (1061) of the present invention may include input data and / or data previously output from at least one machine learning model (e.g., recursive learning feedback).

[0084] At least one parameter of a machine learning model may include at least one of a seed value, a model node, a model layer, an algorithm, a function, connections between different machine learning models, connections between parameters, machine learning model constraints, and other digital components that influence the output of the machine learning model. In this case, model connections between different machine learning models may include or represent relationships between model parameters and / or models, which may be dependent or interdependent, hierarchical, and / or static or dynamic. The combinations and configurations of model parameters described herein may be too complex to be maintained or utilized by human cognitive abilities.

[0085] In the present invention, the machine learning parameters described according to the embodiments are not limited, and a single machine learning model may further include a plurality of model parameters.

[0086] Meanwhile, FIG. 4 illustrates an example of a block diagram of a computing device (1100) that may be included in a user computing device (1010), a server computing system (1030), and a training computing system (1050), as an embodiment of a computing system (10000) in which the present invention can be implemented.

[0087] As illustrated in FIG. 4, the computing device (1100) may include at least one application (e.g., Application 1 to Application N), and each of the at least one application may include a machine learning library and a model execution environment for performing machine learning-based crack classification and measuring the severity of the crack. The at least one application included in the computing device (1100) may communicate with the sensor, context manager, device state manager, or additional component(s) within the computing device (1100) via an Application Programming Interface (API). In one embodiment, the at least one application may interface with device components, such as receiving sensor data or state data via a public or private API, or transmitting prediction results to an output device.

[0088] Meanwhile, FIG. 5 illustrates an example of a block diagram in another aspect of a computing device (1200), which is one of the components of a computing system (10000) that performs crack classification and measurement of crack severity according to an embodiment of the present invention.

[0089] A computing device (1200) according to the present invention may include at least one application (e.g., Application 1 to Application N), and at least one application may communicate with a central intelligence layer (1210). Each application may interact with a shared model within the central intelligence layer (1210) through an API (e.g., a common API).

[0090] The central intelligence layer (1210) includes one or more machine learning models and may share them among multiple applications or provide them independently to each. In one embodiment, the central intelligence layer (1210) may be integrated as part of an operating system or implemented as a separate logical layer.

[0091] Additionally, the central intelligence layer (1210) can communicate with the central device data layer (1220). The central device data layer (1220) can integrate and store bridge crack information, etc., necessary for classifying and measuring the severity of cracks stored within the computing device (1200), and provide this as input data necessary for classifying and measuring the severity of cracks. Each device component (e.g., sensor, state manager, etc.) can communicate with the central device data layer (1220) through a private API, etc.

[0092] The technology described herein may be composed of a single or multiple computing devices, and a machine learning model that performs crack classification and crack severity measurement may be executed sequentially or in parallel on one or multiple distributed components. Data storage, machine learning models, and applications may be distributed and operated locally or over a network, and these configurations can be flexibly applied to various system architectures.

[0093] First, the crack classification system according to the present invention processes a concrete bridge image as input to an encoder (S610, see FIG. 6), and the process of obtaining a multiscale feature map for the concrete bridge image from the encoder may proceed (S620, see FIG. 6).

[0094] As illustrated in FIG. 1, the crack classification system may receive a concrete bridge image (110) from an external device. The concrete bridge image (110) may refer to an image taken of the surface of major structural parts of a bridge, such as a deck, wall, or pavement. The concrete bridge image (110) may include cracks and may include not only the areas where cracks have occurred but also normal surfaces where no cracks exist. The concrete bridge image (110) may be configured to include surface information in various states so that precise identification of the presence of cracks is possible during the subsequent feature extraction and classification process, and the concrete bridge image (110) may be referred to in this specification as 'concrete image', 'bridge image', 'concrete crack image', or 'bridge crack image'.

[0095] Furthermore, the crack classification system according to the present invention may apply subsequent filtering after performing a preprocessing process on the bridge image (110). The preprocessing process may include at least one of techniques such as rotation, scaling, brightness change, position shift, and grayscale conversion on the bridge image (110), and a filtering method including at least one of a Sobel filter, an Otsu threshold filter, a Canny edge filter, and a Chanvese filter may be performed on the preprocessed bridge image (110).

[0096] In this specification, a configuration including a Sobel filter is described as one embodiment as a filtering method for a bridge image (110), but the present invention is not limited thereto.

[0097] Furthermore, the crack classification system according to the present invention may be composed of three encoder (120) and decoder (150) blocks, and the operation method of the encoder (120) block may include at least one of a convolution operation, a Switchable Atrous Convolution (SAC) operation, a Multi-scale Convolution operation, a Channel Attention Mechanism operation, and a Spatial Attention Mechanism operation. Although the SAC operation is described as one embodiment in this specification, the present invention is not limited thereto.

[0098] The control unit (230) can process the filtered bridge image (hereinafter referred to as the bridge image) as input to the encoder (120) to generate a multi-scale feature map (121) for the bridge image (110).

[0099] Here, a multi-scale feature map (121) is configured by combining multiple feature maps extracted at different resolutions from an input image in parallel, and each feature map may include different levels of visual characteristics of the image, such as edges, textures, shapes, and semantic object information. At this time, the multi-scale feature map (121) may be configured to obtain information regarding detailed structures from high-resolution feature maps and abstract semantic-based information from low-resolution feature maps, respectively.

[0100] More specifically, referring to FIG. 7, the encoder (120) may include at least one of a pre-context block (410), a post-context block (430), and an SAC block (420), and the pre-context block (410) and the post-context block (430) may each include at least one convolutional block without a non-linear activation function.

[0101] The control unit (230) can process the bridge image (110) as an input to the pre-context block (410). The pre-context block (410) can provide basic information for generating a multi-scale feature map (121) through an importance determination based on global information regarding the bridge image (110). The pre-context block (410) is composed of a convolution with a kernel weight of 1 and can perform preprocessing operations on the bridge image (110). The pre-context block (410) can perform a global average pooling operation on the bridge image (110) to calculate a global average value for the bridge image (110), and perform a convolution operation on the global average value to perform preprocessing operations on the features (see FIG. 9 (a)).

[0102] In this case, the kernel weights of the convolution can be represented as K, and the height, width, and number of channels of the bridge image as H, W, and C, respectively.

[0103] The control unit (230) can process the output of the pre-context block (410) as input to the SAC block. The encoder (120) can effectively detect cracks of various sizes by including pore dilation convolution operations of various sizes using SAC operations, and can minimize loss of resolution and accurately identify detailed features of the cracks by performing multiple pore operations.

[0104] For example, the SAC block (420) may include at least one of a plurality of convolutional operation paths (423, 424) having dilation rates of 1 and 3, respectively.

[0105] The convolutional operation paths (423, 424) having void coefficients of 1 and 3, respectively, can perform convolutional operations with void coefficients of 1 and 3, respectively, for the same input value.

[0106] Here, performing a convolution operation with gap coefficients of 1 and 3 can be understood as performing a convolution operation on an input feature map through kernels with filter application intervals set to 1 and 3, respectively, and generating a feature map with different receptive fields as a result.

[0107] For example, if a convolution operation with a void coefficient of 1 is performed on the bridge image (110), a feature map containing fine features within adjacent regions, such as crack boundaries or edges, can be obtained, and if a convolution operation with a void coefficient of 3 is performed on the bridge image (110), a global feature representation reflecting structural associations between separated or widely distributed cracks can be obtained.

[0108] That is, the SAC block (420) can be understood as an operation block that can adaptively respond to cracks of various sizes and shapes by performing multiple convolution operations with different acceptance regions in parallel and selectively reflecting the results.

[0109] More specifically, the SAC block (420) may apply 5×5 average pooling to the feature map (see (d) and (e) of FIG. 9) and apply 1×1 convolution (see (f) of FIG. 9).

[0110] Furthermore, the SAC block can generate two gap feature maps by simultaneously performing a 3×3 kernel filter and a gap convolution on the feature map, with gap coefficients (dilated rate or atrous rate) of 1 and 3, respectively (see FIG. 10 (a) and (b)). The two generated gap feature maps can each be combined with the result of the 1×1 convolution operation and then merged to generate an integrated feature map (see FIG. 10 (c), (d), and (e)).

[0111] The above integrated feature map is input into a post-context block with a kernel weight of 1 to perform Global Average Pooling (GAP), and a 1×1 convolution operation is performed on the result of the GAP to finally generate a multi-scale feature map (121) for the bridge image (110) (see (f), (g), and (h) of FIG. 10).

[0112] That is, the SAC encoder (120) can be understood as being for acquiring multi-scale spatial data by detecting cracks at various resolutions and multiple scales (or sizes), and for generating a multi-scale feature map (121) for a bridge image (110) without increasing computational parameters by applying various void coefficients.

[0113] Meanwhile, the crack classification system according to the present invention may process the acquired multi-scale feature map (121) as an input to each of the first module and the second module (S630, see FIG. 6), generate a first feature map and a second feature map using the first module and the second module (S640, S650, see FIG. 6), process the first feature map and the second feature map as an input to a decoder (S660, see FIG. 6), and generate a segment mask using the first feature map and the second feature map (S670, see FIG. 6).

[0114] Referring to FIG. 1, the control unit (230) can process the multi-scale feature map (121) as an input to the first module (250) to generate a first feature map (134) for the bridge image (110).

[0115] The first module (250) may be a module for distinguishing between cracked and non-cracked regions of a bridge image (110) by performing self-attention. Here, self-attention may refer to an operation that calculates the correlation between each element of the bridge image (110) and all other elements and generates an output that reflects weights.

[0116] Additionally, the first feature map (134) may refer to a feature map that includes a feature representation reflecting semantic importance based on the interaction between the positions of the feature map.

[0117] For example, among multiple regions having the same visual form, higher attention weights may be assigned to more important regions to include emphasized feature information.

[0118] That is, the first feature map (134) can be understood as a feature map for distinguishing between the cracked area and the non-cracked (or background) area of ​​the bridge image (110).

[0119] Additionally, the first module (250) may include at least one of a Transformer Self Attention (TSA) module, a Window-based Self Attention module, a Non-local Attention module, and a Performer module. Although the TSA is described as one embodiment in this specification, the invention is not limited thereto.

[0120] The first module (250) can calculate Query, Key, and Value vectors by applying a previously learned weight matrix to each of the multi-scale feature maps (121) (see FIG. 11 (a)), and the weight matrices can be represented as W_Q ?, W ?_K ?, W ?_V, respectively. Additionally, the Query, Key, and Value can be represented as q (131), k (132), and v (133), respectively.

[0121] Here, applying weights can be understood as determining the reflection degree of a feature by evaluating the importance of the multi-scale feature map (121) and performing an operation to apply weights to each feature map according to the importance.

[0122] The first module (250) can calculate an attention score to evaluate the correlation between each location of the bridge image (110) based on the query, key, and value (see (b) of FIG. 11).

[0123] Furthermore, the attention score can be calculated as an attention weight through a softmax function to probabilistically express the importance between each position (see (c) in FIG. 11), and a first feature map (134) can be generated by multiplying the attention weight and the value (see (e) in FIG. 11).

[0124] The control unit (230) can generate a second feature map (144) by processing the multi-scale feature map (121) as an input to the second module (260).

[0125] The second module (260) may be a module for generating a feature map including correlations between cracks in a bridge image (110) by performing global self-attention.

[0126] Additionally, the second feature map (144) may refer to a feature map that includes the relationship between features. For example, the second feature map (144) may include information regarding similarity between crack patterns that are far apart from each other, or information regarding structurally related image regions. That is, the second feature map (144) can be understood as a feature map that includes information that allows determining whether the cracks are connected even if they are broken or far apart.

[0127] More specifically, the second module (260) can calculate a query, key, and value using a weight matrix learned for the multi-scale feature map (121), and the query, key, and value can be represented as Q (142), R (143), and P (141), respectively.

[0128] The second module (260) can calculate a context score based on the query, key, and value (see (f) in FIG. 11).

[0129] Furthermore, the above context score is calculated as a context weight through a softmax function, and a second feature map (144) can be generated through the multiplication of the context weight and the value (see (i) in FIG. 11).

[0130] The control unit (230) can process the first feature map (134) and the second feature map (144) generated through the first module (250) and the second module (260) as inputs to the decoder.

[0131] For example, the control unit (230) can input the second feature map (144) into a decoder block and perform upsampling (e.g., Bilinear upsampling) on ​​the second feature map (144) (see (a) of FIG. 12).

[0132] At this time, the upsampling process of the bridge classification system can be performed by utilizing skip connections between the encoder (120) and the decoder block.

[0133] Here, skip connection can be understood as being configured between the encoder (120) and the decoder block to improve crack segmentation performance by integrating the upsampled feature map with various semantic levels. The skip connection can integrate semantic information of various sizes through operations such as upsampling, convolution, and concatenation.

[0134] The upsampling process for the second feature map (144) can expand the size of the second feature map (144) by a specified scale factor based on the target output resolution for the second feature map (144) (see FIG. 12 (b)). At this time, Hin and Win are the height and width of the second feature map (144), and SF is the scale factor.

[0135] Furthermore, the upsampled second feature map (144) can be normalized with a softmax function (see (c) and (d) of FIG. 12), and a final feature map can be generated by merging the normalized second feature map (144) with the second feature map (144) (see (e) of FIG. 12).

[0136] Furthermore, the final output layer of the decoder may be composed of a plurality (6) of neurons, and may classify cracks or non-cracks in the walls, decks, and pavements of the bridge image (110), and generate a segment mask (160) for the bridge image (110) using the final feature map.

[0137] Meanwhile, the crack classification system according to the present invention may proceed with a process of determining the severity of the bridge image using the segment mask (S680, see FIG. 6).

[0138] Referring to FIG. 8, the control unit (230) can process the generated segment mask (160) as an input to the post-processing unit (270).

[0139] Here, the segment mask (160) can be understood as an image that visually represents the result of classifying whether there is a crack for each pixel of the bridge image (110), and may include a crack area classified as at least one of wall cracks, deck cracks, and pavement cracks.

[0140] The above post-processing unit (270) can extract pixels corresponding to classes classified as cracks (wall, deck, pavement) from the segment mask (160). The total crack length can be calculated by performing distance calculations between the extracted crack pixels.

[0141] Finally, the severity of the bridge image (110) can be evaluated based on the calculated crack length.

[0142] For example, it can be configured to enable severity assessment for bridge maintenance decisions by determining a crack as a severe crack if the calculated crack length is greater than or equal to a preset threshold value, and classifying it as a minor or moderate crack if the calculated crack length is less than or equal to the threshold value.

[0143] Meanwhile, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a computer-readable medium (or recording medium).

[0144] Furthermore, the present invention described above can be implemented as computer-readable code or instructions on a medium on which a program is recorded. That is, the present invention can be provided in the form of a program.

[0145] Meanwhile, computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0146] Furthermore, the computer-readable medium may be a server or cloud storage that includes a storage and is accessible to an electronic device via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage via wired or wireless communication.

[0147] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, namely a CPU (Central Processing Unit), and no special limitations are placed on its type.

[0148] Meanwhile, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

Claim 1 A deep learning model-based method for classifying cracks in concrete bridge components, characterized by comprising: a step of inputting a concrete bridge image into an encoder; a step of obtaining a multi-scale feature map for the concrete bridge image from the encoder; a step of processing the obtained multi-scale feature map as an input to each of a first module and a second module; a step of obtaining a first feature map corresponding to the concrete bridge image using the multi-scale feature map in the first module; a step of obtaining a second feature map corresponding to the concrete bridge image using the multi-scale feature map in the second module; a step of processing the first feature and the second feature as inputs to a decoder, respectively; a step of generating a segment mask for the concrete bridge image using the first feature and the second feature in the decoder; and a step of determining the severity of the concrete bridge image using the segment mask. Claim 2 A method for classifying cracks in concrete bridge components based on a deep learning model according to claim 1, wherein the step of acquiring the multi-scale feature map comprises: a step of performing preprocessing on the concrete bridge image; a step of inputting the preprocessed concrete bridge image into a specific block of the encoder; and a step of acquiring a multi-scale feature map for the concrete bridge image from the specific block. Claim 3 A deep learning model-based method for classifying cracks in concrete bridge components, characterized in that, in paragraph 2, the specific block is configured to perform a plurality of void coefficient calculations on the concrete bridge image to generate a multi-scale feature map for the concrete bridge image. Claim 4 A deep learning model-based method for classifying cracks in concrete bridge components, characterized in that, in claim 1, the first module performs self-attention on the multi-scale feature map to determine the severity of the concrete bridge image and generates a first feature map corresponding to the concrete bridge image. Claim 5 A deep learning model-based method for classifying cracks in concrete bridge components, wherein, in claim 4, the first feature map includes information for distinguishing at least one of a cracked area and a non-cracked area of ​​the concrete bridge image. Claim 6 A deep learning model-based method for classifying cracks in concrete bridge components, wherein, in claim 1, the second module is configured to perform Global Self-attention on the multi-scale feature map to determine the severity of the concrete bridge image and to generate a second feature map corresponding to the concrete bridge image. Claim 7 A deep learning model-based method for classifying cracks in concrete bridge components, characterized in that, in claim 6, the second feature map includes correlations between cracks in the concrete bridge image. Claim 8 A method for classifying cracks in concrete bridge components based on a deep learning model according to claim 1, wherein the step of determining severity comprises: a step of extracting pixels classified as cracks among the segment masks; a step of calculating the crack length using the pixels; and a step of determining the severity of the crack according to the length of the crack. Claim 9 A deep learning model-based method and system for classifying cracks in concrete bridge components, wherein the system inputs a concrete bridge image into an encoder, obtains a multi-scale feature map for the concrete bridge image from the encoder, processes the obtained multi-scale feature map as input to a first module and a second module, respectively, obtains a first feature map corresponding to the concrete bridge image using the multi-scale feature map in the first module, obtains a second feature map corresponding to the concrete bridge image using the multi-scale feature map in the second module, processes the first feature and the second feature as inputs to a decoder, respectively, generates a segment mask for the concrete bridge image using the first feature and the second feature in the decoder, and determines the severity of the concrete bridge image using the segment mask. Claim 10 A program stored on a computer-readable recording medium that is executed by one or more processes in an electronic device, the program comprising instructions for performing the steps of: inputting a concrete bridge image into an encoder; obtaining a multi-scale feature map for the concrete bridge image from the encoder; processing the obtained multi-scale feature map as input to a first module and a second module, respectively; obtaining a first feature map corresponding to the concrete bridge image using the multi-scale feature map in the first module; obtaining a second feature map corresponding to the concrete bridge image using the multi-scale feature map in the second module; processing the first feature and the second feature as input to a decoder, respectively; generating a segment mask for the concrete bridge image using the first feature and the second feature in the decoder; and determining the severity of the concrete bridge image using the segment mask.

Images