A system comprising a mobile application for post-earthquake damage assessment in reinforced concrete structural members

Abstract

After earthquakes, buildings experience damages at different levels depending on various parameters. The extent of damage is determined based on observations such as cracks in structural elements, material crushing, and reinforcement damages. Decisions made are crucial for ensuring human safety and preventing economic losses. The invention pertains to a system which comprises a mobile application containing an automatic decision support system capable of assisting experts conducting field observations after earthquakes and performing damage classification at the element level.

Description

[0001] A SYSTEM COMPRISING A MOBILE APPLICATION FOR POST-EARTHQUAKE DAMAGE ASSESSMENT IN REINFORCED CONCRETE STRUCTURAL

[0002] MEMBERS

[0003] TECHNICAL FIELD

[0004] After earthquakes, buildings experience damages at various levels depending on various parameters. The extent of damage is determined based on observations such as cracks in structural elements, material crushing, and reinforcement damages. Decisions made regarding these observations are crucial for ensuring human safety and preventing economic losses. The invention pertains to a system which comprises a mobile application containing an automatic decision support system capable of assisting experts conducting field observations after earthquakes and performing damage classification at the element level.

[0005] BACKGROUND

[0006] Artificial intelligence can be defined as the process by which certain cognitive activities performed by humans are learned by computers through coding, analyzed, and then adapted to new inputs they haven't encountered before, enabling them to learn autonomously.

[0007] Deep learning is a subset of machine learning that involves deep neural network models. One of the most popular algorithms in machine learning, deep learning requires a large amount of data for high performance. Deep learning, which is used in problem-solving across various fields, has been successfully employed in tasks such as classification (objects, words, emails), segmentation (objects, scenes), recognition (faces, objects, speech), detection (faces, diseases, behaviors, objects), prediction (weather, traffic), recommendation (products and services), among others. Additionally, in the field of computer vision, deep learning can autonomously learn features of images and videos without the need for external intervention, distinguishing it from traditional machine learning techniques.

[0008] Deep learning algorithms are implemented through Convolutional Neural Networks (CNN), which have advanced network structures in computer vision. The data type in convolutional neural networks is generally visual, and images are typically obtained from digital cameras, smartphones, or online websites. AIM OF THE INVENTION

[0009] The aim of the invention is to classify the earthquake damages of reinforced concrete elements using an innovative deep learning-based system integrated into a mobile application. Towards this goal, images of 2455 damaged elements exposed to various destructive earthquakes have been collected. The algorithm described in the invention has been applied to the images, and pretrained Convolutional Neural Network (CNN) algorithms (VGG16, ResNet-50, MobileNetV2, and EfficientNet) have been compared based on performance metrics. While other models achieved a maximum of 89% test accuracy, the deep learning algorithm proposed in the invention achieved a test accuracy of 92% in damage classification.

[0010] Rapid and accurate identification of the type of damage after an earthquake, along with minimizing human errors, is crucial for national economies. In this context, another aim of the invention is to provide significantly more support to evaluators' decision-making processes through the proposed system.

[0011] With the advancement of artificial intelligence, there is a significant potential for improving the accuracy and efficiency of current procedures for assessing building structures after earthquakes. The invention aims to classify the damages of reinforced concrete elements using Convolutional Neural Networks-based software specific to computer vision. In the software within the system, the damage classification criteria proposed by llki et al. (2021) have been utilized. However, the system can also be used based on any other classification criteria.

[0012] LIST OF FIGURES

[0013] Figure 1 . Schematic View of the Network Architecture

[0014] DETAILED DESCRIPTION OF THE INVENTION

[0015] The system which comprises a device further comprising a mobile application which is subject to invention has three stages:

[0016] 1- Photographing the damages occurring in field reinforced concrete structural elements after earthquakes worldwide and classifying them according to damage groups,

[0017] 2- Developing a deep learning algorithm capable of classifying the severity of damage based on the photographs subjected to image processing algorithms, 3- Enabling the usage of this process on portable devices through the application operated by the developed algorithm.

[0018] The classification criteria proposed by llki et al. (2021 ) have been considered for use in the system within the mobile application. However, these criteria are not binding, and any classification criteria can be used. For the sake of clarity and supporting the explanation within the invention, the description has been provided using the aforementioned classification criteria.

[0019] The system comprising the device within the invention will utilize a dataset for comparing photographs. The dataset comprises of examples of damaged structural elements, obtained from photographs of twenty severely damaged elements from the "datacenterhub" database following earthquakes from around the world. However, this dataset can be replaced with another dataset for the invention, as it is not binding. This example dataset consisting of 2455 images from the "datacenterhub" database containing images of structural elements at various levels of damage was provided to illustrate the effectiveness and operational principle of the invention. The images pertain to column, beam, column-beam connection, and shear wall structural elements.

[0020] With the system described in the invention, structural damage images obtained from field studies conducted after earthquakes were examined, and the damages occurring only in structural elements with a reinforced concrete load-bearing system were evaluated. Using a dataset related to the structural elements intended for analysis with the algorithm contained in the system, it is possible to obtain assessments regarding the desired parts of the structures. However, in terms of accuracy of detection, when evaluating damage types such as cracks in structural elements, only damages occurring on the concrete surface have been taken into account, and cracks occurring on plaster or paint have been excluded. Additionally, in the damage detection method, since the width of cracks occurring on the surface of reinforced concrete elements due to earthquake effects is an important parameter in determining the severity of damage, data is recorded in which crack widths has been particularly emphasized in images where micro cracks are present. Therefore, images where cracks were measured with a crack gauge have been considered. These images obtained are high-resolution images acquired through devices such as digital cameras or smartphone cameras.

[0021] In order to facilitate the detection of damages by artificial intelligence algorithms, the areas where damages are concentrated in the images of elements containing visible damages have been cropped by zooming in on the images in the dataset that will be compared to the displayed damaged elements. After the preprocessing steps, a new dataset consisting of a total of 2455 images, with 491 images for each damage level (for example, 0-A-B-C-D classes), has been created. As mentioned above, the content of damage level classification here is variable and is not a restrictive element of the invention. The number of classifications can be increased or decreased, and the number of images used for each class can also be modified. These numbers do not have a technical effect. What matters is that the algorithm responsible for detecting damage in the invention should contain classes that the algorithm can compare photographs with and classify damages in the photographs according to their severity levels, along with having a dataset available for this purpose.

[0022] According to the damage classification criteria in the damage detection algorithm: if there is no crack or compression damage, it is labeled as 0; if the crack width is less than 0.5mm, it is labeled as A; if the crack width is less than 3mm and there are compressive damages in the compression zone, it is labeled as B; if cracks larger than 3mm have occurred or there are spalls in the cover concrete and longitudinal reinforcements are exposed, it is labeled as C; and if the core concrete is crushed and longitudinal reinforcements are buckled, the damages are labeled as group D. Therefore, the algorithm classifies the captured images based on this information. This classification is not restrictive, and different classifications can be made.

[0023] In deep learning models, the performance of the model is tested on data that the model has not seen before training. The more and varied images the model sees during training, the better its performance will be on test data based on what it has learned. Therefore, the number of images used in training is usually higher than the number of test images. In the exemplification of the invention, a relatively small dataset was used, so 85% of the dataset (2085 images) was allocated for training, and 15% (370 images) for testing.

[0024] The technical effect of the invention is the development of a solution suitable for earthquake damage categorization in reinforced concrete structures, with improved performance criteria and high classification accuracy. It involves a deep learning architecture that allows inference to be made by processing photos selected from the device's photo gallery or taken with the device's camera, and the algorithm responsible for classification also utilizes this architecture. The mentioned deep learning architecture consists of 13 convolutional layers and 3 classification layers. The convolutional layers are similar to the VGG16 model, comprising 5 blocks, each consisting of 2 or 3 convolutional layers with (3x3) size filters, and newly added classification layers.

[0025] The first block consists of two convolutional layers with 64 filters each, followed by a "Max Pooling" layer to reduce the dimensions of the feature maps. The second block comprises two convolutional layers with 128 filters each and a "Max Pooling" layer. The third block consists of three convolutional layers with 256 filters each and a "Max Pooling" layer. The 4th and 5th blocks consist of three convolutional layers with 512 filters each and "Max Pooling" layers. Following the convolutional layers, the model includes three "Fully Connected" layers. The first two fully connected layers consist of 512 neurons each, and between these two layers, there is a "Dropout" layer (0.4) to reduce overfitting. In the last layer, there are as many neurons as the number of classes in the damage classification dataset (in our example, there are 5 classes).

[0026] In this method, the weight values learned on the ImageNet dataset are utilized in the convolutional layers of pretrained networks, and the input dimensions are set to (224, 224, 3). In the classification layers, new layers have been added to adapt to transfer learning and address issues arising from deficiencies in the dataset, differing from traditional model architectures. One of these problems is the problem of overfitting. Overfitting can be defined as the model making correct predictions during training but failing during testing. In order to eliminate this problem, which is frequently encountered in limited data sets, the random deactivation (dilution-dropout) method of some neurons in the classification layers was used during training. The dropout rate, which usually takes a value between 0 and 1 , is set to 0.4 in this study. During the training of the models in the classification layers, the Rectified Linear Unit (ReLU) activation function is used to introduce non-linearity to the network. The categorical crossentropy loss function, commonly used in multi-class classification tasks to minimize the difference between the actual and predicted values, is employed. Additionally, the Adam optimizer is utilized with a learning rate of 0.00001 , which updates the weight values through backpropagation based on the loss values to minimize the difference between the error values in real-time for each parameter. In order to reduce computational load, the data was trained in batches of size 32 over 100 epochs. In this method, only the convolutional layers of pretrained network models' geometries were utilized, and new network architectures were created by adding layers necessary for solving the damage classification problem (Figure 1 ).

[0027] Blocks Layer Type Filtre size Output size Parameters

[0028] Input Input Image - 224x224x3 0

[0029] ™e"in9 25088 0 c Fu nlly r Connect ,e Hd S (512)lflca,lon512 Q 12,845,568

[0030] Layers Dropout (0.4) ’512262,656

[0031] Classification _ „

[0032] Classification5 2’565

[0033] Total Parameters 27,825,477

[0034] Table 1. Neural network architecture developed for damage classification.

[0035] The classification accuracy of the system, which offers a new Convolutional Neural Network structure with different parameters and layer numbers specifically tailored for the damage severity classification problem, is observed to be 92%. Thus, the architecture of the system has been incorporated into an algorithm capable of automatic damage assessment at the level of reinforced concrete structural elements after the earthquakes, with low error rates according to a specific classification methodology. Within the scope of the invention, a deep learning-based algorithm developed for determining the levels of damage in structural elements within the framework of a damage detection algorithm based on a classification system (e.g., the methodology proposed by llki et al.) is present in a mobile application. This algorithm applies to evaluating the damage in structural elements occurring after earthquakes. The trained parameter values in the developed deep learning-based algorithm model have been utilized to develop a mobile application compatible with mobile devices. The saved parameter file in the developed deep learning-based algorithm has been converted to TensorFlow Lite, which is a version of the TensorFlow library that enables the creation of artificial intelligence and machine learning applications on mobile devices and provides APIs (Application Programming Interfaces) in languages such as Python, Java, and Kotlin. Thanks to this developed application, the degree of damage can be estimated by using images from the gallery of the device where the mobile application is installed or by capturing photos of the damaged area on the structural element through the device's camera. During the estimation of the damage level, images are processed in the background within the developed artificial intelligence model, allowing the damage classification process to be completed quickly and achieving high accuracy in predicting the damage class. Thus, a decision support system has been developed for use in post-earthquake fieldwork with this application.

Claims

CLAIMS1 . A system capable of estimating the degree of damage on a structural element based on photographs taken using the camera of the device or selected from the gallery of the device on which it is installed, characterized by comprising a device further comprising a mobile application wherein;- The device comprising a dataset further comprising sample photographs of damaged structural elements, which will be compared with photographs selected from the device’s photo gallery or taken with the device camera,- The device comprising classes that can rate the damage in photographs selected from the device’s photo gallery or taken with the device camera according to the extent of the damage,- The device comprising a deep learning architecture further comprising 13 convolutional layers and 3 classification layers, enabling inference processing of selected photographs from the device's photo gallery or taken with the device camera, wherein the convolutional layers further comprise 5 blocks and each block comprises 2 or 3 convolutional layers with filters of size (3x3) and newly added classification layers.

2. The system according to Claim 1 , characterized by comprising the device further comprising the mobile application wherein the dataset for comparison with photographs of damaged structural elements selected from the device's gallery or captured with the camera comprises images where the regions of intensified damage are zoomed in and cropped to facilitate easier detection of damages by the algorithm.

3. The system according to claim 1 , characterized by comprising the device further comprising the mobile application wherein the deep learning architecture has a first block consisting of two convolutional layers containing 64 filters each, followed by a "Pooling (Max Pooling)" layer to reduce the dimensions of the feature map.

4. The system according to claim 1 , characterized by comprising the device further comprising the mobile application wherein the deep learning architecture has a second block consisting of two convolutional layers containing 128 filters each, followed by a "Pooling (Max Pooling)" layer.

5. The system according to claim 1 , characterized by comprising the device further comprising the mobile application wherein the deep learning architecture has a third block consisting of three convolutional layers containing 256 filters each, followed by a "Pooling (Max Pooling)" layer.

6. The system according to claim 1 , characterized by comprising the device further comprising the mobile application wherein the deep learning architecture has the fourth and fifth blocks comprise three convolutional layers containing 512 filters each, followed by "Pooling (Max Pooling)" layers.

7. The system according to claim 1 , characterized by comprising the device further comprising the mobile application wherein the deep learning architecture has three "Fully Connected (dense)" layers after the convolutional layers.

8. The system according to claim 1 , characterized by comprising the device further comprising the mobile application wherein the deep learning architecture has a "Dropout" layer with a dropout rate of 0.4 between the first two fully connected layers to reduce the "overfitting" problem.

9. The system according to claim 1 , characterized by comprising the device further comprising the mobile application wherein the deep learning architecture has a number of neurons in the final layer corresponding to the number of classes in the damage classification dataset.

10. The system according to claim 1 , characterized by comprising the device further comprising the mobile application wherein the deep learning architecture uses the Rectified Linear Unit (ReLU) as the activation function to disrupt the linearity of the network in the classification layers, employing categorical crossentropy as the loss function to minimize the difference between the actual and predicted values, updating the weight values based on the loss values through feedback, and using a real-time learning rate (Adam) of 0.00001 for each parameter to minimize the difference between the error values during model training.11 . The system according to claim 1 , characterized by comprising the device further comprising the mobile application wherein the deep learning architecture has trained batches in sizes of 32 for 100 epochs to reduce computational load.

Images