BERT-based Schema Matching Method for Flood Data Unification

Abstract

The BERT-based schema matching technique for flood data integration can efficiently reduce data construction costs during flood data integration through a BERT-based schema matching model that can be utilized for flood disaster data integration.

Description

Technology Field

[0001] This document concerns a BERT-based schema matching technique for flood data integration. Background Technology

[0002] Urban flooding is a disaster whose frequency and intensity are gradually increasing in tandem with climate change, causing significant damage to property and infrastructure. As the scale and diversity of such disasters grow, various agencies and sources in Korea collect and manage vast amounts of flood data; however, a problem has emerged where integration is difficult due to the mixing of differing formats and variable names. In particular, Korean data often contains many compound words mixed with Sino-Korean characters, leading to inconsistent expressions. Consequently, even when referring to the same concept, it is recorded under completely different names. The table in Figure 2 shows some of the column names in sewer network data provided by different Korean local governments. In this table, 'Individual Number' and 'Unique Identifier' in the first line, and 'Pipe Type' and 'Structural Type' in the third line, are variables with the same meaning, but inconsistencies arise because they use different Sino-Korean characters. In this environment, accurately matching and integrating data schemas is an essential yet challenging task. The problem to be solved

[0003] According to one embodiment of the present document, a BERT-based schema matching technique for flood data integration may be provided, which proposes a BERT-based schema matching model that can be utilized for flood disaster data integration.

[0004] The problem to be solved in one embodiment of this document is not limited to the problem mentioned above, and can be extended in various ways without departing from the spirit and scope of one embodiment of this document. means of solving the problem

[0005] A computing device according to one embodiment of the present document includes: a memory for storing one or more programs; and one or more processors that perform operations according to the one or more programs stored in the memory, wherein the processors automatically integrate non-standard column names of flood data using a BERT-based schema matching model.

[0006] The above BERT-based schema matching model may include a text augmentation module that transforms variable names of the flood data into various expressions by utilizing a text augmentation technique based on a generative language model.

[0007] The text augmentation module above generates alternative expressions based on the original variable name, calculates the embedding similarity between the generated alternative expressions and the original variable name, filters out alternative expressions with an embedding similarity below a specific threshold as noise, and inputs the unfiltered alternative expressions to the embedding generation module.

[0008] The above BERT-based schema matching model may further include an embedding generation module based on Korean BERT that converts the variable names of the flood data derived from the text augmentation module into high-dimensional vectors.

[0009] The above BERT-based schema matching model may further include an automatic matching module that automatically matches non-standard column names to standard variable names based on cosine similarity, based on embedding vectors generated by the embedding generation module. Effects of the invention

[0010] According to one embodiment of the present document, data construction costs can be efficiently reduced when integrating flood data.

[0011] The effects according to the various embodiments of this document are not limited to those described above, and it is obvious to a person skilled in the art that various effects are inherent in one embodiment of this document. Brief explanation of the drawing

[0012] FIG. 1 is a block diagram for explaining a computing device according to one embodiment of the present document. Figure 2 is a diagram showing variable names of sewage network data by local government according to one embodiment of the present document. FIG. 3 is a diagram illustrating an overview of a proposed schema matching model according to one embodiment of the present document. FIG. 4 is a drawing for explaining a text augmentation module according to one embodiment of the present document. FIG. 5 is a diagram illustrating a BERT-based embedding module according to one embodiment of the present document. FIG. 6 is a drawing for explaining an automatic matching module according to one embodiment of the present document. FIG. 7 is a diagram showing an algorithm 1 that performs LLM-based schema matching according to one embodiment of the present document. FIG. 8 is a drawing showing standard parameter names according to one embodiment of the present document. Figure 9 is a diagram showing a comparison of the performance of our model and a conventional word embedding model according to one embodiment of the present document. Specific details for implementing the invention

[0013] Hereinafter, embodiments of this document will be described in detail with reference to the attached drawings. The advantages and features of the embodiments of this document, and the methods for achieving them, will become clear by referring to the details described below in conjunction with the attached drawings. However, the embodiments of this document are not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments of this document are defined only by the scope of the claims.

[0015] A BERT-based schema matching technique for flood data integration according to one embodiment of the present document will be described in detail below with reference to the attached drawings.

[0017] First, a computing device according to one embodiment of the present document will be described with reference to FIG. 1.

[0018] FIG. 1 is a block diagram for explaining a computing device according to one embodiment of the present document.

[0019] Referring to FIG. 1, a computing device (100) according to one embodiment of the present document can perform operations using a BERT-based schema matching technique for flood data integration.

[0020] To this end, the computing device (100) may include one or more processors (110), a computer-readable storage medium (130), and a communication bus (150).

[0021] The processor (110) can control the computing device (100) to operate. For example, the processor (110) can execute one or more programs (131) stored in a computer-readable storage medium (130). One or more programs (131) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (100) to perform an operation using a BERT-based schema matching technique for flood data integration when executed by the processor (110).

[0022] A computer-readable storage medium (130) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information for performing operations using a BERT-based schema matching technique for immersion data integration. A program (131) stored in the computer-readable storage medium (130) includes a set of instructions executable by a processor (110). In one embodiment, the computer-readable storage medium (130) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other forms of storage media that are accessed by the computing device (100) and capable of storing desired information, or a suitable combination thereof.

[0023] The communication bus (150) interconnects various other components of the computing device (100), including the processor (110) and the computer-readable storage medium (130).

[0024] The computing device (100) may also include one or more input / output interfaces (170) and one or more communication interfaces (190) that provide interfaces for one or more input / output devices. The input / output interfaces (170) and the communication interfaces (190) are connected to a communication bus (150). An input / output device (not shown) may be connected to other components of the computing device (100) through the input / output interface (170).

[0026] Then, with reference to FIGS. 2 to 9, a BERT-based schema matching technique for flood data integration according to one embodiment of the present document will be described.

[0027] FIG. 2 is a diagram showing variable names of sewage network data by local government according to one embodiment of the present document, FIG. 3 is a diagram explaining an overview of a proposed schema matching model according to one embodiment of the present document, FIG. 4 is a diagram explaining a text augmentation module according to one embodiment of the present document, FIG. 5 is a diagram explaining a BERT-based embedding module according to one embodiment of the present document, FIG. 6 is a diagram explaining an automatic matching module according to one embodiment of the present document, FIG. 7 is a diagram showing Algorithm 1 performing LLM-based schema matching according to one embodiment of the present document, FIG. 8 is a diagram showing standard parameter names according to one embodiment of the present document, and FIG. 9 is a diagram showing a performance comparison between an existing word embedding model and our model according to one embodiment of the present document.

[0028] Overview (Introduction)

[0029] Urban flooding is a disaster whose frequency and intensity are gradually increasing in tandem with climate change, causing significant damage to property and infrastructure. As the scale and diversity of such disasters grow, various agencies and sources in Korea collect and manage vast amounts of flood data; however, a problem has emerged where integration is difficult due to the mixing of differing formats and variable names. In particular, Korean data often contains many compound words mixed with Sino-Korean characters, leading to inconsistent expressions. Consequently, even when referring to the same concept, it is recorded under completely different names. The table in Figure 2 shows some of the column names in sewer network data provided by different Korean local governments. In this table, 'Individual Number' and 'Unique Identifier' in the first line, and 'Pipe Type' and 'Structural Type' in the third line, are variables with the same meaning, but inconsistencies arise because they use different Sino-Korean characters. In this environment, accurately matching and integrating data schemas is an essential yet challenging task.

[0030] To address these issues, various existing schema matching methodologies have been proposed. Rule-based approaches take a relatively simple approach, analyzing column names using regular expressions or conditional statements and considering the corresponding schema as the same concept if specific patterns or keywords are satisfied. While this method has the advantage of being easy to implement and apply, it exhibits limitations, such as the need to add or modify rules as data grows and the difficulty of immediately responding to the appearance of expressions not previously considered. Dictionary-based methods aim to create a specialized terminology dictionary or thesaurus and map column names appearing in the data to this dictionary to connect them with standardized expressions. Although this provides effective results in the initial stages, in fields like disaster management where terminology changes rapidly or new concepts emerge, the dictionary must be constantly updated, and matching fails if an expression not in the dictionary appears. Similarity measurement-based methods are approaches that primarily calculate the distance between strings or compare column names using simple statistical techniques to select pairs with high similarity scores as matching candidates. Among existing methods, it can be applied relatively quickly without domain knowledge, but due to the Korean language's conjugation of endings, the use of Chinese characters alongside Korean, and complex word structures, it is difficult to fully determine semantic similarity based solely on character-level similarity.

[0031] Ontology-based approaches facilitate schema integration by hierarchically defining domain concepts and their relationships; once established, they can guide different data sources into a unified conceptual framework. For example, an ontology-based data framework was proposed for urban flood disaster management in Zhengzhou, China, enabling the construction of a system that structures and visualizes various concepts related to flood disasters. The challenge lies in the fact that designing and maintaining such an ontology requires significant knowledge and effort from domain experts. Furthermore, in fields like disaster management, where specialized terminology is constantly being created and existing terms are prone to change, ontologies often fail to reflect the rapidly evolving reality in a timely manner. While traditional methodologies may be effective temporarily or within specific scopes, they have distinct limitations when it comes to comprehensively processing and integrating disaster data in the Korean language. In particular, rule-based and dictionary-based methods are weak in terms of scalability and maintenance, while similarity-measurement-based methods struggle to handle linguistic complexity and semantic correspondence. Moreover, the implementation and maintenance costs of ontology-based approaches are burdensome in situations where domain changes are frequent, such as in the disaster sector.

[0032] Recent advancements in deep learning, particularly Natural Language Processing (NLP), have led to the presentation of new approaches in the field of schema matching. These technologies go beyond simply processing text data to effectively learn the semantic representations of text by utilizing word embedding models and pre-trained large language models. This enables the precise identification of semantic similarities between variable names and attributes, allowing for the efficient matching of similar schemas. For example, pre-trained language models are used to automate schema matching in building energy management, and semantic similarity techniques are applied to reduce the burden of manual work. Furthermore, the study demonstrates that word embedding technology can effectively solve schema matching problems in industrial environments. In addition, it proves that combining domain-specific and cross-domain embeddings improves matching between heterogeneous data sources, and that large language models can perform schema matching with high accuracy even in real-time environments. A two-stage matching method utilizing embeddings at both the table and attribute levels demonstrates superior matching performance compared to existing methods.

[0033] However, these methods have several limitations. First, many existing models rely on specific data formats or structures, making it difficult to apply them flexibly to diverse datasets. Second, some methods require deep domain knowledge. Third, they focus on semantic similarity rather than effectively handling diverse data representations or contextual differences. This problem is particularly pronounced in languages ​​like Korean, where technical terms are complex and based on Sino-Korean characters. In the field of disaster management, variations in technical terminology lead to changes in term expressions during the data integration process. To address this issue, we propose an automated flood data integration model utilizing a BERT-based column name matching method. Our approach fine-tunes the BERT model using public documents and news data related to flooding to train it on the specific terms and expressions of Korean flood data. BERT is used to generate embedding vectors for column names, and the most similar column names are matched by comparing the similarity between these vectors. Additionally, the proposed method employs text augmentation techniques to aid the model in understanding various expressions. Through this, data integration is achieved by automatically converting the column names of the target data into standard column names. The contributions of this study are as follows:

[0034] This document presents an automated schema matching technique utilizing the BERT model to propose a method for efficiently reducing data construction costs during flood data integration.

[0035] We develop a model that can convert non-standard names from various data sources without domain expertise.

[0036] - The application of text augmentation based on generative language models improves matching accuracy by enabling the model to learn various expressions.

[0037] - The proposed model improves the efficiency and reliability of schema matching by achieving a 00% improvement in accuracy compared to existing embedding models.

[0038] - A case study applying model results to a simulation of an actual Korean case demonstrates the practicality of the proposed model.

[0040] Methodology

[0041] This document presents a BERT-based schema matching model that can be utilized for the integration of flood disaster data. The proposed BERT-based schema matching model is structured to automatically integrate heterogeneous column names in flood data, focusing on resolving the complexity of Korean data representation and non-standard naming issues arising from various data sources. The model consists of a text augmentation module, an embedding generation module, and an automatic matching module; each module focuses on ensuring data diversity and effectively reflecting semantic similarity. An overview of the detailed schema matching process is shown in Figure 3. Our model is composed of a total of three modules: a text augmentation module, a Korean BERT-based embedding module, and an automatic matching module. The model performs schema matching by passing the input multi-source flood data through each module in sequence to derive the final integrated column names. Detailed information regarding each module is presented in the following section.

[0042] A. Text augmentation module

[0043] The first module, the text augmentation module, utilizes text augmentation techniques based on a generative language model to enhance the model's understanding of variable names in flood data. Standard and target variable names given as input are transformed into various expressions and used for model training. For example, given the standard variable name 'shape of a pipe,' the generative model generates alternative expressions such as 'shape of a sewer pipe' or 'shape of a pipe' based on it. To reduce the possibility that the generated expressions do not semantically match the original variable names during this process, the embedding similarity between the generated expressions and the original variable names is calculated. Expressions with a similarity below a specific threshold are considered noise and filtered out, and only the refined alternative expressions are subsequently input into the BERT-based embedding module. Figure 4 illustrates the augmentation process. Through this module, the model is equipped with robustness and generalization performance capable of matching non-standard data column names even if they use expression methods different from standard variable names. This approach is particularly effective in Korean data environments where there are many complex expressions based on Sino-Korean characters.

[0044] B. BERT-based embedding module

[0045] The second module is a Korean BERT-based embedding module that converts the variable names of the flood data derived in the previous step into high-dimensional vectors. The Korean BERT model was built in-house by collecting news articles from various categories and is further fine-tuned into a language model specialized for disaster management through additional training on news and public data. Through this, the model acquires the ability to more effectively understand and process flood data containing specialized terminology or various expression styles. In the data preprocessing stage, column names are normalized and unnecessary special characters and spaces are removed to enhance data consistency. Additionally, variable names existing in the original data in various forms—such as English words, English abbreviations, and Korean words—are standardized according to naming conventions like camel case and converted into English and Korean formats. This preprocessing process supports matching to the same concept even if variable names are composed in different formats. In the BERT embedding process, the normalized variable names are taken as input to generate high-dimensional vectors that reflect semantic similarity, which are subsequently used in the automatic matching module. Figure 5 illustrates the preprocessing and embedding generation process. This evaluates the semantic similarity between standard and non-standard variable names and prepares for matching in the next step.

[0046] C. Automatic matching module

[0047] The third module performs the task of matching column names of non-standard data with column names of standard data, thereby automating data integration. The embedding vectors generated in the previous step are compared based on cosine similarity, and if the similarity exceeds a set threshold, it is determined that a match has been made. Through this matching process, non-standard column names can be automatically converted into their corresponding standard variable names. Rather than simply comparing the superficial similarity of strings, this approach reflects contextual meaning and the relationships between data to provide more sophisticated and reliable matching results. This approach effectively handles heterogeneity between various representation methods and data sources, contributing to enhancing the reliability and accuracy of data integration. Figure 6 visually illustrates the matching process and demonstrates how non-standard column names are matched to standard column names based on cosine similarity. Target documents integrated with standard document column names through the proposed model can subsequently be used in simulations to contribute to decision-making regarding flood response strategies.

[0049] Algorithm 1 in Figure 7 illustrates the three main steps of the proposed model—text augmentation, embedding generation, and automatic matching—from an input-output perspective. Through these processes, our model improves the accuracy of disaster data integration and derives optimal results using a Top-K matching strategy. This is effective in resolving column name discrepancies between non-standard and standard data.

[0051] Experiments

[0052] A. Experimental setting

[0053] To evaluate the performance of the proposed model, a column name matching process is performed between the target conversion table and the standard table. The data used consists of actual flood-related data provided by the Korean local governments of Busan and Incheon, along with standard variable name tables. The table in Figure 8 shows the English and Korean expressions of the standard variable names that serve as the criteria for data integration. Since the target documents for conversion—the sewer network data of Busan and Incheon—contain non-standard variable names, it is necessary to match them with standard variable names.

[0054] Word2Vec and fine-tuned BERT models are used as comparison models, and performance differences based on the presence or absence of an augmentation module are compared to verify the validity of the proposed text augmentation technique. The description of the baseline model is as follows.

[0055] - Word2Vec: Word2Vec learns embeddings by considering the surrounding context of words through Skip-gram and CBOW methods. While it can identify word meanings based on context by reflecting local context, its performance is limited compared to modern models like BERT.

[0056] - BERT_finetuned, BERT_aug: The BERT model learns context bidirectionally to generate embeddings that reflect the contextual meaning of words. In this experiment, we compare the performance difference between a BERT model with only fine-tuning applied and a model with text augmentation applied.

[0057] B. Experimental evaluation

[0058] For each model, embeddings of the input data are calculated, and the similarity between non-standard and standard variable names is evaluated through cosine similarity. Accuracy@k is used as the evaluation metric, measuring the proportion of standard variable names included in the top k ∈ {1, 3, 5} matching candidates. Figure 9 shows the matching accuracy of the baseline models and the proposed model for each dataset. In the Incheon City dataset, the fine-tuned BERT model shows accuracy of 0.43, 0.57, and 0.71 for top-1, 3, and 5, respectively. Additionally, the proposed model with text augmentation applied achieved accuracy of 0.33, 0.86, and 0.95, respectively, proving the validity of the proposed method. In the case of Top-1 in the Incheon City dataset, the accuracy of the model without augmentation was higher; this is because the data generated during the augmentation process may have weakened the direct match with the standard variable name, or because the model may have been trained with data containing some noise. Furthermore, there may have been instances where the augmented data did not contextually match actual non-standard variable names, and this impact is likely to have been more pronounced due to the nature of Top-1 matching. Nevertheless, text augmentation significantly improves model accuracy in Top-3 and Top-5 matching and demonstrates high flexibility and robustness toward non-standard variable names with diverse expressions and complex contexts. This is because the augmented data helped the model learn semantic similarity with standard variable names over a wider range, consequently proving superior performance compared to existing methods in the task of standardizing non-standard variable names.

[0060] The embodiments of this document are intended to illustrate technical concepts, and the scope of the technical concepts of the embodiments of this document is not limited by these embodiments. The scope of protection of the embodiments of this document shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the embodiments of this document. Explanation of the symbols

[0061] 100 : computing device, 110 : Processor, 130 : Computer-readable storage media, 131 : Program, 150 : Communication bus, 170 : Input / Output Interface, 190: Communication interface

Claims

Claim 1 A computing device comprising: a memory for storing one or more programs; and one or more processors that perform operations according to the one or more programs stored in the memory, wherein the processors automatically integrate non-standard column names of flood data using a BERT-based schema matching model. Claim 2 A computing device according to claim 1, wherein the BERT-based schema matching model comprises a text augmentation module that transforms variable names of the flood data into various expressions by utilizing a text augmentation technique based on a generative language model. Claim 3 A computing device according to paragraph 2, wherein the text augmentation module generates alternative expressions based on original variable names, calculates embedding similarity between the generated alternative expressions and the original variable names, filters alternative expressions with embedding similarity below a specific threshold as noise, and inputs unfiltered alternative expressions to the embedding generation module. Claim 4 In paragraph 2, the computing device further comprises an embedding generation module that converts variable names of the flood data derived from the text augmentation module into high-dimensional vectors, wherein the BERT-based schema matching model is based on Korean BERT. Claim 5 In paragraph 4, the computing device further comprises an automatic matching module that automatically matches non-standard column names to standard variable names based on cosine similarity, based on embedding vectors generated by the embedding generation module, wherein the BERT-based schema matching model comprises an embedding vector generated by the embedding generation module.

Images