A data asset classification management and storage method suitable for the engineering field

Abstract

The present invention discloses a data asset classification management and storage method applicable to the engineering field, comprising: determining the category of data sources; converting data into SQL statements; splitting the SQL statements for creating data forms into data catalog documents for the data forms through SQL semantic analysis, and appending source, classification, classification, and scope information; converting classification fields into data form names, and cataloging new data forms for automatic creation; extracting data form fields, performing similarity calculations on the data fields, and classifying similar results; creating table creation statements based on different classifications, and establishing data forms of different classifications; after creating the data form, automatically storing the name of the data form in the corresponding data catalog, and establishing a mapping relationship between the catalog and the entity data form. This method solves the problem of scattered and trivial data source tables, realizes the management and control of data assets, and maximizes the value of data.

Description

Technical Field

[0001] The present invention relates to the technical field of information management in the engineering field, and in particular to a data asset classification management and storage method applicable to the engineering field. Background Art

[0002] The construction industry has entered a period of stable growth. The traditional model can no longer meet the needs of high-quality development in the industry, and transformation and upgrading are imperative. As the construction industry must seize the opportunity of the integrated development of digitalization, informatization, and intelligentization, the development path of the integration of "construction industry" and "Internet" clarifies the direction of digital technology empowering the high-quality development of the construction industry and will surely comprehensively promote the transformation and upgrading of the construction industry towards digitalization, informatization, and intelligentization.

[0003] The common data classification management methods in existing technologies cannot meet the current segmentation requirements of the construction industry.

[0004] Specifically, the shortcomings of existing technologies in business applications are the lack of more fine-grained data classification and storage methods, which results in the current timely data being stored in the database, but the data is still scattered across various business departments. Employees need to log in to each business system separately to enter information, and leaders also need to log in to different systems for approval and viewing information. The lack of a comprehensive master data management platform leads to low data quality, making it impossible to conduct effective data analysis and form the company's data assets to assist managers in decision-making. Summary of the Invention

[0005] In response to the above problems, the present invention proposes a data asset classification management and storage method suitable for the engineering field, which solves the problem of low data quality and inability to conduct effective data analysis in the construction industry, and provides a fine-grained data classification and storage method to assist in analysis and decision-making.

[0006] The present invention is achieved through the following technical solutions:

[0007] A data asset classification management and storage method applicable to the engineering field comprises the following steps:

[0008] The first step is to determine the category of data sources based on different business processes and label different business processes;

[0009] The second step is to convert the data into SQL statements for creating a data form that can automatically extract SQL statements and form names;

[0010] The third step is to use SQL semantic analysis to split the SQL statement used to create the data form into a data catalog document for the data form, and append the source, classification, category, and scope information. The data catalog document is processed into a data form, annotated with categories and scopes, and the standards for filling in the data fields are stored in the form of regular expressions in the database.

[0011] The fourth step is to convert the classification fields into data form names after the labeling work is completed, and to catalog the newly created data forms for automation;

[0012] The fifth step is to extract the data form fields, calculate the similarity of the data fields, and classify the similar results;

[0013] Step 6: Create table creation statements based on different categories and create data tables for different categories;

[0014] Step 7: After creating the data form, automatically save the name of the data form into the corresponding data catalog to establish a mapping relationship between the catalog and the entity data form.

[0015] In an embodiment of the present invention, the categories of data sources in the first step are divided into manual data, management data, evaluation data, process data and result data.

[0016] In an embodiment of the present invention, the conversion of the classification field into the data form name in the fourth step includes:

[0017] The classification field is stored in the format of XX.XX.XX.XX, with "." as the separator to mark the classification level;

[0018] The hierarchical naming is converted into the naming of the data table, and the hierarchical management is reflected in the naming.

[0019] In an embodiment of the present invention, in the fifth step, similar results are divided into three categories: reference data, homologous data, and entity data.

[0020] In an embodiment of the present invention, in the sixth step, for the data form created by referencing data, the first form is selected as the entity to create a table, and the other fields are all associated with the fields of the first form as foreign keys; for the data form created by the same source data, the table is created in the entity manner, and an additional association relationship table is established to store similar fields and similarity numbers; for entity data, the data form is created directly.

[0021] By adopting the above technical solution, the present invention has the following beneficial effects:

[0022] On the basis of fine-grained management of data classification and grading, the data segmentation dimensions are enriched, and standardized management of data asset cataloging is added. At the same time, the asset cataloging specifications and database storage methods are matched, forming a unique technical solution in the engineering field that can separate cataloging and storage management in a more detailed manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a model architecture diagram of a data asset classification management and storage method applicable to the engineering field provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following is a further description of specific embodiments of the present invention in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is intended to facilitate understanding of the present invention and does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0026] Before that, let’s first provide the following analysis of the professional terms that appear in the article:

[0027] SQL statement: Structured Query Language (Structured Query Language), Structured Query Language is a database query and programming language used to access data as well as query, update and manage relational database systems; in short, SQL statement is a language for operating databases.

[0028] Cataloging is the process of establishing a local or remote client-server database connection. Its purpose is to obtain catalog information, that is, to generate a directory for accessing the database. The system database directory contains a list and pointers that enable DB2 (a relational database management system) to locate known databases, whether they are on the local system or on remote systems.

[0029] SQL semantic analysis: Semantic analysis is a logical stage in the SQL parsing process. Its main task is to examine the context-related properties based on grammatical correctness. During the SQL parsing process, this stage completes the legality judgment of elements such as table names, operators, and types, and detects semantic ambiguity. In this embodiment, the semantic analysis operation can be implemented using an existing SQL semantic analyzer.

[0030] Text similarity calculation: Provides the ability to calculate the semantic similarity between two short texts. The output similarity is a real value between 0 and 1. The larger the output value, the higher the semantic similarity. In this embodiment, TF-IDF can be used to calculate text similarity. The steps are as follows:

[0031] Use the TF-IDF algorithm to find the keywords of the two articles

[0032] 1. Use the TF-IDF algorithm to find the keywords of the two articles;

[0033] 2. Take a number of keywords (for example, 20) from each article, combine them into a set, and calculate the frequency of each article for the words in this set (to avoid differences in article length, relative frequency can be used);

[0034] 3. Generate the word frequency vectors of the two articles;

[0035] 4. Calculate the cosine similarity of the two vectors. The larger the value, the more similar they are.

[0036] Project management information technology is widely used in the engineering field. From project establishment, design, construction, procurement, materials management, to acceptance, a large amount of structured and unstructured data is generated. However, the question is how to store project information in a computer to facilitate its access. If storage is based on the requirements of the application software itself, the information will be too dispersed, easily forming information islands. If it is designed for easy access, compatibility issues with the application software will arise.

[0037] To address these issues, this paper designs a data asset classification method and storage and access method suitable for the engineering field. Based on database SQL statements, it implements secondary cataloging of stored data, integrates the concept of data assetization, establishes data standards, classification methods, and storage and access methods, and effectively addresses the shortcomings of existing methods, such as unclear data resources, inconsistent data standards, and insufficient visualization. This improves data utilization and reduces the difficulty of data use.

[0038] See Figure 1 The present invention provides a data asset classification management and storage method applicable to the engineering field, which mainly includes the following steps:

[0039] The first step is to determine the categories of data sources according to different business processes and mark different business processes; in this embodiment, the categories of data sources are mainly divided into manual data, management data, evaluation data, process data and result data.

[0040] The second step is to convert the data into SQL statements in the database so as to create a data form that can automatically extract the SQL statements and form names.

[0041] The third step is to split the SQL statement for creating the data form into the original form of the data form (becoming a data catalog document) through SQL semantic analysis, and append the source, classification, classification and scope information; process the data catalog document into a data form, mark the classification and scope, and store the standards for filling in the data field in the form of regular expressions in the database; this step realizes the process of automatically creating catalog documents and restoring data tables in a programmatic way.

[0042] In the fourth step, after labeling is complete, the classification fields are first converted to data form names. Classification fields are stored in the format of XX.XX.XX.XX, with "." as a delimiter to indicate the classification hierarchy. The hierarchical naming is then converted to data table names, reflecting the hierarchical management, such as Data Table 1.1, Data Table 1.1.1. This step automatically catalogs the creation of new data tables.

[0043] The fifth step is to extract data form fields and calculate the similarity of all data fields using a text similarity method (using an open source algorithm). The similarity results are divided into three categories: 100%, 60%-99%, and 0-59%. They are classified as reference data, homologous data, and entity data respectively.

[0044] Step 6: Automatically create table creation statements based on the three different categories. For tables referencing data, select the first table as the entity table and associate all other fields with the fields of this table as foreign keys. For tables with the same source data, create the table as an entity, and create an additional relationship table to store the four similarity segments and similarity numbers. For entity data tables, create the table directly.

[0045] Step 7: After creating the data form, automatically save the name of the data form into the corresponding data catalog to establish a mapping relationship between the catalog and the entity data form.

[0046] Using the table creation catalog document above, you can find the transformation relationship between the original table and the final storage table. You can also reversely restore the correspondence between the physical storage table and the original table, facilitating the investigation of discrepancies. Data health data can also be output to facilitate operations and maintenance.

[0047] From the foregoing it will be seen that the present invention is well adapted to attain all of the objects and aims set forth above, as well as other advantages which are obvious and inherent to the structure. It will be understood that certain features and subcombinations are useful and may be employed without reference to other features and subcombinations. This is within the scope of the present invention.

Claims

1. A data asset classification management and storage method applicable to the engineering field, characterized in that: The following steps are involved: The first step is to determine the category of data sources based on different business processes and label different business processes; The second step is to convert the data into SQL statements for creating a data form that can automatically extract SQL statements and form names; The third step is to use SQL semantic analysis to split the SQL statement used to create the data form into a data catalog document for the data form, and append the source, classification, category, and scope information. The data catalog document is processed into a data form, annotated with categories and scopes, and the standards for filling in the data fields are stored in the form of regular expressions in the database. The fourth step is to convert the classification fields into data form names after the labeling work is completed, and to catalog the newly created data forms for automation; The fifth step is to extract the data form fields, calculate the similarity of the data fields, and classify the similar results; Step 6: Create table creation statements based on different categories and create data tables for different categories; Step 7: After creating the data form, automatically save the name of the data form into the corresponding data catalog to establish a mapping relationship between the catalog and the entity data form; In the fifth step, similar results are divided into three categories: citation data, homology data, and entity data; In the sixth step, for the data forms created by referencing data, select the first form as the entity to create the table, and all other fields are associated with the fields of the first form as foreign keys; for data forms created by the same source data, create the table in the same way as the entity, and create an additional association table to store similar fields and similarity numbers; for entity data, create the data form directly.

2. A data asset classification management and storage method applicable to the engineering field according to claim 1, characterized in that: In the first step, the categories of data sources are divided into manual data, management data, evaluation data, process data and result data.

3. The data asset classification management and storage method applicable to the engineering field according to claim 1 is characterized in that: The conversion of classification fields into data form names in the fourth step includes: The classification field is stored in the format of XX.XX.XX.XX, with "." as the separator to mark the classification level; The hierarchical naming is converted into the naming of the data table, and the hierarchical management is reflected in the naming.

Images