Reconnaissance and design enterprise asset multivariate heterogeneous data fusion and knowledge graph construction method

Abstract

The invention discloses an investigation and design enterprise asset multivariate heterogeneous data fusion and knowledge graph construction method, which comprises the following steps: acquiring enterprise asset data information including structured data and unstructured data; invalid and missing value removal processing is carried out on the structured data, and meanwhile, the cleaned numerical data is converted into a text format; performing text data standardization processing and optimization on the unstructured data and the structured data after the text format, and constructing word vectors while performing text disambiguation standardization; respectively extracting features of the structured data and the unstructured data by using a decoder BERT model; fusing all kinds of data characteristics by using a Lite Transform to construct a knowledge graph; the problems that existing investigation and design enterprises are huge in data scale and uneven in data quality, corresponding design histories and collected field related information cannot be rapidly obtained from historical data, and consequently, early-stage investigation data are repeated, and later-stage design is inconsistent are solved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a method for integrating multi-dimensional heterogeneous data of assets of a survey and design enterprise and constructing a knowledge graph. Background Art

[0002] Both China and foreign countries have made some progress in the research of multi-source heterogeneous data aggregation technology. Domestic universities and research institutions as well as foreign universities and research institutions have conducted in-depth research in this field, involving ETL tools, data integration algorithms, etc., and have continuously proposed new data integration methods and algorithms. At the same time, domestic and foreign companies are also exploring the value of multi-source data aggregation technology in practical applications, especially in the fields of finance, e-commerce, and intelligent manufacturing, and are trying and accumulating experience.

[0003] The existing survey and design companies have huge data volumes and uneven data quality. They are unable to quickly obtain the corresponding design history and the collected field-related information from historical data, resulting in a large number of survey and design tasks with problems such as duplication of early survey data and inconsistent later designs. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for fusion of multi-heterogeneous data and construction of knowledge graph of survey and design enterprise assets, so as to solve the problem that the existing survey and design enterprise data is huge in scale, the data quality is uneven, and the corresponding design history and the collected field-related information cannot be quickly obtained from the historical data, resulting in a large number of survey and design tasks with problems such as duplication of early survey data and inconsistent later design.

[0005] The technical solution of the present invention: A method for integrating multi-dimensional heterogeneous data and constructing a knowledge graph of survey and design enterprise assets, the method comprising: Step 1: Collect enterprise asset data information, including structured data and unstructured data; Step 2: Data preprocessing: For structured data, remove invalid and missing values, and convert the cleaned numerical data into text format; for unstructured data and structured data in text format, perform text data normalization and optimization, text disambiguation and normalization, and construct word vectors; Step 3: Use the decoder BERT model to extract the features of structured data and unstructured data respectively; Step 4: Use Lite Transformer to fuse various data features to build a knowledge graph.

[0006] The structured data includes the company's existing operational data and project data table data, and the unstructured data includes survey and design documents, various reports and design procedure descriptive text data.

[0007] The data preprocessing in step 2 specifically includes: Step 2.1: The preprocessing method for structured data includes: performing data cleaning to remove noise and inconsistency in the data, and converting the cleaned structured numerical data into text format; constructing a vocabulary table suitable for the feature values ​​of the data set according to different structured data features, so as to convert the collected structured data into a standard text format; Step 2.2, use word segmentation, part-of-speech tagging and named entity recognition for normalization processing; use the jieba Chinese word segmentation database to segment the collected text content; after word segmentation, use the PKU tag set for part-of-speech tagging; finally, perform named entity recognition, extract professional entity names, use word-based NER technology, select the maximum entropy model as the training algorithm training model, and process the text data after word segmentation, and extract the institutions, organizations and engineering entities in the text based on rules and pattern matching; Step 2.3, construct word vectors: Word vectorization enables the machine to understand and process unstructured language data, realize automatic relationship extraction and semantic analysis, and use the SkipGram mode of Word2vec to construct word vectors.

[0008] The method of using the decoder BERT model to extract features of structured data and unstructured data respectively in step 3 includes: BERT uses the self-attention mechanism in the architecture to simultaneously consider the global context of the text during the training phase, so as to better understand the text, extract the corresponding data of the text, and pre-train on a large-scale corpus.

[0009] The method of using Lite Transformer to fuse various data features to construct a knowledge graph described in step 4 includes: the Lite Transformer layer is used to fuse relevant data features extracted by the decoder BERT model and comprehensively create a knowledge graph, and the knowledge graph is constructed by receiving abstract tensors from BERT1 and BERT2.

[0010] Beneficial effects of the present invention: By integrating information from different data sources, the present invention enables survey and design companies to obtain more comprehensive business insights, promote data sharing and communication across departments and systems, and thus enhance understanding of the business. Secondly, the effective integration of multi-source data can provide more sufficient objective analysis data for the enterprise management and provide auxiliary support for the decision-making of the enterprise management. In addition, multi-source data convergence can also help optimize resource utilization, reduce costs, and improve work efficiency and production benefits. Having the ability to integrate multi-source data can make companies more competitive, better adapt to market changes, and respond quickly to customer needs. At the same time, multi-source data fusion is also the basis for promoting the intelligent development of enterprises, which helps to build intelligent decision-making systems, realize automated operation management, and enhance the innovation capabilities of enterprises. Finally, by integrating and cleaning multi-source data, data quality can be improved, errors and redundancy can be reduced, and the credibility and accuracy of data can be enhanced.

[0011] This invention is based on artificial intelligence and computer natural language processing analysis technology. It uses the BERT model to learn the corresponding features contained in various types of data in the asset data of survey and design enterprises, and then uses Lite Transformer to fuse various data features to build a knowledge graph. By integrating data from different sources, different formats, and different qualities, it helps enterprises establish a comprehensive, consistent and reliable data foundation, thereby supporting more intelligent decision-making and business operations.

[0012] The present invention can clean and fuse various types of enterprise asset data collected by the enterprise, and use these heterogeneous data to create a knowledge base suitable for survey and design enterprises.

[0013] It solves the problem that the existing survey and design companies have huge data scale and uneven data quality, and are unable to quickly obtain the corresponding design history and collected field-related information from historical data, resulting in a large number of survey and design tasks with problems such as duplication of early survey data and inconsistent later designs. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Schematic diagram for multi-heterogeneous data fusion and knowledge graph construction DETAILED DESCRIPTION

[0015] A method for integrating multi-heterogeneous data and constructing a knowledge graph for survey and design enterprise assets, specifically comprising: Step 1: Collect the asset data of the survey and design companies, including structured data (existing operational data, project data and other tabular data of the company) and unstructured data (such as survey and design documents, various reports, design specifications and other descriptive text data).

[0016] Step 2: Data preprocessing: For structured data, remove invalid and missing values, and convert the cleaned numerical data into text format; for unstructured data and structured data in text format, perform text data normalization and optimization, text disambiguation and normalization, and construct word vectors; The specific method is: Step 2.1: The preprocessing method for structured data is: perform data cleaning to remove noise and inconsistency in the data, and convert the cleaned structured numerical data into text format. This step is achieved by constructing descriptive sentences. For example, "2023 / 7 / 11 10:00 3.7m / s" can be uniformly converted to "The instantaneous wind speed at 10:00 on July 11, 2023 is 3.7 meters per second". According to different structured data features, a vocabulary suitable for the feature values ​​of the data set is constructed to convert the collected structured data into a standard text format.

[0017] Step 2.2: Use word segmentation, part-of-speech tagging, and named entity recognition for normalization: Word segmentation refers to the process of splitting text data into separate words according to certain rules and algorithms for subsequent processing and analysis. This method uses the jieba Chinese word segmentation library to segment the collected text content; after word segmentation, the PKU tagging set developed by the Institute of Computational Linguistics of Peking University is used for part-of-speech tagging; finally, named entity recognition is performed to extract professional entity names to ensure the formatting of text data. This method uses the word-based NER (Named Entities Recognition) technology and selects the maximum entropy model as the training algorithm training model to process the text data after word segmentation (such as text materials such as regulations and management standards), and extracts entities such as institutions, organizations, and projects in the text based on rules and pattern matching.

[0018] Step 2.3, build word vector: Word vectorization of text enables machines to understand and process unstructured data such as language, and realize automatic relationship extraction and semantic analysis. This method uses the SkipGram mode of Word2vec to construct word vectors.

[0019] Step 3: Use the decoder BERT model (Bidirectional Encoder Representations from Transformers) to extract the features of structured data and unstructured data respectively.

[0020] The decoder BERT model (Bidirectional Encoder Representations from Transformers) is used to extract features of structured data and unstructured data respectively. Through the self-attention mechanism in its architecture, BERT can simultaneously consider the global context of the text during the training phase, thereby better understanding the text. The corresponding data of the text is extracted. And through pre-training on a large-scale corpus in advance, the model already has the basic knowledge of the corresponding language representation and can be put into use without providing a large amount of data for training.

[0021] Step 4: Use Lite Transformer to fuse various data features to build a knowledge graph.

[0022] Use Lite Transformer to fuse various data features to build a knowledge graph. The Lite Transformer layer is used to fuse the relevant data features extracted by the decoder BERT model in step 3, and comprehensively create a knowledge graph. The present invention uses Lite Transformer. Since most of the knowledge features have been decoded in the previous BERT model, only a simplified lightweight Transformer network is used here for data feature fusion. The knowledge graph is constructed by receiving abstract tensors from BERT1 and BERT2. These tensors are high-dimensional representations of the data after being processed by the BERT model, and they contain rich semantic information and contextual relationships.

Claims

1. A method for integrating multi-dimensional heterogeneous data and constructing a knowledge graph for survey and design enterprise assets, characterized by: The method comprises: Step 1: Collect enterprise asset data information, including structured data and unstructured data; Step 2: Data preprocessing: For structured data, remove invalid and missing values, and convert the cleaned numerical data into text format; for unstructured data and structured data in text format, perform text data normalization and optimization, text disambiguation and normalization, and construct word vectors; Step 3: Use the decoder BERT model to extract the features of structured data and unstructured data respectively; Step 4: Use Lite Transformer to fuse various data features to build a knowledge graph.

2. According to claim 1, a method for fusion of multi-heterogeneous data and construction of knowledge graph of survey and design enterprise assets is characterized by: The structured data includes the company's existing operational data and project data table data, and the unstructured data includes survey and design documents, various reports and design procedure descriptive text data.

3. The method for fusion of multi-heterogeneous data and construction of knowledge graph of survey and design enterprise assets according to claim 1 is characterized by: The data preprocessing in step 2 specifically includes: Step 2.1: The preprocessing method for structured data includes: performing data cleaning to remove noise and inconsistency in the data, and converting the cleaned structured numerical data into text format; constructing a vocabulary table suitable for the feature values ​​of the data set according to different structured data features, so as to convert the collected structured data into a standard text format; Step 2.2, use word segmentation, part-of-speech tagging and named entity recognition for normalization processing; use the jieba Chinese word segmentation database to segment the collected text content; after word segmentation, use the PKU tag set for part-of-speech tagging; finally, perform named entity recognition, extract professional entity names, use word-based NER technology, select the maximum entropy model as the training algorithm training model, and process the text data after word segmentation, and extract the institutions, organizations and engineering entities in the text based on rules and pattern matching; Step 2.3, construct word vectors: Word vectorization enables the machine to understand and process unstructured language data, realize automatic relationship extraction and semantic analysis, and use the SkipGram mode of Word2vec to construct word vectors.

4. The method for fusion of multi-heterogeneous data and construction of knowledge graph of survey and design enterprise assets according to claim 1 is characterized by: The method of using the decoder BERT model to extract features of structured data and unstructured data respectively in step 3 includes: BERT uses the self-attention mechanism in the architecture to simultaneously consider the global context of the text during the training phase, so as to better understand the text, extract the corresponding data of the text, and pre-train on a large-scale corpus.

5. The method for fusion of multi-heterogeneous data and construction of knowledge graph of survey and design enterprise assets according to claim 1 is characterized by: The method of using Lite Transformer to fuse various data features to construct a knowledge graph described in step 4 includes: the Lite Transformer layer is used to fuse relevant data features extracted by the decoder BERT model and comprehensively create a knowledge graph, and the knowledge graph is constructed by receiving abstract tensors from BERT1 and BERT2.

Images