Data asset management and control method and system based on multi-source heterogeneous data source

Abstract

The invention relates to the field of asset management, and provides a data asset management and control method and system based on a multi-source heterogeneous data source, and the method comprises the steps: constructing an engineering data asset card based on the engineering elements of a current engineering project; determining a data topological structure according to the engineering data asset card; wherein the data topological structure comprises a plurality of data fields; based on an open source ETL script and a data topological structure, docking a multi-source heterogeneous data source and a data field of a current engineering project, and cleaning project data output by the multi-source heterogeneous data source to generate business data; establishing a data distribution sharing monitoring mechanism based on the engineering data asset card; and when the business data is distributed to different project businesses by using a data distribution sharing monitoring mechanism, executing data distribution monitoring.

Description

Technical Field

[0001] The present invention relates to the field of data asset control, and in particular to a data asset management and control method and system based on multi-source heterogeneous data sources. Background Art

[0002] At present, data is regarded as an important production factor, playing a key role in the digital economy era. It is the foundation of digitization, networking, and intelligence, and has penetrated into various links such as production, distribution, circulation, consumption, and social service management, changing the mode of production, lifestyle, and social governance. The management and control of digital assets has gradually become the core asset of various enterprises. At present, enterprises usually establish data warehouses and extract all different data into the data warehouses. They only manage the cards of data assets, and do not continuously control the update, iteration, and value application of data assets throughout the whole process.

[0003] The existing digitalization of large-scale projects is often established in a chimney-like manner, such as project management, smart construction sites, and intelligent construction, which leads to a large amount of redundancy in the source of data and the application of data, as well as inconsistent data sources, which leads to the inability to unify data management and control, and even multiple systems for processing based on certain engineering elements. Some projects are aggregated and distributed through the bus, some projects use ETL tools to extract data at regular intervals to achieve unified management of data warehouses, and some projects use data middle platforms and other models for overall management. They often focus on result management, and indeed the full life cycle management of data asset cards, ignoring the reuse value of data, and failing to sort out the source of data and the distribution of data. In the end, data is not transformed into an important factor for improving the quality and efficiency of engineering project management. Summary of the invention

[0004] The present invention proposes a data asset management and control method and system based on multi-source heterogeneous data sources, which is used to solve the problem that enterprises usually establish data warehouses, extract all different data into the data warehouses, and only manage data asset cards, but have no continuous management of the entire process of data asset updates, iterations, and value applications.

[0005] The present invention proposes a data asset management and control method based on multi-source heterogeneous data sources, comprising:

[0006] Based on the engineering elements of the current project, build engineering data asset cards;

[0007] Determine a data topology structure according to the engineering data asset card; wherein the data topology structure includes a plurality of data fields;

[0008] Based on open source ETL scripts and data topology, the multi-source heterogeneous data sources and data fields of the current engineering project are connected, and the project data output by the multi-source heterogeneous data sources is cleaned to generate business data;

[0009] Establish a data distribution, sharing and monitoring mechanism based on engineering data asset cards;

[0010] The data distribution sharing monitoring mechanism is used to perform data distribution monitoring when business data is distributed to different project businesses.

[0011] In combination with the first aspect, the engineering elements based on the current engineering project and constructing the engineering data asset card include:

[0012] Based on the existing project management and control model, a static knowledge base of project element data assets is constructed;

[0013] Build an engineering element data model based on the static knowledge base;

[0014] Determine the engineering elements of the current project based on the engineering element data model;

[0015] According to the engineering elements of the current project, construct element data asset cards for each engineering element; wherein each element data asset card is configured with corresponding data fields;

[0016] According to the element data asset cards, the engineering asset data of the engineering elements is integrated to generate engineering data asset cards.

[0017] In combination with the first aspect, the engineering elements based on the current engineering project and constructing the engineering data asset card also include:

[0018] According to the static knowledge base, sort out the data relationship of each factor data asset card;

[0019] According to the data relationship, determine the data source of the corresponding engineering data in the current engineering data asset card, and determine the multi-source heterogeneous data source based on the data source.

[0020] In combination with the first aspect, determining the data topology structure according to the engineering data asset card includes:

[0021] According to the engineering data asset card, element description information and element relationships of different engineering elements are determined, wherein: the element description information includes multiple data fields corresponding to the engineering elements;

[0022] According to the preset topology parsing script, the structural data in the feature description information is parsed and identified to obtain the topological structure features;

[0023] Generate data topology structure based on topological structure characteristics and feature relationships.

[0024] In combination with the first aspect, the multi-source heterogeneous data sources and data fields of the current engineering project are connected based on the open source ETL technology and data topology structure, including:

[0025] Obtain open source ETL scripts and build multi-heterogeneous docking tools, including:

[0026] The multi-heterogeneous docking tool includes a multi-mode data acquisition component and a directional component. The multi-mode data acquisition component includes a data type identification component, a data transmission identification component and a data encryption identification component. The directional component includes a real-time control component, a timing control component, an automatic control component, a manual control component, a one-way control component and a two-way control component.

[0027] Based on the multi-heterogeneous docking tool, an automatic parsing method is established for each element data asset card and the corresponding data source, and the card content of the corresponding element data asset card is generated through the automatic parsing method, where:

[0028] Automatic parsing methods include: automatic authentication method, automatic encryption and decryption method, automatic filtering method and automatic transmission method.

[0029] In combination with the first aspect, the step of cleaning the project data outputted from the multiple heterogeneous data sources to generate business data includes:

[0030] Connect multiple heterogeneous data sources with pre-trained business data classification models to obtain business classification data and data loss functions;

[0031] Perform gradient calculation on the business classification data to determine the corresponding gradient vector;

[0032] According to the gradient vector, the business classification data is matched with the preset business cleaning rules to determine the cleaning process;

[0033] According to the cleaning process, the business classification data is cleaned, and the cleaning process is optimized through the data loss function to generate business data.

[0034] In combination with the first aspect, the optimizing the cleaning process by using the data loss function includes:

[0035] Calculate the cleaning process of business classification data and determine the outliers in the cleaning process;

[0036] Determine a cleaning feature graph according to the outlier and the cleaning feature of at least one business classification data;

[0037] At least one business classification data is cleaned and optimized according to the cleaning feature graph and the data loss function.

[0038] In combination with the first aspect, the data distribution sharing monitoring mechanism is established based on the engineering data asset card, including:

[0039] According to the project asset cards, determine the distribution nodes of different businesses; the distribution nodes include: management layer nodes, processing layer nodes and application layer nodes;

[0040] Configure the distributed shared task supervision mechanism at the management level node;

[0041] Configure the execution supervision mechanism of distribution sharing in the processing layer nodes;

[0042] Configure the response supervision mechanism for distribution sharing at the application layer nodes;

[0043] Upload the task supervision mechanism, execution acceptance mechanism and response supervision mechanism to the data alliance chain to generate a data distribution and sharing monitoring mechanism.

[0044] In combination with the first aspect, the data distribution sharing monitoring mechanism is used to perform data distribution monitoring when the business data is distributed to different project businesses, including:

[0045] Based on data asset cards and data distribution and sharing monitoring mechanisms, establish dynamic monitoring mechanisms for data asset functions and applications;

[0046] Determine the distribution mode, update log, and version iteration information of business data based on the dynamic functional supervision mechanism;

[0047] According to the dynamic monitoring mechanism of the application, determine the data embedding point and data tracking mechanism, and determine the calling location and application location of each business data.

[0048] In a second aspect, the present invention proposes a data asset management and control system based on multi-source heterogeneous data sources, comprising:

[0049] Asset card module: used to pass the engineering elements of the current project and build engineering data asset cards;

[0050] Data topology module: used to determine the data topology structure according to the engineering data asset card; wherein the data topology structure includes multiple data fields;

[0051] Data cleaning module: used to connect the multi-source heterogeneous data sources and data fields of the current project through open source ETL scripts and data topology structures, and clean the project data output by the multi-source heterogeneous data sources to generate business data;

[0052] Data monitoring module: used to establish a data distribution and sharing monitoring mechanism through engineering data asset cards;

[0053] Data distribution module: used to perform data distribution monitoring when business data is distributed to different project businesses using the data distribution sharing monitoring mechanism.

[0054] The beneficial effects of the present invention are:

[0055] The present invention uses the TGPETL tool to automatically capture, distribute, and clean multi-source heterogeneous data sources, and refine engineering elements into data asset cards. Through the subdivision and modeling of data asset cards, data integration with various application systems in engineering digitization is achieved. Based on the perfect data cards, distribution, sharing and full-life monitoring are achieved. Finally, the efficient application of data elements is submitted to the digital archive.

[0056] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0057] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0059] Figure 1 This is a method flow chart of a data asset management and control method based on multi-source heterogeneous data sources in an embodiment of the present invention;

[0060] Figure 2 This is a system composition diagram of a data asset management and control system based on multi-source heterogeneous data sources in an embodiment of the present invention;

[0061] Figure 3 It is the task monitoring interface of the data asset management and control platform in the embodiment of the present invention;

[0062] Figure 4 A schematic diagram of a data distribution monitoring interface according to an embodiment of the present invention;

[0063] Figure 5 is a schematic diagram of a metadata management interface in an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of the interface of the data warehouse list in an embodiment of the present invention;

[0065] Figure 7 Schematic diagram of different data item interfaces in an embodiment of the present invention;

[0066] Figure 8 is a schematic diagram of a task monitoring interface in an embodiment of the present invention;

[0067] Fig. 9 A schematic diagram of a data service in an embodiment of the present invention;

[0068] Fig.10 Schematic diagram of the TGPETL operation interface in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0070] like Figure 1 As shown, this embodiment provides a data asset management method based on multi-source heterogeneous data sources, including:

[0071] Based on the engineering elements of the current project, build engineering data asset cards;

[0072] Determine a data topology structure according to the engineering data asset card; wherein the data topology structure includes a plurality of data fields;

[0073] Based on open source ETL scripts and data topology, the multi-source heterogeneous data sources and data fields of the current engineering project are connected, and the project data output by the multi-source heterogeneous data sources is cleaned to generate business data;

[0074] Establish a data distribution, sharing and monitoring mechanism based on engineering data asset cards;

[0075] The data distribution sharing monitoring mechanism is used to perform data distribution monitoring when business data is distributed to different project businesses.

[0076] The principle of the above technical solution is:

[0077] In the actual implementation process of the present invention, the engineering elements of the engineering construction are pre-set, and engineering data asset cards are established based on the engineering elements. The engineering elements include safety hazard data, quality acceptance data, investment estimate data, etc., and the data model is sorted out in the business management and control mode.

[0078] Build the corresponding data structure based on the data asset card, annotate the data source for each data field, establish the relationship between the data field and the multi-source heterogeneous data source, and build the data topology structure. Fig.10 , Figure 5 and Figure 7As shown, Figure 7 Indicates different data items, Figure 5 Indicates the metadata corresponding to different data items, Fig.10 For the operation interfaces of different operation components), data labeling is achieved through certain data fields with corresponding data characteristics.

[0079] In this process, the multi-source heterogeneous docking tool TGPETL (such as Fig.10 As shown in the figure, the data fields are connected with multi-source heterogeneous data sources, such as real-time interface connection, database connection, document connection and crawler connection, etc. The connection is realized based on the TGPETL tool, and the data cleaning rules are edited during the connection process to realize real-time cleaning and data acquisition. Data distribution sharing monitoring is established based on data asset cards, and data is distributed to different businesses for application and real-time monitoring.

[0080] The beneficial effects of the present invention are:

[0081] The present invention uses the independently developed TGPETL tool to realize the automatic capture, distribution, and cleaning of multi-source heterogeneous data sources, and refines and splits the engineering elements into data asset cards. Through the subdivision and modeling of data asset cards, data integration with various application systems in engineering digitization is realized. And based on the perfect data cards, distribution, sharing and full life cycle monitoring are realized. Finally, the efficient application of data elements is submitted to the database of digital archives.

[0082] As an embodiment of the present invention, the engineering elements based on the current engineering project and constructing the engineering data asset card include:

[0083] Based on the existing project management and control model, a static knowledge base of project element data assets is constructed;

[0084] Build an engineering element data model based on the static knowledge base;

[0085] Determine the engineering elements of the current project based on the engineering element data model;

[0086] According to the engineering elements of the current project, construct element data asset cards for each engineering element; wherein each element data asset card is configured with corresponding data fields;

[0087] According to the element data asset cards, the engineering asset data of the engineering elements is integrated to generate engineering data asset cards.

[0088] The principle of the above technical solution is:

[0089] In the actual implementation process, the present invention builds a static knowledge base of data assets (such as Figure 6 (As shown in the figure) Sort out the data relationship corresponding to each data asset card, and clean up the data sources of the elements in the current data asset card based on the data relationship. For example, for security data asset cards, some sources come from engineering management systems, some come from smart construction sites, and even some come from BIM systems.

[0090] The static knowledge base construction process is based on the existing engineering project management and control model. By collecting, organizing and classifying key information in engineering projects, a knowledge base containing standardized data elements, attributes and relationships is constructed. The static knowledge base uses a database management system (DBMS) to store the knowledge base to ensure data consistency and accessibility. The information in the static knowledge base is structured and a data model is defined, including entities, attributes and relationships, to support the logical representation and operation of data. The construction of the engineering element model is realized.

[0091] Based on the constructed data model, identify and extract the key elements of the current project, such as project scope, cost, schedule, quality, resources, etc. Finally, create a detailed data record card for each project element, recording all its relevant data fields and attributes. Integrate the data in each element data asset card into a unified view to form a project data asset card.

[0092] In the process of constructing engineering data asset cards, the present invention constructs a static knowledge base and a data model, automatically determines engineering elements and generates data asset cards, simplifies the data asset management process, configures specific data fields for each engineering element, and ensures the consistency and accuracy of data assets. The establishment of a static knowledge base and a data model provides a basis for data standardization and helps eliminate data heterogeneity problems.

[0093] As an embodiment of the present invention, the construction of engineering data asset cards based on engineering elements of the current engineering project also includes:

[0094] According to the static knowledge base, sort out the data relationship of each factor data asset card;

[0095] According to the data relationship, determine the data source of the corresponding engineering data in the current engineering data asset card, and determine the multi-source heterogeneous data source based on the data source.

[0096] The principle of the above technical solution is:

[0097] In the specific implementation, sorting out the data relationships of factor data asset cards is to analyze the logical relationships between the data fields of each data asset card in the static knowledge base, including hierarchical structures, dependencies, association rules, etc.

[0098] Data mapping and association rule mining techniques are used to identify and define the relationships between various data elements in data asset cards.

[0099] Based on the sorted out data relationships, trace and determine the specific data source of each data field in the engineering data asset card, which may include internal systems, external databases, sensors, documents, etc.

[0100] Finally, identify and integrate data from different sources and formats that may differ in structure, semantics, and storage methods.

[0101] As an embodiment of the present invention, determining the data topology structure according to the engineering data asset card includes:

[0102] According to the engineering data asset card, element description information and element relationships of different engineering elements are determined, wherein: the element description information includes multiple data fields corresponding to the engineering elements;

[0103] According to the preset topology parsing script, the structural data in the feature description information is parsed and identified to obtain the topological structure features;

[0104] Generate data topology structure based on topological structure characteristics and feature relationships.

[0105] The principle of the above technical solution is:

[0106] In the actual implementation process, the specific description information of each engineering element is identified through the engineering data asset card. This information usually consists of multiple data fields, and the relationship between different engineering elements is determined. Specifically, the metadata in the data asset card is used to extract the element description information, and the relationship between the elements is represented by a relational model or a graph model. The structured data in the element description information is parsed using a pre-set topology parsing script to identify and extract the topological features of the data. The script usually contains a series of parsing rules and algorithms for extracting key information from data fields and identifying hierarchical structures, dependencies, etc. between data. Based on the parsed topological structure features and known element relationships, the data topology structure of the entire engineering project is constructed.

[0107] As an embodiment of the present invention, the method of docking the multi-source heterogeneous data sources and data fields of the current engineering project based on the open source ETL technology and data topology structure includes:

[0108] Obtain open source ETL scripts and build multi-heterogeneous docking tools, including:

[0109] The multi-heterogeneous docking tool includes a multi-mode data acquisition component and a directional component. The multi-mode data acquisition component includes a data type identification component, a data transmission identification component and a data encryption identification component. The directional component includes a real-time control component, a timing control component, an automatic control component, a manual control component, a one-way control component and a two-way control component.

[0110] Based on the multi-heterogeneous docking tool, an automatic parsing method is established for each element data asset card and the corresponding data source, and the card content of the corresponding element data asset card is generated through the automatic parsing method, where:

[0111] Automatic parsing methods include: automatic authentication method, automatic encryption and decryption method, automatic filtering method and automatic transmission method.

[0112] The principle of the above technical solution is:

[0113] In specific implementation, the present invention implements an autonomous and controllable multi-source heterogeneous docking tool TGPETL based on open source ETL technology. TGPETL can implement a variety of services, such as Fig. 9 As shown. The data fields are connected with multi-source heterogeneous data sources, such as real-time interface connection, database connection, document connection and crawler connection. In the specific implementation process, the self-controllable multi-source heterogeneous connection tool TGPETL based on open source ETL technology is built. The tool can collect data according to the data type, data transmission method and data encryption method. In the collection process, it can realize real-time, scheduled, automatic, manual, one-way and two-way acquisition. Based on the TGPETL tool, each field of the data asset card is connected with the corresponding source to establish data automatic authentication, automatic encryption and decryption, automatic filtering and automatic transmission, and realize automatic cleaning, so as to construct the corresponding digital card content.

[0114] As an embodiment of the present invention, the step of cleaning the project data outputted from the multi-dimensional heterogeneous data sources to generate business data includes:

[0115] Connect multiple heterogeneous data sources with pre-trained business data classification models to obtain business classification data and data loss functions;

[0116] Perform gradient calculation on the business classification data to determine the corresponding gradient vector;

[0117] According to the gradient vector, the business classification data is matched with the preset business cleaning rules to determine the cleaning process;

[0118] According to the cleaning process, the business classification data is cleaned, and the cleaning process is optimized through the data loss function to generate business data.

[0119] The principle of the above technical solution is:

[0120] In the actual implementation process, the connection between multi-source heterogeneous data sources and business data classification models is to integrate data from different sources and formats into a unified framework, and use the pre-trained business data classification model to classify these data, and obtain the data loss function to evaluate the classification effect. By calculating the gradient of the data loss function with respect to the business classification data, the sensitivity information of the data classification error is obtained, thereby guiding the direction of data cleaning. According to the gradient vector, the business classification data is matched with the preset cleaning rules to determine which rules are applicable to the current data cleaning task. According to the determined cleaning process, the business classification data is cleaned to remove noise and correct errors and inconsistent data. The data loss function is used to evaluate the quality of the cleaned data, and the loss function is minimized by iteratively optimizing the cleaning process.

[0121] As an embodiment of the present invention, the optimization of the cleaning process by using the data loss function includes:

[0122] Calculate the cleaning process of business classification data and determine the outliers in the cleaning process;

[0123] Determine a cleaning feature graph according to the outlier and the cleaning feature of at least one business classification data;

[0124] At least one business classification data is cleaned and optimized according to the cleaning feature graph and the data loss function.

[0125] The principle of the above technical solution is:

[0126] In the specific implementation process, the outliers in the cleaning process are determined, and the outliers in the data are identified by calculating the cleaning process of the business classification data. These outliers may be noise, erroneous data or abnormal patterns. According to the identified outliers and the cleaning characteristics of the business classification data (such as the distribution, frequency, trend, etc. of the data), a cleaning feature graph is constructed, which shows the relationship between the data characteristics and the outliers. Based on the cleaning feature graph and the data loss function, the business classification data is cleaned and optimized to reduce data loss and improve data quality.

[0127] As an embodiment of the present invention: the data distribution sharing monitoring mechanism is established based on the engineering data asset card, including:

[0128] According to the project asset cards, determine the distribution nodes of different businesses; the distribution nodes include: management layer nodes, processing layer nodes and application layer nodes;

[0129] Configure the distributed shared task supervision mechanism at the management level node;

[0130] Configure the execution supervision mechanism of distribution sharing in the processing layer nodes;

[0131] Configure the response supervision mechanism for distribution sharing at the application layer nodes;

[0132] Upload the task supervision mechanism, execution acceptance mechanism and response supervision mechanism to the data alliance chain to generate a data distribution and sharing monitoring mechanism.

[0133] The principle of the above technical solution is:

[0134] In the specific implementation process, the distribution nodes for different businesses are determined by identifying the distribution nodes corresponding to different business needs based on the engineering data asset cards. These nodes are divided according to the different responsibilities of the management layer, processing layer, and application layer. At the management layer node, a supervision mechanism is established to ensure that the data distribution tasks are executed in accordance with the established policies and rules. At the processing layer node, the actual execution process of data distribution is supervised to ensure the quality and efficiency of data processing. At the application layer node, the response of the data recipient is monitored to ensure the effective use and feedback of the data. Finally, the supervision mechanism is uploaded to the data alliance chain, and the immutability and transparency of the blockchain are used to enhance the trust and security of the monitoring mechanism.

[0135] As an embodiment of the present invention, the data distribution sharing monitoring mechanism is used to perform data distribution monitoring when the business data is distributed to different project businesses, including:

[0136] Based on data asset cards and data distribution and sharing monitoring mechanisms, establish dynamic monitoring mechanisms for data asset functions and applications;

[0137] Determine the distribution mode, update log, and version iteration information of business data based on the dynamic functional supervision mechanism;

[0138] According to the dynamic monitoring mechanism of the application, determine the data embedding point and data tracking mechanism, and determine the calling location and application location of each business data.

[0139] The technical principle of the above scheme is: in the final distribution process, data distribution sharing monitoring is established based on data asset cards, data is distributed to different businesses for application and real-time monitoring, and based on the data asset cards, data asset monitoring, distribution, update, version iteration and other functions are established to dynamically monitor data assets throughout their life cycle. Based on the data asset card monitoring, dynamic monitoring is performed on who uses each card, how to call it, and where it is applied through embedding and tracking.

[0140] like Figure 2 to Figure 10 As shown, in a second aspect, the present invention proposes a data asset management and control system based on multi-source heterogeneous data sources, comprising:

[0141] Asset card module: used to pass the engineering elements of the current project and build engineering data asset cards;

[0142] Data topology module: used to determine the data topology structure according to the engineering data asset card; wherein the data topology structure includes multiple data fields;

[0143] Data cleaning module: used to connect the multi-source heterogeneous data sources and data fields of the current project through open source ETL scripts and data topology structures, and clean the project data output by the multi-source heterogeneous data sources to generate business data;

[0144] Data monitoring module: used to establish a data distribution and sharing monitoring mechanism through engineering data asset cards;

[0145] Data distribution module: used to perform data distribution monitoring when business data is distributed to different project businesses using the data distribution sharing monitoring mechanism.

[0146] The principle of the above technical solution is:

[0147] In the actual real-time process, the present invention establishes engineering data asset cards based on engineering elements, such as safety hazard data, quality acceptance data, investment estimate data, etc., and sorts out the data model in the business management and control mode. Construct the corresponding data structure based on the data asset card, annotate the data source for each data field, and establish the relationship between the data field and the multi-source heterogeneous data source. Implement an autonomous and controllable multi-source heterogeneous docking tool TGPETL based on open source ETL technology, and implement docking methods between data fields and multi-source heterogeneous data sources, such as real-time interface docking, database docking, document docking, and crawler docking; implement docking based on the TGPETL tool, and annotate data cleaning rules during the docking process to achieve real-time cleaning and data acquisition. Establish data distribution and sharing monitoring based on data asset cards (such as Figure 3 and Figure 4 as well as Figure 8 As shown), the data is distributed to different businesses for application and real-time monitoring.

[0148] The beneficial effects of the present invention are:

[0149] The present invention uses the independently developed TGPETL tool to realize the automatic capture, distribution, and cleaning of multi-source heterogeneous data sources, and refines and splits the engineering elements into data asset cards. Through the subdivision and modeling of data asset cards, data integration with various application systems in engineering digitization is realized. And based on the perfect data cards, distribution, sharing and full life cycle monitoring are realized. Finally, the efficient application of data elements is submitted to the digital archive.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data asset management and control method based on multi-source heterogeneous data sources, characterized in that: include: Based on the engineering elements of the current project, build engineering data asset cards; Determine a data topology structure according to the engineering data asset card; wherein the data topology structure includes a plurality of data fields; Based on open source ETL scripts and data topology, the multi-source heterogeneous data sources and data fields of the current engineering project are connected, and the project data output by the multi-source heterogeneous data sources is cleaned to generate business data; Establish a data distribution, sharing and monitoring mechanism based on engineering data asset cards; The data distribution sharing monitoring mechanism is used to perform data distribution monitoring when business data is distributed to different project businesses.

2. A data asset management and control method based on multi-source heterogeneous data sources as claimed in claim 1, characterized in that: The engineering elements based on the current engineering project and the construction of engineering data asset cards include: Based on the existing project management and control model, a static knowledge base of project element data assets is constructed; Build an engineering element data model based on the static knowledge base; Determine the engineering elements of the current project based on the engineering element data model; According to the engineering elements of the current project, construct element data asset cards for each engineering element; wherein each element data asset card is configured with corresponding data fields; According to the element data asset cards, the engineering asset data of the engineering elements is integrated to generate engineering data asset cards.

3. A data asset management and control method based on multi-source heterogeneous data sources as claimed in claim 2, characterized in that: The engineering elements based on the current engineering project and constructing the engineering data asset card also include: According to the static knowledge base, sort out the data relationship of each factor data asset card; According to the data relationship, determine the data source of the corresponding engineering data in the current engineering data asset card, and determine the multi-source heterogeneous data source based on the data source.

4. A data asset management and control method based on multi-source heterogeneous data sources as claimed in claim 2, characterized in that: Determining the data topology structure according to the engineering data asset card includes: According to the engineering data asset card, element description information and element relationships of different engineering elements are determined, wherein: the element description information includes multiple data fields corresponding to the engineering elements; According to the preset topology parsing script, the structural data in the feature description information is parsed and identified to obtain the topological structure features; Generate data topology structure based on topological structure characteristics and feature relationships.

5. A data asset management and control method based on multi-source heterogeneous data sources as claimed in claim 2, characterized in that: Based on the open source ETL script and data topology structure, the multi-source heterogeneous data sources and data fields of the current engineering project are connected, including: Obtain open source ETL scripts and build multi-heterogeneous docking tools, including: The multi-heterogeneous docking tool includes a multi-mode data acquisition component and a directional component. The multi-mode data acquisition component includes a data type identification component, a data transmission identification component and a data encryption identification component. The directional component includes a real-time control component, a timing control component, an automatic control component, a manual control component, a one-way control component and a two-way control component. Based on the multi-heterogeneous docking tool, an automatic parsing method is established for each element data asset card and the corresponding data source, and the card content of the corresponding element data asset card is generated through the automatic parsing method, where: Automatic parsing methods include: automatic authentication method, automatic encryption and decryption method, automatic filtering method and automatic transmission method.

6. A data asset management and control method based on multi-source heterogeneous data sources as claimed in claim 1, characterized in that: The project data output from the multi-source heterogeneous data sources is cleaned to generate business data, including: Connect multiple heterogeneous data sources with pre-trained business data classification models to obtain business classification data and data loss functions; Perform gradient calculation on the business classification data to determine the corresponding gradient vector; According to the gradient vector, the business classification data is matched with the preset business cleaning rules to determine the cleaning process; According to the cleaning process, the business classification data is cleaned, and the cleaning process is optimized through the data loss function to generate business data.

7. A data asset management and control method based on multi-source heterogeneous data sources as claimed in claim 6, characterized in that: The cleaning process is optimized by using a data loss function, including: Calculate the cleaning process of business classification data and determine the outliers in the cleaning process; Determine a cleaning feature graph according to the outlier and the cleaning feature of at least one business classification data; At least one business classification data is cleaned and optimized according to the cleaning feature graph and the data loss function.

8. A data asset management and control method based on multi-source heterogeneous data sources as claimed in claim 1, characterized in that: The data distribution sharing monitoring mechanism is established based on the engineering data asset card, including: According to the project asset cards, determine the distribution nodes of different businesses; the distribution nodes include: management layer nodes, processing layer nodes and application layer nodes; Configure the distributed shared task supervision mechanism at the management level node; Configure the execution supervision mechanism of distribution sharing in the processing layer nodes; Configure the response supervision mechanism for distribution sharing at the application layer nodes; Upload the task supervision mechanism, execution acceptance mechanism and response supervision mechanism to the data alliance chain to generate a data distribution and sharing monitoring mechanism.

9. A data asset management and control method based on multi-source heterogeneous data sources as claimed in claim 1, characterized in that: The data distribution sharing monitoring mechanism is used to perform data distribution monitoring when the business data is distributed to different project businesses, including: Based on data asset cards and data distribution and sharing monitoring mechanisms, establish dynamic monitoring mechanisms for data asset functions and applications; Determine the distribution mode, update log, and version iteration information of business data based on the dynamic functional supervision mechanism; According to the dynamic monitoring mechanism of the application, determine the data embedding point and data tracking mechanism, and determine the calling location and application location of each business data.

10. A data asset management and control system based on multi-source heterogeneous data sources, characterized in that: include: Asset card module: used to pass the engineering elements of the current project and build engineering data asset cards; Data topology module: used to determine the data topology structure according to the engineering data asset card; wherein the data topology structure includes multiple data fields; Data cleaning module: used to connect the multi-source heterogeneous data sources and data fields of the current project through open source ETL scripts and data topology structures, and clean the project data output by the multi-source heterogeneous data sources to generate business data; Data monitoring module: used to establish a data distribution and sharing monitoring mechanism through engineering data asset cards; Data distribution module: used to perform data distribution monitoring when business data is distributed to different project businesses using the data distribution sharing monitoring mechanism.

Images