Asset checking method and system based on data consanguinity

Abstract

The invention discloses an asset checking method and system based on data consanguinity, and the method comprises the steps: extracting key information from a plurality of external systems, including business metadata information, a basic data link relationship and basic operation index data; performing data conversion on the collected data according to blood relationship standard information, wherein the converted data is used as a basic element of data blood relationship construction and asset inventory; metadata information and data link relation information are automatically obtained from a data conversion result, data consanguinity is constructed according to a template, a consanguinity map is generated, and the integrity of a consanguinity link is analyzed, so that the consanguinity quality is guaranteed; performing weight evaluation, data marking and real-time blood relationship life cycle tracking and recording on the data based on a blood relationship map, and further perfecting blood relationship result data; and performing asset identification, asset classification and asset updating according to the blood relationship result data to complete asset inventory. According to the invention, enterprises are helped to comprehensively understand and track the source, the flow direction and the use condition of assets, so that accurate asset checking is carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of asset inventory, and in particular to an asset inventory method and system based on data lineage. Background Art

[0002] In the context of the increasing importance of data element transactions and data asset management, asset inventory is becoming increasingly important as a core link for enterprises to understand the digital assets they own. Many enterprises use traditional tools to scan data sources (such as databases, file systems, and data lakes) to identify and understand data distribution. There are certain limitations in data relevance, making it difficult to fully identify and integrate all data assets, especially in large enterprises where data sources are diverse and data is distributed across multiple subsystems and platforms. Asset inventory is often viewed as a one-time task, without long-term continuous monitoring, and a regular update and maintenance asset inventory mechanism has not been established, resulting in outdated data asset information, lags and omissions in the data element participation link, and cannot play an important role in development.

[0003] When enterprises need to take inventory of a large number of data assets, they cannot automatically identify them and need to spend a lot of manpower to process them. It is impossible to directly obtain the connection and importance level between assets through business data relationships. The resulting data assets are too single and there is a serious disconnect between the data value, which affects the effect of asset inventory. At the same time, when the business data relationship changes due to changes in business system requirements, the asset relationship cannot be adjusted in time, resulting in assets not being consistent with the actual situation, affecting asset operation decisions. Summary of the invention

[0004] The purpose of the present invention is to provide a data lineage-based asset inventory method and system, which is a technical method for identifying and managing assets through data lineage analysis and tracking data flow relationships. This method is mainly used to optimize data asset management in enterprises, helping enterprises to fully understand and track the source, flow and use of assets, so as to conduct accurate asset inventory.

[0005] The technical solution adopted by the present invention is: An asset inventory method based on data lineage includes the following steps: Step 1: Extract key information from multiple external systems, including business metadata information, basic data link relationships, and basic operating indicator data; Furthermore, the business metadata information in step 1 includes metadata information and sample data.

[0006] Step 2: Convert the collected data according to the lineage standard information, and use the converted data as the basic element for data lineage construction and asset inventory; Furthermore, the data in step 2 is converted to obtain metadata, data links, and indicator data.

[0007] Furthermore, the collection process is tracked and monitored in real time. When business data relationships are changed due to changes in business system requirements, the basic data is compared for differences to ensure timely adjustment of data link relationships.

[0008] Step 3: Automatically obtain metadata information and data link relationship information from the data conversion results, build data lineage according to the template, generate a lineage map, and analyze the integrity of the lineage link to ensure the quality of lineage; Step 4: Based on the blood relationship map, the data is weighted and labeled, and the blood relationship life cycle is tracked and recorded in real time to further improve the blood relationship result data; Furthermore, step 4 specifically includes the following steps: Step 4-1: Configure weight rules according to the complexity of blood relationship, output weight indicators according to the rule algorithm, perform weight evaluation based on the weight indicators, and generate data evaluation results; Step 4-2: Based on the bloodline standard information and weight value, extract the mapping data label data and generate the bloodline node labeling result; Step 4-3: Track and record changes in blood relationships in real time, adjust blood link relationships in a timely manner, ensure the integrity and accuracy of the blood life cycle, automatically trigger change applications for changed relationships, and synchronize them to asset inventory.

[0009] Step 5: Perform asset identification, asset classification, and asset update based on the lineage result data to complete the asset inventory.

[0010] Furthermore, step 5 specifically includes the following steps: Step 5-1: Perform asset identification operations based on the weight results, automatically push qualified asset data, and then confirm and improve the weight rules based on the data; Step 5-2: Classify assets based on data labeling results to provide data support for asset statistics, unified management and asset tracking capabilities; Step 5-3: Implement data asset update operations based on lifecycle change applications. During the update process, perform data review to ensure data accuracy and completeness.

[0011] An asset inventory system based on data lineage, comprising: Pre-processing unit: collects metadata information, data link information, and indicator information of the business system, converts and processes the collected data, and generates basic data for data lineage construction and asset inventory; at the same time, it monitors the changes in data relationships in real time and provides data input for data conversion in a timely manner; Data lineage unit: It constructs lineage, maps lineage, and analyzes lineage links based on the data from the pre-processing unit, and outputs lineage data results in combination with weight evaluation and data labeling capabilities. It also sets a life cycle for lineage, monitors the current topology of lineage in real time, and automatically reports and applies for changes when there are any. Asset inventory unit: Identify assets based on the output lineage data and classify assets by lineage to facilitate subsequent unified management, statistics, maintenance and updates.

[0012] Furthermore, the asset inventory unit performs automatic identification and manual identification configuration by rules during the identification process, and after the identification indicators are updated and manually processed by the assets, the identification rules are automatically updated according to the data algorithm.

[0013] The present invention adopts the above technical solution, which has the following technical features compared with the existing technology: 1. Data traceability: By tracking the source and flow path of the data, data lineage can clearly show the formation and change process of the asset, ensure the finality and traceability of the data, and improve the data quality. 2. Accuracy and consistency: Data lineage helps identify the relationship between data, ensure the accuracy and consistency of asset information, and reduce the risks caused by data errors. 3. Automation and efficiency improvement: Through data lineage technology, asset inventory can be automated, reduce manual intervention, and improve efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments; Figure 1 The figure is a schematic diagram of the principle architecture of an asset inventory system based on data lineage according to the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0016] like Figure 1 As shown, the present invention discloses an asset inventory method based on data lineage, which includes the following steps: Step 1: Extract key information from multiple external systems, including business metadata information, basic data link relationships, and basic operating indicator data; Furthermore, the business metadata information in step 1 includes metadata information and sample data.

[0017] Step 2: Convert the collected data according to the lineage standard information, and use the converted data as the basic element for data lineage construction and asset inventory; Furthermore, the data in step 2 is converted to obtain metadata, data links, and indicator data.

[0018] Furthermore, the collection process is tracked and monitored in real time. When business data relationships are changed due to changes in business system requirements, the basic data is compared for differences to ensure timely adjustment of data link relationships.

[0019] Step 3: Automatically obtain metadata information and data link relationship information from the data conversion results, build data lineage according to the template, generate a lineage map, and analyze the integrity of the lineage link to ensure the quality of lineage; Step 4: Based on the blood relationship map, the data is weighted and labeled, and the blood relationship life cycle is tracked and recorded in real time to further improve the blood relationship result data; Furthermore, step 4 specifically includes the following steps: Step 4-1: Configure weight rules according to the complexity of blood relationship, output weight indicators according to the rule algorithm, perform weight evaluation based on the weight indicators, and generate data evaluation results; Step 4-2: Based on the bloodline standard information and weight value, extract the mapping data label data and generate the bloodline node labeling result; Step 4-3: Track and record changes in blood relationships in real time, adjust blood link relationships in a timely manner, ensure the integrity and accuracy of the blood life cycle, automatically trigger change applications for changed relationships, and synchronize them to asset inventory.

[0020] Step 5: Perform asset identification, asset classification, and asset update based on the lineage result data to complete the asset inventory.

[0021] Furthermore, step 5 specifically includes the following steps: Step 5-1: Perform asset identification operations based on the weight results, automatically push qualified asset data, and then confirm and reversely improve the weight rules based on the data to improve accuracy; Step 5-2: Classify assets based on data labeling results to provide data support for asset statistics, unified management and asset tracking capabilities; Step 5-3: Implement data asset update operations based on lifecycle change applications. During the update process, perform data review to ensure data accuracy and completeness.

[0022] An asset inventory system based on data lineage, comprising: Pre-processing unit: collects metadata information, data link information, and indicator information of the business system, converts and processes the collected data, and generates basic data for data lineage construction and asset inventory; at the same time, it monitors the changes in data relationships in real time and provides data input for data conversion in a timely manner; Data lineage unit: It constructs lineage, maps lineage, and analyzes lineage links based on the data from the pre-processing unit, and outputs complete lineage result data in combination with weight evaluation and data labeling capabilities. It also sets a life cycle for lineage, monitors the current topology of lineage in real time, and automatically reports and applies for processing when there are changes. Asset inventory unit: Identify assets based on the output lineage data and classify assets by lineage to facilitate subsequent unified management, statistics, maintenance and updates.

[0023] Furthermore, the asset inventory unit performs automatic identification and manual identification configuration by rules during the identification process, and after the identification indicators are updated and manually processed by the assets, the identification rules are automatically updated according to the data algorithm.

[0024] The present invention adopts the above technical solution to identify and manage assets through data lineage analysis and tracking of data flow relationships. This method is mainly used to optimize data asset management in enterprises, helping enterprises to fully understand and track the source, flow and use of assets, so as to conduct accurate asset inventory.

[0025] Compared with the existing technology, the present invention has the following technical features: 1. Data traceability: By tracking the source and flow path of data, data lineage can clearly show the formation and change process of assets, ensure the finality and traceability of data, and improve data quality. 2. Accuracy and consistency: Data lineage helps identify the relationship between data, ensure the accuracy and consistency of asset information, and reduce the risks caused by data errors. 3. Automation and efficiency improvement: Through data lineage technology, asset inventory can be automated, reduce manual intervention, and improve efficiency.

[0026] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians of the art without making creative work are within the scope of protection of the present application.

Claims

1. An asset inventory method based on data lineage, characterized by: It includes the following steps: Step 1: Extract key information from multiple external systems, including business metadata information, basic data link relationships, and basic operating indicator data; Step 2: Convert the collected data according to the lineage standard information, and use the converted data as the basic element for data lineage construction and asset inventory; Step 3: Automatically obtain metadata information and data link relationship information from the data conversion results, build data lineage according to the template, generate a lineage map, and analyze the integrity of the lineage link to ensure the quality of lineage; Step 4: Based on the blood relationship map, the data is weighted and labeled, and the blood relationship life cycle is tracked and recorded in real time to further improve the blood relationship result data; Step 5: Perform asset identification, asset classification, and asset update based on the lineage result data to complete the asset inventory.

2. The method for asset inventory based on data lineage according to claim 1, characterized in that: The business metadata information in step 1 includes metadata information and sample data.

3. The method for asset inventory based on data lineage according to claim 1 is characterized by: In step 2, data conversion is performed to obtain metadata, data links, and indicator data.

4. The method for asset inventory based on data lineage according to claim 1, characterized in that: The collection process is tracked and monitored in real time. When business data relationships change due to changes in business system requirements, the basic data is compared for differences to ensure timely adjustment of data link relationships.

5. The method for asset inventory based on data lineage according to claim 1, characterized in that: Step 4 specifically includes the following steps: Step 4-1: Configure weight rules according to the complexity of blood relationship, output weight indicators according to the rule algorithm, perform weight evaluation based on the weight indicators, and generate data evaluation results; Step 4-2: Based on the bloodline standard information and weight value, extract the mapping data label data and generate the bloodline node labeling result; Step 4-3: Track and record changes in blood relationships in real time, adjust blood link relationships in a timely manner, ensure the integrity and accuracy of the blood life cycle, automatically trigger change applications for changed relationships, and synchronize them to asset inventory.

6. The method for asset inventory based on data lineage according to claim 1, characterized in that: Step 5 specifically includes the following steps: Step 5-1: Perform asset identification operations based on the weight results, automatically push qualified asset data, and then confirm and improve the weight rules based on the data; Step 5-2: Classify assets based on data labeling results to provide data support for asset statistics, unified management and asset tracking capabilities; Step 5-3: Implement data asset update operations based on lifecycle change applications. During the update process, perform data review to ensure data accuracy and completeness.

7. An asset inventory system based on data lineage, according to an asset inventory method based on data lineage according to any one of claims 1 to 6, characterized in that: The system includes: Pre-processing unit: collects metadata information, data link information, and indicator information of the business system, converts and processes the collected data, and generates basic data for data lineage construction and asset inventory; at the same time, it monitors the changes in data relationships in real time and provides data input for data conversion in a timely manner; Data lineage unit: It constructs lineage, maps lineage, and analyzes lineage links based on the data from the pre-processing unit, and outputs lineage data results in combination with weight evaluation and data labeling capabilities. It also sets a life cycle for lineage, monitors the current topology of lineage in real time, and automatically reports and applies for changes when there are any. Asset inventory unit: Identify assets based on the output lineage data and classify assets by lineage to facilitate subsequent unified management, statistics, maintenance and updates.

8. The asset inventory system based on data lineage according to claim 7 is characterized by: During the identification process, the asset inventory unit is automatically identified and manually identified by rules. The identification indicators are automatically updated according to the data algorithm after asset updates and manual processing.

Images