Intelligent data blood relationship tracing method and device based on clustering analysis

A cluster analysis and data technology, applied in the field of big data, can solve problems such as inability to complete, data performance impact, and inability to process data lineage, etc., to achieve the effect of improving accuracy and efficiency

CN110083639AActive Publication Date: 2019-08-02中电科嘉兴新型智慧城市科技发展有限公司 +1
6 Cites 12 Cited by

Patent Information

Authority / Receiving Office
CN · China
Current Assignee / Owner
Publication Date
2019-08-02

Smart Images

  • Figure 1
    Figure 1
Patent Text Reader

Abstract

An intelligent data blood relationship tracing method based on clustering analysis comprises the steps that 1, a table structure and data are read, and forming data characteristics of all fields through a data engineering means; step 2, learning the data sample by using a clustering analysis algorithm in machine learning by taking a field as a unit and a field data feature set as a feature; 3, repeatedly executing the clustering analysis in the step 2 until the optimal classification number and the optimal classification number are found; step 4, under the optimal classification, automaticallyjudging the data fields in the same classification as fields which may have a blood relationship; 5, for each blood relationship, inferring the direction of the blood relationship according to the sequence of table creation time pointed by the relationship, namely inferring which field is a source and which field is a target, and if the field of the blood relationship comes from the same table, marking the blood relationship as an invalid blood relationship; and 6, calculating the table blood relationship according to the effective field blood relationship.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the technical field of big data, and in particular relates to a method and device for intelligent traceability of blood relationship of data based on cluster analysis. Background technique

[0002] With the development and popularization of big data and machine learning technology, the amount of data used, managed and generated by data analysis software is increasing, and the dependence on the format, content and quantity of data is also increasing. Before the data analysis system runs, it needs to perform various extraction, cleaning, conversion and desensitization operations on the data. The complexity of these businesses determines that there are many procedures, long processes and complicated methods in the data processing process. It is necessary to trace back the lineage of the data to judge the credibility of the data, analyze the influence of the data, and analyze and process the source of the wrong data. Therefore, e...

Examples

Embodiment Construction

[0021] According to one or more embodiments, such as figure 1 As shown, a method for intelligent traceability of data kinship based on cluster analysis, including steps:

[0022] Step 1: Read the table structure and data, and form the data characteristics of each field through data engineering methods. The specific method is as follows:

[0023] Step 1.1: Analyze the data characteristics of the original data into structured sample data, including field type, field length, field content mode, etc.

[0024] Step 1.2: Combine the existing features in the sample data to form high-dimensional features;

[0025] Step 1.3: Analyze high-dimensional features, form new dimensions and sort the influence of new dimensions;

[0026] Step 1.4: Reduce the dimension of the sample data according to the new dimension, and use the minimum number of dimensions under the premise of ensuring that the distortion rate of the sample data is lower than the set value;

[0027] Step 1.5: Normalize the...