Heterogeneous data sets based on field value priority connection method based on mic

Abstract

The invention relates to a field value prior connection method based on MIC of a heterogeneous data set comprising following steps: fitting parameters of heterogeneous data set SE distribution; calculating a MIC coefficient between field A and field B; generating a set StA and a set StB of all occurrences of the values in the fields A and B respectively; establishing cumulative distribution functions PA (x) and PB (y) corresponding to the set StA and the set StB; determining whether total record I is zero or not, if the total record I is zero, then turning to the last step, otherwise, turning to the next step; calculating and obtaining a field value Ax corresponding to the field A according to PA(x); calculating and obtaining a field value By corresponding to the field B based on a field prior connection model; saving {Ax, By} as a record; updating the total record number I=I-1, and returning to step 5; completing all connections of the heterogeneous data. This method is helpful for realistic simulation of heterogeneous data sets to make the connected data set to maintain a reasonable balance between the fields and the similarity between nodes.

Description

Technical field [0001] The invention relates to the technical field of heterogeneous data field value connection, in particular to a method for preferential connection of field values ​​of heterogeneous data sets based on MIC. Background technique [0002] Reasonable analysis of the field content of heterogeneous data sets is helpful for the construction and testing of their domain systems. However, heterogeneous data sets usually reach TB or even PB level, which consumes network resources extremely, and the user behavior and related item attributes in the data are related The content of the fields involves private information, so companies and governments are rarely willing to share their data for researchers to use. With the continuous expansion of the Internet, the heavy tail phenomenon in heterogeneous data has become more and more common, and the connection relationship between various fields has become more and more complicated. It is extremely difficult to generate heterog...

AI Technical Summary

Problems solved by technology

The disadvantage is that proWGen only implements simple positive / negative correlation for field connection, which makes it difficult to realistically simulate complex and diverse heterogeneous data in practice

With the explosive increase of Internet data volume, Zipf-like is no longer suitable for describing heterogeneous data distribution with heavy tail characteristics. If Zipf-like is used for data generation, for the system used to generate data, its test There will be overestimated results in performance evaluation, and there will be a large error compared with the real data situation, which means that unreliable data is generated

Images