Data source selection method for multi-source heterogeneous data fusion

Abstract

The invention discloses a data source selection method for multi-source heterogeneous data fusion, and belongs to the technical field of big data analysis. The problems that an existing big data analysis mode adopts a manual collection mode, the analysis efficiency is low, and a large amount of resources are wasted are solved. The method includes: on the basis of a heterogeneous data source set, establishing an attribute set of a data analysis task target data set, randomly extracting a target attribute from an attribute set A to serve as a search attribute, and a data source set P is obtainedthrough searching in a data source; constructing each element Pi in the data source set P into a set {Pi} to form a set T; calculating the score of each subset in the set T; obtaining a subset Tmax with the maximum score; judging whether the attribute of the set Tmax contains all target attributes or not; and if yes, judging whether redundant attributes are contained or not, reselecting the redundant attributes, if not, continuing searching, calculating a discrimination function value of the target data source set, continuing searching if the function value is increased, and otherwise, stopping searching to obtain the target data source set.

Description

technical field [0001] The invention belongs to the technical field of big data analysis. Background technique [0002] With the advent of the era of big data, hundreds of millions of data are generated every moment. Based on massive data, people need to extract useful information to understand and even guide people's daily life and work. Hence, big data analytics was born and is becoming an increasingly popular field. [0003] However, for a big data analysis task, how to obtain the data set required by the task is a very critical issue. In many data analysis algorithms, especially most machine learning algorithms, data plays a key role, and data plays a decisive role in the quality of the analysis results. However, people often assume that the dataset is given. However, most of the data sets for data analysis tasks are still obtained manually by experts or institutions in this field. Although manual acquisition of data sets can ensure data quality and is feasible when...

AI Technical Summary

Problems solved by technology

[0004] The present invention aims to solve the problems of low analysis efficiency and a large amount of waste of resources in the existing big data analysis using manual collection

Images