Multi-source data aggregation sampling method and system based on big data environment

Abstract

The invention belongs to the technical field of big data, and discloses a multi-source data aggregation sampling method and system based on a big data environment, and the method comprises the steps:collecting a plurality of original data sources, wherein each original data source comprises a data source name and at least one association domain; cleaning and identifying the acquired data source,and removing redundant of the acquired data source; obtaining an original strategy list by utilizing a construction program according to the original data source, and sorting the original strategies in the original strategy list to form a strategy list between the data sources; carrying out fusion processing on different source data sets by utilizing a fusion program; carrying out word segmentation on the fused file to form a two-dimensional word frequency matrix of file words; setting a balance verification numerical value, circularly matching each word, and carrying out snowball sampling; and displaying the acquired multi-source data by using a display. According to the method, distributed computing is completed by scheduling the computing nodes by the Spark through the preprocessing module, more efficient data preprocessing can be achieved, practicability is high, and the application range is wide.

Description

technical field [0001] The invention belongs to the technical field of big data, and in particular relates to a multi-source data aggregation sampling method and system based on a big data environment. Background technique [0002] Multi-source data fusion technology refers to the use of relevant means to integrate all the information obtained through investigation and analysis, and to make a unified evaluation of the information, and finally to obtain unified information. The purpose of this technology is to integrate various data information, absorb the characteristics of different data sources, and then extract unified, better and richer information than single data. However, in the process of multi-source data aggregation sampling in the existing big data environment, there is insufficient research on structured data, semi-structured and unstructured data preprocessing, and usually only includes two modules: data acquisition and data cleaning. Moreover, the method of da...

AI Technical Summary

Problems solved by technology

[0004] In the existing multi-source data aggregation sampling process under the big data environment, there is insufficient research on structured data, semi-structured and unstructured data preprocessing, and usually only includes two modules of data acquisition and data cleaning, and the data The cleaning method is also relatively simple, which cannot well meet the needs of users; at the same time, when the data is fused, there is no open link data set as prior knowledge, and it is impossible to efficiently and accurately perform large-scale heterogeneous data sources while reducing the complexity. fusion of

Images