Multi-source data aggregation sampling strategy based on big data environment

Abstract

The invention discloses a multi-source data aggregation sampling strategy based on a big data environment. The method comprises the following steps that on the basis of a multi-source data set encoded by GBK, fusion of multiple attributes in one data set and fusion of different source data sets are achieved, and multi-source data fusion operation is completed; fused files are segmented to form a two-dimensional word frequency matrix of file words; high-frequency words are displayed to a user for reference, and random migratory selection of seed root node words required by snowball sampling is conducted; business-goal-oriented seed root node keywords are selected, and the depth of snowball sampling is input; on the basis of seed root node data, a balance check value is set, and the words are circularly matched for snowball sampling; a directed acyclic graph and an adjacent matrix are constructed; a root node clustering network diagram and a logical reasoning diagram which are related to business goals can be obtained. The interference of sample noise on subsequent reasoning can be reduced through the strategy.

Description

technical field [0001] The invention relates to the field of big data, in particular to a multi-source data aggregation sampling strategy based on the big data environment. Background technique [0002] In the big data environment, data related to decision-making goals has the characteristics of multi-source heterogeneity, heterogeneous association, hierarchical nesting, and dynamic evolution. Decision-making goal-oriented multi-source heterogeneous data aggregation and reasoning sampling technology has great practical application value in risk warning, business opportunity prediction and anomaly detection. How to select the range of samples and their attribute features, how to determine the correlation between sample attribute features, how to construct the logical reasoning structure between samples and their attribute features, etc. . [0003] At present, the sampling technology of big data is mainly embodied in single-source or single-form data sampling and random samp...

AI Technical Summary

Problems solved by technology

[0005] (1) Sampling is mostly single-source data or single-form data, and most of them lack the fusion and cross-validation of multi-source data and multi-form data;

[0006] (2) Sampling is mostly random sampling. In a big data environment, random sampling has certain limitations, because in a multi-field, multi-source, multi-carrier, and multi-form big data environment, cross-domain and cross-platform sampling is required;

[0007] (3) Random sampling that ignores the decision-making goal will lead to increased sample noise, resulting in an increase in the proportion of redundant or missing samples;

[0008] (4) Situational reasoning that ignores decision-making goal-oriented sampling and its attribute characteristics will not be able to effectively reveal its potential, future or key samples and their attribute characteristics

Images