Construction method for big-data acceleration structure

Abstract

The invention relates to a construction method for a big-data acceleration structure. The method comprises the following steps that: A: preprocessing data, and forming a dataset which conforms to an operation process; B: carrying out clustering processing, calculating similarity of records in categories, and enabling the most similar records in a group to be minimum on an aspect of space distanceaccording to the grouping result of a clustering algorithm; C: establishing a mapping relationship among a transaction attribute, transaction attribute weight and a transaction record according to three-level indexes, and circulating the process until all data finish being mapped; and D: carrying out initial compression an index structure, a transaction attribute weight index and the transaction attribute, determining the range of continuously recorded shared attribute weight values, traversing an inverted index mapping structure, and compressing the continuous records under the shared attribute weight values through a process compression algorithm. By use of the method, the acceleration structure of big data correlation analysis can be quickly established, and model processing speed and data loading speed can be obviously quickened.

Description

technical field

[0001] The invention relates to a data acceleration processing method, in particular to a method for building a large data acceleration structure. Background technique

[0002] Big data technology has become the most effective and common technology for processing massive data. Police big data, as one of the most representative scenarios in big data processing scenarios, has attracted more and more attention. In the process of mass data analysis, the processing speed and processing performance of the general big data platform is one of the urgent problems to be solved. In big data analysis scenarios, especially in police big data, the most common analysis method is correlation analysis. Comprehensive correlation analysis of relevant factors involved in the analysis object can effectively improve the accuracy of police analysis. The commonly used big data correlation analysis The algorithm model is an association analysis algorithm model. Association analysis...

Examples

Embodiment Construction

[0040] Such as figure 1The construction method of the large data acceleration structure of the present invention shown includes:

[0041] A. Data preprocessing: Data cleaning, data integration and data conversion are performed on the original data to form a data set that conforms to the operation process.

[0042] Data cleaning described therein includes deletion of raw data, filling of missing values ​​and smoothing / filtering of noisy data.

[0043] In deleting original data and filling in missing values, the missing data is judged by the attribute weight coverage p of the original data, and the attribute weight coverage p is:

[0044]

[0045] Where A is the data attribute, ε is the attribute weight, indicating the importance of the attribute, and ε 1 +ε 2 +...+ε k =1, α indicates whether the attribute value is missing, α∈{0,1}, set the threshold ω, the value range of the threshold ω is (0,1), if p≥ω, fill in the missing value for the missing data, otherwise, then de...

technical field

[0001] The invention relates to a data acceleration processing method, in particular to a method for building a large data acceleration structure. Background technique

[0002] Big data technology has become the most effective and common technology for processing massive data. Police big data, as one of the most representative scenarios in big data processing scenarios, has attracted more and more attention. In the process of mass data analysis, the processing speed and processing performance of the general big data platform is one of the urgent problems to be solved. In big data analysis scenarios, especially in police big data, the most common analysis method is correlation analysis. Comprehensive correlation analysis of relevant factors involved in the analysis object can effectively improve the accuracy of police analysis. The commonly used big data correlation analysis The algorithm model is an association analysis algorithm model. Association analysis...

Examples

Embodiment Construction

[0040] Such as figure 1The construction method of the large data acceleration structure of the present invention shown includes:

[0041] A. Data preprocessing: Data cleaning, data integration and data conversion are performed on the original data to form a data set that conforms to the operation process.

[0042] Data cleaning described therein includes deletion of raw data, filling of missing values ​​and smoothing / filtering of noisy data.

[0043] In deleting original data and filling in missing values, the missing data is judged by the attribute weight coverage p of the original data, and the attribute weight coverage p is:

[0044]

[0045] Where A is the data attribute, ε is the attribute weight, indicating the importance of the attribute, and ε 1 +ε 2 +...+ε k =1, α indicates whether the attribute value is missing, α∈{0,1}, set the threshold ω, the value range of the threshold ω is (0,1), if p≥ω, fill in the missing value for the missing data, otherwise, then de...

Images