Social security big data OLAP pre-processing method and on-line analysis and query method

A preprocessing, big data technology, applied in the field of big data processing, can solve the problems of time information, geographic information redundancy, inefficiency, unsatisfactory and other problems, and achieve the effect of overcoming the inefficiency of space

Inactive Publication Date: 2017-05-17
SOUTH CHINA UNIV OF TECH
View PDF4 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Partial materialization is a typical method of exchanging space for time. By establishing CUBE in advance to reduce the time consumed by table connection, when it comes to multi-dimensional social security data, creating a view for each column will bring time information, geographic information, etc. Huge redundancy, so this method is not satisfactory, and CUBE compression also has low efficiency in high dimensions

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Social security big data OLAP pre-processing method and on-line analysis and query method
  • Social security big data OLAP pre-processing method and on-line analysis and query method
  • Social security big data OLAP pre-processing method and on-line analysis and query method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0040] This example proposes a generalized model of tuples by splitting corresponding tuples through an OLAP query processing engine and a proof of sound mathematics.

[0041] Then, it can be shown that the algorithm can contain or derive all data cubes. Then, through the general query model of CUBE, arbitrary segmentation calculations can be performed, thus realizing the aggregation algorithm. Ultimately, through those single data partition structures, all aggregated combinations can be dynamically extracted, and any desired results can be queried from them. The key features of the algorithm of the present invention can be summarized as follows:

[0042] {query result}=∧∨{all extracted one-dimensional key-value pair data, that is, all single-dimensional segmentation} ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a social security big data OLAP pre-processing method and an on-line analysis and query method. The social security big data OLAP pre-processing method comprises the following steps that data is extracted from a raw data base to construct a data warehouse; one-dimensional data is extracted from the data warehouse; according to the one-dimensional data and combinations of the one-dimensional data, inverse mapping from data attributes to IDs is constructed, and a key-value database is constructed through the inverse mapping; proper combinations of the data attributes are selected so as to guarantee that results of multi-dimensional data aggregation can be obtained by combining multi-dimensional attributes; different partitioning is obtained by combinations of the data attributes, and repeating and redundant partitioning is removed through various partitioning combinations, so that it is ensured that all partitioning combinations are achieved by fewest partitioning, and then one-dimensional CUBE is constructed to complete the pre-processing process. By means of the social security big data OLAP pre-processing method and the on-line analysis and query method, the multi-dimension query result of a data model can be effectively expressed and the storage space occupied by the data model is reduced.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to an OLAP preprocessing method for social security big data and an online analysis and query method. Background technique [0002] With the advent of the era of information data, the government, enterprises and other institutions have accumulated a large amount of social security data. These data contain a lot of information, but reasonable mining is required to form useful information that can be processed, so as to predict the future and Make timely decisions. Many enterprise-level systems can reach terabytes and petabytes of data, and for analysis and decision makers, it is necessary to extract information for reference. The current common practice is to use OLAP (Online Analytical Processing, Online Analytical Processing) data warehouse to store historical data in a data warehouse with relatively small data changes through a series of extraction, cleaning, loading ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 王弘剑张星明
Owner SOUTH CHINA UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap