Self-adaption table dimension division based big data processing method
A big data processing and self-adaptive technology, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of reducing the number of dimensions and slow analysis of high-dimensional data, so as to reduce the number of dimensions and reduce data processing. The effect of less storage space
Active Publication Date: 2016-04-13
HANGZHOU DIANZI UNIV
View PDF6 Cites 20 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0005] In view of this, it is indeed necessary to provide a big data processing method based on adaptive table dimension division, which can be automatically divided according to the number of dimensions of the big data storage table, thereby effectively reducing the number of dimensions and solving the shortcomings of slow analysis of high-dimensional data
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0043] see figure 1 , is shown as a flow chart of a method for processing big data based on adaptive table dimension division in the present invention, including the following steps:
[0044] Step S1: Configure and access data sources according to user needs to generate multiple required data tables and import them into the data warehouse. Each data table has a unique tableName;
[0045] Step S2: performing a data cleaning operation on the generated data table;
[0046] Step S3: Perform data preprocessing operations on the data table and pre-store the preprocessed result information in the columnar database;
[0047] Step S4: Perform a data query operation according to the result information pre-stored in the database.
[0048] see figure 2 , is shown as a detailed flowchart of step S3 in a large data processing method based on adaptive table dimen...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a self-adaptation table dimension division based big data processing method. The method comprises steps as follows: S1: configuration is performed according to requirements of a user, a data source is accessed, and multiple required data tables are generated and imported into a data warehouse; S2, data cleaning operation is performed on the generated data tables; S3, data pre-processing operation is performed on the data tables, and result information acquired after pre-processing is pre-stored in a column-oriented database; S4, data inquiry operation is performed on the result information pre-stored in the database. According to the technical scheme, dimensions of the imported data tables can be divided into multiple sub-dimensions automatically, accordingly, the dimensions can be effectively reduced and can be quickly analyzed, and the memory space is little occupied; meanwhile, a dimension support degree set is set, the support degree of the dimensions can be calculated according to inquiry operation of the user, closely related dimensions can be automatically put into one sub-dimension, and the data processing capacity is greatly reduced.
Description
technical field [0001] The invention relates to the technical field of big data query analysis, in particular to a big data processing method based on adaptive table dimension division. Background technique [0002] With the advent of the information society, the scale of global data has grown rapidly in an explosive manner. The so-called "big data era" has arrived. With the massive data generated, on the one hand, traditional data processing methods have been unable to meet Rapid analysis and processing of large-scale data; on the other hand, the value of valuable data in this massive data needs to be tapped urgently. In this context, how to realize the rapid analysis and query of big data is a technical problem to be solved urgently in this field, which mainly includes the following two aspects: 1. There are various ways of data storage and management in the big data platform, how to design a system that can It is difficult to adapt to most business needs and store with a...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F17/30
CPCG06F16/215G06F16/2282G06F16/2456G06F16/248
Inventor 袁友伟陈魏欣黄彬彬俞东进鄢腊梅李黎
Owner HANGZHOU DIANZI UNIV