A big data processing method based on adaptive table dimension division

A big data processing and self-adaptive technology, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of reducing the number of dimensions and slow analysis of high-dimensional data, so as to reduce the number of dimensions and reduce data processing. The effect of less storage space

Active Publication Date: 2018-07-20
HANGZHOU DIANZI UNIV
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, it is indeed necessary to provide a big data processing method based on adaptive table dimension division, which can be automatically divided according to the number of dimensions of the big data storage table, thereby effectively reducing the number of dimensions and solving the shortcomings of slow analysis of high-dimensional data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A big data processing method based on adaptive table dimension division
  • A big data processing method based on adaptive table dimension division
  • A big data processing method based on adaptive table dimension division

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0043] see figure 1 , is shown as a flow chart of a method for processing big data based on adaptive table dimension division in the present invention, including the following steps:

[0044] Step S1: Configure and access data sources according to user needs to generate multiple required data tables and import them into the data warehouse. Each data table has a unique tableName;

[0045] Step S2: performing a data cleaning operation on the generated data table;

[0046] Step S3: Perform data preprocessing operations on the data table and pre-store the preprocessed result information in the columnar database;

[0047] Step S4: Perform a data query operation according to the result information pre-stored in the database.

[0048] see figure 2 , is shown as a detailed flowchart of step S3 in a large data processing method based on adaptive table dimen...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a large data processing method based on adaptive table dimension division, which includes the following steps: S1: configuring and accessing data sources according to user requirements to generate multiple required data tables and importing them into a data warehouse; Perform data cleaning operations on the generated data tables; S3: perform data preprocessing operations on the data tables and pre-store the pre-processed result information in the columnar database; S4: perform data query operations based on the pre-stored result information in the database. By adopting the technical solution of the present invention, it can be automatically divided into multiple sub-dimensions according to the number of dimensions of the imported data table, thereby effectively reducing the number of dimensions, achieving rapid analysis of dimensions and occupying less storage space; at the same time, setting a set of dimension support degrees can According to the user's query operation, the support between dimensions is counted, and the closely related dimensions are automatically placed in a sub-dimension, thereby greatly reducing the amount of data processing.

Description

technical field [0001] The invention relates to the technical field of big data query analysis, in particular to a big data processing method based on adaptive table dimension division. Background technique [0002] With the advent of the information society, the scale of global data has grown rapidly in an explosive manner. The so-called "big data era" has arrived. With the massive data generated, on the one hand, traditional data processing methods have been unable to meet Rapid analysis and processing of large-scale data; on the other hand, the value of valuable data in this massive data needs to be tapped urgently. In this context, how to realize the rapid analysis and query of big data is a technical problem to be solved urgently in this field, which mainly includes the following two aspects: 1. There are various ways of data storage and management in the big data platform, how to design a system that can It is difficult to adapt to most business needs and store with a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/215G06F16/2282G06F16/2456G06F16/248
Inventor 袁友伟陈魏欣黄彬彬俞东进鄢腊梅李黎
Owner HANGZHOU DIANZI UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap