Unlock instant, AI-driven research and patent intelligence for your innovation.

Multi-dimensional processing method and system for massive data

A technology for massive data and processing systems, applied in the field of big data processing, it can solve problems such as data delay and analysis system lag, and achieve the effect of real-time reflection of business changes

Inactive Publication Date: 2017-03-22
BEIJING GEO POLYMERIZATION TECH
View PDF10 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The obvious disadvantage of this approach is the data delay problem
Because the business data will not be extracted and processed until the evening of the day, the results of the analysis system will lag behind

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-dimensional processing method and system for massive data
  • Multi-dimensional processing method and system for massive data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0023] Such as figure 1 As shown, the multi-dimensional processing method of this massive data, the method includes the following steps:

[0024] (1) Real-time data reception: access to raw data through Kafka;

[0025] Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action streaming data in consumer-scale websites. Such actions (web browsing, searching and other user actions) are a key factor in many social functions on the modern web. These data are usually addressed by processing logs and log aggregation due to throughput requirements. This is a viable solution for systems like Hadoop that log data and analyze offline, but require real-time processing constraints. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time consumption through cluster machines.

[0026] (2) Real-time data processing: the original data is processed in real time thr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a multi-dimensional processing method for massive data, through which analysis results of the latest business data can be obtained within minutes, and business changes can be reflected in a real-time manner. The method comprises the following steps: (1) real-time data receiving: original data is accessed through Kafka; (2) real-time data processing: real-time processing of the original data is carried out through Storm, and after aggregated data is generated, an index of the aggregated data is saved in Redis, and the aggregated data is saved in Hbase; (3) data indexing: index data in Redis is read to provide an efficient retrieval function based on a memory; (4) historic data processing: positioned reading of Hbase data is carried out according to the index data in Redis; and (5) querying: the massive data is queried according to user requirements. The invention further provides a multi-dimensional processing system for the massive data.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a multi-dimensional processing method for massive data and a multi-dimensional processing system for massive data. Background technique [0002] For a long time, software systems have been divided into two categories: OLTP and OLAP. OLTP is called a transaction system, which mainly deals with business data, and focuses on how to efficiently increase or query business data. OLAP is called an analysis system, mainly for data analysis, focusing on the multi-dimensional analysis of massive data. [0003] With the development of the Internet, especially after the outbreak of the mobile Internet, more and more data is generated, and the data is generated faster and faster. How to efficiently analyze massive data has become an urgent need. [0004] Existing method: extract the original business data from the OLTP system through the regular ETL process, and store the result...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/2228G06F16/245G06F16/254G06F16/283
Inventor 范卫卫张翼温宗臣何良均严亮
Owner BEIJING GEO POLYMERIZATION TECH