A real-time data analysis method and device

Abstract

Embodiments of the present invention provide a real-time data analysis method and device. The method includes that the application module sends a retrieval request to the distributed cache system, including data identification; if the distributed cache system finds the corresponding index and data in the index structure, it sends it to the application module, wherein the data includes real data and \ or pointer data; if the application module determines that the data includes pointer data, then send a query request to the Spark computing system including the pointer data; if the Spark computing system does not find the corresponding real data, then send a load command to the HDFS storage system that includes at least the pointer Data; the HDFS storage system sends the real data corresponding to the pointer data to the Spark computing system; the Spark computing system summarizes the real data to obtain summary data and sends it to the application module. The embodiment of the present invention enables the analysis system to process massive data while ensuring calculation efficiency and satisfying real-time online calculation.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of the Internet, and in particular to a real-time data analysis method and device. Background technique [0002] With the increasing popularity of smart phones and smart devices and the development of Internet technology, the background data analysis system has to face massive business data every day, and needs to analyze and calculate a large amount of data. [0003] Currently, the Hadoop off-line analysis system architecture is mainly used in the current technology. The data acquisition program directly saves the data to the Hadoop Distributed File System (HDFS); the big data calculation is completed through the MapReduce or Spark architecture; the analysis results are saved to RDBMS; the application module completes business processing by retrieving the RDBMS. Another main computing model is the Storm streaming big data computing model. There are two types of components in the c...

AI Technical Summary

Problems solved by technology

[0004] The former Hadoop big data analysis module is a high-volume, offline analysis and calculation tool that cannot complete online real-time analysis, and the processing system based on RDBMS cannot be too large in hot data volume

The latter Strom streaming big data calculation cannot complete single-point data analysis and calculation

Neither of them can meet the requirements of real-time analysis of large data volumes today.

Images