High-performance data processing system based on memory calculation

Abstract

The invention discloses a high-performance data processing system based on memory calculation. The high-performance data processing system comprises a data collecting interface defining module, a data real-time collecting module, a data memory storage module and a data processing module. By means of the data collecting interface defining module, a data table is selected, related fields of the data table are automatically extracted, and a user defines a compression mode according to the type and range of the fields. The data real-time collecting module is used for extracting related data from a data source according to metadata defined by the data collecting interface defining module, wherein the metadata normally refer to data in a defined data format, and the data source refers to specific service data and normally a relation database of an enterprise service system. The data memory storage module is used for storing the data in a memory through a dictionary mode and a shaping, coding and compressing algorithm in a syllable code mode. The data processing module comprises an SQL analyzing module and is used for extracting records in statistics according to needs. According to the high-performance data processing system based on memory calculation, the data are stored in the memory in a byte mode through compression and reconstruction on the premise that the statistical precision and speed are not influenced.

Description

technical field [0001] The invention relates to a high-performance data processing system based on memory computing, which uses Java platform and JavaNIO technology to realize high-speed statistics of big data. Background technique [0002] Existing statistical technologies are generally implemented through relational database storage procedures or SQL statistical procedures, which are limited by network I / O and disk I / O bottlenecks, and problems such as slow response and excessive database pressure will occur when large data are generated. [0003] For larger data, Hadoopmap / reduce parallel computing technology is used. This technology requires huge hardware resources. Generally, 5 to 15 nodes or more have application value (depending on hardware performance), and the statistical task startup time is at least 10 seconds. Scenarios with high real-time performance requirements cannot be met. Contents of the invention [0004] The technical problem to be solved by the prese...

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved by the present invention is to provide a high-performance data processing system based on memory computing, to solve the problem of too slow big data efficiency in the process of data statistics, and to use memory computing technology to compress data into memory in a predefined way, Eliminate the I / O bottleneck in the statistical process, and finally realize the second-level statistical calculation of tens of millions to billions of data on ordinary single-node hardware

Images