Data processing system and method based on distributed caching

Abstract

The invention relates to a data processing system based on distributed caching. The data processing system comprises a mapreduce data processing module, a map task memory processing module and a reduce distributed caching module, wherein the mapreduce data processing module is used for decomposing submitted user jobs into multiple map tasks and multiple reduce tasks, the map task memory processing module is used for processing the map tasks, and the reduce distributed caching module is used for processing the map tasks through the reduce tasks. The invention further relates to a data processing method based on distributed caching. The data processing system and method have the advantages of mainly serving for the map tasks, optimizing map task processing data, ensuring that the map can find target data within the shortest time and transmitting an intermediate processing result at the highest speed; data transmission quantity can be reduced, the data can be processed in a localized mode, the data hit rate is increased, and therefore the execution efficiency of data processing is promoted.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a distributed cache-based data processing system and processing method. Background technique [0002] Apache Hadoop (generally abbreviated as hadoop) is an open source distributed data processing platform. Its core mainly includes two parts, distributed file system (HDFS) and mapreduce computing model. [0003] HDFS shows great advantages in the storage of large-scale data, however, it has great shortcomings in dealing with real-time data reading. Since a large amount of data needs to be read during the execution of mapreduce tasks, it will cause huge pressure on network transmission and disk I / O (Input / Output) bandwidth, so it is necessary to set up a cache system on the basis of HDFS to reduce data transmission. Quantity to improve the execution efficiency of mapreduce. [0004] The mapreduce data calculation process can be divided into two stages: map and red...

AI Technical Summary

Problems solved by technology

However, memcache also has shortcomings. Distributed storage systems in the field of file backup rarely use memcache for distributed caching

[0007] These mechanisms are all oriented to the traditional data center platform architecture. However, the tightly coupled deployment of computing resources and storage resources on the map / reduce platform and the characteristics of data localization processing make the statistical results of data access characteristics based on data fast be affected by computing resource allocation strategies and The interference of real-time load makes it difficult to completely and truly reflect the characteristics of data access

[0008] Aiming at the need to read a large amount of data during the execution of the mapreduce task, the storage and transmission of the intermediate processing results put a huge pressure on network transmission and I / O bandwidth, and there is no good solution in the prior art

Images