Hadoop heterogeneity method and system based on storage and acceleration optimization

Abstract

The invention discloses a Hadoop heterogeneity method and system based on storage and acceleration optimization, and belongs to the field of distributed calculation. According to the technical scheme, aiming at data processing requirements, a storage medium is divided into three types, including a solid storage medium, a common storage medium and a high-density storage medium, and the most appropriate storage mode is found for data of different types; meanwhile, an acceleration application of which calculation performance needs to be improved is positioned to an FPGA accelerator with a specific algorithm or a GPU accelerator to complete calculation so as to improve processing performance of the application, and algorithm functions and layouts of the FPGA and GPU accelerator can be statically switched. The invention further discloses the Hadoop heterogeneity system based on the storage and acceleration optimization. According to the Hadoop heterogeneity method and system based on the storage and acceleration optimization, reading and writing performance of a whole cluster and execution performance of an application task and the resource utilization rate of an acceleration device are improved.

Description

technical field [0001] The invention belongs to the field of distributed computing, and more specifically relates to a Hadoop heterogeneous system based on storage and acceleration optimization. Background technique [0002] Data mining and machine learning are attracting more and more attention in the industry, and the MapReduce (a distributed programming model) framework for big data processing applications is an extremely easy task due to the characteristics of its own Map (mapping) and Reduce (reduction) calculation stages. Parallel programming model. Due to the simplification of the Map and Reduce interfaces provided by the MapReduce framework to developers, many problems such as parallelism, scalability, and portability can be solved. Its open source implementation is Hadoop (a distributed system infrastructure). Single point of failure problem, decoupling and upgrading functions to YARN (another resource coordinator). [0003] With the limitation of the CPU process ...

AI Technical Summary

Problems solved by technology

[0004] At present, many companies including Microsoft and Intel are doing the practice of integrating FPGA accelerators in large-scale data centers, but large-scale deployment of accelerators will bring many adverse effects

First of all, the cost of FPGA refactoring programming and optimization will be magnified during large-scale deployment. Since the customization of FPGA functions needs to be manually described in hardware language and finally compiled on the board, it will bring a lot of extra burden to developers; secondly , due to the high cost of FPGA itself, it is necessary to make a choice when weighing the acceleration of big data analysis applications and the cost of FPGA clusters. It is necessary to minimize the use of accelerators and improve the computing performance of applications as much as possible; then, due to the existing In the design scheme of the MapReduce programming model, the resource utilization rate of ReduceFPGA (protocol accelerator) is extremely low, and large-scale deployment of FPGA may cause a great waste of resources of ReduceFPGA itself

Images