Parallel processing system for small operation optimizing

Abstract

The invention relates to a parallel processing system for small operation optimizing. The parallel processing system sequentially comprises a resource management module, a resource scale module, a computing system module, a model translating module and an operation logic module from bottom to top. The resource management module realizes resource management of the whole system and provides a management platform. The resource scale module provides working nodes on the management platform for the computing system module. The computing system module is the environment where the system runs and is used for realizing direct data sharing among different threads in a JVMJava virtual machine. The model translating module is used for translating operation logics into workflows which can be recognized by the operation environment. The operation logic module is used for setting data transmission modes and logic expression models. One set of parallel processing system Binos with programs being extensible and resources being scalable and two sets of different MapReduce programming models which are respectively Binos-HashMR and Binos-FileMR are realized, and through the optimization of data storage, data transmission, logic processing and other relevant aspects, executing efficiency of small operation is obviously improved.

Description

technical field [0001] The invention relates to a parallel processing system oriented to small job optimization, which belongs to the field of distributed computers. Background technique [0002] In recent years, distributed data processing systems have become a solution for massive data processing. Aiming at different application scenarios, academia and industry have launched multiple parallel processing systems. After studying and analyzing different parallel processing systems, it is concluded that the programming model is composed of three parts: processing logic, data organization, and transmission mode. With the development of application logic, it is difficult for the existing parallel processing system to quickly realize the combination customization of data organization and transmission mode suitable for processing logic and application characteristics. [0003] According to the analysis of job characteristics in the actual production cluster, most jobs are small ...

AI Technical Summary

Problems solved by technology

Although Mesos and Yarn have proposed methods to support multiple programming models, in terms of supporting small jobs, resources are requested on demand, and the actual production cluster has a large number of small jobs and occupies few resources, making the execution of jobs difficult. Efficiency is affected by frequent application and release of resources

figure 1 For the impact of different input data volumes on execution efficiency under Yarn, through figure 1 It can be seen that when the amount of input data is small, the system still has a large time overhead due to the dynamic application and release of resources within the job.

Images