A system to solve fierce competition for memory resources in big data processing systems

Abstract

The invention discloses a system for settling fierce competition of memory resources in a big data processing system. A memory information feedback module is used for monitoring the memory using condition for a running thread task, converting collected memory information and then feeding back the converted memory information to an information sampling and analyzing module; the information sampling and analyzing module is used for dynamically controlling the number of sampling times of information of all working nodes, analyzing data after the assigned number of sampling times is achieved, and calculating the optimal CPU to memory proportion of the current working node; a decision making and task distributing module is used for making a decision and controlling whether new tasks are distributed to the working nodes for calculation operation according to the information obtained through analysis and the task running information of the current working node to achieve effective limit on the CPU and memory use relation. By means of the system, a memory-perceptive task distribution mechanism can be achieved on a universal big data platform, the I / O expenditure generated when data overflows to a disc due to fierce competition of memory resources can be reduced, and the integral performance of the system can be effectively improved.

Description

technical field [0001] The invention belongs to the field of I / O performance optimization under the computer system structure, and more specifically relates to a system for solving fierce competition of memory resources in a big data processing system. Background technique [0002] With the advent of the big data era, cluster programming models MapReduce and Dryad are mostly used for data processing to process growing data sets. These models can provide automatic task scheduling mechanisms, fault tolerance mechanisms, and load balancing mechanisms, and the implementation details are transparent to users. Among them, the MapReduce model is widely used. [0003] The open source distributed processing system Hadoop, as the most typical representative of the MapReduce model, has been used by the industry to process a variety of offline batch processing applications. However, Hadoop has designed a complete set of execution processes based on disks. Intermediate data needs to be ...

AI Technical Summary

Problems solved by technology

[0006] Aiming at the defects of the prior art, the purpose of the present invention is to provide a system to solve the fierce competition of memory resources in the big data processing system, aiming to solve the problems of excessive competition of memory resources and unnecessary data overflow to Disk I / O operations, technical problems that cannot dynamically adjust the ratio of CPU and memory resources

Images