Transparent efficiency for in-memory execution of map reduce job sequences

Abstract

Executing a map reduce sequence may comprise executing all jobs in the sequence by a collection of a plurality of processes with each process running zero or more mappers, combiners, partitioners and reducers for each job, and transparently sharing heap state between the jobs to improve metrics associated with the job. Processes may communicate among themselves to coordinate completion of map, shuffle and reduce phases, and completion of said all jobs in the sequence.

Description

FIELD[0001]The present application relates generally to computers, and computer applications, and more particularly to MapReduce job sequences in distributed processing.BACKGROUND[0002]Increasingly inter-connected, global computing systems are generating an enormous amount of irregular, unstructured data. Mining such data for actionable business intelligence can give an enterprise a significant competitive advantage. High-productivity programming models that enable programmers to write small pieces of sequential code to analyze massive amounts of data are particularly valuable in mining this data.[0003]Over the last several years, MapReduce has emerged as an important programming model in this space. In this model, the programming problem is broken up into specifying mappers (map operation) and reducers (reduce operation). A mapper takes a small chunk of data (typically in the form of pairs of (key,value)), and produces zero or more additional key value pairs. Multiple mappers are e...

AI Technical Summary

Benefits of technology

This patent describes methods and systems for executing a map reduce sequence, which is a type of processing used in big data applications. The methods involve using a collection of multiple processes with each process running various tasks like mappers, combiners, partitioners, and reducers for every job in the sequence. The processes communicate with each other to coordinate the execution of the entire sequence. The result is improved performance and metrics associated with the job. The system includes a map reduce module that can execute all jobs in the sequence and coordinate the map, shuffle, and reduce phases. The computer readable storage medium includes the program instructions for performing the methods. Overall, this patent provides technical means for efficient and effective big data processing.

Problems solved by technology

Increasingly inter-connected, global computing systems are generating an enormous amount of irregular, unstructured data.

Within limits, of course, if there are a large number of failures, the job controller may give up.

This incurs I / O cost as well as (de-) serialization cost.

Mappers and reducers for each job are started in new JVMs (JVMs typically have high startup cost).

Solid black lines represent expensive out of memory (disk or network) operations.

Serializing and then deserializing this data wastes central processing unit (CPU) cycles.

It is not possible to combine output across JVM instances.

Images