Fault-Tolerant Big Data Processing

Abstract

A method for improving the fault tolerance of MapReduce computing systems that process extremely large data sets uses custom mapper scripts that output results to shuffle and reduce processes. The reducers write their results to multiple files, which are then compressed. The resulting files are processed by a map-only reducer job that aggregates and reduces the results of the single file into a single output. Non-blocking compression prevents files from being split across multiple mappers. The job can recover if a single mapper fails without requiring each map node to be restarted, thus improving fault tolerance.

Description

Technical Field

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 068,641, filed Aug. 21, 2020. Such application is incorporated by reference herein as if fully set forth herein.

Background Art

[0002] A MapReduce system provides a framework for parallelized processing tasks to be performed on very large data (i.e., “big data”) sets on a computing cluster. The MapReduce framework allocates processing resources (processors or processor clusters in a cloud environment) to function as mappers and reducers that each execute a customizable map function and a reduce function, respectively. The mappers operate in parallel to process input data according to the map function, and the reducers operate in parallel to process the mapper output according to the reduce function to produce output data from the MapReduce program.

[0003] The map function performs filtering and sorting, thereby producing key-value pairs. The shuffle step redistributes the data based on the keys such that all data belonging to one key is placed on the same worker node in the distributed computing cluster for the reduce function. The reduce function is then an aggregation procedure applied to the key-value pairs, converting the set of those key-value pairs into a final set of key-value pairs. The MapReduce framework organizes the various tasks that operate in parallel and manages the data transfer between the various parts of the system. The MapReduce framework may be seen as inefficient in the sense that it needs to run multiple instances of the reduction process, but its highly parallelized structure enables operations on data sets that are simply too large to process on conventionally designed frameworks. In this way, this framework trades off increased parallelization of the computational efficiency on individual processors, thereby leading to improved overall throughput for many big data applications.

[0004] In a deliberately simplified example given for illustrative purposes, a MapReduce program can count the number of occurrences of each word that occurs in a set of documents. The map function assigns each unique word as a key and counts each occurrence of each word in a part (i.e., a subset) of one or more documents within the subset of the documents. The count then becomes the value in each key-value pair. The input data set is split into splits so that each mapper counts the occurrences of words within a subset of the document set, and the mappers operate in parallel in a distributed manner to count the occurrences of words across the entire data set. It may be understood that even terabyte or petabyte data can be processed in this way if there are sufficient processors or processor clusters to which the map function can be applied to each specific subset of the data set. The shuffle phase sends key-value pairs from the mappers to the reducers, and each reducer is responsible for a specific set of keys (i.e., a specific subset of words from the total set of unique words that occurred in the input document set). The reducers operate in parallel in a distributed manner to compute the total count of all words in the data set. The reduce function sums the values received from all of the mappers for a particular key and outputs that sum as the total count of occurrences of that particular word in the input document set. The output data from all of the reducers, organized by key, thus contains the total count of occurrences of each unique word in the input data set.

[0005] Conventional MapReduce frameworks include some minimal level of fault tolerance. For example, if a failure occurs in a map or reduce worker node, the work may be rescheduled as long as the input data is still available for operation. However, in a conventional MapReduce process, a failure in a single reduce node typically means that all map nodes need to be restarted and reshuffled in order to recover. Considering the huge costs of using very large cloud-based systems that typically process these big data sets, this type of recovery is very costly both in terms of computing resources and in terms of the financial cost to the parties using those resources. Restarting a processing job after such a failure can also be impractical simply due to time constraints when the data set grows to extremely large amounts of data. This is a result of the fact that certain data results are vulnerable to time, and if the results are not delivered within the required time frame, the value of the computational results can be reduced or even completely lost. These are the data sets that are most likely to experience faults during execution due to the very large number of nodes required to process them within the framework. Therefore, in order to avoid these problems, it is desirable to develop a system and method for processing extremely large data sets in a MapReduce framework with improved fault tolerance.

[0006] Any reference mentioned in this background section of the technology is not admitted to be prior art with respect to the present invention. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0007] The present invention is directed to a system and method for improving fault tolerance of a computing system that processes extremely large data sets (i.e., big data applications) within a MapReduce framework. In certain implementations, the system uses a custom mapper script that outputs results to a shuffle process and a reduce process. The reducer writes the results to a number of files (or what are known as fragments or parts) that are compressed using a non-block compression algorithm. The resulting files are then processed by a map-only reduce job that aggregates and reduces the results of the single file to a single output, which is the final output of the job.

[0008] In these implementations, non-block compression is used to prevent files from being split across multiple mappers. This enables the entire file to be sent to the same mapper in the next phase. By having the entire file to be processed by each mapper (performed by the reduce nodes), the shuffle step may be completely skipped since there is no need to shuffle the data before concatenating it to produce the final output. This saves time and, by extension, money in the use of the MapReduce framework for these operations. Additionally, the fact that the shuffle step is not performed means that in the event of a failure of a single mapper, the job can recover without each map node having to be restarted, thus significantly improving the overall fault tolerance of the system.

[0009] It should be noted that the systems and methods according to these certain implementations are incompatible with the conventional thinking in big data processing. A typical big data processing cluster (such as one that can be implemented using the Apache Hadoop® environment provided by the Apache Software Foundation) operates using the conventional MapReduce functionality as described in the background art above. By proceeding in a way that is actually in a sense exactly the opposite of the conventional and generally recommended method of data processing, the present invention provides higher stability and reliability (i.e., greater fault tolerance) in processing extremely large data sets.

[0010] These and other features, objects, and advantages of the present invention will be better understood from the following detailed description of the preferred embodiments in conjunction with the drawings and the appended claims.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Modes for Carrying Out the Invention

[0012] Before the present invention is described in more detail, it should be understood that the scope of the present invention is limited only by the claims, so the present invention is not limited to the specific embodiments described, and the terms used in describing the specific embodiments are for the purpose of describing those specific embodiments only and are not intended to be limiting.

[0013] The process according to the implementation of the present invention may be described with reference to FIG. 1. The process begins with the custom mapper script 10. This script is used to drive the first reduce function 12 (shown as the "pure reduce" 12 function in FIG. 1) in the MapReduce framework. The reducer writes the results to the non-block compressed fragment storage 14 in a number of files (which may also be referred to as file fragments or file parts). In this first phase, the reducer is simply the identity function, i.e., f(x)=x. Each of these files is then compressed using a non-block compression algorithm. This compression algorithm is a non-block compression to ensure that the files cannot be split across multiple mappers in the next phase, but rather that each of the files (or fragments or parts) resulting from the first mapper function and reducer function stay together in the next phase. This use of the entire output file for each of the first reduce functions takes into account the "simulation" of the reduce operation in the next phase.

[0014] In the next phase, the resulting file is processed by one or more map-only reduce jobs. In the example of FIG. 1, there are three such tasks, namely, map-only reducer A, map-only reducer B, and map-only reducer C, which are referenced by reference numerals 16, 18, and 20 respectively. Map-only reducer A (16), map-only reducer B (18), and map-only reducer C (20) all operate under the same map-only reducer job, but they are separated and partitioned logical tasks within the cluster, and each task takes on only one part / fragment file. For clarity, three jobs are shown in FIG. 1, but it will be understood that the present invention is equally applicable to any number of such jobs. This process aggregates and reduces the result of a single file into a single output with less lines of data. The map-only reduce process is split across a number of custom map-only reducers in a compute cluster environment, but the files are not split, and each file is directed to a specific single reducer node. The final result is then simply the aggregation of the outputs of each of the map-only reducer processes. As shown in FIG. 1, map-only reducer A maps to Out A, map-only reducer B maps to Out B, and map-only reducer C maps to Out C. In FIG. 1, Out A, Out B, and Out C are referenced by reference numerals 22, 24, and 26 respectively. Again, in this example there are three outputs, but it may be understood that the present invention is equally applicable to any number of outputs.

[0015] This process just described takes the logic of the reducer node and runs it instead as a map-only job in a way that resembles it being a map node in the MapReduce framework. As a result, no shuffle is required to concatenate the data at the end of the process. The outputs Out A22, Out B24, and Out C26 can simply be directly combined. This saves a significant amount of time and computational resources. Additionally, this enables the job to recover because if a single mapper operation fails, only that mapper operation has to be restarted. For example, in the instance of FIG. 1, if a failure occurs in map-only reducer B18, only that map-only reduce operation has to be restarted while the other outputs Out A22 and Out C26 remain valid. When map-only reducer B is restarted to recreate Out B24 after a failure and recovery, the outputs can be combined as previously mentioned.

[0016] Using the process described herein and shown in FIG. 1, the process can be aggregated and grouped by a large number of keys, so that extremely large data sets may be processed more efficiently. If a single node experiences a failure during the map-only reducer phase, the single node can recover and restart that part of the process without forcing all that preceded it to restart (i.e., without forcing each map-only reducer to restart). In a conventional MapReduce system, in order to recover on a single reducer node that has failed, all mappers would need to be restarted and shuffled. As the size of the data set becomes extremely large, the likelihood of failure becomes increasingly high. For this reason, the process described herein enables some extremely large data sets to operate in a conventional MapReduce framework cluster system where the likelihood of failure due to the total size of the process becomes so large that it may not even seem possible to complete after several attempts. Using conventional methods, the process may not complete within a computationally feasible time frame that is reasonable for the business because it is too prone to failure. The inventors have found that this approach enables the shuffle of 150 terabytes or more of data to be performed in a timely and consistent manner, something that was not possible in a conventional MapReduce environment.

[0017] In one particular example, consider the case when data is split across 15,000 files. There are 15,000 nodes being used in this way, and in a conventional process, a failure of any one of those nodes that produces non-corrupted data would force a restart, and all 14,999 of the other nodes would need to be re-executed. In the implementation of the invention described herein, only the one node where the failure occurred needs to be restarted, which means a reduction in cost and lost time of over 99%. The fault tolerance of the system in these very large use cases is thereby dramatically increased, and as the size of the data set increases, so does the return. The inventors have found that jobs performed on very large data sets (150 terabytes or more) on commercial MapReduce clusters often fail completely because there are too many retries after the failure of an individual node. In these cases, cost savings of up to 50 or 60% may be achieved by using the implementation of the invention described herein.

[0018] FIG. 2 provides a general architecture diagram of a MapReduce system in which the present invention may be implemented. This is the minimum hardware topology required for the implementation described herein. The custom mapper 10, pure reducer 12, and custom map-only reducer 16, 18, and 20 functions from FIG. 1 occur within the context of the Hadoop® cluster of FIG. 2. Apache Hadoop® is a software framework that is widely used for processing extremely large data sets and is designed particularly for distributed processing. Out A, B, and C (reference numerals 22, 24, and 26) of FIG. 1 persist within the context of the distributed storage 36 of FIG. 2. Generally speaking, a Hadoop® cluster minimally requires a master server and worker servers. In this case, the cluster includes within the framework of the big data processing compute cluster 34 both a master 30 and a plurality of workers 32. The present invention is not limited to implementation in the Hadoop® framework, and in alternative implementations of the present invention, other software packages may be used.

[0019] It will be apparent from the description provided herein that the customer mapper 10 function (initial map process) from FIG. 1 is scalable. The minimal architecture shown in FIG. 2 could theoretically be used to complete the process for a small data set. However, in the case of a large data set, the number of worker servers will increase proportionally, thereby also increasing the amount of hardware components required for those workers. In a specific example where 150 terabytes of data is being processed, the inventors used a master node with 8 CPU cores along with 52 GB of memory and 512 GB of disk storage space. For this same example, 512 workers may be required with 32 cores per worker, thus resulting in a total of 16384 CPU cores for the worker servers. The workers in this example could have 120 GB of memory per worker server and a total of 1200 GB of storage space is required. For example, there could be 2 x 375 GB solid state drive (SSD) disk storage and 450 GB of magnetic hard drive (HDD) storage per worker server. This is only one specific example and the amounts described herein may be scaled down or up as needed to accommodate a particular data set and particular data set size.

[0020] In one particular example using the Google Cloud Platform (trademark) (GCP) architecture from Google, the Apache Hadoop (registered trademark) cluster shown in FIG. 2 may be implemented on a Google Cloud Dataproc Hadoop (registered trademark) cluster. Google Cloud Dataproc is a managed Hadoop (registered trademark) service that enables cloud computing users to utilize a variety of data tools for batch processing, querying, streaming, and machine learning in a cluster computing environment. This service provides autoscaling to meet performance requirements and automatically powers down unnecessary resources. Further in this case, the distributed storage 36 may be implemented as Google Cloud Storage.

[0021] Using the problem of a simple word count example, three scripts are provided below for the implementation of the present invention, particularly for mappers, pure reducers, and map-only reducers.

Number

[0022] The implementations described above provide the specific hardware and software components used to implement the present invention in such implementations, but the present invention is not so limited. Hadoop® is used to process large amounts of data, but the present invention in alternative implementations may be adapted to other software frameworks. Google Cloud Platform is used to create Hadoop® clusters, but the present invention is likewise not so limited since any cluster of a sufficiently large size for the data set can be used.

[0023] The systems and methods described herein may be implemented in various embodiments by any combination of hardware and software. For example, in one embodiment, the systems and methods may be implemented by a computer system or a collection of computer systems, each of which includes one or more processors that execute program instructions stored on a computer-readable storage medium coupled to the processor. The program instructions can implement the functionality described herein. As illustrated in the drawings and described herein, various systems and displays represent exemplary implementations. Any order of methods may be changed, and various elements may be added, modified, or omitted.

[0024] The computing system or computing device described in this specification can implement the hardware part of a cloud computing system or a non-cloud computing system as forming parts of various implementations of the present invention. The computer system can include a commodity server, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a consumer device, an application server, a storage device, a telephone, a mobile phone, other mobile computing devices, or generally any type of computing node, computing node, computing device, and / or computing device, but is not limited thereto. The computing system includes one or more processors (any of which can include a number of processing cores that can be single-threaded or multi-threaded) coupled to system memory via an input / output (I / O) interface. The computer system can further include a network interface coupled to the I / O interface.

[0025] In various embodiments, the computer system may be a single-processor system that includes one processor or a multi-processor system that includes multiple processors. The processor may be any suitable processor capable of executing computing instructions. For example, in various embodiments, the processor may be a general-purpose or embedded processor that implements any of a variety of instruction set architectures. In a multi-processor system, each of the processors may, but typically does not necessarily, implement the same instruction set. The computer system may also include one or more network communication devices (e.g., network interfaces) for communicating with other systems and / or components over a communication network such as a local area network, a wide area network, or the Internet. For example, a client application running on a computing device may use a network interface to communicate with a server application running on a single server or a cluster of servers that implements one or more of the components of the system described herein in a cloud computing or non-cloud computing environment, as implemented in various subsystems. In another example, an instance of a server application running on a computer system may use a network interface to communicate with other instances of applications that may be implemented on other computer systems.

[0026] The computing device also includes one or more persistent storage devices and / or one or more I / O devices. In various embodiments, the persistent storage device may correspond to a disk drive, a tape drive, solid state memory, other mass storage devices, or any other persistent storage device. A computer system (or a distributed application or operating system operating thereon) can store instructions and / or data on a persistent storage device as desired, and can retrieve the stored instructions and / or data as needed. For example, in some embodiments, the computer system can implement one or more nodes of a control plane or control system, and the persistent storage can include an SSD attached to that server node. A number of computer systems may share the same persistent storage device, or may share a pool of persistent storage devices, where the devices in the pool may represent the same or different storage technologies.

[0027] A computer system includes one or more system memories that can store code / instructions and data accessible by a processor. The system memory can include multiple levels of memory and memory caches, for example, in a system designed to exchange information within the memory based on access speed. Interleaving and exchanging can be extended to persistent storage in a virtual memory implementation. Technologies used to implement the memory can include, by way of example, static random-access memory (RAM), dynamic RAM, read-only memory (ROM), non-volatile memory, or flash-type memory. Similar to persistent storage, multiple computer systems can share the same system memory or share a pool of system memory. The system memory or memory can accommodate program instructions executable by the processor to implement the routines described herein. In various embodiments, the program instructions can be encoded in binary, assembly language, any interpreter-type language such as Java®, a compiled language such as C / C++, or any combination thereof, and the specific languages listed herein are for illustration only. In some embodiments, the program instructions can implement multiple separate clients, server nodes, and / or other components.

[0028] In some implementations, the program instructions can include instructions executable to implement an operating system (not shown) that can be any of a variety of operating systems, such as UNIX (registered trademark), LINUX, Solaris (trademark), MacOS (trademark), or Microsoft Windows (registered trademark) (trademark), or a mobile computing device operating system such as iOS (trademark). Any or all of the program instructions may be provided as a computer program product or software that can include a non-transitory computer-readable storage medium storing the instructions, and the instructions are used to program a computer system (or other electronic device) to perform processes according to various implementations. The non-transitory computer-readable storage medium can include any mechanism for storing information in a machine-readable form (e.g., software, processing application) by a machine, such as a computer. Generally speaking, the non-transitory computer-accessible medium can include magnetic or optical media, such as computer-readable storage media or memory media like a disk or DVD / CD-ROM coupled to a computer system via an I / O interface. The non-transitory computer-readable storage medium can also include any volatile or non-volatile media such as RAM or ROM that may be included in some embodiments of a computer system as system memory or another type of memory. In other implementations, the program instructions may be communicated using optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) that can be transmitted via a network that can be implemented via a network interface and / or a communication medium such as a wired or wireless link. The network interface may be used to interface with other devices that can include other computer systems or any type of external electronic device.Generally, system memory, persistent storage, and / or remote storage accessible on other devices through a network can store data blocks, replicas of data blocks, metadata associated with data blocks and / or their states, database configuration information, and / or any other information usable in implementing the routines described herein.

[0029] In certain implementations, the I / O interface can coordinate I / O traffic among the processor, system memory, and any peripheral devices in the system, including via a network interface or other peripheral interface. In some embodiments, the I / O interface can perform any necessary protocol, timing, or other data transformation to convert a data signal from one component (e.g., system memory) into a format suitable for use by another component (e.g., the processor). In some embodiments, the I / O interface can include support for devices attached through various types of peripheral buses, such as, for example, a Peripheral Component Interconnect (PCI) bus standard or a variant of the Universal Serial Bus (USB) standard. Also, in some embodiments, some or all of the functionality of the I / O interface, such as an interface to system memory, may be directly incorporated within the processor.

[0030] The network interface can enable data to be exchanged between a computer system and other devices attached to the network, such as, for example, (one or more storage system server nodes, primary nodes, read-only nodes, and / or other computer systems that can implement clients of the database systems described herein) other computer systems. Additionally, the I / O interface can enable communication between the computer system and various I / O devices and / or remote storage. The input / output devices can include, in some embodiments, one or more display terminals, keyboards, keypads, touch pads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or retrieving data by one or more computer systems. These can be directly connected to a particular computer system or widely connected to a number of computer systems in a cloud computing environment, a grid computing environment, or other systems with a number of computer systems. A number of input / output devices can exist in communication with the computer system or be distributed across various nodes of a distributed system including the computer system. The user interface described herein can be visualized to the user using various types of display screens including CRT displays, LCD displays, LED displays, and other display technologies. In some implementations, the input can be received through the display using touch screen technology, and in other implementations, the input can be received through a keyboard, mouse, touch pad, or other input technology, or any combination of these technologies.

[0031] In some embodiments, similar input / output devices may be separate from the computer system and may interact with one or more nodes of a distributed system including the computer system through a wired or wireless connection via a network interface such as a network interface. The network interface can typically support one or more wireless networking protocols (e.g., Wi-Fi / IEEE802.11, or another wireless networking standard). The network interface can support communication via any suitable wired or wireless general data network, such as, for example, other types of Ethernet® networks. Additionally, the network interface can support communication via a telecommunications / telephony network, such as an analog voice network or a digital fiber communication network, via a storage area network such as a fiber channel SAN, or via any other suitable type of network and / or protocol.

[0032] Any of the distributed system embodiments described herein, or any of their components, may be implemented as one or more network-based services in a cloud computing environment. For example, the read / write nodes and / or read-only nodes within the database tier of a database system may present database services and / or other types of data storage services using the distributed storage system described herein to a client as network-based services. In some embodiments, the network-based services may be implemented by software and / or hardware systems designed to support machine-to-machine interactions that are interoperable over a network. A web service may have an interface described in a machine-processable format such as the Web Services Description Language (WSDL). Other systems may interact with the network-based services in a manner defined by the description of the interface of the network-based services. For example, the network-based services may define various operations that other systems can call, and may define a specific application programming interface (API) that other systems are expected to conform to when requesting the various operations.

[0033] In various embodiments, a network-based service may be requested or invoked through the use of a message that includes parameters and / or data associated with the network-based service request. Such a message may be formatted according to a particular markup language such as Extensible Markup Language (XML), and / or encapsulated using a protocol such as Simple Object Access Protocol (SOAP). To implement a network-based service request, a client of the network-based service assembles a message that includes the request and transmits the message to an addressable endpoint corresponding to the web service (e.g., a Uniform Resource Locator (URL)) using an Internet-based application layer transfer protocol such as the Hypertext Transfer Protocol (HTTP). In some embodiments, the network-based service may be implemented using Representational State Transfer (REST) techniques rather than message-based techniques. For example, a network-based service implemented according to REST techniques may be invoked through parameters included within HTTP methods such as PUT, GET, or DELETE.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, only a limited number of exemplary methods and materials are described herein. It will be apparent to those skilled in the art that many more modifications are possible without departing from the inventive concepts herein.

[0035] All terms used in this specification should be interpreted in the broadest possible manner consistent with the context. When groupings are used in this specification, all individual members of the group, as well as all possible combinations and sub - combinations of the group, are intended to be individually included in this disclosure. All references cited in this specification are incorporated herein by reference to the extent that there is no conflict with the disclosure of this specification. When ranges are used in this specification, all points within the range, as well as all sub - ranges within the range, are intended to be included in this disclosure.

[0036] Although the invention has been described with reference to certain preferred and alternative implementations, the implementations are intended to be illustrative only and are not intended to limit the full scope of the invention.

Claims

A method for processing large data sets in a distributed parallel computing environment, comprising: A plurality of mappers operating in parallel in a computing cluster, each mapper configured to perform a map function that filters and sorts a subset of an input data set to produce key-value pairs, and wherein, to produce an initial mapped data set, executing the map function in parallel for each subset of the input data set; Performing a shuffle phase on the initial mapped data set to produce a shuffled data set by redistributing the initial mapped data set based on the key-value pairs such that all data belonging to a particular key is placed on a common worker node in the distributed parallel computing environment; A plurality of reducers operating in parallel in the computing cluster, each reducer configured to apply an aggregation to key-value pairs of the shuffled data set to convert the key-value pairs to a final set of key-value pairs, and wherein, to produce a reduced data set comprising the final set of key-value pairs, executing the reduce function in parallel at each reducer on the shuffled data set; Compressing the reduced data set using a non-block compression algorithm to produce a plurality of compressed files; Assigning each of the plurality of compressed files to separate map-only tasks within a single map-only reducer to produce a plurality of partial results in parallel; Aggregating each of the plurality of partial results to produce a final result set; and In response to detecting a failure in one of the map-only tasks, restarting only the failed map-only task without restarting all map functions and reduce functions, thereby improving fault tolerance of the distributed computing environment. A method as described above. Claim 2 The method according to claim 1, wherein the reducer reads the shuffled data set from the shuffle phase and the mapper writes the initial mapped data set to the shuffle phase.

3. The method according to claim 1, wherein the step of executing the reduce function includes applying an identity function.

4. The method according to claim 1, wherein each of the compressed files is assigned to a unique map-only reducer throughout.

5. The method according to claim 1, excluding the step of performing a shuffle prior to the step of aggregating each of the partial results to produce the final result.

6. a. at least one computing cluster comprising at least one processor; b. a distributed parallel computing environment comprising at least one non-volatile memory coupled to the at least one processor, wherein when the at least one non-volatile memory is executed by the processor, for each of a plurality of mappers operating in parallel, each mapper configured to perform a map function comprising filtering and sorting a subset of input data to produce a key-value pair, and executing the map function in parallel for each subset of the input data set to produce an initial mapped data set; performing a shuffle phase on the initial mapped data set to produce a shuffled data set by re-distributing the initial mapped data set based on the key-value pair such that all data belonging to a particular key is placed on a common worker node in the distributed parallel computing environment. A plurality of reducers operating in parallel, each reducer being configured to apply an aggregation to a pair of key values of the shuffled data set to convert the pair of key values into a set of final key value pairs, the plurality of reducers generating a plurality of files in a plurality of map nodes comprising the set of final key value pairs, the step of parallelly executing a map function on the shuffled data set at each reducer, The step of compressing each of the plurality of files using a non-block compression algorithm to generate a plurality of compressed files, The step of assigning each of the plurality of compressed files to a separate map-only job from among a plurality of separate map-only jobs to produce partial results in parallel with each of the other separate map-only jobs, each of the plurality of separate map-only jobs being assigned to one of a plurality of reduce nodes, The step of aggregating each of the partial results to produce a final result, In response to detecting a failure in one of the map-only jobs, restarting only the map-only job in which the failure occurred without restarting all map functions and reduce functions, thereby improving the fault tolerance of the distributed computing environment, A computer-readable program code embodied in the non-volatile memory, causing the processor to perform operations including A distributed parallel computing environment. **Claim 7** The distributed parallel computing environment according to claim 6, wherein the computer-readable program code embodied in the non-volatile memory, when executed by the processor, further causes the processor to perform operations including the step of executing a reducer function by applying an identity function. **Claim 8** The distributed parallel computing environment according to claim 6, wherein the computer-readable program code embodied in the non-volatile memory, when executed by the processor, further causes the processor to perform operations including the step of assigning each of the compressed files in its entirety to a single reduce node. Claim 9 The distributed parallel computing environment according to claim 6, wherein when the computer-readable program code embodied in the non-volatile memory is executed by the processor, the processor is further caused to perform operations other than the step of performing shuffling prior to the step of aggregating each of the partial results in order to produce the final result.

Images