MapReduce bandwidth optimization method

Abstract

The invention discloses a MapReduce bandwidth optimization method which comprises the following steps: after Hadoop work is submitted, an OpenFlow controller, according to task execution node information sent by JobTracker and through a Map intermediate value routing strategy, determines nodes for carrying out a Map intermediate value merge task, updates corresponding flow entry and sends the flow entry to an OpenFlow switch; and the OpenFlow switch receives and installs the flow entry through a secure channel, then, carries out flow entry matching on the received data packets, and if behavior type for merging the Map intermediate values is matched therewith, carries out merging of the Map intermediate values. Through the combination of the OpenFlow and Hadoop, and by utilizing data handling capacity of the OpenFlow switch, the Map intermediate values, that is, the intermediate tuple data obtained after the action of a map() function, are subjected to merge processing in advance, thereby greatly relieving the problem of network congestion in the data migration process and improving Hadoop work efficiency substantially.

Description

technical field [0001] The design of the present invention belongs to the Hadoop cloud computing field, and more specifically relates to a MapReduce bandwidth optimization method. Background technique [0002] MapReduce distributed computing requires a large number of one-to-many or many-to-many communications between servers. This makes the data center network often congested in the current technical environment, resulting in increased packet loss, increased transmission delay, and decreased throughput. Especially in the MapReduce process of Hadoop cloud computing, when the main server assigns the Map task and the Reduce task, the Map server starts to perform calculation work and migrates the calculated intermediate value to the Reduce server. During the migration process of a large amount of data, it is very difficult to It is easy to cause network congestion, making the Reduce server wait for a long time or even fail. The time it takes for data to travel across the netw...

AI Technical Summary

Problems solved by technology

In response to this problem, researchers have proposed some optimization schemes, such as MapReduce column storage optimization, MapReduce connection optimization, MapReduce scheduling optimization, etc., but most of the above schemes have the following problems: task scheduling and resource allocation are not separated, scheduling deployment is not automatic enough, and cannot be quickly implemented. Ok, optimize scheduling according to network conditions, etc.

However, when a large number of data streams emerge, the congestion avoidance path is also unable to change the congestion situation.

Images