Distributed operation cluster dynamic energy consumption management method based on data coverage set

Abstract

The invention discloses a distributed operation cluster dynamic energy consumption management method based on a data coverage set. The method comprises the following steps: step 1, a preparation stage: dividing a plurality of different numbers of data coverage sets in a cluster, wherein a union set of all data coverage sets is required to cover all nodes in the cluster during division; step 2, a working stage: when the cluster is started, selecting one data coverage set in the list, starting all nodes in the data coverage set, and after starting is completed, dynamically adjusting the number of the working nodes according to needs except the data coverage set; step 3, an adjustment stage: after the cluster runs for a period of time, replacing the used data coverage set, wherein the switching interval takes days as a unit and is combined with the shutdown maintenance of the cluster, i.e., the data coverage set is selected to be replaced during the periodic maintenance of the cluster. According to the method, the number of the working nodes is dynamically adjusted, and the energy consumption and the equipment loss can be reduced under the condition of ensuring the data availability,so that the operation cost is greatly reduced.

Description

technical field [0001] The invention belongs to the field of energy management of data centers, in particular to a method for managing dynamic energy consumption of distributed computing clusters based on data coverage sets. Background technique [0002] Distributed computing frameworks such as Yahoo's Hadoop can efficiently process massive amounts of data. The implementation of Hadoop relies on a highly fault-tolerant distributed file system (HDFS). HDFS is designed to be deployed on a large number of cheap hardware, which distributes multiple copies of data blocks, provides users with fast data access and can continue to provide services through copies when some machines fail. [0003] HDFS adopts a master / slave node structure, and a typical architecture is as follows figure 1 shown. Each cluster contains a master node and multiple data nodes. The master node is responsible for managing the file system namespace and user file access; the data nodes storing data blocks a...

AI Technical Summary

Problems solved by technology

[0008] However, in the current technical practice, when the data coverage set is used, usually only one data coverage set is established in the cluster. At this time, the availability of the cluster has a great dependence on the reliability of the machines in the data coverage set.

At the same time, the continuous work of the data coverage set will also lead to a faster decline in machine life / performance

Images