A dynamic energy management method for distributed computing clusters based on data coverage sets

Abstract

A method for managing dynamic energy consumption of distributed computing clusters based on data coverage sets, comprising the following steps: Step 1, preparation stage: divide a number of mutually different data coverage sets in the cluster; when dividing, all data coverage sets are required The union of should cover all nodes in the cluster; step 2, working stage: when starting the cluster, select a data coverage set in the list, and enable all the nodes in it, after the startup is complete, except the data coverage set, the working nodes The number of is dynamically adjusted according to the needs; step 3, adjustment stage: after the cluster has been running for a period of time, the data coverage set used should be replaced; the switching interval should be in days, combined with the shutdown and maintenance of the cluster, that is, select Replace the data coverage collection during regular maintenance. The present invention dynamically adjusts the number of working nodes, can reduce energy consumption and equipment loss under the condition of ensuring data availability, thereby greatly reducing operating costs.

Description

technical field [0001] The invention belongs to the field of energy management of data centers, in particular to a method for managing dynamic energy consumption of distributed computing clusters based on data coverage sets. Background technique [0002] Distributed computing frameworks such as Yahoo's Hadoop can efficiently process massive amounts of data. The implementation of Hadoop relies on a highly fault-tolerant distributed file system (HDFS). HDFS is designed to be deployed on a large number of cheap hardware, which distributes multiple copies of data blocks, provides users with fast data access and can continue to provide services through copies when some machines fail. [0003] HDFS adopts a master / slave node structure, and a typical architecture is as follows figure 1 shown. Each cluster contains a master node and multiple data nodes. The master node is responsible for managing the file system namespace and user file access; the data nodes storing data blocks a...

AI Technical Summary

Problems solved by technology

[0008] However, in the current technical practice, when the data coverage set is used, usually only one data coverage set is established in the cluster. At this time, the availability of the cluster has a great dependence on the reliability of the machines in the data coverage set.

At the same time, the continuous work of the data coverage set will also lead to a faster decline in machine life / performance

Images