Data cluster computing node expansion and contraction method and device, equipment and medium

Abstract

The invention discloses a data cluster computing node expansion and contraction method and device, equipment and a medium. The method comprises the steps that multiple computing nodes and data nodes used for storing data are deployed in a cluster; obtaining the current resource utilization rate of the cluster, the number of tasks to be executed and a resource scheduling strategy; based on the resource utilization rate, the number of the to-be-executed tasks and the resource scheduling strategy, capacity expansion or capacity reduction is carried out on the computing nodes; wherein the expanded computing node and the data node are separately deployed. Thus, a plurality of computing nodes are deployed to guarantee normal use of the cluster, and on the basis of separate deployment of the expanded computing nodes and the data nodes, the computing nodes are dynamically expanded or reduced according to the current resource utilization rate of the cluster, the number of tasks to be executed and the resource scheduling strategy. According to the invention, resource demands in the peak period and the low peak period of the task are reasonably coped, resource waste can be avoided, manual operation is reduced, and cost is reduced.

Description

technical field [0001] The present application relates to the technical field of data clusters, and in particular, to a method, apparatus, device and medium for scaling computing nodes of a data cluster. Background technique [0002] The big data platform involved in the existing enterprise data business either uses the data cluster of the cloud manufacturer, or builds the data cluster in its own local computer room, and uses the data cluster of the cloud manufacturer. Taking the Hadoop data cluster as an example, all Hadoop ecosystems And related components are heavily dependent on the version of the cloud manufacturer, the cost is high, the version is fixed, the response time to problems is slow, and it is difficult to carry out customized secondary development and data migration. However, if you use a self-built computer room to deploy a data cluster, you need to evaluate the resources to deploy the data cluster in advance according to the peak period of the task, but mos...

AI Technical Summary

Problems solved by technology

[0002] The big data platforms involved in the existing enterprise data business either use the data clusters of cloud vendors, or build data clusters in their own local computer rooms, and use the data clusters of cloud vendors. Taking Hadoop data clusters as an example, all Hadoop ecosystems And related components are heavily dependent on the version of the cloud vendor, the cost is high, the version is fixed, the response speed to the problem is slow, and it is difficult to carry out customized secondary development and data migration

If you use a self-built computer room to deploy data clusters, you need to evaluate the resource deployment data clusters in advance according to the peak period of the task, but most of the machines are idle during the day, resulting in a huge waste of cluster resources, and you need to manually expand nodes in the later stage. increase, artificial expansion will be more frequent and maintenance time will be longer

Images