Resource allocation method and device, and electronic equipment

Abstract

The invention discloses a resource allocation method and device, and electronic equipment, and relates to the field of resource allocation. The method comprises the steps of obtaining the resource quantity of a target service in a current period according to operation data of the target service in a historical period and operation data of an upstream service in the current period, and the target service and the upstream service have a data dependence relationship; operating the target service by using the resource quantity in the current period, and monitoring the change of the operation dataof the target service in each actuator; and adjusting the resource quantity of the target service and/or the data distribution of the target service in each actuator in real time according to the change of the operation data of the target service in each actuator. And the resource utilization rate of the cluster is improved, so that the service operation performance is relatively stable and configurable. Even if the underlying environment changes, the service operation performance can be kept relatively stable by automatically increasing or decreasing the resource quantity. Meanwhile, the service operation performance can be configured by adjusting configuration parameters of the service operation performance.

Description

technical field [0001] This application relates to the technical field of big data business, especially to the field of resource allocation. Background technique [0002] With the continuous development and application of Internet technology, massive amounts of data and various data computing engines have been generated. The data calculation engine is a component used to perform calculations on data. In order to extract valuable data from massive data, it is necessary to develop a large number of data ETL (Extract-Transform-Load) services, and run these ETL services online according to a certain period. Among them, ETL is used to describe the process of extracting, transforming, and loading data from a data source to a target data source. [0003] As a popular big data solution, Spark has many excellent features. Include data calculation engine in Spark. The resource allocation scheme generally adopted by the existing Spark business, the resource amount used to run the b...

AI Technical Summary

Problems solved by technology

However, if you define a relatively loose resource pre-allocation scheme and use this resource pre-allocation scheme for all ETL services, then for ETL services with small business volumes, there will be excess resources, resulting in low utilization of cluster resources

For the ETL business with a large business volume, there will be insufficient resources, resulting in low performance and long time-consuming business operation

[0004] Therefore, none of the existing resource allocation schemes can allocate an appropriate amount of resources to run the ETL business

Images