Resource scheduling method and device and computer equipment

Abstract

The invention relates to a resource scheduling method and device and computer equipment. The method comprises the following steps: aiming at a first task to be processed, constructing a plurality of computing units for a resource based on the applied resource; further processing each computing unit to at least comprise a static type and a dynamic type, then data is stored by utilizing resources ofthe static computing units according to the data storage requirements of the first task, and releasing at least part of the dynamic computing units and the resources of the dynamic computing units according to the resource release requirements of the first task when the first task is idle. Therefore, the computing unit of the first task is processed into a static type and a dynamic type; differentially and selectively utilizing the resources of the static computing unit to perform data storage, and utilizing the dynamic computing unit and the resources thereof to participate in resource release; under the condition of ensuring the integrity of cached (or persistent) data, the computing resources are dynamically managed, and the problem of contradiction between computing performance and resource occupation in computing platforms such as Spark can be effectively relieved.

Description

technical field [0001] The present application belongs to the technical field of distributed resource management, and in particular relates to a resource scheduling method, device and computer equipment. Background technique [0002] Spark is a fast and general-purpose computing engine designed for large-scale data processing. It can be used to complete various operations, including SQL (Structured Query Language, Structured Query Language) query, text processing, machine learning, etc. [0003] Spark provides an open source distributed cluster computing platform. One of the reasons why Spark is very fast is that it can cache or persist key RDDs (Resilient Distributed Datasets, Elasticity) in the memory of its computing unit (Spark Executor) in different operations Distributed data set), when an RDD is cached or persisted, the RDD can be reused in other subsequent actions on the RDD, which makes subsequent actions more rapid; that is, Spark uses The key RDDs are cached or p...

AI Technical Summary

Problems solved by technology

However, the dynamic resource management mechanism of dynamically applying for / releasing resources, in the Spark application environment, either does not release the idle computing units that have been cached with RDDs based on Spark’s computing performance, or releases the cached RDDs with a focus on saving resources. The computing unit is idle, so the cached RDD will also be lost with the release of its corresponding computing unit, which makes it impossible to guarantee subsequent computing performance

Therefore, for the Spark application environment, there is a problem of conflicting computing performance and resource occupation

Images