Distributed cluster resource scheduling method based on user operation process

A distributed cluster and resource scheduling technology, which is applied in the field of distributed cluster resource scheduling based on user job processes, can solve the problem of not fully considering the real-time resource status of distributed clusters, sub-job execution sequence relationship, and unable to maximize cluster performance, etc. problems, to achieve the effect of increasing parallelism and improving service quality

Pending Publication Date: 2021-07-23
ZHEJIANG UNIV OF TECH
View PDF0 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The commonly used resource scheduling algorithm for user jobs with internal processes is static, and does not fully consider the real-time resource status of the distributed cluster and the execution sequence relationship between sub-jobs, and cannot maximize the performance of the cluster

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed cluster resource scheduling method based on user operation process
  • Distributed cluster resource scheduling method based on user operation process
  • Distributed cluster resource scheduling method based on user operation process

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0049] Embodiment: the application scenario of this embodiment is as figure 1shown. The user submits an assignment application to the cloud service provider and gives a deadline for completing the assignment. In order to ensure the quality of service, the cloud service provider needs to predict the completion time of the job according to the resource situation of the current cluster system. When it is confirmed that the job can be completed within the time specified by the user, the cloud service provider uses the optimal resource configuration to execute the user job.

[0050] In this embodiment, the homework submitted by the user to the cloud service provider is a student's academic analysis and evaluation, which is required to be completed within 10 hours. This job can be decomposed into six sub-jobs, and the relationship between them is shown in Table 1, and there is a data dependency between each sub-job. Successor jobs cannot start executing until the predecessor job ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a distributed cluster resource scheduling method based on a user operation process. The method comprises the following steps: firstly, analyzing an execution sequence constraint relationship among sub-jobs included in the job, and determining a serial-parallel execution sequence of the sub-jobs; and according to the execution sequence of the sub-jobs, performing logic allocation of resources for the sub-jobs and predicting the execution time of each sub-job under the resource allocation, and further predicting the execution time of the job by calculating a key path in the job process. Usually, the job submitted by the user has the constraint of the completion time, so that the predicted job completion time can be used as a basis whether the cluster can provide services for the user in time or not. Experiments prove that compared with a default resource allocation algorithm of Spark, the algorithm provided by the invention can shorten the job execution time by 16.81%. According to the algorithm provided by the invention, the degree of parallelism of sub-job operation can be improved, the job execution time is shortened, and the improvement of the service quality of mechanisms such as a cloud service platform, a supercomputing center and a data center is facilitated.

Description

technical field [0001] The present invention relates to the field of distributed computing, in particular to a distributed cluster resource scheduling method based on user operation flow. Background technique [0002] Supercomputing centers and cloud service providers are paying more and more attention to improving their own quality of service (QoS, Quality of Service). Completing tasks submitted by users on time is the basic guarantee for high-quality services. Therefore, before providing services to users, it is necessary to predict the execution time of user jobs. If the predicted completion time of the job cannot meet the needs of the user, the supercomputing center, etc. should inform the user, so that the user can decide whether to still submit the job. [0003] In order to predict the running time of a job, a common method is to run a part of the job first, and then calculate the predicted value of the user's job execution time according to the proportion of the numb...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/48G06F9/50G06F16/901
CPCG06F9/4806G06F9/5005G06F16/9024
Inventor 胡亚红邱圆圆毛家发
Owner ZHEJIANG UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap