Check patentability & draft patents in minutes with Patsnap Eureka AI!

A two-stage job scheduling method and system for big data platforms

A technology of big data platform and phase operation, applied in the field of big data computing, can solve problems such as lack of solutions, limited resources of the platform, and inability to satisfy all users

Active Publication Date: 2020-04-24
SHANDONG UNIV
View PDF12 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Ideally, the resources of the platform are sufficient to meet the needs of all users. At this time, the service provider’s revenue is also the largest; but in reality, the resources of the platform are limited, and it is very likely that they cannot meet the needs of all users. For the platform Service providers will face the following problems:
[0008] To sum up, how to simultaneously solve the problem of maximizing platform revenue and maximizing platform resource utilization in the big data platform still lacks an effective solution

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A two-stage job scheduling method and system for big data platforms
  • A two-stage job scheduling method and system for big data platforms
  • A two-stage job scheduling method and system for big data platforms

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0072] As introduced in the background technology, the problems of maximizing platform revenue and maximizing platform resource utilization cannot be effectively solved in the prior art, and a two-stage job scheduling method for big data platforms is provided. The scheduling method of the present invention comprehensively considers constraints such as job deadline constraints, maximum platform revenue, and maximum resource utilization, and adopts a two-stage job scheduling method based on revenue and resources. This scheduling method can not only meet the needs of users of the big data platform Job deadline requirements can also ensure the maximum utilization of platform resources while maximizing platform revenue.

[0073] In order to achieve the above object, the present invention adopts the following technical scheme:

[0074] Such as figure 2 as shown,

[0075] A two-stage job scheduling method for big data platforms, the method includes the following steps:

[0076] (...

Embodiment 2

[0168] In the big data platform, in order to make full use of the resources of the platform and improve the revenue of the service provider, the present invention provides a two-stage job scheduling system oriented to the big data platform. The scheduling system is based on a big data platform-oriented Two-stage job scheduling method.

[0169] In order to achieve the above object, the present invention adopts the following technical scheme:

[0170] Such as Figure 5 as shown,

[0171] A two-stage job scheduling system for big data platforms, the system includes:

[0172]The first stage scheduling module, the first stage scheduling module is used to form the job submitted by the user into a job set to be scheduled, each job in the job set has its own SLA information, based on the latest start time constraint of each job and The overall maximum revenue of the service provider, using the maximum revenue scheduling based on the latest start time of the job to adjust and schedu...

Embodiment 3

[0177] In this embodiment 3, from the influence of various factors such as the average job size, job set size, total number of platform resources, and job urgency on the algorithm, the two-stage job oriented to the big data platform in the invention embodiment 1 and embodiment 2 The scheduling method and system are compared with the EDF algorithm and the original FIFO scheduling algorithm in Hadoop.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention relates to a two-stage job scheduling method and system for a big data platform. Jobs submitted by users are formed into a set of jobs to be scheduled, and the maximum benefit scheduling based on the latest start time of the job: the resources of the platform are scheduled according to the deadline of the job. Pre-allocation, and adjust and schedule the results of resource pre-allocation according to the income ratio of the job, and obtain the resource pre-allocation job scheduling result queue that maximizes the service provider's income; job scheduling based on the platform's maximum resource utilization: according to the platform's resources According to the usage situation, fine-tune the above resource pre-allocation job scheduling result queue to obtain the final scheduling result queue, so as to maximize the resource utilization rate of the platform under the premise of ensuring the maximum profit of the platform. Experimental results show that the present invention not only realizes the maximization of platform revenue, but also improves the resource utilization rate of the platform and improves the comprehensive performance of the platform.

Description

technical field [0001] The invention belongs to the technical field of big data computing, and in particular relates to a two-stage job scheduling method and system for a big data platform. Background technique [0002] In recent years, with the vigorous development of cloud computing and Internet technology, data has shown an explosive and continuous growth model, and the era of big data has quietly arrived. Traditional data processing technologies and tools can no longer meet the data processing requirements of the new era, so big data platforms emerged as the times require. The big data platform supports multiple computing frameworks and can provide services for multiple users at the same time. However, in the big data platform, multiple users share platform resources. For platform providers, how to efficiently schedule multi-user jobs can not only make full use of platform resources, but also meet the SLA requirements of most users, so that The biggest benefit for ones...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48G06F9/50G06Q10/06
Inventor 史玉良胡静李庆忠张世栋
Owner SHANDONG UNIV
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com