Resource scheduling method and device, electronic equipment and storage medium

A resource scheduling and resource technology, applied in the field of big data, can solve the problems of no preemption and release, the queue cannot enjoy redundant resources, and the work efficiency of the Hadoop cluster is reduced, so as to avoid preemption and release, improve the efficiency of allocation and recovery, and improve the The effect of utilization efficiency

Pending Publication Date: 2020-06-26
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF0 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this solution, when a certain queue consumes more resources, other queues will not be able to enjoy redundant resources, and when the cluster resources are tight, unnecessary preemption and release are carried out, which reduces the working efficiency of the Hadoop cluster.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Resource scheduling method and device, electronic equipment and storage medium
  • Resource scheduling method and device, electronic equipment and storage medium
  • Resource scheduling method and device, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and fully convey the concept of example embodiments to those skilled in the art.

[0029] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the invention. However, those skilled in the art will appreciate that the technical solutions of the present invention may be practiced without one or more of the specific details, or other methods, components, means, steps, etc. may be employed. In other instances, well-known methods, apparatus, implem...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a resource scheduling method and device based on a Hadoop cluster, electronic equipment and a storage medium, and relates to the technical field of big data. The resource scheduling method comprises the steps: receiving a resource preemption request of a resource preemption thread; in response to the resource preemption request, obtaining a target job needing to calculate resources; judging whether the whole cluster has enough residual computing resources or not and whether the computing resources can be preempted or not according to the target operation; when it is determined that the overall cluster is not sufficient for the remaining computing resources and computing resources may be preempted, excessively allocating the computing resources to the target job. According to the technical scheme of the embodiment of the invention, preemption in allocation is independent through the adjustment logic, the resource allocation efficiency of the cluster is improved, and the working efficiency of the system is improved.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a Hadoop cluster-based resource scheduling method, a resource scheduling device, electronic equipment, and a computer-readable storage medium. Background technique [0002] With the development of Internet technology, the field of Big Data technology has also developed rapidly. Big data refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain period of time. It is a massive, high-growth rate that requires a new processing model to have stronger decision-making power, insight and discovery, and process optimization capabilities. and diverse information assets. [0003] In the existing technical solution, the resource management and job scheduling of distributed big data is to use the scheduler to check whether resource preemption is needed each time a computing node heartbeats, and if it is determined ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
CPCG06F9/5022G06F9/5027
Inventor 吴怡燃
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap