Resource scheduling method and identification system based on key resources and data preprocessing
A data preprocessing and resource scheduling technology, applied in the computer field, can solve the problems of high failure rate of cluster jobs, low node execution efficiency, low resource utilization rate, etc., and achieve the effect of shortening waiting time, reducing failure rate, and improving utilization rate
Active Publication Date: 2019-05-28
厦门科易网科技有限公司
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0008] In view of this, the purpose of the embodiments of the present invention is to provide a resource scheduling method and identification system based on key resources and data preprocessing, aiming to solve the problem of high cluster job failure rate, low node execution efficiency and resource constraints in the prior art. The problem of low usage
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0054] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.
[0055] The specific embodiment of the present invention provides a resource scheduling method based on key resources and data preprocessing, which mainly includes the following steps:
[0056] S11. Generate a node queue during the cluster execution process;
[0057] S12. Select a target node and a preparatory job task according to the generation node queue;
[0058] S13. Find the target node through a scheduling algorithm, read corresponding task information in the preparatory job task, and schedule jobs in the preparatory job task to execute according to the task information.
[0059] The resour...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a resource scheduling method based on key resources and data preprocessing. The method comprises the steps of: generating a node queue in the operation execution process of clusters; selecting a target node and a preparation operation task according to the generated node queue; searching the target node through a scheduling algorithm, reading corresponding task information in the preparation operation task, and scheduling an operation in the preparation operation task according to the task information so as to execute the operation. The invention furthermore discloses a resource scheduling system based on key resources and data preprocessing. According to the method and system provided by the invention, the waiting time of submitted operations in the clusters can be shortened, the operation failure rate is reduced, the node execution efficiency is improved and the resource usage rate is improved.
Description
technical field [0001] The invention relates to the field of computers, in particular to a resource scheduling method and identification system based on key resources and data preprocessing. Background technique [0002] The job scheduling algorithm is responsible for resource allocation within the cluster and related operations of data localization. The job scheduling algorithm realizes the efficient allocation of system resources, reduces the cluster load pressure, and has a great impact on the cluster. Since the industry first proposed the first-come-first-out scheduling algorithm in the Hadoop cluster, scholars have proposed various The job scheduling algorithm of a sample Hadoop cluster, the most important of which are as follows: [0003] (1) Fair scheduling algorithm: The fair scheduling algorithm is suitable for multiple users to use the cluster at the same time, ensuring that jobs can allocate the resources they need for jobs relatively fairly. The process of ensu...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48
CPCG06F9/4875G06F9/4881
Inventor 刘刚李旭炜毛睿
Owner 厦门科易网科技有限公司