Work distribution system and method based on hadoop multi-cluster environment
A work and cluster technology, applied in the field of work distribution system based on Hadoop multi-cluster environment
Inactive Publication Date: 2017-09-22
CHUNGHWA TELECOM CO LTD
View PDF4 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
It can be seen that there are still many deficiencies in the above-mentioned traditional methods.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0022] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments:
[0023] Such as figure 1 Shown, be the architecture schematic diagram of a kind of implementation example of the work assignment system based on Hadoop multi-cluster environment of the present invention, comprise:
[0024] Feature database module 11, in order to store the matrix equation of cluster feature module 12, cluster monitoring module 13, work data analysis module 14, work procedure analysis module 15;
[0025] The cluster feature module 12 is used to collect static features that do not change over time in the cluster, and describe the collected static features with the cluster static feature matrix equation;
[0026] The cluster monitoring module 13 is used to regularly collect the dynamic characteristics of each cluster, and analyze the d...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a system and a method for job assignment based on Hadoop multi-cluster environment. The system and the method are applied to a plurality of distributed computer clusters for mass data processing, and capable of realizing the selection of the optimal execution environment according to the characteristics of an executive program, the characteristics of data to be processed and the dynamic behaviors of the computer clusters; the system and the method have the advantages that the scheduling waiting time of jobs of different operation characteristics can be reduced, the speed of operational analysis can be effectively increased and the rare of overall resource utilization can be increased. The system comprises a cluster monitoring module, a cluster characteristic module, a job data analysis module, a job program analysis module and an execution environment selection module. The method comprises finding out the most appropriate clusters by virtue of operation and comparison by controlling the cluster characteristics, monitoring the operating conditions of the clusters and analyzing influence parameters such as operational data characteristics and program operation characteristics, finding the corresponding cluster by use of the execution environment selection module, and assigning user jobs, including user programs and input data, to the corresponding cluster for execution.
Description
technical field [0001] The invention relates to the technical field of computer clusters, in particular to a work distribution system and method based on a Hadoop multi-cluster environment. Background technique [0002] In recent years, due to a large amount of informatization, general enterprises and government agencies are faced with an explosive growth in the amount of data. No matter in the field of data storage, database, or data retrieval and data mining, they all encounter the same problem. Data filtering and The huge and time-consuming work of sorting can no longer be carried by a supercomputer, and it is directed to a large number of group computers to perform calculations at the same time, so as to obtain the maximum benefit. Today's information field uses cloud service technology to provide distributed computing to solve the above problems, and Apache Hadoop is one of the main open source solutions. [0003] Hadoop implements a distributed computing processing fr...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30G06Q10/06
CPCG06F9/5083G06F16/25
Inventor 林威廷黄俊翔林修民黄瀞莹蔡庆堂
Owner CHUNGHWA TELECOM CO LTD