Configuration parameter determination method and apparatus of big data processing system
A technology for big data processing and configuration parameters, applied in the computer field, can solve the problems of big data processing system execution performance impact, inaccurate set of configuration parameter groups, etc., to achieve the effect of optimizing the set of configuration parameter values and improving operating efficiency
Inactive Publication Date: 2017-02-08
BEIHANG UNIV
View PDF6 Cites 12 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0007] The present invention provides a method and device for determining configuration parameters of a large data processing system, which are used to solve the problem that the execution performance of the large data processing system is affected due to the inaccurate set of configuration parameter groups determined by the existing configuration parameter optimization method
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.
[0032] The following first introduces the relevant knowledge of the big data processing system. It is worth noting that the big data processing system in the embodiments of the present invention is described by taking the big data processing system using the MapReduce programming model as an example.
[0033] With the rapid development of e-comme...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
Embodiments of the invention provide a configuration parameter determination method and apparatus of a big data processing system. The method comprises the steps of obtaining N job execution time corresponding to N jobs of the big data processing system by changing values of configuration parameters in a configuration parameter set, wherein each piece of the job execution time comprises a sum of execution time of all execution stages comprised in all tasks of each job; determining N actual execution time corresponding to the N jobs according to the execution time of the execution stages comprised in the tasks of each job and concurrent execution time of the execution stages; determining optimal execution time from the N actual execution time; and determining a configuration parameter set formed by configuration parameter values corresponding to the optimal execution time. According to the technical scheme, the parameter set of the big data processing system can be effectively and quickly optimized, so that the job running efficiency in the big data processing system is improved.
Description
technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for determining configuration parameters of a big data processing system. Background technique [0002] MapReduce is a programming model for parallel computing, which is used for parallel computing of large-scale data sets. It is currently one of the most popular and efficient big data processing frameworks. It provides a simple programming interface, and users can purposefully Big data applications that need to be processed implement these interfaces. Hadoop is one of the most commonly used open source implementations of MapReduce. Users can process various big data applications on the Hadoop platform, such as log analysis, index construction, and data mining. [0003] A MapReduce job is an execution instance of a MapReduce application on the Hadoop platform, and it consists of the following three parts: a user-defined MapReduce program, input data to be p...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/50
CPCG06F9/5066
Inventor 刘旭东孙海龙吕中厚唐宇
Owner BEIHANG UNIV