A Method for Optimizing Configuration Parameters of a Big Data Framework Based on Prior Knowledge

Abstract

The present invention discloses a method for optimizing configuration parameters of a big data framework based on prior knowledge. The method includes: selecting multiple types of tasks from the task distributions adapted by the big data framework, executing them under different parameter configurations, and obtaining the correlation relationship between the running times of different types of tasks and the configuration parameters; selecting the configuration parameters related to resources according to the correlation relationship, constructing a prior configuration set, initializing the prior configuration set, and obtaining the minimum running time by changing the values of the parameters in the prior configuration set and changing the computing power of the big data framework; determining the expected time for task running based on the minimum running time as a constraint for controlling the termination of subsequent task running, and dynamically updating the expected time according to the task running results; among all the tasks that have run successfully, finding the configuration parameters corresponding to the task with the shortest running time as the optimal configuration. The present invention improves the performance of big data processing through small sample optimization.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and more specifically, to a method for optimizing configuration parameters of a big data framework based on prior knowledge. Background Art

[0002] In recent years, with the popularization of big data applications, general parallel frameworks for big data have received increasing attention. For example, the Hadoop framework uses the MapReduce model, and through a distributed system architecture, it can use the resources of different nodes for large-scale parallel computing. The Spark framework further expands the MapReduce model. Compared with Hadoop, it stores a lot of data in memory for computing, reducing the very time-consuming input and output operations, so it significantly improves the speed of big data processing and has been widely used.

[0003] Taking Spark as an example, the Spark parallel computing framework needs to allocate tasks to computing nodes for operation. Different tasks need to cooperate with different resources to complete. For example, some tasks for rapid data processing require a large amount of processor resources; while some tasks generate more temporary data during operation and require a large amount of memory. Therefore, different configuration parameters need to be used for different tasks to run.

[0004] Existing Spark configuration parameter optimization methods generate different configuration parameters and use these parameters to obtain operation results, and then use machine learning methods to train using the operation results, and then find the optimal set of configuration parameters among them. Specifically, existing Spark configuration parameter optimization methods need to generate configuration parameters multiple times for each new program, and then use these configuration parameters to run tasks, so as to obtain the task execution time under different configuration parameters. Further, through machine learning algorithms, the obtained task time and the corresponding configuration parameters are used as training samples, and through large-scale sample training, the relationship between configuration parameters and task execution time is obtained, so as to find the optimal set of configurations. Existing machine learning methods require a large number of training samples, and when running Spark for big data processing, the time required to complete a task is sometimes very long (up to several hours or even longer), so the process of collecting training samples is very time-consuming. Summary of the Invention

[0005] The object of the present invention is to overcome the defects of the above-mentioned existing technologies, and provide a method for optimizing configuration parameters of a big data framework based on prior knowledge, which is a new technical solution for small-sample optimization of configuration parameters for a big data processing framework.

[0006] The technical solution of the present invention is to provide a method for optimizing configuration parameters of a big data framework based on prior knowledge, and the method includes the following steps:

[0007] Select multiple types of tasks from the task distribution adapted to the big data framework, execute them under different parameter configurations, and obtain the correlation relationship between the running times of different types of tasks and the configuration parameters.

[0008] Select configuration parameters related to resources according to the correlation relationship, construct them into a prior configuration set, initialize the prior configuration set, and obtain the minimum running time by changing the values of the parameters in the prior configuration set and changing the computing power of the big data framework.

[0009] Determine the expected time for task running based on the minimum running time, use it as a constraint to control the termination of subsequent task running, and dynamically update the expected time according to the task running results.

[0010] Among all the tasks that have run successfully, find the configuration parameters corresponding to the task with the shortest running time as the optimal configuration.

[0011] Compared with the prior art, the advantages of the present invention are as follows: for typical big data processing tasks, by running multiple times, the relatively critical parameters are found and used as prior knowledge. Then, when processing new tasks, the prior knowledge is used to initialize the parameters and run the tasks. According to the running time of each task, a new expected time is obtained and used as the maximum allowed running time for subsequent runs. The present invention can significantly shorten the initialization time by using prior knowledge, and during the running process, using the expected time can avoid tasks with too long running times.

[0012] Through the following detailed description of the exemplary embodiments of the present invention with reference to the accompanying drawings, other features and advantages of the present invention will become clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings incorporated in and constituting a part of this specification illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

[0014] Figure 1 is a flowchart of a method for optimizing configuration parameters of a big data framework based on prior knowledge according to an embodiment of the present invention;

[0015] Figure 2 is a schematic diagram of the process of a method for optimizing configuration parameters of a big data framework based on prior knowledge according to an embodiment of the present invention;

[0016] Figure 3 is a schematic diagram of the time spent searching for the optimal configuration according to an embodiment of the present invention;

[0017] Figure 4 is a schematic diagram of the task execution time obtained using the optimal configuration according to an embodiment of the present invention;

[0018] Figure 5 It is to set a for modifying the expected time according to an embodiment of the present invention, and compare the ratio of the search time to the optimal performance when a takes different values. Detailed implementation manners

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present invention.

[0020] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way a limitation on the present invention or its application or use.

[0021] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and devices should be regarded as part of the specification.

[0022] In all the examples shown and discussed herein, any specific values should be construed as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

[0023] It should be noted that: similar reference numerals and letters denote similar items in the following drawings, and thus, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0024] The big data processing framework to which the present invention is applied includes but is not limited to Hadoop, Flink, Spark, etc. For clarity, the Spark platform will be taken as an example for illustration below.

[0025] Combined with Figure 1 and Figure 2 As shown, the method for optimizing configuration parameters of a big data framework based on prior knowledge provided by this embodiment includes the following steps:

[0026] Step S110, for multiple typical types of tasks, run them multiple times with different configuration parameters to obtain the correlation relationship between the running times of different types of tasks and the configuration parameters.

[0027] In this step, several different typical Spark programs are selected from the task distributions adapted by Spark for analysis, and the analysis results are used as prior knowledge.

[0028] Existing Spark programs can handle a variety of different tasks, such as conventional data processing tasks, database-related tasks, artificial intelligence algorithm tasks, etc. For example, using various types of tasks such as sorting tasks, database scanning tasks, and Bayesian classification, running multiple times under different parameter configurations, and analyzing the running time t of the i-th run i and the corresponding configuration parameter <c 1,i , C 2,i , …>, where c m,i represents the m-th parameter during the i-th run.

[0029] In one embodiment, for each type of task, the correlation relationship between the task running time and the configuration parameters includes a linear correlation degree, a non-linear correlation degree, or a combination of both.

[0030] For example, when conducting analysis, the correlation relationship between the running time and the configuration parameters is expressed as:

[0031] R(t, c) = |PCC(t, c)| + MIC(t, c) (1)

[0032] where t represents the running time, c represents the configuration parameter, PCC represents the Pearson correlation coefficient, which is used to measure the linear correlation degree between the running time and the configuration parameter, and MIC represents the maximum mutual information coefficient, which is used to measure the non-linear correlation degree between the running time and the configuration parameter.

[0033] Using formula (1), the correlation degrees between the running times of different types of tasks and each configuration parameter can be obtained, and the correlation degrees are arranged in descending order. In this way, the parameters that have a great impact on performance can be selected. For example, through analysis, it is found that the configuration parameters related to resources (such as memory usage, etc.) have a very large impact on performance. When the value of each parameter increases, the performance of some tasks increases, while the performance of some tasks decreases. Therefore, both the increase and decrease of the value need to be considered.

[0034] In this step S110, taking the characteristics of Spark typical programs as prior knowledge can quickly find the configuration parameters that have a great impact on performance. However, the existing Spark configuration parameter optimization algorithms collect a large number of samples for training according to each task, and do not use the characteristics of Spark typical programs as prior knowledge.

[0035] Step S120, select the configuration parameters related to resources according to the correlation relationship, construct them into a prior configuration set, initialize the prior configuration set, and obtain the minimum running time by changing the values of the parameters in the prior configuration set and changing the computing power of the big data platform.

[0036] After obtaining the prior knowledge (i.e., the correlation between the task running time and the configuration parameters), initialize the resource-related configuration parameters obtained from the prior knowledge, that is, initialize the constructed prior configuration set.

[0037] Specifically, first, for the prior configuration set, run all resources with maximized configuration, then change the memory usage rate parameter and run it with both an increase and a decrease respectively, and finally change the number of cores and the memory size of each executor, and try to run tasks using multiple executors simultaneously on each computing node. During this process, the minimum running time obtained is T.

[0038] In this step S120, initializing the prior configuration set obtained by using prior knowledge can significantly shorten the initialization process, and compared with existing machine learning that uses a large number of training samples, it significantly saves the search time.

[0039] Step S130, run the task, and determine the expected time for the task to run based on the minimum running time, as a constraint to control the termination of subsequent task runs.

[0040] For example, set the expected time to T*a, where 0 < a ≤ 1. After obtaining the expected time, use this expected time as the maximum permitted time for the next task run, and tasks that exceed this maximum permitted time will automatically terminate, which can effectively avoid the situation where individual tasks run for too long.

[0041] Furthermore, for the new running time T obtained after the task runs successfully, if this new running time is less than the stored minimum running time, update the expected time and set the new expected time to T*a.

[0042] In practical applications, by adjusting the parameter a, a balance can be found between the search time for the optimal configuration and the performance. The smaller the value of a, the shorter the search time, but the worse the optimal performance will be. Therefore, by adjusting the value of a, users can find a suitable point between the search time and the optimal performance.

[0043] In this step, use the expected time to control whether the task terminates, thereby reducing the time for each task run. It should be noted that other forms of expected time can also be designed using the minimum running time, such as T*b 2 etc.

[0044] Step S140, use the set constraint conditions to determine whether to end the task execution.

[0045] For example, after each task is executed, check whether it has been completed. If the number of tasks executed reaches the number set by the user, exit the task execution, otherwise continue to execute the task.

[0046] Step S150: Select the optimal configuration parameters according to the task running results.

[0047] After exiting the task execution, find the task with the shortest running time among all successfully executed tasks. The configuration corresponding to this task is the optimal configuration. The performance of the big data processing framework corresponding to this optimal configuration is also referred to as the optimal performance in this article.

[0048] To further verify the feasibility and effectiveness of the present invention, experiments were conducted. Specifically, by using the benchmark tasks of HiBench, the optimal configuration search was performed for the tasks under the Spark framework. In the experiment, all 19 Spark data processing tasks in HiBench were compared (such as sort for the sorting task, Scan for the database scanning task, which will not be elaborated one by one here). For each task, a small sample search for the optimal configuration was carried out, and the sample size was set to 30.

[0049] Figure 3 It represents the time spent on searching for the optimal configuration. Among them, A represents the method of the present invention, and a = 1 is used when setting the expected time; CherryPick represents the existing method of small sample search through Bayesian optimization and Gaussian process; RL represents an existing method of optimal configuration search through reinforcement learning. It can be seen that the average search speed of the present invention is 2.39 and 1.68 times that of CherryPick and RL respectively, significantly reducing the search time for the optimal configuration.

[0050] Figure 4 It represents the task time obtained by using the optimal configuration. It can be seen that the optimal task times obtained by the three methods are very close. The average task time obtained by the present invention is 3% and 8% less than that of CherryPick and RL respectively, without sacrificing performance due to the short search time.

[0051] Figure 5 It represents modifying the setting of the expected time a, taking the case of a = 1 as the reference value, and comparing the ratio of the search time and the optimal performance of the present invention when a takes different values. It can be seen that when the value of a is relatively small, the search time is also relatively short, but the optimal performance is relatively poor. By adjusting the value of a, users can flexibly balance the optimal configuration and the optimal performance.

[0052] In summary, the present invention proposes the determination of prior knowledge. By analyzing typical Spark tasks, parameters that are relatively important for performance optimization are obtained, and these parameters are used as prior knowledge. For each new task, the prior knowledge is first used for initialization, and then configuration parameters are randomly generated and the task is run. An expected time can be obtained each time the task is run, and this expected time is used as the maximum allowed running time for the next task run. Compared with the existing machine learning methods that use a large number of training samples, the small-sample optimization method based on prior knowledge provided by the present invention has simple parameters and is easy to adjust. The optimization relying on small samples improves the processing performance of the big data platform and reduces the search time for the optimal configuration.

[0053] The present invention can be a system, a method, and / or a computer program product. The computer program product can include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0054] The computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as an instantaneous signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0055] The computer-readable program instructions described herein can be downloaded from the computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include a copper transmission cable, an optical fiber transmission, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. The network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0056] The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present invention.

[0057] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0058] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create a means for implementing the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium, which instructions cause a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises a manufacture, including instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0059] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0060] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or by combinations of special purpose hardware and computer instructions. As will be apparent to those of ordinary skill in the art, implementations by hardware, by software, and by a combination of software and hardware are equivalent.

[0061] The embodiments of the present invention have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for optimizing configuration parameters of a big data framework based on prior knowledge, comprising the following steps: Select multiple types of tasks from the task distributions adapted by the big data framework, execute them under different parameter configurations, and obtain the correlation relationship between the running times of different types of tasks and the configuration parameters; Select configuration parameters related to resources according to the correlation relationship, construct them into a prior configuration set, initialize the prior configuration set, and obtain the minimum running time by changing the values of the parameters in the prior configuration set and changing the computing power of the big data framework; Determine the expected running time of the task based on the minimum running time, use it as a constraint to control the termination of subsequent task runs, and dynamically update the expected running time according to the task running results; Among all the tasks that have run successfully, find the configuration parameters corresponding to the task with the shortest running time as the optimal configuration; Among them, the correlation relationship between the running times of different types of tasks and the configuration parameters is obtained according to the following steps: Select multiple types of tasks, run them multiple times under different configurations, and analyze the running time \(t\) of the \(i\)-th run i and the corresponding configuration parameter \(c\) 1,i , \(c\) 2,i , …>, where \(c\) m,i represents the \(m\)-th parameter during the \(i\)-th run; According to the running results, determine the correlation degree R(t, c) between the task running time and the configuration parameters, where t represents the running time and c represents the configuration parameter; Among them, the correlation degree R(t, c) between the task running time and the configuration parameters is expressed as: R(t, c) = |PCC(t, c)| + MIC(t, c) Among them, t represents the running time, c represents the configuration parameter, PCC represents the Pearson correlation coefficient, and MIC represents the maximum mutual information coefficient.

2. The method according to claim 1, wherein, Set the expected time to T * a, where 0 < a ≤ 1 and T is the obtained minimum running time.

3. According to the method described in claim 1, the correlation relationship is used to characterize the linear correlation degree, non-linear correlation degree or a combination of both between the task running time and the configuration parameters.

4. The method according to claim 1, wherein, Initializing the prior configuration set and obtaining the minimum running time by changing the values of the parameters in the prior configuration set and changing the computing power of the big data framework includes: For the prior configuration set, run with the maximum resource configuration, and change the memory usage parameter, run it with an increase and a decrease respectively; Change the number of cores and the memory size of each executor, and try to run tasks using multiple executors on each computing node at the same time, thereby obtaining the minimum running time.

5. The method according to claim 1, wherein The big data framework includes Hadoop, Spark, and Flink.

6. The method according to claim 2, wherein, Set the value of a according to the search time of the optimal configuration and the processing performance corresponding to the optimal configuration.

7. A computer-readable storage medium having a computer program stored thereon, wherein, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer device, comprising a memory and a processor, wherein a computer program capable of running on the processor is stored on the memory, characterized in that When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 6.

Images