Automatic optimization method for configuration of big data processing platform

A big data processing and automatic optimization technology, applied in the computer field, can solve the problems of long running time and high time cost, achieve the effect of reducing time and avoiding long-term running configuration with poor parameters

CN113032033APending Publication Date: 2021-06-25SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
0 Cites 1 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2021-06-25

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention provides an automatic optimization method for configuration of a big data processing platform. The method collects a configuration parameter training set according to the following steps: randomly generating a group of configuration parameters of the big data processing platform, running the group of configuration parameters, monitoring execution time t, and stopping running when the execution time t exceeds dynamically set maximum permitted time Tmax; determining whether to exit the process of collecting the configuration parameter training set or not according to the fluctuation condition of the execution time t in the configuration parameter operation process; and after determining to quit the process of collecting the configuration parameter training set, selecting a group of configuration parameters with the shortest execution time from all the successfully operated configuration parameters as optimal configuration parameters. According to the method, the data collection time can be remarkably shortened on the premise of accurately finding out the optimal configuration.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the field of computer technology, in particular to an automatic optimization method for configuration of a big data processing platform. Background technique

[0002] In recent years, with the rapid development of Internet technology, big data has been used more and more. For example, in order to quickly process big data, the AMP Lab at the University of California, Berkeley designed a general parallel framework for Spark big data systems. It uses the BSD open source license and was donated to the Apache Software Foundation in 2013. Spark covers various workloads such as batch programs, user-interactive programs, iterative algorithms, etc. It extends the MapReduce model, and greatly reduces disk read / write operations through memory cluster computing, thereby greatly improving data processing speed.

[0003] The performance of the Spark framework will be affected by configuration parameters during operation. Due to the diffe...

Examples

Embodiment Construction

[0027] In order to make the purpose, technical solution, design method and advantages of the present invention clearer, the present invention will be further described in detail through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0028] In all examples shown and discussed herein, any specific values ​​should be construed as exemplary only, and not as limitations. Therefore, other instances of the exemplary embodiment may have different values.

[0029] Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods and devices should be considered part of the description.

[0030] see figure 1 As shown, the method for automatically optimizing the configuration of the big data processing platform prov...