Spark parameter adaptive optimization method and system

An optimization method and self-adaptive technology, which can be applied in the fields of electrical digital data processing, resource allocation, program control design, etc., and can solve problems such as complex parameter tuning of Spark.
CN110825522APending Publication Date: 2020-02-21武汉联图时空信息科技有限公司
4 Cites 3 Cited by

Patent Information

Authority / Receiving Office
CN · China
Current Assignee / Owner
武汉联图时空信息科技有限公司
Publication Date
2020-02-21

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a Spark parameter adaptive optimization method and system, and the method comprises the steps: 1, building a prediction model of Spark task execution time, and training the prediction model through sample data; 2, sequentially decomposing a Spark application to be executed into a job, a stage and a task according to a Spark execution mechanism, allocating each task to eachcore of each Executor node, predicting the execution time of each task by utilizing a task execution time prediction model, simulating the task scheduling process of the Spark stage based on the taskexecution time, and calculating the execution time of the stage under different parameter combinations; and 3, determining a final optimized parameter combination according to the stage execution timepredicted under different parameter combinations. According to the parameter self-adaptive optimization method, the execution time required by each parameter combination in an actual test is avoidedin a prediction mode, so that the feasible parameter self-adaptive optimization method is realized.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the technical field of spatiotemporal big data computing, and in particular relates to a Spark parameter self-adaptive optimization method and system. Background technique

[0002] In the past few years, we have entered the era of big data. The growth of massive data puts forward new requirements for storage management and computing analysis, and promotes the development of big data technology. The mainstream computing framework has also evolved from Hadoop based on MapReduce to Spark based on memory computing. As a complex general-purpose distributed computing framework, Spark provides a large number of configurable parameters to meet the various needs of using Spark in different application scenarios and maximize the computing performance of Spark. However, too many parameters and complex parameter settings also bring great challenges to program developers to optimize Spark applications. Contents of the invention

[0003] ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More