SPARK parameter automatic adjustment and optimization method based on cost model

A cost model and automatic optimization technology, applied in the direction of program startup/switching, electrical digital data processing, program control design, etc., can solve problems such as dynamic upgrade of performance model, inability to adapt to large parameter space, and inability to accurately establish models, etc. question

Active Publication Date: 2020-01-24
BEIHANG UNIV
View PDF3 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0009] For experienced operation and maintenance personnel, when there are few parameters to be adjusted, the task performance can be greatly increased by adjusting the configuration parameters, but this requires very familiarity with the system to achieve, and it is impossible to determine whether there is a better Configuration
But with the increase of parameters, even experienced operation and maint...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • SPARK parameter automatic adjustment and optimization method based on cost model
  • SPARK parameter automatic adjustment and optimization method based on cost model
  • SPARK parameter automatic adjustment and optimization method based on cost model

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0083] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

[0084] The basic idea of ​​the present invention is to obtain the optimal parameters of the Spark task by establishing a cost-based performance model combined with a parameter space search algorithm; and to provide the reference value of the optimized parameter of the unknown task by judging the similarity of tasks.

[0085] figure 1 A schematic diagram of the system architecture for implementing the cost model-based SPARK parameter automatic tuning metho...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a Spark parameter automatic adjustment and optimization method based on a cost model. The method comprises the following steps: 1, constructing a cost-based performance model byobtaining configuration of task execution and corresponding cost information, and obtaining optimal configuration in a given parameter space; 2, for tasks of unknown types, using default parameters for one-time operation, and giving a reference value of optimal configuration by judging the similarity of the tasks. According to the invention, a cost-based performance model is provided for solvingproblems possibly existing in current configuration parameter adjustment and optimization. A performance model is generated by analyzing Spark historical tasks, optimization parameters are obtained through a parameter space search algorithm. Meanwhile, the model is continuously upgraded and adjusted along with running of new tasks, the accuracy of the model is improved, and parameter reference values are provided after the model runs once for tasks of unknown types.

Description

technical field [0001] The invention relates to establishment of a Spark task performance model, space search of configuration parameters of a big data system, and task similarity judgment. Background technique [0002] With the continuous development of science and technology, as small as a mobile phone, a tablet, as large as an astronomical telescope, and the Large Hadron Collider, they are all data producers. massive data. How to store, process, and analyze these data has also become a realistic topic before everyone. Since Google published the Google File System paper in 2003, a number of distributed computing frameworks have emerged, such as hadoop and sparkSpark—cluster computing on working sets. It plays an important role in various application scenarios. In response to the low efficiency of hadoop in iterative processing, Spark came into being. Although spark has a good performance in processing iterative computing tasks, the performance optimization of spark tas...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/48
CPCG06F9/4881
Inventor 杨海龙马群李云春
Owner BEIHANG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap