Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Spark cluster optimal configuration parameter determination method, device and apparatus

A technology for optimal configuration and parameter determination, applied in the field of spark clusters, can solve problems such as long time spent, low efficiency, and easy to fall into local optimum for optimal configuration parameters, so as to achieve the effect of improving accuracy and determining efficiency

Active Publication Date: 2020-09-04
LANGCHAO ELECTRONIC INFORMATION IND CO LTD
View PDF8 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] At present, the optimal configuration parameters of the spark cluster are obtained only by the user's continuous modification, trial, and comparison of the results based on the experience value. Easy to fall into local optimum

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Spark cluster optimal configuration parameter determination method, device and apparatus
  • Spark cluster optimal configuration parameter determination method, device and apparatus
  • Spark cluster optimal configuration parameter determination method, device and apparatus

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0048] see figure 1 , which shows a flowchart of a method for determining an optimal configuration parameter of a spark cluster provided in an embodiment of the present application. A method for determining an optimal configuration parameter of a spark cluster provided in an embodiment of the present application may include:

[0049] S11: Use the cloud platform to create spark cluster test environments with different host configurations.

[0050] Considering the existing problems o...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a spark cluster optimal configuration parameter determination method, device and apparatus, and a computer readable storage medium. The method comprises the steps of creating spark cluster test environments configured by different hosts by utilizing a cloud platform; obtaining an optimal configuration parameter sample corresponding to each spark cluster test environment through a preset sampling method and a machine learning algorithm, and combining the optimal configuration parameter samples into a sample set; training the sample set to obtain an optimization model; and inputting the configuration data of a target spark cluster into a tuning model to obtain an optimal configuration parameter corresponding to the target spark cluster. Through the disclosed technicalscheme, an optimization model is obtained through spark cluster test environment creation, a sample set composed of optimal configuration parameter samples acquired and the sample set is trained, andoptimal configuration parameters corresponding to a target spark cluster are obtained through the optimization model, so that the efficiency and accuracy of optimal configuration parameter determination are improved.

Description

technical field [0001] The present application relates to the technical field of spark clusters, and more specifically, to a method, device, equipment, and computer-readable storage medium for determining optimal configuration parameters of spark clusters. Background technique [0002] In order to improve the running performance of the spark cluster, its configuration parameters (such as driver-memory, num-executors, executor-cores, executor-memory, shuffle.partitions) need to be tuned. [0003] At present, the optimal configuration parameters of the spark cluster are obtained only by the user’s continuous modification, trial, and comparison of the results based on the experience value. It is easy to fall into local optimum. [0004] To sum up, how to improve the efficiency and accuracy of determining the optimal configuration parameters of the spark cluster is a technical problem to be solved urgently by those skilled in the art. Contents of the invention [0005] In vi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08H04L12/24H04L12/26
CPCH04L67/10H04L41/0823H04L43/08H04L41/145H04L41/0803
Inventor 王小珂张东
Owner LANGCHAO ELECTRONIC INFORMATION IND CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com