Unlock instant, AI-driven research and patent intelligence for your innovation.

Performance optimization and parameter configuration method based on in-memory computing framework spark

A parameter configuration method and memory computing technology, applied in computing, computer components, resource allocation, etc., can solve the problem that performance is greatly affected by configuration parameters, and achieve the effect of satisfying accuracy

Active Publication Date: 2022-07-01
CHONGQING UNIV OF POSTS & TELECOMM
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] The purpose of the present invention is to propose a performance optimization and parameter configuration based on the memory computing framework Spark for the existing distributed computing framework Spark due to the large number of configuration parameters, the performance is greatly affected by the configuration parameters, and the application program has different characteristics. method

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Performance optimization and parameter configuration method based on in-memory computing framework spark
  • Performance optimization and parameter configuration method based on in-memory computing framework spark
  • Performance optimization and parameter configuration method based on in-memory computing framework spark

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0023] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

[0024] like figure 1 As shown, a performance optimization and parameter configuration strategy method based on the memory computing framework Spark includes the following four steps:

[0025] One, if image 3 As shown, the Spark resource scheduling process, the specific resource scheduling process is shown in the following three steps:

[0026] (1) Driver is the main() function that runs Spark Applicaion (Spark application), which creates SparkContext (Spark context object). SparkContext is responsible for communicating with the Cluster Manager (client manager) for resource application,...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a performance optimization and parameter configuration method based on a memory computing framework Spark. By first determining the Spark application type and the Spark performance parameters affecting different types, randomly combining the configuration parameters to obtain a training set, and establishing the training set through the LightGBM algorithm The configuration parameter model is used to search for the optimal combination of hyperparameters of the LightGBM algorithm through the Bayesian optimization algorithm, which further enables the configuration model to select the optimal configuration parameters. The present invention can find the optimal configuration parameters for different types of application programs running in different cluster environments for users without requiring users to understand the Spark operation mechanism, parameter meaning operations and value ranges, as well as application type characteristics and input sets. Compared with the previous parameter configuration method, it is more simple, clear and convenient.

Description

technical field [0001] The invention belongs to the technical fields of big data, cloud computing, distributed systems and the like, and particularly relates to a performance optimization and parameter configuration method based on a memory computing framework Spark. Background technique [0002] Distributed memory computing framework Spark is a big data parallel computing framework based on memory computing. The characteristics of massive data and real-time processing requirements brought by big data have created a huge contradiction with the traditional computing-centric model, making it difficult for traditional computing models to adapt to data processing in today's big data environment. In general, processing has also shifted from computational processing to data processing. Therefore, the problem of data processing speed becomes more and more prominent, and the real-time efficiency is not strong. The characteristics of big data, such as fast increment speed and low t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50G06F9/445G06K9/62
CPCG06F9/5016G06F9/5083G06F9/4451G06F18/24155G06F18/214
Inventor 范天文龙昭华沈励芝余快崔永明
Owner CHONGQING UNIV OF POSTS & TELECOMM