Spark performance optimization control method, device and equipment and storage medium

A technology for optimizing control and performance, applied in the field of big data computing, can solve problems such as unreasonable resource utilization and Spark's inability to achieve better performance, and achieve the effect of improving performance

Pending Publication Date: 2021-02-19
ZTE CORP
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] A Spark performance optimization control method, device, equipment, and storage medium provided by the embodiments of the present invention solve the problem that Spark tasks in related technologies are run with fixed configuration parameters, resulting in unreasonable resource utilization, which in turn causes Spark to fail to achieve better performance. performance problem

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Spark performance optimization control method, device and equipment and storage medium
  • Spark performance optimization control method, device and equipment and storage medium
  • Spark performance optimization control method, device and equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0028] The Spark performance optimization control method provided in this embodiment can first determine the initial running configuration parameters of the Spark task to be optimized, and then control the running of the Spark task according to the running configuration parameters, thereby according to the actual resource occupation of the Spark task running process Dynamically adjust the running configuration parameters, and control the Spark task to run according to the adjusted running configuration parameters, so as to quickly improve the performance of Spark.

[0029] For ease of understanding, this embodiment is described below in conjunction with a specific process of a Spark performance optimization control method. Please refer to figure 1 shown, including:

[0030] S101: Determine the running configuration parameters of the Spark task to be optimized.

[0031] In this embodiment, the Spark task to be optimized may be at least one Spark task in the current Spark execu...

Embodiment 2

[0082] This embodiment provides a Spark performance optimization control device, which can be set in various communication devices (for example, the communication device can be but not limited to a server), please refer to Figure 4 shown, including:

[0083] The configuration parameter determination module 41 is configured to determine the running configuration parameters of the Spark task to be optimized. For the specific determination process, please refer to the above-mentioned embodiments, which will not be repeated here. The function of the configuration parameter determination module 41 in this embodiment can be realized by a processor, and the specific structure and implementation of the configuration parameter determination module 41 can also be flexibly set, for example, please refer to Figure 5 As shown, the configuration parameter determination module 41 includes a task computational complexity acquisition module 411, which can be used to calculate the task comput...

Embodiment 3

[0125] This embodiment also provides a communication device, which can be a server, see Figure 7 As shown, it includes a processor 701, a memory 702 and a communication bus 703;

[0126] The communication bus 703 is used to realize the communication connection between the processor 701 and the memory 702;

[0127] In an example, the processor 701 may be configured to execute one or more computer programs stored in the memory 702, so as to implement the steps of the Spark performance optimization control method in the above embodiments.

[0128] The present embodiment also provides a computer-readable storage medium, which includes information implemented in any method or technology for storing information, such as computer-readable instructions, data structures, computer program modules, or other data. volatile or nonvolatile, removable or non-removable media. Computer-readable storage media include but are not limited to RAM (Random Access Memory, random access memory), RO...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a Spark performance optimization control method and device, equipment and a storage medium, and the method comprises the steps of determining an initial operation configuration parameter of a to-be-optimized Spark task, controlling the operation of the Spark task according to the operation configuration parameter, and obtaining the resource occupation information in the operation process of the Spark task; when it is determined that the operation configuration parameters need to be adjusted according to the obtained resource occupation information, adjusting the operation configuration parameters of the Spark task based on the resource occupation information, that is, closed-loop dynamic adjustment optimization is conducted on the operation configuration parameters of the Spark task according to the specific operation condition of the Spark task, therefore, resources are reasonably utilized as much as possible in the Spark task running process,and the performance of Spark can be rapidly improved.

Description

technical field [0001] The present invention relates to the technical field of big data computing, in particular to a Spark performance optimization control method, device, equipment and storage medium. Background technique [0002] Big data systems need to process massive amounts of data, which inevitably requires a large number of servers and consumes a lot of power resources. How to improve the computing performance of big data computing systems and reduce the use of servers is an urgent problem to be solved. Spark is a fast and general computing engine designed for large-scale data processing. In related technologies, in the process of using Spark to process massive data, the corresponding Spark tasks are run with fixed configuration parameters, resulting in unreasonable resource utilization, which in turn causes Spark to fail to achieve better performance. Contents of the invention [0003] A Spark performance optimization control method, device, equipment, and stora...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/445G06F9/50
CPCG06F9/44505G06F9/5016G06F9/5027G06F9/445G06F9/50
Inventor 陈艳山
Owner ZTE CORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap