Spark performance optimization control method, device and equipment and storage medium

Abstract

The embodiment of the invention provides a Spark performance optimization control method and device, equipment and a storage medium, and the method comprises the steps of determining an initial operation configuration parameter of a to-be-optimized Spark task, controlling the operation of the Spark task according to the operation configuration parameter, and obtaining the resource occupation information in the operation process of the Spark task; when it is determined that the operation configuration parameters need to be adjusted according to the obtained resource occupation information, adjusting the operation configuration parameters of the Spark task based on the resource occupation information, that is, closed-loop dynamic adjustment optimization is conducted on the operation configuration parameters of the Spark task according to the specific operation condition of the Spark task, therefore, resources are reasonably utilized as much as possible in the Spark task running process,and the performance of Spark can be rapidly improved.

Description

technical field

[0001] The present invention relates to the technical field of big data computing, in particular to a Spark performance optimization control method, device, equipment and storage medium. Background technique

[0002] Big data systems need to process massive amounts of data, which inevitably requires a large number of servers and consumes a lot of power resources. How to improve the computing performance of big data computing systems and reduce the use of servers is an urgent problem to be solved. Spark is a fast and general computing engine designed for large-scale data processing. In related technologies, in the process of using Spark to process massive data, the corresponding Spark tasks are run with fixed configuration parameters, resulting in unreasonable resource utilization, which in turn causes Spark to fail to achieve better performance. Contents of the invention

[0003] A Spark performance optimization control method, device, equipment, and stora...

Examples

Embodiment 1

[0028] The Spark performance optimization control method provided in this embodiment can first determine the initial running configuration parameters of the Spark task to be optimized, and then control the running of the Spark task according to the running configuration parameters, thereby according to the actual resource occupation of the Spark task running process Dynamically adjust the running configuration parameters, and control the Spark task to run according to the adjusted running configuration parameters, so as to quickly improve the performance of Spark.

[0029] For ease of understanding, this embodiment is described below in conjunction with a specific process of a Spark performance optimization control method. Please refer to figure 1 shown, including:

[0030] S101: Determine the running configuration parameters of the Spark task to be optimized.

[0031] In this embodiment, the Spark task to be optimized may be at least one Spark task in the current Spark execu...

Embodiment 2

[0082] This embodiment provides a Spark performance optimization control device, which can be set in various communication devices (for example, the communication device can be but not limited to a server), please refer to Figure 4 shown, including:

[0083] The configuration parameter determination module 41 is configured to determine the running configuration parameters of the Spark task to be optimized. For the specific determination process, please refer to the above-mentioned embodiments, which will not be repeated here. The function of the configuration parameter determination module 41 in this embodiment can be realized by a processor, and the specific structure and implementation of the configuration parameter determination module 41 can also be flexibly set, for example, please refer to Figure 5 As shown, the configuration parameter determination module 41 includes a task computational complexity acquisition module 411, which can be used to calculate the task comput...

Embodiment 3

[0125] This embodiment also provides a communication device, which can be a server, see Figure 7 As shown, it includes a processor 701, a memory 702 and a communication bus 703;

[0126] The communication bus 703 is used to realize the communication connection between the processor 701 and the memory 702;

[0127] In an example, the processor 701 may be configured to execute one or more computer programs stored in the memory 702, so as to implement the steps of the Spark performance optimization control method in the above embodiments.

[0128] The present embodiment also provides a computer-readable storage medium, which includes information implemented in any method or technology for storing information, such as computer-readable instructions, data structures, computer program modules, or other data. volatile or nonvolatile, removable or non-removable media. Computer-readable storage media include but are not limited to RAM (Random Access Memory, random access memory), RO...

technical field

[0001] The present invention relates to the technical field of big data computing, in particular to a Spark performance optimization control method, device, equipment and storage medium. Background technique

[0002] Big data systems need to process massive amounts of data, which inevitably requires a large number of servers and consumes a lot of power resources. How to improve the computing performance of big data computing systems and reduce the use of servers is an urgent problem to be solved. Spark is a fast and general computing engine designed for large-scale data processing. In related technologies, in the process of using Spark to process massive data, the corresponding Spark tasks are run with fixed configuration parameters, resulting in unreasonable resource utilization, which in turn causes Spark to fail to achieve better performance. Contents of the invention

[0003] A Spark performance optimization control method, device, equipment, and stora...

Examples

Embodiment 1

[0028] The Spark performance optimization control method provided in this embodiment can first determine the initial running configuration parameters of the Spark task to be optimized, and then control the running of the Spark task according to the running configuration parameters, thereby according to the actual resource occupation of the Spark task running process Dynamically adjust the running configuration parameters, and control the Spark task to run according to the adjusted running configuration parameters, so as to quickly improve the performance of Spark.

[0029] For ease of understanding, this embodiment is described below in conjunction with a specific process of a Spark performance optimization control method. Please refer to figure 1 shown, including:

[0030] S101: Determine the running configuration parameters of the Spark task to be optimized.

[0031] In this embodiment, the Spark task to be optimized may be at least one Spark task in the current Spark execu...

Embodiment 2

[0082] This embodiment provides a Spark performance optimization control device, which can be set in various communication devices (for example, the communication device can be but not limited to a server), please refer to Figure 4 shown, including:

[0083] The configuration parameter determination module 41 is configured to determine the running configuration parameters of the Spark task to be optimized. For the specific determination process, please refer to the above-mentioned embodiments, which will not be repeated here. The function of the configuration parameter determination module 41 in this embodiment can be realized by a processor, and the specific structure and implementation of the configuration parameter determination module 41 can also be flexibly set, for example, please refer to Figure 5 As shown, the configuration parameter determination module 41 includes a task computational complexity acquisition module 411, which can be used to calculate the task comput...

Embodiment 3

[0125] This embodiment also provides a communication device, which can be a server, see Figure 7 As shown, it includes a processor 701, a memory 702 and a communication bus 703;

[0126] The communication bus 703 is used to realize the communication connection between the processor 701 and the memory 702;

[0127] In an example, the processor 701 may be configured to execute one or more computer programs stored in the memory 702, so as to implement the steps of the Spark performance optimization control method in the above embodiments.

[0128] The present embodiment also provides a computer-readable storage medium, which includes information implemented in any method or technology for storing information, such as computer-readable instructions, data structures, computer program modules, or other data. volatile or nonvolatile, removable or non-removable media. Computer-readable storage media include but are not limited to RAM (Random Access Memory, random access memory), RO...

Images