A method, system, storage medium and device for controlling the parallelism of a single task

Abstract

The present invention discloses a method, system, storage medium and device for controlling the parallelism of a single task. The method is based on the SparkThriftServer service and includes the following steps: by configuring the parallelism enabling parameter, setting whether to enable the core quantity usage limit; by configuring a setExecutionParallelism method, registering the spark.sql.execution.parallelism configuration parameter as a threshold in the corresponding local thread, if the parallelism enabling parameter is true, judging whether the corresponding thread parallelism has been reached according to the threshold, if reached, terminating the process, if not reached, allocating resources. The present invention proposes a SparkThriftServer service that can control resource allocation, which more autonomously and reasonably allocates the current self-use usage, thereby improving resource utilization.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a method, system, storage medium and device for controlling the parallelism of a single task. Background Art

[0002] Spark Thrift Server is a distributed service based on Apache Spark and HiveServer2. It allows users to submit SQL queries to Spark clusters through standard HiveServer2 interfaces. SparkThrift Server combines the functions of Apache Thrift and Apache Hive to provide users with a high-performance and scalable way to query large-scale data sets.

[0003] However, the SQL statements currently running in SparkThriftServer will use all resources in the ThriftServer service to process the corresponding SQL statements when there are enough stages. If one of the SQL statements is blocked, all subsequent queries will wait for resources, and some SQL queries do not need to use so many resources to run, which will cause a waste of resources. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method for specifying the number of cores in a Spark Thrift Server to be used so as to allocate resources more reasonably.

[0005] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0006] A method for controlling the parallelism of a single task, based on the SparkThriftServer service, includes the following steps:

[0007] By configuring the parallelism enabling parameters, set whether to enable the core number usage limit;

[0008] By configuring a setExecutionParallelism method, the spark.sql.execution.parallelism configuration parameter is registered as a threshold to the corresponding local thread.

[0009] If the parallelism enabling parameter is true, it is determined whether the corresponding thread parallelism has been reached according to the threshold. If it has been reached, the process ends; if not, resources are allocated.

[0010] Preferably, if the parallelism enabling parameter is true, then judging whether the corresponding thread parallelism has been reached according to the threshold, terminating the process if reached, and allocating resources if not reached, the method comprises the steps of:

[0011] Get the parallelism enable parameter in SparkConf, and determine whether to enable task parallelism through the value of the parallelism enable parameter. If the parallelism enable parameter is true, get the thread parallelism threshold corresponding to each task, and determine whether the parallelism has been reached. If not, allocate resources. If yes, end the process.

[0012] Preferably, the method of setting whether to enable the core quantity usage restriction by configuring the parallelism enabling parameter specifically includes:

[0013] Configure the spark.sql.execution.parallelism.enable parameter value. If the parameter value is true, it means that the core usage limit is enabled. If the parameter value is false, it means that the current SQL is allowed to use all resources.

[0014] Preferably, the thread parallelism threshold corresponding to each task is obtained, and it is determined whether the parallelism has been reached. If not, the method for allocating resources specifically includes:

[0015] Filter the currently running Job tasks to see if there is a task with the same ID as the current task running;

[0016] If it is running, get the current spark.sql.execution.parallelism parameter value of the task in the corresponding running tasks data set, and determine whether the threshold is exceeded during resource application. If it is exceeded, stop resource allocation; if it is not exceeded, allocate resources to the current task.

[0017] Correspondingly, the present invention also proposes a system for controlling the parallelism of a single task, based on the SparkThriftServer service system, comprising:

[0018] The enabling configuration unit is used to set whether to enable the core quantity usage limit by configuring the parallelism enabling parameters;

[0019] The resource allocation unit is used to register the spark.sql.execution.parallelism configuration parameter as a threshold to the corresponding local thread by configuring a setExecutionParallelism method;

[0020] The running unit is used to realize that if the parallelism enabling parameter is true, it determines whether the corresponding thread parallelism has been reached according to the threshold value, ends the process if it has been reached, and allocates resources if it has not been reached.

[0021] Preferably, the method for allocating resources by the running unit specifically includes: obtaining the parallelism enabling parameter in SparkConf, judging whether it supports enabling and setting task parallelism through the value of the parallelism enabling parameter, if the parallelism enabling parameter is true, obtaining the thread parallelism threshold corresponding to each task, judging whether the parallelism has been reached, if not, allocating resources, and if so, ending the process.

[0022] Preferably, the specific method of configuring the parallelism enabling parameter of the enabling configuration unit includes configuring the spark.sql.execution.parallelism.enable parameter value. If the parameter value is true, it means that the usage restriction of the number of cores is enabled. If the parameter value is false, it means that the current sql is allowed to use all resources.

[0023] Preferably, the method for allocating resources by the operation unit specifically includes: screening whether there is a task with the same task ID as the current task running from the currently running Job tasks;

[0024] If it is running, get the current spark.sql.execution.parallelism parameter value of the task in the corresponding running tasks data set, and determine whether the threshold is exceeded during resource application. If it is exceeded, stop resource allocation; if it is not exceeded, allocate resources to the current task.

[0025] A computer storage medium stores a computer program, wherein the computer program is called by a processor to implement a method for controlling the parallelism of a single task.

[0026] An electronic device comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method for controlling the parallelism of a single task.

[0027] Beneficial effects: Compared with the existing technology, this application implements a SparkThriftServer service that can control resource allocation, allocates the current self-usage more autonomously and reasonably, and thus improves resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 This is a flowchart of the method for controlling the parallelism of a single task disclosed in Example 1. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with embodiments. The following embodiments are for explanation of the present invention but the present invention is not limited to the following embodiments.

[0031] Glossary:

[0032] Job: In Spark, a job is a collection of stages connected by dependencies. Each job represents a computing task, such as reading data, performing filtering, aggregation, etc. The task identifier proposed in this article is also called JobID during execution.

[0033] In the context of Spark Thrift Server, a "session connection" refers to a session connection established between a client and a Spark Thrift Server.

[0034] The number of cores (CPU cores) available when running Spark applications

[0035] The setExecutionParallelism method is used to set the parallelism of Spark Thrift Server when executing queries. Parallelism refers to the number of tasks that Spark can run simultaneously when executing a query or task.

[0036] Task: Task is the finest-grained parallel computing unit in Spark.

[0037] Embodiment 1: This embodiment discloses a method for controlling the parallelism of a single task, which is based on the SparkThriftServer service and includes the following steps:

[0038] S001, by configuring the parallelism enabling parameters, set whether to enable the core number usage limit;

[0039] S002, by configuring a setExecutionParallelism method, register the spark.sql.execution.parallelism configuration parameter as a threshold in the corresponding local thread, and when the content in the session is processed by the Spark kernel, the spark.sql.execution.parallelism parameter in the corresponding local thread is used for corresponding processing.

[0040] This solution modifies the interface of SparkThriftServer to execute SQL tasks and adds an interface that can set the number of cores of the current session thread. When the task parallelism setting is enabled, the parameter of the number of cores required for the corresponding task is set to the corresponding local thread in the current session. Add an entry for parameter injection to pass the parameters of resource control to the actual resource allocation location, so as to achieve a more reasonable allocation of resources and avoid large SQL queries being blocked when serving small SQL queries.

[0041] S101, by configuring the parallelism enabling parameter, setting whether to enable the core quantity usage restriction method, specifically includes:

[0042] Configure the spark.sql.execution.parallelism.enable parameter value. If the parameter value is true, it means that the core usage limit is enabled. If the parameter value is false, it means that the current SQL is allowed to use all resources.

[0043] Exemplarily, the default value of the spark.sql.execution.parallelism.enable parameter is configured to be 2, and the parallelism threshold of the number of cores is configured to be 2. The parallelism threshold can be set accordingly according to the computer performance, the required data volume, etc. It can be set as a default value, or it can be set independently according to the business scenario requirements. For example, the parallelism threshold parameter configuration function is provided to the user, and the user adjusts the parallelism threshold parameter by the data volume of his own task; preferably, the parallelism threshold of a single task can be set according to the number of tasks generated within a period of time, the resources required for a single task, and the total resource configuration. Comprehensive analysis to achieve reasonable resource allocation and rapid response. Exemplarily, if the amount of data is large, the spark.sql.execution.parallelism.enable parameter value can be configured to 5, 8, 10, etc., and it is necessary to ensure that the sum of the parallelism of different tasks is less than or equal to the total service threads, so that no waiting will occur.

[0044] A preferred solution is further disclosed, based on the above solution, and further includes a method for allocating a corresponding number of cores to each task, including:

[0045] Get the parallelism enablement parameter in SparkConf, and determine whether to enable task parallelism through the value of the parallelism enablement parameter. If supported, get the thread parallelism corresponding to each task and determine whether the parallelism has been reached. If not, allocate resources. If yes, end the process.

[0046] SparkConf is the configuration file class used in Spark, which reads the parameters set in the entire Spark environment and at startup.

[0047] The method of obtaining the thread parallelism threshold corresponding to each task and determining whether the parallelism has been reached, and if not, allocating resources specifically includes:

[0048] Filter the currently running Job tasks to see if there is a task with the same ID as the current task running;

[0049] If it is running, get the current spark.sql.execution.parallelism parameter value of the task in the corresponding running tasks data set, and determine whether the spark.sql.execution.parallelism configuration parameter (threshold) is exceeded during resource application. If it is exceeded, stop resource allocation; if not, allocate resources to the current task.

[0050] Specifically, if the task corresponding to the current task identifier is decomposed into 10 steps (stages), if the parallelism is not controlled, the system needs to allocate 10 execution units by default for parallel execution. When the number of cores is insufficient, all of them will be occupied. As a result, the SQL statement running in the SparkThriftServer in the prior art proposed in the background technology will use all resources in the ThriftServer service to process the corresponding SQL when there are enough stages, causing a blocking problem.

[0051] Based on the scheme disclosed above, the specific method of this preferred scheme includes:

[0052] S201, obtain the spark.sql.execution.parallelism.enable parameter value in SparkConf to determine whether it is True to determine whether the current SparkThriftserver supports enabling the setting of task parallelism. If the parameter value is True, execute S202.

[0053] S202, filter out from the running Job tasks whether there is a task with the same ID as the current task running, if there is a corresponding task running, execute S203

[0054] S203, obtaining the spark.sql.execution.parallelism value of the task in the corresponding running tasks data set and determining whether it exceeds the threshold when applying for resources. If so, stop allocating resources; if not, continue to allocate resources to the current task.

[0055] For example, if the task identified as a101 is decomposed into 50 tasks, a first data set is obtained; the task identified as a102 is decomposed into 8 tasks, a second data set is obtained; the task identified as a103 is decomposed into 12 tasks, a third data set is obtained;

[0056] The task in the first data set carries the corresponding task identifier a101, the task in the second data set carries the corresponding task identifier a102, and the task in the third data set carries the corresponding task identifier a103. And there is a sequence in the submission time of task a101, task a102 and task a103; at this time, if the enable parameter is true, it means that the single task parallelism control scheme is turned on. When allocating resources, it is determined whether the corresponding task is running according to the task identifier a101. If so, it is further determined whether the number reaches the threshold of 2. If it does not reach 2, resources will continue to be allocated until the threshold is reached; if the threshold has been reached, the resource allocation will stop. Correspondingly, tasks a102 and a103 are subsequently submitted, and resources can be allocated according to the same logic.

[0057] If there are 20 CPU cores, 10 tasks can be executed at the same time. If it is not enabled, only one task can be executed, which will cause congestion. Even small tasks a102 and a103, which could have responded quickly, are blocked because task a101 occupies all resources.

[0058] It should be noted that the above parameter values ​​are only described as examples for reference.

[0059] This solution proposes a resource allocation logic to allocate the corresponding number of CPU cores for each task. In Spark, the current spark.sql.execution.parallelism parameter value is used when obtaining the number of CPU cores for each subquery. A SparkThriftServer service that can control resource allocation is implemented, which allocates the current self-use amount more autonomously and reasonably, thereby improving resource utilization.

[0060] Embodiment 2: A system for controlling the parallelism of a single task is disclosed, based on the SparkThriftServer service system, including:

[0061] The enabling configuration unit is used to set whether to enable the core quantity usage limit by configuring the parallelism enabling parameters;

[0062] The resource allocation unit is used to register the spark.sql.execution.parallelism configuration parameter as a threshold to the corresponding local thread by configuring a setExecutionParallelism method;

[0063] The running unit is used to realize that if the parallelism enabling parameter is true, it determines whether the corresponding thread parallelism has been reached according to the threshold value, ends the process if it has been reached, and allocates resources if it has not been reached.

[0064] Preferably, the method for allocating resources by the running unit specifically includes: obtaining the parallelism enabling parameter in SparkConf, judging whether it supports enabling and setting task parallelism through the value of the parallelism enabling parameter, if the parallelism enabling parameter is true, obtaining the thread parallelism threshold corresponding to each task, judging whether the parallelism has been reached, if not, allocating resources, and if so, ending the process.

[0065] Preferably, the specific method of configuring the parallelism enabling parameter of the enabling configuration unit includes configuring the spark.sql.execution.parallelism.enable parameter value. If the parameter value is true, it means that the usage restriction of the number of cores is enabled. If the parameter value is false, it means that the current sql is allowed to use all resources.

[0066] Preferably, the method for allocating resources by the operation unit specifically includes: screening whether there is a task with the same task ID as the current task running from the currently running Job tasks;

[0067] If it is running, get the current spark.sql.execution.parallelism parameter value of the task in the corresponding running tasks data set, and determine whether the threshold is exceeded during resource application. If it is exceeded, stop resource allocation; if it is not exceeded, allocate resources to the current task.

[0068] Also disclosed is a computer storage medium storing a computer program, wherein the computer program is called by a processor to implement a method for controlling the parallelism of a single task.

[0069] Also disclosed is an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method for controlling the parallelism of a single task.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0071] The units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0072] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0073] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0074] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for controlling the parallelism of a single task, characterized in that: The method is based on the SparkThriftServer service and includes the following steps: By configuring the parallelism enabling parameters, set whether to enable the core number usage limit; Register the spark.sql.execution.parallelism configuration parameter as a threshold in the corresponding local thread, where the spark.sql.execution.parallelism configuration parameter includes the spark.sql.execution.parallelism.enable parallelism enablement parameter; Get the parallelism enable parameter in SparkConf, and determine whether to enable task parallelism through the value of the parallelism enable parameter. If the parallelism enable parameter is true, get the thread parallelism threshold corresponding to each task, and determine whether the parallelism has been reached. If not, allocate resources. If yes, end the process. If the parallelism enable parameter is false, it means that the current SQL is allowed to use all resources; The method of allocating resources includes: screening whether there is a task with the same task ID as the current task running from the currently running Job tasks; If it is running, get the current spark.sql.execution.parallelism parameter value of the task in the corresponding running tasks data set, and determine whether the threshold is exceeded during resource application. If it is exceeded, stop resource allocation; if it is not exceeded, allocate resources to the current task.

2. A system for controlling the parallelism of a single task, characterized in that: Based on SparkThriftServer service system, including: The enabling configuration unit is used to set whether to enable the core quantity usage limit by configuring the parallelism enabling parameters; A resource allocation unit, used to register the spark.sql.execution.parallelism configuration parameter as a threshold to a corresponding local thread; wherein the spark.sql.execution.parallelism configuration parameter includes the spark.sql.execution.parallelism.enable parallelism enabling parameter; The run unit is used to obtain the parallelism enable parameter in SparkConf, and determine whether to enable task parallelism through the value of the parallelism enable parameter. If the parallelism enable parameter is true, obtain the thread parallelism threshold corresponding to each task, and determine whether the parallelism has been reached. If not, allocate resources. If so, end the process. If the parallelism enable parameter is false, it means that the current SQL is allowed to use all resources. The method of allocating resources includes: screening whether there is a task with the same task ID as the current task running from the currently running Job tasks; If it is running, get the current spark.sql.execution.parallelism parameter value of the task in the corresponding running tasks data set, and determine whether the threshold is exceeded during resource application. If it is exceeded, stop resource allocation; if it is not exceeded, allocate resources to the current task.

3. A computer storage medium, characterized in that: The computer program is stored therein, and the computer program is called by a processor to implement a method for controlling the parallelism of a single task as claimed in claim 1.

4. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method for controlling the parallelism of a single task as claimed in claim 1.

Images