Parameter configuration methods, devices, equipment and storage media

Abstract

This application provides a parameter configuration method, apparatus, device, and storage medium. The parameter configuration method includes: obtaining the data volume of a task to be processed corresponding to a computing engine and obtaining basic parameters of the cluster corresponding to the computing engine, wherein the cluster includes multiple computing nodes; obtaining a first parameter corresponding to the task to be processed, wherein the first parameter represents the number of data blocks that each core can execute within a set time period; determining the number of executors corresponding to the task to be processed based on the data volume of the task to be processed, the basic parameters of the cluster corresponding to the computing engine, and the first parameter corresponding to the task to be processed, so as to control each computing node to process the task to be processed through the computing engine based on the number of executors corresponding to the task to be processed, thereby realizing automatic configuration of computing engine parameters, improving the efficiency of parameter configuration, and thus improving the efficiency of data processing.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a parameter configuration method, apparatus, device and storage medium. Background Technology

[0002] Big data processing is characterized by large data volume, fast data generation speed, and diverse data types. In order to improve the processing efficiency of big data, distributed computing is usually adopted.

[0003] Common distributed computing frameworks include computing engines such as Spark and Flink. When processing big data tasks based on the offline computing functions of these computing engines, it is necessary to specify the resources required for the task, that is, to configure the sampling parameters. Based on the specified parameters, the processes to be started and the memory resources occupied by each process are determined.

[0004] In existing technologies, engineers typically need to manually set the parameters corresponding to a task based on experience when submitting the task, which results in long parameter configuration time and low efficiency. Summary of the Invention

[0005] This application provides a parameter configuration method, apparatus, device, and storage medium to solve the problem of low efficiency in computing engine parameter configuration.

[0006] Firstly, this application provides a parameter configuration method, which includes:

[0007] The system obtains the data volume of the task to be processed corresponding to the computing engine and the basic parameters of the cluster corresponding to the computing engine, wherein the cluster includes multiple computing nodes; it obtains a first parameter corresponding to the task to be processed, wherein the first parameter represents the number of data blocks that each core can execute within a set time period; based on the data volume of the task to be processed, the basic parameters of the cluster corresponding to the computing engine, and the first parameter corresponding to the task to be processed, it determines the number of executors corresponding to the task to be processed, so as to control each computing node to process the task to be processed based on the number of executors corresponding to the task to be processed through the computing engine.

[0008] Optionally, obtain the data volume of the task to be processed corresponding to the computing engine, including:

[0009] Using data warehouse tools, the tasks to be processed are converted into database tables; based on the database tables, the amount of data for the tasks to be processed is determined.

[0010] Optionally, obtaining the first parameter corresponding to the task to be processed includes:

[0011] Based on the task type of the task to be processed, determine the first parameter corresponding to the task to be processed.

[0012] Optionally, based on the task type of the task to be processed, the first parameter corresponding to the task to be processed is determined, including:

[0013] Based on the task type of the task to be processed, at least one historical task is determined, wherein the historical task is in the same cluster as the task to be processed; based on the first parameter corresponding to the at least one historical task, the actual processing time, and the expected processing time, the first parameter corresponding to the task to be processed is determined.

[0014] Optionally, the first parameter corresponding to the task to be processed is determined based on the first parameter corresponding to the at least one historical task, the actual processing time, and the expected processing time, including:

[0015] For each historical task, the first parameter corresponding to the historical task is adjusted based on the difference between the actual processing time and the expected processing time of the historical task; the first parameter corresponding to the task to be processed is determined based on the adjusted first parameter corresponding to each historical task.

[0016] Optionally, the basic parameters include the memory of a single data block and the number of cores of the executor.

[0017] Optionally, the number of executors corresponding to the task to be processed is determined based on the data volume of the task to be processed, the basic parameters of the cluster corresponding to the computing engine, and the first parameter corresponding to the task to be processed, including:

[0018] The total number of tasks corresponding to the tasks to be processed is determined based on the data volume of the tasks to be processed and the memory of a single data block; the number of executors corresponding to the tasks to be processed is determined based on the expected processing time of the tasks to be processed, the total number of tasks corresponding to the tasks to be processed, the first parameter corresponding to the tasks to be processed, and the number of cores of the executors corresponding to the tasks to be processed.

[0019] Optionally, the method further includes:

[0020] The memory of the executor corresponding to the task to be processed is determined based on the number of cores of the executor and the memory of a single data block.

[0021] Optionally, the task to be processed is a data file generated based on a distributed file system.

[0022] Secondly, this application provides a parameter configuration device, the device comprising:

[0023] The data volume acquisition module is used to acquire the data volume of the task to be processed corresponding to the computing engine and the basic parameters of the cluster corresponding to the computing engine, wherein the cluster includes multiple computing nodes; the first parameter acquisition module is used to acquire the first parameter corresponding to the task to be processed, wherein the first parameter represents the number of data blocks that each core can execute within a set time period; the executor number determination module is used to determine the number of executors corresponding to the task to be processed based on the data volume of the task to be processed, the basic parameters of the cluster corresponding to the computing engine, and the first parameter corresponding to the task to be processed, so as to control each computing node to process the task to be processed based on the number of executors corresponding to the task to be processed through the computing engine.

[0024] Optional, a data acquisition module, specifically used for:

[0025] Using data warehouse tools, the tasks to be processed are converted into database tables; based on the database tables, the amount of data for the tasks to be processed is determined.

[0026] Optional, the first parameter acquisition module is specifically used for:

[0027] Based on the task type of the task to be processed, determine the first parameter corresponding to the task to be processed.

[0028] Optionally, the first parameter acquisition module includes:

[0029] The historical task determination unit is used to determine at least one historical task based on the task type of the task to be processed, wherein the historical task is the same as the cluster corresponding to the task to be processed; the first parameter determination unit is used to determine the first parameter corresponding to the task to be processed based on the first parameter corresponding to the at least one historical task, the actual processing time, and the expected processing time.

[0030] Optionally, the first parameter determination unit is specifically used for:

[0031] For each historical task, the first parameter corresponding to the historical task is adjusted based on the difference between the actual processing time and the expected processing time of the historical task; the first parameter corresponding to the task to be processed is determined based on the adjusted first parameter corresponding to each historical task.

[0032] Optional, the actuator number determination module is specifically used for:

[0033] The total number of tasks corresponding to the tasks to be processed is determined based on the data volume of the tasks to be processed and the memory of a single data block; the number of executors corresponding to the tasks to be processed is determined based on the expected processing time of the tasks to be processed, the total number of tasks corresponding to the tasks to be processed, the first parameter corresponding to the tasks to be processed, and the number of cores of the executors corresponding to the tasks to be processed.

[0034] Optionally, the device further includes:

[0035] The executor memory determination module is used to determine the memory of the executor corresponding to the task to be processed based on the number of cores of the executor and the memory of a single data block.

[0036] Thirdly, this application also provides a parameter configuration device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the parameter configuration method provided in the first aspect of this application.

[0037] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the parameter configuration method provided in the first aspect of this application.

[0038] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the parameter configuration method provided in the first aspect of this application.

[0039] The parameter configuration method, apparatus, device, and storage medium provided in this application, targeting a computing engine for big data processing, automatically determine the number of executors required for a task based on the data volume of the task to be processed, the first parameter of the task, and the relevant parameters of the cluster. This enables parameter configuration at the application level using data volume, achieving automatic parameter configuration of the computing engine, improving the efficiency of parameter configuration, and ensuring high accuracy. It also reduces resource waste and improves data processing efficiency. Furthermore, by automatically configuring parameters, the complexity of big data processing based on the computing engine is reduced. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] Figure 1 A schematic diagram of a communication system architecture provided in one embodiment of this application;

[0042] Figure 2 A flowchart illustrating a parameter configuration method provided in one embodiment of this application;

[0043] Figure 3 A flowchart illustrating a parameter configuration method provided in another embodiment of this application;

[0044] Figure 4 This is a schematic diagram of the structure of a parameter configuration device provided in one embodiment of this application;

[0045] Figure 5 This is a schematic diagram of the structure of a parameter configuration device provided in one embodiment of this application.

[0046] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0048] Figure 1 This is a schematic diagram illustrating an application scenario of the parameter configuration method provided in one embodiment of this application, such as... Figure 1 As shown, when the computing engine receives a user-submitted task, it needs to pre-configure its parameters, such as the number of processes to be started and the memory usage of each process. The computing engine then schedules the various computing nodes in the cluster based on these configuration parameters. Figure 1 Taking a cluster consisting of 4 computing nodes as an example, the physical resources of the cluster are used to process the tasks to be processed, thereby realizing the distributed processing of the tasks to be processed.

[0049] In related technologies, users typically need to configure the computing engine parameters based on experience or by using fixed parameters. Using fixed parameters can easily lead to resource waste or insufficient resources, resulting in excessively long execution times for pending tasks. Manually configuring parameters based on experience requires users to be familiar with the characteristics of the computing engine and cluster, placing high demands on users and requiring a considerable amount of time for parameter configuration, resulting in low efficiency and making it unsuitable for scenarios involving large-scale task processing.

[0050] The parameter configuration method provided in this application aims to solve the above-mentioned technical problems of the prior art. Its main concept is as follows: after receiving a task to be processed, a first parameter is set for the task. This first parameter is used to set the number of databases that can be processed per core unit of time. Based on the data volume of the task to be processed, the basic parameters of the cluster, and the first parameter set for the task, the number of executors required for the task to be processed is automatically determined, thereby achieving automatic parameter configuration and improving the efficiency and accuracy of parameter configuration.

[0051] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0052] Figure 2 This is a flowchart illustrating a parameter configuration method provided in one embodiment of this application. This method can be executed by a parameter configuration device, which can be a computer, server, etc., such as one or more computing nodes in a cluster. Figure 2 As shown, the parameter configuration method includes the following steps:

[0053] Step S201: Obtain the data volume of the task to be processed corresponding to the computing engine and obtain the basic parameters of the cluster corresponding to the computing engine.

[0054] The tasks to be processed are big data processing tasks, which can be distributed processing tasks. The cluster consists of multiple computing nodes, and the number of cores and memory of each computing node are known. The data volume of the tasks to be processed describes the amount of disk space required by the computing nodes for each task. The basic parameters of the cluster corresponding to the computing engine are the basic parameters of the cluster environment corresponding to that computing engine.

[0055] Optionally, the task to be processed is a data file generated based on the distributed file system (HDFS, Hadoop Distributed File System) to achieve high-throughput data access and streaming data reading.

[0056] In the HDFS file system, files are stored in the form of data blocks, which are the smallest storage units in HDFS.

[0057] In some embodiments, the computing engine can be the Spark computing engine. The basic parameters include the memory of a single data block and the number of cores of the executor. Using the Spark computing engine can improve the processing speed of pending tasks and enhance the versatility of the platform.

[0058] Specifically, the memory of a single data block can be determined based on the disk transfer speed of the cluster. When the disk transfer speed is 100MB / s, the memory of a single data block can be 128M; when the disk transfer speed is 200MB / s, the memory of a single data block can be 256M, so that the addressing time of data in HDFS is 1% of the transfer time.

[0059] Specifically, the number of cores for the executor can be determined based on the number of compute nodes in the cluster, with the number of cores per compute node, or it can be set to a default value, such as 5, 4, or other values.

[0060] In some embodiments, the computing engine can be the Flink computing engine, and the number of cores of the executor can be replaced by the number of task slots of the container (Taskmanager). In the Flink computing engine, the executor can be replaced by a container.

[0061] Specifically, the amount of data for the task to be processed can be obtained using database tools such as Hive and Impala. Alternatively, the user can input the amount of data for the task when submitting it.

[0062] Step S202: Obtain the first parameter corresponding to the task to be processed.

[0063] The first parameter represents the number of data blocks that each core can execute within a set time period. A data block is the smallest storage unit that the computing engine can divide a task into, thereby splitting the task into multiple tasks. The first parameter can be any decimal greater than 0 and can be used to control the processing time of the task; the larger the first parameter, the shorter the processing time. The set time period can be 1 minute, 100 seconds, or other time intervals.

[0064] In some embodiments, the first parameter can be set manually, or the first parameter of the previous task to be processed can be used, or a fixed first parameter can be adopted.

[0065] In some embodiments, the cluster corresponding to each platform is fixed, and the big data processing system composed of the cluster and the corresponding computing engine is used to process fixed types of pending tasks. The first parameters of the pending task can be set according to the first parameters of historical tasks. For example, the first parameters of the pending task can be set based on the first parameters of the previous task (which has been completed) and the difference between the expected processing time and the actual processing time of that task.

[0066] Optionally, obtaining the first parameter corresponding to the task to be processed includes:

[0067] Based on the task type of the task to be processed, determine the first parameter corresponding to the task to be processed.

[0068] The task type can be divided according to the amount of data to be processed and / or the expected processing time, or according to the various atomic operations performed by the task and their quantity, or according to the processing operations corresponding to the task. Processing operations may include behavior analysis, call charge settlement, etc.

[0069] Specifically, a first correspondence between various task types and the first parameter can be established in advance, and then the first parameter corresponding to the task to be processed can be determined based on the task type of the task to be processed and the first correspondence.

[0070] By setting the first parameter of the task to be processed in a reasonable way, data is provided for configuring the number of executors in the future, thereby improving the accuracy of parameter configuration.

[0071] Step S203: Based on the data volume of the task to be processed, the basic parameters of the cluster corresponding to the computing engine, and the first parameter corresponding to the task to be processed, determine the number of executors corresponding to the task to be processed, so as to control each computing node to process the task to be processed through the computing engine based on the number of executors corresponding to the task to be processed.

[0072] Specifically, the amount of data of the task to be processed, the basic parameters of the computing engine, and the first parameter of the task to be processed can be input into a preset formula to obtain the number of executors corresponding to the task to be processed.

[0073] Specifically, when the computing engine is Flink, the number of containers corresponding to the task to be processed can be determined based on the amount of data of the task to be processed, the number of task slots of the container, the memory of a single data block, and the first parameter corresponding to the task to be processed. Then, the maximum parallelism is determined based on the product of the number of containers and the number of task slots of the container.

[0074] Furthermore, the memory of each executor can be determined based on the number of cores in each executor and the memory of each data block.

[0075] For example, the memory E of each executor can be: E = C × B + ΔE, where C is the number of cores of each executor, B is the size of the data block or memory, and ΔE is the set redundancy, such as 1024M, to reserve memory resources for other computations besides the assigned tasks.

[0076] Furthermore, during the processing of the task to be processed by the computing engine, the first parameter can be adjusted according to the difference between the actual processing time and the expected processing time of the completed subtasks of the task to be processed, so as to realize the dynamic adjustment of resources, such as adjusting the first parameter according to a set step size, wherein the set step size can be 0.1.

[0077] Specifically, if the actual processing time of the completed subtask of the task to be processed is less than the expected processing time, the first parameter is reduced by a set step size; if the actual processing time of the completed subtask of the task to be processed is greater than the expected processing time, the first parameter is increased by a set step size.

[0078] Furthermore, if the absolute value of the difference between the actual processing time and the expected processing time of the completed subtask of the task to be processed is greater than the preset time, the preset time can be a fixed value, such as 10s, 30s, etc., or a set percentage of the expected processing time, such as 1%, 5%, etc., then the first parameter mentioned above is adjusted according to the difference between the actual processing time and the expected processing time of the completed subtask of the task to be processed.

[0079] For example, if the task to be processed is divided into 100 tasks (subtasks), the first parameter is 1.3, 45 tasks have been completed at the current time, the actual processing time is 11 minutes, and the expected processing time is 10 minutes, then the first parameter can be increased by a set step size, that is, the first parameter can be adjusted to 1.4.

[0080] The parameter configuration method provided in this embodiment, targeting a computing engine for big data processing, automatically determines the number of executors required for a task based on the data volume of the task to be processed, the first parameter of the task, and the relevant parameters of the cluster. This enables parameter configuration at the application level using data volume, achieving automatic parameter configuration of the computing engine, improving the efficiency of parameter configuration, and ensuring high accuracy. It also reduces resource waste and improves data processing efficiency. Furthermore, by automatically configuring parameters, the complexity of big data processing based on the computing engine is reduced.

[0081] Figure 3This is a flowchart illustrating a parameter configuration method provided in another embodiment of this application. This embodiment is for the Spark computing engine. Figure 2 Based on the illustrated embodiment, further refinements are made to steps S201 to S203, and an actuator memory determination step is added after step S203, as follows: Figure 3 As shown, the parameter configuration method provided in this embodiment may include the following steps:

[0082] Step S301: Based on the data warehouse tool Hive, the task to be processed is converted into a database table.

[0083] Hive is a data warehouse tool based on Hadoop that can be used for data extraction, transformation, loading, and other operations. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop.

[0084] Specifically, data files containing structured data (such as tasks to be processed) can be mapped to a database table using the data warehouse tool Hive, and query functionality can be provided.

[0085] Step S302: Determine the amount of data for the task to be processed based on the database table.

[0086] Specifically, the size of each volume file in the database tables corresponding to the task to be processed can be counted to determine the amount or size of data in the task to be processed.

[0087] Determining the amount of data for a task to be processed using Hive improves the speed and accuracy of data volume determination.

[0088] Step S303: Obtain the memory of a single data block and the number of cores of the executor.

[0089] Furthermore, after obtaining the memory of a single data block and the number of cores of the executor, the memory of the executor corresponding to the task to be processed can be determined based on the number of cores of the executor and the memory of a single data block.

[0090] Specifically, the memory E of the executor corresponding to the task to be processed is the sum of the product of the number of cores C of the executor and the memory B of a single data block, plus the preset redundancy ΔE, i.e., E = C × B + ΔE. By setting the preset redundancy, insufficient memory is avoided when allocating memory to the executor, thus preventing a reduction in data processing efficiency.

[0091] Step S304: Determine at least one historical task based on the task type of the task to be processed.

[0092] The historical tasks are those that have already been processed, and the clusters or platforms corresponding to the tasks to be processed are the same.

[0093] In some embodiments, historical tasks may include completed tasks of the same type as the task to be processed.

[0094] Specifically, from the historical tasks that have been processed in the cluster, based on the task type of the task to be processed, we can find historical tasks with the same task type as the task to be processed, such as historical tasks with the same data volume as the task to be processed, or historical tasks with the same expected processing time as the task to be processed, or historical tasks with the same processing operations as the task to be processed, etc.

[0095] Step S305: Determine the first parameter corresponding to the task to be processed based on the first parameter corresponding to the at least one historical task, the actual processing time, and the expected processing time.

[0096] Specifically, at least one historical task can be filtered based on the difference between the actual processing time and the expected processing time to obtain various target historical tasks. The absolute value of the difference between the actual processing time and the expected processing time for each target historical task is less than a preset time. Then, based on the first parameter of each target historical task, the data volume of the task to be processed is determined.

[0097] Specifically, the first parameter of the task to be processed can be determined based on the average value of the first parameter corresponding to each target's historical task.

[0098] Furthermore, the first parameter corresponding to the task to be processed can be determined based on the first parameter corresponding to each target historical task, the data volume of each target historical task, and the data volume of the task to be processed.

[0099] Specifically, the weight coefficient of the target historical task can be determined based on the data volume of the target historical task and the data volume of the task to be processed. Then, the first parameter of the task to be processed can be determined based on the first parameter and weight coefficient corresponding to each target historical task. The closer the data volume of the target historical task is to the data volume of the task to be processed, the larger the weight coefficient of the target historical task.

[0100] By determining the first parameter of the current task to be processed based on the historical tasks that have been processed by the computing engine, the accuracy of the first parameter determination is improved, which in turn improves the accuracy of the determination of the number of subsequent executors.

[0101] Optionally, the first parameter corresponding to the task to be processed is determined based on the first parameter corresponding to the at least one historical task, the actual processing time, and the expected processing time, including:

[0102] For each historical task, the first parameter corresponding to the historical task is adjusted based on the difference between the actual processing time and the expected processing time of the historical task; the first parameter corresponding to the task to be processed is determined based on the adjusted first parameter corresponding to each historical task.

[0103] Specifically, based on a pre-established second correspondence and the difference between the actual processing time and the expected processing time of a historical task, an adjustment value for the first parameter corresponding to the historical task can be determined. Then, based on this adjustment value, the first parameter of the historical task is adjusted to obtain the adjusted first parameter corresponding to that historical task. The second correspondence describes the relationship between the difference between the actual processing time and the expected processing time and the adjustment value of the first parameter.

[0104] Furthermore, the adjustment value of the first parameter corresponding to the historical task can be determined based on the difference between the actual processing time and the expected processing time of the historical task, as well as the workload of the historical task.

[0105] Specifically, the first parameter corresponding to the task to be processed can be determined based on the average value of the adjusted first parameter corresponding to each historical task.

[0106] Specifically, the first parameter corresponding to the task to be processed can be determined based on the adjusted first parameter corresponding to each historical task and the data volume of each historical task.

[0107] Specifically, the weighting coefficient of historical tasks can be determined based on the data volume of historical tasks and the data volume of tasks to be processed. Then, the first parameter of the task to be processed can be determined based on the adjusted first parameter and weighting coefficient of each historical task. The closer the data volume of historical tasks is to the data volume of tasks to be processed, the larger the weighting coefficient of the target historical task.

[0108] By analyzing the difference between the actual processing time and the expected processing time from historical task feedback, the first parameter is adjusted. Based on the adjusted first parameter from each historical task, the first parameter of the task to be processed is determined, further improving the accuracy of the first parameter determination.

[0109] In some embodiments, steps S301, S303 and S304 may be executed in parallel, or in any order, or in partial parallel. This application does not limit the execution order of steps S301, S303 and S304.

[0110] Step S306: Determine the total task quantity corresponding to the task to be processed based on the data volume of the task to be processed and the memory of a single data block.

[0111] Specifically, the total task quantity N corresponding to the tasks to be processed is the ratio of the data quantity X of the tasks to be processed to the memory B of a single data block, that is, N = X / B.

[0112] Step S307: Determine the number of executors corresponding to the task to be processed based on the expected processing time of the task to be processed, the total number of tasks corresponding to the task to be processed, the first parameter corresponding to the task to be processed, and the number of cores of the executor corresponding to the task to be processed.

[0113] The expected processing time for the pending tasks can be set by the user or determined based on the task type and data volume.

[0114] Specifically, the formula for calculating the number Y of executors corresponding to the tasks to be processed is as follows:

[0115]

[0116] Where T is the expected processing time of the task to be processed; Z is the first parameter corresponding to the task to be processed; C is the number of cores of the executor; N is the total task quantity corresponding to the task to be processed, N = X / B; X is the data quantity of the task to be processed; and B is the memory of a single data block.

[0117] Based on the number of executors corresponding to the tasks to be processed and the memory of the executors corresponding to the tasks to be processed, the computing engine controls each computing node of the cluster to process the tasks to be processed.

[0118] Furthermore, after determining the number of executors corresponding to or required for the task to be processed and the memory required by the executors, the memory of the driver can be set, thereby obtaining the various parameters of the Spark computing engine corresponding to the task to be processed. Then, based on the various parameters of the Spark computing engine set for the task to be processed, the computing engine controls the various computing nodes in the corresponding cluster to process the task to be processed, so as to obtain the processing result.

[0119] Furthermore, the processing results corresponding to the tasks to be processed can be stored in the cluster, and the processing results can be displayed through one or more computing nodes in the cluster. The processing results can also be sent to a designated terminal, such as a user terminal.

[0120] In this embodiment, for a task to be processed using the Spark computing engine, the data volume of the task is obtained through the Hive data warehouse tool. Based on the data volume of the task and the memory of a single data block, the memory of the executor corresponding to the task is determined. Based on the first parameters, actual processing time, and expected processing time of a completed historical task of the same type as the task to be processed, the first parameters of the task to be processed are determined, improving the accuracy of the determination of the first parameters. Furthermore, based on the data volume, expected processing time, first parameters, memory of a single data block corresponding to the Spark computing engine, and the number of cores of the executor, the number of executors corresponding to the task to be processed is automatically determined, thereby realizing the automatic configuration of the number of executors and the memory of the executors for the task to be processed, improving the efficiency and accuracy of parameter configuration, reducing resource waste, and improving the efficiency of data processing.

[0121] Figure 4 This is a schematic diagram of the structure of a parameter configuration device provided in one embodiment of this application, as shown below. Figure 4 As shown, the device includes: a data acquisition module 410, a first parameter acquisition module 420, and an actuator quantity determination module 430.

[0122] The data volume acquisition module 410 is used to acquire the data volume of the task to be processed corresponding to the computing engine and the basic parameters of the cluster corresponding to the computing engine, wherein the cluster includes multiple computing nodes; the first parameter acquisition module 420 is used to acquire the first parameter corresponding to the task to be processed, wherein the first parameter represents the number of data blocks that each core can execute within a set time period; the executor number determination module 430 is used to determine the number of executors corresponding to the task to be processed based on the data volume of the task to be processed, the basic parameters of the cluster corresponding to the computing engine, and the first parameter corresponding to the task to be processed, so as to control each computing node to process the task to be processed based on the number of executors corresponding to the task to be processed through the computing engine.

[0123] Optional, the data acquisition module 410 is specifically used for:

[0124] Using data warehouse tools, the tasks to be processed are converted into database tables; based on the database tables, the amount of data for the tasks to be processed is determined.

[0125] Optionally, the first parameter acquisition module 420 is specifically used for:

[0126] Based on the task type of the task to be processed, determine the first parameter corresponding to the task to be processed.

[0127] Optionally, the first parameter acquisition module 420 includes:

[0128] The historical task determination unit is used to determine at least one historical task based on the task type of the task to be processed, wherein the historical task is the same as the cluster corresponding to the task to be processed; the first parameter determination unit is used to determine the first parameter corresponding to the task to be processed based on the first parameter corresponding to the at least one historical task, the actual processing time, and the expected processing time.

[0129] Optionally, the first parameter determination unit is specifically used for:

[0130] For each historical task, the first parameter corresponding to the historical task is adjusted based on the difference between the actual processing time and the expected processing time of the historical task; the first parameter corresponding to the task to be processed is determined based on the adjusted first parameter corresponding to each historical task.

[0131] Optionally, the actuator quantity determination module 430 is specifically used for:

[0132] The total number of tasks corresponding to the tasks to be processed is determined based on the data volume of the tasks to be processed and the memory of a single data block; the number of executors corresponding to the tasks to be processed is determined based on the expected processing time of the tasks to be processed, the total number of tasks corresponding to the tasks to be processed, the first parameter corresponding to the tasks to be processed, and the number of cores of the executors corresponding to the tasks to be processed.

[0133] Optionally, the device further includes:

[0134] The executor memory determination module is used to determine the memory of the executor corresponding to the task to be processed based on the number of cores of the executor and the memory of a single data block.

[0135] The parameter configuration device provided in this application embodiment can execute the parameter configuration method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0136] Figure 5 This is a schematic diagram of the structure of a parameter configuration device provided in one embodiment of this application, as shown below. Figure 5 As shown, the parameter configuration device includes: memory 510, processor 520, and computer program.

[0137] The computer program is stored in memory 510 and configured to be executed by processor 520 to implement this application. Figures 2 to 3 The parameter configuration method provided in any of the corresponding embodiments.

[0138] The memory 510 and the processor 520 are connected via a bus 530.

[0139] For relevant instructions, please refer to the corresponding text. Figures 2 to 3 The relevant descriptions and effects corresponding to the steps will be understood, and will not be elaborated on here.

[0140] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a parameter configuration device, the parameter configuration device is able to perform the above-described parameter configuration method.

[0141] For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0142] This application also provides a computer program product, including an executable computer program stored in a readable storage medium. At least one processor of the parameter configuration device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the parameter configuration device to implement the parameter configuration methods provided in the various embodiments described above.

[0143] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0144] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A parameter configuration method, characterized in that, The method includes: The data volume of the task to be processed corresponding to the computing engine and the basic parameters of the cluster corresponding to the computing engine are obtained. The cluster includes multiple computing nodes, and the basic parameters include the memory of a single data block and the number of cores of the executor. Based on the task type of the task to be processed, obtain the first parameter corresponding to the task to be processed, wherein the first parameter represents the number of data blocks that can be executed by each core within a set time period, and the task type can be divided according to the data flow and / or expected processing time of the task to be processed; The total number of tasks corresponding to the tasks to be processed is determined based on the amount of data in the tasks to be processed and the memory of a single data block. Based on the expected processing time of the task to be processed, the total number of tasks corresponding to the task to be processed, the first parameter corresponding to the task to be processed, and the number of cores of the executor corresponding to the task to be processed, the number of executors corresponding to the task to be processed is determined, so as to control each computing node to process the task to be processed based on the number of executors corresponding to the task to be processed through the computing engine. During the processing of the task to be processed based on the computing engine: If the actual processing time of the subtasks that have been completed for the task to be processed is less than the expected processing time, then the first parameter is reduced by a set step size. If the actual processing time of the subtasks that have been completed for the task to be processed is greater than the expected processing time, then the first parameter is increased by a set step size.

2. The method according to claim 1, characterized in that, Obtain the amount of data for the tasks to be processed corresponding to the computing engine, including: Based on data warehouse tools, the tasks to be processed are converted into database tables; The amount of data for the task to be processed is determined based on the database table.

3. The method according to claim 1, characterized in that, Based on the task type of the task to be processed, obtain the first parameter corresponding to the task to be processed, including: Based on the task type of the task to be processed, at least one historical task is determined, wherein the historical task is the same as the cluster corresponding to the task to be processed. The first parameter corresponding to the task to be processed is determined based on the first parameter corresponding to the at least one historical task, the actual processing time, and the expected processing time.

4. The method according to claim 3, characterized in that, Based on the first parameter corresponding to the at least one historical task, the actual processing time, and the expected processing time, the first parameter corresponding to the task to be processed is determined, including: For each historical task, the first parameter corresponding to the historical task is adjusted based on the difference between the actual processing time and the expected processing time of the historical task. The first parameter corresponding to the task to be processed is determined based on the adjusted first parameter corresponding to each historical task.

5. The method according to claim 1, characterized in that, The method further includes: The memory of the executor corresponding to the task to be processed is determined based on the number of cores of the executor and the memory of a single data block.

6. The method according to any one of claims 1-5, characterized in that, The task to be processed is a data file generated based on a distributed file system.

7. A parameter configuration device, characterized in that, The device includes: The data volume acquisition module is used to acquire the data volume of the task to be processed corresponding to the computing engine and to acquire the basic parameters of the cluster corresponding to the computing engine. The cluster includes multiple computing nodes, and the basic parameters include the memory of a single data block and the number of cores of the executor. The first parameter acquisition module is used to acquire the first parameter corresponding to the task to be processed according to the task type of the task to be processed. The first parameter represents the number of data blocks that each core can execute within a set time period. The task type can be divided according to the data flow and / or expected processing time of the task to be processed. An executor quantity determination module is used to determine the total task quantity corresponding to the task to be processed based on the data volume of the task to be processed and the memory of a single data block; and to determine the number of executors corresponding to the task to be processed based on the expected processing time of the task to be processed, the total task quantity corresponding to the task to be processed, the first parameter corresponding to the task to be processed, and the number of cores of the executor corresponding to the task to be processed, so as to control each computing node to process the task to be processed through the computing engine based on the number of executors corresponding to the task to be processed. The first parameter acquisition module is further configured to: During the processing of the task to be processed based on the computing engine: If the actual processing time of the subtasks that have been completed for the task to be processed is less than the expected processing time, then the first parameter is reduced by a set step size. If the actual processing time of the subtasks that have been completed for the task to be processed is greater than the expected processing time, then the first parameter is increased by a set step size.

8. A parameter configuration device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the parameter configuration method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the parameter configuration method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images