Parameter configuration method and device, computing equipment, system and readable storage medium

Abstract

The invention provides a parameter configuration method and device, computing equipment, a system and a readable storage medium, and relates to the field of big data. The method comprises the following steps: aiming at a to-be-executed task needing to run on a big data platform, firstly determining a nearest task closest to a coding distance of the to-be-executed task in at least one executed task; and then, setting the parameter configuration of the to-be-executed task as the parameter configuration of the nearest neighbor task in the big data platform, so that the big data platform executes the to-be-executed task by using the configuration parameter of the nearest neighbor task. Thus, the recommended parameter configuration of the to-be-executed task is obtained by using the operation information of the historical task of the big data platform based on the rapid matching of the to-be-executed task and the executed task, and the time consumed for parameter configuration optimization of the to-be-executed task can be reduced while the parameters enabling the execution efficiency of the to-be-executed task to be relatively high can be obtained.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a parameter configuration method, apparatus, computing device, system and readable storage medium. Background Art

[0002] Big data platforms are tools used to manage, process and analyze large amounts of data, such as big data analysis (BDA) platforms. When performing tasks such as data management and analysis, big data platforms need to adjust the relevant parameter configurations of tasks. The different values ​​of these parameter configurations have a great impact on the overall performance of the big data platform. Some suitable parameter configurations can greatly shorten the running time of tasks.

[0003] In addition to manual tuning of the parameter configuration of the big data platform by staff based on prior knowledge, there are also automatic parameter tuning methods such as offline tuning based on neural network models and online tuning based on the coefficient of variation (CV) to screen the parameter set that affects the performance of the big data platform for iterative optimization. However, the above traditional offline tuning or online tuning methods have the problem of taking a long time to determine the parameter configuration. Summary of the invention

[0004] The embodiments of the present application provide a parameter configuration method, apparatus, computing device, system and readable storage medium, which can solve the problem of time-consuming parameter configuration and tuning of a big data platform.

[0005] In a first aspect, a parameter configuration method is provided. The parameter configuration method includes: for a task to be executed that needs to be run on a big data platform, determining a nearest task that is closest to the coding distance of the task to be executed in at least one executed task, and then using the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, so that when the big data platform executes the task to be executed, the parameters of the task to be executed are set according to the parameter configuration of the nearest task, and the task to be executed is executed.

[0006] Based on the above parameter configuration method, the parameter configuration of the nearest neighboring task with the closest encoding distance to the task to be executed in at least one executed task is used as the recommended parameter configuration of the task to be executed, without the need to conduct time-consuming training of the neural network model with a complex architecture, nor is there any need to perform parameter sensitivity screening and iterative optimization each time the parameter configuration recommendation is made. Therefore, the method is based on the rapid matching of the task to be executed and the executed tasks, and uses the operation information of the historical tasks of the big data platform to obtain the recommended parameter configuration of the task to be executed. While being able to obtain parameters that make the execution efficiency of the task to be executed higher, it reduces the time spent on optimizing the parameter configuration of the task to be executed.

[0007] This application does not limit the execution subject of the parameter configuration method. Optionally, the execution subject of the parameter configuration method provided in this application can be a computing device (or computing node) used for big data analysis and processing in a big data system, or a computing device connected to a big data system.

[0008] For example, when the parameter configuration method is executed by a computing node within a big data system, the computing node can determine that the parameter configuration of the task to be executed in the big data platform is the parameter configuration of the nearest task, and then execute the task to be executed based on the parameter configuration of the nearest task in the big data platform.

[0009] For another example, when the parameter configuration method is executed by a computing node outside the big data system, the computing node can send the parameter configuration to the computing node used to execute the task on the big data platform after determining that the parameter configuration of the task to be executed in the big data platform is the parameter configuration of the nearest task, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

[0010] As a possible implementation, the encoding distance refers to the similarity between two tasks. The computing node determines the nearest neighbor task as the executed task with the highest similarity to the task to be executed among at least one executed task.

[0011] Optionally, the similarity in this embodiment may be represented by cosine similarity, Manhattan distance, Chebyshev distance, simple matching coefficient, etc.

[0012] Taking cosine similarity as an example, the computing node can convert the task to be executed and at least one executed task into vector form respectively, and then determine the executed task with the highest cosine similarity with the task to be executed in vector form among the at least one executed task in vector form, and use the executed task as the nearest task to be executed.

[0013] In this way, the computing node only needs to calculate the similarity such as cosine similarity between tasks to determine the nearest task to be executed from at least one executed task. The overall calculation amount is small, which improves the overall efficiency of parameter configuration.

[0014] As a possible implementation, at least one executed task includes the historical data of each executed task. The historical data of each executed task includes the historical record item corresponding to the task when it was previously executed in the big data platform, and the historical record item includes parameter configuration and execution time. The computing node determines the historical record item with the shortest execution time from the historical data of the nearest task, and determines the parameter configuration of the nearest task as the parameter configuration of the historical record item with the shortest execution time.

[0015] In this way, the execution time of multiple executions of historical tasks is used as a reference factor for selecting parameter configurations, and the parameter configuration that makes the execution time of the nearest task shortest is selected from the historical records of the nearest task as the recommended parameter configuration for the task to be executed. This parameter configuration can enable the nearest task to be executed efficiently. Since the nearest task has a high similarity with the task to be executed, this parameter configuration can also improve the execution efficiency of the task to be executed, thereby shortening its execution time.

[0016] As a possible implementation method, after the computing node executes the task to be executed based on the parameter configuration, or after sending the parameter configuration to the big data platform so that the big data platform executes the task to be executed based on the parameter configuration, the computing node can also update the historical data of at least one executed task according to the execution result of the task to be executed.

[0017] The execution result of the task to be executed includes the execution time and parameter configuration of the task, and the data of at least one executed task includes the execution time and parameter configuration of each executed task.

[0018] In this way, the historical data of at least one executed task is continuously updated, ensuring the data breadth of the historical data, so that the computing node can match the nearest task with a closer encoding distance in the historical data as much as possible, thereby improving the degree to which the parameter configuration is suitable for the task to be executed.

[0019] Optionally, the computing node adds the execution result of the task to be executed, i.e., the execution duration and parameter configuration, as historical data to the data of at least one executed task, and then determines the optimal parameter configuration in the data of at least one executed task after the added data, and then adds the execution result of the nearest task based on the optimal parameter configuration as historical data to the data of at least one executed task.

[0020] This application does not limit the method for determining the optimal parameter configuration. For example, the computing node can determine the optimal parameter configuration using a neural network model. The input of the neural network model is the feature vector and parameter configuration of each executed task, and the output of the neural network model is the execution time of each executed task. The above-mentioned neural network model can be a neural network model constructed based on a regression model.

[0021] In this way, by first recommending parameter configuration and then updating historical data, the two steps of parameter configuration recommendation and historical data update are decoupled, which can not only realize the iterative update of the neural network model, but also avoid the influence of the model update process on the computing node's determination of the recommended parameter configuration for the task to be executed.

[0022] Optionally, the training set of the above-mentioned neural network model can be historical data, i.e., data of at least one executed task, obtained after Bayesian optimization (BO) based on conformal prediction (CP). In this way, statistical factors are introduced into the Bayesian optimization process through conformal prediction, and the parameter configuration in the historical data is subjected to Bayesian optimization with a high confidence level to sample the training set. Thus, while simplifying the training set to increase the training speed of the neural network model, the deviation in the sampling process of the training set is reduced, the representation ability of the training set is guaranteed, and the neural network model is prevented from falling into a local optimum, thereby ensuring the accuracy and universality of the neural network model.

[0023] In a second aspect, a parameter configuration device is provided, which includes modules for executing the data processing method in the first aspect or any possible design of the first aspect.

[0024] As a possible implementation, the parameter configuration device includes a parameter recommendation module. The parameter recommendation module is used to: determine the nearest neighboring task with the closest coding distance to the task to be executed in at least one executed task, and use the parameter configuration of the nearest neighboring task as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest neighboring task.

[0025] Optionally, the parameter recommendation module is specifically used to: determine that the nearest neighbor task is the executed task with the highest similarity to the task to be executed among at least one executed task.

[0026] The above similarity is cosine similarity.

[0027] For example, the parameter recommendation module is specifically used to: convert the task to be executed and at least one executed task into vector form respectively, and determine the nearest neighbor task as the executed task whose vector form has the highest cosine similarity with the vector form of the task to be executed in at least one executed task.

[0028] As a possible implementation, the parameter recommendation module is further used to: determine the historical record item with the shortest execution time from the historical data of the nearest task, and determine the parameter configuration of the nearest task as the parameter configuration of the historical record item with the shortest execution time.

[0029] As a possible implementation method, the execution subject of the parameter configuration method is used to execute tasks on the big data platform, and the parameter recommendation module is specifically used to: execute the task to be executed based on the parameter configuration of the nearest task in the big data platform.

[0030] As a possible implementation method, the execution subject of the parameter configuration method is not used to execute the tasks of the big data platform, and the parameter recommendation module is specifically used to: use the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, and send the parameter configuration of the nearest task to the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

[0031] As a possible implementation manner, the parameter configuration device further includes a historical task tuning module, which is used to update data of at least one executed task according to an execution result of the task to be executed.

[0032] As a possible implementation method, the execution result includes the execution time and parameter configuration of the task. The historical task tuning module is specifically used to: add the execution time and parameter configuration of the task to be executed to the data of at least one executed task; determine the optimal parameter configuration in the data of at least one executed task after adding the execution time and parameter configuration of the task to be executed; and add the execution result of the nearest task based on the optimal parameter configuration to the data of at least one executed task.

[0033] Optionally, the historical task tuning module is specifically used to: determine the optimal parameter configuration using a neural network model, the input of the neural network model is the feature vector and parameter configuration of each executed task, and the output is the execution time of each executed task.

[0034] Optionally, the neural network model is a regression model.

[0035] Optionally, the training set of the neural network model is obtained through Bayesian optimization of conformal prediction based on data of at least one executed task.

[0036] In a third aspect, a computing device is provided, which includes at least one processor and a memory, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions, the operating steps of the parameter configuration method in the first aspect or any possible implementation of the first aspect are executed.

[0037] In a fourth aspect, a big data system is provided, which includes a first computing device and a second computing device, the first computing device being used to execute the operating steps of the parameter configuration method in the first aspect or any possible implementation of the first aspect, and the second computing device being used to execute the task to be executed based on the parameter configuration of the nearest task.

[0038] In addition, the technical effects of the parameter configuration device described in the second aspect, the technical effects of the computing equipment described in the third aspect, and the technical effects of the big data system described in the fourth aspect can refer to the technical effects of the parameter configuration method described in the first aspect, and will not be repeated here.

[0039] In a fifth aspect, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a computing device, the computing device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0040] In a sixth aspect, a computer program product is provided. When the computer program product is run on a computer, the computing device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0041] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the structure of a neural network provided for this application;

[0043] Figure 2 A schematic diagram of the architecture of a big data system provided for this application;

[0044] Figure 3 A schematic diagram of the structure of a computing cluster provided for this application;

[0045] Figure 4 A flow chart of a parameter configuration method provided in this application;

[0046] Figure 5 A schematic diagram of a process for determining the nearest task step provided for this application;

[0047] Figure 6 A flowchart of the historical data optimization steps provided in this application;

[0048] Figure 7 A schematic diagram of the structure of a parameter configuration device provided in this application;

[0049] Figure 8 A schematic diagram of the structure of a computing device and system provided in this application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b and c can be single or multiple.

[0051] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0052] The parameter configuration method provided in the embodiment of the present application can be applied to the big data analysis platform scenario in the big data field. The following is a brief introduction to the technologies that may be involved in this application.

[0053] (1) Neural Network

[0054] A neural network can be composed of neurons, and a neuron can be a s The output of this operation unit satisfies the following formula:

[0055]

[0056] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s, b is the bias of the neuron. f is the activation function of the neuron, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neuron into the output signal. The output signal of the activation function can be used as the input of the next layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neurons mentioned above, that is, the output of one neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neurons. The weight characterizes the strength of the connection between different neurons. The weight determines the influence of the input on the output. A weight close to 0 means that changing the input does not change the output. A negative weight means that increasing the input reduces the output.

[0057] like Figure 1 , which is a schematic diagram of the structure of a neural network provided in an embodiment of the present application. The neural network 100 includes N processing layers, where N is an integer greater than or equal to 3. The first layer of the neural network 100 is the input layer 110, which is responsible for receiving the input signal x i The last layer of the neural network 100 is the output layer 130, which is responsible for outputting the processing result h of the neural network. i The other layers except the first layer and the last layer are intermediate layers 140, which together constitute hidden layers 120. Each intermediate layer 140 in the hidden layer 120 can both receive input signals and output signals. The hidden layer 120 is responsible for the processing of input signals. Each layer represents a logical level of signal processing. Through multiple layers, data signals can be processed by multiple levels of logic.

[0058] In some feasible embodiments, the input signal of the neural network can be a signal in various forms such as a video signal, a voice signal, a text signal, an image signal or a temperature signal. The input signal of the neural network also includes various other computer-processable engineering signals, which are not listed here one by one.

[0059] (2) Big Data

[0060] With the development of the Internet, the Internet of Things, network bandwidth, smart terminals, cloud computing and other services, the data types and data scales have developed at an unprecedented speed, and big data has emerged. Data has changed from a single processing object to a basic resource. Big data is a collection of data that cannot be captured, managed and processed by conventional software tools within a period of time.

[0061] The sources of big data can include management systems, web systems, physical systems, and scientific experiment systems. Management systems include transaction processing systems and office automation systems. Web systems include Internet systems, such as search engines. Physical systems refer to systems about physical objects and physical processes, such as real-time monitoring, real-time detection, or production scheduling. Scientific experiment systems refer to physical systems based on experimental environments.

[0062] Big data can be classified according to data structure characteristics, including structured data, unstructured data and semi-structured data. Structured data refers to data represented by a unified structure, such as data stored in a database such as a financial system, enterprise resource planning system or customer relationship management system. Unstructured data refers to data with variable field lengths that are not easily expressed using structured data, such as audio, video or image data. Semi-structured data is data between structured data and unstructured data. Semi-structured data belongs to the same category and has different attributes, such as data in a mail system.

[0063] Since the large amount of data contained in big data has correlations, data analysis methods, models or tools are used to analyze big data, explore the data relationships in big data, and use the data relationships in big data to make predictions or decisions. For example, by analyzing user shopping trend data, users can be pushed items that they may buy to improve their shopping experience. Therefore, big data has the characteristics of large data volume, fast data growth rate, diverse data types and high utilization value.

[0064] (3) Big Data Platform

[0065] Big data platforms are tools used to manage, process, and analyze large amounts of data, such as big data analysis (BDA) platforms. Take Spark as an example. It is one of the most widely used big data analysis platforms. There are many adjustable parameter configuration items on Spark. The different values ​​of these parameter configurations have a great impact on the overall performance of the Spark system. A set of appropriate parameter configurations can greatly shorten the running time of Spark tasks. However, the optimal parameter configurations for each Spark task are different, and most Spark tasks are difficult to achieve good performance under the default parameter configuration. Other big data platforms (such as Hive, Hadoop, etc.) also have similar phenomena.

[0066] Therefore, recommending appropriate parameter configurations for tasks on the big data platform can increase the running speed of the tasks, thereby improving the overall operating efficiency of the big data platform.

[0067] However, the design of parameter tuning systems on big data platforms is difficult. The main reasons are as follows: First, the optimal configuration of a task is difficult to reuse, which means that the optimal configuration obtained after tuning a task (or application) is difficult to migrate to another task (or application), and sometimes even requires tuning from scratch; second, there are complex relationships between different configuration items. Even for the same task (or application), the optimal parameters obtained under a single configuration item are difficult to apply to multiple configuration items; third, the tuning parameter space is very complex. Taking Spark as an example, among the parameters that can be tuned in Spark, there are discrete ones (such as "executor.cores" whose values ​​can only be integers) and continuous ones (such as "executor.memory"). These parameters constitute a high-dimensional space where continuous and discrete values ​​coexist, which adds difficulty to operations such as sampling and finding the maximum value.

[0068] The parameter tuning of traditional big data platforms is achieved through manual tuning methods, that is, professionals set the value of each configuration item based on prior knowledge through repeated experiments. However, this manual tuning method has very few tunable configuration items, and the tuning effect is not good enough, and it also has a high labor cost. Therefore, automatic parameter tuning methods have also emerged in the industry. Most of these methods are divided into two methods: offline tuning and online tuning. The offline tuning method requires the establishment of an additional offline system based on the actual system, and it takes a lot of time to collect data and train complex models in advance, so the time and cost are relatively high. The online tuning method requires running the tasks to be tuned on the actual system multiple times. It takes a long time to obtain the optimal parameter configuration, and the performance fluctuations during the operation are difficult to control, and the risk of new task failure is high.

[0069] Next, we will use Schemes 1 to 3 as examples to explain the commonly used automatic parameter tuning methods.

[0070] Solution 1 includes two stages: offline training and online recommendation.

[0071] The offline training phase mainly includes the following steps a-c:

[0072] a. Collect data: Input some Spark applications, run the applications on a small-scale data set and collect data, and obtain the directed acyclic graph (DAG) scheduling diagram and stage-level code snippets corresponding to the input Spark applications.

[0073] b. Spark application feature extraction: The stage-level code snippets are converted into the embedding matrix of the vocabulary, and the convolutional neural network (CNN) model is used for feature encoding to obtain the feature vector representing the code information; the DAG scheduling graph is converted into the embedding matrix of the node, and the graph convolutional neural network (GCN) model is used for feature encoding to obtain the feature vector representing the scheduler.

[0074] c. Training model: concatenate the two vectors obtained in step b with the three vectors representing data features, environment features, and configuration features to obtain a feature vector representing the Spark task. Based on this feature vector, train a multi-layer perceptron to predict the running time of the Spark task. At the same time, for each parameter configuration item that needs to be tuned, train a regression model to map the input Spark task and feature vector to the value of a parameter configuration item.

[0075] The online recommendation stage mainly includes the following steps d-g:

[0076] d. Input a Spark task, run it on a large-scale data set and collect data, and obtain the DAG scheduling graph and stage-level code snippet corresponding to the input Spark task.

[0077] e. Adaptively generate candidate parameter configurations: For each parameter configuration item that needs to be tuned, construct a sampling interval. Finally, sample candidate parameter configurations in the high-dimensional space composed of the sampling intervals of all parameter configuration items.

[0078] f. Recommend optimal parameter configuration: Under each candidate parameter configuration sampled in step e, use the NECS model to predict the application's running time, and select the parameter configuration with the shortest predicted running time as the optimal parameter configuration.

[0079] g. Adaptive model update: Run the Spark application under the optimal parameter configuration recommended in the previous step to collect data. After collecting a certain amount of data, combine the data collected in the online recommendation phase with the data set in the offline training phase, and use adversarial learning to learn indistinguishable representations of training instances and test instances to update the NECS model.

[0080] Solution 2 includes three steps: query selection, parameter selection, and Bayesian optimization.

[0081] The specific process of the query selection step includes: using the coefficient of variation (CV), that is, the ratio of the standard deviation to the mean, to represent the sensitivity of the query to different parameter configurations. Generally speaking, a higher coefficient of variation indicates that the query is more sensitive to parameter configuration adjustments. When collecting training samples, in order to save time, some more effective queries can be selected from the Spark structured query language server database (SQL) application and other queries can be deleted. To this end, a suitable threshold for the coefficient of variation is required to determine which queries to retain. This solution adopts a dynamic method to determine its threshold and divides all coefficients of variation into three non-overlapping partitions. Subsequently, queries in the first interval are defined as parameter configuration insensitive queries, and the remaining queries are defined as parameter configuration sensitive queries. Finally, the insensitive queries are removed, and only sensitive queries are retained to replace all original queries for subsequent operations, which can greatly save time costs.

[0082] The specific process of the parameter selection step includes: combining feature selection and feature extraction to screen out the parameter set that most affects the performance of the SparkSQL task. This process is based on the running data of the new application. First, some unimportant parameter configuration items are removed through the Spearman correlation coefficient (SCC), and then the important parameter configuration items are screened using kernel-based principle component analysis (KPCA). KPCA is a powerful nonlinear feature extractor that can not only reduce the feature dimension by calculating the eigenvectors of the covariance matrix of the original input, but also extract nonlinear information by using the kernel method, thereby achieving dimensionality reduction of parameters. After this process, the parameters that have a greater impact on the execution time of the Spark SQL task can be obtained.

[0083] The specific process of the Bayesian optimization step includes: This step designs a Gaussian process (GP) that takes into account the data size based on the traditional Bayesian optimization, so that it can adapt to the changes in the size of the input data. Specifically, the Gaussian process is used to model the execution time of Spark SQL tasks with specific configurations as a function of parameter configuration values ​​and input data size. At the same time, it uses the Markov chain Monte Carlo method to reduce the number of required iterations, which to some extent solves the problem that traditional Bayesian optimization is difficult to run in an online environment.

[0084] Solution 3 includes three steps: model training, configuration recommendation, and similarity judgment.

[0085] The specific process of the model training step includes: sampling N sets of parameter configurations for a given new type of task and executing the task based on the parameter configuration, using the cost information of task execution to build a cost-based performance model. Each type of task corresponds to a performance model, which will be gradually updated during the configuration recommendation process.

[0086] The specific process of the configuration recommendation step includes: searching for a set of better parameter configurations in the parameter configuration space based on the trained performance model. Specifically, the simulated annealing algorithm (SAA) is used during the search, which can not only improve the search efficiency but also avoid falling into the local optimum to a certain extent. The search process is continuously iterated. When the parameter configuration obtained by the search meets the expectations, the tuning is terminated and the parameter configuration is recommended to the user.

[0087] The specific process of the similarity determination step includes: inputting a new task of unknown type, which is first executed once with the default configuration, and then based on the runtime data, searching the database for tasks of similar types. If there are, call the performance model corresponding to the similar type of task, and directly recommend parameter configuration, saving the time of model training; if not, treat the running task as a new type, use the model training module to train a new performance model, and then recommend parameter configuration.

[0088] The above solution 1 requires the support of offline systems, and the overall architecture is relatively complex. It takes a lot of time to collect training samples in advance, resulting in low overall parameter tuning efficiency and long time consumption. The above solution 2 uses the coefficient of variation of the execution time of Spark SQL tasks to determine the sensitivity of queries. There is no theoretical support, and the configuration-sensitive queries obtained in the end are not necessarily correct. In addition, each time a new task is tuned, it needs to start from scratch, and historical data cannot be used. It is necessary to repeat the task with different parameters to obtain a set of parameter configurations, which takes a long time. The performance model of each type of task in the above solution 3 only considers the configuration item value and the amount of task data, and does not consider the characteristic information of different tasks. Therefore, the prediction of performance is inaccurate. At the same time, the convergence process of the simulated annealing algorithm is relatively slow, so it takes a long time to recommend a good set of parameter configurations for new tasks.

[0089] It can be seen that traditional automatic parameter tuning methods have the problems of low parameter tuning efficiency and taking too long to recommend parameter configurations for new tasks.

[0090] The present application provides a parameter configuration method, in particular, a parameter configuration method for "recommending parameter configuration based on the similarity between historical tasks and tasks to be executed on a big data platform". The parameter configuration method includes: for a task to be executed that needs to be executed on a big data platform, determining the nearest task with the closest coding distance to the task to be executed from at least one executed task, and then using the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, so that when the big data platform executes the task to be executed, the parameters of the task to be executed are set according to the parameter configuration of the nearest task, and the task to be executed is executed.

[0091] Based on the parameter configuration method provided by the present application, the parameter configuration of the nearest neighboring task with the closest coding distance to the task to be executed in at least one executed task is used as the recommended parameter configuration of the task to be executed. Compared with the above-mentioned schemes 1 to 3, there is no need to conduct time-consuming training of the neural network model with complex architecture, nor is there any need to perform parameter sensitivity screening and iterative optimization every time the parameter configuration recommendation is made. Therefore, the parameter configuration method provided by the present application is based on the rapid matching of the task to be executed with the executed tasks in the historical data, and uses the operation information of the historical tasks executed by the big data platform to match to obtain the recommended parameter configuration of the task to be executed. While being able to obtain parameters that make the execution efficiency of the task to be executed higher, it reduces the time-consuming optimization of the parameter configuration of the task to be executed, and improves the overall operation efficiency of the parameter configuration big data platform.

[0092] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0093] Since the amount of big data is very large, a single computing node cannot meet the computing needs of big data. Usually, a distributed file system (DFS) is used to store and manage big data, and multiple computing nodes in a computing cluster are used to process big data services. Figure 2 This is a schematic diagram of the architecture of a big data system provided in an embodiment of the present application. Figure 2 As shown, the big data system 200 includes a client 210 , a computing cluster 220 , and a storage cluster 230 .

[0094] The storage cluster 230 includes at least two storage nodes 231. A storage node 231 includes one or more controllers, a network card and multiple hard disks. The hard disk is used to store data. The hard disk can be a magnetic disk or other types of storage media, such as a solid-state hard disk or a shingled magnetic recording hard disk. The network card is used to communicate with the computing node 221 included in the computing cluster 220. The controller is used to write data to the hard disk or read data from the hard disk according to the read / write data request sent by the computing node 221. In the process of reading and writing data, the controller needs to convert the address carried in the read / write data request into an address that the hard disk can recognize.

[0095] The client 210 communicates with the computing cluster 220 and the storage cluster 230 through the network 240. For example, the client 210 sends a service request of a big data service to the computing cluster 220 through the network 240, requesting the computing cluster 220 to perform distributed processing on the service data included in the service request. The network 240 may refer to an internal network of an enterprise (such as a local area network (LAN)) or the Internet.

[0096] In some embodiments, the storage cluster 230 stores and manages big data based on the distributed file system 232 and the distributed database 233. The client 210 is installed with a client program 211. The client 210 runs the client program 211 to display a user interface (UI). The user 250 operates the user interface to access the distributed file system 232 and the distributed database 233 to obtain data, and instructs the computing cluster 220 to process the big data business. The client 210 refers to a computer connected to the network 240, which can also be called a workstation. Different clients can share resources on the network (such as computing resources and storage resources). In other embodiments, the system administrator 260 can access the distributed file system 232 and the distributed database 233 through the client 210 to configure system information, such as the deployment information and storage strategy of the storage medium configured for the computing node or storage node provided in the embodiment of the present application.

[0097] It should be noted that the storage medium may be a memory, which is a memory device for storing programs and various data. The larger the capacity of the memory, the slower the access speed. Conversely, the smaller the capacity of the memory, the faster the access speed. The access speed refers to the data transmission speed when writing data to or reading data from the memory. The access speed may also be referred to as the read and write speed. The memory can be divided into different levels according to the storage capacity and access speed.

[0098] The computing cluster 220 includes at least two computing nodes 221, and the computing nodes 221 can communicate with each other. The computing node 221 is a computing device, such as a server, a desktop computer, or a controller of a storage array. The big data business submitted by the client 210 can be called a job. A job can be divided into multiple tasks, and multiple computing nodes execute multiple tasks in parallel. The completion of all tasks marks the completion of a job. A task is generally a processing process for a part of data or a stage in a job, and all tasks are scheduled to be completed in parallel or serially. In some embodiments, the computing cluster 220 performs distributed processing on big data services based on the map-reduce (MapReduce) model 234. The MapReduce model is a distributed programming model, which decomposes the big data business into multiple tasks, and multiple computing nodes 221 execute the tasks. In some embodiments, such as Figure 3 As shown, the computing cluster 220 includes a control node 222 and at least two computing nodes 221. The control node and the computing node can be independent physical devices. In this case, the control node can also be called a control device or a naming node, and the computing node can be called a computing device or a data node. The control node 222 is used to manage the namespace of the distributed file system and the access of the client 210 to the distributed file system. In addition, the control node 222 indicates the computing node that executes the task. The computing node executes the task based on the MapReduce model 234. Optionally, the storage cluster 230 can also process the data according to the MapReduce model 234 and then store it.

[0099] As a possible implementation method, the parameter configuration method provided in the present application is executed by the control node 222, and the control node 222 is used to determine the nearest task with the closest coding distance to the task to be executed in at least one executed task, and send the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform to the computing node 221. The computing node 221 is a node for executing the task to be executed based on the received parameter configuration.

[0100] As another possible implementation method, the parameter configuration method provided in the present application is executed by the computing node 221, and the computing node 221 is used to determine the nearest task with the closest coding distance to the task to be executed in at least one executed task, and use the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, and then execute the task to be executed based on the parameter configuration.

[0101] The parameter configuration method provided in this application is described in detail below with reference to the accompanying drawings.

[0102] Figure 4A flow chart of a parameter configuration method provided by the present application. Here, the client 210 and the computing cluster 220 are used as examples for explanation. The computing cluster 220 includes a control node 222 and at least two computing nodes 221. The control node 222 is used to recommend parameter configurations of tasks to be executed to the computing node 221, and to control the computing node 221 to perform distributed processing of big data. Figure 4 As shown, the method includes the following steps 410-460.

[0103] Step 410 : The control node 222 receives a service request for a big data service sent by the client 210 .

[0104] As a possible example, the client 210 responds to a user operation and sends a service request for a big data service to the control node 222. The user operation may refer to an operation in which a user operates a big data user interface to submit a big data service. The control node 222 may receive a service request for a big data service sent by the client 210 through a local area network or the Internet. The service request may include a service identifier and service data. The service identifier is used to uniquely indicate a big data service. The service data may be data for a computing node to perform distributed processing of big data or identification data indicating data to be processed.

[0105] Big data services include data analysis services, data query services, and data modification services. For example, big data services refer to analyzing customers' personal data and purchase behavior data to describe user portraits and classify customers, so that targeted products or preferential products can be recommended to specific customers, improving customer satisfaction and consolidating customer relationships. Another example is that big data services refer to analyzing the historical sales volume of products to predict future sales volume, discovering the reasons for the decline or increase in sales volume, and recommending constructive suggestions to increase sales.

[0106] Step 420: The control node 222 determines, from at least one executed task, a most adjacent task having a closest coding distance to the task to be executed.

[0107] As a possible example, the control node 222 may divide the big data service into multiple tasks to be executed, determine the computing node 221 that executes each task to be executed, and then determine the nearest neighboring task with the shortest encoding distance to each task to be executed from at least one executed task.

[0108] As a possible implementation, taking a task to be executed as an example, the control node 222 determines the nearest neighboring task as the executed task with the highest similarity to the task to be executed among at least one executed task. The similarity may be cosine similarity, Manhattan distance, Chebyshev distance, simple matching coefficient, etc.

[0109] Optionally, the control node 222 determines the similarity between tasks based on the similarity between the feature vectors of the tasks. The control node 222 converts the task to be executed and at least one executed task into vector form, obtains the feature vector of the task to be executed and at least one executed task feature vector, and then determines the nearest neighbor task as the executed task corresponding to the executed task feature vector with the highest cosine similarity to the feature vector of the task to be executed in the at least one executed task feature vector.

[0110] The present application does not limit the above-mentioned method of converting the program file of the task into the feature vector. For example, the method of converting the program file of the task into the feature vector in the present application can refer to step b of the offline phase in the above-mentioned solution 1.

[0111] In this way, the control node 222 only needs to perform a simple cosine similarity calculation between feature vectors to determine the recommended parameter configuration that matches the task to be executed. There is no need to use a complex neural network architecture to determine the recommended parameter configuration through long-term training or calculation, nor is there a need to perform multiple iterations of parameter screening and optimization for each new task. This reduces the time spent on parameter configuration optimization and improves the overall efficiency of the big data platform in executing tasks.

[0112] In a possible embodiment of the present application, before executing the above step 420, the control node 222 also needs to obtain the parameters that need to be tuned for the task to be executed, and the historical data of at least one executed task. For specific steps, please refer to Figure 5 Steps 421 to 423 in the above are not described in detail here.

[0113] Step 430: The control node 222 uses the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform.

[0114] As a possible example, the control node 222 selects the optimal parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform.

[0115] As a possible implementation manner, the above optimal parameter configuration is the parameter configuration that enables the big data platform to perform best in running the nearest task in the historical data of the nearest task.

[0116] Optionally, the historical data of a task (e.g., any one of at least one executed task) includes a historical record item corresponding to when the task is executed in the big data platform. The historical record item includes a parameter configuration corresponding to when the task is executed, and an execution time of the task executed by the big data platform based on the parameter configuration.

[0117] For example, the historical data of the nearest task contains at least two historical record items. Considering that the shorter the task execution time, the higher the task execution efficiency corresponding to the parameter configuration, the control node 222 selects the parameter configuration contained in the historical record item with the shortest execution time in the historical data of the nearest task as the optimal parameter configuration, so that the parameter configuration subsequently recommended to the big data platform can better improve the task execution efficiency and improve the overall performance of the big data platform.

[0118] For another example, the history data of the nearest task includes a history record item, and the control node 222 selects the parameter configuration included in the history record item of the nearest task as the optimal parameter configuration.

[0119] Step 440 : The control node 222 sends a control instruction to at least one computing node 221 .

[0120] As a possible example, the control node 222 sends a control instruction to at least one computing node 121 that executes a task to be executed, and the control instruction is used to instruct the computing node 221 to execute the task to be executed based on the parameter configuration of the nearest task to be executed.

[0121] Step 450 , the computing node 221 executes the pending tasks in the big data distributed processing on the business data according to the control instruction sent by the control node 222 .

[0122] As a possible example, computing node 221 executes the task to be executed in big data distributed processing on the business data of the big data platform according to the control instruction and based on the parameter configuration of the nearest task to be executed.

[0123] The computing nodes 221 execute the pending tasks in the big data distributed processing on the business data of the big data platform, which can be regarded as the big data platform executing the pending tasks.

[0124] As a possible implementation, after executing step 450 , the computing node 221 may further execute step 460 , that is, returning a service response of the big data service to the client 210 according to the processing result of the task to be executed.

[0125] Based on the above steps 410-460, the control node 222 uses the parameter configuration of the nearest task in at least one executed task that is closest to the encoding distance of the task to be executed as the recommended parameter configuration of the task to be executed, without the need to conduct time-consuming training of the neural network model of the complex architecture, and without the need to perform parameter sensitivity screening and iterative optimization each time the parameter configuration recommendation is made. Therefore, the method is based on the rapid matching of the task to be executed and the executed task, and uses the operation information of the historical tasks of the big data platform to obtain the recommended parameter configuration of the task to be executed, which ensures that the recommended parameter configuration of the task to be executed obtained by the computing node 221 is a parameter that can make the execution efficiency of the task to be executed higher, while reducing the time-consuming parameter configuration optimization of the task to be executed, and improving the overall performance of the big data platform.

[0126] Combination of the above Figures 2 to 4 , the parameter configuration method provided in this application is generally described. Figure 5 The specific steps for the control node 222 to determine the nearest task to be executed are described in detail.

[0127] like Figure 5 As shown, the step of determining the nearest task may include the following steps 421 to 425 .

[0128] Step 421: The control node 222 determines the parameters that need to be tuned for the big data platform.

[0129] As a possible example, the control node 222 determines, according to the type of the big data platform, the parameters that need to be tuned when the big data platform executes the task to be executed.

[0130] This application does not limit the specific type and quantity of the parameter, and the type of the parameter may vary depending on the type of the big data platform.

[0131] For example, if the big data platform is a Spark platform, the parameters that need to be tuned include but are not limited to spark.driver.cores, spark.driver.memory, spark.executor.memory, spark.executor.memoryOverhead, spark.shuffle.file.buffer, etc.

[0132] For example, the big data platform is the Hive platform, hive.new.job.grouping.set.cardinality, hive.map.aggr.hash.force.flush.mem ory.threshold, hive.map.aggr.hash.min.reduction, hive.optimize.index.filter, hive.mapjoin.bucket.cache.size, etc.

[0133] Step 422: The control node 222 obtains historical data.

[0134] As a possible example, the control node 222 obtains historical data from a historical database, wherein the historical data includes at least one executed task that has been executed by the big data platform.

[0135] As a possible implementation, the historical database may be deployed in any storage node 231 of the cloud platform or the storage cluster 230 .

[0136] Optionally, the historical data may be a program code file of at least one executed task, or may be a feature vector of at least one executed task program code file after conversion. If the historical data is a feature vector, then in the subsequent step 424, feature conversion of at least one executed task program code file is not required.

[0137] Step 423: The control node 222 obtains the task to be executed.

[0138] As a possible example, the task to be executed acquired by the control node 222 may be one task to be executed, or may be a collection of multiple tasks to be executed. The present application takes the control node 222 acquiring one task to be executed as an example for subsequent description.

[0139] Step 424: The control node 222 respectively calculates the coding distance between the task to be executed and each of the at least one executed task.

[0140] Step 425 : The control node 222 determines the nearest neighboring task with the highest similarity to the task to be executed according to the coding distance.

[0141] For details of the above steps 424-425, please refer to Figure 4 Step 420 shown is not repeated here.

[0142] Based on the above steps 421 to 425, it can be seen that the parameter configuration method provided in the present application can determine different parameters that need to be tuned according to different types of big data platforms to execute subsequent historical data, data acquisition of tasks to be executed, and steps for determining the nearest tasks. Therefore, this method can be applicable to different types of big data platforms such as Spark platform and Hive platform.

[0143] Combination of the above Figure 5 The following describes in detail how the control node 222 determines the nearest task to be executed. After determining the nearest task, the control node 222 can execute the following Figure 4 Steps 430 to 460 shown in the figure complete the parameter configuration recommendation. Considering that after the computing node 221 completes the task to be executed, the task to be executed also becomes historical data when the computing node 221 needs to execute a new task to be executed, the update of historical data can improve the accuracy of subsequent parameter configuration recommendations. Therefore, the parameter configuration method provided by the present application is in addition to Figure 4 The steps shown can also include historical data tuning steps. Figure 6 Steps 610 to 670 are shown to provide detailed descriptions of the historical data tuning steps.

[0144] Step 610: The control node 222 obtains historical data.

[0145] As a possible example, the control node 222 obtains the historical data from a historical database.

[0146] The difference between the above historical data and the historical data in step 422 is that the historical data in step 610 includes the historical record items corresponding to the executed tasks when they are executed in the big data platform. The historical record items include parameter configuration and execution duration.

[0147] Optionally, the feature vectors of executed tasks in the historical data may be stored in a historical database, and the historical record items in the historical data may be stored in a historical operation database, wherein the historical operation database may be deployed in any storage node 231 of the cloud platform or the storage cluster 230 .

[0148] Optionally, the historical data of step 610 also includes data of the task to be executed obtained after the computing node 221 executes the task to be executed according to the parameter configuration of the nearest task. The data of the task to be executed can be stored in the historical database by the computing node 221 after executing the task to be executed.

[0149] Step 620: The control node 222 constructs a training set based on historical data.

[0150] As a possible example, the control node 222 uses the feature vector and parameter configuration of each executed task in the historical data as a sample in the training set, and uses the execution time of each executed task as a label (ie, output result) to obtain the training set.

[0151] As a possible implementation method, the control node 222 can also perform Bayesian optimization based on conformal prediction on the above training set to obtain the training set for model training. In this way, statistical factors are introduced into the Bayesian optimization process through conformal prediction, and the parameter configuration in the historical data is subjected to Bayesian optimization with a high confidence level to sample and obtain the training set, thereby reducing the number of sampling times, reducing the dependence on samples, simplifying the training set to improve the training speed of the neural network model, and reducing the deviation in the training set sampling process, ensuring the representation ability of the training set, thereby avoiding the neural network model from falling into the local optimum, and ensuring the accuracy and universality of the neural network model.

[0152] Step 630: The control node 222 uses the training set to train the task performance regression model.

[0153] As a possible example, the control node 222 inputs the training set into the task performance regression model to iteratively train the task performance regression model until a model convergence condition or an iteration number threshold is met.

[0154] Optionally, the task performance regression model may be any type of regression model, such as a linear regression model, a random forest regression model, etc.

[0155] Step 640: The control node 222 uses the task performance regression model to output the optimal parameter configuration in the historical data.

[0156] As a possible example, the control node 222 inputs the historical data into the task performance regression model to obtain the optimal parameter configuration in the historical data output by the task performance regression model.

[0157] Step 650 : The control node 222 sends a control instruction to at least one computing node 221 .

[0158] As a possible example, the control node 222 sends a control instruction to at least one computing node 121 that executes a task to be executed, where the control instruction is used to instruct the computing node 221 to execute the nearest task based on the optimal parameter configuration.

[0159] Step 660 , the computing node 221 executes the nearest neighbor task in big data distributed processing on the business data according to the control instruction sent by the control node 222 .

[0160] As a possible example, computing node 221 performs the nearest neighbor task in big data distributed processing on the business data of the big data platform based on the optimal parameter configuration according to the control instruction.

[0161] Step 670 : The computing node 221 adds the historical record item of the nearest task to the historical data.

[0162] As a possible example, the computing node 221 stores the feature vector of the nearest task in a history database, and stores the history record item in a history operation database.

[0163] As a possible implementation manner, the computing node 221 obtains the historical record item of the nearest task based on log analysis and task diagnosis after the nearest task is executed.

[0164] Optionally, the history record items include execution duration and parameter configuration.

[0165] Regarding the above steps 610 to 670, steps 630 to 670 can be repeated multiple times to complete the update of the nearest task. Steps 610 to 670 can be executed when the big data platform is not executing tasks, and the idle time and computing power of the big data platform are used to update the historical data, thereby realizing the decoupling of the two steps of historical data update and parameter configuration recommendation, which not only realizes the update of the performance prediction model, but also avoids the impact of the model update process on the actual online environment of the parameter configuration recommendation.

[0166] Figure 7 Schematic diagram of the structure of a possible parameter configuration device provided in this embodiment. The parameter configuration device can be used to implement the functions of the control node and / or the computing node in the above method embodiment, and thus also has the beneficial effects possessed by the above method embodiment. In this embodiment, the parameter configuration device can be as follows Figure 2 The nodes constituted by the control node 222, computing node 221 and / or other servers shown may also be modules (such as chips) applied to the servers.

[0167] like Figure 7 As shown, the parameter configuration device 700 includes a parameter recommendation module 710 .

[0168] The parameter configuration device 700 can be used to implement the above Figure 4 In the method embodiment shown in FIG. 1 , the functions of the controller 222 are described, and the modules included in the parameter configuration device 700 are specifically used to implement the following functions.

[0169] The parameter recommendation module 710 is used to determine the nearest task with the closest coding distance to the task to be executed in at least one executed task, and use the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task. For example, the parameter recommendation module 710 is used to perform the following steps: Figure 4 Steps 420 and 430 are shown.

[0170] As a possible implementation manner, the parameter recommendation module 710 is specifically configured to: determine that the nearest neighbor task is the executed task having the highest similarity with the task to be executed among at least one executed task.

[0171] Optionally, the similarity is cosine similarity.

[0172] As a possible implementation method, the parameter recommendation module 710 is specifically used to: convert the task to be executed and at least one executed task into vector form respectively; determine the nearest neighboring task as the executed task whose vector form in at least one executed task has the highest cosine similarity with the vector form of the task to be executed.

[0173] As a possible implementation method, the parameter recommendation module 710 is specifically used to: determine the historical record item with the shortest execution time from the historical data of the nearest task, the historical data including the historical record item of the nearest task executed at least once by the big data platform based on different parameter configurations; determine the parameter configuration of the nearest task as the parameter configuration of the historical record item with the shortest execution time.

[0174] As a possible implementation, the execution subject of the parameter configuration method is used to execute tasks on the big data platform, and the parameter recommendation module 710 is specifically used to: execute the task to be executed based on the parameter configuration of the nearest task in the big data platform.

[0175] As a possible implementation method, the execution subject of the parameter configuration method is not used to execute tasks on the big data platform, and the parameter recommendation module 710 is specifically used to: use the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, and send the parameter configuration of the nearest task to the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

[0176] As a possible implementation, the parameter configuration device 700 further includes a historical task tuning module 720, which is used to update the data of at least one executed task according to the execution result of the task to be executed. Figure 6 Steps 610 - 650 are shown.

[0177] As a possible implementation method, the execution result includes the execution time and parameter configuration of the task, and the historical task tuning module 720 is specifically used to: add the execution time and parameter configuration of the task to be executed to the data of at least one executed task; determine the optimal parameter configuration in the data of at least one executed task after adding the execution time and parameter configuration of the task to be executed; add the execution result of the nearest task based on the optimal parameter configuration to the data of at least one executed task.

[0178] Optionally, the historical task tuning module 720 is specifically used to: determine the optimal parameter configuration using a neural network model, the input of the neural network model is the feature vector and parameter configuration of each executed task, and the output is the execution time of each executed task.

[0179] Optionally, the neural network model is a regression model.

[0180] Optionally, the training set of the neural network model is obtained through Bayesian optimization of conformal prediction based on data of at least one executed task.

[0181] It should be understood that the parameter configuration device 700 of the embodiment of the present invention can be implemented by a CPU, an ASIC, or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), an FPGA, a generic array logic (GAL), or any combination thereof. Figure 4 When the parameter configuration method shown in , the parameter configuration device 700 and its various modules can also be software modules.

[0182] It should be understood that the computing nodes, storage nodes and other nodes in the embodiment of the present application may correspond to the parameter configuration device 700 in the embodiment of the application, and may correspond to the corresponding subjects in the method according to the embodiment of the present application, and the above and other operations and / or functions of each module in the parameter configuration device 700 are respectively to achieve Figure 4 For the sake of brevity, the corresponding process of the method in will not be repeated here.

[0183] The present application also provides a Figure 8 The system shown in FIG. 1 includes multiple computing devices 800, each of which includes a memory 801, a processor 802, a communication interface 803, and a bus 804. The memory 801, the processor 802, and the communication interface 803 are connected to each other through the bus 804. The computing device 800 may be Figure 2A computing node, control node, storage node, or server in a cluster.

[0184] The memory 801 may be a read-only memory, a static storage device, a dynamic storage device or a random access memory. The memory 801 may store computer instructions and a data set required to execute the computer instructions. When the computer instructions stored in the memory 801 are executed by the processor 802, the processor 802 and the communication interface 803 are used to execute the parameter configuration method.

[0185] The processor 802 may be a general-purpose central processing unit (CPU), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or any combination thereof. The processor 802 may include one or more chips. The processor 802 may include an AI accelerator, such as a neural processing unit (NPU). In addition, Figure 8 The example in which each computing device 800 includes a processor 802 is used for explanation. In specific implementation, the number and type of processors 802 in each computing device 800 can be set according to business requirements. The same computing device 800 can include one or more processors. When the same computing device 800 includes multiple processors, this application does not limit the type of processor.

[0186] The communication interface 803 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the computing device 800 and other devices or a communication network. For example, a user operation request can be received or sent through the communication interface 803 .

[0187] The bus 804 may include a path for transmitting information between various components of the computing device 800 (eg, the memory 801 , the processor 802 , and the communication interface 803 ).

[0188] A communication path is established between each of the computing devices 800 through a communication network. Any computing device 800 may be a computer (eg, a server) in a distributed storage system, or a computer in an edge data center, or a terminal computing device.

[0189] Each computing device 800 may be deployed with computing nodes and / or storage nodes, etc., for example, to execute Figure 4 Any of steps 410 to 460 shown, or the functions of the parameter recommendation module 710 and / or the historical task tuning module 720 in the parameter configuration device 700 are executed.

[0190] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a terminal device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0191] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD). The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A parameter configuration method, characterized in that: include: Determine, from at least one executed task, a most adjacent task having a closest encoding distance to the task to be executed; The parameter configuration of the nearest task is used as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

2. The method according to claim 1, characterized in that The step of determining the nearest neighboring task having the shortest coding distance to the task to be executed from at least one executed task comprises: The most adjacent task is determined to be an executed task having the highest similarity to the task to be executed among the at least one executed task.

3. The method according to claim 2, characterized in that The similarity is cosine similarity.

4. The method according to claim 2 or 3, characterized in that: The determining that the most adjacent task is the executed task with the highest similarity to the task to be executed among the at least one executed task includes: Converting the to-be-executed task and the at least one executed task into vector forms respectively; The most adjacent task is determined to be an executed task whose vector form has the highest cosine similarity with the vector form of the task to be executed among the at least one executed task.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determine the historical record item with the shortest execution time from the historical data of the nearest task, wherein the historical data includes the historical record item of the execution of the nearest task by the big data platform at least once based on different parameter configurations; The parameter configuration of the most adjacent task is determined to be the parameter configuration of the historical record item with the shortest execution time.

6. The method according to any one of claims 1 to 5, characterized in that: The execution subject of the parameter configuration method is used to execute the task of the big data platform, and the parameter configuration of the nearest task is used as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task, including: The task to be executed is executed in the big data platform based on the parameter configuration of the nearest task.

7. The method according to any one of claims 1 to 5, characterized in that: The execution subject of the parameter configuration method is not used to execute the task of the big data platform, and the parameter configuration of the nearest task is used as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task, including: Using the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform; The parameter configuration of the nearest task is sent to the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The data of the at least one executed task is updated according to the execution result of the task to be executed.

9. The method according to any one of claims 1 to 8, characterized in that: The execution result includes the execution time and parameter configuration of the task, the data of the at least one executed task includes the execution time and parameter configuration of each executed task, and updating the data of the at least one executed task according to the execution result of the task to be executed includes: Adding the execution duration and parameter configuration of the task to be executed to the data of the at least one executed task; Determining an optimal parameter configuration in the data of the at least one executed task after adding the execution duration and parameter configuration of the task to be executed; An execution result of executing the nearest task based on the optimal parameter configuration is added to the data of the at least one executed task.

10. The method according to claim 9, characterized in that The determining of the optimal parameter configuration in the data of the at least one executed task after adding the execution duration and parameter configuration of the task to be executed comprises: The optimal parameter configuration is determined using a neural network model, the input of the neural network model is the feature vector and parameter configuration of each executed task, and the output of the neural network model is the execution time of each executed task.

11. The method according to claim 10, characterized in that The neural network model is a regression model.

12. The method according to claim 9 or 10, characterized in that The training set of the neural network model is obtained by Bayesian optimization of the data of the at least one executed task based on conformal prediction.

13. A parameter configuration device, characterized in that: include: A parameter recommendation module, used to determine the nearest neighboring task with the shortest coding distance to the task to be executed from at least one executed task; The parameter recommendation module is also used to use the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

14. The device according to claim 13, characterized in that The device also includes: The historical task tuning module is used to update the data of the at least one executed task according to the execution result of the task to be executed.

15. A computing device, characterized in that: The computing device includes a memory and a processor, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of any one of the methods described in claims 1-12 are performed.

16. A big data system, characterized in that: The big data system includes a first computing device and a second computing device, wherein the first computing device is used to execute the operation steps of the method described in any one of claims 1 to 12, and the second computing device is used to execute the task to be executed based on the parameter configuration of the nearest task.

17. A readable storage medium, characterized in that: The readable storage medium includes a computer program or instructions. When the computer program or instructions are executed on a computer, the computer is enabled to execute the operation steps of the method described in any one of claims 1 to 12.

Images