Parameter configuration method, apparatus, computing device, system and readable storage medium

Abstract

The present application relates to the field of big data. Provided are a parameter configuration method, an apparatus, a computing device, a system and a readable storage medium. The method comprises: for a task to be executed that needs to be run by a big data platform, first, determining from at least one executed task a nearest neighbor task having a closest coding distance to the task to be executed; and then, setting a parameter configuration of the task to be executed in the big data platform as a parameter configuration of the nearest neighbor task, such that the big data platform executes the task to be executed by using configuration parameters of the nearest neighbor task. Therefore, on the basis of fast matching of the task to be executed and the executed task, by using running information of historical tasks of the big data platform to obtain a recommendation parameter configuration of the task to be executed, parameters making the execution efficiency of the task to be executed relatively high are obtained, and time consumption for performing parameter configuration optimization on the task to be executed are reduced.

Description

Parameter configuration method, device, computing device, system and readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 29, 2023, with application number 202310793027.X and application name “Parameter configuration method, device, computing device, system and readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of big data, and in particular to a parameter configuration method, apparatus, computing device, system, and readable storage medium. Background Art

[0003] Big data platforms, such as big data analysis (BDA) platforms, are tools used to manage, process, and analyze large amounts of data. When performing data management and analysis tasks, these platforms require adjustments to task-related parameter configurations. Different values ​​for these parameter configurations significantly impact the platform's overall performance. Appropriate parameter configurations can significantly shorten task runtimes.

[0004] In addition to manual tuning of big data platform parameters based on prior knowledge, automated parameter tuning methods exist, including offline tuning based on neural network models and online tuning that iteratively optimizes parameter sets based on the coefficient of variation (CV) to identify those that affect big data platform performance. However, both traditional offline and online tuning methods suffer from the time-consuming nature of determining parameter configurations.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a parameter configuration method, apparatus, computing device, system, and readable storage medium, which can solve the problem of time-consuming parameter configuration and tuning of a big data platform.

[0007] In a first aspect, a parameter configuration method is provided. The parameter configuration method includes: for a pending task to be executed on a big data platform, determining a nearest neighboring task from at least one executed task that has the closest coding distance to the pending task, and then using the parameter configuration of the nearest neighboring task as the parameter configuration of the pending task on the big data platform, so that when the big data platform executes the pending task, it sets the parameters of the pending task according to the parameter configuration of the nearest neighboring task, and then executes the pending task.

[0008] Based on the above parameter configuration method, the parameter configuration of the nearest neighboring task in at least one executed task, which is closest to the encoding distance of the task to be executed, is used as the recommended parameter configuration for the task to be executed. This eliminates the need for time-consuming training of complex neural network models, nor does it require parameter sensitivity screening and iterative optimization each time a parameter configuration recommendation is made. Therefore, this method is based on the rapid matching of the task to be executed with the executed tasks, and utilizes the historical task operation information of the big data platform to obtain the recommended parameter configuration for the task to be executed. While being able to obtain parameters that increase the execution efficiency of the task to be executed, it also reduces the time spent on parameter configuration optimization for the task to be executed.

[0009] This application does not limit the execution subject of the parameter configuration method. Optionally, the execution subject of the parameter configuration method provided in this application can be a computing device (or computing node) used for big data analysis and processing in a big data system, or a computing device connected to a big data system.

[0010] For example, when the parameter configuration method is executed by a computing node within a big data system, the computing node can determine that the parameter configuration of the task to be executed in the big data platform is the parameter configuration of the nearest task, and then execute the task to be executed based on the parameter configuration of the nearest task in the big data platform.

[0011] For another example, when the parameter configuration method is executed by a computing node outside the big data system, the computing node can send the parameter configuration to the computing node used to execute the task on the big data platform after determining that the parameter configuration of the task to be executed in the big data platform is the parameter configuration of the nearest task, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

[0012] As a possible implementation, the encoding distance refers to the similarity between two tasks. The computing node determines the nearest neighbor task as the executed task that has the highest similarity to the task to be executed among at least one executed task.

[0013] Optionally, the similarity in this embodiment may be represented by cosine similarity, Manhattan distance, Chebyshev distance, simple matching coefficient, etc.

[0014] Taking cosine similarity as an example, the computing node can convert the task to be executed and at least one executed task into vector form respectively, and then determine the executed task with the highest cosine similarity with the task to be executed in vector form among the at least one executed task in vector form, and use the executed task as the nearest task to be executed.

[0015] In this way, the computing node only needs to calculate the similarity between tasks, such as the cosine similarity, to determine the nearest task to be executed from at least one executed task. The overall computational complexity is small, which improves the overall efficiency of parameter configuration.

[0016] As one possible implementation, at least one executed task includes historical data for each executed task. The historical data for each executed task includes historical records corresponding to previous executions of the task on the big data platform. The historical records include parameter configurations and execution durations. The computing node determines the historical record with the shortest execution duration from the historical data of the nearest task, and determines the parameter configuration of the nearest task to be the parameter configuration of the historical record with the shortest execution duration.

[0017] In this way, the execution time of multiple historical tasks is used as a reference factor for selecting parameter configurations. The parameter configuration that minimizes the execution time of the nearest task is selected from the historical records of the nearest task as the recommended parameter configuration for the task to be executed. This parameter configuration can enable the nearest task to be executed efficiently. Since the nearest task is highly similar to the task to be executed, this parameter configuration can also improve the execution efficiency of the task to be executed, thereby shortening its execution time.

[0018] As a possible implementation method, after the computing node executes the task to be executed based on the parameter configuration, or after sending the parameter configuration to the big data platform so that the big data platform executes the task to be executed based on the parameter configuration, it can also update the historical data of at least one executed task according to the execution result of the task to be executed.

[0019] The execution result of the task to be executed includes the execution time and parameter configuration of the task, and the data of at least one executed task includes the execution time and parameter configuration of each executed task.

[0020] In this way, the historical data of at least one executed task is continuously updated, the data breadth of the historical data is guaranteed, and the computing node can match the nearest task with a closer coding distance in the historical data as much as possible, thereby improving the degree to which the parameter configuration is suitable for the task to be executed.

[0021] Optionally, the computing node adds the execution result of the task to be executed, i.e., the execution duration and parameter configuration, as historical data to the data of at least one executed task, and then determines the optimal parameter configuration in the data of at least one executed task after the added data, and then adds the execution result of the nearest task based on the optimal parameter configuration as historical data to the data of at least one executed task.

[0022] This application does not limit the method for determining the optimal parameter configuration. For example, the computing node may utilize a neural network model to determine the optimal parameter configuration. The input of the neural network model is the feature vector and parameter configuration of each executed task, and the output of the neural network model is the execution time of each executed task. The neural network model may be a neural network model constructed based on a regression model.

[0023] In this way, by first recommending parameter configuration and then updating historical data, the two steps of parameter configuration recommendation and historical data update are decoupled, which can not only realize the iterative update of the neural network model, but also avoid the impact of the model update process on the computing node's determination of the recommended parameter configuration for the task to be executed.

[0024] Optionally, the training set of the above-mentioned neural network model can be historical data, i.e., data of at least one executed task, obtained after Bayesian optimization (BO) based on conformal prediction (CP). In this way, statistical factors are introduced into the Bayesian optimization process through conformal prediction, and the parameter configuration in the historical data is subjected to Bayesian optimization with a high degree of confidence to sample and obtain the training set. In this way, while simplifying the training set to increase the training speed of the neural network model, the deviation in the training set sampling process is reduced, the representation ability of the training set is guaranteed, and the neural network model is prevented from falling into a local optimum, thereby ensuring the accuracy and universality of the neural network model.

[0025] In a second aspect, a parameter configuration device is provided, which includes modules for executing the data processing method in the first aspect or any possible design of the first aspect.

[0026] As a possible implementation, the parameter configuration device includes a parameter recommendation module. The parameter recommendation module is configured to: determine, from at least one executed task, a nearest neighboring task with the closest coding distance to the task to be executed; use the parameter configuration of the nearest neighboring task as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest neighboring task.

[0027] Optionally, the parameter recommendation module is specifically configured to: determine the nearest neighboring task as the executed task having the highest similarity to the task to be executed among the at least one executed task.

[0028] The above similarity is cosine similarity.

[0029] For example, the parameter recommendation module is specifically configured to convert the to-be-executed task and at least one executed task into vector form, and determine the nearest neighbor task as the executed task whose vector form has the highest cosine similarity with the to-be-executed task.

[0030] As a possible implementation, the parameter recommendation module is further configured to determine the historical record item with the shortest execution time from the historical data of the nearest task, and determine the parameter configuration of the nearest task to be the parameter configuration of the historical record item with the shortest execution time.

[0031] As a possible implementation method, the execution subject of the parameter configuration method is used to execute tasks on the big data platform, and the parameter recommendation module is specifically used to: execute the task to be executed based on the parameter configuration of the nearest task in the big data platform.

[0032] As a possible implementation method, the execution subject of the parameter configuration method is not used to execute the tasks of the big data platform. The parameter recommendation module is specifically used to: use the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, and send the parameter configuration of the nearest task to the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

[0033] As a possible implementation manner, the parameter configuration device further includes a historical task tuning module, which is used to update data of at least one executed task according to the execution result of the task to be executed.

[0034] As a possible implementation method, the execution result includes the execution duration and parameter configuration of the task. The historical task tuning module is specifically used to: add the execution duration and parameter configuration of the task to be executed to the data of at least one executed task; determine the optimal parameter configuration in the data of at least one executed task after adding the execution duration and parameter configuration of the task to be executed; and add the execution result of the nearest task based on the optimal parameter configuration to the data of at least one executed task.

[0035] Optionally, the historical task tuning module is specifically used to: determine the optimal parameter configuration using a neural network model, the input of the neural network model is the feature vector and parameter configuration of each executed task, and the output is the execution time of each executed task.

[0036] Optionally, the neural network model is a regression model.

[0037] Optionally, the training set of the neural network model is obtained by Bayesian optimization based on conformal prediction of data of at least one executed task.

[0038] In a third aspect, a computing device is provided, which includes at least one processor and a memory, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions, the operating steps of the parameter configuration method in the first aspect or any possible implementation of the first aspect are executed.

[0039] In a fourth aspect, a big data system is provided, which includes a first computing device and a second computing device, the first computing device is used to execute the operating steps of the parameter configuration method in the first aspect or any possible implementation of the first aspect, and the second computing device is used to execute the task to be executed based on the parameter configuration of the nearest task.

[0040] In addition, the technical effects of the parameter configuration device described in the second aspect, the technical effects of the computing equipment described in the third aspect, and the technical effects of the big data system described in the fourth aspect can refer to the technical effects of the parameter configuration method described in the first aspect and will not be repeated here.

[0041] In a fifth aspect, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a computing device, the computing device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0042] In a sixth aspect, a computer program product is provided, which, when executed on a computer, enables the computing device to execute the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0043] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a schematic diagram of the structure of a neural network provided by this application;

[0045] FIG2 is a schematic diagram of the architecture of a big data system provided by this application;

[0046] FIG3 is a schematic diagram of the structure of a computing cluster provided by this application;

[0047] FIG4 is a flow chart of a parameter configuration method provided by the present application;

[0048] FIG5 is a schematic diagram of a process for determining the nearest neighbor task provided by the present application;

[0049] FIG6 is a flow chart of a historical data optimization step provided by the present application;

[0050] FIG7 is a schematic diagram of the structure of a parameter configuration device provided by the present application;

[0051] FIG8 is a schematic diagram of the structure of a computing device and system provided by this application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b and c can be single or multiple.

[0053] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0054] The parameter configuration method provided in the embodiment of the present application can be applied to the big data analysis platform scenario in the big data field. The following is a brief introduction to the technologies that may be involved in this application.

[0055] (1) Neural Network

[0056] A neural network can be composed of neurons, which can be represented by x s The output of this operation unit satisfies the following formula:

[0057] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x sThe weight of the neuron, b is the bias of the neuron. f is the activation function of the neuron, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neuron into the output signal. The output signal of the activation function can be used as the input of the next layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neurons mentioned above, that is, the output of one neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neurons. The weight represents the strength of the connection between different neurons. The weight determines the influence of the input on the output. A weight close to 0 means that changing the input does not change the output. A negative weight means that increasing the input reduces the output.

[0058] As shown in FIG1 , a schematic diagram of the structure of a neural network provided in an embodiment of the present application is shown. The neural network 100 includes N processing layers, where N is an integer greater than or equal to 3. The first layer of the neural network 100 is the input layer 110, which is responsible for receiving the input signal x i The last layer of the neural network 100 is the output layer 130, which is responsible for outputting the processing result h of the neural network. i Excluding the first and last layers, the remaining layers are intermediate layers 140, which collectively comprise hidden layers 120. Each intermediate layer 140 in hidden layers 120 can both receive input signals and output signals. Hidden layers 120 are responsible for processing input signals. Each layer represents a logical level of signal processing. Through multiple layers, data signals can undergo multi-level logical processing.

[0059] In some feasible embodiments, the input signal of the neural network can be a video signal, a voice signal, a text signal, an image signal, a temperature signal, or other signals in various forms. The input signal of the neural network also includes various other computer-processable engineering signals, which are not listed here one by one.

[0060] (2) Big Data

[0061] The development of the internet, the Internet of Things, network bandwidth, smart devices, and cloud computing has driven the growth of data types and volumes at an unprecedented rate, leading to the emergence of big data. Data has evolved from a single processing object to a fundamental resource. Big data is a collection of data that cannot be captured, managed, and processed within a short period of time using conventional software tools.

[0062] Sources of big data can include management systems, web systems, physical systems, and scientific experiment systems. Management systems include transaction processing systems and office automation systems. Web systems include internet systems, such as search engines. Physical systems refer to systems involving physical objects and processes, such as real-time monitoring, real-time testing, and production scheduling. Scientific experiment systems refer to physical systems based on experimental environments.

[0063] Big data can be categorized based on its structural characteristics, including structured data, unstructured data, and semi-structured data. Structured data refers to data represented by a unified structure, such as data stored in databases like financial systems, enterprise resource planning systems, or customer relationship management systems. Unstructured data, such as audio, video, and image data, has variable field lengths and is not easily represented using structured data. Semi-structured data falls somewhere between structured and unstructured data, consisting of data within the same category but with different attributes, such as data in email systems.

[0064] Because big data contains vast amounts of data with interconnected relationships, it's useful to analyze it using data analysis methods, models, or tools to uncover these relationships and leverage them for predictions and decision-making. For example, analyzing user shopping trends can improve the user's shopping experience by recommending items they're likely to purchase. Therefore, big data is characterized by large volumes, rapid growth, diverse data types, and high utilization value.

[0065] (3) Big Data Platform

[0066] Big data platforms, such as big data analysis (BDA) platforms, are tools used to manage, process, and analyze large amounts of data. Spark, for example, is one of the most widely used big data analysis platforms. Spark has many adjustable parameter configurations, and the different values ​​of these parameters significantly impact the overall performance of the Spark system. A suitable set of parameter configurations can significantly shorten the runtime of Spark tasks. However, the optimal parameter configuration for each Spark task is different, and most Spark tasks struggle to achieve good performance with the default parameter configuration. Similar phenomena also occur in other big data platforms (such as Hive and Hadoop).

[0067] Therefore, recommending appropriate parameter configurations for tasks on the big data platform can increase the running speed of tasks and thus improve the overall operating efficiency of the big data platform.

[0068] However, the design of parameter tuning systems on big data platforms is quite difficult, mainly for the following three reasons: First, the optimal configuration of a task is difficult to reuse, which means that the optimal configuration obtained after tuning one task (or application) is difficult to migrate to another task (or application), and sometimes even requires tuning from scratch; second, there are complex relationships between different configuration items. Even for the same task (or application), the optimal parameters obtained under a single configuration item are difficult to apply to multiple configuration items; third, the tuning parameter space is very complex. Taking Spark as an example, among the parameters that can be tuned in Spark, there are discrete ones (such as "executor.cores" that can only take integer values) and continuous ones (such as "executor.memory"). These parameters constitute a high-dimensional space where continuous and discrete values ​​coexist, which adds difficulty to operations such as sampling and finding the maximum value.

[0069] Traditional parameter tuning for big data platforms is achieved manually, where professionals set the value of each configuration item through trial and error based on prior knowledge. However, this manual tuning method limits the number of tunable configuration items, resulting in limited results and high labor costs. Consequently, automated parameter tuning methods have emerged in the industry, primarily categorized as offline and online tuning. Offline tuning requires the establishment of an additional offline system based on the actual system, and requires significant upfront effort in data collection and complex model training, resulting in high time and cost. Online tuning requires running the task to be tuned multiple times on the actual system, which can be time-consuming and time-consuming to obtain the optimal parameter configuration. Performance fluctuations during operation are also difficult to control, and the risk of new tasks failing is high.

[0070] Next, we will use Schemes 1 to 3 as examples to illustrate the commonly used automatic parameter tuning methods.

[0071] Solution 1 includes two stages: offline training and online recommendation.

[0072] The offline training phase mainly includes the following steps a-c:

[0073] a. Collect data: Input some Spark applications, run the applications on small-scale datasets, collect data, and obtain the directed acyclic graph (DAG) scheduling diagram and stage-level code snippets corresponding to the input Spark applications.

[0074] b. Spark application feature extraction: The stage-level code snippets are converted into a vocabulary embedding matrix, and a convolutional neural network (CNN) model is applied for feature encoding to obtain a feature vector representing the code information. The DAG scheduling graph is converted into a node embedding matrix, and a graph convolutional neural network (GCN) model is applied for feature encoding to obtain a feature vector representing the scheduler.

[0075] c. Train the model: Concatenate the two vectors obtained in step b with the three vectors representing data features, environment features, and configuration features to obtain a feature vector representing the Spark task. Based on this feature vector, a multi-layer perceptron is trained to predict the runtime of the Spark task. At the same time, for each parameter configuration item that needs to be tuned, a regression model is trained to map the input Spark task and feature vector to the value of a parameter configuration item.

[0076] The online recommendation stage mainly includes the following steps d-g:

[0077] d. Input a Spark task, run it on a large-scale dataset and collect data, and obtain the DAG scheduling graph and stage-level code snippet corresponding to the input Spark task.

[0078] e. Adaptively generate candidate parameter configurations: For each parameter configuration item that needs to be tuned, construct a sampling interval. Finally, sample candidate parameter configurations in the high-dimensional space composed of the sampling intervals of all parameter configuration items.

[0079] f. Recommend the optimal parameter configuration: Under each candidate parameter configuration sampled in step e, use the NECS model to predict the application's runtime, and select the parameter configuration with the shortest predicted runtime as the optimal parameter configuration.

[0080] g. Adaptive model update: Run the Spark application with the optimal parameter configuration recommended in the previous step to collect data. After collecting a certain amount of data, combine the data collected in the online recommendation phase with the dataset from the offline training phase. Using adversarial learning, learn indistinguishable representations of training and test instances to update the NECS model.

[0081] Solution 2 includes three steps: query selection, parameter selection, and Bayesian optimization.

[0082] The query selection step involves using the coefficient of variation (CV), the ratio of the standard deviation to the mean, to represent the query's sensitivity to different parameter configurations. Generally speaking, a higher CV indicates a query that is more sensitive to parameter configuration adjustments. To save time when collecting training samples, some more efficient queries can be selected from the Spark Structured Query Language (SQL) server database application and others can be removed. To achieve this, an appropriate threshold for the CV is determined to determine which queries to retain. This solution uses a dynamic method to determine the threshold and partitions all CVs into three non-overlapping partitions. Queries falling within the first interval are then classified as parameter configuration-insensitive queries, while the remaining queries are defined as parameter configuration-sensitive queries. Finally, the insensitive queries are removed, and only the sensitive queries are retained to replace all existing queries for subsequent operations, significantly saving time.

[0083] The specific process of the parameter selection step includes: combining feature selection and feature extraction to screen out the parameter set that most affects the performance of the Spark SQL task. This process is based on the running data of the new application. First, some unimportant parameter configuration items are removed through the Spearman correlation coefficient (SCC), and then kernel-based principle component analysis (KPCA) is used to screen out important parameter configuration items. KPCA is a powerful nonlinear feature extractor that can not only reduce feature dimensions by calculating the eigenvectors of the covariance matrix of the original input, but also extract nonlinear information by using kernel methods, thereby achieving parameter dimensionality reduction. After this process is completed, the parameters that have a significant impact on the execution time of the Spark SQL task can be obtained.

[0084] The Bayesian optimization step involves designing a data-size-adaptive Gaussian process (GP) based on traditional Bayesian optimization, enabling it to adapt to changes in input data size. Specifically, the GP is used to model the execution time of Spark SQL tasks with specific configurations as a function of parameter configuration values ​​and input data size. Furthermore, the Markov Chain Monte Carlo method is used to reduce the number of required iterations, partially addressing the difficulty of running traditional Bayesian optimization in online environments.

[0085] Solution 3 includes three steps: model training, configuration recommendation, and similarity judgment.

[0086] The model training process involves sampling N sets of parameter configurations for a given new task type, executing the task based on these configurations, and building a cost-based performance model using the cost information from the task execution. Each task type corresponds to a performance model, which is gradually updated during the configuration recommendation process.

[0087] The configuration recommendation process involves searching for a set of optimal parameter configurations within the parameter configuration space based on the trained performance model. Specifically, the simulated annealing algorithm (SAA) is used to improve search efficiency while avoiding local optima. The search process is iterative, and when the resulting parameter configuration meets the expectations, tuning is terminated and the parameter configuration is recommended to the user.

[0088] The specific process of the similarity determination step involves the following: A new task of unknown type is input and executed once with the default configuration. Then, based on runtime data, the database is searched for similar tasks. If so, the performance model corresponding to the similar task is invoked to directly recommend parameter configurations, saving time on model training. If not, the running task is treated as a new type, and a new performance model is trained using the model training module before parameter configuration recommendations are made.

[0089] Solution 1 requires the support of an offline system, and the overall architecture is relatively complex. It requires a lot of time to collect training samples in advance, resulting in low overall parameter tuning efficiency and a long time consumption. Solution 2 uses the coefficient of variation of the execution time of Spark SQL tasks to determine the sensitivity of the query. This has no theoretical support, and the resulting configuration-sensitive query may not be correct. In addition, each tuning of a new task requires starting from scratch, and historical data cannot be used. Tasks must be repeated with different parameters to obtain a set of parameter configurations, which is time-consuming. The performance model for each type of task in Solution 3 only considers the configuration item values ​​and the amount of task data, and does not consider the characteristic information of different tasks. Therefore, the performance prediction is inaccurate. At the same time, the simulated annealing algorithm converges slowly, so it takes a long time to recommend a good set of parameter configurations for a new task.

[0090] It can be seen that traditional automatic parameter tuning methods have the problems of low parameter tuning efficiency and taking too long to recommend parameter configurations for new tasks.

[0091] The present application provides a parameter configuration method, in particular, a parameter configuration method for "recommending parameter configurations based on the similarity between historical tasks and pending tasks on a big data platform." The parameter configuration method includes: for a pending task to be run on a big data platform, determining the nearest neighboring task with the closest coding distance to the pending task from at least one executed task, and then using the parameter configuration of the nearest neighboring task as the parameter configuration of the pending task on the big data platform. This allows the big data platform to set the parameters of the pending task according to the parameter configuration of the nearest neighboring task when executing the pending task, and then execute the pending task.

[0092] Based on the parameter configuration method provided by the present application, the parameter configuration of the nearest neighboring task with the closest coding distance to the task to be executed in at least one executed task is used as the recommended parameter configuration for the task to be executed. Compared with the above-mentioned solutions one to three, there is no need to conduct time-consuming training of the neural network model with a complex architecture, nor is there any need to perform parameter sensitivity screening and iterative optimization each time a parameter configuration recommendation is made. Therefore, the parameter configuration method provided by the present application is based on the rapid matching of the task to be executed with the executed tasks in the historical data, and uses the operation information of the historical tasks executed by the big data platform to match to obtain the recommended parameter configuration for the task to be executed. While being able to obtain parameters that make the execution efficiency of the task to be executed higher, it reduces the time-consuming parameter configuration optimization for the task to be executed, and improves the overall operation efficiency of the parameter configuration big data platform.

[0093] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0094] Because the amount of big data is very large, a single computing node cannot meet the computing needs of big data. Typically, a distributed file system (DFS) is used to store and manage big data, and multiple computing nodes in a computing cluster are used to process big data services. Figure 2 is a schematic diagram of the architecture of a big data system provided in an embodiment of the present application. As shown in Figure 2, the big data system 200 includes a client 210, a computing cluster 220, and a storage cluster 230.

[0095] Storage cluster 230 includes at least two storage nodes 231. Each storage node 231 includes one or more controllers, a network card, and multiple hard disks. The hard disks are used to store data. Hard disks can be magnetic disks or other types of storage media, such as solid-state drives or shingled magnetic recording hard disks. The network cards are used to communicate with computing nodes 221 included in computing cluster 220. The controllers are used to write data to or read data from the hard disks based on read / write data requests sent by computing nodes 221. During the data reading and writing process, the controllers need to convert the addresses carried in the read / write data requests into addresses that the hard disks can recognize.

[0096] Client 210 communicates with computing cluster 220 and storage cluster 230 via network 240. For example, client 210 sends a big data service request to computing cluster 220 via network 240, requesting that computing cluster 220 perform distributed processing on the service data included in the service request. Network 240 can be an internal enterprise network (e.g., a local area network (LAN)) or the Internet.

[0097] In some embodiments, the storage cluster 230 stores and manages big data based on a distributed file system 232 and a distributed database 233. The client 210 is installed with a client program 211. The client 210 runs the client program 211 to display a user interface (UI). The user 250 operates the user interface to access the distributed file system 232 and the distributed database 233 to obtain data and instruct the computing cluster 220 to process big data services. The client 210 refers to a computer connected to the network 240, which can also be called a workstation. Different clients can share resources on the network (such as computing resources and storage resources). In other embodiments, the system administrator 260 can access the distributed file system 232 and the distributed database 233 by calling the application platform interface (API) 212 or the command-line interface (CLI) interface 213 through the client 210 to configure system information, etc., such as the deployment information and storage strategy of the storage medium configured for the computing node or storage node provided in the embodiments of the present application.

[0098] It should be noted that the storage medium can be a memory device, which is a memory device used to store programs and various data. The larger the memory capacity, the slower the access speed. Conversely, the smaller the memory capacity, the faster the access speed. Access speed refers to the data transmission speed when writing data to or reading data from the memory. Access speed can also be referred to as read and write speed. Memories can be divided into different tiers based on storage capacity and access speed.

[0099] Computing cluster 220 includes at least two computing nodes 221, which can communicate with each other. A computing node 221 is a computing device, such as a server, desktop computer, or storage array controller. The big data business submitted by client 210 can be called a job. A job can be divided into multiple tasks, which are executed in parallel by multiple computing nodes. The completion of all tasks marks the completion of a job. A task is generally a processing process for a portion of data or a stage within a job. All tasks are scheduled and completed in parallel or serially. In some embodiments, computing cluster 220 performs distributed processing of big data business based on the MapReduce model 234. The MapReduce model is a distributed programming model that decomposes big data business into multiple tasks, which are executed by multiple computing nodes 221. In some embodiments, as shown in Figure 3, computing cluster 220 includes a control node 222 and at least two computing nodes 221. The control node and computing nodes can be independent physical devices. In this case, the control node can also be referred to as a control device or a named node, and the computing nodes can be referred to as computing devices or data nodes. Control node 222 manages the namespace of the distributed file system and client 210's access to the distributed file system. Furthermore, control node 222 instructs the compute nodes to execute tasks. The compute nodes execute tasks based on MapReduce model 234 and obtain the results. Optionally, storage cluster 230 can also process data according to MapReduce model 234 before storing it.

[0100] As a possible implementation method, the parameter configuration method provided in the present application is executed by the control node 222, and the control node 222 is used to determine the nearest task with the closest coding distance to the task to be executed in at least one executed task, and send the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform to the computing node 221. The computing node 221 is a node for executing the task to be executed based on the received parameter configuration.

[0101] As another possible implementation method, the parameter configuration method provided in the present application is executed by the computing node 221, and the computing node 221 is used to determine the nearest task with the closest coding distance to the task to be executed in at least one executed task, and use the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, and then execute the task to be executed based on the parameter configuration.

[0102] The parameter configuration method provided in this application is described in detail below with reference to the accompanying drawings.

[0103] Figure 4 is a flow chart illustrating a parameter configuration method provided herein. This method is illustrated using a client 210 and a computing cluster 220 as an example. Computing cluster 220 includes a control node 222 and at least two computing nodes 221. Control node 222 is used to recommend parameter configurations for tasks to be performed to computing nodes 221 and to control computing nodes 221 in executing distributed big data processing. As shown in Figure 4, the method includes the following steps 410 through 460.

[0104] Step 410 : The control node 222 receives a service request for a big data service sent by the client 210 .

[0105] As one possible example, client 210 sends a service request for a big data service to control node 222 in response to a user operation. The user operation may refer to a user operating a big data user interface to submit a big data service. Control node 222 may receive the service request for the big data service sent by client 210 via a local area network or the internet. The service request may include a service identifier and service data. The service identifier uniquely identifies a big data service. The service data may be data for distributed big data processing by a computing node or identification data indicating data to be processed.

[0106] Big data services include data analysis, data query, and data modification. For example, big data services involve analyzing customer personal data and purchasing behavior data to create user profiles and categorize customers. This allows for the recommendation of targeted products or discounts to specific customers, thereby improving customer satisfaction and strengthening customer relationships. Another example is big data services that analyze historical product sales to predict future sales, identify reasons for declining or increasing sales, and recommend constructive suggestions for increasing sales.

[0107] Step 420: The control node 222 determines, from the at least one executed task, a nearest neighboring task having the shortest coding distance to the task to be executed.

[0108] As a possible example, the control node 222 can divide the big data business into multiple tasks to be executed, determine the computing node 221 to execute each task to be executed, and then determine the nearest neighboring task with the closest coding distance to each task to be executed among at least one executed task.

[0109] As a possible implementation, taking a pending task as an example, the control node 222 determines the nearest neighboring task as the executed task with the highest similarity to the pending task among at least one executed task. The similarity may be cosine similarity, Manhattan distance, Chebyshev distance, simple matching coefficient, or the like.

[0110] Optionally, the control node 222 determines the similarity between tasks based on the similarity between the feature vectors of the tasks. The control node 222 converts the to-be-executed task and at least one executed task into vector form, obtains a feature vector of the to-be-executed task and at least one executed task feature vector, and then determines the nearest neighbor task as the executed task corresponding to the executed task feature vector with the highest cosine similarity to the to-be-executed task feature vector among the at least one executed task feature vectors.

[0111] This application does not limit the above-mentioned method of converting the program file of the task into the feature vector. For example, the method of converting the program file of the task into the feature vector in this application can refer to step b of the offline phase in the first solution shown above.

[0112] In this way, the control node 222 only needs to perform a simple cosine similarity calculation between feature vectors to determine the recommended parameter configuration that matches the task to be performed. There is no need to use a complex neural network architecture to conduct long-term training or calculation to determine the parameter configuration to be recommended, nor is there a need to perform multiple iterations of parameter screening and optimization for each new task. This reduces the time consumption of parameter configuration optimization and improves the overall efficiency of the big data platform in executing tasks.

[0113] In a possible embodiment of the present application, before executing the above step 420, the control node 222 also needs to obtain the parameters that need to be tuned for the task to be executed, as well as historical data of at least one executed task. For specific steps, please refer to steps 421 to 423 in Figure 5, which will not be repeated here.

[0114] Step 430: The control node 222 uses the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform.

[0115] As a possible example, the control node 222 selects the optimal parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform.

[0116] As a possible implementation manner, the above-mentioned optimal parameter configuration is the parameter configuration that enables the big data platform to perform the best performance in running the nearest task in the historical data of the nearest task.

[0117] Optionally, the historical data of a task (e.g., any one of the at least one executed task) includes a historical record item corresponding to when the task was executed in the big data platform. The historical record item includes a parameter configuration corresponding to when the task was executed, and an execution time of the task executed by the big data platform based on the parameter configuration.

[0118] For example, the historical data of the nearest task contains at least two historical record items. Considering that the shorter the task execution time, the higher the task execution efficiency corresponding to the parameter configuration, the control node 222 selects the parameter configuration contained in the historical record item with the shortest execution time in the historical data of the nearest task as the optimal parameter configuration, so that the parameter configuration subsequently recommended to the big data platform can better improve the task execution efficiency and improve the overall performance of the big data platform.

[0119] For another example, the history data of the nearest task includes a history record item, and the control node 222 selects the parameter configuration included in the history record item of the nearest task as the optimal parameter configuration.

[0120] Step 440 : The control node 222 sends a control instruction to at least one computing node 221 .

[0121] As a possible example, the control node 222 sends a control instruction to at least one computing node 121 that executes the task to be executed, and the control instruction is used to instruct the computing node 221 to execute the task to be executed based on the parameter configuration of the nearest task to be executed for the task to be executed.

[0122] Step 450 : The computing node 221 executes the pending tasks in the big data distributed processing on the business data according to the control instruction sent by the control node 222 .

[0123] As a possible example, the computing node 221 executes the task to be executed in the big data distributed processing on the business data of the big data platform according to the control instruction and based on the parameter configuration of the nearest task to be executed.

[0124] The computing nodes 221 execute the pending tasks in the big data distributed processing on the business data of the big data platform, which can be regarded as the big data platform executing the pending tasks.

[0125] As a possible implementation, after executing step 450 , the computing node 221 may further execute step 460 , ie, return a service response of the big data service to the client 210 according to the processing result of the task to be executed.

[0126] Based on the above steps 410-460, the control node 222 uses the parameter configuration of the nearest neighboring task that is closest to the encoding distance of the task to be executed in at least one executed task as the recommended parameter configuration for the task to be executed, without the need for time-consuming training of the neural network model with a complex architecture, and without the need for parameter sensitivity screening and iterative optimization each time the parameter configuration is recommended. Therefore, this method is based on the rapid matching of the task to be executed and the executed tasks, and uses the running information of the historical tasks of the big data platform to obtain the recommended parameter configuration for the task to be executed, ensuring that the recommended parameter configuration for the task to be executed obtained by the computing node 221 is a parameter that can make the execution efficiency of the task to be executed higher, while reducing the time-consuming parameter configuration optimization of the task to be executed and improving the overall performance of the big data platform.

[0127] The above describes the parameter configuration method provided in the present application in general in conjunction with FIG. 2 to FIG. 4 . The following describes in detail the specific steps for the control node 222 to determine the nearest task to be executed in conjunction with FIG. 5 .

[0128] As shown in FIG5 , the step of determining the nearest task may include the following steps 421 to 425 .

[0129] Step 421: The control node 222 determines the parameters that need to be tuned for the big data platform.

[0130] As a possible example, the control node 222 determines, based on the type of the big data platform, parameters that need to be tuned when the big data platform executes the task to be executed.

[0131] This application does not limit the specific type and quantity of the parameters, and the type of the parameters may vary depending on the type of the big data platform.

[0132] For example, if the big data platform is the Spark platform, the parameters that need to be tuned include but are not limited to spark.driver.cores, spark.driver.memory, spark.executor.memory, spark.executor.memoryOverhead, spark.shuffle.file.buffer, etc.

[0133] For example, if the big data platform is Hive, the parameters include hive.new.job.grouping.set.cardinality, hive.map.aggr.hash.force.flush.memory.threshold, hive.map.aggr.hash.min.reduction, hive.optimize.index.filter, and hive.mapjoin.bucket.cache.size.

[0134] Step 422: The control node 222 obtains historical data.

[0135] As a possible example, the control node 222 obtains historical data from a historical database, wherein the historical data includes at least one executed task that has been executed by the big data platform.

[0136] As a possible implementation, the historical database may be deployed in a cloud platform or any storage node 231 of the storage cluster 230 .

[0137] Alternatively, the historical data may be a program code file of at least one executed task, or a feature vector of the program code file of at least one executed task after conversion. If the historical data is a feature vector, then in the subsequent step 424, feature conversion of the program code file of at least one executed task is not required.

[0138] Step 423: The control node 222 obtains the task to be executed.

[0139] As a possible example, the to-be-executed task acquired by the control node 222 may be one to-be-executed task or a collection of multiple to-be-executed tasks. This application uses the example of the control node 222 acquiring one to-be-executed task for subsequent description.

[0140] In step 424 , the control node 222 calculates the coding distance between the task to be executed and each of the at least one executed task.

[0141] Step 425 : The control node 222 determines the nearest neighboring task with the highest similarity to the task to be executed based on the coding distance.

[0142] For the specific contents of the above steps 424 and 425, please refer to step 420 shown in FIG4 , which will not be repeated here.

[0143] Based on the above steps 421 to 425, it can be seen that the parameter configuration method provided in this application can determine different parameters that need to be tuned according to different types of big data platforms to execute subsequent historical data, data acquisition of tasks to be executed, and steps for determining the nearest tasks. Therefore, this method can be applicable to different types of big data platforms such as Spark platform and Hive platform.

[0144] The above text, in conjunction with Figure 5, explains in detail how the control node 222 determines the nearest task to be executed. After determining the nearest task, the control node 222 can execute steps 430-460 as shown in Figure 4 to complete the parameter configuration recommendation. Considering that after the computing node 221 completes the task to be executed, the task to be executed also becomes historical data when the computing node 221 needs to execute a new task to be executed, the update of historical data can improve the accuracy of subsequent parameter configuration recommendations. Therefore, in addition to the steps shown in Figure 4, the parameter configuration method provided by the present application can also include a historical data tuning step. Next, the historical data tuning step is described in detail in conjunction with steps 610-670 shown in Figure 6.

[0145] Step 610: The control node 222 obtains historical data.

[0146] As a possible example, the control node 222 obtains historical data from a historical database.

[0147] The difference between the above historical data and the historical data in step 422 is that the historical data in step 610 includes historical records corresponding to the execution of the executed task in the big data platform. The historical records include parameter configuration and execution time.

[0148] Optionally, the feature vectors of executed tasks in the historical data may be stored in a historical database, and the historical record items in the historical data may be stored in a historical operation database, wherein the historical operation database may be deployed in any storage node 231 of the cloud platform or the storage cluster 230 .

[0149] Optionally, the historical data of step 610 also includes data of the pending task obtained after the computing node 221 executes the pending task according to the parameter configuration of the nearest task. The data of the pending task can be stored in the historical database by the computing node 221 after executing the pending task.

[0150] Step 620: The control node 222 constructs a training set based on historical data.

[0151] As a possible example, the control node 222 uses the feature vector and parameter configuration of each executed task in the historical data as a sample in the training set, and uses the execution time of each executed task as a label (ie, output result) to obtain the training set.

[0152] As a possible implementation, control node 222 can also perform Bayesian optimization based on conformal prediction on the training set to obtain a training set for final model training. In this way, by introducing statistical factors into the Bayesian optimization process through conformal prediction, the parameter configuration in the historical data is subjected to Bayesian optimization with high confidence to sample the training set. This can reduce the number of sampling times, reduce reliance on samples, simplify the training set, and improve the training speed of the neural network model. This also reduces bias in the training set sampling process, ensures the representational capability of the training set, and thus prevents the neural network model from falling into a local optimum, thereby ensuring the accuracy and universality of the neural network model.

[0153] Step 630: The control node 222 uses the training set to train the task performance regression model.

[0154] As a possible example, the control node 222 inputs the training set into the task performance regression model to iteratively train the task performance regression model until a model convergence condition or an iteration number threshold is met.

[0155] Optionally, the task performance regression model may be any type of regression model, such as a linear regression model, a random forest regression model, or the like.

[0156] Step 640: The control node 222 uses the task performance regression model to output the optimal parameter configuration in the historical data.

[0157] As a possible example, the control node 222 inputs the historical data into the task performance regression model to obtain the optimal parameter configuration in the historical data output by the task performance regression model.

[0158] Step 650 : The control node 222 sends a control instruction to at least one computing node 221 .

[0159] As a possible example, the control node 222 sends a control instruction to at least one computing node 121 that executes a task to be executed, where the control instruction is used to instruct the computing node 221 to execute the nearest task based on the optimal parameter configuration.

[0160] Step 660 : The computing node 221 executes the nearest neighbor task in the big data distributed processing on the business data according to the control instruction sent by the control node 222 .

[0161] As a possible example, the computing node 221 performs the nearest neighbor task in big data distributed processing on the business data of the big data platform based on the optimal parameter configuration according to the control instruction.

[0162] Step 670 : The computing node 221 adds the history record item of the nearest task to the historical data.

[0163] As a possible example, the computing node 221 stores the feature vector of the nearest task in a history database, and stores the history record item in a history operation database.

[0164] As a possible implementation manner, the computing node 221 obtains the historical record item of the nearest task based on log analysis and task diagnosis after the nearest task is executed.

[0165] Optionally, the history record items include execution duration and parameter configuration.

[0166] Regarding steps 610-670 above, steps 630-670 can be repeated multiple times to complete the update of the nearest task. Steps 610-670 can be executed when the big data platform is not executing tasks, utilizing the platform's idle time and computing power to update historical data. This decouples the historical data update and parameter configuration recommendation steps, enabling the update of the performance prediction model while preventing the model update process from affecting the actual online environment of the parameter configuration recommendation.

[0167] FIG7 is a schematic diagram of the structure of a possible parameter configuration device provided in this embodiment. This parameter configuration device can be used to implement the functions of the control node and / or computing node in the above-mentioned method embodiment, and thus also has the beneficial effects of the above-mentioned method embodiment. In this embodiment, the parameter configuration device can be a node composed of the control node 222, computing node 221, and / or other servers as shown in FIG2, or a module (such as a chip) applied to the server.

[0168] As shown in FIG. 7 , the parameter configuration device 700 includes a parameter recommendation module 710 .

[0169] The parameter configuration device 700 can be used to implement the functions of the controller 222 in the method embodiment shown in FIG. 4 . The modules included in the parameter configuration device 700 are specifically used to implement the following functions.

[0170] Parameter recommendation module 710 is configured to determine, from at least one executed task, the nearest neighboring task with the closest coding distance to the task to be executed, and to use the parameter configuration of the nearest neighboring task as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest neighboring task. For example, parameter recommendation module 710 is configured to execute steps 420 and 430 as shown in FIG. 4 .

[0171] As a possible implementation manner, the parameter recommendation module 710 is specifically configured to: determine the nearest neighbor task as the executed task having the highest similarity to the task to be executed among at least one executed task.

[0172] Optionally, the similarity is cosine similarity.

[0173] As a possible implementation method, the parameter recommendation module 710 is specifically used to: convert the task to be executed and at least one executed task into vector form respectively; determine the nearest neighboring task as the executed task whose vector form in at least one executed task has the highest cosine similarity with the vector form of the task to be executed.

[0174] As a possible implementation method, the parameter recommendation module 710 is specifically used to: determine the historical record item with the shortest execution time from the historical data of the nearest task, the historical data including the historical record item of the nearest task executed at least once by the big data platform based on different parameter configurations; determine the parameter configuration of the nearest task as the parameter configuration of the historical record item with the shortest execution time.

[0175] As a possible implementation, the execution subject of the parameter configuration method is used to execute tasks on the big data platform, and the parameter recommendation module 710 is specifically used to execute the task to be executed based on the parameter configuration of the nearest task in the big data platform.

[0176] As a possible implementation method, the execution subject of the parameter configuration method is not used to execute the tasks of the big data platform. The parameter recommendation module 710 is specifically used to: use the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, and send the parameter configuration of the nearest task to the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

[0177] As a possible implementation, the parameter configuration device 700 further includes a historical task tuning module 720 for updating data of at least one executed task according to the execution result of the task to be executed. For example, the historical task tuning module 720 is configured to execute steps 610 to 650 as shown in FIG6 .

[0178] As a possible implementation method, the execution result includes the execution duration and parameter configuration of the task. The historical task tuning module 720 is specifically used to: add the execution duration and parameter configuration of the task to be executed to the data of at least one executed task; determine the optimal parameter configuration in the data of at least one executed task after adding the execution duration and parameter configuration of the task to be executed; and add the execution result of the nearest task based on the optimal parameter configuration to the data of at least one executed task.

[0179] Optionally, the historical task tuning module 720 is specifically used to: determine the optimal parameter configuration using a neural network model, the input of the neural network model is the feature vector and parameter configuration of each executed task, and the output is the execution time of each executed task.

[0180] Optionally, the neural network model is a regression model.

[0181] Optionally, the training set of the neural network model is obtained by Bayesian optimization based on conformal prediction of data of at least one executed task.

[0182] It should be understood that the parameter configuration device 700 of the embodiment of the present invention can be implemented by a CPU, an ASIC, or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), an FPGA, a generic array logic (GAL), or any combination thereof. When the parameter configuration device 700 implements the parameter configuration method shown in FIG. 4 through software, the parameter configuration device 700 and its various modules can also be software modules.

[0183] It should be understood that the computing nodes, storage nodes and other nodes in the embodiments of the present application may correspond to the parameter configuration device 700 in the embodiments of the application, and may correspond to the corresponding subjects in executing the method according to the embodiments of the present application, and the above-mentioned and other operations and / or functions of each module in the parameter configuration device 700 are respectively for realizing the corresponding processes of the method in Figure 4. For the sake of brevity, they will not be repeated here.

[0184] The present application also provides a system as shown in FIG8 , which includes multiple computing devices 800, each of which includes a memory 801, a processor 802, a communication interface 803, and a bus 804. The memory 801, the processor 802, and the communication interface 803 are communicatively connected to each other via the bus 804. The computing device 800 may be a computing node, a control node, a storage node, or a server in FIG2 .

[0185] The memory 801 may be a read-only memory, a static storage device, a dynamic storage device, or a random access memory. The memory 801 may store computer instructions and a data set required to execute the computer instructions. When the computer instructions stored in the memory 801 are executed by the processor 802, the processor 802 and the communication interface 803 are used to execute the parameter configuration method.

[0186] The processor 802 can adopt a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or any combination thereof. The processor 802 may include one or more chips. The processor 802 may include an AI accelerator, for example, a neural network processor (NPU). In addition, Figure 8 is illustrated by taking the example of each computing device 800 including one processor 802. During specific implementation, the number and type of processors 802 in each computing device 800 can be set according to business requirements. The same computing device 800 may include one or more processors. When the same computing device 800 includes multiple processors, this application does not limit the type of processor.

[0187] The communication interface 803 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the computing device 800 and other devices or a communication network. For example, a user operation request can be received or sent through the communication interface 803.

[0188] The bus 804 may include a path for transmitting information between various components of the computing device 800 (eg, the memory 801 , the processor 802 , and the communication interface 803 ).

[0189] A communication path is established between each of the computing devices 800 via a communication network. Any computing device 800 may be a computer (eg, a server) in a distributed storage system, a computer in an edge data center, or a terminal computing device.

[0190] The functions of nodes such as computing nodes and / or storage nodes can be deployed on each computing device 800, for example, executing any step 410-step 460 shown in Figure 4, or executing the functions of the parameter recommendation module 710 and / or the historical task tuning module 720 in the parameter configuration device 700.

[0191] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a terminal device. Of course, the processor and storage medium can also exist as discrete components in a network device or a terminal device.

[0192] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is performed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD). The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A parameter configuration method, characterized in that: include: Determine, from at least one executed task, a most adjacent task having a closest encoding distance to the task to be executed; The parameter configuration of the nearest task is used as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

2. The method according to claim 1, characterized in that The step of determining the nearest neighboring task having the shortest coding distance to the task to be executed from at least one executed task comprises: The most adjacent task is determined to be an executed task having the highest similarity to the task to be executed among the at least one executed task.

3. The method according to claim 2, characterized in that The similarity is cosine similarity.

4. The method according to claim 2 or 3, characterized in that: The determining that the most adjacent task is the executed task with the highest similarity to the task to be executed among the at least one executed task includes: Converting the to-be-executed task and the at least one executed task into vector forms respectively; The most adjacent task is determined to be an executed task whose vector form has the highest cosine similarity with the vector form of the task to be executed among the at least one executed task.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determine the historical record item with the shortest execution time from the historical data of the nearest task, wherein the historical data includes the historical record item of the execution of the nearest task by the big data platform at least once based on different parameter configurations; The parameter configuration of the most adjacent task is determined to be the parameter configuration of the historical record item with the shortest execution time.

6. The method according to any one of claims 1 to 5, characterized in that: The execution subject of the parameter configuration method is used to execute the task of the big data platform, and the parameter configuration of the nearest task is used as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task, including: The task to be executed is executed in the big data platform based on the parameter configuration of the nearest task.

7. The method according to any one of claims 1 to 5, characterized in that: The execution subject of the parameter configuration method is not used to execute the task of the big data platform, and the parameter configuration of the nearest task is used as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task, including: Using the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform; The parameter configuration of the nearest task is sent to the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The data of the at least one executed task is updated according to the execution result of the task to be executed.

9. The method according to any one of claims 1 to 8, characterized in that: The execution result includes the execution time and parameter configuration of the task, the data of the at least one executed task includes the execution time and parameter configuration of each executed task, and updating the data of the at least one executed task according to the execution result of the task to be executed includes: Adding the execution duration and parameter configuration of the task to be executed to the data of the at least one executed task; Determining an optimal parameter configuration in the data of the at least one executed task after adding the execution duration and parameter configuration of the task to be executed; An execution result of executing the nearest task based on the optimal parameter configuration is added to the data of the at least one executed task.

10. The method according to claim 9, characterized in that The determining of the optimal parameter configuration in the data of the at least one executed task after adding the execution duration and parameter configuration of the task to be executed comprises: The optimal parameter configuration is determined using a neural network model, the input of the neural network model is the feature vector and parameter configuration of each executed task, and the output of the neural network model is the execution time of each executed task.

11. The method according to claim 10, characterized in that The neural network model is a regression model.

12. The method according to claim 9 or 10, characterized in that: The training set of the neural network model is obtained by Bayesian optimization of the data of the at least one executed task based on conformal prediction.

13. A parameter configuration device, characterized in that: include: A parameter recommendation module, used to determine the nearest neighboring task with the shortest coding distance to the task to be executed from at least one executed task; The parameter recommendation module is also used to use the parameter configuration of the nearest task as the parameter configuration of the task to be executed in the big data platform, so that the big data platform executes the task to be executed based on the parameter configuration of the nearest task.

14. The device according to claim 13, characterized in that The device also includes: The historical task tuning module is used to update the data of the at least one executed task according to the execution result of the task to be executed.

15. A computing device, characterized in that: The computing device includes a memory and a processor, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of any one of the methods described in claims 1-12 are performed.

16. A big data system, characterized in that: The big data system includes a first computing device and a second computing device, wherein the first computing device is used to execute the operation steps of the method described in any one of claims 1 to 12, and the second computing device is used to execute the task to be executed based on the parameter configuration of the nearest task.

17. A readable storage medium, characterized in that: The readable storage medium includes a computer program or instructions. When the computer program or instructions are executed on a computer, the computer is enabled to execute the operation steps of the method described in any one of claims 1 to 12.

Images