Operating parameter recommendation method and apparatus

Abstract

Embodiments of the present application provide an operating parameter recommendation method and apparatus, used for in a big data processing scenario, recommending operating parameters for a computing device on the basis of the resource consumption under different operating parameters, without depending on historical data. The method comprises: first, on the basis of information obtained during the processing of historical tasks in a big data system, establishing a first relationship among the corresponding data information, parameters of a computing device processing the tasks, and time required to process the tasks, and a second relationship among the data information, the parameters, and resources of the computing device consumed for processing the tasks; inputting data information of a task to be processed, and the first relationship and the second relationship to a genetic algorithm, and outputting multiple sets of time values and resource values, i.e., modeling the process of the computing device executing data processing tasks, so as to fit the relationship between the time and resource consumption of the computing device and the operating parameters of the computing device; and determining operating parameters on the basis of actual available resources of the computing device or resource consumption that a user desires.

Description

Method and device for recommending operation parameters

[0001] The present application claims priority to the Chinese patent application No. 202410580888.4, filed on May 10, 2024, and entitled "Method and device for recommending operation parameters", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of big data processing, and in particular to a method and device for recommending operation parameters. BACKGROUND

[0003] Big data processing is a common processing scenario in scenarios requiring large data processing. The computing resources of one or more devices are usually used to perform big data processing. Before performing big data processing, the operation parameters of the one or more devices need to be set, which are parameters configured for the devices when performing tasks, used to support the devices to perform tasks. In some scenarios, historical data generated by big data processing can be used for learning to obtain operation parameters in the current big data processing scenario.

[0004] However, in the face of some cross-month or cross-year scenarios that take a long time to output data, the data processed each time can be different, and the available resources can also be different. Therefore, the operation parameters determined based on historical data can not match the actual data processing scenario, i.e., the operation parameters used by historical tasks do not have reference value for the operation parameters of the current task. Therefore, how to improve the accuracy of the operation parameters used for performing data processing over a long period of time has become a problem to be solved. SUMMARY

[0005] The present application provides a method and device for recommending operation parameters, which are used to recommend operation parameters for computing devices based on resource consumption of different operation parameters without relying on historical data in the big data processing scenario.

[0006] Therefore, in a first aspect, the present application provides a method for recommending operation parameters, comprising: first, establishing a first relationship between data information corresponding to a processing task, parameters of a computing device processing the task, and time required for processing the task, and a second relationship between the processing task and resources consumed by the computing device, according to information in a historical task processing process of a big data system; inputting the data information of the to-be-processed task and the first relationship and the second relationship into a genetic algorithm, and outputting multiple sets of time values and resource values, each set of time values and resource values corresponding to a set of parameters, wherein the specific algorithm processing flow of the genetic algorithm can include: performing multiple transformations on the parameter values of the operation parameters, and performing iterative calculations based on the data information of the to-be-processed task (such as the data volume, the number of data sets, etc. of the to-be-processed task), the first relationship and the second relationship, and outputting the multiple sets of time values and resource values; then determining one set from the multiple sets of time values and resource values according to user requirements, and determining the parameter values of the operation parameters corresponding to the selected time value and resource value, and taking the parameter values as the parameter values of the computing device processing the to-be-processed task; that is, equivalent to modeling the process of the computing device executing the data processing task, thereby fitting the relationship between the time and resource consumption of the computing device and the operation parameters of the computing device by simulating the process of the computing device data processing, so as to determine the operation parameters based on the actual available resources of the computing device or the user expected resource consumption, so that the recommended operation parameters meet the computing power of the computing device or meet the user expectations.

[0007] In the embodiments of the present application, the big data processing process is modeled, thereby simulating the data processing process by modeling, and then fitting the relationship between the consumed time or resources and the operation parameters when the computing device executes the task, so that the time consumption or resource consumption meeting the user requirements and the corresponding operation parameters can be determined based on the relationship, and the operation parameters meeting the user requirements can be recommended for the computing device without relying on historical data.

[0008] In a possible implementation, the to-be-processed task described above can include data extraction, data processing, or data loading, the data extraction model includes a model for extracting the to-be-processed data to the storage space of the computing device, the data processing model includes a model for processing the to-be-processed data according to a pre-set logic, and the data loading model is used to load the model output by the data processing model to a pre-set database. Therefore, in the embodiments of the present application, the operation parameters meeting the resource consumption requirements can be recommended for the computing device in the end-to-end big data processing scene, when the historical data is not referenceable or has low reference value.

[0009] Correspondingly, the operation parameter of the computing device can include at least one of a read batch size, a partition number, an extraction parallelism, a process number, a thread number, a memory size, a write batch size, or a write parallelism, the read batch size including a batch number of data extracted from the data to be processed, the partition number including a number of data partitions to be processed, the extraction parallelism including a number of parallel jobs for extracting the data to be processed to the memory, the process number including a number of processes running when processing the data to be processed, the thread number including a number of threads running when processing the data to be processed, the memory size including a size of memory occupied when processing the data to be processed, the write batch including a batch number of data written to the database after data processing, and the write parallelism including a number of parallel jobs for writing the data to the database after data processing. In the embodiments of the present application, the operation parameter includes parameters of each stage in the end-to-end big data processing of the computing device, and the feasibility of the recommended operation parameter is improved.

[0010] In a possible implementation, the first relationship can be a mathematical model or a machine learning model between the operation parameter of the computing device and the time required for processing the task, and the second relationship can be a mathematical model or a machine learning model between the operation parameter of the computing device and the resources consumed by the computing device for processing the task. Therefore, in the embodiments of the present application, a data model can be constructed, or a machine learning model can be generated by machine learning, so that the model is used to represent the relationship between the operation parameter and the time required for processing the data task, and the relationship between the operation parameter and the consumed resources, so as to subsequently determine the operation parameter meeting the user demand by using the relationship.

[0011] In a possible implementation, the iteration process of the genetic algorithm can include the following steps: performing multiple transformations on the multiple sets of operation parameters output by the last iteration, outputting multiple sets of transformed parameters, obtaining a time value and a resource value corresponding to each set of transformed parameters according to the first relationship and the second relationship; subsequently, filtering multiple sets of Pareto optimal solutions from the multiple sets of transformed parameters according to the time value and the resource value corresponding to each set of transformed parameters, and outputting a Pareto optimal solution set; if the termination iteration condition is met, outputting the Pareto optimal solution set; if the termination iteration condition is not met, taking the Pareto optimal solution set as the input of the next iteration to continue the next iteration. In the embodiments of the present application, when multiple time values and resource values are output based on the genetic algorithm, the transformed parameters can be generated by transformation, and the operation parameters can be optimized by iteration, so that the operation parameters that perform better in the dimensions of time and / or resources are finally output.

[0012] In a possible implementation, the filtering of the plurality of groups of Pareto optimal solutions from the plurality of groups of transformation parameters according to the time value and the resource value corresponding to each group of transformation parameters, and outputting the set of Pareto optimal solutions can include: obtaining a guiding dominance relationship between each group of transformation parameters and other parameters in the plurality of groups of transformation parameters according to the time value and the resource value corresponding to each group of transformation parameters, wherein the evaluation value corresponding to the time value and the resource value of a dominator in the guiding dominance relationship is better than the evaluation value corresponding to the time value and the resource value of a dominated, that is, in the two groups of parameters with the dominance relationship, the parameter with better performance dominates the parameter with worse performance; and filtering a plurality of groups of undominated parameters from the plurality of groups of transformation parameters according to the guiding dominance relationship between each group of transformation parameters and other parameters in the plurality of groups of transformation parameters, and outputting the set of Pareto optimal solutions, which is equivalent to filtering the parameters with better performance from the plurality of groups of transformation parameters, so as to obtain the parameters with better performance in the time dimension or the resource consumption dimension.

[0013] In a possible implementation, the transformation is cross and / or mutation of the parameters, where the cross is to exchange part of the parameter values between two groups of parameters to combine different parameters, and the mutation is to adjust part of the parameter values in the parameters to generate new parameters. Therefore, in the embodiments of the present application, new parameters can be generated through cross and / or mutation, so as to evolve the parameters with better performance in the time dimension or the resource consumption dimension.

[0014] In a possible implementation, in the case where the transformation includes mutation, the processing flow of the mutation includes: mutating the running parameters according to a pre-set parameter range to obtain the mutated running parameters. In the embodiments of the present application, the parameter range for mutation can be pre-set, so that the parameters obtained after mutation are within the constrained parameter range, such as the parameter range supported by the computing device, so as to improve the feasibility of the finally output parameters.

[0015] In a possible implementation, the filtering of the plurality of groups of Pareto optimal solutions from the plurality of groups of transformation parameters according to the time value and the resource value corresponding to each group of transformation parameters, and outputting the set of Pareto optimal solutions can include: obtaining a guiding dominance relationship between each group of transformation parameters and other parameters in the plurality of groups of transformation parameters according to the time value and the resource value corresponding to each group of transformation parameters, wherein the evaluation value corresponding to the time value and the resource value of a dominator in the guiding dominance relationship is better than the evaluation value corresponding to the time value and the resource value of a dominated, that is, in the two groups of parameters with the dominance relationship, the parameter with better performance dominates the parameter with worse performance; and filtering a plurality of groups of undominated parameters from the plurality of groups of transformation parameters according to the guiding dominance relationship between each group of transformation parameters and other parameters in the plurality of groups of transformation parameters, and outputting the set of Pareto optimal solutions, which is equivalent to filtering the parameters with better performance from the plurality of groups of transformation parameters, so as to obtain the parameters with better performance in the time dimension or the resource consumption dimension.

[0016] In a second aspect, the present application provides a running parameter recommendation device, which includes:

[0017] The modeling module is configured to establish, according to information of a historical task processed by the big data system, data information corresponding to the task, a first relationship between an operation parameter of a computing device processing the task and a time required for processing the task, and a second relationship between the operation parameter of the computing device and a resource consumed by the computing device for processing the task.

[0018] The algorithm module is configured to input the data information of the to-be-processed task and the first relationship and the second relationship into a genetic algorithm, where a processing flow of the genetic algorithm includes: performing multiple transformations on parameter values of the operation parameter, and performing iterative calculation based on the data information of the to-be-processed task and the first relationship and the second relationship, to output multiple sets of time values and resource values.

[0019] The selection module is configured to select one set of time values and resource values from the multiple sets of time values and resource values according to a user demand, determine parameter values corresponding to the selected time values and resource values, and use the parameter values as parameter values of the computing device processing the to-be-processed task.

[0020] Effects achieved by the second aspect and any optional implementation of the second aspect can be referred to the description of the first aspect or any optional implementation of the first aspect, which will not be described here.

[0021] In a possible implementation, the to-be-processed task includes data extraction, data processing, or data loading, the data extraction model includes a model of extracting the to-be-processed data to a storage space of the computing device, the data processing model includes a model of processing the to-be-processed data according to a preset logic, and the data loading model is used to load a model output by the data processing model to a preset database. The operation parameter includes at least one of a read batch size, a partition number, an extraction parallelism, a process number, a thread number, a memory size, a write batch size, or a write parallelism. The read batch size includes a batch number of data extracted from the to-be-processed data, the partition number includes a number of data partitions to be processed, the extraction parallelism includes a number of parallel jobs for extracting the to-be-processed data to the memory, the process number includes a number of processes running when the to-be-processed data is processed, the thread number includes a number of threads running when the to-be-processed data is processed, the memory size includes a size of memory occupied when the to-be-processed data is processed, the write batch size includes a batch number of data written to the database after data processing, and the write parallelism includes a number of parallel jobs for writing the data to the database after data processing.

[0022] In a possible implementation, the first relationship is a mathematical model or a machine learning model between the operation parameter of the computing device and the time required for processing the task, and the second relationship is a mathematical model or a machine learning model between the operation parameter of the computing device and the resource consumed by the computing device for processing the task.

[0023] In a possible implementation, the algorithm module performs the process flow of any one iteration in the genetic algorithm, and the process flow specifically includes: performing multiple transformations on the multiple sets of parameters output by the previous iteration, outputting multiple sets of transformed parameters, determining the time value and the resource value corresponding to each set of transformed parameters according to the first relationship and the second relationship, selecting multiple sets of Pareto optimal solutions from the multiple sets of transformed parameters according to the time value and the resource value corresponding to each set of transformed parameters, outputting a Pareto optimal solution set, outputting the Pareto optimal solution set if a termination iteration condition is met, and taking the Pareto optimal solution set as input of the next iteration and performing the next iteration if the termination iteration condition is not met.

[0024] In a possible implementation, the algorithm module is specifically configured to: obtain a guiding dominance relationship between each set of transformed parameters and other parameters in the multiple sets of transformed parameters according to the time value and the resource value corresponding to each set of transformed parameters, where an evaluation value corresponding to the time value and the resource value of a dominator in the guiding dominance relationship is better than an evaluation value corresponding to the time value and the resource value of a dominated, and select multiple sets of non-dominated parameters from the multiple sets of transformed parameters according to the guiding dominance relationship between each set of transformed parameters and other parameters in the multiple sets of transformed parameters, and output the Pareto optimal solution set.

[0025] In a possible implementation, the transformation is a crossover and / or mutation on the parameters.

[0026] In a possible implementation, in the case where the transformation includes mutation, the process flow of the mutation includes: performing mutation on the running parameters according to a pre-set parameter range, to obtain mutated running parameters.

[0027] In a possible implementation, the selection module is specifically configured to: receive user input data, where the user input data includes a parameter selection strategy, and the parameter selection strategy includes parameter selection from a time dimension and / or a resource value consumption dimension; and select one set of time value and resource value from the multiple sets of time value and resource value according to the parameter selection strategy.

[0028] In a third aspect, an embodiment of the present application provides a computing device, including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device performs the method steps in the first aspect and any implementation manner thereof.

[0029] In a fourth aspect, an embodiment of the present application provides a computing device cluster, including at least one computing device, and each computing device includes a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method steps in the first aspect and any implementation manner thereof.

[0030] In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method in the first aspect or any implementation manner of the first aspect.

[0031] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method in the first aspect or any implementation manner of the first aspect.

[0032] In a seventh aspect, an embodiment of the present application provides a chip, including at least one processor and an interface; the at least one processor acquires program instructions or data through the interface; the at least one processor is configured to execute the program instructions to implement the method in the first aspect or any implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0033] FIG. 1 is a schematic diagram of a system architecture provided by the present application;

[0034] FIG. 2 is another schematic diagram of a system architecture provided by the present application;

[0035] FIG. 3 is a schematic diagram of a running parameter recommendation method provided by the present application;

[0036] FIG. 4 is another schematic diagram of a running parameter recommendation method provided by the present application;

[0037] FIG. 5 is a schematic diagram of a big data processing flow provided by the present application;

[0038] FIG. 6 is a schematic diagram of a spark job flow provided by the present application;

[0039] FIG. 7 is another schematic diagram of a running parameter recommendation method provided by the present application;

[0040] FIG. 8 is an example diagram of crossover variation provided by the present application;

[0041] FIG. 9 is a Knee point diagram provided by the present application;

[0042] FIG. 10 is a schematic diagram of a guiding dominance relationship provided by the present application;

[0043] FIG. 11 is a schematic diagram of an application scenario provided by the present application;

[0044] FIG. 12 is another schematic diagram of an application scenario provided by the present application;

[0045] FIG. 13 is a schematic diagram of a client display interface provided by the present application;

[0046] Fig. 14 is a schematic diagram of a client display interface provided by the present application;

[0047] Fig. 15 is a schematic diagram of a structure of a running parameter recommendation device provided by the present application;

[0048] Fig. 16 is a schematic diagram of a structure of a computing device provided by the present application;

[0049] Fig. 17 is a schematic diagram of a structure of a computing device cluster provided by an embodiment of the present application;

[0050] Fig. 18 is a schematic diagram of a structure of another computing device cluster provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0052] First, for the convenience of understanding, some concepts or terms involved in the embodiments of the present application are explained.

[0053] (1) Machine learning

[0054] Construct a statistical model, use an optimization method to fit the model parameters on sample data, and make a prediction on new sample data.

[0055] (2) AutoML: refers to designing a series of high-level control systems to operate machine learning models, so that the models can automatically learn suitable parameters and configurations without human intervention. In a learning model based on a deep neural network, automatic learning mainly includes network architecture search and global parameter setting. Among them, network architecture search is used to generate a neural network architecture that best adapts to the problem according to the data, which has the characteristics of high training complexity and large performance improvement. A machine learning task usually includes a training part and a prediction part. In the prediction part, the parameters of the statistical model can be used to make a prediction on the training sample data, and the update direction of the parameters of the statistical model can be calculated according to the prediction error. Repeat the process until the parameters converge. In the prediction part, the trained model can be used to predict new samples.

[0056] (3) Genetic algorithm (Genetic Algorithm, GA)

[0057] An algorithm based on the evolution law of nature. It is a computational model that simulates the biological evolution process of natural selection and genetics. It searches for the optimal solution by simulating the process of natural evolution. The algorithm uses mathematical methods and computer simulation operations to convert the problem solving process into processes such as chromosome gene crossing and mutation in biological evolution. In solving complex combinatorial optimization problems, it can usually obtain better optimization results faster than some conventional optimization algorithms. In the embodiments of the application, the process of big data processing can be modeled and solved, and in the solving process, the evaluation of the solution is improved by combining the crossing and mutation processes.

[0058] (4) Multi-objective optimization

[0059] Also known as multi-objective programming, it is a branch of mathematical programming. It studies the optimization of more than one objective function in a given region.

[0060] (5) Pareto-optimal solution set (Pareto-optimal front, PoF)

[0061] The so-called Pareto-optimal solution set is that some solutions cannot be further optimized for one or several objectives without degrading other objectives, also known as non-inferior optimal solution set. The Pareto optimal concept is a vector evaluation method for multi-objective solutions based on set theory. Traditional mathematical programming methods and simulated annealing algorithms are characterized by serial algorithms with single-point search, which cannot evaluate solutions using the Pareto optimal concept.

[0062] The method provided by the application can be applied to a big data processing scene. For a big data processing scene, some existing schemes cannot be applied to a data processing scene spanning a long time. In some existing schemes, a reverse optimization scheme can be designed based on the results of a running task. The big data task is suitable for continuous running, and the results of the last running task have reference value. However, for a scene in which data settlement time is relatively long and the current big data processing scene does not have reference value, a scheme that relies on historical data to determine the running parameters of the characters is not applicable.

[0063] For example, in an existing scheme, a task submission interface of a Spark application is encapsulated, it is judged whether there is a hierarchical gray box time prediction model of the Spark application, a database is accessed, the hierarchical gray box time prediction model is read and updated, and it is selected whether to optimize by a user; if optimization is performed, optimization parameters are generated, and if optimization is not performed, the original parameters are executed, and a shell command is called to run the task. However, this scheme is very dependent on historical data, and generates parameters based on a data model trained based on historical data.

[0064] For example, in an existing scheme, a cost-based performance model is constructed by obtaining the configuration of task execution and corresponding cost information, an optimal configuration is obtained in a given parameter space, for unknown types of tasks, a default parameter is run once, and a reference value of the optimal configuration is given by judging the similarity of the tasks. This scheme proposes a cost-based performance model to address the possible problems in current configuration parameter tuning. The performance model is generated by analyzing the historical tasks of Spark, the optimal parameters are obtained by parameter space search algorithm, and the model is continuously upgraded and adjusted with the running of new tasks to increase the accuracy of the model. For unknown types of tasks, a reference value of the parameters is provided after running once. However, in this scheme, the parameter generation based on historical data execution of the parameter space search algorithm is very dependent on historical data, similar to the previous scheme.

[0065] The two aforementioned schemes are very dependent on historical data and can only run in cases where historical data has reference. In some scenarios, there may be a situation of outputting data across a long period of time. For example, in some scenarios involving the financial field, there may be monthly or annual data. For monthly or annual data processing scenarios, the parameters of the computing device need to be configured before processing the data generated by these monthly or annual data. The data volume of the current processing process is different from that of the previous processing task, or the available hardware resources are different, so the running parameters used by the historical task do not have reference for selecting running parameters for the current task.

[0066] Therefore, the present application provides a running parameter recommendation method, which can more accurately determine the running parameters of the computing device to determine the running parameters in the scenario of processing data across a long period of time.

[0067] First, the method provided by the embodiments of the present application can be applied to one or more computing devices. When applied to multiple computing devices, the multiple computing devices can form a computing device cluster, which can include a centralized computing architecture or a distributed computing architecture. For example, in a distributed architecture, the method can be applied to a big data processing architecture based on spark jobs or a big data processing architecture based on flink jobs, etc. Specifically, in a distributed architecture, a server cluster can provide services to users in the form of a client. Users can interact with the server cluster through the client, such as sending data or initiating service requests to the server cluster, etc.

[0068] For example, taking a big data processing architecture based on spark job as an example, as shown in FIG. 1, the architecture 10 can include a client 13 and a server cluster 12, the server cluster can include a plurality of computing nodes, each node can be implemented by one or more servers, and one server can implement one or more nodes. The plurality of nodes can be specifically divided into a driver 121, a cluster manager 122, a spark worker 123, and the like.

[0069] The client 13 can be a program deployed in a user device set, and the user can submit a job request to the server cluster through the client 13.

[0070] The server cluster 12 can process based on the request submitted by the user, and specifically:

[0071] The driver 121 runs the main function of the spark application, creates an application context (SparkContext), and the SparkContext controls the life cycle of the spark application.

[0072] The cluster manager 122, also known as the master, can be specifically one of the servers in the server cluster, or a processor or processing module deployed in one of the servers, and the like. The cluster manager 122 can be used to manage the computing resources in the server cluster, such as allocating the number of computing nodes for each task.

[0073] The spark worker 123 can be specifically a server in the server cluster, or a processor or processing module deployed in the server, such as one of the servers in the cluster. The spark worker 123 can be used to run the computing nodes of the spark application, and the spark worker can be set with a task executor Executor for executing specific tasks (Tasks). The Executor can save the cache and the specific running tasks.

[0074] The big data processing flow based on the spark job can be as shown in FIG. 2. Specifically:

[0075] The client sends a job request to the server cluster, and the driver registers and creates a spark job task (i.e., the corresponding application), and creates a SparkContext to manage the life cycle of the spark job task.

[0076] After the creation of the spark job task is completed, resource scheduling can be performed by the resource manager, and the computing nodes are allocated to the spark job task based on the task executor state of each spark job node, and the spark job node can execute the received spark job task.

[0077] Taking cross-database data processing as an example, that is, processing data in different databases, the process of big data processing can include data extraction, database processing and data loading, etc.

[0078] Among them, data extraction: according to the spark job requirements, the data is extracted from multiple source libraries to the Spark memory.

[0079] Data processing: according to the user's written Spark data processing logic, the data in the Spark memory is calculated, and the final result table is obtained in the Spark.

[0080] Data loading: load the result table in the Spark memory into the target library.

[0081] Before the server cluster executes the spark job task, it is usually necessary to configure the device running parameters, such as configuring the memory resources, CPU resources, etc. for the computing nodes.

[0082] The method provided by the application can recommend running parameters for computing devices based on more optimal targets, such as resource consumption targets or time consumption targets, etc. before big data processing. The method provided by the application can be deployed in a computing device, which can include a standalone server or a server in the server cluster 12, such as the resource manager 122 or the driver 121, etc.

[0083] The method flow provided by the application will be introduced below in combination with the foregoing system architecture.

[0084] Participate in Figure 3, the flow chart of a running parameter recommendation method provided by the application, as follows.

[0085] 301, model with historical task data information, output data information, first relationship between running parameters and time, and second relationship between data information, running parameters and resources.

[0086] In the process of processing historical tasks in a big data processing system, the data involved includes: historical task data information, computing device operating parameters, processing time, and resource consumption. Specifically, the historical task data information may include the amount of data to be processed from historical figures, the number of datasets, or data categories. The computing device operating parameters may include the parameters configured for executing the historical task, such as the number of processes and batch sizes. The computing device resources may include metrics used to measure resource consumption, such as CPU resource consumption, memory consumption, or input / output (I / O) consumption.

[0087] Data from historical tasks executed by a big data processing system can be used to create a model representing the task processing flow. There can be one or more historical tasks. By utilizing relevant data from the execution of one or more historical tasks using computing devices, the data information, operating parameters, duration, and resources of the historical tasks can be modeled. This model outputs the primary relationship between data information, operating parameters, and time, as well as the secondary relationship between data information, operating parameters, and resources.

[0088] Specifically, when performing modeling, either a mathematical model or a machine learning model can be used.

[0089] In the process of modeling using mathematical models, relationships between historical task data, operational parameters, duration, and resources can be fitted. Specifically, the relationships between data, operational parameters, and duration, as well as the relationships between data, operational parameters, and resources, can be fitted separately, thus outputting a first relationship and a second relationship respectively. When using data models for modeling, in the subsequent process of determining operational parameters for the task to be processed, the data information of the task to be processed can be used as input to the first and second relationships, outputting multiple sets of time and resources, as well as the corresponding parameters.

[0090] In the process of using machine learning models for modeling, the initial structure of the machine learning model can be defined, such as setting the input and output channels. Machine learning is then performed using data, runtime parameters, duration, and resources from historical task execution, iteratively updating the model to obtain a usable trained model. During the subsequent determination of runtime parameters for the task at hand, the data from the task can be used as input to the machine learning model, outputting multiple sets of time and resource information, along with corresponding parameters.

[0091] 302. Input the data information of the task to be processed, the first relation and the second relation into the genetic algorithm, and output multiple sets of time values ​​and resource values.

[0092] Firstly, the to-be-processed task and data information of the to-be-processed task are introduced.

[0093] The to-be-processed task can be a task to be executed by the computing device, and specifically can be a task executed on to-be-processed data. The data information of the to-be-processed task can include information of to-be-processed data to be processed when the computing device executes the to-be-processed task, such as data volume, data set quantity, and the like of the to-be-processed data.

[0094] From the dimension of the steps of executing the task, the to-be-executed task can be specifically divided into one or more steps: data extraction, data processing, or data loading, and the like. The data extraction model includes a model for extracting to-be-processed data to a storage space of the computing device. The data processing model includes a model for processing to-be-processed data according to a pre-set logic. The data loading model is used to load a model output by the data processing model to a pre-set database.

[0095] Correspondingly, the running parameters configured by the computing device for processing the to-be-processed task can include but are not limited to one or more of a read batch quantity, a partition quantity, an extraction parallel degree, a process quantity, a thread quantity, a memory size, a write batch quantity, or a write parallel degree. The read batch quantity includes a batch quantity of extracting data from to-be-processed data. The partition quantity includes a quantity of partitions of to-be-processed data. The extraction parallel degree includes a quantity of parallel jobs for executing extraction of to-be-processed data to a memory. The process quantity includes a quantity of processes running when to-be-processed data is processed. The thread quantity includes a quantity of threads running when to-be-processed data is processed. The memory size includes a size of memory occupied when to-be-processed data is processed. The write batch includes a batch quantity of writing data obtained after data processing to a database. The write parallel degree includes a quantity of parallel jobs for writing data obtained after data processing to a database.

[0096] The to-be-processed data can be different with different to-be-processed tasks, and the logic for processing the to-be-processed data can also be different with different processing tasks. For example, in a tax rate calculation task, the to-be-processed data can include income information and tax additional information of multiple users. In an enterprise financial report calculation task, the to-be-processed data can include financial information of an enterprise, such as information of enterprise working capital, flow load, quick assets, total liabilities, and total assets.

[0097] For example, in some possible scenarios, the method provided by the embodiments of the present application can be applied to a tax rate calculation task scenario, such as an application scenario in which a certain enterprise calculates the personal tax rate of an employee on a monthly basis. The data to be processed can specifically include information that can be used to calculate the tax rate of an employee, such as the salary income, length of service, cumulative income this year, social security level, social security amount to be paid, lunch allowance, transportation allowance, housing allowance, comprehensive allowance, sick leave, casual leave, maternity leave, lateness / early departure, or absence, and the like.

[0098] Secondly, the specific algorithm process based on the genetic algorithm is introduced.

[0099] Specifically, the processing process based on the genetic algorithm can include: performing multiple transformations on the parameter values of the running parameters, and performing iterative calculation based on the data information of the to-be-processed task, and the first relationship and the second relationship, and outputting multiple sets of time values and resource values. In the embodiments of the present application, the parameters that perform better in the time or resource dimension can be evolved based on the genetic algorithm.

[0100] Taking any one of the iterative processing processes as an example: performing multiple transformations on the multiple sets of parameters output by the last iteration, and outputting multiple sets of transformed parameters; determining the time value and the resource value corresponding to each set of transformed parameters according to the first relationship and the second relationship; selecting multiple sets of Pareto optimal solutions from the multiple sets of transformed parameters according to the time value and the resource value corresponding to each set of transformed parameters, and outputting a Pareto optimal solution set; if the termination iteration condition is met, outputting the Pareto optimal solution set; if the termination iteration condition is not met, taking the Pareto optimal solution set as the input of the next iteration, and performing the next iteration processing.

[0101] Therefore, in the embodiments of the present application, the initial parameters can be mutated based on the crossover and mutation in the genetic algorithm, so that the running parameters that perform better in the time or resource dimension are evolved, so that the running parameters that better meet the user's needs can be selected subsequently.

[0102] Further, in the process of screening the Pareto optimal solution, the guiding dominance relationship between each set of transformed parameters and other parameters in the multiple sets of transformed parameters can be obtained according to the time value and the resource value corresponding to each set of transformed parameters, the evaluation value corresponding to the time value and the resource value of the dominator in the guiding dominance relationship is better than the evaluation value corresponding to the time value and the resource value of the dominated, and then multiple sets of parameters that are not dominated are selected from the multiple sets of transformed parameters according to the guiding dominance relationship between each set of transformed parameters and other parameters in the multiple sets of transformed parameters, and a Pareto optimal solution set is output.

[0103] It can be understood that, in the process of determining the guiding dominance relationship between the parameters, a coordinate system can be established from the time and resource dimensions, and a dominance area of each group of parameters is constructed in the coordinate system. The dominance area is an area in which the time dimension and the resource dimension corresponding to the parameters are not superior to the group of parameters. In other words, the parameters perform better in the time dimension and the resource dimension in the dominance area, and the group of parameters is the Pareto frontier in the dominance area.

[0104] In the embodiments of the present application, in the process of outputting the Pareto optimal solution set, each group of running parameters can be evaluated to obtain an evaluation result corresponding to each group of running parameters. The evaluation result is used to evaluate the degree to which the consumed time or resource satisfies the optimization target when each group of running parameters is configured on at least one computing device. The optimization target is to minimize the time value and / or the resource value. A group of running parameters is determined according to the evaluation result corresponding to each group of running parameters. Therefore, in the embodiments of the present application, the degree to which the consumed resource of each group of running parameters satisfies the optimization target when the running parameters are configured to the computing device to execute the to-be-processed task can be evaluated, so that the running parameters that better satisfy the optimization target can be finally screened out.

[0105] Therefore, in the embodiments of the present application, the parameters that perform better in the time dimension and the resource dimension can be screened out based on the guiding dominance relationship between the parameters, which is equivalent to screening out the parameters on the Pareto frontier. The parameters that perform better are output, and the search for the parameters that perform better in the time dimension and the resource dimension is realized.

[0106] Optionally, the foregoing transformation can include crossover and / or mutation. The crossover is to exchange different types of parameter values between two groups of parameters to output new parameters. The mutation is to adjust one or more values in a group of parameters to output new parameters. Therefore, in the embodiments of the present application, the parameters can be evolved through crossover or mutation, so that the parameters that perform better can be evolved, so that the parameters that better satisfy the user demand can be obtained subsequently.

[0107] For example, at least one of crossover or mutation is performed on the n individuals output in the last iteration to obtain m individuals, each individual representing a group of running parameters. The m can be a value greater than n, such as 2n or 3n. The crossover can be understood as cutting two individuals at a crossover point and recombining to obtain new individuals. The mutation is to change the gene points in the individual to obtain new individuals. The Pareto frontier is screened out from the m individuals based on the time value and the resource value corresponding to each individual, that is, the better n individuals are output for the next iteration.

[0108] Further, in the case that the foregoing transformation includes variation, when one or more values in the parameters are varied, the variation can generally be performed according to a preset parameter range, which can specifically include a parameter range supported by the computing device, or a range determined according to experience values, etc., so that the generated parameters after variation are within the constrained parameter range, and the output parameters are more practical.

[0109] 303. Select one of the groups of time values and resource values according to the user demand, and take the parameter values corresponding to the group of time values and resource values as the parameters of the computing device for processing the to-be-processed task.

[0110] After the foregoing step 302 is performed, the groups of time values and resource values, and the parameters corresponding to each group of time values and resource values can be output. That is, the groups of time values and resource values can be filtered based on the user demand, the time values and resource values meeting the user demand can be selected, the corresponding parameter values can be extracted, and the parameters of the computing device can be configured.

[0111] In the embodiments of the present application, the process of executing the task by the computing device can be modeled, so as to determine the relationship between the running parameters configured by the computing device and the resource consumption, i.e., the first relationship or the second relationship, based on the modeling, so as to determine the running parameters meeting the demand of the resource consumption by using the relationship. Therefore, even in the data processing scenario spanning a long time period, the running parameters meeting the demand of the resource consumption can be determined by using the modeling, without relying on historical data to determine the running parameters.

[0112] In a possible implementation, the user input data can be specifically received, and the user input data includes a running parameter selection strategy. When a group of running parameters is selected, one of the groups of time values and resource values is selected according to the parameter selection strategy, and the corresponding parameters are determined, such as selecting parameters with less required time, or selecting parameters with less resource consumption, or selecting parameters with more balanced required time and resource consumption, etc. Therefore, the running parameters meeting the user demand can be selected based on the actual preference strategy of the user. For example, the optimization target can include optimization targets in two dimensions of time and hardware resources, and the user can select whether to prefer running parameters with less time consumption, or prefer running parameters with less hardware resource consumption, etc.

[0113] The method process provided by the present application is introduced above, and the method process provided by the present application is further introduced below in combination with a specific application scenario.

[0114] Exemplarily, the method provided in the application is exemplarily introduced below taking the example of a big data architecture based on a spark job. In the spark job scenario, a job task can be divided into one or more independent tasks. When divided into multiple tasks, the multiple tasks can be executed in each computing node in synchronization or in series.

[0115] The method provided in the application can be divided into multiple stages, as shown in FIG. 4, and can be specifically divided into constructing a mathematical model of a spark running parameter and ETL (extract-transform-load) data processing, spark running parameter searching, etc. The steps of each stage are introduced in detail below.

[0116] Stage one, constructing a mathematical model of a spark running parameter and ETL data processing

[0117] First, the data processing flow is introduced.

[0118] The spark job flow can be as shown in FIG. 5. The job flow can specifically include data extraction, data processing and data loading, etc.

[0119] The data extraction: according to the spark job demand, data is extracted from multiple source libraries to the Spark memory.

[0120] The data processing: according to the Spark data processing logic (such as data processing logic, business logic) written by the user, the data in the Spark memory is calculated, and the final result table is obtained in the Spark.

[0121] The data loading: the result table in the Spark memory is loaded into the target library.

[0122] And for different steps, before the computing device processes, the corresponding running parameters need to be configured based on different environment parameters, that is, the data information corresponding to the task.

[0123] For example, in the data extraction stage, the environment parameters and the running parameters to be configured can be as shown in Table 1:

[0124] Table 1

[0125] In the data extraction stage, the spark job faces three source libraries for data extraction, and the data volume of each source library is different, which constitutes the environment parameters of the current job. The extraction batch, the partition number and the parallelism degree set in the data extraction are one of the running parameters of the spark job, which determines the efficiency and resource consumption of the data extraction.

[0126] For example, in the data processing stage, the environment parameters and the running parameters to be configured can be shown in Table 2:

[0127] Table 2

[0128] In the data processing stage, the environment parameters include the amount of data to be processed and the number of operator types used for processing the data to be processed, and for the spark running parameters, the process number, thread number and corresponding memory size can be set to support the computing device to process the data to be processed.

[0129] For example, in the data loading stage, the environment parameters and the running parameters to be configured can be shown in Table 3:

[0130] Table 3

[0131] In the data loading stage, the environment parameters include the amount of output data obtained after data processing, and the running parameters to be configured include the write batch size and the write parallelism for loading the output data to the target library.

[0132] Secondly, the modeling process is introduced.

[0133] For example, according to the job requirements, the optimization target can be set as time consumption and resource consumption, aiming to find a set of running parameters to make the time and resource consumption of the Spark job as small as possible. Among them, resource consumption and time consumption are two mutually exclusive targets, so the running parameters need to consider the balance between the two targets. In order to better express the task target through mathematical model, the symbols in Table 4 can be mathematically modeled.

[0134] Table 4

[0135] Correspondingly, the constructed model can be shown in Figure 6. The model is divided into three stages, namely data extraction, data processing and data loading, and the environment parameters, running parameters and target information involved in the three processes of the Spark job are shown in Figure 6. Among them, the environment parameters involved in the execution history task specifically include the data in the data extraction, data processing and data loading stages, such as the data information of each stage (such as the total number of records of the master table, the data amount table involved in encryption and decryption), the Spark running parameters (such as the Spark task execution unit, the Spark memory execution unit, the parallelism, etc.), the time consumption of each stage (such as the columns shown in Figure 6, such as data reading (s), decryption (s), processing (s), encryption (s), writing (s), total time consumption (s) or JOB time consumption (s) of the process, etc.), and the resource consumption (such as the core consumption, memory consumption, etc. shown in Figure 6).

[0136] In the data extraction stage, the relationship between the environment parameters (data information from historical tasks), Spark running parameters and the target is constructed, the Spark running parameters of the data extraction stage can include the extraction batch size, the partition number, the extraction parallelism, etc., and the target includes the time consumption and the resource consumption, i.e. the relationship between the environment parameters, the Spark running parameters of the data extraction stage and the time consumption and the resource consumption is constructed by using the data of the data extraction stage in the process of executing the historical tasks; in the data processing stage, the environment parameters include the data size, the Spark running parameters include the process number, the thread number and the memory, and the target also includes the time consumption and the resource consumption, i.e. the relationship between the environment parameters, the Spark running parameters of the data processing stage and the time consumption and the resource consumption is constructed by using the data of the data processing stage in the process of executing the historical tasks; in the data transfer stage, the environment parameters include the data size to be loaded, the Spark running parameters include the write batch size and the parallelism, etc., and the target also includes the time consumption and the resource consumption, i.e. the relationship between the environment parameters, the Spark running parameters of the data loading stage and the time consumption and the resource consumption is constructed by using the data of the data loading stage in the process of executing the historical tasks.

[0137] Since the Spark job is composed of multiple processes, the mathematical representation of the two targets is the sum of each stage, and the optimization target can be represented as follows:

[0138] Target 1 (time consumption, or called time consumption):

[0139] Target 2 (resource consumption):

[0140] Among them, target 1 represents the time consumption of the Spark task running in the (t) environment with parameters, t1 represents the time consumption of the data extraction stage, t2 represents the time consumption of the data processing stage, and t3 represents the time consumption of the data loading stage, and this target is usually smaller, i.e. the time taken by the computing device to execute the task is shorter. Similarly, target 2 represents the resource consumption of the Spark task running in the (t) environment with parameters, which includes the weighted value of multiple resource consumptions, wherein, n α *α represents the weighted value of CPU consumption, n β *β represents the weighted value of IO consumption, and n γ *γ represents the weighted value of memory consumption, and the smaller the target is, the less resource the computing device consumes to execute the task.

[0141] Based on the above mathematical representation and sample, a mathematical model of Spark running parameters and ETL data processing is constructed. As shown in FIG. 5, the input of the mathematical model is the Spark job running parameters (data extraction batch, process number, memory number, etc.), the environment parameter is the running environment (data volume, data set number, etc.), and the output is the target (time consumption and resource consumption).

[0142] Further, in constructing the data model, various modeling methods can be used, such as modeling using data relationship or using a machine learning model.

[0143] Among them, mathematical relationship modeling: based on the information of historical tasks to set corresponding parameters to express the mathematical relationship between input and output, and the parameters are affected by the environment. It is equivalent to constructing the data relationship under each target based on the information of historical tasks, and expressing the relationship between the environment parameters, Spark running parameters and targets in each stage through the data relationship. In the following spark running parameter search stage, the mathematical relationship is used, the data information of the task to be processed is taken as the input of the data relationship, and multiple groups of Spark parameters and the time consumption and resource consumption corresponding to each group of Spark parameters are output after calculation.

[0144] Machine learning model: based on historical information, 8 Spark running parameters, 2 target outputs and 3 environment parameters in historical tasks are used for machine learning, and a trained machine learning model is output. The machine learning model can select a basic machine learning model, such as K-Nearest Neighbor (KNN) classification algorithm, eXtreme Gradient Boosting (XGBoost), random forest and other machine learning models. In the following spark running parameter search stage, the data information of the task to be processed can be directly taken as the input of the machine learning model, and multiple groups of Spark parameters and the time consumption and resource consumption corresponding to each group of Spark parameters are output.

[0145] Of course, the optimization target mentioned above can also be set as stability or other targets related to actual business scenarios, etc., which are only exemplary introduced here.

[0146] Stage two, spark running parameter search

[0147] In the running parameter search stage, the embodiment of the application can search the user preferred spark running parameters based on the knee-oriented genetic algorithm.

[0148] In the running parameter search stage, the application provides a parameter selection method based on genetic algorithm. For example, the flow of parameter search can be as shown in FIG. 7.

[0149] 701、constructing an initial population P t (g).

[0150] Based on the modeling output result of the foregoing stage one, a constructed spark running parameter and a mathematical model of ETL data processing can be obtained. The initial population P t (g) can be obtained based on the constructed spark running parameter and the mathematical model of ETL data processing in the foregoing stage one. For example, the information of the to-be-processed data of the to-be-processed task can be input into the foregoing mathematical relationship or machine learning model, and a plurality of groups of parameters and the time consumption and resource consumption corresponding to each group of parameters are output.

[0151] Specifically, the data information of the to-be-processed data can be taken as an environmental parameter, and the environmental parameter (for example, information such as a to-be-processed data amount, a data set number, a data amount, a data processing operator number, and a data loading data amount) can be input into the foregoing mathematical relationship or machine learning model, and a plurality of individuals and the time consumption and resource consumption corresponding to each individual are output, that is, the initial population P t (g) can be composed. t (g) contains n individuals, each individual represents a group of spark running parameters, and each group of spark running parameters has corresponding time consumption and resource consumption.

[0152] 702、outputting a new population based on a knee-oriented genetic algorithm iteration.

[0153] In the initial stage, the n individuals in the initial population P t (g) can be subjected to evolution operations such as crossover and / or mutation to generate a new offspring population containing 2n individuals; then, better individuals are screened based on a knee-oriented dominance relationship, and n elite individuals are reserved to generate a new population, that is, an elite offspring population. In the case where the convergence condition is not met, iteration is continued, that is, the evolution operations such as crossover and / or mutation are repeatedly performed on the new population, until the convergence condition is met, and the new population is output.

[0154] The step 702 is described below.

[0155] 1、Individuals in the population

[0156] Exemplarily, the individuals can be as shown in Table 5,

[0157] Table 5

[0158] Among them, the mathematical model running parameters established according to the foregoing stage one are divided into 3 parts, a total of 8 parameters. For example, the Spark running parameters included in data extraction can include: extraction batch size, partition number, extraction parallelism; the Spark running parameters included in data processing can include: process number, thread number, memory; the Spark running parameters included in data loading can include: write batch, parallelism. Among them, data extraction is divided into multiple groups of parameters, the number of groups is determined by the data extraction source, and 3 data sources are involved in the example, so data extraction is divided into 3 groups.

[0159] 2. Crossover and mutation

[0160] In the genetic algorithm, the purpose of crossover is to generate new excellent individuals while preserving the excellent genes of the parents. The purpose of mutation is to expand the search range of the solution, retain a certain randomness for the offspring inheritance, and avoid falling into local optimum.

[0161] In the process of Spark running parameter crossover and mutation, the crossover and mutation logic can be as shown in FIG. 8, and specifically as follows:

[0162] Crossover: randomly select two parents in the population, randomly select parameters for crossover, and combine new offspring. In the example, 2 groups of parameters of data extraction and 2 parameters of data loading of parent 1 and 1 group of parameters of data extraction and 3 parameters of data processing of parent 2 are selected for crossover to form offspring.

[0163] Mutation: the offspring selects random parameters and randomly mutates within the allowed parameter range. In the example, the memory parameter in the offspring is selected for mutation, and the inherited 16 from parent 2 is mutated to 32. Among them, the mutation range of each parameter is limited within the effective parameter range to prevent useless mutation.

[0164] 3. Optimal selection based on knee-oriented dominance relationship

[0165] First, introduce the Knee points.

[0166] Knee points are natural preferred solutions in the Pareto-optima front (PoF). Knee points are usually the most prominent solutions in PoF because they can significantly benefit most objectives by reasonably sacrificing a small part of the objective fitness. In other words, for multi-objective problems without clear preferences, knee points can perform better than their field solutions.

[0167] For example, Fig. 9 is a Knee point illustration, where the solid line represents the PoF, and the dots B and C represent solutions in the PoF, and the dot A represents the knee point in the PoF. From a geometric perspective, the knee point A achieves a significant improvement in objective 2 at the expense of a small improvement in objective 1 compared to solution B. Similarly, the knee point A achieves a significant improvement in objective 1 at the expense of a small improvement in objective 2 compared to solution C. From a performance index perspective, knee points tend to achieve the best utility and hyper-volume. A set of predefined reference vectors can be used to decompose the objective space into a series of subspaces, and a local alpha dominance relationship can be used to drive the search process to search for potential knee regions in the subspaces to accurately locate the knee solutions and eliminate the effects of interfering solutions.

[0168] Therefore, embodiments of the present application provide an individual selection based on knee-oriented dominance relationship, which can maximize the natural preference solution by reasonably using computing resources and narrowing the solution range to achieve the maximum disassembly benefit. Assuming that there are solution A and solution B in the problem space, the decisive index μ in the knee-oriented dominance relationship is defined as follows:

[0169] wherein,

[0170] is the acute angle determined by the fitness of the i-th objective in solution A, maxf i (E) is the extreme value of the i-th objective fitness in the population. N id is the reference point, and ε is a very small positive number to ensure that the denominator is not zero.

[0171] f j (A) is the defined fitness of the j-th objective, τ ∈ [0, 1] is a key parameter that controls the knee-oriented dominance region. If μ(A, B) < 0 represents that solution A dominates solution B through the knee-oriented relationship. If μ(A, B) and μ(B, A) are both greater than zero, it means that solution A and solution B cannot dominate each other through the knee-oriented relationship. The shaded areas in the knee-oriented dominance region of solutions A, B and C in Fig. 10 represent the knee-oriented dominance region of solutions A, B and C, respectively. Solution A and solution B cannot dominate each other through the knee-oriented relationship, while solution C is dominated by the knee-oriented relationship of solutions A and B. Moreover, the closer a solution is to the reference point, the larger its dominance region is, and the more likely it is to become a knee point. The solution is less likely to be dominated by other solutions. The expanded dominance region through the knee-oriented relationship can more accurately locate the knee points and eliminate the effects of interfering solutions.

[0172] Therefore, the specific steps of the individual selection process based on the knee-oriented dominance relationship can be represented by the following pseudo code:

[0173]

[0174] Wherein, each row can represent:

[0175] Lines 1-2: For P t (g), perform crossover and mutation to generate offspring population Q t (g), and merge P t (g) and Q t (g) to generate population R t (g);

[0176] Line 3: Sort the individuals in population R t (g) according to the knee-oriented relationship, and divide population R t (g) into multiple front faces according to the dominance relationship between individuals, and assign each individual a front face number;

[0177] Lines 4-6: Take out individuals according to front face number in ascending order to form P t (g+1) layer by layer until the edge front face is reached. The concept of edge front face is proposed, which is the last front face that forms P t (g+1). Since the number of individuals contained in the edge front face is more than the remaining population space of P t (g+1), only part of the individuals will be selected;

[0178] Lines 8-9: Select individuals from edge front face L t in ascending order of RS to join population P t (g+1). The purpose of this is to select individuals with excellent average fitness while retaining individuals with excellent stability.

[0179] Therefore, in the embodiments of the present application, the knee-oriented genetic algorithm is used to evolve individuals and select elite individuals, so that individuals with better evaluation are obtained as the number of iterations increases, and the final output can be a better individual, i.e. the running parameters of the computing device. According to the algorithm combined with the mathematical model, the Spark running parameters that meet the target can be searched. The algorithm can quickly converge to the knee points that maximize the benefits, and has better performance for multi-objective problems. Therefore, in the future, when there is a demand for optimization of more targets in the process of system evolution, the current two targets can be expanded to multiple targets, which also has good adaptation ability.

[0180] 703、Select spark running parameters based on user preferences.

[0181] After the final population iteration is completed, a set of parameters is selected for running according to a user preference strategy. The user preference strategy can be input by the user at the client side.

[0182] For example, the user preference strategy can include:

[0183] Strategy 1: Select the parameter with the least time consumption

[0184] Strategy 2: Select the parameter with the least resource consumption

[0185] Strategy 3: Select the parameter with a balance between time and resource consumption

[0186] Therefore, multiple preference settings can be provided for the user without the user configuring parameter details, the user can set the preference strategy, and the user can adapt to different optimization strategies to achieve automatic parameter optimization and adaptation of Spark jobs.

[0187] In the embodiments of the present application, a knee-oriented genetic algorithm is used, which can quickly converge to the knee points of maximum benefit and has better performance for multi-objective problems. Therefore, in the future, when there is a need to optimize more targets in the process of system evolution, the current two targets can be extended to multiple targets, and the system also has good adaptation ability.

[0188] For the scenario where the big data Spark job is not frequently scheduled, for example, the monthly / annual settlement of finance, which results in the historical data not having reference value, the big data Spark job needs to obtain the best parameters before running to achieve the goal of minimizing the time consumption and resource consumption of the job. By using the method provided in the embodiments of the present application, a mathematical model is constructed and a knee-oriented genetic algorithm is used to search for the Spark running parameters preferred by the user. The method can be applied to the scenario of non-frequent Spark tasks that need to be run in the forward optimization mode.

[0189] For ease of understanding, the effects achieved by the embodiments of the present application are further introduced as follows.

[0190] For example, for the financial top secret data processing scene, the method provided by the embodiment of the present application can be applied to the data encryption and decryption service based on Spark. The specific process can be as shown in FIG. 11. The process involves data extraction, data processing, data loading and other main processes. The encryption and decryption service performs encryption and decryption of top secret data during data extraction and data loading based on encryption and decryption algorithms, thereby achieving the requirement of top secret data processing. In the data extraction stage, data can be extracted from different databases, such as the data warehouse service (DWS) database and the hadoop distributed file system (HDFS) database as shown in FIG. 11. Each database can include top secret data and non-top secret data. The non-top secret data is data that cannot be leaked. For non-top secret data, the data can be directly extracted. For top secret data, a decryption algorithm can be added in the embodiment of the present application to process the top secret data to output non-top secret data and top secret data. In the data processing stage, various encrypted data are processed to output the processing result. In the data loading stage, data that needs to be encrypted, such as data obtained after decryption of top secret data, can be encrypted. The unencrypted data and the encrypted data are loaded into the target database, thereby avoiding leakage of top secret data by encrypting the data that needs to be encrypted.

[0191] And the data encryption and decryption in the Spark memory ensures the efficiency of data processing under the premise of ensuring data security. The parameters and resource consumption of the method provided by the application applied in the encryption performance test are shown in FIG. 12. Among them, different data volume scenarios can be tested, and environment parameters and running parameters in the data extraction stage, data processing stage and data loading stage are set respectively. Among them, different environment parameters can be set based on different data volumes; the running parameters are set to adapt to the parameters in different stages, such as in the data extraction stage, the configured running parameters include data extraction stage parameters (extraction batch size, partition number), data processing stage parameters (process number, thread number, CPU, shuffle parallelism) and data loading stage parameters (write batch size), and the corresponding resource consumption conditions can include spark computing time consumption, task preparation time consumption and scheduling total time consumption, etc. Of course, time resource consumption is taken as an example for introduction here, and actual scenarios can also be replaced by hardware resources or other resources, etc. According to the parameter changes under different data volumes in FIG. 12, it is obvious that the timeliness of Spark job is greatly affected by parameters. Therefore, the data encryption and decryption job based on Spark needs to set reasonable parameters before running to ensure the balance between running timeliness and resource consumption. Through the method provided by the application embodiment, the timeliness and resource consumption can be set as optimization targets, so as to filter out the running parameter scheme meeting the actual demand.

[0192] Generally, the scene of financial data encryption and decryption is often for financial monthly / yearly settlement scene, and many Spark jobs run once every month / quarter / year to meet the monthly / yearly settlement demand. The data volume changes greatly every month / year, which leads to that the fixed parameter setting cannot adapt to all scenarios, and the historical job running results do not have reference value due to different environments (data volume). Therefore, before the task runs, the method provided by the application embodiment can search the user's preferred Spark running parameters based on the knee-oriented genetic algorithm without relying on historical data for reverse adjustment.

[0193] And, a more concise and easy-to-operate client display interface can be provided for the user, that is, in the configuration stage, the recommended running parameters can be realized, or the running parameters can be manually set by the user through the rule configuration page for adjustment. For example, the client display interface can be as shown in FIG. 13 and FIG. 14. In the client interface shown in FIG. 13 and FIG. 14, a scheme for recommending running parameters for the user is displayed. For example, in the interface in FIG. 13, for the parameters to be configured, such as the extraction size, the partition field, the partition quantity, the partition lower limit or the partition upper limit, the recommended configuration parameters are displayed. In FIG. 14, the process number, the thread number, the running memory, the batch size, the parallelism and other parameters to be configured display the recommended configuration parameters, so as to realize the automatic configuration of the running parameters, provide a more easy-to-operate client for the user, and improve the configuration experience of the user.

[0194] Therefore, the embodiment of the present application provides a mathematical model for constructing the Spark running parameters and the ETL data processing in the Spark job scenario. Based on the mathematical model, the knee-oriented genetic algorithm can be used to search the Spark running parameters preferred by the user, so as to realize the positive parameter recommendation for the Spark job which cannot reuse the historical data information.

[0195] The foregoing introduces the method flow provided by the present application. The device for executing the method flow provided by the present application is introduced as follows.

[0196] Referring to FIG. 15, a structural schematic diagram of a running parameter recommendation device provided by the present application is provided. The running parameter recommendation device comprises:

[0197] The modeling module 1501 is configured to establish a first relationship between the running parameters of the computing device for processing the task and the time required for processing the task and a second relationship between the running parameters of the computing device for processing the task and the resources consumed by the computing device for processing the task according to the information of the historical tasks processed by the big data system.

[0198] The algorithm module 1502 is configured to input the data information of the to-be-processed task and the first relationship and the second relationship into the genetic algorithm. The processing flow of the genetic algorithm comprises: performing multiple transformations on the parameter values of the running parameters, and performing iterative calculation based on the data information of the to-be-processed task and the first relationship and the second relationship, and outputting multiple sets of time values and resource values.

[0199] The selection module 1503 is configured to select one set of time values and resource values from the multiple sets of time values and resource values according to the user demand, determine the parameter values corresponding to the selected time values and resource values, and take the parameter values as the parameter values of the computing device for processing the to-be-processed task.

[0200] In a possible implementation, the to-be-processed task comprises data extraction, data processing, or data loading, the data extraction model comprises a model corresponding to extraction of to-be-processed data to a storage space of the computing device, the data processing model comprises a model for processing the to-be-processed data according to a preset logic, and the data loading model is used for loading a model output by the data processing model to a preset database. The running parameter comprises at least one of a read batch size, a partition quantity, an extraction parallelism, a process quantity, a thread quantity, a memory size, a write batch size, or a write parallelism. The read batch size comprises a batch quantity of data extracted from the to-be-processed data. The partition quantity comprises a quantity of partitions of the to-be-processed data. The extraction parallelism comprises a quantity of parallel jobs for extracting the to-be-processed data to the memory. The process quantity comprises a quantity of processes running when the to-be-processed data is processed. The thread quantity comprises a quantity of threads running when the to-be-processed data is processed. The memory size comprises a size of memory occupied when the to-be-processed data is processed. The write batch comprises a batch quantity of data written to the database after the data is processed. The write parallelism comprises a quantity of parallel jobs for writing the data to the database after the data is processed.

[0201] In a possible implementation, the first relationship is a mathematical model or a machine learning model between the running parameter of the computing device and the time required for processing the task, and the second relationship is a mathematical model or a machine learning model between the running parameter of the computing device and the resource consumed by the computing device for processing the task.

[0202] In a possible implementation, the algorithm module 1502 performs any iteration process in the genetic algorithm, specifically comprising: performing a plurality of transformations on a plurality of sets of parameters output by a previous iteration, outputting a plurality of sets of transformed parameters, determining a time value and a resource value corresponding to each set of transformed parameters according to the first relationship and the second relationship; selecting a plurality of sets of Pareto optimal solutions from the plurality of sets of transformed parameters according to the time value and the resource value corresponding to each set of transformed parameters, and outputting a Pareto optimal solution set; if a termination iteration condition is met, outputting the Pareto optimal solution set; and if the termination iteration condition is not met, taking the Pareto optimal solution set as an input of a next iteration and performing the next iteration.

[0203] In a possible implementation, the algorithm module 1502 is specifically configured to: according to the time value and the resource value corresponding to each set of transformed parameters, obtain a guiding dominance relationship between each set of transformed parameters and other parameters in the plurality of sets of transformed parameters, wherein an evaluation value corresponding to the time value and the resource value of a dominator in the guiding dominance relationship is better than an evaluation value corresponding to the time value and the resource value of a dominated; and according to the guiding dominance relationship between each set of transformed parameters and other parameters in the plurality of sets of transformed parameters, select a plurality of sets of non-dominated parameters from the plurality of sets of transformed parameters, and output the Pareto optimal solution set.

[0204] In a possible implementation, the foregoing transformation is cross and / or mutation of the parameters.

[0205] In a possible implementation, in the case where the transformation includes mutation, the processing procedure of the mutation includes: mutating the running parameters according to a preset parameter range to obtain mutated running parameters.

[0206] In a possible implementation, the selection module 1503 is specifically configured to: receive user input data, the user input data including a parameter selection strategy, the parameter selection strategy including parameter selection from a time dimension and / or a resource value consumption dimension; and select one group of time values and resource values according to the parameter selection strategy.

[0207] Each of the foregoing modules in the running parameter recommendation apparatus can be implemented by software or by hardware. For example, the implementation of the algorithm module is introduced as follows. Similarly, the implementation of other modules, such as the selection module and the modeling module, can refer to the implementation of the algorithm module.

[0208] As an example of a software functional unit, the algorithm module can include code running on a computing instance. The computing instance can be at least one of a physical host (computing device), a virtual machine, a container, and the like. Further, the computing device can be one or more. For example, the algorithm module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region, or can be distributed in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same AZ, or can be distributed in different AZs, each AZ including one data center or multiple data centers in close geographical proximity. Generally, one region can include multiple AZs.

[0209] Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same VPC, or can be distributed in multiple VPCs. Generally, one VPC is set in one region. Communication between two virtual private clouds (VPCs) in the same region and between VPCs in different regions needs to set a communication gateway in each VPC to realize the interconnection between VPCs through the communication gateway.

[0210] As an example of a hardware functional unit, an algorithm module can include at least one computing device, such as a server or the like. Alternatively, an algorithm module can also be a device implemented using a central processing unit (CPU), an application specific integrated Circuit (ASIC), a programmable logic device (PLD), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), an embedded neural network processing unit (NPU), a system on chip (SoC), an offload card, an acceleration card, or the like. Among them, the above-mentioned PLD can be a CPLD, an FPGA, a GAL, or any combination thereof.

[0211] The multiple computing devices included in the algorithm module can be distributed in the same region or in different regions. The multiple computing devices included in the algorithm module can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the algorithm module can be distributed in the same VPC or in multiple VPCs. Among them, the multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, acceleration cards, and the like.

[0212] As an example of a software functional unit, the running parameter recommendation device can include code running on a computing instance. Among them, the computing instance can be at least one of a physical host (computing device), a virtual machine, a container, and the like. Further, the above-mentioned computing device can be one or more. For example, the running parameter recommendation device can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same AZ or in different AZs, and each AZ includes a data center or multiple data centers with similar geographical locations. Among them, a region usually includes multiple AZs.

[0213] Similarly, the plurality of hosts / virtual machines / containers for running the code can be distributed in the same VPC or in multiple VPCs. Among them, usually one VPC is set in one region. The communication gateway needs to be set in each VPC for the interconnection between two VPCs in the same region or between VPCs in different regions.

[0214] As an example of a hardware functional unit, the running parameter recommendation device can include at least one computing device, such as a server. Alternatively, the running parameter recommendation device can also be a device implemented by ASIC or PLD. Among them, the PLD can be implemented by CPLD, FPGA, GAL or any combination thereof.

[0215] The plurality of computing devices included in the running parameter recommendation device can be distributed in the same region or in different regions. The plurality of computing devices included in the running parameter recommendation device can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the running parameter recommendation device can be distributed in the same VPC or in multiple VPCs. Among them, the plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs and GALs.

[0216] The application also provides a chip system including a processor and a power supply circuit, the power supply circuit is used for performing power supply to the processor, and the processor is used for performing the operation steps corresponding to the method provided by the application. For brevity, it will not be repeated here. Among them, the processor can be implemented by GPU, or by DPU, NPU, XPU, SoC, offload card, acceleration card and other computing devices.

[0217] The application also provides a computing device 100. As shown in FIG. 16, the computing device 100 includes a bus 102, a processor 104, a memory 106 and a communication interface 108. The processor 104, the memory 106 and the communication interface 108 communicate through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the application does not limit the number of processors and memories in the computing device 100.

[0218] The bus 102 can be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or the like. The unified bus can also be referred to as a coherent bus. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is shown in FIG. 16, but it does not mean that there is only one bus or only one type of bus. The bus 104 can include a path for transmitting information between various components of the computing device 100 (e.g., the memory 106, the processor 104, the communication interface 108). The unified bus can also be referred to as a coherent bus.

[0219] The processor 104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), an ASIC, a FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, or the like computing device.

[0220] The memory 106 can include a volatile memory, such as a random access memory (RAM) including a dynamic or static RAM. The processor 104 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). In addition, the memory 106 can also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0221] It is worth mentioning that the same type of storage medium can be configured in the same computing device to implement the function of the storage 106, or two or more types of storage media can be configured to implement the function of the storage 106, and the present application does not limit this.

[0222] The storage 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the modules mentioned in the foregoing FIG. 15, thereby implementing the method provided by the present application. That is, the storage 106 stores instructions for executing the method provided by the present application.

[0223] Alternatively, the storage 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the foregoing selection module, modeling module, or algorithm module, thereby implementing the method provided by the present application. That is, the storage 106 stores instructions for executing the method provided by the present application.

[0224] The communication interface 103 uses a transceiver module such as but not limited to a network interface card and a transceiver to implement communication between the computing device 100 and other devices or communication networks.

[0225] As a possible implementation manner, the computing device 100 can also include a chip system including a processor and a power supply circuit for performing power supply to the processor, and the processor is used to perform the operation steps corresponding to the method provided by the present application. For the sake of brevity, it will not be repeated here. The processor can be implemented by a GPU, or can be implemented by a DPU, NPU, XPU, SoC, offload card, acceleration card, or other computing devices or AI chips.

[0226] As a possible implementation manner, the computing device 100 can include multiple types of processors 104, that is, the computing device 100 is a heterogeneous device, for example, the computing device 100 includes a CPU and a GPU, and at least one of the processors 104 in the computing device 100 can perform the operation steps corresponding to the method provided by the present application. For the sake of brevity, it will not be repeated here.

[0227] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a notebook computer, or a terminal device such as a smart phone.

[0228] As shown in FIG. 17, the computing device cluster includes at least one computing device 100. The storage 106 in one or more computing devices 100 in the computing device cluster can store the same instructions for executing the method provided by the present application.

[0229] In some possible implementation, the memory 106 of one or more of the computing devices 100 in the computing device cluster can also respectively store partial instructions for performing the method provided by the present application. In other words, the combination of one or more computing devices 100 can collectively execute the instructions for performing the method provided by the present application.

[0230] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, respectively for performing partial functions of the joint testing device. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more of the aforementioned selection module, modeling module or algorithm module.

[0231] In some possible implementation, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 18 shows one possible implementation. As shown in FIG. 18, two computing devices 100A and 100B are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for performing the functions of the selection module, the modeling module or the algorithm module. Meanwhile, the memory 106 in the computing device 100B stores instructions for performing the functions of the selection module, the modeling module or the algorithm module.

[0232] It should be understood that the functions of the computing device 100A shown in FIG. 18 can also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be completed by multiple computing devices 100.

[0233] The connection manner between the computing device cluster shown in FIG. 18 can be that the method provided by the present application requires a large amount of computing power, needs to achieve load balancing or store a large amount of data, etc. Therefore, different modules are deployed in different computing devices, such as the functions implemented by the algorithm module are executed by the computing device 100A, and the functions implemented by the modeling module are executed by the computing device 100B.

[0234] The present application also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manners of the computing device clusters described with reference to FIG. 17 and FIG. 18. The difference is that the memory 106 in one or more computing devices 100 in the computing device cluster can store the same instructions for performing the method provided by the present application.

[0235] In some possible implementation manners, the memory 106 of one or more of the computing devices 100 in the computing device cluster can also respectively store partial instructions for performing the method provided in the present application. In other words, the combination of the one or more computing devices 100 can collectively execute the instructions for performing the method provided in the present application.

[0236] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions for performing part of the functions of the operating parameter recommendation apparatus provided in the present application. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more modules such as the selection module, the modeling module, or the algorithm module.

[0237] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the method provided in the present application.

[0238] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to perform the method provided in the present application.

[0239] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of operating parameter recommendation, characterized in that, The method comprises: establishing data information corresponding to a task according to data information of a historical task processed by a big data system, running parameters of a computing device processing the task, a first relationship between time required for processing the task and a second relationship between resources consumed by the computing device for processing the task; inputting data information of a task to be processed and the first relationship and the second relationship into a genetic algorithm, wherein a processing flow of the genetic algorithm comprises: performing multiple transformations on parameter values of the running parameters, and performing iterative calculation based on the data information of the task to be processed and the first relationship and the second relationship, and outputting multiple sets of time values and resource values; selecting one set of time values and resource values from the multiple sets of time values and resource values according to user requirements, determining parameter values corresponding to the selected time values and resource values, and taking the parameter values as parameter values of a computing device processing the task to be processed.

2. The method of claim 1, wherein, The task to be processed comprises data extraction, data processing or data loading, the data extraction model comprises a model for extracting data to be processed to a storage space of the computing device, the data processing model comprises a model for processing data to be processed according to a pre-set logic, and the data loading model is used to load a model output by the data processing model to a pre-set database, the running parameters comprise at least one of read batch size, partition number, extraction parallelism, process number, thread number, memory size, write batch size or write parallelism, the read batch size comprises a batch number of data extracted from data to be processed, the partition number comprises a number of data partitions to be processed, the extraction parallelism comprises a number of parallel jobs for extracting data to be processed to memory, the process number comprises a number of processes running when processing data to be processed, the thread number comprises a number of threads running when processing data to be processed, the memory size comprises a size of memory occupied when processing data to be processed, the write batch comprises a batch number of data written to a database after data processing, and the write parallelism comprises a number of parallel jobs for writing data to a database after data processing.

3. The method according to claim 1 or 2, characterized in that, The first relationship is a mathematical model or a machine learning model between the running parameters of the computing device and the time required for processing the task, and the second relationship is a mathematical model or a machine learning model between the running parameters of the computing device and the resources consumed by the computing device for processing the task.

4. The method according to any one of claims 1 to 3, characterized in that, The genetic algorithm comprises: performing multiple transformations on multiple sets of parameters output by the last iteration, and outputting multiple sets of transformed parameters; determining time values and resource values corresponding to each set of transformed parameters according to the first relationship and the second relationship; selecting multiple sets of Pareto optimal solutions from the multiple sets of transformed parameters according to the time values and resource values corresponding to each set of transformed parameters, and outputting a Pareto optimal solution set; if a termination iteration condition is met, outputting the Pareto optimal solution set. If the termination iteration condition is not met, the pareto optimal solution set is taken as the input of the next iteration, and the next iteration processing is performed.

5. The method of claim 4, wherein, The filtering of the multiple groups of pareto optimal solutions from the multiple groups of transformation parameters according to the time values and the resource values corresponding to each group of transformation parameters and the output of the pareto optimal solution set include: According to the time value and the resource value corresponding to each group of transformation parameters, the guiding dominance relationship between each group of transformation parameters and other parameters in the multiple groups of transformation parameters is obtained, and the evaluation value corresponding to the time value and the resource value of the dominator in the guiding dominance relationship is better than the evaluation value corresponding to the time value and the resource value of the dominated. According to the guiding dominance relationship between each group of transformation parameters and other parameters in the multiple groups of transformation parameters, multiple groups of undominated parameters are filtered from the multiple groups of transformation parameters, and the pareto optimal solution set is output.

6. The method according to claim 4 or 5, characterized in that, The transformation is cross and / or mutation of the parameters.

7. The method of claim 6, wherein, In the case where the transformation includes mutation, the processing flow of the mutation includes: mutating the running parameters according to a pre-set parameter range to obtain the mutated running parameters.

8. The method according to any one of claims 1-7, characterized in that, The selection of one group from the multiple groups of time values and resource values according to the user demand includes: Receiving user input data, wherein the user input data includes a parameter selection strategy, and the parameter selection strategy includes parameter selection from the time dimension and / or the resource value consumption dimension; Selecting one group from the multiple groups of time values and resource values according to the parameter selection strategy.

9. An operating parameter recommendation apparatus characterized by comprising: It includes: The modeling module is configured to establish a first relationship between running parameters of a computing device processing a task and time required for processing the task and a second relationship between the running parameters and resource consumption of the computing device based on data information of the task and historical task processing data information of a big data system; The algorithm module is configured to input data information of a to-be-processed task and the first relationship and the second relationship into a genetic algorithm, wherein the processing flow of the genetic algorithm includes multiple transformations of parameter values of the running parameters, and iterative calculation based on the data information of the to-be-processed task and the first relationship and the second relationship, and output of multiple groups of time values and resource values; The selection module is configured to select one group from the multiple groups of time values and resource values according to user demand, determine parameter values corresponding to the selected time values and resource values, and take the parameter values as parameter values of a computing device processing the to-be-processed task.

10. The apparatus of claim 9, wherein, The to-be-processed task comprises data extraction, data processing or data loading, the data extraction model comprises a model of extracting to-be-processed data to a storage space of the computing device, the data processing model comprises a model of processing to-be-processed data according to a preset logic, and the data loading model is used for loading a model output by the data processing model to a preset database; the running parameter comprises at least one of a reading batch size, a partition number, an extraction parallel degree, a process number, a thread number, a memory size, a writing batch size or a writing parallel degree; the reading batch size comprises a batch number of extracted data from to-be-processed data; the partition number comprises a number of to-be-processed data partitions; the extraction parallel degree comprises a number of parallel jobs of extracting to-be-processed data to a memory; the process number comprises a number of running processes when to-be-processed data is processed; the thread number comprises a number of running threads when to-be-processed data is processed; the memory size comprises a size of a memory occupied when to-be-processed data is processed; the writing batch comprises a batch number of data written to a database after data processing; and the writing parallel degree comprises a number of parallel jobs of writing data to a database after data processing.

11. The apparatus of claim 9 or 10, wherein, The first relationship is a mathematical model or a machine learning model between a running parameter of the computing device and a time required for processing the task, and the second relationship is a mathematical model or a machine learning model between the running parameter of the computing device and a resource consumed by the computing device for processing the task.

12. The apparatus of any one of claims 9-11, wherein, The algorithm module specifically comprises the following steps: performing multiple transformations on the multiple groups of parameters output in the last iteration to output multiple groups of transformed parameters; determining a time value and a resource value corresponding to each group of transformed parameters according to the first relationship and the second relationship; determining a time value and a resource value corresponding to each group of transformed parameters according to the first relationship and the second relationship; screening multiple groups of Pareto optimal solutions from the multiple groups of transformed parameters according to the time value and the resource value corresponding to each group of transformed parameters, and outputting a Pareto optimal solution set; if a termination iteration condition is met, outputting the Pareto optimal solution set; if the termination iteration condition is not met, taking the Pareto optimal solution set as an input of the next iteration and performing the next iteration.

13. The apparatus of claim 12, wherein, The algorithm module is specifically configured to: obtain a guiding dominance relationship between each group of transformed parameters and other parameters in the multiple groups of transformed parameters according to the time value and the resource value corresponding to each group of transformed parameters, wherein an evaluation value corresponding to a time value and a resource value of a dominator in the guiding dominance relationship is better than an evaluation value corresponding to a time value and a resource value of a dominated; screen multiple groups of non-dominated parameters from the multiple groups of transformed parameters according to the guiding dominance relationship between each group of transformed parameters and other parameters in the multiple groups of transformed parameters, and output the Pareto optimal solution set.

14. The apparatus of claim 12 or 13, wherein, The transformation is a crossover and / or mutation of the parameters.

15. The apparatus of claim 14, wherein In a case where the transformation includes variation, a processing flow of the variation includes: varying the operation parameter according to a preset parameter range, to obtain a varied operation parameter.

16. The apparatus of any one of claims 9-15, wherein, The selection module is specifically configured to: receive user input data, the user input data including a parameter selection strategy, the parameter selection strategy including parameter selection from a time dimension and / or a resource value consumption dimension; select one group of time values and resource values according to the parameter selection strategy.

17. A computing device, comprising: The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the computing device performs the operation steps of the method according to any one of claims 1 to 8.

18. A cluster of computing devices, characterized in that, Each computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the operation steps of the method according to any one of claims 1 to 8.

19. A computer program product comprising instructions, characterized in that, When the instructions are executed by the computing device cluster, the computing device cluster performs the operation steps of the method according to any one of claims 1 to 8.

20. A computer-readable storage medium, characterized in that, The computer program instructions are executed by the computing device cluster, and the computing device cluster performs the operation steps of the method according to any one of claims 1 to 8.

Images