Operating parameter recommendation method and apparatus

Abstract

Embodiments of the present application provide an operating parameter recommendation method and apparatus, used for in a big data processing scenario, recommending operating parameters for a computing device on the basis of the resource consumption under different operating parameters, without depending on historical data. The method comprises: first, on the basis of information obtained during the processing of historical tasks in a big data system, establishing a first relationship among the corresponding data information, parameters of a computing device processing the tasks, and time required to process the tasks, and a second relationship among the data information, the parameters, and resources of the computing device consumed for processing the tasks; inputting data information of a task to be processed, and the first relationship and the second relationship to a genetic algorithm, and outputting multiple sets of time values and resource values, i.e., modeling the process of the computing device executing data processing tasks, so as to fit the relationship between the time and resource consumption of the computing device and the operating parameters of the computing device; and determining operating parameters on the basis of actual available resources of the computing device or resource consumption that a user desires.

Description

A method and apparatus for recommending operating parameters

[0001] This application claims priority to Chinese Patent Application No. 202410580888.4, filed on May 10, 2024, entitled "A Method and Apparatus for Recommending Operating Parameters", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of big data processing, and in particular to a method and apparatus for recommending operating parameters. Background Technology

[0003] Big data processing is a common scenario in situations requiring the processing of large amounts of data. It typically utilizes the computing resources of one or more devices to perform big data processing. Before executing big data processing, the operating parameters of these devices need to be set; these parameters are the parameters configured for the devices to support task execution. In some scenarios, historical big data processing data can be used for learning to obtain the operating parameters for the current big data processing scenario.

[0004] However, in situations where data output spans longer periods, such as months or years, the data processed each time may differ, and available resources may also vary. Therefore, operating parameters determined by historical data may not match the actual data processing scenario; in other words, operating parameters used in past tasks are not reliable for the current task. Thus, improving the accuracy of operating parameters used for data processing over long time periods has become a pressing issue. Summary of the Invention

[0005] This application provides a method and apparatus for recommending operating parameters, which is used in big data processing scenarios to recommend operating parameters for computing devices based on the resource consumption of different operating parameters without relying on historical data.

[0006] In view of this, firstly, this application provides a method for recommending operating parameters, comprising: firstly, establishing corresponding data information, a first relationship between the parameters of the computing device used to process the task and the time required to process the task, and a second relationship between the parameters of the computing device consumed by the task and the time required to process the task, based on information from the historical task processing process of the big data system; inputting the data information of the task to be processed and the first and second relationships into a genetic algorithm, outputting multiple sets of time values ​​and resource values, each set of time values ​​and resource values ​​corresponding to a set of parameters, wherein the specific algorithm processing flow of the genetic algorithm may include: performing various transformations on the parameter values ​​of the operating parameters, and based on the data information of the task to be processed (such as the amount of data in the task to be processed, the number of datasets, etc.), the first relationship between the parameters of the computing device used to process the task and the second relationship, and the second relationship between the parameters of the computing device consumed by the task and the time required to process the task, and the second relationship between the parameters of the computing device used to process ... The first and second relations are iteratively calculated to output multiple sets of time and resource values. Then, based on user requirements, one set of time and resource values ​​is selected from these multiple sets, and the parameter value of the running parameter corresponding to the selected time and resource value is determined. This parameter value is used as the parameter value of the computing device to process the task. In other words, it is equivalent to modeling the process of the computing device performing data processing tasks. By simulating the data processing flow of the computing device, the relationship between the time and resource consumption of the computing device and the running parameters of the computing device is fitted. Based on the actual available resources of the computing device or the user's expected resource consumption, the running parameters are determined so that the recommended running parameters meet the computing capabilities of the computing device or meet the user's expectations.

[0007] In this embodiment, a big data processing flow is modeled to simulate the data processing process, thereby fitting the relationship between the time or resources consumed by the computing device when performing a task and the operating parameters. Based on this relationship, the time or resource consumption that meets the user's needs, as well as the corresponding operating parameters, can be determined. This allows for the recommendation of operating parameters that meet the user's needs for the computing device without relying on historical data.

[0008] In one possible implementation, the aforementioned task to be processed may include: data extraction, data processing, or data loading. The data extraction model includes a model that extracts the data to be processed to the storage space corresponding to the computing device. The data processing model includes a model that processes the data to be processed according to a pre-defined logic. The data loading model is used to load the model output by the data processing model into a preset database. Therefore, this implementation can be applied to end-to-end big data processing scenarios. For end-to-end big data processing scenarios, when historical data is not reliable or has low reference value, it recommends operating parameters that meet the resource consumption requirements of the computing device.

[0009] Accordingly, the operating parameters of the computing device may include at least one of the following parameters: read batch size, number of partitions, extraction parallelism, number of processes, number of threads, memory size, write batch size, or write parallelism. Read batch size includes the number of batches extracting data from the data to be processed; number of partitions includes the number of data partitions to be processed; extraction parallelism includes the number of parallel jobs executing the extraction of the data to be processed into memory; number of processes includes the number of processes running while processing the data to be processed; number of threads includes the number of threads running while processing the data to be processed; memory size includes the amount of memory occupied while processing the data to be processed; write batch size includes the number of batches writing the processed data into the database; and write parallelism includes the number of parallel jobs writing the processed data into the database. In this embodiment, the provided operating parameters include parameters for each stage of the end-to-end big data processing process of the computing device, improving the feasibility of the recommended operating parameters.

[0010] In one possible implementation, the aforementioned first relationship can specifically be a mathematical model or machine learning model between the operating parameters of the computing device and the time required to process the task, and the second relationship can specifically be a mathematical model or machine learning model between the operating parameters of the computing device and the resources consumed by the computing device to process the task. Therefore, in this embodiment, a data model can be constructed, or a machine learning model can be generated using machine learning methods. The constructed model is then used to represent the relationship between the operating parameters and the required time, as well as the relationship between the operating parameters and the consumed resources, during the execution of a data task by the computing device, so as to facilitate the subsequent determination of operating parameters that meet user needs using this relationship.

[0011] In one possible implementation, the aforementioned genetic algorithm's iteration process specifically includes: performing various transformations on multiple sets of operating parameters output from the previous iteration, outputting multiple sets of transformed parameters; obtaining the time and resource values ​​corresponding to each set of transformed parameters based on a first relation and a second relation; subsequently, selecting multiple Pareto optimal solutions from the multiple sets of transformed parameters based on the time and resource values ​​corresponding to each set of transformed parameters, and outputting a set of Pareto optimal solutions; if the termination iteration condition is met, then outputting the set of Pareto optimal solutions; if the termination iteration condition is not met, then using the set of Pareto optimal solutions as input for the next iteration to continue executing the next iteration. In this embodiment, when outputting multiple sets of time and resource values ​​based on the genetic algorithm, new operating parameters can be generated through transformation, and the operating parameters can be optimized through iteration, thereby ultimately outputting operating parameters that perform better in terms of time and / or resource consumption.

[0012] In one possible implementation, the aforementioned process of selecting multiple Pareto optimal solutions from the multiple sets of transformation parameters based on the time and resource values ​​corresponding to each set of transformation parameters, and outputting a set of Pareto optimal solutions, includes: obtaining the guiding dominance relationship between each set of transformation parameters and other parameters in the multiple sets of transformation parameters based on the time and resource values ​​corresponding to each set of transformation parameters, wherein the evaluation value corresponding to the time and resource values ​​of the dominant parameter is better than that of the dominated parameter, that is, in two sets of parameters with a dominance relationship, the parameter with better performance dominates the parameter with worse performance; and selecting multiple sets of undominated parameters from the multiple sets of transformation parameters based on the guiding dominance relationship between each set of transformation parameters and other parameters in the multiple sets of transformation parameters, and outputting a set of Pareto optimal solutions, which is equivalent to selecting the parameter with better performance from the multiple sets of transformation parameters, thereby obtaining the parameter with better performance in the time dimension or the resource consumption dimension.

[0013] In one possible implementation, the aforementioned transformation involves cross-parameter manipulation and / or mutation. Cross-parameter manipulation involves exchanging some parameter values ​​between two sets of parameters to create different parameters, while mutation involves adjusting some parameter values ​​to generate new parameters. Therefore, in this embodiment, new parameters can be generated through cross-parameter manipulation and / or mutation, thereby evolving parameters that may perform better in terms of time or resource consumption.

[0014] In one possible implementation, when the transformation includes mutation, the mutation process includes: mutating the operating parameters according to a pre-defined parameter range to obtain the mutated operating parameters. In this embodiment, the parameter range for mutation can be pre-defined, ensuring that the mutated parameters are within a constrained parameter range, such as the range supported by the computing device, thereby improving the feasibility of the final output parameters.

[0015] In one possible implementation, the aforementioned selection of one set of time values ​​and resource values ​​from multiple sets based on user needs may include: receiving user input data, the user input data including a parameter selection strategy, the parameter selection strategy including parameter selection from the time dimension and / or resource value consumption dimension; selecting one set of time values ​​and resource values ​​from multiple sets of time values ​​and resource values ​​according to the parameter selection strategy, and determining the corresponding parameters, such as selecting parameters that require less time, or selecting parameters that consume less resources, or selecting parameters that are more balanced between required time and resource consumption, etc.

[0016] Secondly, this application provides an operating parameter recommendation device, comprising:

[0017] The modeling module is used to establish, based on the information from historical tasks processed by the big data system, the data information corresponding to the task, the operating parameters of the computing device used to process the task, the first relationship between the time required to process the task, and the second relationship between the computing device resources consumed by the task.

[0018] The algorithm module is used to input the data information of the task to be processed, as well as the first and second relations, into the genetic algorithm. The processing flow of the genetic algorithm includes: performing various transformations on the parameter values ​​of the running parameters, and performing iterative calculations based on the data information of the task to be processed, as well as the first and second relations, and outputting multiple sets of time values ​​and resource values.

[0019] The selection module is used to select one set of time and resource values ​​from multiple sets according to user needs, determine the parameter values ​​corresponding to the selected time and resource values, and use the parameter values ​​as the parameter values ​​of the computing device to process the task to be processed.

[0020] The effects achieved by the second aspect and any optional implementation of the second aspect can be referred to the description of the first aspect or any optional implementation of the first aspect, and will not be repeated hereafter.

[0021] In one possible implementation, the task to be processed includes: data extraction, data processing, or data loading. The data extraction model includes a model that extracts the data to be processed to the storage space of the computing device. The data processing model includes a model that processes the data to be processed according to a pre-defined logic. The data loading model is used to load the model output by the data processing model into a preset database. The running parameters include at least one of the following: read batch size, number of partitions, extraction parallelism, number of processes, number of threads, memory size, write batch size, or write parallelism. The read batch size includes the number of batches of data extracted from the data to be processed. The number of partitions includes the number of data partitions to be processed. The extraction parallelism includes the number of parallel jobs that extract the data to be processed into memory. The number of processes includes the number of processes running when processing the data to be processed. The number of threads includes the number of threads running when processing the data to be processed. The memory size includes the amount of memory occupied when processing the data to be processed. The write batch size includes the number of batches of data to be processed that are written into the database. The write parallelism includes the number of parallel jobs that are written into the database.

[0022] In one possible implementation, the first relationship is a mathematical model or machine learning model between the operating parameters of the computing device and the time required to process the task, and the second relationship is a mathematical model or machine learning model between the operating parameters of the computing device and the resources of the computing device consumed by the processing task.

[0023] In one possible implementation, the algorithm module executes any iteration of the genetic algorithm, specifically including: performing various transformations on multiple sets of parameters output from the previous iteration, outputting multiple sets of transformation parameters; determining the time value and resource value corresponding to each set of transformation parameters based on the first relationship and the second relationship; selecting multiple sets of Pareto optimal solutions from the multiple sets of transformation parameters based on the time value and resource value corresponding to each set of transformation parameters, and outputting a set of Pareto optimal solutions; if the termination iteration condition is met, then outputting the set of Pareto optimal solutions; if the termination iteration condition is not met, then using the set of Pareto optimal solutions as the input for the next iteration and executing the next iteration.

[0024] In one possible implementation, the algorithm module is specifically used to: obtain the guiding dominance relationship between each set of transformation parameters and other parameters in the multiple sets of transformation parameters based on the time value and resource value corresponding to each set of transformation parameters, wherein the evaluation value corresponding to the time value and resource value of the dominant parameter in the guiding dominance relationship is better than the evaluation value corresponding to the time value and resource value of the dominated parameter; and select multiple sets of non-dominated parameters from the multiple sets of transformation parameters based on the guiding dominance relationship between each set of transformation parameters and other parameters in the multiple sets of transformation parameters, and output a Pareto optimal solution set.

[0025] In one possible implementation, the aforementioned transformation is to cross and / or mutate the parameters.

[0026] In one possible implementation, when the transformation includes mutation, the mutation process includes: mutating the operating parameters according to a pre-set parameter range to obtain the mutated operating parameters.

[0027] In one possible implementation, the selection module is specifically configured to: receive user input data, the user input data including a parameter selection strategy, the parameter selection strategy including parameter selection from a time dimension and / or a resource value consumption dimension; and select one of multiple sets of time values ​​and resource values ​​according to the parameter selection strategy.

[0028] Thirdly, embodiments of this application provide a computing device, including a processor and a memory; the processor of at least one computing device is configured to execute instructions stored in the memory of at least one computing device, so that the computing device performs method steps as described in the first aspect and any implementation thereof.

[0029] Fourthly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method steps as described in the first aspect and any implementation thereof.

[0030] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform a method as described in the first aspect or any implementation thereof.

[0031] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, enable the cluster of computing devices to perform a method as described in the first aspect or any implementation thereof.

[0032] In a seventh aspect, embodiments of this application provide a chip including at least one processor and an interface; at least one processor obtains program instructions or data through the interface; at least one processor is used to execute program line instructions to implement the method in the first aspect or any implementation thereof. Attached Figure Description

[0033] Figure 1 is a schematic diagram of a system architecture provided in this application;

[0034] Figure 2 is a schematic diagram of another system architecture provided in this application;

[0035] Figure 3 is a flowchart illustrating a method for recommending operating parameters provided in this application;

[0036] Figure 4 is a flowchart illustrating another method for recommending operating parameters provided in this application;

[0037] Figure 5 is a schematic diagram of a big data processing flow provided in this application;

[0038] Figure 6 is a schematic diagram of a Spark job workflow provided in this application;

[0039] Figure 7 is a flowchart illustrating another method for recommending operating parameters provided in this application;

[0040] Figure 8 is an example diagram of a crossover variation provided in this application;

[0041] Figure 9 is a schematic diagram of a Knee point provided in this application;

[0042] Figure 10 is a schematic diagram of a guiding and dominating relationship provided in this application;

[0043] Figure 11 is a schematic diagram of an application scenario provided in this application;

[0044] Figure 12 is a schematic diagram of another application scenario provided by this application;

[0045] Figure 13 is a schematic diagram of a client display interface provided in this application;

[0046] Figure 14 is a schematic diagram of a client display interface provided in this application;

[0047] Figure 15 is a schematic diagram of an operating parameter recommendation device provided in this application;

[0048] Figure 16 is a schematic diagram of the structure of a computing device provided in this application;

[0049] Figure 17 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0050] Figure 18 is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0052] First, for ease of understanding, some concepts or terms involved in the embodiments of this application will be explained.

[0053] (1) Machine Learning

[0054] Build a statistical model, fit the model parameters to the sample data using optimization methods, and make predictions on new sample data.

[0055] (2) Automated Machine Learning (AutoML): This refers to designing a series of advanced control systems to operate machine learning models, enabling the models to automatically learn appropriate parameters and configurations without human intervention. In deep neural network-based learning models, automated computational learning mainly includes network architecture search and global parameter setting. Network architecture search is used to allow the computer to generate the most suitable neural network architecture for the problem based on the data, characterized by high training complexity and significant performance improvement. A machine learning task typically includes a training part and a prediction part. In the prediction part, the parameters of the statistical model can be used to make predictions on the training sample data. The update direction of the statistical model parameters is calculated based on the prediction error, and this process is repeated until the parameters converge. In the prediction part, the trained model can be used to predict new samples.

[0056] (3) Genetic Algorithm (GA)

[0057] This is an algorithm based on the laws of biological evolution in nature. It is a computational model that simulates the natural selection and genetic processes of biological evolution. It searches for the optimal solution by simulating the natural evolutionary process. This algorithm uses mathematical methods and computer simulation to transform the problem-solving process into processes similar to crossover and mutation of chromosomes and genes in biological evolution. When solving complex combinatorial optimization problems, it can typically obtain better optimization results faster than some conventional optimization algorithms. In the embodiments of this application, the process of big data processing can be modeled and solved, and in the solution process, processes such as crossover and mutation are combined to evolve and obtain a better-evaluated solution.

[0058] (4) Multi-objective optimization

[0059] Also known as multi-objective programming, it is a branch of mathematical programming. It studies the optimization of more than one objective function over a given region.

[0060] (5) Pareto-optima front (PoF)

[0061] The Pareto optimal set refers to a set of solutions for which further optimization of one or more objectives is impossible without deteriorating other objectives; hence, it is also called a non-dominated optimal set. The Pareto optimality concept is a vector evaluation method for multi-objective solutions based on set theory. Traditional mathematical programming and simulated annealing algorithms, being serial algorithms characterized by single-point search, cannot utilize the Pareto optimality concept for solution evaluation.

[0062] The method provided in this application can be applied to big data processing scenarios. For big data processing scenarios, some existing solutions are not suitable for data processing scenarios spanning long periods. In some existing solutions, reverse optimization schemes can be designed based on the results of previously run tasks. This is suitable for continuously running big data tasks, where the results of previous runs are valuable for reference. However, for scenarios with long data settlement times and no reference value for the current big data processing scenario, solutions relying on historical data to determine the task's operating parameters are not applicable.

[0063] For example, in one existing solution, the task submission interface of a Spark application is encapsulated. It checks if a hierarchical gray-box time prediction model already exists for the Spark application, accesses the database, reads and updates the hierarchical gray-box time prediction model, and the user chooses whether to optimize. If optimization is performed, optimized parameters are generated; otherwise, the task is executed with the original parameters by calling a shell command. However, this solution heavily relies on historical data, training the data model based on historical data for parameter generation.

[0064] For example, in an existing approach, a cost-based performance model is constructed by obtaining the task execution configuration and corresponding cost information. Optimized configurations are then obtained within a given parameter space. For unknown task types, default parameters are used for a single run, and reference values ​​for the optimized configuration are provided by assessing task similarity. This approach addresses potential issues with current configuration parameter tuning by proposing a cost-based performance model. It generates the performance model through analysis of historical Spark tasks, obtains optimized parameters using a parameter space search algorithm, and continuously upgrades and adjusts the model as new tasks are run, increasing its accuracy. For unknown task types, reference parameter values ​​are provided after a single run. However, this approach, similar to the aforementioned approach, heavily relies on historical data for parameter generation through a parameter space search algorithm.

[0065] Both of the aforementioned approaches rely heavily on historical data and can only operate when that data is reliable. In some scenarios, data may be output over a considerable period. For example, in financial scenarios, there may be monthly or yearly data closings. For monthly or yearly data processing, the operating parameters of the computing equipment need to be configured before processing this data. Since the amount of data or available hardware resources may differ between the current processing task and the previous one, the operating parameters used in historical tasks are not relevant for selecting operating parameters for the current task.

[0066] Therefore, this application provides a method for recommending operating parameters, which can more accurately determine the operating parameters of computing devices in scenarios where data is processed over a long period of time.

[0067] First, the method provided in this application can be applied to one or more computing devices. When applied to multiple computing devices, these devices can form a computing device cluster, which may include a centralized computing architecture or a distributed computing architecture. For example, taking a distributed architecture as an example, it can be specifically applied to a big data processing architecture based on Spark jobs or a big data processing architecture based on Flink jobs. Specifically, in a distributed architecture, the server cluster can provide services to users in the form of clients. Users can interact with the server cluster through clients, such as sending data to the server cluster or initiating service requests.

[0068] For example, taking a big data processing architecture based on Spark jobs as an example, as shown in Figure 1, this architecture 10 can include a client 13 and a server cluster 12. The server cluster can include multiple computing nodes, each of which can be implemented by one or more servers, and one server can implement one or more nodes. These multiple nodes can be specifically divided into drivers 121, cluster managers 122, Spark job nodes 123, etc.

[0069] Client 13 can be a program deployed on a user device cluster, through which users can submit job requests to the server cluster.

[0070] Server cluster 12 can process user-submitted requests, specifically:

[0071] Driver 121 runs the main function of the Spark application and creates the application context (SparkContext), which controls the lifecycle of the Spark application.

[0072] Resource Manager 122, also known as the master node, can be one of the servers in a server cluster, or a processor or processing module deployed on one of those servers. Resource Manager 122 is used to manage computing resources within the server cluster, such as allocating the number of compute nodes to various tasks.

[0073] Spark job nodes 123 can be deployed as servers within a server cluster, or as processors or processing modules deployed within servers. For example, a job node can be one of the servers in a cluster. Spark job nodes 123 can be used as compute nodes to run Spark applications. Executors can be set up within Spark job nodes to execute specific tasks. Executors can store cached data along with the actual tasks being run.

[0074] The big data processing workflow based on Spark jobs can be shown in Figure 2. Specifically:

[0075] The client sends a job request to the server cluster. The driver registers and creates Spark job tasks (i.e., the corresponding application) and creates a SparkContext to manage the lifecycle of Spark job tasks.

[0076] After the Spark job task is created, the resource manager can schedule resources and allocate computing nodes to the Spark job task based on the task executor status of each Spark job node. The Spark job node can then execute the received Spark job task.

[0077] Taking cross-database data processing as an example, which involves processing data from different databases, the big data processing workflow can include Spark job tasks such as data extraction, database processing, and data loading.

[0078] Data extraction involves extracting data from multiple source libraries into Spark memory based on the requirements of the Spark job.

[0079] Data processing: According to the Spark data processing logic written by the user, perform calculations on the data in Spark memory and obtain the final result table in Spark.

[0080] Data loading: Loads the result tables from Spark's memory into the target database.

[0081] Before executing Spark jobs on a server cluster, it is usually necessary to configure the device's operating parameters, such as configuring memory resources and CPU resources for the compute nodes.

[0082] The method provided in this application can recommend operating parameters for computing devices based on better objectives, such as resource consumption objectives or time consumption objectives, before performing big data processing. The method provided in this application can be deployed in computing devices, which may include independently deployed servers or servers in the aforementioned server cluster 12, such as the aforementioned resource manager 122 or driver 121.

[0083] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0084] Referring to Figure 3, a flowchart of an operating parameter recommendation method provided in this application is shown below.

[0085] 301. Utilize historical task data to build a model, and output the first relationship between data information, operating parameters, and time, as well as the second relationship between data information, operating parameters, and resources.

[0086] In the process of processing historical tasks in a big data processing system, the data involved includes: historical task data information, computing device operating parameters, processing time, and resource consumption. Specifically, the historical task data information may include the amount of data to be processed from historical figures, the number of datasets, or data categories. The computing device operating parameters may include the parameters configured for executing the historical task, such as the number of processes and batch sizes. The computing device resources may include metrics used to measure resource consumption, such as CPU resource consumption, memory consumption, or input / output (I / O) consumption.

[0087] Data from historical tasks executed by a big data processing system can be used to create a model representing the task processing flow. There can be one or more historical tasks. By utilizing relevant data from the execution of one or more historical tasks using computing devices, the data information, operating parameters, duration, and resources of the historical tasks can be modeled. This model outputs the primary relationship between data information, operating parameters, and time, as well as the secondary relationship between data information, operating parameters, and resources.

[0088] Specifically, when performing modeling, either a mathematical model or a machine learning model can be used.

[0089] In the process of modeling using mathematical models, relationships between historical task data, operational parameters, duration, and resources can be fitted. Specifically, the relationships between data, operational parameters, and duration, as well as the relationships between data, operational parameters, and resources, can be fitted separately, thus outputting a first relationship and a second relationship respectively. When using data models for modeling, in the subsequent process of determining operational parameters for the task to be processed, the data information of the task to be processed can be used as input to the first and second relationships, outputting multiple sets of time and resources, as well as the corresponding parameters.

[0090] In the process of using machine learning models for modeling, the initial structure of the machine learning model can be defined, such as setting the input and output channels. Machine learning is then performed using data, runtime parameters, duration, and resources from historical task execution, iteratively updating the model to obtain a usable trained model. During the subsequent determination of runtime parameters for the task at hand, the data from the task can be used as input to the machine learning model, outputting multiple sets of time and resource information, along with corresponding parameters.

[0091] 302. Input the data information of the task to be processed, the first relation and the second relation into the genetic algorithm, and output multiple sets of time values ​​and resource values.

[0092] First, the task to be processed and its data information will be introduced.

[0093] The task to be processed can be a task to be executed by the computing device, specifically a task to be performed on data to be processed. The data information of the task to be processed can include information about the data to be processed when the computing device executes the task, such as the amount of data to be processed and the number of datasets.

[0094] From the perspective of the steps involved in executing a task, the task to be executed can be divided into one or more steps: data extraction, data processing, or data loading, etc. The data extraction model includes a model that extracts the data to be processed into the storage space of the computing device. The data processing model includes a model that processes the data to be processed according to a pre-defined logic. The data loading model is used to load the model output by the data processing model into a preset database.

[0095] Accordingly, the operating parameters that the computing device needs to configure to process the task to be processed may include, but are not limited to, one or more of the following: read batch size, number of partitions, extraction parallelism, number of processes, number of threads, memory size, write batch size, or write parallelism. Read batch size includes the number of batches of data extracted from the data to be processed. Number of partitions includes the number of data partitions to be processed. Extraction parallelism includes the number of parallel jobs that extract the data to be processed into memory. Number of processes includes the number of processes that run when processing the data to be processed. Number of threads includes the number of threads that run when processing the data to be processed. Memory size includes the amount of memory occupied when processing the data to be processed. Write batch size includes the number of batches of data to be processed that are written into the database. Write parallelism includes the number of parallel jobs that write the data to be processed into the database.

[0096] The data to be processed may vary depending on the specific task, and the processing logic for this data will also differ across tasks. For example, in a tax rate calculation task, the data to be processed may include income information and tax deduction information for multiple users. In a corporate financial report calculation task, the data to be processed may include the company's financial information, such as working capital, current liabilities, quick assets, total liabilities, and total assets.

[0097] For example, in some possible scenarios, the method provided in this application embodiment can be applied to tax rate calculation tasks, such as in the application scenario where a company calculates the individual tax rate of its employees on a monthly basis. The data to be processed may specifically include the employee's salary income, length of service, cumulative income for the year, social security level, social security contribution amount, lunch allowance, transportation allowance, housing allowance, comprehensive allowance, sick leave, personal leave, maternity leave, late / early departure or absenteeism, etc., which can be used to calculate the employee's tax rate.

[0098] Secondly, the specific algorithm flow based on the genetic algorithm is introduced.

[0099] Specifically, the processing flow based on genetic algorithms may include: performing various transformations on the parameter values ​​of the running parameters, and iteratively calculating based on the data information of the task to be processed, as well as the first and second relationships, to output multiple sets of time and resource values. In the embodiments of this application, parameters that perform better in the time or resource dimensions can be evolved based on genetic algorithms.

[0100] Taking any one of the iteration processes as an example: perform various transformations on the multiple sets of parameters output from the previous iteration, and output multiple sets of transformed parameters; determine the time value and resource value corresponding to each set of transformed parameters according to the first relationship and the second relationship; select multiple sets of Pareto optimal solutions from the multiple sets of transformed parameters according to the time value and resource value corresponding to each set of transformed parameters, and output the Pareto optimal solution set; if the termination iteration condition is met, output the Pareto optimal solution set; if the termination iteration condition is not met, use the Pareto optimal solution set as the input for the next iteration and execute the next iteration process.

[0101] Therefore, in this embodiment, the initial parameters can be mutated based on crossover and mutation in the genetic algorithm to evolve better operating parameters in terms of time or resources, so that operating parameters that better meet the user's needs can be selected in the future.

[0102] Furthermore, in the process of selecting Pareto optimal solutions, the guiding dominance relationship between each set of transformation parameters and other parameters in multiple sets of transformation parameters can be obtained based on the time and resource values ​​corresponding to each set of transformation parameters. In the guiding dominance relationship, the evaluation value corresponding to the time and resource values ​​of the dominant parameter is better than that of the dominated parameter. Subsequently, based on the guiding dominance relationship between each set of transformation parameters and other parameters in multiple sets of transformation parameters, multiple sets of undominated parameters are selected from multiple sets of transformation parameters, and the Pareto optimal solution set is output.

[0103] This can be understood as follows: in the process of determining the guiding and dominating relationship between parameters, a coordinate system can be established from the dimensions of time and resources. Within the coordinate system, the dominating region of each set of parameters can be constructed. The dominating region is the region where the time dimension and resource dimension of the parameter are not better than that of the parameter set. In other words, the parameter performs better in the time dimension and resource dimension within its dominating region. The parameter set is the Pareto front within its dominating region.

[0104] In this embodiment of the application, during the process of outputting the Pareto optimal solution set, each set of operating parameters can be evaluated to obtain an evaluation result for each set of operating parameters. This evaluation result is used to evaluate the degree to which the time or resources consumed when configuring each set of operating parameters on at least one computing device meet the optimization objective, which is to minimize the time value and / or resource value. A set of operating parameters is determined based on the evaluation result corresponding to each set of operating parameters. Therefore, in this embodiment of the application, the degree to which the resources consumed by each set of operating parameters, when configured to execute the task to be processed on a computing device, meet the optimization objective can be evaluated, thereby ultimately allowing the selection of operating parameters that better meet the optimization objective.

[0105] Therefore, in the embodiments of this application, parameters that perform better in the time and resource dimensions can be selected based on the guiding and dominating relationship between parameters. This is equivalent to selecting parameters that are at the Pareto front and outputting parameters that perform better, thereby achieving a better parameter search in the time and resource dimensions.

[0106] Optionally, the aforementioned transformations may include crossover and / or mutation. Crossover involves exchanging parameter values ​​of different types between two sets of parameters to output new parameters, while mutation involves adjusting one or more values ​​in a set of parameters to output new parameters. Therefore, in this embodiment, parameters can be evolved through crossover or mutation to produce better-performing parameters, thus enabling the acquisition of parameters that better meet user needs.

[0107] For example, perform at least one operation, such as crossover or mutation, on the n individuals output from the previous iteration to obtain m individuals. Each individual represents a set of operating parameters, where m can be a value greater than n, such as 2n or 3n. Crossover can be understood as splitting two individuals at the intersection point and then recombining them to obtain new individuals. Mutation is the alteration of gene points in an individual to obtain new individuals. Based on the time value and resource value corresponding to each individual, select the Pareto front from the m individuals, that is, output the n better individuals for the next iteration.

[0108] Furthermore, when the aforementioned transformation includes mutation, the mutation of one or more values ​​of the parameters can usually be performed according to a pre-set parameter range. This pre-set parameter range may specifically include the parameter range supported by the computing device, or a range determined based on empirical values, so that the parameters generated after mutation are within the constrained parameter range, making the output parameters more practical.

[0109] 303. Select one set of time and resource values ​​from multiple sets according to user needs, and use the parameter values ​​corresponding to the time and resource values ​​in that set as the parameters for the computing device to process the task to be processed.

[0110] After performing step 302, multiple sets of time and resource values, along with the corresponding parameters for each set, will be output. Users can then filter these sets of time and resource values ​​based on their needs, selecting those that meet their requirements, extracting the corresponding parameter values, and configuring the computing device parameters.

[0111] In this embodiment, the process of a computing device executing a task can be modeled to determine the relationship between the operating parameters configured on the computing device and resource consumption, namely the aforementioned first or second relationship. This relationship allows for the determination of operating parameters with acceptable resource consumption. Therefore, even in data processing scenarios spanning long periods, modeling can be used to determine operating parameters with acceptable resource consumption without relying on historical data.

[0112] In one possible implementation, user input data can be received, including a parameter selection strategy. When selecting a set of parameters, the strategy selects one set from multiple time and resource values, and determines the corresponding parameters. For example, the strategy might choose parameters that require less time, consume fewer resources, or have a better balance between time and resource consumption. This allows for the selection of parameters that better meet the user's needs based on their actual preferences. For instance, optimization goals could include both time and hardware resource optimization objectives; the user could choose to prioritize parameters with lower time consumption or lower hardware resource consumption.

[0113] The method and flow provided in this application have been described above. The following section will further describe the method and flow provided in this application in conjunction with specific application scenarios.

[0114] For example, the following description uses the method provided in this application as an example of applying it to a big data architecture based on Spark jobs. In a Spark job scenario, a job task can be divided into one or more independent tasks. When divided into multiple tasks, these multiple tasks can be executed synchronously or serially on various computing nodes.

[0115] The method provided in this application can be divided into multiple stages, as shown in Figure 4. Specifically, it can be divided into building a mathematical model for Spark runtime parameters and ETL (extract-transform-load) data processing, and searching for Spark runtime parameters. The steps of each stage are described in detail below.

[0116] Phase 1: Constructing a mathematical model for Spark runtime parameters and ETL data processing

[0117] First, let's introduce the data processing flow.

[0118] The Spark job workflow can be illustrated in Figure 5. Specifically, the workflow can include data extraction, data processing, and data loading.

[0119] Data extraction involves extracting data from multiple source libraries into Spark memory based on the requirements of the Spark job.

[0120] Data processing: Based on the user-written Spark data processing logic (such as data processing logic and business logic), perform calculations on the data in Spark memory and obtain the final result table in Spark.

[0121] Data loading: Loads the result tables from Spark's memory into the target database.

[0122] Furthermore, for different steps, before the computing device processes the data, it is necessary to configure the corresponding operating parameters based on different environmental parameters. These environmental parameters are the data information corresponding to the task.

[0123] For example, during the data extraction phase, the environmental parameters and the required operating parameters can be shown in Table 1:

[0124] Table 1

[0125] During the data extraction phase, the Spark job extracts data from three source databases, each with a different amount of data, which constitutes the environment parameters of the current job. The extraction batch size, number of partitions, and parallelism settings are among the Spark job's runtime parameters, determining the efficiency and resource consumption of data extraction.

[0126] For example, during the data processing phase, the environmental parameters and the required operating parameters can be shown in Table 2:

[0127] Table 2

[0128] During the data processing phase, environmental parameters include the amount of data to be processed and the number of operators used to process the data. For Spark runtime parameters, you can set the number of processes, threads, and corresponding memory size to support the computing device in processing the data.

[0129] For example, during the data loading phase, the environmental parameters and the required operating parameters can be shown in Table 3:

[0130] Table 3

[0131] During the data loading phase, the environmental parameters include the amount of output data obtained after data processing, and the configurable runtime parameters include the write batch size and write parallelism for transferring the output data to the target database.

[0132] Secondly, the modeling process will be introduced.

[0133] For example, based on job requirements, the optimization objective can be set as time consumption and resource consumption, aiming to find a set of operating parameters that minimizes both the time and resource consumption of the Spark job. Resource consumption and time consumption are two mutually exclusive objectives, therefore the operating parameters must simultaneously consider the balance between the two objectives. To better represent the task objectives through a mathematical model, the symbols in Table 4 can be used for mathematical modeling.

[0134] Table 4

[0135] Accordingly, the constructed model can be shown in Figure 6. This model is divided into three stages: data extraction, data processing, and data loading. The environmental parameters, runtime parameters, and objectives involved in the three processes of the Spark job are shown in Figure 6. Among them, the environmental parameters involved in executing the historical task specifically include data from the data extraction, data processing, and data loading stages, such as data information for each stage (e.g., the total number of records in the main table, the amount of data involved in encryption and decryption), Spark runtime parameters (e.g., Spark task submission execution units, Spark memory execution units, parallelism, etc.), the time consumption of each stage (e.g., the columns shown in Figure 6, such as data reading(s), decryption(s), processing(s), encryption(s), writing(s), total process time(s) or JOB time(s), etc.), and resource consumption (e.g., core consumption, memory consumption, etc. shown in Figure 6), etc.

[0136] In the data extraction phase, the relationships between environmental parameters (data from historical tasks), Spark runtime parameters, and objectives are constructed. Spark runtime parameters in the data extraction phase can include batch size, number of partitions, and extraction parallelism. Objectives include time consumption and resource consumption. That is, using data from the data extraction phase of historical tasks, the relationships between the environmental parameters, Spark runtime parameters, and time consumption are constructed. In the data processing phase, environmental parameters include data volume, and Spark runtime parameters include the number of processes, threads, and memory. Objectives also include time consumption and resource consumption. That is, using data from the data processing phase of historical tasks, the relationships between the environmental parameters, Spark runtime parameters, and time consumption and resource consumption are constructed. In the data loading phase, environmental parameters include the amount of data to be loaded, and Spark runtime parameters include batch size and parallelism. Objectives also include time consumption and resource consumption. That is, using data from the data loading phase of historical tasks, the relationships between the environmental parameters, Spark runtime parameters, and time consumption and resource consumption are constructed.

[0137] Since a Spark job consists of multiple processes, the mathematical representation of the two objectives is the sum of the individual stages. The optimization objectives can be expressed as follows:

[0138] Objective 1 (Time Consumption, or Time Spent):

[0139] Objective 2 (Resource Consumption):

[0140] Where, objective 1 represents at θ (t) In the environment The parameters represent the time consumption of the Spark task being run, where t1 represents the time spent in the data extraction phase, t2 represents the time spent in the data processing phase, and t3 represents the time spent in the data loading phase. This objective is generally better when it is as small as possible, meaning the shorter the time spent by the computing device to execute the task, the better. Similarly, objective 2 represents the time spent at θ... (t) In the environment The parameters specify the resource consumption of the Spark task being run, including a weighted average of various resource consumption values, where n α *α represents the weighted value of CPU consumption, n β *β represents the weighted value of I / O consumption, n γ *γ represents the weighted value of memory consumption. The smaller this target is, the better, meaning the less resources the computing device consumes to execute tasks.

[0141] Based on the above mathematical representations and examples, a mathematical model for Spark runtime parameters and ETL data processing was constructed. As shown in Figure 5 above, the input of this mathematical model is the Spark job runtime parameters (data extraction batches, number of processes, memory usage, etc.), the environment parameters are the runtime environment (data volume, number of datasets, etc.), and the output is the objective (time consumption and resource consumption).

[0142] Furthermore, when constructing a data model, various methods can be used for modeling, such as modeling using data relationships or using machine learning models.

[0143] The mathematical relationship modeling involves setting corresponding parameters based on historical task information to express the mathematical relationship between input and output. These parameters are influenced by the environment. Essentially, it constructs data relationships under various objectives based on historical task information, using these relationships to represent the relationship between environmental parameters, Spark runtime parameters, and the objectives at each stage. In the subsequent Spark runtime parameter search stage, this mathematical relationship is used, taking the data information of the task to be processed as input, and calculating and outputting multiple sets of Spark parameters, along with the corresponding time and resource consumption for each set.

[0144] Machine Learning Model: Based on historical information, machine learning is performed using 8 Spark runtime parameters, 2 target outputs, and 3 environment parameters from historical tasks, outputting a trained machine learning model. The machine learning model can be a basic machine learning model, such as the K-Nearest Neighbor (KNN) classification algorithm, eXtreme Gradient Boosting (XGBoost), or Random Forest. In the Spark runtime parameter search phase, the data from the task to be processed can be directly used as input to the machine learning model, outputting multiple sets of Spark parameters and the corresponding time and resource consumption for each set.

[0145] Of course, the aforementioned optimization objectives can also be set as stability or other objectives related to actual business scenarios, which are only introduced here as examples.

[0146] Phase 2: Searching for Spark runtime parameters

[0147] During the parameter search phase, embodiments of this application can use a knee-guided genetic algorithm to search for user-preferred Spark runtime parameters.

[0148] During the parameter search phase, this application provides a parameter selection method based on a genetic algorithm. For example, the parameter search process can be shown in Figure 7.

[0149] 701. Construct the initial population P t (g)

[0150] Combining the modeling output from Phase 1 above, we can obtain the mathematical model for constructing Spark runtime parameters and ETL data processing. This initial population P... t (g) can be obtained based on the mathematical model of Spark runtime parameters and ETL data processing constructed in the aforementioned stage one. For example, the information of the data to be processed in the task can be input into the aforementioned mathematical relationship or machine learning model, and multiple sets of parameters and the time and resource consumption corresponding to each set of parameters can be output.

[0151] Specifically, the data information of the data to be processed can be used as environmental parameters. These environmental parameters (such as the amount of data to be processed, the number of datasets, the amount of data, the number of data processing operators, and the amount of data loaded) are input into the aforementioned mathematical relationship or machine learning model. The output consists of multiple individuals, along with the time and resource consumption of each individual, thus forming the initial population P. t (g), for example, P t (g) contains n individuals, each individual representing a set of Spark runtime parameters, and each set of Spark runtime parameters has corresponding time consumption and resource consumption.

[0152] 702. Iterative output of a new population based on a knee-guided genetic algorithm.

[0153] In the initial stage, the initial population P can be initialized. t In step (g), n individuals undergo crossover and / or mutation to generate a new offspring population containing 2n individuals. Subsequently, based on a knee-oriented dominance relationship, superior individuals are selected, and n elite individuals are retained to generate a new generation population, i.e., the elite offspring population. If the convergence condition is not met, iteration continues, i.e., crossover and / or mutation are repeated on the new generation population until the convergence condition is met, at which point the new generation population is output.

[0154] The following section will elaborate on step 702.

[0155] 1. Individuals in a population

[0156] For example, individuals can be as shown in Table 5.

[0157] Table 5

[0158] The runtime parameters of the mathematical model established in Phase 1 are divided into three parts, totaling eight parameters. For example, the Spark runtime parameters for data extraction may include: batch size, number of partitions, and extraction parallelism; the Spark runtime parameters for data processing may include: number of processes, number of threads, and memory; and the Spark runtime parameters for data loading may include: write batch size and parallelism. Data extraction is further divided into multiple sets of parameters, the number of which is determined by the data source. In this example, three data sources are involved, therefore the data extraction is divided into three sets.

[0159] 2. Crossover and Mutation

[0160] In genetic algorithms, crossover aims to generate new superior individuals while preserving the superior genes of the parents. Mutation aims to expand the search range of solutions, retain a certain degree of randomness in offspring inheritance, and avoid getting trapped in local optima.

[0161] During the crossover and mutation process of Spark runtime parameters, the crossover and mutation logic can be illustrated as shown in Figure 8, as follows:

[0162] Crossover: Randomly select two parents from the population, and randomly choose parameters to crossover, creating new offspring. In the example, the two sets of parameters extracted from parent 1 and the two parameters loaded from the data are crossovered with the one set of parameters extracted from parent 2 and the three parameters processed from the data to form offspring.

[0163] Mutation: The offspring generation selects random parameters and performs random mutations within the allowed parameter range. In the example, the memory parameters from the offspring generation are selected for mutation, from 16 inherited from parent generation 2 to 32. The mutation range for each parameter is limited to the valid parameter range to prevent useless mutations.

[0164] 3. Optimal selection based on knee-oriented dominance relationship

[0165] First, let's introduce Knee points.

[0166] Knee points are natural preferred solutions in the Pareto-optima front (PoF). Knee points are often the best-performing solutions in PoF because they can lead to significant gains for most objectives by reasonably sacrificing the fitness of a small subset of objectives. In other words, for multi-objective problems without explicit preferences, knee points can outperform their neighborhood solutions.

[0167] For example, Figure 9 illustrates a knee point. The solid line represents the Proof-of-Flight (PoF), points B and C represent solutions within PoF, and point A represents the knee point within PoF. Geometrically, knee point A, compared to solution B, achieves a significant improvement in objective 2 by sacrificing a small fitness on objective 1. Similarly, compared to solution C, knee point A achieves a significant improvement in objective 1 by sacrificing a small fitness on objective 2. From an evaluation perspective, knee points often achieve the best practical performance and hypervolume. A set of predefined reference vectors can be used to decompose the objective space into a series of subspaces. The search process is driven by local α-dominance relations, searching for potential knee regions within the subspaces to accurately locate knee solutions and eliminate the influence of interfering solutions.

[0168] Therefore, this application provides a method for individual selection based on the knee-guided dominance relation, which maximizes the disassembly efficiency by rationally utilizing computational resources and narrowing the range of solutions. Assuming that solutions A and B exist in the problem space, the determinant μ in the knee-guided dominance relation is defined as follows:

[0169] in,

[0170] The acute angle is determined by the fitness of the i-th objective in solution A, maxf i (E) is the extreme value of the fitness of the i-th target in the population. N id It is the reference point, and ε is a very small positive number to ensure that the denominator is not zero.

[0171] f j (A) is the fitness of the j-th objective, and τ∈[0,1] is the key parameter controlling the knee-guided dominance region. If μ(A,B)<0, it means that solution A dominates solution B through the knee-guided relation. If both μ(A,B) and μ(B,A) are greater than zero, it means that solutions A and B cannot dominate each other through the knee-guided relation. In Figure 10, the shaded areas in the knee-guided dominance regions of solutions A, B, and C represent the respective knee-guided dominance regions of solutions A, B, and C. Solutions A and B cannot dominate each other through the knee-guided relation, while solution C is simultaneously dominated by the knee-guided relations of solutions A and B. Furthermore, the closer a solution is to the reference point, the larger its dominance region, and the more likely it is to become a knee point; such a solution is less likely to be dominated by other solutions. The dominance region expanded through the knee-guided relation can more accurately locate knee points and eliminate the influence of interfering solutions.

[0172] Therefore, the pseudocode corresponding to the specific steps of the individual selection process based on the knee-guided dominance relationship can be represented as follows:

[0173]

[0174] Each row can represent:

[0175] Lines 1-2: Targeting P t (g) Perform crossover and mutation to generate offspring population Q t (g), and P t (g) and Q t (g) Merge to generate population R t (g);

[0176] Line 3: Based on the knee guidance relationship, the population R t The individuals in (g) are ranked, and the population R is divided according to the dominance relationship among the individuals. t (g) Divide into multiple frontiers and assign a frontier number to each individual;

[0177] Lines 4-6: Extract individuals layer by layer in ascending order of the front edge number to form P. t (g+1), until the edge front surface is reached. The concept of the edge front surface is introduced; the edge front surface is the component that makes up P. t The last frontier of (g+1). Because the edge frontier contains more individuals than P... t (g+1) There is a lot of remaining population space, so only some individuals will be selected;

[0178] Lines 8-9: From the leading edge face L t Individuals are selected to join the population P in ascending order of RS. t (g+1). The aim of this is to select individuals with excellent average fitness while retaining individuals with high stability through competitive selection.

[0179] Therefore, in this embodiment, a knee-guided genetic algorithm is used to evolve individuals and select elite individuals. As the number of iterations increases, individuals with better evaluations are obtained, ultimately leading to the output of superior individuals, i.e., the runtime parameters for configuring computing devices. Based on this algorithm and a mathematical model, Spark runtime parameters that meet its objectives can be searched. This algorithm can quickly converge to the knee points that maximize efficiency and exhibits better performance for multi-objective problems. Therefore, in future system evolution where there is a need to optimize more objectives, expanding the current two objectives to multiple objectives also has good adaptability.

[0180] 703. Select Spark runtime parameters based on user preferences.

[0181] After the final population iteration is complete, a set of parameters is selected for execution based on a user preference strategy. This user preference strategy can be obtained by user input on the client side.

[0182] For example, the user preference strategy may include:

[0183] Strategy 1: Select the parameter that consumes the least amount of time.

[0184] Strategy 2: Select the parameter with the lowest resource consumption

[0185] Strategy 3: Choose parameters that balance time and resource consumption.

[0186] Therefore, it can provide users with a variety of preference settings without requiring users to configure parameter details. Users can set preference strategies to adapt to different solution selection strategies, realize automatic parameter selection and adaptation to Spark jobs.

[0187] In this embodiment, a knee-guided genetic algorithm is used. This algorithm can quickly converge to the knee points that maximize benefits and performs better for multi-objective problems. Therefore, in the future, when there is a need to optimize more objectives during the system's evolution, expanding the current two objectives to multiple objectives also has good adaptability.

[0188] For scenarios where big data Spark jobs are not frequently scheduled, such as monthly / yearly financial closings where historical data is not valuable, big data Spark jobs need to obtain optimal parameters before execution to minimize both time and resource consumption. The method provided in this application constructs a mathematical model and utilizes a knee-guided genetic algorithm to search for user-preferred Spark runtime parameters. This approach is applicable to infrequently run Spark tasks that require pre-run parameter optimization.

[0189] To facilitate understanding, the effects achieved by the embodiments of this application will be further described below.

[0190] For example, in the scenario of processing top-secret financial data, the method provided in this application embodiment can be applied to data encryption and decryption services based on Spark. The specific process is shown in Figure 11. This process involves main steps such as data extraction, data processing, and data loading. The encryption and decryption service performs encryption and decryption of top-secret data based on encryption and decryption algorithms during data extraction and data loading, thereby fulfilling the requirements for processing top-secret data. During the data extraction phase, data can be extracted from different databases, such as the data warehouse service (DWS) database and the highly fault-tolerant distributed file system (HDFS) database, as shown in Figure 11. Each database may contain top-secret and non-top-secret data. Non-top-secret data is data that cannot be disclosed. For non-top-secret data, data can be extracted directly. For top-secret data, in this embodiment, a decryption algorithm can be added to process the top-secret data to output non-top-secret and top-secret data. During the data processing phase, various encrypted data are processed and the processing results are output. During the data loading phase, data that needs to be encrypted, such as data obtained after decrypting top-secret data, can be encrypted. The unencrypted data and the encrypted data are loaded into the target database, thereby preventing the top-secret data from being leaked by encrypting the data that needs to be encrypted.

[0191] Furthermore, performing data encryption and decryption in Spark memory ensures both data security and data processing efficiency. Figure 12 shows the parameters and resource consumption of the method provided in this application applied to encryption performance testing. The method allows for testing with different data volumes, setting environment and runtime parameters for the data extraction, data processing, and data loading stages. Environment parameters can be set differently based on the data volume; runtime parameters are set to adapt to different stages. For example, in the data extraction stage, configured runtime parameters include data extraction parameters (batch size, number of partitions), data processing parameters (number of processes, number of threads, CPU, shuffle parallelism), and data loading parameters (batch size). Corresponding resource consumption can include Spark computation time, task preparation time, and total scheduling time. This example uses time resource consumption; in real-world scenarios, hardware resources or other resources can be substituted. Based on the parameter changes under different data volumes in Figure 12, it is clear that the timeliness of Spark jobs is greatly affected by the parameters. Therefore, Spark-based data encryption and decryption jobs require setting reasonable parameters before execution to ensure a balance between timeliness and resource consumption. The method provided in this application allows setting timeliness and resource consumption as optimization targets, thereby selecting an execution parameter scheme that meets actual needs.

[0192] Typically, financial data encryption and decryption scenarios often target monthly / yearly financial closing periods. Many Spark jobs run every month / quarter / year to meet these closing requirements. However, the large variations in data volume each month / year mean that fixed parameter settings cannot adapt to all scenarios, and historical job results are unreliable due to differences in environment (data volume). Therefore, the method provided in this application's embodiments is needed to perform reverse tuning before task execution, without relying on historical data. This allows for the use of a knee-guided genetic algorithm to search for user-preferred Spark runtime parameters.

[0193] Furthermore, a simpler and more user-friendly client interface can be provided. During the configuration phase, recommended runtime parameters can be displayed, and users can also manually set and adjust runtime parameters through the rule configuration page. For example, the client interface can be shown in Figures 13 and 14. These interfaces display a recommended runtime parameter scheme for the user. For instance, in Figure 13, recommended configuration parameters are displayed for parameters that need to be configured, such as extraction size, partition fields, number of partitions, and partition lower or upper bounds. Similarly, in Figure 14, recommended configuration parameters are displayed for parameters such as the number of processes, number of threads, running memory, batch size, and parallelism. This achieves automatic configuration of runtime parameters, providing users with a more user-friendly client and improving the user's configuration experience.

[0194] Therefore, this application provides a mathematical model for constructing Spark runtime parameters and ETL data processing for Spark job scenarios. Based on this mathematical model, a knee-guided genetic algorithm can be used to search for Spark runtime parameters preferred by the user, so as to achieve positive parameter recommendation for Spark jobs that cannot reuse historical data information.

[0195] The foregoing has described the method flow provided in this application. The following describes the equipment for executing the method flow provided in this application.

[0196] Referring to Figure 15, this application provides a schematic diagram of an operating parameter recommendation device. The operating parameter recommendation device includes:

[0197] Modeling module 1501 is used to establish, based on the information of historical tasks processed by the big data system, the data information corresponding to the task, the operating parameters of the computing device used to process the task and the time required to process the task, and the second relationship with the resources of the computing device consumed by the task.

[0198] Algorithm module 1502 is used to input the data information of the task to be processed, the first relationship, and the second relationship into the genetic algorithm. The processing flow of the genetic algorithm includes: performing various transformations on the parameter values ​​of the running parameters, and performing iterative calculations based on the data information of the task to be processed, the first relationship, and the second relationship, and outputting multiple sets of time values ​​and resource values.

[0199] The selection module 1503 is used to select one set of time values ​​and resource values ​​from multiple sets according to user needs, determine the parameter values ​​corresponding to the selected time values ​​and resource values, and use the parameter values ​​as the parameter values ​​of the computing device for processing the task to be processed.

[0200] In one possible implementation, the task to be processed includes: data extraction, data processing, or data loading. The data extraction model includes a model that extracts the data to be processed to the storage space of the computing device. The data processing model includes a model that processes the data to be processed according to a pre-defined logic. The data loading model is used to load the model output by the data processing model into a preset database. The running parameters include at least one of the following: read batch size, number of partitions, extraction parallelism, number of processes, number of threads, memory size, write batch size, or write parallelism. The read batch size includes the number of batches of data extracted from the data to be processed. The number of partitions includes the number of data partitions to be processed. The extraction parallelism includes the number of parallel jobs that extract the data to be processed into memory. The number of processes includes the number of processes running when processing the data to be processed. The number of threads includes the number of threads running when processing the data to be processed. The memory size includes the amount of memory occupied when processing the data to be processed. The write batch size includes the number of batches of data to be processed that are written into the database. The write parallelism includes the number of parallel jobs that are written into the database.

[0201] In one possible implementation, the first relationship is a mathematical model or machine learning model between the operating parameters of the computing device and the time required to process the task, and the second relationship is a mathematical model or machine learning model between the operating parameters of the computing device and the resources of the computing device consumed by the processing task.

[0202] In one possible implementation, the algorithm module 1502 executes any iteration of the genetic algorithm, specifically including: performing various transformations on the multiple sets of parameters output from the previous iteration, outputting multiple sets of transformation parameters; determining the time value and resource value corresponding to each set of transformation parameters according to the first relationship and the second relationship; selecting multiple sets of Pareto optimal solutions from the multiple sets of transformation parameters according to the time value and resource value corresponding to each set of transformation parameters, and outputting a set of Pareto optimal solutions; if the termination iteration condition is met, then outputting the set of Pareto optimal solutions; if the termination iteration condition is not met, then using the set of Pareto optimal solutions as the input for the next iteration and executing the next iteration.

[0203] In one possible implementation, algorithm module 1502 is specifically used to: obtain the guiding dominance relationship between each set of transformation parameters and other parameters in multiple sets of transformation parameters based on the time value and resource value corresponding to each set of transformation parameters, wherein the evaluation value corresponding to the time value and resource value of the dominant parameter in the guiding dominance relationship is better than the evaluation value corresponding to the time value and resource value of the dominated parameter; and select multiple sets of non-dominated parameters from multiple sets of transformation parameters based on the guiding dominance relationship between each set of transformation parameters and other parameters in multiple sets of transformation parameters, and output a Pareto optimal solution set.

[0204] In one possible implementation, the aforementioned transformation is to cross and / or mutate the parameters.

[0205] In one possible implementation, when the transformation includes mutation, the mutation process includes: mutating the operating parameters according to a pre-set parameter range to obtain the mutated operating parameters.

[0206] In one possible implementation, the selection module 1503 is specifically configured to: receive user input data, the user input data including a parameter selection strategy, the parameter selection strategy including parameter selection from a time dimension and / or a resource value consumption dimension; and select one of multiple sets of time values ​​and resource values ​​according to the parameter selection strategy.

[0207] Each module in the aforementioned operating parameter recommendation device can be implemented in software or hardware. For example, the implementation of the algorithm module will be described below. Similarly, the implementation of other modules, such as the selection module and the modeling module, can refer to the implementation of the algorithm module.

[0208] As an example of a software functional unit, an algorithm module may include code running on a computing instance. This computing instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Furthermore, the aforementioned computing devices may be one or more. For example, an algorithm module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed within the same region or in different regions. Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same Availability Zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0209] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Cross-region communication between two Virtual Private Clouds (VPCs) within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC, enabling interconnection between VPCs.

[0210] As an example of a hardware functional unit, an algorithm module may include at least one computing device, such as a server. Alternatively, an algorithm module may also be a device implemented using a central processing unit (CPU), application-specific integrated circuit (ASIC), programmable logic device (PLD), complex programmable logic device (CPLD), field-programmable gate array (FPGA), generic array logic (GAL), data processing unit (DPU), embedded neural network processing unit (NPU), system-on-chip (SoC), offloading card, accelerator card, etc. The aforementioned PLD can be implemented using CPLD, FPGA, GAL, or any combination thereof.

[0211] The algorithm module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the algorithm module includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the algorithm module includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0212] As an example of a software functional unit, the runtime parameter recommendation device may include code running on a computing instance. This computing instance can be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Furthermore, the aforementioned computing device may be one or more. For example, the runtime parameter recommendation device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed within the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed within the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0213] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0214] As an example of a hardware functional unit, the operating parameter recommendation device may include at least one computing device, such as a server. Alternatively, the operating parameter recommendation device may also be a device implemented using an ASIC or a PLD. The aforementioned PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0215] The runtime parameter recommendation device includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multiple computing devices can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0216] This application also provides a chip system including a processor and a power supply circuit. The power supply circuit supplies power to the processor, which executes the operation steps corresponding to the method provided in this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices such as a DPU, NPU, XPU, SoC, offload card, or accelerator card.

[0217] This application also provides a computing device 100. As shown in FIG16, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0218] Bus 102 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in Figure 16, but this does not imply that there is only one bus or one type of bus. Bus 104 can include pathways for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108). The Unified Bus can also be referred to as the Lingqu Bus.

[0219] The processor 104 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0220] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 106 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0221] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 106, or two or more types of storage media can be configured to realize the function of memory 106. This application does not limit this.

[0222] The memory 106 stores executable program code, and the processor 104 executes the executable program code to implement the functions of each module mentioned in FIG15, thereby implementing the method provided in this application. That is, the memory 106 stores instructions for executing the method provided in this application.

[0223] Alternatively, the memory 106 stores executable code, which the processor 104 executes to implement the functions of the aforementioned selection module, modeling module, or algorithm module, thereby implementing the method provided in this application. That is, the memory 106 stores instructions for executing the method provided in this application.

[0224] The communication interface 103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0225] As one possible implementation, the computing device 100 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the method provided in this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offload card, or accelerator card.

[0226] As one possible implementation, the computing device 100 may include various types of processors 104, meaning the computing device 100 is a heterogeneous device. For example, the computing device 100 may include a CPU and a GPU, and at least one of the processors 104 may execute the operation steps corresponding to the method provided in this application. For the sake of brevity, further details are omitted here.

[0227] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0228] As shown in Figure 17, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for executing the methods provided in this application.

[0229] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the methods provided in this application. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the methods provided in this application.

[0230] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the joint test device. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more of the aforementioned selection module, modeling module, or algorithm module.

[0231] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 18 illustrates one possible implementation. As shown in Figure 18, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 106 in computing device 100A stores instructions for executing the functions of a selection module, a modeling module, or an algorithm module. Simultaneously, the memory 106 in computing device 100B stores instructions for executing the functions of a selection module, a modeling module, or an algorithm module.

[0232] It should be understood that the functions of computing device 100A shown in Figure 18 can also be performed by multiple computing devices 100. Similarly, the functions of computing device 100B can also be performed by multiple computing devices 100.

[0233] The connection method between the computing device clusters shown in Figure 18 can be based on the fact that the method provided in this application requires a large amount of computing power, needs to achieve load balancing, or requires a large amount of data storage. Therefore, different modules are deployed in different computing devices. For example, the functions implemented by the algorithm module are executed by computing device 100A, and the functions implemented by the modeling module are executed by computing device 100B.

[0234] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device clusters described in Figures 17 and 18. The difference is that the memory 106 of one or more computing devices 100 in this computing device cluster can store the same instructions for executing the method provided in this application.

[0235] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the methods provided in this application. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the methods provided in this application.

[0236] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions for executing some functions of the operating parameter recommendation device provided in this application. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more modules such as the selection module, modeling module, or algorithm module.

[0237] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method provided in this application.

[0238] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the method provided in this application.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recommending operating parameters, characterized in that, include: Based on the data information of historical tasks processed by the big data system, establish the data information corresponding to the task, the operating parameters of the computing device that processes the task, the first relationship between the time required to process the task and the second relationship between the resources of the computing device consumed in processing the task; The data information of the task to be processed, the first relationship, and the second relationship are input into the genetic algorithm. The processing flow of the genetic algorithm includes: performing various transformations on the parameter values ​​of the running parameters, and performing iterative calculations based on the data information of the task to be processed, the first relationship, and the second relationship, and outputting multiple sets of time values ​​and resource values. According to user requirements, one set of time values ​​and resource values ​​is selected from the multiple sets of time values ​​and resource values, and the parameter values ​​corresponding to the selected time values ​​and resource values ​​are determined. The parameter values ​​are then used as the parameter values ​​of the computing device that processes the task to be processed.

2. The method according to claim 1, characterized in that, The tasks to be processed include: data extraction, data processing, or data loading. The data extraction model includes a model that extracts the data to be processed to the storage space corresponding to the computing device. The data processing model includes a model that processes the data to be processed according to a pre-defined logic. The data loading model is used to load the model output by the data processing model into a preset database. The running parameters include at least one of the following: read batch size, number of partitions, extraction parallelism, number of processes, number of threads, memory size, write batch size, or write parallelism. The read batch size includes the number of batches of data extracted from the data to be processed. The number of partitions includes the number of partitions of the data to be processed. The extraction parallelism includes the number of parallel jobs that extract the data to be processed into memory. The number of processes includes the number of processes that run when processing the data to be processed. The number of threads includes the number of threads that run when processing the data to be processed. The memory size includes the amount of memory occupied when processing the data to be processed. The write batch size includes the number of batches of data to be processed that are written into the database. The write parallelism includes the number of parallel jobs that write the data to be processed into the database.

3. The method according to claim 1 or 2, characterized in that, The first relationship is a mathematical model or machine learning model between the operating parameters of the computing device and the time required to process the task, and the second relationship is a mathematical model or machine learning model between the operating parameters of the computing device and the resources of the computing device consumed in processing the task.

4. The method according to any one of claims 1-3, characterized in that, The specific processing flow for any iteration of the genetic algorithm includes: Perform various transformations on the multiple sets of parameters output from the previous iteration, and output multiple sets of transformed parameters; The time value and resource value corresponding to each set of transformation parameters are determined based on the first relationship and the second relationship; Based on the time value and resource value corresponding to each set of transformation parameters, multiple Pareto optimal solutions are selected from the multiple sets of transformation parameters, and a set of Pareto optimal solutions is output. If the termination iteration condition is met, then the Pareto optimal solution set is output; If the termination iteration condition is not met, the Pareto optimal solution set is used as the input for the next iteration, and the next iteration process is executed.

5. The method according to claim 4, characterized in that, The step involves selecting multiple Pareto optimal solutions from the multiple sets of transformation parameters based on the time and resource values ​​corresponding to each set of transformation parameters, and outputting a set of Pareto optimal solutions, including: Based on the time value and resource value corresponding to each set of transformation parameters, the guiding dominance relationship between each set of transformation parameters and other parameters in the multiple sets of transformation parameters is obtained. In the guiding dominance relationship, the evaluation value corresponding to the time value and resource value of the dominant is better than the evaluation value corresponding to the time value and resource value of the dominated. Based on the guiding dominance relationship between each set of transformation parameters and other parameters in the plurality of transformation parameters, a plurality of undominated parameters are selected from the plurality of transformation parameters, and the Pareto optimal solution set is output.

6. The method according to claim 4 or 5, characterized in that, The transformation involves crossing and / or mutating the parameters.

7. The method according to claim 6, characterized in that, When the transformation includes mutation, the mutation processing flow includes: mutating the operating parameters according to a preset parameter range to obtain the mutated operating parameters.

8. The method according to any one of claims 1-7, characterized in that, The step of selecting one set from the multiple sets of time values ​​and resource values ​​according to user needs includes: Receive user input data, the user input data including a parameter selection strategy, the parameter selection strategy including parameter selection from the time dimension and / or the resource value consumption dimension; Based on the parameter selection strategy, one set of time values ​​and resource values ​​is selected from the multiple sets.

9. A device for recommending operating parameters, characterized in that, include: The modeling module is used to establish, based on the data information of historical tasks processed by the big data system, the data information corresponding to the task, the first relationship between the operating parameters of the computing device processing the task and the time required to process the task, and the second relationship between the computing device resources consumed by processing the task; The algorithm module is used to input the data information of the task to be processed and the first relationship and the second relationship into the genetic algorithm. The processing flow of the genetic algorithm includes: performing various transformations on the parameter values ​​of the running parameters, and performing iterative calculations based on the data information of the task to be processed and the first relationship and the second relationship, and outputting multiple sets of time values ​​and resource values. The selection module is used to select one set of time values ​​and resource values ​​from the multiple sets according to user needs, determine the parameter values ​​corresponding to the selected time values ​​and resource values, and use the parameter values ​​as the parameter values ​​of the computing device for processing the task to be processed.

10. The apparatus according to claim 9, characterized in that, The tasks to be processed include: data extraction, data processing, or data loading. The data extraction model includes a model that extracts the data to be processed to the storage space corresponding to the computing device. The data processing model includes a model that processes the data to be processed according to a pre-defined logic. The data loading model is used to load the model output by the data processing model into a preset database. The running parameters include at least one of the following: read batch size, number of partitions, extraction parallelism, number of processes, number of threads, memory size, write batch size, or write parallelism. The read batch size includes the number of batches of data extracted from the data to be processed. The number of partitions includes the number of partitions of the data to be processed. The extraction parallelism includes the number of parallel jobs that extract the data to be processed into memory. The number of processes includes the number of processes that run when processing the data to be processed. The number of threads includes the number of threads that run when processing the data to be processed. The memory size includes the amount of memory occupied when processing the data to be processed. The write batch size includes the number of batches of data to be processed that are written into the database. The write parallelism includes the number of parallel jobs that write the data to be processed into the database.

11. The apparatus according to claim 9 or 10, characterized in that, The first relationship is a mathematical model or machine learning model between the operating parameters of the computing device and the time required to process the task, and the second relationship is a mathematical model or machine learning model between the operating parameters of the computing device and the resources of the computing device consumed in processing the task.

12. The apparatus according to any one of claims 9-11, characterized in that, The algorithm module executes any iteration of the genetic algorithm, specifically including the following processes: Perform various transformations on the multiple sets of parameters output from the previous iteration, and output multiple sets of transformed parameters; The time value and resource value corresponding to each set of transformation parameters are determined based on the first relationship and the second relationship; The time value and resource value corresponding to each set of transformation parameters are determined based on the first relationship and the second relationship; Based on the time value and resource value corresponding to each set of transformation parameters, multiple Pareto optimal solutions are selected from the multiple sets of transformation parameters, and a set of Pareto optimal solutions is output. If the termination iteration condition is met, then the Pareto optimal solution set is output; If the termination iteration condition is not met, the Pareto optimal solution set is used as the input for the next iteration, and the next iteration process is executed.

13. The apparatus according to claim 12, characterized in that, The algorithm module is specifically used for: Based on the time value and resource value corresponding to each set of transformation parameters, the guiding dominance relationship between each set of transformation parameters and other parameters in the multiple sets of transformation parameters is obtained. In the guiding dominance relationship, the evaluation value corresponding to the time value and resource value of the dominant is better than the evaluation value corresponding to the time value and resource value of the dominated. Based on the guiding dominance relationship between each set of transformation parameters and other parameters in the plurality of transformation parameters, a plurality of undominated parameters are selected from the plurality of transformation parameters, and the Pareto optimal solution set is output.

14. The apparatus according to claim 12 or 13, characterized in that, The transformation involves crossing and / or mutating the parameters.

15. The apparatus according to claim 14, characterized in that, When the transformation includes mutation, the mutation processing flow includes: mutating the operating parameters according to a preset parameter range to obtain the mutated operating parameters.

16. The apparatus according to any one of claims 9-15, characterized in that, The selection module is specifically used for: Receive user input data, the user input data including a parameter selection strategy, the parameter selection strategy including parameter selection from the time dimension and / or the resource value consumption dimension; Based on the parameter selection strategy, one set of time values ​​and resource values ​​is selected from the multiple sets.

17. A computing device, characterized in that, The computing device includes a processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the operational steps of the method as described in any one of claims 1 to 8.

18. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 8.

19. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the operation steps of the method as described in any one of claims 1 to 8.

20. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the operational steps of the method as described in any one of claims 1 to 8.

Images