Heterogeneous computing-based parameter processing method and device

Abstract

The present application provides a heterogeneous computing-based parameter processing method. The method is applied to a heterogeneous big data platform, the heterogeneous big data platform is used for managing an infrastructure, and the infrastructure comprises at least two cores with different computing power. The method comprises: acquiring computational work and configuration parameter tuning settings inputted by a user, wherein the computational work is run by the heterogeneous big data platform, and the configuration parameter tuning settings are used for determining tuning settings of related configuration parameters for running the computational work on the heterogeneous big data platform; acquiring a running index related to the computational work, wherein the running index comprises a software index and a hardware index; and according to the computational work, the running index, and the configuration parameter tuning settings, determining a first configuration parameter set, wherein the first configuration parameter set comprises some or all of the related configuration parameters for running the computational work on the heterogeneous big data platform. The parameter processing method provided by the present application can adapt to a heterogeneous multi-core computing platform, so as to perform timely adjustment and optimization on parameters.

Description

Parameter processing method and device based on heterogeneous computing

[0001] This application claims priority to Russian patent application No. 2023115476, filed with the Russian Federal Intellectual Property Office on June 14, 2023, entitled “Method and Device for Parameter Optimization,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of computer technology, and more specifically, to a parameter processing method and apparatus based on heterogeneous computing. Background Art

[0003] Modern cloud big data engines are based on distributed cluster deployments, enabling efficient data analysis. In practical applications, big data engines and computing platforms are often used together to achieve efficient and scalable big data processing and computing tasks.

[0004] For big data engines, proper parameter configuration is crucial. For example, Spark, as a relatively complex system, requires a large number of parameters, such as the degree of parallelism. Improper parameter configuration can lead to poor job performance, significant resource waste, low fault tolerance, and even task failure. For big data engines running on heterogeneous multi-core computing platforms with uneven computing power, parameter optimization is even more crucial. For example, the requirements for setting parameters such as the number of heterogeneous cores and the degree of parallelism are more stringent.

[0005] Therefore, how to construct a parameter processing method that can adapt to heterogeneous multi-core computing platforms is a technical problem that needs to be solved urgently.

[0006] Summary of the Invention

[0007] The present application provides a parameter processing method and device based on heterogeneous computing, which can adapt to heterogeneous multi-core computing platforms and adjust and optimize parameters in a timely manner.

[0008] In a first aspect, a parameter processing method based on heterogeneous computing is provided, which is applied to a heterogeneous big data platform, where the heterogeneous big data platform is used to manage infrastructure, and the infrastructure includes at least two cores with unequal computing power. The method includes: obtaining a computing job and configuration parameter tuning settings input by a user, where the computing job is run by the heterogeneous big data platform, and the configuration parameter tuning settings are used to determine the tuning settings of relevant configuration parameters for the heterogeneous big data platform to run the computing job; obtaining operating indicators related to the computing job, where the operating indicators include software indicators and hardware indicators; and determining a first configuration parameter set based on the computing job, the operating indicators, and the configuration parameter tuning settings, where the first configuration parameter set includes some or all of the relevant configuration parameters of the heterogeneous big data platform to run the computing job.

[0009] This application provides a parameter processing method based on heterogeneous computing, which can adjust and optimize the parameters of computing jobs running on heterogeneous big data platforms according to the configuration parameter tuning settings input by the user, and adapt to heterogeneous multi-core computing platforms.

[0010] Computing jobs run on heterogeneous big data platforms according to configuration parameters to perform processing tasks. Typically, computing jobs run according to multiple configuration parameters to perform processing tasks, so these multiple configuration parameters can also be referred to as the configuration parameter set of the computing job. Exemplarily, the configuration parameters in the configuration parameter set can be the number of computing nodes used to execute the job, the amount of memory available on the nodes, the parallelism of the job, the maximum execution time of the job, or the parameters of the algorithm. The first configuration parameter set can be a relatively good or optimal configuration parameter set for running the computing job, predicted according to the method provided in this application.

[0011] In some possible application scenarios, users can enter or select the frequency, optimization intensity, or tuning scenario for parameter tuning of computing jobs. Users can also choose whether to enable automatic tuning of configuration parameters.

[0012] Exemplarily, the parameter processing method provided in the present application can obtain computing jobs input by the user, configuration parameter tuning settings, and software indicators and hardware indicators related to the computing jobs. The software indicators may include software indicators collected before running the computing jobs and software indicators collected during running the computing jobs. The hardware indicators may include hardware indicators collected before running the computing jobs and hardware indicators collected during running the computing jobs. The method provided in the present application runs computing jobs on a heterogeneous big data platform according to the configuration parameter tuning settings input by the user, collects and analyzes running indicators, and continuously updates the optimal configuration parameter set corresponding to the computing job obtained in each round of tuning until the configuration parameter tuning settings are met. The optimal configuration parameter set obtained at this time is the first configuration parameter set.

[0013] Exemplarily, the first configuration parameter set can be an optimal or better configuration parameter set obtained by analyzing and predicting the operating indicators of the computing job using a prediction model, or it can be a configuration parameter set preset based on domain knowledge, where the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the operating indicators.

[0014] Exemplarily, a computing job may be a big data engine in each dimension or process of a big data job. Big data engines include, but are not limited to, Apache Hadoop, Apache Spark, Flink, Storm, and Apache Cassandra. A heterogeneous big data platform may be a software platform for managing any infrastructure with uneven computing power. Exemplarily, the infrastructure may be an infrastructure including super-normal cores or an infrastructure including a central processing unit (CPU) plus a graphics processing unit (GPU). The infrastructure may include a Yarn resource pool, which includes multiple cores with different computing powers. For example, the Yarn resource pool may include at least two of super-cores, normal cores, or heterogeneous nodes. Normal cores are cores with computing power lower than a preset threshold, super-cores are cores with computing power higher than a preset threshold, and heterogeneous nodes are mixed nodes of super-cores and normal cores.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the software indicators include a first software indicator and a second software indicator, the hardware indicators include a first hardware indicator and a second hardware indicator, the first software indicator includes a software indicator collected before running the computing job, the second software indicator includes a software indicator collected during the running of the computing job, the first hardware indicator includes a hardware indicator collected before running the computing job, and the second hardware indicator includes a hardware indicator collected during the running of the computing job.

[0016] This application provides a parameter processing method based on heterogeneous computing, which can collect indicators before and during the operation of computing jobs, adjust parameter optimization in real time according to information such as environmental changes and operating conditions, and can adapt to heterogeneous multi-core computing platforms.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the first software indicator includes information on the usage of software resources provided by the infrastructure by running programs in the infrastructure before the computing job runs; the second software indicator includes information on the usage of software resources provided by the infrastructure by the computing job during the running of the computing job; the first hardware indicator includes information on the usage of hardware resources provided by the infrastructure by running programs in the infrastructure before the computing job runs; and the second hardware indicator includes information on the usage of hardware resources provided by the infrastructure by the computing job during the running of the computing job.

[0018] Exemplarily, the software metric includes multiple candidate configuration parameter sets, wherein any one of the multiple candidate configuration parameter sets includes any one or more of configuration parameters preset before running a computing job and configuration parameters collected during the running of the computing job. Some or all of the configuration parameters in the multiple candidate configuration parameter sets belong to the first software metric or the second software metric.

[0019] The running programs in the infrastructure before the computing job runs can be other big data jobs in the environment where the computing job will run, or other programs that are not related to big data.

[0020] For example, the first software indicator includes the operating system load capacity and cluster scheduling load capacity before the computing job is executed, and the second software indicator includes the operating system load capacity and cluster scheduling load capacity during the computing job is executed.

[0021] Exemplarily, the first hardware indicator includes the usage of CPU resources, memory resources, network card resources or disk resources by the running program in the infrastructure before the computing job is run, and the second hardware indicator includes the usage of CPU resources, memory resources, network card resources, disk resources or cores by the computing job during the running of the computing job.

[0022] This application provides a parameter processing method based on heterogeneous computing, which can collect indicators before and during the operation of computing jobs, adjust parameter optimization in real time according to information such as environmental changes, operating conditions, resource usage, etc., and can adapt to heterogeneous multi-core computing platforms.

[0023] In combination with the first aspect, in certain implementations of the first aspect, determining the first configuration parameter set based on the computing job, operating indicators and configuration parameter tuning settings includes: obtaining the second configuration parameter set, the first software indicator and the first hardware indicator of the computing job, the second configuration parameter set being a historical configuration parameter set or a preset configuration parameter set; running the computing job according to the second configuration parameter set to determine the second software indicator and the second hardware indicator of the computing job; obtaining a priori model and multiple candidate configuration parameter sets, some or all of the configuration parameters in the multiple candidate configuration parameter sets belong to the first software indicator or the second software indicator; determining the first configuration parameter set based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, multiple candidate configuration parameter sets, the priori model and the configuration parameter tuning settings, the priori model is used to predict the first configuration parameter set.

[0024] The second configuration parameter set may be an optimal configuration parameter set obtained through historical parameter tuning, or may be a preset configuration parameter set, such as a preset configuration parameter set based on domain knowledge.

[0025] In one possible implementation, the prior model is a Gaussian prior model. Optionally, the prior model can also be a Laplace prior model, an exponential prior model, a uniform prior model, etc. This application does not limit the specific type of the prior model. The method provided in this application can use the prior model to predict the first configuration parameter set.

[0026] The first configuration parameter set includes multiple configuration parameters, which may be from at least two of the multiple candidate configuration parameter sets, or from one of the candidate configuration parameter sets.

[0027] This application provides a parameter processing method based on heterogeneous computing, which can collect indicators before and during the operation of computing jobs, adjust parameter optimization in real time according to information such as environmental changes and operating conditions, and can adapt to heterogeneous multi-core computing platforms.

[0028] In combination with the first aspect, in certain implementations of the first aspect, before determining the first configuration parameter set, the method also includes: obtaining a white box model, the white box model is used to enhance the predictive performance of the prior model, the white box model is constructed based on domain knowledge, the domain knowledge includes performance diagnosis and optimization suggestions corresponding to operating indicators; and strengthening the prior model based on the white box model.

[0029] The domain knowledge base may include domain knowledge of big data engines and / or domain knowledge of heterogeneous platforms. Taking Spark as an example, before starting to tune the computing job, the present application can analyze the principles of Spark, establish a white box performance model, and strengthen the prior model. In the subsequent iterative tuning process, the white box model is continuously updated, and the prior model is strengthened based on the updated white box model. A white box model refers to a model that can explain and understand the internal operating mechanism of the model, and its internal structure and parameters can be directly observed and understood. The white box model of the present application can be based on the principles of Spark, and by analyzing the internal implementation and operating mechanism of Spark, the relationship between input and output is modeled to predict the output result.

[0030] It should be understood that the white box model in this application can be placed in a heterogeneous big data platform, for example, it can be implemented through a plug-in, and can be continuously optimized as the configuration parameters are tuned.

[0031] This application provides a parameter processing method based on heterogeneous computing, which can use a white box model based on domain knowledge to strengthen the prior model, improve modeling accuracy, and thus improve the predictive performance of the prior model.

[0032] In combination with the first aspect, in certain implementations of the first aspect, determining the first configuration parameter set based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, multiple candidate configuration parameter sets, the prior model and the configuration parameter tuning settings includes: obtaining an acquisition function, the acquisition function is used to enhance the predictive performance of the prior model; determining the first configuration parameter set based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, multiple candidate configuration parameter sets, the prior model, the acquisition function and the configuration parameter tuning settings.

[0033] The acquisition function is a method used to guide the selection of the next sampling point, helping to find potentially better configuration parameters near the known optimal configuration parameters. In the method provided by this application, the expected improvement (EI) posterior acquisition function and a Markov Chain Monte Carlo (MCMC)-based method can be used to optimize the EI itself to achieve a better balance between the exploitation of known optimal points and the exploration of new areas.

[0034] This application provides a parameter processing method based on heterogeneous computing. By designing an acquisition function, it can achieve a better balance between the development and utilization of known optimal points and the exploration of new areas in the process of finding the optimal configuration parameter set, thereby improving the predictive performance of the prior model.

[0035] In combination with the first aspect, in certain implementations of the first aspect, before determining the first configuration parameter set based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, multiple candidate configuration parameter sets, the prior model and the configuration parameter tuning settings, the method also includes: matching domain knowledge according to the operating indicators, the domain knowledge including performance diagnosis and optimization suggestions corresponding to the operating indicators; optimizing the computing job or the relevant configuration parameters of the computing job according to the optimization suggestions in the domain knowledge.

[0036] For example, each operational metric can be matched with a performance diagnosis and optimization suggestion feedback. Based on this matching result, jobs can be pre-optimized to improve parameter processing efficiency. For example, the domain knowledge base contains Spark domain knowledge. For example, the Spark domain knowledge matching a certain operational metric indicates excessive memory usage, suggesting appropriate memory reduction. This allows pre-optimization of jobs before modeling, improving overall parameter processing efficiency.

[0037] This application provides a parameter processing method based on heterogeneous computing, which can pre-optimize computing jobs through domain knowledge, can be used universally among different user job optimizations, reduce the overall parameter optimization complexity, and improve the overall parameter processing efficiency.

[0038] In combination with the first aspect, in certain implementations of the first aspect, when running the computing job for the first N times, the second configuration parameter set is a configuration parameter set preset based on domain knowledge, and the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the operating indicators, and N is greater than or equal to 1.

[0039] This application provides a parameter processing method based on heterogeneous computing, which can preset the configuration parameter set for the first N times of running computing jobs through domain knowledge to solve the cold start problem.

[0040] In combination with the first aspect, in some implementations of the first aspect, the prior model is a Gaussian prior model.

[0041] In combination with the first aspect, in certain implementations of the first aspect, the domain knowledge includes any one or more of performance diagnosis and optimization suggestions related to computing jobs, and performance diagnosis and optimization suggestions related to heterogeneous big data platforms.

[0042] In combination with the first aspect, in certain implementations of the first aspect, the performance diagnosis and optimization suggestions related to computing jobs include performance diagnosis and optimization suggestions for any one or more of the following aspects related to computing jobs: memory management, resource management and scheduling, task scheduling, reorganization operations, directed acyclic graphs, dependencies and resilient distributed data sets.

[0043] In combination with the first aspect, in certain implementations of the first aspect, the performance diagnosis and optimization suggestions related to the heterogeneous big data platform include performance diagnosis and optimization suggestions for any one or more of the following aspects related to the heterogeneous big data platform: core utilization information, performance parameters, virtualization layer characteristics, physical machine layer scheduling, and cloud instance design.

[0044] This application provides a parameter processing method based on heterogeneous computing, which can collaboratively analyze software indicators and hardware indicators, consider more dimensional features than existing optimization methods, identify performance bottlenecks in advance, and at the same time reduce parameter optimization costs and improve parameter optimization performance.

[0045] In combination with the first aspect, in certain implementations of the first aspect, before determining the first configuration parameter set based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, multiple candidate configuration parameter sets, the prior model and the configuration parameter tuning settings, the method also includes: performing at least one of important parameter analysis and sensitive query analysis, the important parameter analysis including screening out candidate configuration parameters from multiple candidate configuration parameter sets whose importance is greater than a first preset threshold, and the sensitive query analysis including filtering out scenarios whose sensitivity to parameter tuning is less than a second preset threshold.

[0046] Important parameter analysis includes screening out configuration parameters whose importance is greater than a preset threshold among candidate configuration parameters, and sensitive query analysis includes filtering out scenarios that are insensitive to parameter tuning. For example, important parameters can be analyzed using methods such as kernel principal component analysis (KPCA), and sensitive query analysis can be performed using coefficient of variation calculation. For example, in a possible implementation scenario, 100 collected job configuration parameters can be screened to 20 through important parameter analysis, and 100 query jobs to be optimized can be simplified to 10 through sensitive query analysis.

[0047] The present application provides a parameter processing method based on heterogeneous computing, which can perform at least one of important parameter analysis and sensitive query analysis on computing jobs, reduce parameter optimization costs, and improve parameter optimization performance.

[0048] In combination with the first aspect, in certain implementations of the first aspect, any one of the multiple candidate configuration parameter sets includes any one or more of configuration parameters preset before running the computing job and configuration parameters collected during the running of the computing job.

[0049] In combination with the first aspect, in certain implementations of the first aspect, the first software indicator includes any one or more of the execution plan of the computing job, the directed acyclic graph of the computing job, the data volume of the computing job, the operator of the computing job, and the parallelism of the computing job; the second software indicator includes any one or more of the parallelism of the computing job, the amount of reorganization operation data, and the scale of data overflow.

[0050] In combination with the first aspect, in certain implementations of the first aspect, the second software indicator and the second hardware indicator are obtained by extracting from a running log of the computing job.

[0051] In combination with the first aspect, in some implementations of the first aspect, the second hardware indicator includes any one or more of a performance monitoring unit indicator and a multi-level cache memory cache bandwidth.

[0052] In a second aspect, an embodiment of the present application provides a parameter processing device, which includes a module or unit for implementing the first aspect or any possible implementation method of the first aspect.

[0053] In a third aspect, an embodiment of the present application provides a parameter processing device, which includes a processor, which is used to couple with a memory, read and execute instructions and / or program codes in the memory to execute the first aspect or any possible implementation of the first aspect.

[0054] In a fourth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect.

[0055] Optionally, the processor may be a general-purpose processor, which may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like; when implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated into the processor or located independently of the processor.

[0056] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method in the first aspect and any one of the implementations of the first aspect.

[0057] In a sixth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to the first aspect and any one of the implementations of the first aspect.

[0058] By way of example, these computer-readable storages include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and a hard drive.

[0059] Optionally, as an implementation manner, the above-mentioned storage medium may specifically be a non-volatile storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] FIG1 is an exemplary flowchart of a parameter processing method based on heterogeneous computing provided in an embodiment of the present application.

[0061] FIG2 is a schematic diagram of an application scenario of a parameter processing method based on heterogeneous computing provided in an embodiment of the present application.

[0062] FIG3 is a schematic diagram of an application scenario of another parameter processing method based on heterogeneous computing provided in an embodiment of the present application.

[0063] FIG4 is an exemplary flowchart of another parameter processing method provided in an embodiment of the present application.

[0064] FIG5 is an exemplary flowchart of another parameter processing method provided in an embodiment of the present application.

[0065] FIG6 is a schematic block diagram of a parameter processing device provided in an embodiment of the present application.

[0066] FIG7 is a schematic diagram of the architecture of a computing device provided in an embodiment of the present application.

[0067] FIG8 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0068] FIG9 is a schematic diagram of a network connection between computing devices provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts should fall within the scope of protection of this application.

[0070] In the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0071] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field will know that with the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0072] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0073] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0074] To facilitate understanding of the embodiments of the present application, some definitions involved in the present application are first briefly explained.

[0075] 1. Spark: A fast and general computing engine designed for large-scale data processing.

[0076] 2. Executor: A type of Java virtual machine (JVM) in Spark that is responsible for executing specific tasks in Spark jobs.

[0077] 3. Driver: A type of JVM process in Spark that is responsible for resource application, parsing, and scheduling of specific Spark job tasks.

[0078] 4. Task: The smallest unit of job execution in Spark.

[0079] 5. Stage: An execution stage of a Spark job, which contains multiple tasks.

[0080] 6. Job: A set of stages of a Spark job, triggered by a specific action.

[0081] 7. Yet another resource negotiator (Yarn): A resource manager in a Spark cluster responsible for resource scheduling and management.

[0082] 8. YarnAllocator: The module in Spark that interacts with Yarn and is responsible for resource application.

[0083] 9. Container: The basic unit of processing power in Yarn, which is an encapsulation of resources such as memory and central processing unit (CPU).

[0084] 10.Expectation improvement (EI): A type of acquisition function.

[0085] 11. Bayesian optimization (BO): Bayesian optimization is a black-box optimization algorithm used to solve the extreme value problem of a function with unknown expression.

[0086] 12. Markov Chain Monte Carlo (MCMC): is a set of algorithms that use Markov chains to sample from a random distribution, using the previous steps as a basis. The more steps, the better the results.

[0087] 13. Resilient Distributed Datasets (RDD): This is the most fundamental data abstraction in Spark. In the code, it is an abstract class that represents a resilient, immutable, partitionable collection whose elements can be computed in parallel.

[0088] 14. Shuffle: In Spark, Shuffle is an important data shuffling operation, which usually occurs in data transformation operations (such as groupByKey, reduceByKey, join, etc.) and is used to repartition and merge data for subsequent calculations and analysis.

[0089] 15. Heterogeneous computing architecture (big.LITTLE): In this architecture, "big clusters" consisting of more power-hungry but powerful processor cores (hypercores) are combined with "LITTLE clusters" consisting of lower-power, less powerful processor cores (normal cores). These processor cores share memory segments and can dispatch and switch loads between different CPU clusters in real time.

[0090] 16. Roofline: A performance analysis and optimization tool that maps job performance data and hardware performance parameters into a chart, helping users identify and optimize performance bottlenecks in Spark jobs to improve job execution efficiency and speed.

[0091] 17. Task memory manager (TMA): It helps Spark dynamically manage memory allocation and release of tasks during task execution to avoid memory overflow and performance bottlenecks.

[0092] Modern Cloud Big Data Engine is a big data processing platform based on cloud computing technology. It provides efficient, scalable, elastic, secure, and cost-effective big data processing and analysis services in a cloud environment. Modern Cloud Big Data Engine supports a variety of data processing and analysis tasks, including batch processing, stream processing, machine learning, graph computing, search, and recommendation. It offers a variety of data processing engines and frameworks, such as Apache Hadoop, Apache Spark, Flink, Storm, and Apache Cassandra.

[0093] Take Spark, for example. With hundreds of thousands of jobs running daily on the cloud, Spark has become a leading engine for large-scale data batch processing. Spark, based on a distributed cluster deployment model, transforms large-scale data analysis jobs into numerous highly parallel computing tasks, enabling efficient data analysis. Therefore, scheduling technology for tasks and cluster resources is essential.

[0094] Existing big data user jobs on existing networks waste significant computing power. At the cluster level, idle periods of central processing unit (CPU) resources occur between user jobs. Furthermore, depending on system default or user-defined operating parameters, user jobs may also experience idle CPU resources during execution. At the task level, the computational load is uneven across different tasks, with some tasks consuming less CPU resources. Therefore, integrating Spark task execution with heterogeneous computing platforms with uneven computing power is essential. Better matching computing power with big data workloads through heterogeneous computing platforms can effectively avoid wasted computing power. Furthermore, because the parallelism of Spark big data jobs is a major performance bottleneck, using more cores with average computing power (normal cores) instead of more powerful cores (hypercores) at the same cost can improve parallelism and total computing power, thereby achieving a better price-performance ratio.

[0095] At the same time, Spark, as a relatively complex system, has a large number of parameters (over 400) that require proper configuration, such as the degree of parallelism. Improper parameter configuration can lead to poor job performance, significant resource waste, low fault tolerance, and even task failure. For big data engines running on heterogeneous computing platforms with uneven computing power, optimizing configuration parameters is even more crucial. For example, the requirements for setting parameters such as the number of heterogeneous cores and the degree of parallelism are more stringent, requiring a parameter processing method that can adapt to heterogeneous computing scenarios with uneven computing power.

[0096] Existing cloud computing platforms typically use a general-purpose core architecture, meaning they include multiple cores of equal computing power, resulting in significant waste of computing power. Existing Spark big data job parameter optimization solutions also suffer from the following issues: 1. They fail to consider the characteristics of the computing core environment, making them incapable of adapting to environments with super-general-purpose cores and difficult to find globally optimal parameters; 2. They fail to consider the use of big data engines and domain knowledge of super-general-purpose cores, nor do they integrate the two, leading to cold start issues, high optimization costs, and a lack of universality; 3. Parameter optimization requires extensive offline iterations of the optimized workload, resulting in high optimization costs.

[0097] FIG1 is an exemplary flowchart of a parameter processing method based on heterogeneous computing provided in an embodiment of the present application.

[0098] 110, obtaining the computing job and configuration parameter tuning settings input by the user.

[0099] Computing jobs run on a heterogeneous big data platform according to configuration parameters to perform processing tasks. The heterogeneous big data platform is used to manage the infrastructure, which includes at least two cores with different computing powers.

[0100] Exemplarily, a computing job may be a big data engine in each dimension or process of a big data job. Big data engines include, but are not limited to, Apache Hadoop, Apache Spark, Flink, Storm, and Apache Cassandra. A heterogeneous big data platform may be a software platform for managing any infrastructure with uneven computing power. Exemplarily, the infrastructure may be an infrastructure including super-normal cores or an infrastructure including a central processing unit (CPU) plus a graphics processing unit (GPU). The infrastructure may include a Yarn resource pool, which includes multiple cores with different computing powers. For example, the Yarn resource pool may include at least two of super-cores, normal cores, or heterogeneous nodes. Normal cores are cores with computing power lower than a preset threshold, super-cores are cores with computing power higher than a preset threshold, and heterogeneous nodes are mixed nodes of super-cores and normal cores.

[0101] Configuration parameter tuning settings are used to determine the tuning settings for configuration parameters related to computing jobs running on heterogeneous big data platforms. For example, these settings include user-entered settings for the frequency, optimization intensity, or tuning scenarios for computing jobs on the heterogeneous big data platform. They may also include a user-selected setting for whether to enable automatic tuning of configuration parameters.

[0102] 120, obtain the operation indicators related to the computing job.

[0103] Operational indicators include software indicators and hardware indicators. Software indicators include first and second software indicators, and hardware indicators include first and second hardware indicators. The first software indicator includes the software indicator collected before running the computing job, and the second software indicator includes the software indicator collected during the computing job. The first hardware indicator includes the hardware indicator collected before running the computing job, and the second hardware indicator includes the hardware indicator collected during the computing job.

[0104] Exemplarily, the first software indicator includes information on the usage of software resources provided by the infrastructure by running programs in the infrastructure before the computing job runs; the second software indicator includes information on the usage of software resources provided by the infrastructure by the computing job during the running of the computing job; the first hardware indicator includes information on the usage of hardware resources provided by the infrastructure by running programs in the infrastructure before the computing job runs; the second hardware indicator includes information on the usage of hardware resources provided by the infrastructure by the computing job during the running of the computing job.

[0105] Exemplarily, the software metric includes multiple candidate configuration parameter sets, wherein any one of the multiple candidate configuration parameter sets includes any one or more of configuration parameters preset before running a computing job and configuration parameters collected during the running of the computing job. Some or all of the configuration parameters in the multiple candidate configuration parameter sets belong to the first software metric or the second software metric.

[0106] 130. Determine a first configuration parameter set corresponding to the computing job.

[0107] A first configuration parameter set is determined according to the computing job, operating indicators and configuration parameter tuning settings. The first configuration parameter set includes some or all of the configuration parameters related to the computing job running on the heterogeneous big data platform.

[0108] Exemplarily, the parameter processing method provided in the present application can obtain computing jobs input by the user, configuration parameter tuning settings, and software indicators and hardware indicators related to the computing jobs. The software indicators may include software indicators collected before running the computing jobs and software indicators collected during running the computing jobs. The hardware indicators may include hardware indicators collected before running the computing jobs and hardware indicators collected during running the computing jobs. The method provided in the present application runs computing jobs on a heterogeneous big data platform according to the configuration parameter tuning settings input by the user, collects and analyzes running indicators, and continuously updates the optimal configuration parameter set corresponding to the computing job obtained in each round of tuning until the configuration parameter tuning settings are met. The optimal configuration parameter set obtained at this time is the first configuration parameter set.

[0109] The first configuration parameter set includes multiple configuration parameters, which may be from at least two of the multiple candidate configuration parameter sets, or from one of the candidate configuration parameter sets.

[0110] FIG2 is a schematic diagram of an application scenario of a parameter processing method based on heterogeneous computing provided in an embodiment of the present application.

[0111] The public cloud infrastructure development platform is a platform-as-a-service (PaaS) cloud service within the cloud platform. Leveraging the extensive infrastructure resources and software capabilities of the public cloud service provider, it provides users (also known as tenants, developers, etc.) with assistance in deploying and running big data jobs. As shown in Figure 2, the user interaction with the infrastructure development platform primarily involves logging in to the cloud platform through a client webpage, selecting and purchasing the infrastructure development platform's cloud services, and then performing tasks based on the functionality provided by the infrastructure development platform.

[0112] It should be understood that the basic development platform can use the parameter processing method of the embodiment of the present application to provide parameter tuning services for big data jobs on heterogeneous big data platforms. The user sends optimization request information to the basic development platform. The optimization request information can be the frequency of parameter tuning for computing jobs, optimization intensity, tuning scenarios, or whether to enable automatic tuning of configuration parameters, etc., which are input or selected by the user. The basic development platform can tune the configuration parameters of the computing jobs and return the running results of the computing jobs to the user. In some possible application scenarios, the tuned configuration parameters can also be returned to the user.

[0113] FIG3 is a schematic diagram of an application scenario of another parameter processing method based on heterogeneous computing provided in an embodiment of the present application.

[0114] Users can log in to cloud platform 210 through a client, and cloud platform 210 can manage infrastructure 220. Infrastructure 220 includes multiple cores with varying computing power. For example, infrastructure 220 may include a Yarn resource pool 230, which includes at least two types of nodes: supercores, standard cores, or heterogeneous nodes. Standard cores are cores with computing power below a preset threshold, supercores are cores with computing power above a preset threshold, and heterogeneous nodes are a mixture of supercores and standard cores.

[0115] Cloud platform 210 includes a parameter processing device 240 and a Spark server 250. Optionally, it may also include a domain knowledge base 260. It should be understood that domain knowledge base 260 may not be part of cloud platform 210; cloud platform 210 may obtain domain knowledge base 260 from elsewhere when needed. Parameter processing device 240 includes an acquisition module 241 and a processing module 243. Acquisition module 241 includes an input acquisition submodule 245 and an indicator collection and analysis submodule 246. Processing module 243 includes a performance modeling submodule 247 and a parameter search submodule 248.

[0116] Users log in to the cloud platform 210 through the client and submit computing jobs and corresponding configuration parameter tuning settings on the cloud platform 210. The input acquisition submodule 245 obtains the computing jobs and corresponding configuration parameter tuning settings input by the user and submits them to the Spark server 250. The Spark server 250 submits computing jobs, such as big data jobs (Spark jobs), to the cluster for preparation, and the Yarn resource pool 230 executes the big data jobs. The Spark server 250 performs tasks such as submitting computing jobs and configuration parameter tuning settings, reading parameters, starting the driver, loading the scheduler / environment preparation, and parsing the directed acyclic graph (DAG).

[0117] Spark jobs start running based on a given configuration, and the Yarn resource pool 230 can monitor the job's running status and performance indicators. The parameter processing device 240 can analyze the job's running status and performance indicators in conjunction with the domain knowledge base 260 to obtain the optimal parameter configuration set for the next job run.

[0118] The indicator collection and analysis submodule 246 is a resident process that can be deployed in the server's operating system to collect indicators from the application layer (before and during the execution of each job or stage of the Spark job), virtualization layer, and physical machine layer in real time, perform software and hardware collaborative indicator analysis, identify and tune sensitive queries and important parameters, and provide feedback on analysis results.

[0119] The performance modeling submodule 247, as a resident process, can be deployed in the server's operating system. It can match the results of the collection and analysis module to achieve pre-optimization and reduce the overall optimization complexity. At the same time, it can establish a white box performance model based on the domain knowledge base 260, strengthen Gaussian process modeling, and jointly support the parameter search submodule 248.

[0120] The parameter search submodule 248, a resident process, can be deployed in the server's operating system. It first uses a reinforced Gaussian model, integrating domain knowledge with the white-box model, as the prior model. Secondly, it uses the EI posterior acquisition function to pre-optimize its hyperparameters using a Monte Carlo Markov chain. The parameter search submodule 248 interacts with online Spark jobs in real time, applying the searched optimal parameters to the online job.

[0121] Domain knowledge base 260 may include big data engine domain knowledge and / or heterogeneous platform domain knowledge. Taking Spark as an example, big data engine domain knowledge may include memory management, resource management and scheduling, task scheduling, shuffle, DAG, dependencies, and RDD. Heterogeneous platform domain knowledge may include hypercore and normal core utilization, performance parameters, virtualization layer characteristics, physical machine layer scheduling, and cloud instance design.

[0122] FIG4 is an exemplary flow chart of another parameter processing method provided in an embodiment of the present application. The method flow shown in FIG4 corresponds to FIG3.

[0123] 310, apply for resources according to configuration.

[0124] The user logs in to the cloud platform 210 through the client and submits the computing job and the corresponding configuration parameter tuning settings on the cloud platform 210. The Spark server 250 initializes the environment and applies for resources based on the current computing job and the corresponding configuration parameter tuning settings, and starts executing the Spark job.

[0125] 320, real-time monitoring of operating data.

[0126] Yarn, a distributed resource manager, provides resource management and task scheduling for Spark jobs. When a Spark application runs in Yarn mode, it submits a job request to Yarn, requesting a certain amount of computing resources. Yarn dynamically allocates computing resources based on the job's resource requirements and the cluster's available resources, and monitors Spark's performance data in real time.

[0127] 330, indicator collection and analysis.

[0128] This application collects and analyzes key indicators through the indicator collection and analysis submodule 246. Unlike existing parameter tuning technologies, this application collects both software and hardware indicators to achieve collaborative performance analysis of software and hardware. In one possible application scenario, software and hardware indicators can be directly extracted from the operation log.

[0129] Table 1 shows examples of indicators collected and analyzed by this application at different collection stages.

[0130] Table 1

[0131] It should be understood that candidate configuration parameters can be collected both before and during a job run for use by the parameter search submodule 248 in determining optimal configuration parameters. Configuration parameters collected before a job run are included in the software metrics collected before the job run and are preset configuration parameters. Configuration parameters collected during a job run are included in the software metrics collected during the job run and are configuration parameters collected from the system using a collection tool, such as configuration parameters collected from run logs.

[0132] In some possible application scenarios, configuration parameters may be collected only before a job is run, or only during the job is run, or both before and during the job is run.

[0133] It should be understood that, in addition to collecting configuration parameters, the collected metrics also require understanding the usage of software and hardware resources by running programs in infrastructure 220 before the computing job runs, as well as the usage of software and hardware resources in infrastructure 220 by the computing job during the computing job runs. The remaining metrics are optional. In some possible application scenarios, corresponding metrics can be collected based on specific analysis methods. The collected software and hardware metrics shown in Table 1 should not be construed as limiting this application.

[0134] After collecting software indicators and hardware indicators, any one or more of important parameter analysis and sensitive query analysis can be performed on these software indicators and hardware indicators. Important parameter analysis includes screening out configuration parameters whose importance is greater than a preset threshold among candidate configuration parameters, and sensitive query analysis includes filtering out scenarios that are insensitive to parameter tuning. Exemplarily, important parameters can be analyzed using methods such as kernel principal component analysis (KPCA), and sensitive query analysis can be performed using coefficient of variation calculation. Exemplarily, in a possible implementation scenario, 100 collected job configuration parameters can be screened to 20 through important parameter analysis, and 100 query jobs to be optimized can be simplified to 10 through sensitive query analysis.

[0135] In some possible application scenarios, you can skip the important parameter analysis and sensitive query analysis and proceed directly to the next step of parameter tuning.

[0136] 340, Performance Modeling.

[0137] In some possible application scenarios, the software indicators and hardware indicators output by the indicator collection and analysis submodule 246 can be matched with the domain knowledge base 260. The domain knowledge base 260 contains performance diagnosis and some preset optimization suggestion feedback. For example, a performance diagnosis and optimization suggestion feedback can be matched according to each software indicator and hardware indicator, and then the job is pre-optimized based on the matching result to improve the efficiency of parameter processing. For example, the domain knowledge base 260 contains Spark domain knowledge and super-general core domain knowledge. The Spark domain knowledge matched by a certain software indicator is too much memory usage, and it is recommended to appropriately reduce memory usage. In this way, the job can be pre-optimized before modeling to improve the overall parameter processing efficiency.

[0138] The embodiment of the present application utilizes a Gaussian prior model to determine the optimal configuration parameter set among multiple configuration parameter sets. The present application can establish a white-box performance model based on the analysis of Spark principles in domain knowledge, and strengthen the Gaussian prior model in the parameter search submodule 248. A white-box model refers to a model that can explain and understand the internal operating mechanism of the model, and its internal structure and parameters can be directly observed and understood. The white-box model of the present application is based on Spark principles. By analyzing Spark's internal implementation and operating mechanism, the relationship between input and output is modeled to predict the output result.

[0139] It should be understood that the performance modeling in step 340 is for pre-optimization and strengthening of the Gaussian prior model in the parameter search submodule 248. This step may be omitted and step 350 may be directly executed.

[0140] 350, parameter search.

[0141] By using Gaussian process modeling, we can obtain the posterior probability distribution of the objective function, and through methods such as maximum a posteriori estimation (MAP) or expectation maximization (EM), we can obtain the optimal solution of the objective function, that is, the optimal parameter configuration.

[0142] However, in practical applications, the objective function usually has multiple input variables and noise, so its optimization process is relatively complex and time-consuming. To solve this problem, the Bayesian optimization method can be used to continuously use Gaussian process modeling and Bayesian inference to find the global optimal solution of the objective function. Specifically, the Bayesian optimization method treats the objective function as a black box and gradually narrows the search space to find the optimal solution by continuously sampling and evaluating. In each sampling, the Bayesian optimization method uses Gaussian process modeling to calculate the posterior probability distribution of the objective function and selects the point that is most likely to be the global optimal solution for sampling.

[0143] During the Bayesian optimization process, the expected improvement (EI) posterior acquisition function and Markov Chain Monte Carlo (MCMC)-based methods can be used to optimize the EI itself to achieve a better balance between the exploitation of known optimal points and the exploration of new areas. The EI posterior acquisition function is a method used to guide the selection of the next sampling point, which can help find better points near the known optimal point. However, relying solely on the EI posterior acquisition function may lead to an overly concentrated search near the known optimal point, failing to explore new areas and potentially missing the global optimal solution. To address this issue, a Monte Carlo Markov chain-based method can be used to explore new areas to find the global optimal solution.

[0144] The present invention can also improve the efficiency of parameter search by sparsifying the covariance matrix. Covariance matrix sparsification is a method used in Gaussian process modeling that can reduce the computational complexity and storage space of the covariance matrix, thereby improving the computational efficiency of the Gaussian process model.

[0145] It should be understood that the Gaussian process modeling and EI posterior acquisition function used in the embodiment of the present application are merely examples, and other prior models and other acquisition functions may also be used. This example should not be understood as a limitation to the present application.

[0146] 360, determine the optimal configuration parameter set.

[0147] The optimal configuration parameter set searched by the parameter search submodule 248 is inserted into the online job of the Spark server 250 .

[0148] FIG5 is an exemplary flow chart of another parameter processing method provided in an embodiment of the present application. The method flow shown in FIG5 corresponds to FIG3.

[0149] 410, the user submits a big data job and the corresponding configuration parameter tuning settings.

[0150] Exemplarily, the configuration parameter tuning settings include the frequency, optimization intensity, and tuning scenarios of parameter tuning for big data jobs on a heterogeneous big data platform that can be input by the user, or whether to enable automatic tuning of configuration parameters.

[0151] 420, the parameter processing device starts and monitors the operation status in real time.

[0152] The parameter processing device 240 provided in this application is deployed in a big data job scenario on the cloud. After the user submits a big data job and before the job is executed, the parameter processing device 240 provided in this application is started and begins to monitor the current job in real time.

[0153] 430, determine whether to enable parameter tuning.

[0154] If parameter tuning is enabled, execute step 440, otherwise execute step 431. It should be understood that during the process of big data job parameter tuning, the user can choose to disable the automatic parameter tuning option at any time to execute the job with the optimal configuration parameters in the current database.

[0155] 431, Start the job according to the current configuration parameters.

[0156] 440, the inserted parameters are the optimal parameters, and the job is started.

[0157] Initialize the environment according to the current optimal configuration parameters and start the job, reusing Spark's own parameter management interface. Prioritize using the optimal configuration parameters in the collection and analysis database 465 to execute the job. If the optimal configuration parameters in the current collection and analysis database 465 are empty, the current optimal configuration parameters based on the domain knowledge base 260 are used to solve the cold start problem.

[0158] 450, the indicator collection and analysis submodule performs operations.

[0159] For the Spark big data job that starts the parameter processing device provided by this application, during the period from job submission to job execution completion, the indicator collection and analysis submodule 246 collects and analyzes indicators before or during the Spark job stage or job execution.

[0160] For example, in the Spark job dimension, the indicator collection and analysis submodule 246 collects corresponding software indicators by interacting with Spark's metrics interface. The metrics interface can monitor and measure the performance and resource usage of Spark applications; in the cluster dimension, the indicator collection and analysis submodule 246 collects hardware indicators, such as PMU and core utilization information to support Roofline and TMA analysis.

[0161] The indicator collection and analysis submodule 246 can also perform important parameter analysis and sensitive query analysis, and store the collected indicators and analysis results in the collection and analysis database 465.

[0162] The functions of the indicator collection and analysis submodule 246 can be found in the description of step 330 in FIG4 , and will not be repeated in this application.

[0163] 460, the performance modeling submodule performs operations.

[0164] The software indicators and hardware indicators output by the indicator collection and analysis submodule 246 can be matched with the domain knowledge base 260. The domain knowledge base 260 contains performance diagnosis and some preset optimization suggestion feedback. For example, a performance diagnosis and optimization suggestion feedback can be matched according to each software indicator and hardware indicator, and then the device status can be pre-optimized based on the matching results to improve the efficiency of parameter processing.

[0165] The embodiment of the present application utilizes a Gaussian prior model to determine the optimal candidate configuration parameter set among multiple candidate configuration parameter sets. The present application can establish a white box performance model based on Spark principle analysis to strengthen the Gaussian prior model in the parameter search submodule 248.

[0166] The function of the performance modeling submodule 247 can be found in the description of step 340 in FIG4 , and will not be repeated in this application.

[0167] 470, the parameter search submodule performs operations.

[0168] By using Gaussian process modeling, we can obtain the posterior probability distribution of the objective function, and through methods such as maximum a posteriori estimation (MAP) or expectation maximization (EM), we can obtain the optimal solution of the objective function, that is, the optimal parameter configuration.

[0169] The function of the parameter search submodule 248 can be found in the description of step 350 in FIG4 , and will not be repeated in this application.

[0170] 480, output the current optimal configuration parameters.

[0171] After the optimal parameter search reaches the convergence condition, the current optimal configuration parameters are output and then stored in the acquisition and analysis database 465.

[0172] 490, the parameter processing device waits for the execution of the next job.

[0173] It should be understood that the technical solution of the present application can be applied and deployed in public clouds, hybrid clouds, and private clouds. The technical solution of the present application can be applied not only to the framework of Spark big data, but also to big data frameworks such as Hadoop, Flink, Storm, and Cassandra. The above examples for Spark should not be understood as limitations on the present application. At the same time, the parameter processing device provided by the present application can perform parameter tuning on other frameworks of a single node or the entire cluster, or even the operating system (operating system, OS) in the cluster dimension. As long as there is a parameter setting interface and there is a software with parameter tuning requirements, the technical solution provided by the present application can be used. Because the technical solution of the present application is actually parameter tuning performed on an unbalanced computing power infrastructure. Therefore, the technical solution of the present application can be applied not only to the infrastructure including the super-general core in the above embodiment, but also to other infrastructures with unbalanced computing power, and even the infrastructure of the central processing unit (CPU) plus the graphics processing unit (GPU).

[0174] The parameter processing method provided by the embodiment of the present application is described in detail above in conjunction with Figures 1 to 5. The embodiment of the device of the present application will be described in detail below in conjunction with Figures 6 to 9. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment. Therefore, for parts not described in detail, reference can be made to the above method embodiment.

[0175] Figure 6 is a schematic block diagram of a parameter processing device provided in an embodiment of the present application. It should be understood that the parameter processing device shown in Figure 6 can be regarded as an example of the parameter processing device 240, and the division of specific units should not be understood as a limitation to the present application.

[0176] The parameter processing device 240 can be implemented by software, hardware or a combination of both. The parameter processing device 240 provided in the embodiment of the present application can implement the method flow shown in Figures 1 to 5 of the embodiment of the present application. The parameter processing device 240 is applied to a heterogeneous big data platform, which is used to manage infrastructure. The infrastructure includes at least two cores with different computing powers. The parameter processing device 240 includes: an acquisition module 241 and a processing module 243. Among them, the acquisition module 241 is used to obtain the computing job and configuration parameter tuning settings input by the user. The computing job is run by the heterogeneous big data platform. The configuration parameter tuning settings are used to determine the tuning settings of the relevant configuration parameters of the heterogeneous big data platform running the computing job; the acquisition module 241 is also used to obtain the operating indicators related to the computing job. The operating indicators include software indicators and hardware indicators; the processing module 243 is used to determine the first configuration parameter set based on the computing job, the operating indicators and the configuration parameter tuning settings. The first configuration parameter set includes some or all parameters in the relevant configuration parameters of the heterogeneous big data platform running the computing job.

[0177] Optionally, the software indicators include a first software indicator and a second software indicator, and the hardware indicators include a first hardware indicator and a second hardware indicator. The first software indicator includes the software indicator collected before running the computing job, and the second software indicator includes the software indicator collected during the running of the computing job. The first hardware indicator includes the hardware indicator collected before running the computing job, and the second hardware indicator includes the hardware indicator collected during the running of the computing job.

[0178] Optionally, the first software indicator includes information on the usage of software resources provided by the infrastructure by the running programs in the infrastructure before the computing job runs, and the second software indicator includes information on the usage of software resources provided by the infrastructure by the computing job during the running of the computing job. The first hardware indicator includes information on the usage of hardware resources provided by the infrastructure by the running programs in the infrastructure before the computing job runs, and the second hardware indicator includes information on the usage of hardware resources provided by the infrastructure by the computing job during the running of the computing job.

[0179] Optionally, the acquisition module 241 is also used to acquire a second configuration parameter set, a first software indicator and a first hardware indicator of the computing job, where the second configuration parameter set is a historical configuration parameter set or a preset configuration parameter set; the processing module 243 is also used to run the computing job according to the second configuration parameter set to determine the second software indicator and the second hardware indicator of the computing job; the acquisition module 241 is also used to acquire a priori model and multiple candidate configuration parameter sets, where some or all of the configuration parameters in the multiple candidate configuration parameter sets belong to the first software indicator or the second software indicator; the processing module 243 is also used to determine the first configuration parameter set based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, multiple candidate configuration parameter sets, the priori model and the configuration parameter tuning settings, and the priori model is used to predict the first configuration parameter set.

[0180] Optionally, the acquisition module 241 is also used to obtain a white box model, which is used to enhance the predictive performance of the prior model. The white box model is constructed based on domain knowledge, which includes performance diagnosis and optimization suggestions corresponding to operating indicators; the processing module 243 is also used to enhance the prior model based on the white box model.

[0181] Optionally, the acquisition module 241 is also used to obtain an acquisition function, which is used to enhance the predictive performance of the prior model; the processing module 243 is also used to determine the first configuration parameter set based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, multiple candidate configuration parameter sets, the prior model, the acquisition function and the configuration parameter tuning settings.

[0182] Optionally, the processing module 243 is further used to match domain knowledge according to the operating indicators, where the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the operating indicators; the processing module 243 is further used to optimize computing jobs according to the optimization suggestions in the domain knowledge.

[0183] Optionally, when the computing job is run for the first N times, the second configuration parameter set is a configuration parameter set preset based on domain knowledge, where the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the running indicators, and N is greater than or equal to 1.

[0184] Optionally, the prior model is a Gaussian prior model.

[0185] Optionally, the domain knowledge includes any one or more of performance diagnosis and optimization suggestions related to computing jobs and performance diagnosis and optimization suggestions related to heterogeneous big data platforms.

[0186] Optionally, the performance diagnosis and optimization suggestions related to the computing job include performance diagnosis and optimization suggestions for any one or more of the following aspects related to the computing job: memory management, resource management and scheduling, task scheduling, reorganization operations, directed acyclic graphs, dependencies, and resilient distributed datasets.

[0187] Optionally, the performance diagnosis and optimization suggestions related to the heterogeneous big data platform include performance diagnosis and optimization suggestions for any one or more of the following aspects related to the heterogeneous big data platform: core utilization information, performance parameters, virtualization layer characteristics, physical machine layer scheduling, and cloud instance design.

[0188] Optionally, the processing module 243 is also used to perform at least one of important parameter analysis and sensitive query analysis. The important parameter analysis includes screening out candidate configuration parameters whose importance is greater than a first preset threshold in a set of multiple candidate configuration parameters. The sensitive query analysis includes filtering out scenarios whose sensitivity to parameter tuning is less than a second preset threshold.

[0189] Optionally, any one of the multiple candidate configuration parameter sets includes any one or more of configuration parameters preset before running the computing job and configuration parameters collected during the running of the computing job.

[0190] Optionally, the first software indicator includes any one or more of the execution plan of the computing job, the directed acyclic graph of the computing job, the data volume of the computing job, the operator of the computing job, and the parallelism of the computing job; and the second software indicator includes any one or more of the parallelism of the computing job, the amount of reorganization operation data, and the scale of data overflow.

[0191] Optionally, the second software indicator and the second hardware indicator are obtained by extracting from a running log of the computing job.

[0192] Optionally, the second hardware indicator includes any one or more of a performance monitoring unit indicator and a multi-level cache memory cache bandwidth.

[0193] It should be understood that the parameter processing device 240 here can be embodied in the form of a functional module. The term "module" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0194] For example, a "module" can be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functions. For example, the implementation of acquisition module 241 will be described below, using acquisition module 241 as an example. Similarly, the implementation of other modules, such as processing module 243, can refer to the implementation of acquisition module 241.

[0195] As an example of a software functional unit, the acquisition module 241 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the acquisition module 241 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0196] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0197] As an example of a hardware functional unit, the acquisition module 241 may include at least one computing device, such as a server. Alternatively, the acquisition module 241 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0198] The multiple computing devices included in acquisition module 241 can be distributed in the same region or in different regions. The multiple computing devices included in acquisition module 241 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in acquisition module 241 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0199] Therefore, the modules of each example described in the embodiments of this application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0200] It should be noted that the parameter processing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when executing the parameter processing method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the acquisition module 241 can be used to execute any step in the parameter processing method, and the processing module 243 can be used to execute any step in the parameter processing method. The steps that the acquisition module 241 and the processing module 243 are responsible for implementing can be specified as needed. The full functions of the parameter processing device are realized by having the acquisition module 241 and the processing module 243 respectively implement different steps in the parameter processing method.

[0201] In addition, the parameter processing device and the parameter processing method provided in the above embodiment belong to the same concept, and their specific implementation process is detailed in the method embodiment above, which will not be repeated here.

[0202] The parameter processing method provided in the embodiment of the present application can be performed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory and a memory control unit, and the function and structure of the hardware are subsequently described in detail. The operating system is any one or more computer operating systems that implement business processing through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. Furthermore, optionally, the computer system is a handheld device such as a smart phone, or a terminal device such as a personal computer, and the present application is not particularly limited, as long as the method provided by the embodiment of the present application can be used. The execution subject of the parameter processing method provided in the embodiment of the present application can be a computing device, or a functional module in a computing device that can call a program and execute a program.

[0203] 7 , a computing device provided in an embodiment of the present application is described in detail below.

[0204] Figure 7 is a schematic diagram of the architecture of a computing device provided in an embodiment of the present application. The computing device 1700 can be a server, a computer, or other device with computing capabilities. The computing device 1700 shown in Figure 7 includes: a processor 1720, a memory 1730, a communication interface 1740, and a bus 1710. The processor 1720, the memory 1730, and the communication interface 1740 communicate with each other via the bus 1710. The computing device 1700 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1700.

[0205] Bus 1710 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG7 shows a single bus line, but this does not imply a single bus or type of bus. Bus 1710 may include a path for transmitting information between various components of computing device 1700 (e.g., memory 1730, processor 1720, and communication interface 1740).

[0206] The processor 1720 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0207] The memory 1730 may include a volatile memory, such as a random access memory (RAM). The processor 1720 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0208] Memory 1730 stores executable program code, which processor 1720 executes to implement the functions of acquisition module 241 and processing module 243, respectively, thereby implementing the parameter processing method described in the above embodiment. That is, memory 1730 stores instructions for executing the parameter processing method described in the above embodiment.

[0209] Alternatively, the memory 1730 stores executable program code, and the processor 1720 executes the executable program code to implement the functions of the acquisition module 241 and the processing module 243, respectively, thereby implementing the parameter processing method described in the above embodiment. In other words, the memory 1730 stores instructions for executing the parameter processing method described in the above embodiment.

[0210] The communication interface 1740 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1700 and other devices or a communication network.

[0211] The structure of the computing device 1700 listed above is merely an example, and the present application is not limited thereto. The computing device 1700 of the embodiment of the present application includes various hardware in the computer system of the prior art. For example, the computing device 1700 also includes other memories in addition to the memory 1730, such as disk storage, etc. Those skilled in the art will understand that the computing device 1700 may also include other devices necessary to achieve normal operation. At the same time, according to specific needs, those skilled in the art will understand that the computing device 1700 may also include hardware devices that implement other additional functions. In addition, those skilled in the art will understand that the computing device 1700 may also include only the devices necessary to implement the embodiment of the present application, and does not necessarily include all the devices shown in Figure 7.

[0212] FIG8 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0213] The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0214] As shown in Figure 8, the computing device cluster includes at least one computing device 1700. The memory 1730 in one or more computing devices 1700 in the computing device cluster may store the same instructions for executing the parameter processing method described in the above embodiment.

[0215] In some possible implementations, the memory 1730 of one or more computing devices 1700 in the computing device cluster may also store some instructions for executing the parameter processing method described in the above embodiment. In other words, the combination of one or more computing devices 1700 can jointly execute the instructions for executing the parameter processing method described in the above embodiment.

[0216] It should be noted that the memory 1730 in different computing devices 1700 in the computing device cluster may store different instructions, each for executing part of the functions of the computing device 1700. In other words, the instructions stored in the memory 1730 in different computing devices 1700 may implement the functions of one or more modules among the acquisition module 241 and the processing module 243.

[0217] Alternatively, the memories 1730 in different computing devices 1700 in the computing device cluster may store different instructions, each for executing a portion of the functions of the parameter processing apparatus 240. In other words, the instructions stored in the memories 1730 in different computing devices 1700 may implement the functions of one or more of the acquisition module 241 and the processing module 243.

[0218] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG. 9 shows a possible implementation. FIG. 9 is a schematic diagram of a connection between computing devices 1700A and 1700B provided in an embodiment of the present application via a network. The two computing devices 1700A and 1700B are connected via a network. Specifically, the connection to the network is made via a communication interface in each computing device. In this type of possible implementation, the memory 1730 in the computing device 1700A stores instructions for executing the functions of the acquisition module 241. At the same time, the memory 1730 in the computing device 1700B stores instructions for executing the functions of the processing module 243.

[0219] It should be understood that the functionality of the computing device 1700A shown in FIG9 may also be implemented by multiple computing devices 1700. Similarly, the functionality of the computing device 1700B may also be implemented by multiple computing devices 1700.

[0220] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the computing device executes the parameter processing method provided above, or implements the functions of the parameter processing apparatus 240 provided above.

[0221] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device cluster or stored in any available medium. When executed by a computing device cluster, the computer program product causes the computing device cluster to perform the parameter processing method provided above, or causes the computing device cluster to implement the functions of the parameter processing apparatus 240 provided above.

[0222] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the parameter processing method provided above.

[0223] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed by a computing device cluster, cause the computing device cluster to execute the parameter processing method provided above.

[0224] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0225] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0226] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses, and computing device clusters can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0227] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0228] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0229] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0230] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0231] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A parameter processing method based on heterogeneous computing, characterized in that: Applied to a heterogeneous big data platform, the heterogeneous big data platform is used to manage infrastructure, the infrastructure includes at least two cores with unequal computing power, the method includes: Obtaining a computing job and configuration parameter tuning settings input by a user, wherein the computing job is run by the heterogeneous big data platform, and the configuration parameter tuning settings are used to determine tuning settings of relevant configuration parameters for the heterogeneous big data platform to run the computing job; Acquire operation indicators related to the computing job, wherein the operation indicators include software indicators and hardware indicators; A first configuration parameter set is determined according to the computing job, the operating index and the configuration parameter tuning settings, wherein the first configuration parameter set includes part or all of the configuration parameters related to the computing job running on the heterogeneous big data platform.

2. The method according to claim 1, characterized in that The software indicators include a first software indicator and a second software indicator, and the hardware indicators include a first hardware indicator and a second hardware indicator. The first software indicator includes a software indicator collected before running the computing job, and the second software indicator includes a software indicator collected during the running of the computing job. The first hardware indicator includes a hardware indicator collected before running the computing job, and the second hardware indicator includes a hardware indicator collected during the running of the computing job.

3. The method according to claim 2, characterized in that The first software indicator includes usage information of software resources provided by the infrastructure by the running programs in the infrastructure before the computing job runs; the second software indicator includes usage information of software resources provided by the infrastructure by the computing job during the running of the computing job; the first hardware indicator includes usage information of hardware resources provided by the infrastructure by the running programs in the infrastructure before the computing job runs; the second hardware indicator includes usage information of hardware resources provided by the infrastructure by the computing job during the running of the computing job.

4. The method according to claim 2 or 3, characterized in that The determining a first configuration parameter set according to the computing job, the operation index and the configuration parameter tuning setting includes: Acquire a second configuration parameter set of the computing job, the first software indicator, and the first hardware indicator, where the second configuration parameter set is a historical configuration parameter set or a preset configuration parameter set; Run the computing job according to the second configuration parameter set to determine the second software indicator and the second hardware indicator of the computing job; Acquire a priori models and multiple candidate configuration parameter sets, where some or all of the configuration parameters in the multiple candidate configuration parameter sets belong to the first software indicator or the second software indicator; The first configuration parameter set is determined based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, the multiple candidate configuration parameter sets, the priori model and the configuration parameter tuning settings, and the priori model is used to predict the first configuration parameter set.

5. The method according to claim 4, characterized in that Before determining the first configuration parameter set, the method further includes: Obtaining a white box model, the white box model is used to enhance the predictive performance of the prior model, the white box model is constructed based on domain knowledge, the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the operating indicators; The prior model is enhanced according to the white-box model.

6. The method according to claim 4 or 5, characterized in that The determining the first configuration parameter set according to the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, the multiple candidate configuration parameter sets, the priori model and the configuration parameter tuning setting includes: Acquiring an acquisition function, wherein the acquisition function is used to enhance the prediction performance of the prior model; The first configuration parameter set is determined according to the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, the multiple candidate configuration parameter sets, the priori model, the acquisition function and the configuration parameter tuning settings.

7. The method according to any one of claims 4 to 6, characterized in that Before determining the first configuration parameter set according to the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, the multiple candidate configuration parameter sets, the priori model, and the configuration parameter tuning setting, the method further includes: Matching domain knowledge according to the operation indicators, wherein the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the operation indicators; The computing operation is optimized according to the optimization suggestions in the domain knowledge.

8. The method according to any one of claims 4 to 7, characterized in that When the computing job is run for the first N times, the second configuration parameter set is a configuration parameter set preset according to domain knowledge, and the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the running indicators, and N is greater than or equal to 1.

9. The method according to any one of claims 4 to 8, characterized in that The prior model is a Gaussian prior model.

10. The method according to any one of claims 5, 7 or 8, characterized in that: The domain knowledge includes any one or more of performance diagnosis and optimization suggestions related to the computing job and performance diagnosis and optimization suggestions related to the heterogeneous big data platform.

11. The method according to claim 10, characterized in that The performance diagnosis and optimization suggestions related to the computing job include performance diagnosis and optimization suggestions for any one or more of the following aspects related to the computing job: memory management, resource management and scheduling, task scheduling, reorganization operations, directed acyclic graphs, dependencies and resilient distributed data sets.

12. The method according to claim 10 or 11, characterized in that The performance diagnosis and optimization suggestions related to the heterogeneous big data platform include performance diagnosis and optimization suggestions in any one or more of the following aspects related to the heterogeneous big data platform: core utilization information, performance parameters, virtualization layer characteristics, physical machine layer scheduling, and cloud instance design.

13. The method according to any one of claims 4 to 12, characterized in that: Before determining the first configuration parameter set according to the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, the multiple candidate configuration parameter sets, the priori model, and the configuration parameter tuning setting, the method further includes: Perform at least one of important parameter analysis and sensitive query analysis, wherein the important parameter analysis includes screening out candidate configuration parameters whose importance is greater than a first preset threshold in the set of multiple candidate configuration parameters, and the sensitive query analysis includes filtering out scenarios whose sensitivity to parameter tuning is less than a second preset threshold.

14. The method according to any one of claims 4 to 13, characterized in that: Any one of the multiple candidate configuration parameter sets includes any one or more of configuration parameters preset before running the computing job and configuration parameters collected during the running of the computing job.

15. The method according to any one of claims 2 to 14, characterized in that The first software indicator includes any one or more of the execution plan of the computing job, the directed acyclic graph of the computing job, the data volume of the computing job, the operator of the computing job, and the parallelism of the computing job; the second software indicator includes any one or more of the parallelism of the computing job, the amount of reorganization operation data, and the scale of data overflow.

16. The method according to any one of claims 2 to 15, characterized in that The second software indicator and the second hardware indicator are obtained by extracting from the running log of the computing job.

17. The method according to any one of claims 2 to 16, characterized in that The second hardware indicator includes any one or more of a performance monitoring unit indicator and a multi-level high-speed cache memory Cache bandwidth.

18. A parameter processing device, characterized in that: The parameter processing device is used to manage an infrastructure, wherein the infrastructure includes at least two cores with different computing powers, and the device includes: An acquisition module, for a computing job and a configuration parameter tuning setting input by a user, wherein the computing job is run by the heterogeneous big data platform, and the configuration parameter tuning setting is used to determine a tuning setting of relevant configuration parameters for the heterogeneous big data platform to run the computing job; The acquisition module is further used to acquire operation indicators related to the computing job, wherein the operation indicators include software indicators and hardware indicators; A processing module is used to determine a first configuration parameter set based on the computing job, the operating indicators and the configuration parameter tuning settings, wherein the first configuration parameter set includes part or all of the relevant configuration parameters of the heterogeneous big data platform running the computing job.

19. The device according to claim 18, characterized in that The software indicators include a first software indicator and a second software indicator, and the hardware indicators include a first hardware indicator and a second hardware indicator. The first software indicator includes a software indicator collected before running the computing job, and the second software indicator includes a software indicator collected during the running of the computing job. The first hardware indicator includes a hardware indicator collected before running the computing job, and the second hardware indicator includes a hardware indicator collected during the running of the computing job.

20. The device according to claim 19, characterized in that The first software indicator includes usage information of software resources provided by the infrastructure by the running programs in the infrastructure before the computing job runs; the second software indicator includes usage information of software resources provided by the infrastructure by the computing job during the running of the computing job; the first hardware indicator includes usage information of hardware resources provided by the infrastructure by the running programs in the infrastructure before the computing job runs; the second hardware indicator includes usage information of hardware resources provided by the infrastructure by the computing job during the running of the computing job.

21. The device according to claim 19 or 20, characterized in that The acquisition module is further used to acquire a second configuration parameter set of the computing job, the first software indicator, and the first hardware indicator, where the second configuration parameter set is a historical configuration parameter set or a preset configuration parameter set; The processing module is further configured to run the computing job according to the second configuration parameter set, and determine the first The second software indicator and the second hardware indicator; The acquisition module is further used to acquire a priori models and multiple candidate configuration parameter sets, where some or all of the configuration parameters in the multiple candidate configuration parameter sets belong to the first software indicator or the second software indicator; The processing module is further used to determine the first configuration parameter set based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, the multiple candidate configuration parameter sets, the prior model and the configuration parameter tuning settings, and the prior model is used to predict the first configuration parameter set.

22. The device according to claim 21, characterized in that The acquisition module is further used to acquire a white box model, the white box model is used to enhance the prediction performance of the prior model, the white box model is constructed according to domain knowledge, and the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the operation index; The processing module is further used to strengthen the prior model according to the white box model.

23. The device according to claim 21 or 22, characterized in that The acquisition module is further used to acquire an acquisition function, and the acquisition function is used to enhance the prediction performance of the prior model; The processing module is further used to determine the first configuration parameter set based on the first software indicator, the first hardware indicator, the second software indicator, the second hardware indicator, the multiple candidate configuration parameter sets, the prior model, the acquisition function and the configuration parameter tuning settings.

24. The device according to any one of claims 21 to 23, characterized in that The processing module is further used for: Matching domain knowledge according to the operation indicators, wherein the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the operation indicators; The computing operation is optimized according to the optimization suggestions in the domain knowledge.

25. The device according to any one of claims 21 to 24, characterized in that When the computing job is run for the first N times, the second configuration parameter set is a configuration parameter set preset according to domain knowledge, and the domain knowledge includes performance diagnosis and optimization suggestions corresponding to the running indicators, and N is greater than or equal to 1.

26. The device according to any one of claims 21 to 25, characterized in that The prior model is a Gaussian prior model.

27. The device according to any one of claims 22, 24 or 25, characterized in that The domain knowledge includes any one or more of performance diagnosis and optimization suggestions related to the computing job and performance diagnosis and optimization suggestions related to the heterogeneous big data platform.

28. The device according to claim 27, characterized in that The performance diagnosis and optimization suggestions related to the computing job include performance diagnosis and optimization suggestions for any one or more of the following aspects related to the computing job: memory management, resource management and scheduling, task scheduling, reorganization operations, directed acyclic graphs, dependencies and resilient distributed data sets.

29. The device according to claim 27 or 28, characterized in that The performance diagnosis and optimization suggestions related to the heterogeneous big data platform include performance diagnosis and optimization suggestions in any one or more of the following aspects related to the heterogeneous big data platform: core utilization information, performance parameters, virtualization layer characteristics, physical machine layer scheduling, and cloud instance design.

30. The device according to any one of claims 21 to 29, characterized in that The processing module is also used to perform at least one of important parameter analysis and sensitive query analysis. The important parameter analysis includes screening out candidate configuration parameters whose importance is greater than a first preset threshold in the set of multiple candidate configuration parameters, and the sensitive query analysis includes filtering out scenarios whose sensitivity to parameter tuning is less than a second preset threshold.

31. The device according to any one of claims 21 to 30, characterized in that Any one of the multiple candidate configuration parameter sets includes any one or more of configuration parameters preset before running the computing job and configuration parameters collected during the running of the computing job.

32. The device according to any one of claims 19 to 31, characterized in that The first software indicator includes any one or more of the execution plan of the computing job, the directed acyclic graph of the computing job, the data volume of the computing job, the operator of the computing job, and the parallelism of the computing job; the second software indicator includes any one or more of the parallelism of the computing job, the amount of reorganization operation data, and the scale of data overflow.

33. The device according to any one of claims 19 to 32, characterized in that The second software indicator and the second hardware indicator are obtained by extracting from the running log of the computing job.

34. The device according to any one of claims 19 to 33, characterized in that The second hardware indicator includes any one or more of a performance monitoring unit indicator and a multi-level high-speed cache memory Cache bandwidth.

35. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 17.

36. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 17.

37. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method as claimed in any one of claims 1 to 17.

Images