Software system performance adjusting and optimizing method and device based on automatic parameter configuration optimization

Abstract

The invention discloses a software system performance adjusting and optimizing method and device based on automatic parameter configuration optimization. The method comprises the steps that a to-be-adjusted and optimized configuration set oriented to a to-be-adjusted and optimized system is collected; sampling a plurality of candidate configuration instances from the configuration set to be optimized; utilizing the performance samples under different candidate configuration examples to configure a performance prediction model of the system to be tuned and optimized, and obtaining recommended configuration of the current configuration example based on the performance prediction model; after the to-be-adjusted and optimized system is reset according to the recommended configuration, the current working load of the to-be-adjusted and optimized system is recognized, and the performance value of the current working load is evaluated; adjusting the proxy model, the sampling function and the optimization strategy based on the performance value of the current working load, and returning to sample a plurality of candidate configuration instances from the configuration set to be adjusted and optimized; and outputting the corresponding recommended configuration until the first set stop condition is met. The problem that the performance of a software system is reduced can be solved, and the software system obtains the optimal performance.

Description

Technical Field

[0001] The present invention belongs to the field of system software performance optimization, and in particular to a software system performance tuning method and device based on automatic parameter configuration optimization. Background Art

[0002] With the rapid development of the Internet and computer science and technology, technologies such as artificial intelligence, cloud computing, and big data have become major development trends. Large enterprises are increasingly using large-scale software systems, such as big data analysis components, databases, and Web servers, to solve problems in actual business scenarios. However, due to the complexity of software systems, parameter configuration has become a challenge. The application of default parameter configuration has led to performance degradation and affected service quality.

[0003] To solve the problem of software system performance degradation, we can consider both hardware and software aspects. Some companies solve performance problems by increasing hardware resources, but this will increase cost pressure. For small and medium-sized companies, it is necessary to optimize performance from the perspective of the software system itself, that is, to improve performance by reasonably configuring parameters.

[0004] However, software system parameter configuration faces multiple challenges. First, manual parameter optimization is time-consuming and expensive, requiring multiple tests, analyses, and resource reallocations. Second, software systems usually have a large number of configuration parameters, which may have dependencies between them and are difficult to understand and learn. In addition, as software system versions and features are released, the number of configuration parameters continues to increase, making optimization more difficult. Finally, the optimal configuration of software system parameters depends on the hardware configuration and workload of the application scenario, and the optimal configuration for one application scenario may not be applicable to other scenarios. Summary of the invention

[0005] Based on the above reasons, the present invention discloses a software system performance tuning method and device based on automatic parameter configuration optimization, aiming to solve the problem of software system performance degradation. For a given software system and a specific workload, the goal of software system performance optimization (PTSS) is to find the optimal configuration by exploring high-dimensional space under limited time and constraints, so as to obtain the best performance.

[0006] The technical solution of the present invention includes:

[0007] A software system performance tuning method based on automatic parameter configuration optimization, comprising:

[0008] Collect the configuration set to be tuned for the system to be tuned;

[0009] Sampling a number of candidate configuration instances from the configuration set to be tuned;

[0010] Using performance samples under different candidate configuration instances, configuring a performance prediction model of the system to be tuned, and obtaining a recommended configuration of the current configuration instance based on the performance prediction model; wherein the performance prediction model is constructed based on a proxy model, a sampling function and an optimization strategy;

[0011] After resetting the system to be tuned according to the recommended configuration, identifying the current workload of the system to be tuned, and evaluating the performance value of the current workload;

[0012] After adjusting the proxy model, the sampling function and the optimization strategy based on the performance value of the current workload, returning to sampling a plurality of candidate configuration instances from the set of configurations to be tuned;

[0013] When the first set stop condition is met, the corresponding recommended configuration is output.

[0014] Furthermore, configuring a performance prediction model of the system to be tuned by using performance samples under different candidate configuration instances, and obtaining a recommended configuration of the current configuration instance based on the performance prediction model, includes:

[0015] Creating a configuration space based on sampling a number of candidate configuration instances, wherein the configuration space includes configuration parameters to be optimized, and a value range, parameter type, dependency condition, and condition restriction of each configuration parameter to be optimized;

[0016] Creating a container, wherein the container is used to record the performance evaluation results corresponding to each configuration set;

[0017] According to the configuration parameters to be optimized in the configuration space and the value range of each parameter to be optimized, a plurality of candidate configuration sets are generated using a random sampling algorithm based on a low-discrepancy sequence;

[0018] The candidate configuration sets are sequentially input into the system to be tuned for operation, and the performance evaluation results corresponding to each set of candidate configuration sets are used as training samples and updated into the container;

[0019] Create or get the adjusted proxy model and use the new proxy model to update the sampling function;

[0020] Training a proxy model based on the candidate configuration set and the performance indicator of the candidate configuration set to obtain a new proxy model;

[0021] Update the candidate configuration set based on the updated optimization strategy and the updated sampling function;

[0022] Based on the updated candidate configuration set, returning to the step of sequentially inputting the candidate configuration set into the system to be tuned for execution until a second set stop condition is reached;

[0023] By performing feature analysis on the performance evaluation results in the container, configuration options having an impact on performance improvement that is less than a set value are excluded and the value range of the parameter to be optimized is adjusted;

[0024] Based on the adjusted configuration options and the adjusted value range of the parameter to be optimized, return to the step of creating a configuration space based on sampling a number of candidate configuration instances, and output a recommended configuration for the current configuration instance after a third set stop condition is reached.

[0025] Furthermore, the sampling function includes: a probability lifting sampling function, an expectation lifting sampling function, a lower confidence bound sampling function, an expectation lifting sampling function per second, a probability lifting sampling function per second and / or a constrained expectation lifting sampling function.

[0026] Furthermore, the optimization strategy includes: a sorting optimization algorithm based on a sampling function, a local search optimization algorithm based on a local optimal configuration, or an optimization algorithm based on a gradient.

[0027] Furthermore, the generating of multiple sets of candidate configuration sets using a random sampling algorithm based on a low-discrepancy sequence according to the configuration parameters to be optimized in the configuration space and the value range of each parameter to be optimized includes:

[0028] Determine the value range and type of hyperparameters;

[0029] Select a random sampling algorithm based on low-discrepancy sequences according to the application requirements and hyperparameter type of the system to be tuned;

[0030] Mapping and scaling operations are performed on each hyperparameter to convert the value range of the hyperparameter into a set range;

[0031] Using a selected random sampling algorithm based on a low-discrepancy sequence, a series of sampling points are generated, wherein each sampling point represents a configuration;

[0032] The sampling points are reversely mapped to the value range of the hyperparameter after transformation to obtain multiple sets of candidate configuration sets,

[0033] Furthermore, the random sampling algorithm based on low-discrepancy sequences includes: a random sampling algorithm based on a Halton sequence, a random sampling algorithm based on a Sobol sequence, a random sampling algorithm based on a Hammersley sequence, or a Latin hypercube sampling algorithm.

[0034] Further, the identifying the current workload of the system to be tuned includes:

[0035] Monitor the performance indicator data of the system to be tuned under different workloads; wherein the performance indicator data includes: CPU usage, memory usage, disk read and write speed, and network bandwidth;

[0036] Extracting features from the performance indicator data to obtain statistical features; wherein the statistical features include: average value, maximum value, minimum value, variance, correlation, trend and periodicity;

[0037] Using the performance indicator data and the statistical characteristics, establishing a system load model;

[0038] By using the established system load model, the performance indicator data under the current workload is classified and identified to obtain the current workload.

[0039] Further, the evaluating the performance value of the current workload includes:

[0040] Get the system to be tuned, the current workload, and the current recommended configuration;

[0041] Based on the system to be tuned, the current workload, and the current recommended configuration, calculate the performance value of the current workload.

[0042] A software system performance tuning device based on automatic parameter configuration optimization, comprising:

[0043] A configuration collection module is used to collect the configuration set to be tuned for the system to be tuned;

[0044] A configuration sampling module, used to sample a number of candidate configuration instances from the configuration set to be tuned;

[0045] A performance tuning module, configured to configure a performance prediction model of the system to be tuned using performance samples under different candidate configuration instances, and obtain a recommended configuration of the current configuration instance based on the performance prediction model; wherein the performance prediction model is constructed based on an agent model, a sampling function and an optimization strategy;

[0046] A load identification module, configured to identify a current workload of the system to be tuned after resetting the system to be tuned according to the recommended configuration, and to evaluate a performance value of the current workload;

[0047] A model updating module, configured to adjust the proxy model, the sampling function and the optimization strategy based on the performance value of the current workload, and then return to sampling a plurality of candidate configuration instances from the set of configurations to be tuned;

[0048] The configuration output module is used to output the corresponding recommended configuration until the first set stop condition is met.

[0049] A computer device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the software system performance tuning method based on automatic parameter configuration optimization described in any one of the above items is implemented.

[0050] Compared with the prior art, the present invention has at least the following technical advantages:

[0051] 1. Histogram-based Gradient Boosting Quantile Regression has the following advantages over Gaussian process regression and random forest regression:

[0052] (1) Handling nonlinear relationships and complex data distributions: Histogram gradient boosting quantile regression can better capture nonlinear relationships and complex data distributions. By combining multiple decision tree models, it can adapt to a wider range of data patterns, while Gaussian process regression and random forests may be limited by data distribution and relationships in some cases.

[0053] (2) Robustness: The histogram method is robust to outliers and noise and can reduce their impact on the model. In contrast, Gaussian process regression is more sensitive to outliers, while random forests may have poor robustness.

[0054] (3) Processing large-scale data sets: The histogram method enables the model to process large-scale data sets by blocking and compressing the eigenvalues. In contrast, Gaussian process regression has a large computational overhead on large-scale data sets, and the performance of random forests may be affected by the size of the data.

[0055] (4) Computational efficiency: The histogram method uses histogram approximation when constructing decision tree nodes, which has high computational efficiency. In contrast, Gaussian process regression may require solving more complex mathematical problems and has high computational overhead.

[0056] (5) Confidence interval estimation: Histogram gradient boosting quantile regression can provide confidence intervals for prediction results at different quantiles, helping to quantify uncertainty. In Gaussian process regression and random forests, obtaining confidence intervals may require additional computational or statistical methods.

[0057] (6) Feature importance: The histogram method can provide an assessment of feature importance, helping to understand which features contribute more to the prediction results. In Gaussian process regression and random forests, obtaining feature importance may require additional analysis.

[0058] 2. Histogram-based gradient boosting quantile regression has some advantages in hyperparameter optimization and software system tuning, and is particularly suitable for dealing with problems such as asymmetric distributions, outliers, and quantitative uncertainty. The following are its advantages in these two areas:

[0059] (1) Robustness: Histogram gradient boosting quantile regression is robust to outliers and noise, and can reduce these effects during the optimization process. This makes the model more capable of being optimized in unstable data environments and has a certain degree of stability in the selection of hyperparameters. When tuning a software system, you may encounter the impact of outliers on performance results. The histogram-based method is robust to outliers, which can reduce their impact and improve the stability of performance results.

[0060] (2) Quantifying uncertainty: The optimization process usually needs to consider the uncertainty of the model under different hyperparameters. The histogram method can provide confidence intervals for prediction results at different quantiles, which helps to evaluate the performance differences and stability of different hyperparameter configurations. Software system tuning needs to consider uncertainty, such as performance fluctuations under different hardware configurations or load conditions. The histogram method can provide confidence intervals for prediction results at different quantiles, which helps to evaluate the reliability and stability of performance results.

[0061] (3) Parameter adjustment guidance: The histogram method can provide an assessment of feature importance and help understand which hyperparameters have a greater impact on the optimization results. This helps guide parameter adjustment and optimization strategies.

[0062] (4) Efficient computing: The histogram method uses histogram approximation when constructing decision tree nodes, which has high computational efficiency. This is particularly important for efficient optimization in the hyperparameter search space, which can save computing time and resources.

[0063] (5) Asymmetric distribution: Software performance results often present an asymmetric distribution. Histogram-based methods can more accurately adapt to data distribution at different quantiles and provide more accurate performance predictions.

[0064] (6) Performance Guidance: The histogram method can provide an assessment of feature importance and help understand which factors have a greater impact on software performance. This helps guide the priority and direction of system tuning.

[0065] In summary, histogram-based gradient boosting quantile regression can better handle problems such as asymmetric distribution, outliers, and quantitative uncertainty in hyperparameter optimization and software system tuning, and provide more accurate and stable prediction results and optimization guidance.

[0066] 3. Using historical data to accelerate software system tuning has many advantages, including improving tuning efficiency, optimizing the accuracy of results, and reducing resource consumption. The following are some of the advantages of using historical data to accelerate software system tuning:

[0067] (1) Accelerate the optimization process: Historical data contains information about previous system configurations and performance, which can be used to build prediction models to accelerate the current system tuning process. By predicting the performance under different configurations, expensive actual operations and measurements can be avoided, greatly shortening the optimization cycle.

[0068] (2) Resource saving: The actual operation of different system configurations requires a lot of time and computing resources. Using historical data for prediction can avoid unnecessary waste of resources and reduce the occupation of system resources.

[0069] (3) Efficient search space: The configuration parameters of a software system may constitute a huge search space, and models that use historical data can guide the optimization algorithm to search for high-performance areas more efficiently, thereby reducing the size of the search space.

[0070] (4) Rapid exploration of new strategies: Models that leverage historical data can help quickly evaluate new system configuration strategies, thereby more quickly discovering and validating new optimization solutions.

[0071] (5) Adaptive optimization: The model of historical data can be continuously updated and iterated with new performance data, thereby achieving adaptive optimization and adapting to changes and evolution of system performance.

[0072] (6) Decision support: Models using historical data can provide support for decision making, helping system administrators and developers make more informed decisions and guiding the direction of system tuning.

[0073] In summary, using historical data to accelerate software system tuning can greatly improve tuning efficiency and reduce resource consumption. It can also improve the accuracy and reliability of optimization results, effectively improving system performance.

[0074] 4. Low Discrepancy Sequence Sampling has some advantages over random sequence sampling in hyperparameter optimization, especially in optimization problems with high dimensions and large search spaces. The following are the advantages of low discrepancy sequence sampling in hyperparameter optimization:

[0075] (1) More uniform exploration: Low-discrepancy sequence sampling fills the hyperparameter space in a uniformly distributed manner, making the sample points more evenly distributed throughout the space. This helps to explore the hyperparameter space more comprehensively, making it more likely to find the global optimal solution or avoid missing important areas in high-dimensional situations.

[0076] (2) Reduce redundant samples: Low-difference sequence sampling can reduce redundant sample points and avoid repeated sampling in similar areas. This helps the optimization algorithm to make more effective use of limited samples and improve search efficiency.

[0077] (3) Efficient convergence: Due to the uniform distribution of low-discrepancy sequence sampling in the hyperparameter space, the optimization algorithm can converge to the local optimal solution faster. In contrast, random sequence sampling may cause the optimization algorithm to oscillate or converge more slowly during training.

[0078] (4) Stability: Low-discrepancy sequence sampling can also maintain good uniformity in high-dimensional space, while random sequence sampling is easily affected by the curse of dimensionality in high-dimensional space. This makes low-discrepancy sequence sampling more stable in high-dimensional problems.

[0079] (5) Improved sampling efficiency: Low-discrepancy sequence sampling can usually fill the hyperparameter space faster. Compared with random sampling, it can obtain a global perspective on the impact of hyperparameters earlier, thereby guiding the optimization process faster.

[0080] (6) Optimizing performance prediction models: When constructing performance prediction models, low-discrepancy sequence sampling can better cover the hyperparameter space, allowing the model to more accurately capture the relationship between hyperparameters and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is the interaction diagram of each module of the software system performance tuning model.

[0082] Figure 2 Diagram of the automatic tuning workflow for the performance tuning engine. DETAILED DESCRIPTION

[0083] Exemplary embodiments will be described in detail below with reference to the accompanying drawings.

[0084] The present invention improves the performance and service quality of the system to be tuned through automatic parameter configuration optimization, reduces the burden of manual configuration, reduces hardware resource costs, and improves user experience. The system to be tuned (System Under Tune, SUT) can be selected from many kinds of software systems, including but not limited to data analysis systems, databases, message systems, web application servers, and Linux operating systems. These provide users with a large number of parameters for configuration.

[0085] The software system performance tuning device of the present invention generally includes several components: configuration space, load identification, performance evaluation, and performance tuning engine:

[0086] 1. Configuration Space (CS)

[0087] The configuration space is a parameter space used to describe feasible configurations. It defines various configuration options and their value ranges for the system to be optimized. In this paper, the configuration of the system to be optimized is represented as a set of n parameters C = (c1, c2, ..., c n ), the value of each parameter must be within the configuration boundaries and condition constraints predefined by the system to be tuned, and the system to be tuned assigns a valid value to each parameter in C. For example, Spark has about 200 configurable parameters, each of which is stored in the configuration file in the form of a key-value pair, such as execution memory size, serialization method, number of execution cores, execution cache size, etc.

[0088] In the configuration space, each configuration option is an independent dimension, and the value range defines the selectable values ​​on the dimension. The configuration space can also define various constraints and dependencies, such as conditional constraints (a configuration option is only available when certain conditions are met), mutually exclusive constraints (mutually exclusive relationships between certain configuration options), etc. These constraints and dependencies can further limit the feasible configuration combinations. The present invention considers the following conditions and constraints:

[0089] (1) Parameter range: For each hyperparameter, its value range can be defined. For example, the learning rate can be limited to between 0 and 1.

[0090] (2) Parameter type: Some hyperparameters may need to be integer or discrete values ​​rather than continuous. For example, the maximum depth of a decision tree may only be an integer value.

[0091] (3) Parameter relationships: Certain hyperparameters may have relationships that need to satisfy specific mathematical or logical conditions. For example, the weight of the regularization term may need to be inversely proportional to the learning rate.

[0092] (4) Constraints: In addition to the relationship between hyperparameters, other constraints can also be defined. For example, the training time cannot exceed a certain limit, or the memory usage of the model needs to be within a certain range.

[0093] 2. Workload Recognition (WR).

[0094] Workloads represent specific tasks running on the system to be tuned. For example, different programs running on Spark, a group of transactions on MySQL, a group of requests on Tomcat, or different message lengths, the number of producers, the message sending and receiving modes, and the number of logically available resources in the software system on Kafka. Different workloads represent different operating environments and application scenarios of the software system. Under different workloads, the software system will have different performance.

[0095] The present invention monitors and analyzes various indicators of the system and automatically determines the current system load through a machine learning model. The specific steps are as follows:

[0096] (1) Monitoring system indicators: First, it is necessary to monitor and record various system indicators, including but not limited to CPU usage, memory usage, disk read and write speed, network bandwidth, etc. These indicators are obtained through the interfaces provided by the operating system, system performance monitoring tools, and third-party monitoring tools.

[0097] (2) Feature extraction: Feature extraction is performed based on the monitored indicator data. Statistical features are calculated, including but not limited to average, maximum, minimum, variance, correlation, etc., and more advanced features are extracted through time series analysis, including but not limited to trends, periodicity, etc.

[0098] (3) Establishing a load model: Using monitoring data and features, establish a system load model. The machine learning algorithms used in the present invention include, but are not limited to, random forest, extreme tree, XGboost, LightGBM, recurrent neural network, convolutional neural network, and self-attention neural network, etc., to train and build a load automatic identification model. During training, it is necessary to prepare a labeled data set, which contains indicator data under known load conditions and corresponding load labels.

[0099] (4) Load classification and identification: Use the established load model to classify and identify the system indicators monitored in real time. By inputting the current indicator data, the load model can predict the current load status of the system.

[0100] (5) Continuous optimization: Automatic load identification is a dynamic process, and the system load situation may change over time. Therefore, it is necessary to regularly evaluate and optimize the load model to adapt to changes in the system load and update and adjust the model according to actual conditions.

[0101] 3. Performance Evaluation (PE)

[0102] Software system performance evaluation is the process of measuring the performance and operation effect of software systems in different aspects. Its purpose is to identify bottlenecks and performance problems in the system and provide guidance on how to improve system performance. Performance is an important non-functional characteristic of the system to be tuned, which directly affects the user experience and the cost of operation. In this paper, performance is regarded as a black box function, and the formula P = (S, W, C) is used to represent the performance value. Given a software system S to be tuned, a workload W and a parameter configuration C, different configurations of the system to be tuned have different performance goals. For MySQL data access load, the performance goal is to increase throughput, while for data analysis tasks on Spark, reducing execution time is its performance goal. Without losing the versatility of the software system, the present invention hopes to optimize the performance value on all configurations. For the case of maximizing the performance value, minimize the negative value of its performance.

[0103] 4. Performance Tuning Engine (PTE).

[0104] For a given system to be tuned, it is difficult to find a good configuration to understand its performance characteristics due to the high dimensionality of the configuration space, limited tuning time, and lack of prior knowledge. To address these challenges, this paper designs a method based on machine learning. The key step is to use the collected performance samples under different configurations to establish a performance prediction model for the configuration tuning system. The following are the main contents of each functional module:

[0105] (1) Parameter Space (PS): Parameter space is an important component of PTE. Its goal is to convert the configuration parameters of the software system into a continuous hyperparameter space to facilitate subsequent machine learning modeling and optimization.

[0106] (2) Sampler: The present invention uses low-discrepancy sequence sampling to replace the traditional random sequence generation method to generate high-quality, uniformly distributed random sampling points. The following are the general steps for configuration generation based on low-discrepancy sequence sampling:

[0107] a. Determine the value range and type of the hyperparameter: First, you need to define the value range and data type of each hyperparameter, including floating point numbers, integers, or discrete data.

[0108] b. Select a suitable low-discrepancy sequence generation algorithm: Select a suitable low-discrepancy sequence generation algorithm based on specific application requirements and hyperparameter types.

[0109] c. Map and scale each hyperparameter: According to the value range of the hyperparameter, map and scale each hyperparameter to convert it to the appropriate range. For example, for floating-point parameters, they can be mapped to the range of [0,1].

[0110] d. Generate sampling points using low-discrepancy sequences: Generate a series of sampling points using the selected low-discrepancy sequence generation algorithm. Each sampling point represents a configuration.

[0111] e. Back-mapping restoration: Back-map the generated sampling points to the value range of the original hyperparameters to obtain the final configuration.

[0112] The low-discrepancy sequence sampling method provided by the present invention includes Halton sequence, Sobol sequence, Hammersley sequence and Latin hypercube sampling.

[0113] The basic idea of ​​the Halton sequence is to use different prime number bases and generate sequences according to certain rules. The following are the general steps for sampling the Halton sequence to generate sampling points:

[0114] 1. Select a prime basis: First, select a set of different prime numbers as the basis. Each prime number corresponds to a sequence of one dimension. The prime numbers usually selected are relatively small and unrelated to ensure low variability of the sequence.

[0115] 2. Convert the serial number to a decimal fraction: For the i-th sample point, express its serial number i as a decimal fraction based on different prime numbers. For example, for prime numbers p1 and p2, i can be expressed as (i p1 ,i p2 ), where i p1 and i p2 represent the decimal fraction of i in the basis p1 and p2 respectively.

[0116] 3. Generate sampling points: For each dimension, use the fractional part to generate sampling points. Usually, you can use recursion or other methods to convert the fractional part into a binary representation and then generate sampling points.

[0117] 4. Scaling and mapping: According to the actual needs of the problem, the generated sampling points are scaled and mapped so that they fall within the specified hyperparameter space.

[0118] The generation process of the Sobol sequence is relatively complicated, but it can maintain better uniformity and low variance in high dimensions. The following are the general steps for generating sampling points for Sobol sequence sampling:

[0119] 1. Select prime number basis: Sobol sequence uses a set of prime numbers as basis, each prime number corresponds to a dimension of the sequence. These prime numbers need to meet certain conditions to ensure that the generated sequence has good properties.

[0120] 2. Calculate the generator polynomial: For each dimension, the Sobol sequence uses a generator polynomial to calculate the next value in the sequence. The generator polynomial is a polynomial related to the basis that is used to generate each element in the sequence.

[0121] 3. Initialization: Initialize the starting point of the sequence for each dimension. Usually, the starting point can be chosen to be zero or other appropriate value.

[0122] 4. Calculate sequence values: For each dimension, use the generator polynomial to calculate the next sequence value. The generator polynomial uses the previous sequence value and the basis to calculate the current value.

[0123] 5. Reverse Bit: For each dimension of the sequence value, perform a bit reversal operation. This is one of the key steps of the Sobol sequence, which ensures that the sequence is evenly distributed in the hyperparameter space.

[0124] 6. Scaling and mapping: According to the actual needs of the problem, the generated sampling points are scaled and mapped so that they fall within the specified hyperparameter space.

[0125] Latin Hypercube Sampling (LHS) is a sampling method commonly used in hyperparameter optimization and experimental design to generate uniformly distributed sampling points. LHS can help better explore the hyperparameter space and reduce the correlation between sampling points. The following are the general steps for generating sampling points in the Latin Hypercube Sampling sequence:

[0126] 1. Determine the dimension and number of samples: First, determine the dimension of the hyperparameters to be optimized and the number of samples that need to be generated.

[0127] 2. Generate random uniformly distributed values: For each dimension, generate a sample number of uniformly distributed random values, usually in the range of [0,1).

[0128] 3. Construct the initial sampling matrix: The random numbers of each dimension are organized into columns to form the initial sampling matrix. Each column in this matrix corresponds to a hyperparameter dimension, and each row corresponds to a sample point.

[0129] 4. Rearrange the values ​​of each column: For each column, rearrange its values ​​to ensure that the values ​​of each column are unique and random. This can avoid repeated sampling in each dimension and reduce correlation.

[0130] 5. Map values ​​to hyperparameter ranges: Map the values ​​of each row of each dimension so that they fall within the corresponding hyperparameter range. This can be achieved by a linear transformation, mapping random values ​​to the specified range.

[0131] 6. Generate sampling points: Take the mapped value of each row as a sampling point to obtain the final Latin hypercube sampling sequence.

[0132] (3) Surrogate Model: In PTE, a surrogate model is a method that replaces the objective function (performance evaluation function) by establishing an approximate function. The surrogate model uses the generated configuration samples and corresponding performance data to train a machine learning model to predict the relationship between configuration and performance. The trained model can quickly predict the objective function value under different hyperparameter combinations, thereby accelerating the optimization process. The following are the surrogate models used in the PTE, which can be selected and applied according to specific problems and needs.

[0133] a. Decision tree-based proxy model: The decision tree model can be used to predict the value of discrete or continuous objective functions. It divides the objective function according to the value of the hyperparameter and constructs a tree structure for prediction. It includes CART regression tree, Extremely Randomized Trees, Random Forest, Gradient Boosting Regression Tree, XGboost, and LightGBM.

[0134] b. Surrogate model based on Gaussian process: The Gaussian process model is a probabilistic model used to model the uncertainty of the target function. It makes predictions by fitting the joint distribution of observed hyperparameters and target function values. The Gaussian process model can provide the mean and variance of the prediction, thereby providing an estimate of the uncertainty of the prediction results. Including Gaussian Process Regression.

[0135] c. Neural network-based proxy models: Neural network models can be used to learn complex nonlinear relationships between hyperparameters and objective functions. By using deep learning techniques, neural networks can be trained on large amounts of data and provide accurate objective function predictions. These include feedforward neural networks, convolutional neural networks, recurrent neural networks, and self-attention neural networks.

[0136] d. Histogram-based Gradient Boosting Quantile Regression: HGBQR combines the ideas of Gradient Boosting Tree and Quantile Regression to predict the conditional distribution of the target variable at different quantiles. The core idea of ​​this method is to introduce the histogram method to process the distribution of data on the basis of the gradient boosting tree to better adapt to the quantile regression problem. Specifically, its steps are as follows:

[0137] 1. Data segmentation and histogram construction: In the construction process of each weak learner (usually a decision tree), the data is divided into multiple bins, and then a histogram is constructed in each bin to approximate the distribution of the target variable in that bin.

[0138] 2. Quantile loss function: In each iteration, the quantile loss function of the target variable in each interval is calculated. This loss function is used to measure the prediction effect of the model at different quantiles to guide the update of the model.

[0139] 3. Optimization process: Through optimization algorithms such as gradient descent, the model parameters are updated to reduce the quantile loss function. This allows the model to gradually approach the distribution of the target variable at different quantiles.

[0140] 4. Model ensemble: The final prediction result is the weighted average of multiple weak learners, and the weight is determined by the derivative of the quantile loss function. This ensemble model can provide prediction results and confidence intervals at different quantiles.

[0141] HGBQR performs well in dealing with quantile regression problems, asymmetric distribution, quantitative uncertainty, etc. It combines the advantages of gradient boosting tree and histogram methods, and can more accurately model the conditional distribution of target variables at different quantiles, providing more accurate prediction results and confidence intervals.

[0142] (4) Sampling function: Based on the trained model, a set of candidate configurations are sampled in the parameter space. These configurations are considered to have higher potential performance and can be further evaluated in the next optimization step. In PTE, the following sampling functions are selected to select the next hyperparameter combination to be evaluated.

[0143] a. Probability of Improvement: The probability improvement sampling function calculates the probability of each hyperparameter combination reaching or exceeding the current optimal result. It estimates the probability of the objective function exceeding the threshold based on the predicted distribution of the surrogate model and selects the hyperparameter combination with the highest probability improvement. The probability improvement function pays more attention to the probability of reaching a specific goal and is suitable for optimization problems that need to reach a certain target level. The calculation formula is as follows:

[0144] impove(x)=best(x * )-μ(x)-ε

[0145]

[0146] Where best(x * ) represents the current optimal configuration x * The corresponding performance value, μ(x) represents the posterior expected value corresponding to configuration x, ε is a decimal greater than or equal to 0, which is used to adjust the performance fluctuation value. impove(x) represents the improvement of the current configuration x, σ(x) represents the posterior standard deviation corresponding to configuration x, represents the cumulative distribution function.

[0147] b. Expected Improvement: The expected improvement sampling function selects the hyperparameter combination with the largest expected improvement as the next point to be evaluated based on the prediction results of the surrogate model and the uncertainty of the difference. This allows us to explore unknown areas while tending to select hyperparameters that are expected to bring greater improvements. The calculation formula is as follows:

[0148]

[0149] Where impove(x) represents the improvement of the current configuration x, and σ(x) represents the standard deviation of the posterior period corresponding to configuration x. ρ(·) is the cumulative distribution function and ρ(·) is the probability density function.

[0150] c. Lower Confidence Bound: The lower confidence bound sampling function comprehensively considers the expectation and uncertainty of the objective function in the prediction results of the surrogate model. It is based on the concept of confidence bound and balances between optimization and exploration. The lower confidence bound function selects the hyperparameter combination with the highest confidence bound as the next point to be evaluated. This allows exploration in areas where good results are known while taking into account the uncertainty of the objective function. The calculation formula is as follows:

[0151]

[0152] Where μ(x) represents the posterior expected value corresponding to configuration x, σ(x) represents the posterior standard deviation corresponding to configuration x, α and β represent adjustment coefficients, n represents the number of optimization targets, and m represents the total number of observations that have been evaluated.

[0153] d. Expected Improvement per Second (EIps) is a metric used in optimization problems to measure the degree of expected improvement per unit time. EIps can be used to evaluate the pros and cons of candidate configurations. It combines two key factors: expected improvement and time. Time is the posterior expected value of the time series proxy model. The calculation formula is as follows:

[0154]

[0155] where μ t (x) represents the posterior time expectation corresponding to configuration x.

[0156] e. Probability Improvement per Second (PIps) is a metric used in optimization problems to measure the degree of probability improvement per unit time. It combines two key factors: probability improvement and time. The calculation formula is as follows:

[0157]

[0158] where μ t (x) represents the posterior time expectation corresponding to configuration x.

[0159] f. Constrained Expected Improvement: It is a metric used to measure candidate solutions in constrained optimization problems. It combines expected improvement and constraints to evaluate the optimization potential of candidate solutions under the premise of satisfying the constraints. In constrained optimization problems, in addition to the optimization objective, there are also some constraints that need to be satisfied. Candidate solutions must have good performance under the constraints. CEI considers the expected improvement of the objective function at the same time.

[0160] The quality of candidate solutions is evaluated by the degree of progress and satisfaction of constraints. The calculation formula is as follows:

[0161]

[0162] Where μ(x) represents the posterior expected value corresponding to configuration x, σ(x) represents the posterior standard deviation corresponding to configuration x, represents the cumulative distribution function.

[0163] (5) Optimization strategy: The optimization strategy refers to evaluating and ranking candidate configurations during the optimization process, and selecting the best configuration as the final optimization result based on the performance results. The following methods can be selected and combined according to the specific situation and the characteristics of the problem. By using these optimization strategies, the configuration can be effectively searched and improved during the optimization process to find the configuration with the best performance as the final result. The choice of optimization strategy depends on the nature of the problem, the characteristics of the configuration space, and the requirements of the optimization goal. The optimization strategies mentioned in this article include the following methods:

[0164] a. Sorting optimization based on sampling function: The sampling function generates configuration samples based on the prior knowledge and sampling strategy in the configuration space. These samples are evaluated for performance and ranked according to the performance results.

[0165] b. Local search optimization based on the local optimal configuration: Starting from the current configuration with the best performance, a local search is performed near it. The local search method can further improve performance by fine-tuning the configuration parameters. By continuously adjusting and optimizing the configuration, the local optimal solution can be gradually approached.

[0166] c. LBFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno) method: This is a gradient-based optimization algorithm used to find local optimal solutions in continuous parameter space. The LBFGS method updates the configuration parameters by estimating the gradient of the objective function and the inverse of the Hessian matrix to minimize the objective function.

[0167] Specifically, the interaction process of each module of the software system performance tuning model, such as Figure 1 As shown, it can be divided into the following steps:

[0168] (0) Preparation: Before starting performance tuning, some preparation work needs to be completed, including starting the system to be tuned SUT, preparing the configuration set to be tuned CS, starting PTE, running the load identification engine WR, and preparing the performance evaluation standard program PE.

[0169] (1) Sampling configuration instances: Sample a batch of candidate configuration instances from the configuration set to be tuned CS and provide them to PTE. Generate a recommended configuration C based on the current configuration instance.

[0170] (2) Reset the system: Use the recommended configuration C in step (1) to reset the system to be tuned SUT to ensure that the system runs with the latest configuration.

[0171] (3) Workload identification: Run the load identification engine WR to identify the current workload. The load identification engine analyzes the current system load and determines the type and characteristics of the workload currently faced by the system.

[0172] (4) Performance evaluation: Run the performance evaluation standard program PE to evaluate the performance of the current workload and obtain the performance value P.

[0173] (5) Feedback and iteration: The performance value P obtained in step (4) is fed back to PTE. The performance tuning engine adjusts the proxy model, sampling function, optimization strategy and recommendation process according to the feedback performance value P. Then, steps (1) to (4) are repeated until the optimization process is stopped when the set conditions and constraints are met.

[0174] (6) Verification and deployment: Further verify and test the optimal configuration to ensure feasibility and performance improvement in the actual production environment. Deploy the optimal configuration to the production system and monitor performance indicators.

[0175] Through this iterative process, the performance tuning model continuously samples configurations, resets the system, identifies loads, evaluates performance, and provides feedback and optimization based on performance results. This can gradually improve system performance and find the optimal configuration to meet preset performance goals and constraints.

[0176] like Figure 2 The automatic tuning workflow of the performance tuning engine is divided into the following steps:

[0177] (0) Preparation: Before performance tuning, some necessary preparation is required. First, determine the software system to be configured and tuned, and find the set of all configurable parameters based on the official information of the system. Then, select the configuration set CS to be tuned that has the highest correlation with the performance of the software system.

[0178] (1) Generate configuration space: Create a configuration space based on the selected configuration set CS to be optimized. The configuration space contains the configuration parameters to be optimized and defines the value range, parameter type, dependency conditions and conditional constraints for each parameter. This helps to define the scope and constraints of the search.

[0179] (2) Create a historical container: Create a container for storing the configuration and historical observation data of performance indicators recommended during the tuning process. This container is used to record the results of each performance evaluation for subsequent analysis and optimization.

[0180] (3) Generate initial configurations: Based on the parameter set and the defined value range in the configuration space, a random sampling method based on low-discrepancy sequences is used to generate multiple sets of original configuration combinations. These initial configurations will serve as the starting point of the optimization process.

[0181] (4) Performance evaluation: The generated initial configuration combinations are sequentially input into the software system to run and measure their performance indicators. The performance indicators of each configuration group are recorded and used as training samples for the parameter optimization method and updated to the observation history container.

[0182] (5) Create or adjust algorithms: Automatically create or adjust algorithms such as proxy models, sampling functions, and optimization strategies based on scenario information such as user settings, configuration type and quantity, and optimization history. These algorithms will help make the optimization process more efficient and accurate based on existing data and requirements.

[0183] (6) Update the proxy model: The observed configuration and performance history are preprocessed to meet some target conditions and constraints. The preprocessed data is then used to train the proxy model, which can predict performance indicators based on the input configuration parameters.

[0184] (7) Update sampling function: Use the new proxy model to update the sampling function. The sampling function is responsible for generating a set of candidate configurations based on the current proxy model for use by the optimization strategy.

[0185] (8) Run the optimization strategy: Generate a set of candidate configurations based on the optimization strategy and the updated sampling function, and sort them according to the requirements of the optimization strategy. Then recommend the currently predicted optimal configuration as the next configuration choice.

[0186] (9) Stop condition: Determine whether to terminate the optimization process based on the preset stop condition. The stop condition can be reaching a predetermined number of iterations, the objective function reaching an expected threshold, etc.

[0187] (10) Repeat steps (4)-(9): According to the selected acquisition strategy, the performance evaluation, agent model training, sampling function update, running optimization strategy and configuration recommendation steps are repeated. In this way, the configuration space can be gradually explored and the objective function can be optimized.

[0188] (11) Feature analysis: Perform feature analysis on observed historical configuration data to explore the relationship between configuration and performance indicators. By analyzing the impact of each configuration item on the performance of the software system, configuration items that have little impact on performance improvement can be excluded.

[0189] (12) Adjust configuration: Based on the results of feature analysis, adjust configuration options and ranges to further optimize system performance.

[0190] (13) Repeat (1)-(12): Continue the cycle of configuration optimization and performance improvement. After the optimization process is completed, the configuration with the best performance is selected as the recommended configuration C based on the agent model and the observed historical data.

[0191] Through the cyclic iteration of the above steps, the configuration optimization system can continuously perform performance evaluation, model training, configuration recommendation, and parameter adjustment to gradually optimize the performance of the software system, find the best configuration, and meet the preset performance goals.

[0192] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof.

Claims

1. A software system performance tuning method based on automated parameter configuration optimization, characterized in that: The method comprises: Collect the configuration set to be tuned for the system to be tuned; Sampling a number of candidate configuration instances from the configuration set to be tuned; Using performance samples under different candidate configuration instances, configuring a performance prediction model of the system to be tuned, and obtaining a recommended configuration of the current configuration instance based on the performance prediction model; wherein the performance prediction model is constructed based on a proxy model, a sampling function and an optimization strategy; After resetting the system to be tuned according to the recommended configuration, identifying the current workload of the system to be tuned, and evaluating the performance value of the current workload; After adjusting the proxy model, the sampling function and the optimization strategy based on the performance value of the current workload, returning to sampling a plurality of candidate configuration instances from the set of configurations to be tuned; When the first set stop condition is met, the corresponding recommended configuration is output.

2. The method according to claim 1, characterized in that The using performance samples under different candidate configuration instances to configure the performance prediction model of the system to be tuned, and obtaining the recommended configuration of the current configuration instance based on the performance prediction model, includes: Creating a configuration space based on sampling a number of candidate configuration instances, wherein the configuration space includes configuration parameters to be optimized, and a value range, parameter type, dependency condition, and condition restriction of each configuration parameter to be optimized; Creating a container, wherein the container is used to record the performance evaluation results corresponding to each configuration set; According to the configuration parameters to be optimized in the configuration space and the value range of each parameter to be optimized, a plurality of candidate configuration sets are generated using a random sampling algorithm based on a low-discrepancy sequence; The candidate configuration sets are sequentially input into the system to be tuned for operation, and the performance evaluation results corresponding to each set of candidate configuration sets are used as training samples and updated into the container; Create or get the adjusted proxy model and use the new proxy model to update the sampling function; Training a proxy model based on the candidate configuration set and the performance indicator of the candidate configuration set to obtain a new proxy model; Update the candidate configuration set based on the updated optimization strategy and the updated sampling function; Based on the updated candidate configuration set, returning to the step of sequentially inputting the candidate configuration set into the system to be tuned for execution until a second set stop condition is reached; By performing feature analysis on the performance evaluation results in the container, configuration options having an impact on performance improvement that is less than a set value are excluded and the value range of the parameter to be optimized is adjusted; Based on the adjusted configuration options and the adjusted value range of the parameter to be optimized, return to the step of creating a configuration space based on sampling a number of candidate configuration instances, and output a recommended configuration for the current configuration instance after a third set stop condition is reached.

3. The method according to claim 1 or 2, characterized in that The sampling functions include: a probability boost sampling function, an expectation boost sampling function, a lower confidence bound sampling function, an expectation boost sampling function per second, a probability boost sampling function per second and / or a constrained expectation boost sampling function.

4. The method according to claim 1 or 2, characterized in that: The optimization strategies include: a sorting optimization algorithm based on a sampling function, a local search optimization algorithm based on a local optimal configuration, or an optimization algorithm based on a gradient.

5. The method according to claim 2, characterized in that The method of generating multiple sets of candidate configuration sets using a random sampling algorithm based on a low-discrepancy sequence according to the configuration parameters to be optimized in the configuration space and the value range of each parameter to be optimized includes: Determine the value range and type of hyperparameters; Select a random sampling algorithm based on low-discrepancy sequences according to the application requirements and hyperparameter type of the system to be tuned; Mapping and scaling operations are performed on each hyperparameter to convert the value range of the hyperparameter into a set range; Using a selected random sampling algorithm based on a low-discrepancy sequence, a series of sampling points are generated, wherein each sampling point represents a configuration; The sampling points are inversely mapped to the value range of the hyperparameter after transformation to obtain multiple groups of candidate configuration sets.

6. The method according to claim 5, characterized in that The random sampling algorithm based on low-discrepancy sequence includes: a random sampling algorithm based on Halton sequence, a random sampling algorithm based on Sobol sequence, a random sampling algorithm based on Hammersley sequence or a Latin hypercube sampling algorithm.

7. The method according to claim 1, characterized in that The identifying the current workload of the system to be tuned includes: Monitor the performance indicator data of the system to be tuned under different workloads; wherein the performance indicator data includes: CPU usage, memory usage, disk read and write speed, and network bandwidth; Extracting features from the performance indicator data to obtain statistical features; wherein the statistical features include: average value, maximum value, minimum value, variance, correlation, trend and periodicity; Using the performance indicator data and the statistical characteristics, establishing a system load model; By using the established system load model, the performance indicator data under the current workload is classified and identified to obtain the current workload.

8. The method according to claim 1, characterized in that The evaluating the performance value of the current workload includes: Get the system to be tuned, the current workload, and the current recommended configuration; Based on the system to be tuned, the current workload, and the current recommended configuration, calculate the performance value of the current workload.

9. A software system performance tuning device based on automatic parameter configuration optimization, characterized in that: The device comprises: A configuration collection module is used to collect the configuration set to be tuned for the system to be tuned; A configuration sampling module, used to sample a number of candidate configuration instances from the configuration set to be tuned; A performance tuning module, configured to configure a performance prediction model of the system to be tuned using performance samples under different candidate configuration instances, and obtain a recommended configuration of the current configuration instance based on the performance prediction model; wherein the performance prediction model is constructed based on an agent model, a sampling function and an optimization strategy; A load identification module, configured to identify a current workload of the system to be tuned after resetting the system to be tuned according to the recommended configuration, and to evaluate a performance value of the current workload; A model updating module, configured to adjust the proxy model, the sampling function and the optimization strategy based on the performance value of the current workload, and then return to sampling a plurality of candidate configuration instances from the set of configurations to be tuned; The configuration output module is used to output the corresponding recommended configuration until the first set stop condition is met.

10. A computer device, characterized in that: The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the software system performance tuning method based on automatic parameter configuration optimization as described in any one of claims 1 to 8 is implemented.

Images