An automated tuning system and method based on machine learning
By using an automated tuning system based on machine learning, which leverages the XGBoost model to train feature importance ranking and incremental learning, the problems of poor flexibility and high computational cost in system parameter tuning are solved. This achieves efficient and stable parameter optimization and feedback loop, making it suitable for online environments.
Patent Information
- Application Number
- CN202611237523.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies lack continuous learning capabilities in system parameter tuning, have high computational overhead, cannot form a tuning closed loop, and lack effective protection against failures, resulting in poor flexibility and insufficient stability.
An automated tuning system based on machine learning is adopted. Through offline model building, online parameter recommendation and feedback closed-loop mechanism, the XGBoost regression model is used to train feature importance ranking, generate candidate configurations and monitor the effect, and realize incremental learning and safe rollback.
It achieves efficient and intelligent parameter tuning, reduces the search space, has a latency in the second range, is suitable for online use, has continuous learning capabilities and a feedback loop, and ensures system stability.
Smart Images

Figure CN123114908A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of system and application performance optimization, specifically to an automated tuning system and method based on machine learning. Background Technology
[0002] In modern computing systems, system parameters and application configurations have a significant impact on performance and require fine-tuning based on the actual load and hardware environment. Commonly used tuning methods mainly include the following categories: 1. Manual experience-based optimization: This method relies on the domain knowledge and historical experience of operations and maintenance personnel to manually adjust parameter configurations. It is highly dependent on personal experience, lacks universality and reproducibility, and is extremely inefficient when dealing with high-dimensional parameter spaces, making it difficult to find the globally optimal solution.
[0003] 2. Rule Engine Tuning: Pre-written fixed tuning rules (e.g., "increase the buffer when memory usage exceeds 80%"). This method has poor flexibility, cannot adapt to dynamically changing loads and environments, has high rule maintenance costs, and is prone to missing optimal configurations in complex interaction scenarios.
[0004] 3. Grid Search vs. Random Search: Both search methods seek the optimal configuration by exhaustively searching or randomly sampling the parameter space. The computational complexity of grid search increases exponentially with the number of parameters (O(m...). n This approach is not feasible in high-dimensional scenarios; while random search reduces computational cost, it lacks directionality, has a slow convergence speed, and cannot utilize historical tuning experience.
[0005] 4. Bayesian Optimization: Sequential optimization based on Gaussian process surrogate model. The computation cost of Gaussian processes in high-dimensional parameter spaces is extremely high (O(n³)), and the choice of kernel function is sensitive to the optimization effect, making it difficult to extend to more than a dozen parameters in practical applications.
[0006] 5. Deep Reinforcement Learning Approach: This approach models optimization as a Markov decision process, allowing the agent to learn the optimal policy through interaction with the environment. However, deep reinforcement learning training is unstable, requires a large number of trial and error samples, and struggles to guarantee the security of the online environment. Optimization failures can lead to a significant degrade in system performance.
[0007] Common drawbacks of existing methods: 1. Most methods are one-time optimizations and cannot adaptively adjust to continuous changes in the system's operating state, lacking continuous learning capabilities; 2. Failure to establish a closed-loop mechanism of "recommendation-application-monitoring-feedback" prevents the feedback of optimization successes and failures to the model, thus hindering its ability to self-evolve. 3. In high-dimensional parameter spaces, there is a lack of effective dimensionality reduction methods, resulting in excessive computational overhead and making it difficult to meet the needs of online real-time optimization; 4. There is a lack of effective security measures to protect against tuning failures. If an incorrect configuration is recommended, it may lead to system crashes or a sharp deterioration in performance. Summary of the Invention
[0008] The technical problem to be solved by the present invention is to provide an automated tuning system and method based on machine learning, which addresses the above-mentioned problems in the prior art. The present invention aims to overcome the shortcomings of the prior art, such as reliance on experience, poor flexibility, high computational cost, lack of continuous learning ability, and inability to form a tuning closed loop.
[0009] To address the aforementioned technical problems, this invention employs an offline model construction stage, an online parameter recommendation stage, and a feedback closed-loop mechanism, wherein: The offline model building phase includes: extracting samples from the historical tuning database, concatenating the running indicators F and parameter configuration P in the samples as input features, using performance scores S as labels, training the XGBoost regression model, evaluating the mean absolute error MAE and the coefficient of determination R² on the validation set, and when MAE is less than a preset threshold θ and R² is greater than a preset threshold, the model passes the validation and outputs a feature importance ranking table. The online parameter recommendation stage includes: obtaining the current system operating metric F_cur and parameter configuration P_cur; selecting the top K parameters by importance as the key parameter subspace based on the feature importance ranking table, while keeping the current values of the remaining non-key parameters unchanged; generating candidate values for each key parameter in the key parameter subspace according to its type; generating a candidate configuration set C by Cartesian product; concatenating the current system operating metric F_cur with the candidate configuration P_ci to obtain X_ci = [F_cur| P_ci]; using the trained XGBoost model to predict the performance score S_ci of each candidate configuration in batches; and outputting a recommended configuration list based on the predicted performance. The feedback loop mechanism includes: after applying the recommended configuration, start monitoring the effect; at the end of the monitoring window, calculate the actual optimization effect and judge and roll back it; when the actual optimization effect is excellent, keep the current configuration; otherwise, make fine adjustments or automatically roll back to the configuration P_old before optimization; generate optimization records and write them to the historical optimization database; and set a differentiated weight 'a' for the failed optimization samples, where 'a' is lower than the weight of normal samples.
[0010] Furthermore, it also includes a parameter space definition table and a tuning record table. The parameter space definition table records the name, type, value range, and default value of each adjustable parameter. The tuning record table records complete information for each tuning operation, including tuning time, parameter configuration before tuning P_old, parameter configuration after tuning P_new, performance before tuning S_old, performance after tuning S_new, and tuning effect E, where tuning effect E = S_old. S_new.
[0011] Furthermore, the historical optimization database stores all historical optimization samples. Each optimization sample is a triple, which includes F, P, and S; where F is the system operation index feature vector, P is the current parameter configuration vector, and S is the corresponding performance score.
[0012] Furthermore, the XGBoost model training first acquires samples from the historical tuning database. The running metric F and parameter configuration P from these samples are concatenated as input features, and the performance score is used as the label. The sample features are normalized, One-Hot encoded, and missing values are filled. The training and validation sets are then divided, and the XGBoost model is trained. Its principle is a gradient boosting tree, with built-in L1 / L2 regularization constraints on the weights and number of leaf nodes. The MAE and R² are evaluated on the validation set. If the MAE is less than a preset threshold θ (which is configurable, set to 5% of the target performance metric), and R² is greater than a preset threshold (also configurable, set to 0.85), the model passes validation and enters the online state. Otherwise, the hyperparameters are adjusted and retrained. The trained model outputs a feature importance ranking table, marking the top K most important parameters as key parameters, where K = min(total number of parameters, 10).
[0013] Furthermore, the parameter types include continuous uniform sampling, enumeration traversal, and Boolean enumeration; the prediction performance ranking is performed by sorting in ascending order when the objective is to minimize, and taking the top N configurations as the recommendation list, where N=5. The configuration set C is generated by Cartesian product, where the configuration set C={P_c1, P_c2, ..., P_cM}. The recommended configuration list is output, and the prediction performance improvement ΔS_i = S_cur– S_ci for each recommended configuration is recorded, where S_cur is the performance score corresponding to the current configuration, S_ci is the prediction performance score corresponding to the i-th candidate configuration P_ci, and the performance score S_ci = f(X_ci), X_ci= [F_cur | P_ci], is the input feature constructed for each candidate configuration P_ci.
[0014] Furthermore, the determination and rollback of the actual optimization effect includes: When E_real>0: the optimization is effective, and the current configuration is maintained When -Δ_threshold ≤ E_real ≤ 0, the effect is not obvious but no degradation occurs, the current configuration is maintained and marked as "fine-tuning"; When E_real< Δ_threshold: the performance degradation exceeds the threshold, where the threshold Δ_threshold = 10% of the target indicator, an automatic rollback is triggered to restore to the pre-optimization configuration P_old; Write the complete record of this optimization into the historical optimization database; for samples with optimization failure, i.e., E_real<0, set their sample weight to α during subsequent incremental training, where α=0.5, which is lower than the normal sample weight of 1.0, so that the model reduces the recommendation tendency for similar parameter configurations.
[0015] Further, an incremental learning mechanism is also included; when the number of new samples added to the historical optimization database or the time elapsed since the last model update meets the trigger condition, the Warm Start strategy or full replacement strategy will be adopted for incremental training of the model, the MAE of the old model and the new model are respectively evaluated on the verification set, and if MAE_new<MAE_old × (1-ε), the new model is replaced and launched, otherwise the old model is retained, where ε=0.02.
[0016] Further, the incremental learning mechanism includes: When the number of new samples added after the last model update is not less than N_incr, or the time elapsed since the last model update is not less than D_incr, incremental training is triggered, where N_incr=200, and D_incr=7 days; When the number of decision trees T_cur of the current online model is less than T_max, the Warm Start strategy is adopted, all parameters of the current online model and the T_cur trained decision trees are loaded, new samples are appended to the training set, and T_add decision trees are continuously trained on the basis of the existing T_cur decision trees, where T_max=500, T_add=50, and the total number of decision trees after incremental training does not exceed T_max; When the number of decision trees T_cur of the current online model is greater than or equal to T_max, the full replacement training strategy is adopted, all historical data including new samples are used to train a new model containing T_max decision trees from scratch to replace the old model; Evaluate the MAE of the old model and the new model on the verification set respectively, if MAE_new<MAE_old × (1 ε), then replace the online model with the new model; otherwise, retain the old model, where ε is set to 0.02, MAE_new is the mean absolute error of the new model on the same validation set, and MAE_old is the mean absolute error of the old model on the same validation set. Further, a certain proportion of early historical samples are retained in the validation set in the incremental learning mechanism to prevent catastrophic forgetting, and the proportion is set to twenty percent.
[0017] The present invention further provides an automatic tuning system based on machine learning, comprising: a data acquisition module, configured to periodically collect a system operation indicator F_cur and a parameter configuration P_cur vector, and form a sample in combination with a corresponding performance score; a historical tuning database, configured to store all historical tuning samples, each sample being a triple (F, P, S), where F is a system operation indicator feature vector, P is a parameter configuration vector, and S is a performance score; a model training module, configured to train an XGBoost regression model, output a feature importance ranking table after verification passes, and mark the first K parameters as a key parameter subspace; a parameter recommendation module, configured to select the key parameter subspace according to the feature importance ranking table, generate a candidate configuration set C in the subspace, use the trained XGBoost model to predict the performance score S_ci of each candidate configuration in batches, and output a recommended configuration list sorted by predicted performance; a feedback closed-loop module, configured to start a monitoring window after applying the recommended configuration, calculate the actual tuning effect when the window ends, perform determination of effective retention, fine-tuning retention and degradation rollback according to the actual effect, and write the current tuning record into the historical tuning database; an incremental learning module, configured to perform incremental training on the model by adopting a WarmStart strategy or a full replacement strategy when the number of new samples or the time elapsed since the last update meets a trigger condition, evaluate MAE_new of the new model and MAE_old of the old model on the validation set, and replace and deploy the new model online if MAE_new < MAE_old × (1-ε).
[0018] Compared with existing technologies, the advantages of this invention are: strong versatility, not limited to specific systems or applications, users only need to define the tuning target and the range of adjustable parameters to use it; high level of intelligence, automatically learning the nonlinear relationship between parameters and performance based on the XGBoost algorithm, without the need for human experience; continuous evolution through the Warm Start incremental learning mechanism, the model accuracy continuously improves with usage time; feedback closed-loop mechanism ensures automatic rollback when tuning fails, guaranteeing system stability; XGBoost can output feature importance ranking, with good interpretability; narrowing the search space through key parameter identification, parameter recommendation latency is in the second range, suitable for online use in production environments, can automatically identify the mapping relationship between system parameters and performance by learning historical tuning data, automatically recommend and apply the optimal parameter configuration, and has incremental learning and effect feedback closed-loop capabilities, so that the tuning effect continuously improves with the increase of usage time. Attached Figure Description
[0019] Figure 1 This is the overall system architecture diagram of this invention.
[0020] Figure 2 This is a flowchart of the XGBoost model training process in this invention.
[0021] Figure 3 This is a flowchart of the parameter recommendation algorithm in this invention.
[0022] Figure 4 This is a flowchart of the incremental learning and feedback closed loop in this invention.
[0023] Figure 5 This is a flowchart of the method in this invention. Detailed Implementation
[0024] To enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.
[0025] This embodiment discloses an automated tuning system and method based on machine learning, comprising the following steps: Taking parameter tuning of a MySQL database system as an example, the complete workflow of this invention is illustrated. Assume the database is an OLTP application, and the tuning objective is to minimize the average query response time. There are 20 adjustable parameters and 10 collected performance metrics. Figure 1 This is the overall architecture diagram of the automated tuning system based on machine learning of this invention, showing the relationship and data flow between the four core modules: data acquisition, model training, parameter recommendation, and feedback loop. Figure 2This is a flowchart of the XGBoost model training process, showing the complete workflow from historical data extraction, feature preprocessing, model training, validation and evaluation to feature importance ranking. Figure 3 This is a flowchart of the parameter recommendation algorithm, illustrating the decision-making process of key parameter identification → parameter space search → batch prediction → ranking output. Figure 4 It is a flowchart of incremental learning and feedback closed loop, which shows the complete closed loop process of optimization effect evaluation, rollback determination, data feedback, Warm Start incremental training, and model validation and replacement.
[0026] like Figure 5 As shown, the method flowchart of the present invention includes the following steps: The offline model building phase includes: extracting samples from the historical tuning database, concatenating the running indicators F and parameter configuration P in the samples as input features, using performance scores S as labels, training the XGBoost regression model, evaluating the mean absolute error MAE and the coefficient of determination R² on the validation set, and when MAE is less than a preset threshold θ and R² is greater than a preset threshold, the model passes the validation and outputs a feature importance ranking table. The online parameter recommendation stage includes: obtaining the current system operating metric F_cur and parameter configuration P_cur; selecting the top K parameters by importance as the key parameter subspace based on the feature importance ranking table, while keeping the current values of the remaining non-key parameters unchanged; generating candidate values for each key parameter in the key parameter subspace according to its type; generating a candidate configuration set C by Cartesian product; concatenating the current system operating metric F_cur with the candidate configuration P_ci to obtain X_ci = [F_cur| P_ci]; using the trained XGBoost model to predict the performance score S_ci of each candidate configuration in batches; and outputting a recommended configuration list based on the predicted performance. The feedback loop mechanism includes: after applying the recommended configuration, start monitoring the effect; at the end of the monitoring window, calculate the actual optimization effect and judge and roll back it; when the actual optimization effect is excellent, keep the current configuration; otherwise, make fine adjustments or automatically roll back to the configuration P_old before optimization; generate optimization records and write them to the historical optimization database; and set a differentiated weight 'a' for the failed optimization samples, where 'a' is lower than the weight of normal samples.
[0027] In a specific embodiment, the user registers the target MySQL database in the system, defines a list of adjustable parameters and their value ranges, sets the optimization goal to minimize avg_query_latency, and sets the sampling frequency to 60 seconds.
[0028] In a specific embodiment, the system periodically collects MySQL running metrics and current parameter configuration values through a collection probe, and stores them together with the corresponding performance scores as samples in a historical tuning database. After accumulating to the initial sample size threshold, model training begins.
[0029] In a specific embodiment, a parameter space definition table and a tuning record table are also included. The parameter space definition table records the name, type, value range, and default value of each adjustable parameter. The tuning record table records complete information for each tuning operation, including tuning time, parameter configuration before tuning P_old, parameter configuration after tuning P_new, performance before tuning S_old, performance after tuning S_new, and tuning effect E, where tuning effect E = S_old. S_new. Taking the minimization target as an example, E>0 indicates improvement; for example: the parameter "buffer_pool_size" is of continuous type, with a value range of [128MB, 16384MB], and a default value of 128MB.
[0030] In a specific embodiment, step S1, the historical tuning database, stores all historical tuning samples. Each tuning sample is a triple, comprising F, P, and S; where F is a system performance indicator feature vector, P is the current parameter configuration vector, and S is the corresponding performance score. For example, F = {CPU utilization, memory utilization, disk I / O, network throughput, concurrent connections, ...}, P = {value of parameter 1, value of parameter 2, ..., value of parameter n}, and S is the target performance value (such as response time) under this configuration.
[0031] In this specific embodiment, the model is trained for the first time. 1000 samples are extracted, and 10 performance metrics and 20 parameter configurations are concatenated into a 30-dimensional input feature, with `avg_query_latency` as the label. The training and validation sets are divided in an 8:2 ratio to train the XGBoost regression model. After validation, the MAE is 2.3ms and R² is 0.91, indicating the model has passed validation and is ready for deployment.
[0032] In a specific embodiment, the XGBoost model training first acquires samples from the historical tuning database. The performance metric F and parameter configuration P from these samples are concatenated as input features, and the performance score is used as the label. The sample features are normalized, One-Hot encoded, and missing values are filled. The training and validation sets are then divided, and the XGBoost model is trained. Its principle is a gradient boosting tree, with built-in L1 / L2 regularization constraints on the weights and number of leaf nodes. The MAE and R² are evaluated on the validation set. If the MAE is less than a preset threshold θ (which is configurable, set to 5% of the target performance metric), and R² is greater than a preset threshold (also configurable, set to 0.85), the model passes validation and enters the deployment phase. Otherwise, the hyperparameters are adjusted and retrained. The trained model outputs a feature importance ranking table, marking the top K most important parameters as key parameters, where K = min(total number of parameters, 10). Subsequent parameter recommendations prioritize the subspace of these K key parameters.
[0033] In a specific implementation, feature importance analysis is performed. The model outputs a ranking of feature importance, marking the top K parameters as key parameters, and subsequently recommending priority searches within the subspaces of these key parameters.
[0034] In a specific embodiment, the parameter types include continuous uniform sampling, enumeration traversal, and Boolean enumeration; the prediction performance ranking is performed by sorting in ascending order when the objective is to minimize, and taking the top N configurations as the recommendation list, where N=5. The configuration set C is generated by Cartesian product, where the configuration set C={P_c1, P_c2, ..., P_cM}. The recommended configuration list is output, and the prediction performance improvement ΔS_i = S_cur– S_ci for each recommended configuration is recorded, where S_cur is the performance score corresponding to the current configuration, S_ci is the prediction performance score corresponding to the i-th candidate configuration P_ci, and the performance score S_ci = f(X_ci), X_ci= [F_cur | P_ci], is the input feature constructed for each candidate configuration P_ci.
[0035] In a specific embodiment, the parameter recommendation is to output a recommended configuration list when ΔS_i>0; otherwise, there is no need to change the current configuration.
[0036] In a specific implementation, the current system's avg_query_latency is 52ms. The system searches and performs batch predictions in subspaces of five key parameters. It is recommended to adjust the innodb_buffer_pool_size to 8GB and the max_connections to 300, etc., which can reduce the predicted avg_query_latency to 34ms, an expected improvement of 34.6%.
[0037] In a specific embodiment, the time of the monitoring window is set to 30 minutes, and at the end of the monitoring window, the actual tuning effect E_real = S_before is calculated S_after, where S_before is the actual performance score before configuration adjustment, and S_after is the actual performance score after configuration adjustment.
[0038] In a specific embodiment, parameter application and monitoring: after user confirmation, the system automatically applies the recommended configuration. A 30-minute monitoring window is started after the application. When the monitoring ends, the actual avg_query_latency is counted as 37ms, and the actual improvement E_real=52 37=15ms (28.8%), the tuning is valid, and the current configuration is maintained.
[0039] In a specific embodiment, the determination and rollback of the actual tuning effect include: When E_real>0: the tuning is valid, and the current configuration is maintained When -Δ_threshold ≤ E_real ≤ 0, the effect is not obvious but no degradation occurs, the current configuration is maintained and marked as "fine-tuning"; E_real< Δ_threshold: the performance degradation exceeds the threshold, wherein the threshold Δ_threshold = 10% of the target indicator, an automatic rollback is triggered to restore the pre-tuning configuration P_old; The complete record of this tuning is written into the historical tuning database; for samples with failed tuning, that is, E_real<0, the sample weight is set to α in the subsequent incremental training, where α=0.5, which is lower than the normal sample weight of 1.0, so that the model reduces the recommendation tendency for similar parameter configurations.
[0040] In a specific embodiment, feedback recording: the system writes the complete record of this tuning (configurations before and after tuning, performance data, actual effect) into the historical tuning database In a specific embodiment, an incremental learning mechanism is further included; when the number of newly added samples in the historical tuning database or the time elapsed since the last model update meets the trigger condition, the Warm Start strategy or the full replacement strategy is adopted to perform incremental training on the model, the MAE of the old model and the new model are respectively evaluated on the verification set, and if MAE_new<MAE_old × (1-ε), the new model is replaced and launched, otherwise the old model is retained, where ε=0.02.
[0041] In a specific embodiment, the incremental learning mechanism includes: When the number of new samples added after the last model update ≥ N_incr or the time elapsed since the last model update ≥ T_incr, set D_incr=200 and T_incr=7 days, then incremental training is triggered. In S402, when the number of trees has not reached the upper limit, that is, T_cur<T_max, set T_max=500, adopt the Warm Start strategy, load all parameters of the current online model, including the T_cur trained decision trees, append the new samples to the training set, continue to train an additional T_incr decision trees starting from the T_cur+1-th round, and set T_add=50. This approach preserves the knowledge already learned by the model, only needs to learn the incremental information in the new data, and has low computational overhead.
[0042] When the number of trees reaches the upper limit (T_cur ≥ T_max): adopt a full replacement training strategy, use all historical data including new samples to train a new model with T_max trees from scratch to replace the old model, and the number of trees will not continue to grow at this time. This method ensures that the model scale is always controllable and the inference latency is stable. At the same time, since more historical data is accumulated, the newly trained model is superior to the old model in terms of data sufficiency, which can make up for the information loss caused by discarding the old trees.
[0043] Evaluate the MAE of the old model and the new model on the validation set respectively. If MAE_new<MAE_old × (1 ε), the online model is replaced with the new model; otherwise, the old model is retained, where ε is set to 0.02, MAE_new is the mean absolute error of the new model on the same validation set, and MAE_old is the mean absolute error of the old model on the same validation set.
[0044] In a specific embodiment, incremental learning; the system continues to run, and 230 sets of tuning samples are accumulated after 7 days, which meets the incremental training trigger condition (≥200 sets). Execute Warm Start incremental training: load the current model (200 trees), and continue to train 50 new trees. The MAE of the new model on the validation set is 1.8ms, and the MAE of the old model is 2.3ms, which is reduced by 21.7% (exceeding the 2% threshold), and the new model is replaced and launched online.
[0045] In a specific embodiment, rollback scenario; after the system recommends a certain parameter adjustment, it is monitored that the performance degradation exceeds the rollback threshold, the system automatically rolls back to the pre-tuning configuration, and writes the failed sample into the historical data with a lower weight, so that subsequent models will reduce the recommendation tendency for similar configurations.
[0046] In a specific embodiment, a certain proportion of early historical samples are retained in the validation set in the incremental learning mechanism to prevent catastrophic forgetting, and the proportion is set to twenty percent.
[0047] In a specific embodiment, the present invention further provides an automated tuning system based on machine learning, comprising: a data acquisition module, configured to periodically collect a system operation indicator F_cur and a parameter configuration P_cur vector, and form a sample in combination with a corresponding performance score; a historical tuning database, configured to store all historical tuning samples, where each sample is a triplet (F, P, S), wherein F is a system operation indicator feature vector, P is a parameter configuration vector, and S is a performance score; a model training module, configured to train an XGBoost regression model, and output a feature importance ranking table after verification is passed, marking the first K parameters as a key parameter subspace; a parameter recommendation module, configured to select the key parameter subspace according to the feature importance ranking table, generate a candidate configuration set C in the subspace, use the trained XGBoost model to batch predict the performance score S_ci of each candidate configuration, and output a recommended configuration list sorted by predicted performance; a feedback closed-loop module, configured to start a monitoring window after the recommended configuration is applied, calculate the actual tuning effect when the window ends, perform determination of effective retention, fine-tuning retention, and degradation rollback according to the actual effect, and write the current tuning record into the historical tuning database; an incremental learning module, configured to perform incremental training on the model by adopting a WarmStart strategy or a full replacement strategy when the number of new samples or the time elapsed since the last update meets a trigger condition, evaluate the MAE_new of the new model and the MAE_old of the old model on a verification set, and replace the online model if MAE_new < MAE_old × (1-ε).
[0048] Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer usable program code embodied therein. The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus produce means for implementing the flow Figure 1One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0049] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A machine learning based automated tuning method, characterized in that, include: The offline model building phase, the online parameter recommendation phase, and the feedback closed-loop mechanism include: The offline model building phase includes: extracting samples from the historical tuning database, concatenating the running indicators F and parameter configuration P in the samples as input features, using performance scores S as labels, training the XGBoost regression model, evaluating the mean absolute error MAE and the coefficient of determination R² on the validation set, and when MAE is less than a preset threshold θ and R² is greater than a preset threshold, the model passes the validation and outputs a feature importance ranking table. The online parameter recommendation stage includes: obtaining the current system operating metric F_cur and parameter configuration P_cur; selecting the top K parameters by importance as the key parameter subspace based on the feature importance ranking table, while keeping the current values of the remaining non-key parameters unchanged; generating candidate values for each key parameter in the key parameter subspace according to its type; generating a candidate configuration set C by Cartesian product; concatenating the current system operating metric F_cur with the candidate configuration P_ci to obtain X_ci = [F_cur | P_ci]; using the trained XGBoost model to predict the performance score S_ci of each candidate configuration in batches; and outputting a recommended configuration list based on the predicted performance. The feedback loop mechanism includes: after applying the recommended configuration, start monitoring the effect; at the end of the monitoring window, calculate the actual optimization effect and judge and roll back it; when the actual optimization effect is excellent, keep the current configuration; otherwise, make fine adjustments or automatically roll back to the configuration P_old before optimization; generate optimization records and write them to the historical optimization database; and set a differentiated weight 'a' for the failed optimization samples, where 'a' is lower than the weight of normal samples.
2. The automated tuning method based on machine learning according to claim 1, characterized in that, Also included are a parameter space definition table, which records the name, type, value range and default value of each adjustable parameter, and a tuning record table, which records complete information of each tuning operation, including tuning time, parameter configuration before tuning P_old, parameter configuration after tuning P_new, performance before tuning S_old, performance after tuning S_new, and tuning effect E, where tuning effect E=S_old S_new.
3. The method of claim 1, wherein, The historical tuning database stores all historical tuning samples. Each tuning sample is a triplet, which includes F, P, and S; where F is the system operation indicator feature vector, P is the current parameter configuration vector, and S is the corresponding performance score.
4. The method of claim 1, wherein, The XGBoost model training first acquires samples from the historical tuning database. The performance metric F and parameter configuration P from these samples are concatenated as input features, and the performance score is used as the label. The sample features are normalized, One-Hot encoded, and missing values are filled. The training and validation sets are then divided, and the XGBoost model is trained. Its principle is a gradient boosting tree, with built-in L1 / L2 regularization constraints on the weights and number of leaf nodes. The MAE and R² are evaluated on the validation set. If the MAE is less than a preset threshold θ (which is configurable, set to 5% of the target performance metric), and R² is greater than a preset threshold (also configurable, set to 0.85), the model passes validation and enters the deployment phase. Otherwise, the hyperparameters are adjusted and retrained. The trained model outputs a feature importance ranking table, marking the top K most important parameters as key parameters, where K = min(total number of parameters, 10).
5. The method of claim 1, wherein, The parameter types include continuous uniform sampling, enumeration traversal, and Boolean enumeration; the prediction performance ranking is performed in ascending order when the target is minimization, and the top N ranked configurations are taken as the recommendation list, where N=5. The configuration set C is generated by Cartesian product, where the configuration set C={P_c1, P_c2, ..., P_cM}, the recommended configuration list is output, and the predicted performance improvement margin of each recommended configuration is recorded as ΔS_i = S_cur– S_ci, where S_cur is the performance score corresponding to the current configuration, S_ci is the predicted performance score corresponding to the i-th candidate configuration P_ci, the performance score S_ci = f(X_ci), and X_ci= [F_cur |P_ci] is the input feature constructed for each candidate configuration P_ci.
6. The method of claim 1, wherein, The determination and rollback of the actual tuning effect include: When E_real>0: the tuning is effective, and the current configuration is maintained When -Δ_threshold ≤ E_real ≤ 0, the effect is not obvious but no degradation occurs, the current configuration is maintained and marked as "fine-tuning"; When E_real < 0 Δ_threshold: performance degradation exceeds threshold, where threshold Δ_threshold = 10% of target metric, triggers automatic rollback to pre-tuning configuration P_old. The complete record of this tuning is written into the historical tuning database; for samples with tuning failure, that is, E_real<0, the sample weight is set to α in the subsequent incremental training, where α=0.5, which is lower than the normal sample weight of 1.0, so that the model reduces the recommendation tendency for similar parameter configurations.
7. The method of claim 1, wherein, It also includes an incremental learning mechanism; when the number of newly added samples in the historical tuning database or the time elapsed since the last model update meets the trigger condition, the Warm Start strategy or full replacement strategy will be adopted for incremental training of the model, the MAE of the old model and the new model will be evaluated on the verification set respectively. If MAE_new<MAE_old × (1-ε), the new model will be replaced and launched, otherwise the old model will be retained, where ε=0.
02.
8. The method of claim 7, wherein, The incremental learning mechanism includes: When the number of newly added samples after the last model update is not less than N_incr, or the time elapsed since the last model update is not less than D_incr, incremental training is triggered, where N_incr=200 and D_incr=7 days; When the number of decision trees T_cur of the current online model is less than T_max, the Warm Start strategy is adopted, all parameters of the current online model and the T_cur trained decision trees are loaded, new samples are appended to the training set, and T_add decision trees are continuously trained on the basis of the existing T_cur decision trees, where T_max=500, T_add=50, and the total number of decision trees after incremental training does not exceed T_max; When the number of decision trees T_cur of the current online model is greater than or equal to T_max, the full replacement training strategy is adopted, all historical data including newly added samples are used to train a new model containing T_max decision trees from scratch to replace the old model; The MAE of the old model and the new model are respectively evaluated on the validation set, if MAE_new < MAE_old x (1 ε), the new model replaces the online model; otherwise, the old model is retained, wherein ε = 0.02, wherein MAE_new is the average absolute error of the new model on the same validation set, and MAE_old is the average absolute error of the old model on the same validation set.
9. The method of claim 7, wherein, In the incremental learning mechanism, a certain proportion of early historical samples are retained in the verification set to prevent catastrophic forgetting, where the proportion is set to twenty percent.
10. An automated tuning system based on machine learning, characterized in that, comprises: a data acquisition module, configured to periodically collect a system operation indicator F_cur and a parameter configuration P_cur vector, and form a sample in combination with a corresponding performance score; a historical tuning database, configured to store all historical tuning samples, where each sample is a triplet (F, P, S), where F is a system operation indicator feature vector, P is a parameter configuration vector, and S is a performance score; a model training module, configured to train an XGBoost regression model, output a feature importance ranking table after verification is passed, and mark the first K parameters as a key parameter subspace; a parameter recommendation module, configured to select the key parameter subspace according to the feature importance ranking table, generate a candidate configuration set C in the subspace, use the trained XGBoost model to batch predict the performance score S_ci of each candidate configuration, and output a recommended configuration list according to the predicted performance ranking; a feedback closed-loop module, configured to start a monitoring window after the recommended configuration is applied, calculate an actual tuning effect when the window ends, perform judgment of effective retention, fine-tuning retention and degradation rollback according to the actual effect, and write the current tuning record into the historical tuning database; an incremental learning module, configured to perform incremental training on the model by using a WarmStart strategy or a full replacement strategy when the number of new samples or the time elapsed since the last update meets a trigger condition, evaluate MAE_new of the new model and MAE_old of the old model on a verification set, and replace the model online if MAE_new < MAE_old × (1-ε).
