A performance prediction method for configurable software systems based on active learning
By constructing a heterogeneous selector that combines novelty and uncertainty indices, the performance prediction of configurable software systems is optimized, solving the problems of single learning models and stochasticity, and achieving more stable and efficient performance prediction results.
Patent Information
- Application Number
- CN202411175764.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-26
AI Technical Summary
In existing technologies, single learning models exhibit inconsistent performance in performance prediction tasks across different software systems. The randomness of the initial sampling configuration leads to unstable prediction results, and uncertainty as the objective function may fail to fully cover the global configuration space, resulting in insufficient prediction efficiency and accuracy.
A heterogeneous selector is constructed using multiple learning models. An objective function is constructed by combining novelty and uncertainty indices. Through uniform sampling and iterative learning, the configuration with the greatest novelty and uncertainty is selected and added to the training set to optimize the learning process. The heterogeneous selector is then used for performance prediction.
It improves the stability and accuracy of performance prediction for configurable software systems, reduces the impact of learning models and randomness on prediction results, and significantly improves prediction efficiency and accuracy.
Smart Images

Figure CN118689779B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of software system performance prediction, and in particular to a performance prediction method for a configurable software system based on active learning. Background Art
[0002] With the rapid development of informatization, the development scale and complexity of most modern software systems are increasing day by day, and they are highly configurable and can combine hundreds of configuration options, functions or plug-ins. End users, system administrators, software engineers and scientists can select these feature parameters to configure various software systems to meet their functional or non-functional requirements, such as execution time, output quality, security, energy consumption, etc. Although it is very meaningful to customize software in this way, the system complexity increases significantly due to the huge set of feasible configurations. For example, the Linux kernel has more than 15,000 features, most of which can have three values: "yes", "no" or "module". If some other constraints are not considered, there are at least 315,000 possible feature combinations. Due to the explosion of the number of combinations and the high cost of executing software, it is almost impossible to fully explore all software configurations.
[0003] To this end, a method for performance prediction using machine learning technology is proposed. Its basic idea is to learn from a small number of configuration samples that have been explored, and to predict the performance of the entire configuration space. The prediction of configurable software systems using machine learning methods is mainly divided into the steps of "sampling, measurement, learning, and evaluation". For the further introduced active learning method, its basic idea is to iteratively perform the two steps of "measurement" and "learning", so as to guide the measurement target of the next round according to the results of each round of learning until it reaches the preset measurement budget.
[0004] However, in the prior art, there are the following problems when using active learning methods to predict the performance of configurable software systems:
[0005] (1) Single learning model: Existing methods usually use a single learning model to predict performance. Different learning models have different performances in prediction tasks of different software systems, and no learning model can perform well in all tasks. Therefore, when faced with an unknown software system, it is impossible to determine which learning model is most suitable for a specific prediction task;
[0006] (2) Randomness of initial sampling configuration: Before the initial iteration of active learning, it is often necessary to randomly select the initial sampling configuration from all configurations. All subsequent iterations are based on these initial configurations. This randomness may lead to instability in the prediction results, and may produce completely different prediction accuracy in two identical prediction tasks;
[0007] (3) Uncertainty as an objective function: Uncertainty is often used as an objective function to guide the next round of sampling in the active learning process, hoping to find a configuration that can significantly improve the prediction accuracy. However, the use of uncertainty may result in sampling that fails to fully cover the global configuration space, requiring more iterations to achieve the desired prediction accuracy. Summary of the invention
[0008] In view of the problems existing in the prior art, the present invention provides a performance prediction method for a configurable software system based on active learning, which can achieve more stable, efficient and accurate performance prediction of the software system when only a relatively small number of configuration sample measurements are provided.
[0009] The technical solution of the present invention is achieved in this way:
[0010] A performance prediction method for a configurable software system based on active learning, wherein the configurable software system includes a plurality of feasible configurations, and the feasible configurations include feature options; the method includes the following steps:
[0011] T1. Preprocess the feature options; the preprocessing includes setting the value range of the feature options; the values of multiple feature options of a feasible configuration constitute a feature vector; the feasible configuration is a combination of all feature options that meet the constraints of the software system, which can be configured in the software system and work normally. Uniformly sample multiple feasible configurations according to a preset number to obtain a sampling configuration; measure the actual performance of the sampling configuration; construct a training set based on the sampling configuration and its actual performance; select several learning models and construct a heterogeneous selector; and use the training set to initialize the learning model.
[0012] T2. Calculate the novelty value and uncertainty value of each of the feasible configurations;
[0013] For each of the feasible configurations, the minimum Manhattan distance between the feature vector and the existing configurations in the training set is calculated as the novelty value; the larger the distance value, the higher the novelty. Novelty means that configurations with similar feature options to the configuration have not been tested, and adding them to the training set helps explore new configuration space, thereby improving prediction accuracy.
[0014] The heterogeneous selector predicts each of the feasible configurations to obtain a prediction result; the uncertainty value of each configuration is calculated based on the prediction result; wherein each of the learning models calculates a prediction value for each of the feasible configurations; for the same feasible configuration, the variance calculated based on the prediction values of several learning models is the uncertainty value; the larger the uncertainty value, the greater the uncertainty, indicating that the greater the divergence of a configuration in these learning models, and adding it to the training set is more likely to improve the local prediction accuracy.
[0015] Select feasible configurations with the largest novelty value and uncertainty value to add to the training set; use the training set to retrain the heterogeneous selector; the training process is the process of active learning of the learning model.
[0016] T3. Repeat step T2 according to the preset number of iterations.
[0017] T4. Select any one of the learning models in the heterogeneous selector as the performance prediction model; select a learning model from the trained heterogeneous selector as the trained performance prediction model, so as to achieve the effect of mutual feedback between learning and sampling. Use the performance prediction model to perform performance prediction on all the feasible configurations to obtain the final prediction result.
[0018] As a further optimization of the above scheme, the feature options include binary features and numerical features; the value of the binary feature is 0 or 1; the value of the numerical feature is an integer in a set of integers; and the preprocessing includes mapping the integer to a decimal between 0 and 1.
[0019] The mapping is to unify the value ranges of binary features and numerical features so that they have the same weight in subsequent uncertainty calculations and novelty calculations. For example, if the values of a numerical feature are: -8, -4, 0, 4, 8, the corresponding mapping is 0, 0.25, 0.5, 0.75, 1.
[0020] As a further optimization of the above scheme, the uniform sampling is that, for multiple feasible configurations, the Manhattan distances between them are calculated according to the feature vectors, and multiple feasible configurations with the largest average Manhattan distances are selected as sampling configurations according to a preset number.
[0021] As a further optimization of the above scheme, the learning model is a random forest, a decision tree, a gradient boosting or an extreme gradient boosting tree, etc.
[0022] As a further optimization of the above scheme, in T2, for all feasible configurations, weighted calculation is performed based on the novelty value and the uncertainty value to obtain a unique value and then sort them, and finally the feasible configuration with the largest novelty value and uncertainty value is screened out.
[0023] As a further optimization of the above scheme, the feasible configurations are sorted in descending order according to the novelty value and the uncertainty value, and each of the feasible configurations obtains a first rank and a second rank; the first rank and the second rank are weightedly calculated according to preset weights to obtain the unique value, and then sorted according to the unique value, and finally the feasible configuration with the smallest unique value has the largest novelty value and uncertainty value.
[0024] Compared with the prior art, the present invention achieves the following beneficial effects:
[0025] (1) The present invention introduces a heterogeneous selector constructed with multiple learning models and applies it to the iteration of active learning, which effectively improves the applicability and prediction effect in the performance prediction tasks of most configurable software systems;
[0026] (2) The present invention proposes an active learning method based on uncertainty index and novelty index. It expands on the existing performance prediction of configurable software systems using active learning methods and constructs a new objective function based on novelty value and uncertainty value, so that the sampling process can explore the global / novel and uncertain space at the same time, and reduce the impact of learning models and randomness on the stability of prediction results, thereby significantly improving the efficiency and accuracy of performance prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a flowchart of a performance prediction method of a configurable software system based on active learning provided by an embodiment of the present invention;
[0028] Figure 2 is a schematic diagram of the process of initializing a heterogeneous selector provided by an embodiment of the present invention;
[0029] Figure 3 is a schematic diagram of a process of updating a heterogeneous selector provided by an embodiment of the present invention;
[0030] Figure 4 It is a schematic diagram of a process for performing final performance prediction provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] like Figures 1 to 4 As shown, this embodiment provides a performance prediction method for a configurable software system based on active learning, the configurable software system includes multiple feasible configurations, and the feasible configurations include feature options; the method includes the following steps:
[0033] T1. Preprocess the feature options; preprocessing includes setting the value range of the feature options; the values of multiple feature options of a feasible configuration constitute a feature vector; the feasible configuration is a combination of all feature options that meet the constraints of the software system, which can be configured in the software system and work normally.
[0034] In this embodiment, the feature options include binary features and numerical features; the value of the binary feature is 0 or 1; the value of the numerical feature is an integer in a set of integers; and the preprocessing includes mapping the integer to a decimal between 0 and 1.
[0035] The mapping is to unify the value ranges of binary features and numerical features so that they have the same weight in subsequent uncertainty calculations and novelty calculations. For example, if the values of a numerical feature are: -8, -4, 0, 4, 8, the corresponding mapping is 0, 0.25, 0.5, 0.75, 1.
[0036] Uniformly sample multiple feasible configurations according to a preset number to obtain a sampling configuration; measure the actual performance of the sampling configuration; construct an initial training set based on the sampling configuration and its actual performance;
[0037] In this embodiment, uniform sampling is to calculate the Manhattan distances between multiple feasible configurations according to the feature vectors, and select multiple feasible configurations with the largest average Manhattan distance as sampling configurations according to a preset number.
[0038] For example, for the three existing configurations, the corresponding eigenvectors are (0, 1, 1, 0), (1, 0, 0, 1) and (1, 1, 0, 0), then the Manhattan distances between them are , and , then the average Manhattan distance of the three configurations is .
[0039] The resulting uniform training set helps improve the efficiency of the subsequent active learning process while reducing the randomness of the algorithm and the instability of the prediction results.
[0040] Select several learning models and construct a heterogeneous selector; use the training set to initialize the learning model, that is, initialize the heterogeneous selector.
[0041] In this embodiment, the construction selector includes three learning models: random forest, decision tree, and gradient boosting.
[0042] T2. Calculate the novelty value and uncertainty value of each feasible configuration;
[0043] For each feasible configuration, the minimum Manhattan distance with the existing configurations in the training set is calculated based on the feature vector, which is the novelty value; the larger the distance value, the higher the novelty.
[0044] Novelty means that configurations with similar feature options have not been tested. Adding it to the training set helps explore new configuration space and thus improve prediction accuracy.
[0045] The heterogeneous selector predicts each feasible configuration and obtains the prediction result; the uncertainty value of each configuration is calculated based on the prediction result; each learning model calculates a prediction value for each feasible configuration; for the same feasible configuration, the variance calculated based on the prediction values of several learning models is the uncertainty value;
[0046] The larger the uncertainty value, the greater the uncertainty, indicating that there is more disagreement about a configuration among these learning models, and adding it to the training set is more likely to improve local prediction accuracy.
[0047] For example, the heterogeneous selector includes three learning models. The predicted values for a certain configuration are 5, 5, and 8. The average predicted value of the configuration is (5+5+8) / 3=6, and the uncertainty index is .
[0048] Select the feasible configuration with the largest novelty value and uncertainty value to add to the training set;
[0049] Specifically, for all feasible configurations, weighted calculations are performed based on the novelty value and uncertainty value to obtain unique values and then sort them. Finally, the feasible configuration with the largest novelty value and the largest uncertainty value is screened out, that is, the most novel and most uncertain feasible configuration.
[0050] In this embodiment, the feasible configurations are sorted in descending order according to the novelty value and the uncertainty value, and each feasible configuration is given a first rank and a second rank;
[0051] The first and second rankings are weighted at a ratio of 1 to 1 to obtain a unique value, and then sorted according to the unique value. Finally, the feasible configuration with the smallest unique value has the largest novelty value and uncertainty value.
[0052] For example, there are 4 configurations whose uncertainty values are 2, 6, 7, and 4, and they are ranked 4, 2, 1, and 3 in descending order. Their novelty values are 5, 2, 4, and 1, and they are ranked 1, 3, 2, and 4 in descending order. The ranks of the two indicators are weighted at a ratio of 1:1 to obtain (4+1), (2+3), (1+2), and (3+4). The configuration with the smallest weighted rank is selected as the most uncertain and novel configuration, that is, configuration 3.
[0053] Retrain and update the heterogeneous selector using the training set;
[0054] T3, repeat step T2 according to the preset number of iterations;
[0055] T4. Select any learning model from the heterogeneous selector as the performance prediction model. This embodiment uses random forest. Select a learning model from the trained heterogeneous selector as the trained performance prediction model, so as to achieve the effect of mutual feedback between learning and sampling. Use the performance prediction model to predict the performance of all feasible configurations and obtain the final prediction result.
[0056] According to the disclosure and teaching of the above description, those skilled in the art to which the present invention belongs may also make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the scope of protection of the claims of the present invention. In addition, although some specific terms are used in this specification, these terms are only for the convenience of description and do not constitute any limitation to the present invention.
Claims
1. A performance prediction method for a configurable software system based on active learning, wherein the configurable software system includes a plurality of feasible configurations, and the feasible configurations include feature options; Features: The method comprises the following steps: T1. Preprocessing the feature options; the preprocessing includes setting the value range of the feature options; the values of multiple feature options of a feasible configuration constitute a feature vector; uniformly sampling multiple feasible configurations according to a preset number to obtain a sampling configuration; measuring the actual performance of the sampling configuration; constructing a training set based on the sampling configuration and its actual performance; selecting several different learning models to construct a heterogeneous selector; initializing the learning model using the training set; T2. Calculate the novelty value and uncertainty value of each of the feasible configurations; For each of the feasible configurations, calculating the minimum Manhattan distance with the existing configurations in the training set according to the feature vector, which is the novelty value; Each of the learning models calculates a prediction value for each of the feasible configurations; for the same feasible configuration, the variance calculated based on the prediction values of several learning models is the uncertainty value; The feasible configurations are sorted in descending order according to the novelty value and the uncertainty value, and each of the feasible configurations obtains a first rank and a second rank; the first rank and the second rank are weightedly calculated according to preset weights to obtain unique values, and then sorted according to the unique values, and finally the feasible configuration with the smallest unique value has the largest novelty value and uncertainty value; Select feasible configurations with the largest novelty value and uncertainty value to add to the training set; retrain the heterogeneous selector using the training set; T3, repeat step T2 according to the preset number of iterations; T4. Select any one of the learning models in the heterogeneous selector as a performance prediction model; use the performance prediction model to perform performance prediction on all the feasible configurations to obtain a final prediction result.
2. The performance prediction method of a configurable software system based on active learning according to claim 1, It is characterized in that The feature options include binary features and numerical features; The value of the binary feature is 0 or 1; the value of the numerical feature is an integer in a set of integers; and the preprocessing includes mapping the integer to a decimal between 0 and 1.
3. The performance prediction method of a configurable software system based on active learning according to claim 1, It is characterized in that The uniform sampling is to calculate the Manhattan distances between the multiple feasible configurations according to the feature vectors, and select multiple feasible configurations with the largest average Manhattan distance as sampling configurations according to a preset number.
4. The performance prediction method of a configurable software system based on active learning according to claim 1, It is characterized in that The learning model is a random forest, a decision tree, a gradient boosting, or an extreme gradient boosting tree.
