A performance change point classification method based on multi-dimensional features, medium and equipment

Abstract

The application discloses a kind of based on the classification method of performance change point of multi-dimensional feature, medium and equipment.The method of the application is by establishing the multi-dimensional feature model of software version performance change point, and using machine learning method is classified to performance change point.The application describes performance change point from four-dimensional features, including environmental configuration dimension, time series dimension, version dimension and content dimension.This method has the characteristics of high precision, strong explainability, etc., can effectively distinguish the change point in performance test, greatly improves the ability of identifying performance defects.Different from existing performance change point classification method, the classification method of performance change point proposed by the present application has better performance, can effectively identify the real performance change point, reduce the repair cost of performance defects.

Description

Technical Field

[0001] This invention relates to the field of software performance testing, and specifically to a method, medium, and device for classifying performance change points based on multidimensional features. Background Technology

[0002] Continuous integration systems (CIS) enable performance testing infrastructures to automate performance testing for each release. As code changes are made to the repository, the CIS runs numerous benchmark tests, each producing one or more scalar values ​​as a measure of its performance. Previously, developers often relied on manually observing time-series graphs to identify points of change. Threshold-based methods later replaced this manual process, but these methods depended on expert subjective experience. Currently, based on extensive performance test results and accumulated historical data, methods exist that automate the detection of performance change points using statistical and machine learning approaches.

[0003] However, a major problem with automatic change point detection technology is the high rate of false alarms; only a small percentage of the changes detected by this technology are actually recognized by developers. On one hand, due to inherent errors in performance metric measurement, performance test results often do not reflect the "true" performance. Therefore, the process of evaluating performance changes inevitably introduces bias, leading to false alarms. On the other hand, to ensure project schedules, some "unimportant" performance changes may be overlooked by developers. Although these "unimportant" performance changes have no negative impact on system usability, fixing them can be costly. Therefore, providing a method and system for classifying performance change points, and further identifying the truly necessary performance changes from the candidate change points detected by automatic change point detection technology, has become a pressing need. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, a classification method for performance change points based on multi-dimensional features is proposed. This invention uses a machine learning model to solve the classification problem of performance change points in performance testing, effectively identifying performance changes that need to be addressed. The specific technical solution adopted by this invention is as follows:

[0005] In a first aspect, the present invention provides a method for classifying performance change points based on multidimensional features, which includes the following steps:

[0006] S1: The performance test results obtained from different versions of the software under different performance test environment configurations are used to form a performance test result sequence. The performance test result sequence is further divided into result subsequences based on the performance test environment configuration. Change point detection technology is used to detect each result subsequence, and all versions with performance changes are detected and a candidate performance change point set is formed. Each candidate performance change point corresponds to a software version and a performance test environment configuration.

[0007] S2: For each candidate performance change point in the candidate performance change point set, extract features from the environment configuration dimension, time series dimension, version dimension, and content dimension respectively;

[0008] S3: For each candidate performance change point in the candidate performance change point set, the four-dimensional features are encoded into a feature vector as input to the training sample. At the same time, the feature vector is labeled to indicate whether it belongs to the performance change point that needs to be repaired, thereby constructing the training sample data. Then, the obtained training sample data is used to train the random forest model to obtain the performance change point classification model.

[0009] S4: For candidate performance change points to be classified, extract features from the environment configuration dimension, time series dimension, version dimension, and content dimension. Then, encode the features of the four dimensions into feature vectors and input them into the performance change point classification model trained in S3, thereby classifying whether the candidate performance change point belongs to the performance change point that needs to be repaired.

[0010] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the classification method for performance change points based on multidimensional features as described in any of the solutions of the first aspect above.

[0011] Thirdly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0012] The memory is used to store computer programs;

[0013] The processor is configured to, when executing the computer program, implement the classification method for performance change points based on multidimensional features as described in any of the solutions of the first aspect above.

[0014] This invention classifies performance change points by establishing a multi-dimensional feature model of these points and using machine learning methods to learn their features. This invention has the following beneficial effects:

[0015] 1. This invention utilizes multi-dimensional features to describe performance change points, thereby improving the accuracy of identifying performance change points;

[0016] 2. This invention uses a machine learning model to construct a classification model for performance change points, making the classification results of performance change points more interpretable. Attached Figure Description

[0017] Figure 1 This is a flowchart of the performance change point classification method based on multidimensional features in an embodiment of the present invention;

[0018] Figure 2 This is a block diagram of a classification system for performance change points based on multidimensional features in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings.

[0020] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide the public with a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0021] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0022] like Figure 1 As shown, in a preferred embodiment of the present invention, a method for classifying performance change points based on multidimensional features is provided, which includes the following steps:

[0023] S1: The performance test results obtained from different versions of the software under different performance test environment configurations are used to form a performance test result sequence. The performance test result sequence is divided into result subsequences based on the performance test environment configuration. Change point detection technology is used to detect each result subsequence, and all versions with performance changes are detected and constitute a candidate performance change point set. Each candidate performance change point corresponds to a software version and a performance test environment configuration.

[0024] In an embodiment of the present invention, the change point detection technology used in step S1 above is the E-Divisivemeans detection algorithm.

[0025] S2: For each candidate performance change point in the candidate performance change point set, extract features from the environment configuration dimension, time series dimension, version dimension, and content dimension respectively.

[0026] In an embodiment of the present invention, the method for extracting the four dimensions of features—environmental configuration, time series, version, and content—for each candidate performance change point in step S2 is as follows:

[0027] S21: Based on the performance test environment configuration corresponding to the current candidate performance change point, obtain the features corresponding to the environment configuration dimension, which includes 6 sub-feature dimensions, where: the first environment sub-feature is the project performing the performance test, the second environment sub-feature is E #thread The first environment sub-feature indicates the number of threads used for performance testing. The second environment sub-feature indicates the method for measuring performance metrics. The third environment sub-feature indicates the name of the performance test task. The fourth environment sub-feature indicates the name of the performance test. The fifth environment sub-feature indicates the name of the performance test. The sixth environment sub-feature indicates the construction parameters of the performance test task.

[0028] S22: The result subsequence containing the current candidate performance change point is taken as a time series. The time series feature extraction method is used to obtain the features of the time series dimension, which includes 8 sub-feature dimensions, where: the first time sub-feature is the number of times the time series crosses the midline, the second time sub-feature is the Shannon entropy of the time series spectral density, the third time sub-feature is the variance of the sub-time series, the fourth time sub-feature is the maximum average offset between two consecutive sliding windows of the time series, the fifth time sub-feature is the average of the minimum and maximum normalized values ​​of the time series, the sixth time sub-feature is the standard deviation of the minimum and maximum normalized values ​​of the time series, the seventh time sub-feature is the autocorrelation of the time series, and the eighth time sub-feature is the variance of the mean of the sub-time series; the above sub-time series are the sequence segments from the current candidate performance change point to the previous candidate performance change point in the above time series.

[0029] S23: By analyzing the detailed information in the version logs of the current candidate performance change points, the features of the version dimension are obtained, which include three sub-feature dimensions: the first version sub-feature is the change in the performance metric value of the current candidate performance change point, the second version sub-feature is the probability of the current version being changed, and the third version sub-feature is the correlation between the text content in the messages of the current version and the performance change.

[0030] In an embodiment of the present invention, the three sub-feature dimensions of the version dimension feature in step S23 above are as follows:

[0031] S231: The first version sub-feature mentioned above consists of two feature values. The first feature value is the performance change ratio of the current candidate performance change point relative to the previous version. The second feature value is the result of the performance metric of the current candidate performance change point after z-score standardization.

[0032] S232: The above second version sub-feature consists of four feature values. The first feature value indicates whether the previous version of the current candidate performance change point in the result subsequence is a candidate performance change point. The second feature value is the number of versions of the current candidate performance change point in the result subsequence that are far from the previous candidate performance change point. The third feature value is the number of versions in the result subsequence that are before the current candidate performance change point. The fourth feature value is the proportion of the number of candidate performance change points in the result subsequence that are before the current candidate performance change point to the total number of versions.

[0033] S233: The above third version sub-feature consists of four feature values, which are the classification probabilities output by the four version classification models. The four version classification models are the trained Gaussian Naive Bayes model, complementary Naive Bayes model, multinomial Naive Bayes model, and Bernoulli Naive Bayes model. The message text content in the version log of the current candidate performance change point is segmented and vectorized as the input of each model. The output of each model is the classification probability of whether there is a performance change in the version.

[0034] In one embodiment of the present invention, words in the message text content can be vectorized using the TF-ID method.

[0035] S24: By analyzing the set of modified files corresponding to the current candidate performance change points. The features of the content dimension are obtained, which include three sub-feature dimensions: the first content sub-feature is the confidence of the association rule between the version sequence and the modified file; the second content sub-feature is the scale of the modified content in the modified file; and the third content sub-feature is the evolutionary feature of the modified content in the modified file.

[0036] In an embodiment of the present invention, the three sub-feature dimensions of the content dimension in step S24 above are as follows:

[0037] S241: The first content sub-feature mentioned above is the aforementioned set of modified files. The sum of the relevance scores between each modified file and the resulting subsequence containing the current candidate performance change point; the relevance score between any modified file and the resulting subsequence is the ratio of the modulus of the first intersection to the modulus of the second intersection, the first intersection being the version intersection of the subset and the set of candidate performance change points in the resulting subsequence, the second intersection being the version intersection of the subset and the resulting subsequence, and the aforementioned subset being the set of versions of the currently modified file in the resulting subsequence that have been modified;

[0038] S242: The second content sub-feature mentioned above consists of six feature values, the first of which is the set of modified files mentioned above. The number of modified files in the set, and the second characteristic value is the set of modified files mentioned above. The number of source code files, and the third characteristic value is the aforementioned set of modified files. The number of configuration files, and the fourth characteristic value is the set of modified files mentioned above. The number of resource files, the fifth feature value is the number of text lines added to the modified file, and the sixth feature value is the number of text lines deleted from the modified file;

[0039] S243: The third content sub-feature mentioned above consists of three feature values. The first feature value is the value of the modified file set up to the current candidate performance change point. The total number of developers who contributed to each file, and the second characteristic is the set of modified files mentioned above. The frequency of changes to each file in the dataset, and the third characteristic value is the set of modified files mentioned above. The age of each document in the document.

[0040] S3: For each candidate performance change point in the candidate performance change point set, the four-dimensional features are encoded into a feature vector as input to the training sample. At the same time, the feature vector is labeled to indicate whether it belongs to the performance change point that needs to be repaired, thereby constructing training sample data. Then, the obtained training sample data is used to train a random forest model to obtain a performance change point classification model.

[0041] It should be noted that the Random Forest in this invention is a classic Bagging model, and its specific principles and model structure belong to existing technology. The Random Forest model randomly samples from the original dataset to form n different sample datasets, then builds n different decision tree models based on these datasets, and finally obtains the final result based on the average of these decision tree models (for regression models) or the voting results (for classification models).

[0042] In addition, in the embodiments of the present invention, in step S3 above, when labeling candidate performance change points, they are divided into positive samples and negative samples. Positive samples are performance change points that are confirmed to need to be repaired, and negative samples are false positive performance change points that do not need to be repaired. Positive samples and negative samples can be labeled with corresponding tags to serve as the true values ​​during model training.

[0043] During the training process of the random forest model, each training sample uses the encoded vectors of four features as model input, and the optimization of model parameters is supervised using pre-labeled ground truth values, thereby optimizing the model parameters. The specific training process of the random forest model is prior art and will not be described in detail here. In the embodiment of the present invention, in step S3 above, a grid search method is used to obtain the optimal parameters of the model when training the random forest model.

[0044] S4: For candidate performance change points to be classified, extract features from the environment configuration dimension, time series dimension, version dimension, and content dimension. Then, encode the features of the four dimensions into feature vectors and input them into the performance change point classification model trained in S3, thereby classifying whether the candidate performance change point belongs to the performance change point that needs to be repaired.

[0045] It should be noted that the method steps S1 to S4 described above can essentially be implemented in the form of a computer program functional model.

[0046] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a classification system for performance change points based on multidimensional features, corresponding to the classification method for performance change points based on multidimensional features provided in the above embodiments. For example... Figure 2 As shown, this classification system for performance change points based on multidimensional features includes four basic modules, namely:

[0047] The data preprocessing module is used to assemble a performance test result sequence from the performance test results obtained by different versions of the software under different performance test environment configurations. The performance test result sequence is divided into result subsequences based on the performance test environment configuration. Change point detection technology is used to detect each result subsequence, thereby detecting all versions that have performance changes and forming a candidate performance change point set. Each candidate performance change point corresponds to a software version and a performance test environment configuration.

[0048] The multidimensional feature extraction module is used to extract features from each candidate performance change point in the candidate performance change point set, respectively, in terms of environment configuration dimension, time series dimension, version dimension, and content dimension.

[0049] The classification model training module is used to encode the four-dimensional features into a feature vector for each candidate performance change point in the candidate performance change point set, and use it as the input of the training sample. At the same time, it labels whether it belongs to the performance change point that needs to be repaired, thereby constructing the training sample data. Then, the obtained training sample data is used to train the random forest model to obtain the performance change point classification model.

[0050] The classification and prediction module is used to extract features from the environment configuration, time series, version, and content dimensions of candidate performance change points to be classified. These features are then encoded into feature vectors and input into the performance change point classification model trained in the classification model training module. This allows the module to classify whether a candidate performance change point is a performance change point that needs to be repaired.

[0051] Since the principle of the performance change point classification system based on multidimensional features in this embodiment of the invention is similar to the performance change point classification method based on multidimensional features in the above embodiment of the invention, any unresolved details of the specific implementation of each module of the system in this embodiment can also be found in the specific implementation of the above method, and repeated details will not be repeated.

[0052] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an electronic device corresponding to the classification method of performance change points based on multidimensional features provided in the above embodiments, which includes a memory and a processor;

[0053] The memory is used to store computer programs;

[0054] The processor is configured to implement the previously described classification method for performance change points based on multidimensional features when executing the computer program.

[0055] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0056] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the classification method of performance change points based on multidimensional features provided in the above embodiments. The storage medium stores a computer program that, when executed by a processor, can implement the classification method of performance change points based on multidimensional features as described above.

[0057] Specifically, in the computer-readable storage medium of the above two embodiments, the stored computer program is executed by a processor, which can perform the aforementioned steps S1 to S3.

[0058] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0059] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0060] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0061] The classification method for performance change points based on multidimensional features shown in S1 to S4 of the above embodiments will be applied to a specific example to demonstrate its technical effect.

[0062] Example

[0063] In this embodiment, the steps of the performance change point classification method based on multi-dimensional features are the same as those described in S1 to S4 above, and the specific process and parameter definitions can also be found above. The specific implementation process in the example is shown below, with detailed steps as follows:

[0064] S1. Data Preprocessing: Input the performance test result sequence and use change point detection technology to obtain a set of candidate performance change points from the software performance test result data. The specific steps of S1 include S11-S12, which are shown below.

[0065] S11. Obtain the performance test result sequence. First, assume V = [v1, v2, ...] is a time-ordered version sequence in a project, where v i This represents the version with the i-th number. C = [c1, c2, ...] is a configuration option for a performance testing environment, where c... k This represents the configuration options for the k-th test environment. Each version v i Multiple tests will be conducted under different configurations. It is a version sequence. All versions were tested for performance using configuration c. Indicates version v i In c k The performance metrics were obtained from performance tests conducted under the same configuration. Finally, the performance test results for each version under the same configuration were compiled into a performance test result sequence. in c k The performance test results of each version under the configuration can be regarded as a result subsequence in T.

[0066] S12. Obtain the set of candidate performance change points. For each result subsequence... The E-Divisive means technique, an automatic changepoint detection technology, is used to obtain a set of candidate performance changepoints. The candidate changepoints are the set of versions with performance changes automatically detected by the changepoint detection technology, denoted as...

[0067] S2. Multidimensional Feature Extraction: For each candidate performance change point CP i , Features are extracted across four dimensions: environment configuration, time series, version, and content. S2 specifically includes steps S21-S24, which are shown below.

[0068] S21. Obtain features at the environment configuration level: Obtain features at the environment configuration level based on the performance test environment configuration. Candidate performance change points (CPs) i The environmental configuration characteristic is identified as: E = < E project E #thread E measurement E task E test E variant >. Among them, E project This indicates the project for which performance testing is performed, where E #thread E represents the number of threads used for performance testing. measurement The measurement method representing the performance index, E task and E test Indicates the task name and test name for the performance test, E variant This indicates the build parameters for running the performance test task, i.e., the parameters used when building the project (the build parameters include hardware information).

[0069] S22. Obtaining features of the time series dimension: For time series... The time series feature extraction method (using the tsfresh tool in this embodiment) is applied to obtain features in the time series dimension. Candidate performance change points (CPs) are then selected.i Time series feature identifier: T = < T crossing_points T entropy T lumpiness T max_level_shift T norm_mean T norm_sd T serial_correlation T stability >. T crossing_points T represents the number of times the time series crosses the midline. entropy Shannon entropy, T, represents the spectral density of a time series. lumpiness T represents the variance of the sub-time series. max_level_shift T represents the maximum average offset between two consecutive sliding windows of a time series. norm_mean and T norm_sd T represents the mean and standard deviation of the minimum-maximum normalized values ​​of the time series, respectively. serial_correlation T represents the autocorrelation of a time series. stability This represents the variance of the mean of a sub-time series. It's important to note that a sub-time series refers to... The sequence of test results between the previous candidate performance change point and the current candidate performance change point.

[0070] S23. Obtain version-level characteristics: Obtain version-level characteristics by analyzing detailed version log information. Candidate performance change points (CPs) i The version feature identifier is:

[0071] V = < V execution_results V possible_changes V likelihood_changes >.

[0072] S231.V execution_results Indicates the candidate performance change point CP i Changes in performance metrics, V execution_results = <V percent_change V z_score >. Assumption For CP i Performance test results. V percent_change The percentage representing the change in performance is calculated using the following formula: V z_score This represents the result of standardizing the performance metric using z-scores.

[0073] S232.V possible_changes Indicates the possibility of version changes, V possible_changes = <V last_status V last_change_age V age V change rate >, where Vlast_status Indicates in version sequence Chinese CP i Is the previous version a candidate performance change point? last_change_age Indicates in version sequence The distance between the previous candidate performance change point and CP i The number of versions at intervals; V age Indicates in version sequence Chinese CP i Number of previous versions; V change rate Indicates in version sequence Chinese CP i The percentage of previous candidate performance change points out of the total number of versions.

[0074] S233.V likelihood_changes V represents the correlation between the text content in the version message and the performance changes. likelihood_changes = <V gnb-score V mnb-score V bnb-score V cnb-score >, where V gnb-score V mnb-score V bnb-score V cnb-score These represent the text score results obtained through Gaussian Naive Bayes, Complementary Naive Bayes, Multinomial Naive Bayes, and Bernoulli Naive Bayes, respectively. The specific steps for obtaining these four sub-features include S2331-S2334, which are shown below.

[0075] S2331. Retrieve historical versions and tags. Retrieve all CPs from V. i All previous versions are denoted as V′. If v j ∈V′ in a certain configuration c k If there is a performance change under ∈C, then v j Marked as class T, otherwise v j It is marked as class F.

[0076] S2332. Obtain the text of each version of the message in V′, convert the content of the message into words, and then use TF-IDF to convert them into a numerical representation in vector form.

[0077] S2333. Using the vector obtained in S2332 as input features, train the model using Gaussian Naive Bayes, Complementary Naive Bayes, Multinomial Naive Bayes, and Bernoulli Naive Bayes respectively to obtain the classification model.

[0078] S2334. CP iAfter the message text is processed in step S2332, it is input into the four classification models obtained in step S2333 to output CP. i The probability V of being classified into class T gnb-score V mnb-score V bnb-score V cnb - score .

[0079] S24. Obtain content characteristics: By analyzing a set of files that have been modified in the version. Candidate performance change points CP i Content feature identifier: CT=<CT Corr CT Size CT Evolution The following sections will present the forms of the sub-features in the three dimensions.

[0080] S241.CT Corr The confidence level of the association rule between version sequences and files is calculated as follows:

[0081]

[0082] Represents version sequence and a file The correlation score between them is defined as: in yes A subset of, representing Chinese f j There exists a set of modified versions.

[0083] S242.CT Size Indicates the scale of the modifications. CT Size = <CT #file CT #code CT #config CT #resource CT ALOC CT DLOC >. Each commit version v i There is a collection of modified files. CT #file This indicates the number of files modified in version 1, i.e. The number of files contained; based on The filename extensions of each file in the CT algorithm are used to categorize the files into three classes and count the number of files in each class. code CT represents the number of source code files. #config Indicates the number of files used for configuration, CT #resourc Indicates the number of resource files; CTALOC and CT DLOC This indicates the number of lines of text added and deleted in the modified file within the specified version. The calculation method is as follows:

[0084]

[0085]

[0086] Where addLOC(f j V i ) and delLOC(f j V i ) represents f j In V i The number of rows added and deleted.

[0087] S243.CT Evolution This indicates the evolutionary characteristics of the modified content. CT Evolution = <CT #dev CT f_fre CT f_age >, including CT #dev Indicates the end of CP i right The total number of different developers who contributed to each document; CT f_fre express The frequency of changes to each file in the file, i.e. How many versions of each file in V have been modified? CT f_age Indicates the age of the document, i.e. The number of versions between the first modification in each file V and the current version.

[0088] S3. Train a classification model for performance change points using random forest. Specific steps include:

[0089] S31. Construct the dataset. Candidate performance change points are manually divided into two categories: TCPs and FCPs. Candidate performance change points in TCPs are considered positive samples, i.e., performance change points that need to be fixed. Candidate performance change points in FCPs are considered negative samples, i.e., false positive performance change points.

[0090] S32. Feature Encoding. Encode the four features of each candidate performance change point in the dataset to form a feature vector. See step S2 above for the specific acquisition of the four features. The final feature vector input to the model is formed by concatenating the four encoded feature vectors.

[0091] S33. Using cross-validation, a random forest model is trained on the constructed dataset to obtain the final performance change point classification model. In this embodiment, the candidate performance change points in the dataset are first sorted according to time sequence. Then, a time-series-based K-fold cross-validation (CV_Ts) method is used, as follows:

[0092] First, the dataset is divided into K blocks. In the first iteration, we use the first block for training and the second block for testing. These two blocks are then merged together as the training dataset for the second iteration, while the data from the third block serves as the test dataset for that iteration. This process is repeated until the last block has been tested. The CV_Ts method uses forward testing, where the test dataset is constructed by expanding the training dataset; that is, the new training dataset is a superset of the old training dataset.

[0093] Once trained, the random forest model can be used as a performance change point classification model for practical applications or testing.

[0094] S4. In practical applications or testing, S2 is executed on the candidate performance change points to be classified to obtain their corresponding four-dimensional features; then the four-dimensional features of the candidate performance change points to be classified are encoded according to S32 to obtain the encoded feature vector; finally, the encoded feature vector is input into the performance change point classification model trained in S3 to classify whether the candidate performance change point belongs to the performance change point that needs to be repaired, and the final classification result is obtained.

[0095] The following section presents the specific dataset and validation results for the classification method described above in this embodiment:

[0096] Data source acquisition: The publicly available dataset provided by MongoDB used in this embodiment includes test results from MongoDB's internal performance testing system (detected performance change points and classification results).

[0097] Table 1. Dataset Description

[0098] Number of performance test results 722461 153.98 10 837 Number of candidate performance change points 16460 3.51 1 25 Number of TCPs 4080 0.87 0 6 Number of FCPs 5563 1.19 0 25

[0099] Table 1 provides an overview of the dataset. The first column in Table 1 is a statistical description of the dataset, and the following four columns describe the total number of objects, mean, minimum, and maximum values, respectively.

[0100] To verify the technical effectiveness of the present invention, five commonly used indicators were selected: area under the curve (AUC), Matthews correlation coefficient (MCC), precision, false alarm rate (FAR), and proportional gain (Recall@K).

[0101] This embodiment also selected logistic regression (LR), decision tree (DT), k-nearest neighbors (KNN), support vector machine (SVM), random forest (RF), and multilayer neural network (MLP). As comparative algorithms to demonstrate the effectiveness of the proposed method, the experimental results are shown in Table 2.

[0102] Table 2 Description of Experimental Results

[0103]

[0104] The results show that, in terms of overall model performance, the Random Forest (RF)-based model performs best in terms of AUC and MCC metrics. Specifically, the Multilayer Neural Network (MLP) model has the highest recall, but its precision is only 0.814. The low AUC of the MLP model indicates an imbalance between recall and precision. Therefore, the RF-based method of this invention is an effective way to predict point-of-change (TCP) types, which can help developers identify TCPs while minimizing inspection time.

[0105] Furthermore, to verify the impact of the proposed multidimensional features on the detection results, this embodiment also analyzes whether different feature categories have an important role. Experiments were conducted using features that included only each category (features for environment configuration, time series, version, and content dimensions are labeled Configuration, TimeSeries, Version, and Content in the table, respectively) and features that excluded each category individually, to evaluate the significance and effect size of the differences.

[0106] Table 3 shows the performance differences when considering specific feature categories.

[0107]

[0108] Table 4 Performance differences when ignoring specific feature categories

[0109]

[0110] Table 3 shows the average difference, significance level, and effect size between models built using the complete feature set and those using only a single feature class. The results demonstrate that no single feature class can provide good predictive performance. When only one type of feature is used, the model's AUC and MCC values ​​drop sharply, and the predictive model's performance significantly declines. This proves the effectiveness of the multi-dimensional feature-based approach proposed in this invention.

[0111] Table 4 shows the average difference, significance, and effect size between the model built using all features and the model built by excluding only one type of feature. The results show that excluding any one type of feature either results in a worse prediction model than using all features or remains unchanged. This demonstrates that the feature dimensions proposed in this invention are indispensable.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A classification method for performance change points based on multidimensional features, characterized in that... Includes the following steps: S1: The performance test results obtained from different versions of the software under different performance test environment configurations are used to form a performance test result sequence. The performance test result sequence is divided into result subsequences based on the performance test environment configuration. Change point detection technology is used to detect each result subsequence, and all versions with performance changes are detected to form a set of candidate performance change points; each candidate performance change point corresponds to a software version and a performance test environment configuration; S2: For each candidate performance change point in the candidate performance change point set, extract features from the environment configuration dimension, time series dimension, version dimension, and content dimension respectively; S3: For each candidate performance change point in the candidate performance change point set, the four-dimensional features are encoded into a feature vector as input to the training sample. At the same time, the feature vector is labeled to indicate whether it belongs to the performance change point that needs to be repaired, thereby constructing the training sample data. Then, the obtained training sample data is used to train the random forest model to obtain the performance change point classification model. S4: For candidate performance change points to be classified, extract features from the environment configuration dimension, time series dimension, version dimension, and content dimension. Then, encode the features of the four dimensions into feature vectors and input them into the performance change point classification model trained in S3 to classify whether the candidate performance change point belongs to the performance change point that needs to be repaired. In S2, for each candidate performance change point, the method for extracting the four dimensions of features—environment configuration, time series, version, and content—is as follows: S21: Based on the performance test environment configuration corresponding to the current candidate performance change point, obtain the features corresponding to the environment configuration dimension, which includes 6 sub-feature dimensions, where: the first environment sub-feature is the project executing the performance test, the second environment sub-feature is the number of threads used to perform the performance test, the third environment sub-feature is the measurement method of the performance metric, the fourth environment sub-feature is the name of the performance test task, the fifth environment sub-feature is the name of the performance test, and the sixth environment sub-feature is the construction parameters of the performance test task; S22: The result subsequence containing the current candidate performance change point is taken as a time series. Time series feature extraction methods are used to obtain the features of the time series dimension, which includes eight sub-feature dimensions: the first time sub-feature is the number of times the time series crosses the midline; the second time sub-feature is the Shannon entropy of the time series spectral density; the third time sub-feature is the variance of the sub-time series; the fourth time sub-feature is the maximum average offset between two consecutive sliding windows of the time series; the fifth time sub-feature is the average of the minimum and maximum normalized values ​​of the time series; the sixth time sub-feature is the standard deviation of the minimum and maximum normalized values ​​of the time series; the seventh time sub-feature is the autocorrelation of the time series; and the eighth time sub-feature is the variance of the mean of the sub-time series. The sub-time series is the sequence segment from the current candidate performance change point to the previous candidate performance change point in the time series. S23: By analyzing the detailed information in the version logs of the current candidate performance change points, the features of the version dimension are obtained, which include three sub-feature dimensions: the first version sub-feature is the change in the performance metric value of the current candidate performance change point, the second version sub-feature is the probability of the current version being changed, and the third version sub-feature is the correlation between the text content in the messages of the current version and the performance change. S24: By analyzing the set of modified files corresponding to the current candidate performance change point, the features of the content dimension are obtained, which include three sub-feature dimensions: the first content sub-feature is the confidence of the association rule between the version sequence and the modified file; the second content sub-feature is the scale of the modified content in the modified file; and the third content sub-feature is the evolutionary feature of the modified content in the modified file. The third version sub-feature consists of four feature values, which are the classification probabilities output by the four version classification models. The four version classification models are a trained Gaussian Naive Bayes model, a complementary Naive Bayes model, a multinomial Naive Bayes model, and a Bernoulli Naive Bayes model. The message text content in the version log of the current candidate performance change point is segmented and vectorized as the input of each model, and the output of each model is the classification probability of whether there is a performance change in the version.

2. The classification method for performance change points based on multidimensional features according to claim 1, characterized in that, In S1, the change point detection technology used is the E-Divisive means detection algorithm.

3. The classification method for performance change points based on multidimensional features according to claim 1, characterized in that, In S23, the first version sub-feature and the second version sub-feature in the version dimension features are as follows: The first version sub-feature consists of two feature values. The first feature value is the performance change ratio of the current candidate performance change point relative to the previous version, and the second feature value is the result of the performance metric of the current candidate performance change point after z-score standardization. The second version sub-feature consists of four feature values. The first feature value indicates whether the previous version of the current candidate performance change point in the result sub-sequence is a candidate performance change point. The second feature value is the number of versions of the current candidate performance change point in the result sub-sequence that are far from the previous candidate performance change point. The third feature value is the number of versions in the result sub-sequence that are before the current candidate performance change point. The fourth feature value is the proportion of the number of candidate performance change points in the result sub-sequence that are before the current candidate performance change point to the total number of versions.

4. The classification method for performance change points based on multidimensional features according to claim 3, characterized in that, The third version of the sub-feature uses the TF-ID method to vectorize words in the message text content.

5. The classification method for performance change points based on multidimensional features according to claim 1, characterized in that, In S24, the three sub-feature dimensions of the content dimension are as follows: S241: The first content sub-feature is the sum of the relevance scores between each modified file in the modified file set and the result subsequence where the current candidate performance change point is located; the relevance score between any modified file and the result subsequence is the ratio of the modulus of the first intersection to the modulus of the second intersection, the first intersection being the version intersection of the subset and the set of candidate performance change points in the result subsequence, the second intersection being the version intersection of the subset and the result subsequence, and the subset being the set of versions in the result subsequence where the current modified file has been modified; S242: The second content sub-feature consists of six feature values. The first feature value is the number of modified files in the modified file set. The second feature value is the number of source code files in the modified file set. The third feature value is the number of configuration files in the modified file set. The fourth feature value is the number of resource files in the modified file set. The fifth feature value is the number of lines of text added to the modified file. The sixth feature value is the number of lines of text deleted from the modified file. S243: The third content sub-feature consists of three feature values. The first feature value is the total number of developers who have contributed to each file in the modified file set up to the current candidate performance change point. The second feature value is the change frequency of each file in the modified file set. The third feature value is the age of each file in the modified file set.

6. The classification method for performance change points based on multidimensional features according to claim 1, characterized in that, In step S3, when labeling candidate performance change points, they are divided into positive samples and negative samples. Positive samples are performance change points that are confirmed to need to be repaired, while negative samples are false alarm performance change points that do not need to be repaired.

7. The classification method for performance change points based on multidimensional features according to claim 1, characterized in that, In S3, a grid search method is used to obtain the optimal parameters of the model when training the random forest model.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the classification method for performance change points based on multidimensional features as described in any one of claims 1 to 7.

9. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the classification method for performance change points based on multidimensional features as described in any one of claims 1 to 7 when executing the computer program.

Images