Construction method of software defect prediction model

Abstract

The invention mainly provides a method for constructing a software defect prediction model. The method comprises the following steps: collecting historical multi-source data in a software function test process and preprocessing the historical multi-source data; extracting a plurality of features from the preprocessed data, screening out key features having significant influence on defect prediction by adopting an information gain or mutual information method, and forming a data set; selecting a proper machine learning algorithm or a deep learning algorithm as a basic model, training and verifying the basic model by using the data set, determining an optimal hyper-parameter combination, and constructing a defect prediction model; and evaluating the defect prediction model by using a test data set obtained in real time, and if the performance of the model does not reach an expected result, analyzing reasons and performing optimization. The prediction model constructed by the technical scheme can predict which modules or test cases are more likely to expose defects before the test of a new version each time, so that the test resource configuration is optimized, and the test efficiency and the software quality are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of software detection, in particular to a method for constructing a software defect prediction model. TECHNICAL BACKGROUND

[0002] With the increasing complexity of software systems, it is particularly important to ensure their quality and reliability. The traditional functional testing method has the following problems: (1) uneven resource allocation: it is difficult to accurately determine which modules are more prone to defects, resulting in unreasonable distribution of testing focus; (2) low testing efficiency: lack of defect risk prediction mechanism, leading to inefficient and repetitive testing by the testing team; (3) defect discovery lag: many critical defects are discovered in the later stages, increasing the cost of repair and the risk of project delay; (4) relying on historical experience: some existing defect prediction methods mainly rely on expert experience or static code analysis, lacking dynamic adaptability and data-driven capability.

[0003] Although some research has attempted to apply machine learning to defect prediction, most of them focus on defect prediction at the source code level, lacking comprehensive modeling and prediction capabilities for test cases, modules and execution results in the functional testing process.

[0004] TECHNICAL CONTENT

[0005] The present application aims to solve at least one of the above technical problems.

[0006] To solve the above problems, the present application provides a method for constructing a software defect prediction model, comprising:

[0007] Collecting historical multi-source data in the software functional testing process and performing cleaning and standardization preprocessing;

[0008] Extracting multiple features from the preprocessed data, using information gain or mutual information method to select key features that have significant impact on defect prediction, and forming a data set;

[0009] Selecting a suitable machine learning algorithm or deep learning algorithm as a base model, training and verifying the base model using the data set to determine the optimal hyperparameter combination, and constructing a defect prediction model;

[0010] Using real-time obtained test data set to evaluate the defect prediction model, and if the model performance does not meet the expected results, analyzing the reasons and optimizing.

[0011] Optionally, the multi-source data includes test case execution result data, bug data, software code metric data, software module dependency relationship data and other related data features.

[0012] Optionally, if the model performance does not meet the expected results, the cause is analyzed and optimized, including: if the cause is under-fitting, the method of adjusting the model structure or increasing the model complexity is used for optimization; if the cause is over-fitting, the method of regularization technique or increasing the amount of training data is used for optimization.

[0013] Optionally, the test data set obtained in real time is used to evaluate the defect prediction model, including: the accuracy or recall or F1 value index is used to measure the prediction performance of the model.

[0014] Optionally, the optimal combination of hyperparameters is determined after training and verifying the base model using the data set, including:

[0015] If the base model is a random forest, the number of trees is adjusted as a hyperparameter; if the base model is a support vector machine, the kernel function parameter is adjusted; if the base model is a neural network, the number of hidden layer nodes is adjusted;

[0016] The optimal combination of hyperparameters is determined by cross-validation method.

[0017] Optionally, a plurality of features are extracted from the preprocessed data, including: the bug data features include bug type, bug level, module; the test case execution result data features include test case execution times, pass rate, failure rate; the software code metric data features include code lines, cyclomatic complexity, code coupling degree; the software module dependency relationship data features include module in-degree, module out-degree, dependency depth; the other related data features include project features, team features.

[0018] Optionally, the method of information gain is used to screen out key features that have a significant impact on defect prediction, including:

[0019] All extracted features are combined into a feature set, from which the features and target variables are extracted;

[0020] Then the SelectKBest and mutual_info_classif functions in the scikit-learn library are used for information gain-based feature selection, and the top 10 features with the highest selection scores are specified as the most important feature set;

[0021] The selector.fit_transform(X,y) method is used to train the input features X and target variables y, and perform feature selection according to the previously set selection criteria.

[0022] The technical scheme of the present application fuses test case execution results, software code metrics, module dependency relationships and other multi-source data, captures software features from multiple angles, more accurately describes software system running conditions and potential risks, and thus can predict which modules or test cases are more likely to expose defects before each new version is tested, thereby optimizing test resource allocation and improving test efficiency and software quality.

[0023] Drawings

[0024] Figure 1 A flowchart of a software defect prediction model construction method according to an embodiment of the present application is shown in FIG. 1.

[0025] Figure 2 A flowchart of multi-source data preprocessing according to an embodiment of the present application is shown in FIG. 2.

[0026] Figure 3 A flowchart of key feature screening according to an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0027] In order to make the above objectives, features and advantages of the present application more apparent, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided to make the present application more thorough and complete. It should be understood that the drawings and embodiments of the present application are for exemplary purposes only, and are not intended to limit the scope of protection of the present application.

[0028] It should be understood that each step described in the method embodiments of the present application can be executed in different orders, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0029] The term “comprising” and variations thereof as used herein are open-ended, that is, “comprising but not limited to”; the term “based on” is “based at least in part on”; the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments”; the term “optionally” means “optional embodiments.” Related definitions are given throughout the description. It should be noted that the concepts “first”, “second”, etc. mentioned in the present application are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modification of "one" and "multiple" mentioned in the present application is illustrative rather than restrictive, and those skilled in the art should understand that "one or more" should be understood unless otherwise explicitly indicated in the context.

[0031] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present application are only for illustrative purposes, and are not used to limit the scope of the messages or information.

[0032] With reference to the drawings. The present embodiment provides a method for constructing a software defect prediction model, comprising:

[0033] S1, collect historical multi-source data in the software function test process, and perform cleaning and standardization preprocessing. The multi-source data includes but is not limited to: test case execution result data, bug data, software code metric data, software module dependency relationship data and other related data characteristics. The collected data is cleaned to remove noise data and outliers, and missing values are filled. Then the data is standardized to make different types of data have the same dimension.

[0034] S2, extract multiple features from the preprocessed data, and use information gain or mutual information method to screen out key features that have significant influence on defect prediction, and form a data set. The bug data characteristics include bug type, bug level, module; The test case execution result data characteristics include test case execution times, pass rate, failure rate; The software code metric data characteristics include code lines, cyclomatic complexity, code coupling degree; The software module dependency relationship data characteristics include module in-degree, module out-degree, dependency depth; Other related data characteristics include project characteristics, team characteristics.

[0035] S3, select a suitable machine learning algorithm or deep learning algorithm as a base model, train and verify the base model using the data set to determine the optimal hyperparameter combination, and construct a defect prediction model. The base model can select random forest, support vector machine, neural network, etc. The training data processed by feature engineering is used to train the base model, and the hyperparameters of the model are adjusted, such as the number of trees in random forest, the kernel function parameter in support vector machine, the number of hidden layer nodes in neural network, etc. The optimal hyperparameter combination is determined by cross-validation method, and a good defect prediction model is constructed.

[0036] S4, using the test data set obtained in real time to evaluate the defect prediction model, if the model performance does not achieve the expected result, analyze the reason and optimize. Among them, the accuracy, recall rate, F1 value and other indicators are used to measure the prediction performance of the model, if the model performance does not achieve the expected result, analyze the reason and optimize: if the reason is under fitting, adjust the model structure or increase the model complexity; if the reason is over fitting, use regularization techniques or increase the amount of training data.

[0037] In order to further understand the scheme, the embodiment provides specific operations of the actual construction process, which are described as follows.

[0038] In the test case management system, the connection between the data collection script and the Jira or TestRail system is configured through the API key and interface address provided by the system. In the Git repository, the hook script is deployed to the corresponding code repository directory to ensure that the script can be executed to capture data when the code is committed. For continuous integration tools such as Jenkins, configure plugins or write scripts to regularly obtain test execution records from their databases or log files.

[0039] According to the development rhythm of the project and the data update frequency, the data collection frequency of the test case management system and the continuous integration tool is set to once a day to ensure that the latest test cases and execution result information can be obtained in time. For the code submission information of the Git repository, since it changes frequently, the hook script is set to capture data in real time to ensure that each code submission can be recorded.

[0040] In the Python environment, the following code is written for data cleaning and preprocessing, see Figure 2 . First, import the pandas library and MinMaxScaler function:

[0041] import pandas as pd

[0042] from sklearn.preprocessingimport MinMaxScaler

[0043] Then read the original data stored in the MySQL database and store the data in the table named raw_data:

[0044] raw_data=pd.read_sql('SELECT*FROM raw_data',con=mysql_connection)

[0045] Data cleaning is performed to remove noisy data and outliers, for example, for the test case priority field, if values less than 1 or greater than 5 are considered outliers and are deleted:

[0046] raw_data = raw_data[(raw_data['priority'] >= 1) & (raw_data['priority'] <= 5)]

[0047] Missing values are handled, and for the test case description field, missing values are marked as "missing":

[0048] raw_data['description'].fillna('missing', inplace=True)

[0049] For the numerical code modification frequency field, the mean value is filled:

[0050] mean_value = raw_data['change_count'].mean()

[0051] raw_data['change_count'].fillna(mean_value, inplace=True)

[0052] Finally, data standardization is performed, assuming that the numerical fields to be standardized are numeric_columns:

[0053] scaler = MinMaxScaler()

[0054] raw_data[numeric_columns] = scaler.fit_transform(raw_data[numeric_columns])

[0055] The preprocessed data is stored back into the database for subsequent feature extraction analysis.

[0056] See Figure 3 . Read the preprocessed data and calculate the pass rate and failure rate of the test cases. Assume that the test case execution result field is result, and the pass result is marked as "pass" and the failure result is marked as "fail":

[0057] test_case_data = pd.read_sql('SELECT * FROM preprocessed_data WHERE data_type="test_case"', con=mysql_connection)

[0058] test_case_data['pass_rate']=test_case_data.groupby('test_case_id')['result'].apply(lambda x:(x=='pass').sum() / len(x))

[0059] test_case_data['fail_rate']=test_case_data.groupby('test_case_id')['result'].apply(lambda x:(x=='fail').sum() / len(x))

[0060] Use code analysis tools such as Pylint for Python code, execute command line instructions in the project code repository directory to extract code metrics features. Save the extracted results as a CSV file and read them into a pandas dataframe to merge with other data.

[0061] Store module call relationship data in a table named module_dependency, containing caller_module and callee_module fields, representing calling modules and called modules. Calculate the in-degree and out-degree of modules, and merge the calculated in-degree and out-degree data with other feature data.

[0062] Merge all extracted features into a feature set feature_set, then use the SelectKBest and mutual_info_classif functions in the scikit-learn library for information gain-based feature selection. Assume the target variable is defect, indicating whether the software has defects, as follows:

[0063] The program extracts features and target variables from the dataset in feature_set. It creates a new DataFrame X by calling the feature_set.drop('defect', axis=1) method, which includes all columns except the 'defect' column as features. At the same time, the target variable y is defined as feature_set['defect'], which is the column in the original data that indicates whether the software has defects.

[0064] Next, initialize the feature selector selector, which is an object using the SelectKBest method with the scoring function set to mutual_info_classif, a mutual information-based method for assessing the importance of features. Specify the parameter k = 10, which will select the top 10 features with the highest scores as the most important feature set.

[0065] Subsequently, train the input features X and target variable y using the selector.fit_transform(X, y) method, and perform feature selection based on the previously set selection criteria. The fit_transform method not only calculates the importance score of each feature relative to the target variable, but also returns a new dataset X_selected containing only the selected number of the most important features, preferably 10 in this implementation.

[0066] To determine which features are selected, continue to call the selector.get_support() method, which returns a boolean array indicating which columns are important. Then, combine this boolean array with the original feature name list X.columns to obtain all the selected feature names through the index operation X.columns[selector.get_support()], and store them in the variable selected_features.

[0067] Save the selected key features to the database or file for subsequent model training.

[0068] In this implementation, take the random forest model as an example, where c is the number of categories, and Pi is the probability of occurrence of a certain category:

[0069]

[0070] Each node in the decision tree will select the optimal feature and split point based on a certain criterion, as follows

[0071] In Python, use the scikit-learn library for model selection and hyperparameter initialization:

[0072] from sklearn.ensembleimport RandomForestClassifier

[0073] rf = RandomForestClassifier(n_estimators = 100, max_depth = 10, random_state = 42)

[0074] where n_estimators represents the number of trees, max_depth represents the maximum depth of the trees, and random_state is used to set the random seed to ensure experimental reproducibility.

[0075] Read the key feature set data, and use 30% of the data as the validation set and 70% of the data as the training set.

[0076] Train the random forest model using the training set and evaluate the model performance using the validation set:

[0077] If the model performance does not meet the expected standard, for example, the accuracy is less than 0.8, adjust the hyperparameters and retrain and evaluate until the model performance meets the standard.

[0078] When the model performance meets the expected standard, use the test set for final evaluation. Finally, save the trained optimal model as a pickle file for use in actual applications:

[0079] In the actual deployment stage, whenever a new software version is ready to enter the testing phase, the defect prediction model can be run. After inputting the relevant feature data of the current version, the model will output the prediction results, indicating which modules or test cases are more likely to expose defects. Based on this, the test team can prioritize resources to conduct in-depth testing in high-risk areas, while reducing attention to low-risk parts, thereby achieving a more efficient and accurate testing strategy.

[0080] In summary, the defect prediction model based on software functional testing can provide valuable decision support closely around software testing, helping testers accurately predict software defects, optimize test resource allocation and test scheme improvement, and thus improve the overall software quality.

[0081] Although the present application is disclosed as above, the protection scope of the present application is not limited to this. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and these changes and modifications will fall within the protection scope of the present application.

Claims

1. A method for constructing a software defect prediction model, characterized in that... include: Collect historical multi-source data during the software functional testing process, and perform cleaning and standardization preprocessing; Multiple features are extracted from the preprocessed data, and key features that have a significant impact on defect prediction are selected using information gain or mutual information methods to form a dataset. Select a suitable machine learning algorithm or deep learning algorithm as the base model, train and validate the base model using the dataset, determine the optimal hyperparameter combination, and construct a defect prediction model. The defect prediction model is evaluated using a real-time test dataset. If the model performance does not meet expectations, the reasons are analyzed and optimization is performed.

2. The method for constructing a software defect prediction model as described in claim 1, characterized in that... The multi-source data includes: test case execution result data, bug data, software code metric data, software module dependency data, and other relevant data features.

3. The method for constructing a software defect prediction model as described in claim 1, characterized in that... If the model performance does not meet the expected results, the reasons will be analyzed and optimized, including: if the reason is underfitting, optimization will be carried out by adjusting the model structure or increasing the model complexity; if the reason is overfitting, optimization will be carried out by regularization techniques or increasing the amount of training data.

4. A method for constructing a software defect prediction model as described in claim 1 or 3, characterized in that... The evaluation of the defect prediction model using a real-time obtained test dataset includes: using accuracy, recall, or F1 score metrics to measure the model's predictive performance.

5. The method for constructing a software defect prediction model as described in claim 1, characterized in that... The process of determining the optimal hyperparameter combination after training and validating the base model using the dataset includes: If the base model is a random forest, the number of trees is adjusted using hyperparameters; if the base model is a support vector machine, the kernel function parameters are adjusted; if the base model is a neural network, the number of hidden layer nodes is adjusted. The optimal combination of hyperparameters is determined by cross-validation.

6. The method for constructing a software defect prediction model as described in claim 2, characterized in that... The extraction of multiple features from the preprocessed data includes: The bug data features include bug type, bug level, and module; the test case execution result data features include the number of test case executions, pass rate, and failure rate; the software code metric data features include lines of code, cyclomatic complexity, and code coupling; the software module dependency relationship data features include module in-degree, module out-degree, and dependency depth; and other relevant data features include project features and team features.

7. The method for constructing a software defect prediction model as described in claim 2, characterized in that... The method of information gain is used to screen out key features that have a significant impact on defect prediction, including: All extracted features are merged into a feature set, from which features and target variables are extracted; Then, the SelectKBest and mutual_info_classif functions from the scikit-learn library are used to perform information gain-based feature selection, specifying the parameters to select the top 10 features with the highest scores as the most important feature set. The input feature X and target variable y are trained using the selector.fit_transform(X,y) method, and feature selection is performed according to the previously set selection criteria.

Images