Diabetes prediction model construction method and prediction system based on ensemble learning

Abstract

The invention belongs to the technical field of diabetes diagnosis, and particularly relates to a diabetes prediction model construction method based on ensemble learning and a prediction system. The method comprises the following steps: step 1, carrying out preprocessing and feature screening on a data set; 2, dividing a training set and a test set; 3, searching and extracting positive samples in the training set, randomly generating positive samples obeying normal distribution according to the mean vector and the covariance matrix of the positive samples in the training set, adding the newly generated positive samples into the training set, and continuously repeating until the proportion of the negative samples to the positive samples is balanced; and 4, learning the model by using the processed training set machine. The invention also provides a diabetes prediction system constructed by using the machine learning model. The method can effectively improve the prediction performance of the prediction model through the balance of the sample data and the integrated learning of the single classifier, and has a good application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of diabetes diagnosis, and in particular relates to a diabetes prediction model construction method and a prediction system based on ensemble learning. Background Art

[0002] With the continuous development of science and technology, the application of machine learning algorithms in the medical field is becoming more and more extensive. Compared with the traditional medical decision-making system based on doctors and experts, the disease prediction model based on machine learning can more effectively utilize massive physical examination data, fully explore the value of these data, analyze the potential connections within the data, and solve the problems of complex physical examination data structure and large amount of information, thereby improving the accuracy of prediction and the decision-making efficiency of doctors.

[0003] At present, there are two main types of methods for disease prediction based on machine learning: (1) methods based on weak classifier algorithm learning; (2) methods based on ensemble learning, such as bagging, boosting, and stacking. Generally speaking, it is often difficult to achieve higher accuracy when using weak classifier algorithms for diabetes prediction; while methods based on ensemble learning comprehensively consider the results of multiple weak classifiers, can achieve higher accuracy, and have more stable prediction performance.

[0004] In the process of building diabetes prediction models based on machine learning, single classifier models are favored by relevant researchers because of their simple implementation and fast running speed. Therefore, various single classifier algorithms have become the choice of many scholars in this research field: The literature "BJOG: An International Journal of Obstetrics & Gynaecology, 2010, 117 (1): 69-75." uses the multivariate logistic regression method to model diabetes data, taking into account factors such as race, family genetic history, and personal physical examination indicators. The AUC value of the prediction result is 0.77, and it passes the Hosmer-Lemeshow goodness of fit test. The literature "Journal of Medical Systems, 2011, 35 (2): 277-281." uses clinical data from a hospital in Turin, Italy to train six models such as logistic regression and artificial neural network, and concludes that simple models such as logistic regression perform better than artificial neural networks. The literature "Global Journal of Health Science, 2015, 7(5): 304-310." used the decision tree J48 algorithm and more than 20,000 samples to predict diabetes, and the results showed that the model achieved an accuracy of 71.7%. The literature "British Journal of Applied Science & Technology, 2014, 4(33): 4591-4606." used a convolution algorithm to fill in missing values. This method improved the diabetes prediction accuracy of the decision tree algorithm, the neighbor algorithm, and the naive Bayes algorithm by 7%.

[0005] However, the classification indicators of the above machine learning weak classifier models are poor, and it is difficult to meet the requirements of prediction accuracy and robustness in medical work. Therefore, ensemble learning algorithms that integrate the results of multiple classifiers have emerged. Ensemble learning mainly includes methods such as bagging and boosting. This type of algorithm comprehensively considers the classification results of multiple classification algorithms and has more stable prediction performance.

[0006] The literature "Procedia Computer Science, 2015, 69: 132-142." uses bagging and boosting algorithm frameworks to integrate multiple machine learning algorithms. The experimental results show that the random forest algorithm proposed in the literature "Machine Learning, 2001, 45 (1): 5-32." has the best prediction effect. This algorithm belongs to the bagging series of algorithms. The main idea is to generate multiple sampling sets in the training set to train multiple decision trees in parallel, and finally decide the final classification result by voting. The literature "Procedia Computer Science, 2019, 165: 292-299." compares the diabetes prediction performance of random forest, logistic regression and AdaBoost algorithms, and concludes that the AdaBoost algorithm proposed in the literature "Journal of Computer and System Sciences, 1997, 55 (1): 119-139." has the highest prediction accuracy. AdaBoost is an integrated learning algorithm in the boosting series. This algorithm is an additive model. Each iteration will increase the weight of the misclassified samples in the weak classifier, and has a higher prediction accuracy. The literature "Intelligent Computers and Applications, 2019, 9(5): 176-179." combines the existing K nearest neighbor algorithm, support vector machine, logistic regression and random forest algorithms by voting. The voting rule is Hard Voting, that is, the weight of each model is the same when voting, and if the votes of the four classifiers are the same, a classification result is randomly generated. Finally, by comparing the relevant evaluation indicators, it is verified that this fusion model has better prediction effect for diabetes prediction than a single model. The literature "IEEE Access, 2020, 8: 76516-76531." Based on the bagging idea, a rule for weighted fusion of different machine learning algorithms is proposed. The weight is the AUC value of the prediction results of each classifier. The numerical experimental results also verify that the ensemble learning model using weighted voting with AdaBoost and XGBoost algorithms performs better.

[0007] In summary, there have been many studies on machine learning prediction methods for diabetes in the existing technology. However, these existing studies have focused on how to improve the accuracy and stability of the model by improving the model method, but still have not proposed a method to solve the problem of sample class imbalance in diabetes datasets, and further improvement is needed. Summary of the invention

[0008] In response to the problems of the prior art, the present invention provides a diabetes prediction model construction method and prediction system based on ensemble learning, aiming to improve the diabetes prediction model construction method, solve the problem of sample class imbalance in the diabetes data set, and thereby improve the prediction performance of the final established prediction model.

[0009] A method for constructing a diabetes prediction model based on ensemble learning comprises the following steps:

[0010] Step 1: preprocess and feature screen the data set;

[0011] Step 2, divide the training set and test set;

[0012] Step 3: Find and extract positive samples in the training set, randomly generate positive samples that obey the normal distribution according to the mean vector and covariance matrix of the positive samples in the training set, and add the newly generated positive samples to the training set, and repeat until the ratio of negative and positive samples reaches a balance;

[0013] Step 4: Use the processed training set machine learning model.

[0014] Preferably, in step 1, the preprocessing step includes: missing value filling, outlier processing and data standardization operations.

[0015] Preferably, the missing value filling method is: using the downward filling method in the adjacent value filling method, and deleting the samples that are still missing after the downward filling;

[0016] The outlier processing adopts the box plot method principle, and the theoretical upper and lower bounds of each variable are calculated through the upper and lower quartiles, and outliers outside the theoretical upper and lower bounds are excluded.

[0017] Preferably, in step 1, the feature screening method is selected from: principal component analysis, independent component analysis or a method based on correlation coefficient.

[0018] Preferably, in step 2, the method of dividing the training set and the test set is a stratified sampling and 5-fold crossover method.

[0019] Preferably, in step 3, the formula for randomly generating positive samples that obey the normal distribution is:

[0020] x=m+N(0,C)

[0021] Where m and C are the mean vector and covariance matrix of the positive samples in the training set, respectively; N(0,C) is a multidimensional normal random variable with a mean vector of 0 and a covariance matrix of C; and x is a newly generated positive sample.

[0022] Preferably, in step 4, the machine learning model is obtained by fusing a single classifier algorithm through a weighted voting rule, and the weighted voting rule includes:

[0023] 1) Calculate a single classifier G m The misprediction error rate e m , the calculation formula is:

[0024]

[0025] Among them, I is the indicative function, N is the total number of predicted samples, yi is the sample label value, is the model prediction value;

[0026] 2) Calculate the weight ω of each single classifier m , the calculation formula is:

[0027]

[0028] 3) Normalize the weights of all single classifiers, and the calculation formula is:

[0029]

[0030] 4) Perform weighted voting on the classification results of the single classifier according to the normalized weights, and define the voting result as the prediction result of the integrated machine learning model.

[0031] Preferably, the single classifier algorithm is selected from at least two of the following algorithms: K nearest neighbor, support vector machine, BP neural network, decision tree, random forest, Bagging + logistic regression, Bagging + artificial neural network, AdaBoost, Gradient Boost, LightGBM, CatBoost, Naive Bayes or XGBoost.

[0032] The present invention also provides a diabetes prediction system based on ensemble learning, comprising:

[0033] An input module, used for inputting characteristic data of patients;

[0034] A prediction module, used to predict whether a patient has diabetes by using a machine learning model, wherein the machine learning model is obtained by using the above-mentioned diabetes prediction model construction method;

[0035] The output module is used to output the prediction results of the prediction module.

[0036] The present invention also provides a computer-readable storage medium, on which is stored: a computer program for implementing the above-mentioned diabetes prediction model construction method, or for implementing the above-mentioned diabetes prediction system.

[0037] In the model training stage, the present invention generates new positive samples based on the mean vector and covariance matrix of the positive samples, so that the number of negative and positive samples in the training set is balanced, thereby effectively solving the problem of sample imbalance in the diabetes data set and effectively improving the prediction performance of the diabetes prediction model. In a preferred embodiment, the present invention further provides a method for integrating different types of machine learning classifiers to further improve the classification accuracy of the model. The present invention can effectively improve the prediction performance of the diabetes prediction model and has a good application prospect.

[0038] Obviously, according to the above contents of the present invention, in accordance with common technical knowledge and customary means in the art, without departing from the above basic technical ideas of the present invention, other various forms of modification, replacement or change may be made.

[0039] The above contents of the present invention are further described in detail below through specific implementation methods in the form of embodiments. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following examples. All technologies realized based on the above contents of the present invention belong to the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is the flow chart of the integrated learning model;

[0041] Figure 2 This is a flow chart of a single classifier model;

[0042] Figure 3 It is a schematic diagram of a two-layer 5-fold cross method;

[0043] Figure 4 This is a diagram illustrating the confusion matrix;

[0044] Figure 5 Flowchart for solving the sample class imbalance problem. DETAILED DESCRIPTION

[0045] It should be noted that the algorithms of data collection, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structure, circuit connection, etc. not specifically described can all be implemented through the contents disclosed in the prior art.

[0046] Example 1 Method for constructing a diabetes prediction model based on ensemble learning

[0047] This embodiment provides a method for constructing a diabetes prediction model. Figure 1 As shown, the specific steps include:

[0048] First, the Pima-Idian dataset is preprocessed and feature screened. Then, the training set and test set are divided by stratified sampling and 5-fold crossover. The positive samples in the training set are processed to balance the negative and positive sample classes. The processed training set is then used to train six machine learning models. The training process of the six single classifiers is as follows: Figure 2 As shown, finally, based on the weighted rule proposed in this embodiment, voting is performed to obtain an average model and perform evaluation.

[0049] Specifically, the characteristic data of diabetes were first preprocessed by filling missing values, handling outliers, and standardizing data. Among them, the missing value processing method adopted the downward filling method in the adjacent value filling method, that is, the value of the previous sample was used to fill in from top to bottom, and the samples that were still missing after the downward filling were deleted. The outlier processing operation adopted the box plot method principle, that is, the theoretical upper and lower limits of each variable were calculated through the upper and lower quartiles, as shown in formula (1), and then the outliers outside the theoretical upper and lower limits were excluded, as shown in formula (2).

[0050]

[0051] In formula (1), IQR represents the distance between the upper quartile Q3 and the lower quartile Q1, and LB and UB are defined as the theoretical lower bound and theoretical upper bound of the characteristic variable, respectively.

[0052]

[0053] Formula (2) shows the outlier rejection method used in this embodiment, that is, the theoretical upper and lower bounds are calculated based on the 75% quantile and the 25% quantile of each feature column, and the sample data outside the bounds are judged as outliers and deleted; the remaining normal samples within the upper and lower bounds are retained.

[0054] For data normalization, this embodiment adopts the extreme value normalization operation shown in formula (3).

[0055]

[0056] In formula (3), Max and Min are the maximum and minimum values ​​of the corresponding feature columns respectively. This method can map each feature column data to the interval [0, 1] and can retain the distribution of the original data to a large extent.

[0057] Secondly, this technology uses three methods, namely principal component analysis (PCA), independent component analysis (ICA) and correlation coefficient (Corr), to screen the characteristic variables, and selects the optimal method based on the numerical experimental results.

[0058] The feature screening algorithm process of PCA is as follows:

[0059] Input: The feature matrix X (n×m dimensions) with m samples and n variables in the original data

[0060] Output: Feature matrix Y after dimensionality reduction (k×m dimensions)

[0061] 1. Accept the input matrix X and calculate the mean of X row by row:

[0062]

[0063] 2. Calculate the sample covariance matrix:

[0064]

[0065] 3. Perform eigendecomposition on the covariance matrix:

[0066] C=PDP -1

[0067] Among them, P is the eigenvector matrix and D is the diagonal matrix composed of eigenvalues.

[0068] 4. Sort the eigenvalues ​​in descending order, select the eigenvectors corresponding to the first k largest eigenvalues, and form a new projection matrix W n×k

[0069] 5. Reduce the dimension of the feature matrix according to the projection matrix W:

[0070] Y=W T X, Y∈R k×m

[0071] 6. Output the matrix Y after feature screening

[0072] The feature screening algorithm process of ICA is as follows:

[0073] Input: Feature matrix X in the original data

[0074] Output: Feature matrix Y after dimensionality reduction operation

[0075] 1. Define the calculation formula G(X) of negative entropy as shown in formula (2-5) as the objective function

[0076] 2. Initialize the ICA formula H = W T The unmixing matrix W in X, where H represents the separated independent component matrix

[0077] 3. Repeat the following process until W does not change (i.e. FastICA algorithm):

[0078] ·W *=mean(X·G(W T ·X))-mean(G′(W T ·X))

[0079] Among them, G' is the first-order derivative of the objective function G

[0080] W=W * / ||W * ||

[0081] 4. Reduce the dimension of the feature matrix according to the projection matrix W:

[0082]

[0083] 5. Output the matrix Y after feature screening

[0084] Corr's feature screening algorithm process is:

[0085] Input: feature matrix X and label column L in the original data

[0086] Output: Feature matrix Y after dimensionality reduction operation

[0087] 1. According to the calculated coefficient of determination of each variable, i = 1, 2..., n:

[0088]

[0089] 2. Sort r, select the first k largest coefficients of determination, and retain the corresponding k characteristic variables

[0090] 3. Output the retained feature matrix and label column

[0091] In order to improve the prediction accuracy of the model and verify the stability of the model, this embodiment adopts a two-layer 5-fold cross structure. The first layer is to divide the training set and the test set, and verify the stability of the model by calculating the average results and standard deviations of the indicators; and the second layer is to select the verification set from the training set again by 5-fold cross and use the grid search method to perform parameter tuning to improve the model through parameter tuning. The structural diagram is shown in Figure 3 As shown in the figure, the outer loop first uses the stratified sampling method to divide the original data set into five parts, so that the ratio of negative and positive samples in each data set is similar, and then uses the 5-fold cross-validation idea to divide the five samples into training set and sample set in a ratio of 4:1; the inner loop uses the 5-fold cross-validation method to divide each fold of the training set in the outer loop into five parts again, and takes one of them as the validation set, thereby realizing the hyperparameter tuning of the model.

[0092] This embodiment is based on the confusion matrix ( Figure 4) The average performance (AVG) of the five evaluation indicators shown in Table 1 is selected to evaluate the prediction effect of each model. The calculation method of the average performance AVG_M of any indicator M is shown in formula (4).

[0093]

[0094] K=5 in formula (4) indicates the number of crossovers between the training set and the test set in the outer loop. Formula (4) shows that the average performance of the indicator is the mean ± standard deviation of each indicator in the outer 5-fold crossover process. This technology uses the average performance and standard deviation of the indicator to evaluate the model, which is conducive to judging the stability of the model.

[0095] Table 1 Selection of model evaluation indicators

[0096]

[0097]

[0098] In order to solve the common sample class imbalance problem in diabetes prediction, this embodiment proposes a method flow for generating minority class samples, such as Figure 5 First, we need to find and extract the positive (minority class) samples in the training set, randomly generate positive samples that obey the normal distribution according to the mean vector and covariance matrix of the positive samples in the training set, as shown in formula (5), and add the newly generated positive samples to the training set. Repeat this process until the ratio of negative to positive samples reaches 1:1.

[0099] x=m+N(0,C) (5)

[0100] In formula (5), m and C are the mean vector and covariance matrix of the positive samples in the training set, respectively, N(0, C) is a multidimensional normal random variable with a mean vector of 0 and a covariance matrix of C, and x is a newly generated positive sample. This formula can randomly generate new samples that are close to the average positive sample, thus balancing the proportion of the two types of samples.

[0101] This embodiment uses the above method to process data and adopts Figure 2 The process shown is used to train six existing machine learning algorithms, including K nearest neighbor (KNN), decision tree (DT), random forest (RF), AdaBoost (AB), naive Bayes (NB) and XGBoost (XB), and perform model evaluation to select the optimal single classifier algorithm.

[0102] In order to further improve the prediction effect of the existing algorithm, this technology uses Figure 1The model training method shown combines the above six algorithms, and the weighted voting rules of the model are as follows:

[0103] 1) Calculate six optimal single classifiers G m The misprediction error rate (i.e. the proportion of the number of samples with misprediction to the total number of samples) is shown in formula (3-6):

[0104]

[0105] In formula (6), I is the indicative function, and N is the total number of prediction samples (m=1, 2, ..., 6).

[0106] 2) Calculate the weight of each classifier:

[0107]

[0108] Formula (7) achieves the goal of assigning a small weight to a classifier with a high error rate and a larger weight to a classifier with a low error rate.

[0109] 3) Normalize the weight of a single classifier:

[0110]

[0111] 4) Perform weighted voting (Soft Voting) on ​​the classification results of the single classifier according to the weight calculated by formula (8), and define the voting result as the prediction result of the ensemble learning model.

[0112] The models that are weighted and fused using the above method include single classifier models (KNN, NB, DT), Bagging series ensemble learning models (RF), and Boosting series ensemble learning models (AB, XB), which improve the accuracy and reliability of the system.

[0113] In the Pima-Indian diabetes prediction public data set used in this embodiment, the construction process and prediction performance results of each step are as follows:

[0114] 1) The data were preprocessed by filling missing values ​​(P), rejecting outliers (Q) and standardizing data (R), and then the best feature screening method was selected by voting based on the top three results of AUC mean ± standard deviation in Table 2, which was the screening method based on the correlation coefficient (Corr) method to retain 6 feature variables.

[0115] Table 2 Selection of optimal feature screening method

[0116]

[0117] 2) By Figure 2The algorithm flow chart shows that six single classifier models, including K nearest neighbor (KNN), decision tree (DT), random forest (RF), AdaBoost (AB), naive Bayes (NB) and XGBoost (XB), were trained and the results shown in Table 3 were obtained.

[0118] Table 3 Experimental results of single classifier parameter tuning

[0119]

[0120] 3) According to the results of AUC mean ± standard deviation in Table 3, the single classifier algorithms are ranked as follows: NB>AB>KNN>XB>RF>DT, and the ranking results are ranked according to Figure 1 The process shown integrates two, three, or even six single classifier algorithms to obtain the results shown in Table 4. The optimal integrated learning algorithm is selected from the results in Tables 4 and 5 as NB+AB+KNN+XB+RF.

[0121] Table 4 The first screening of the optimal ensemble learning classifier

[0122]

[0123]

[0124] Table 5 Second screening of the best ensemble learning classifier

[0125]

[0126] 4) Similarly, according to the results in Tables 6 and 7, the best single classifier algorithm can be selected as AB.

[0127] Table 6 The first screening of the best single classifier

[0128]

[0129] Table 7 The second screening of the best single classifier

[0130]

[0131]

[0132] 5) Table 8 compares the performance results of the optimal single classifier algorithm AB and the optimal ensemble learning algorithm NB+AB+KNN+XB+RF, and concludes that the ensemble learning algorithm has better performance results in 3 of the 5 selected indicators, thereby verifying that the ensemble learning method proposed in this embodiment is more effective and stable.

[0133] Table 8 Comparison of the results of the best single classifier and ensemble learning classifier

[0134]

[0135] 6) Table 9 compares the performance results of the optimal ensemble learning algorithm NB+AB+KNN+XB+RF with and without sample balancing processing, and concludes that the algorithm after sample balancing processing has better performance results in 4 of the 5 selected indicators, thereby verifying that the sample balancing processing method proposed in this embodiment is more effective.

[0136] Table 9 Comparison of results before and after sample balancing for NB+AB+KNN+XB+RF classifier

[0137]

[0138] Example 2 Diabetes prediction system based on ensemble learning

[0139] The system of this embodiment includes:

[0140] An input module, used for inputting characteristic data of patients;

[0141] A prediction module, used to predict whether a patient has diabetes by using a machine learning model, wherein the machine learning model is obtained by the diabetes prediction model construction method of Example 1;

[0142] The output module is used to output the prediction results of the prediction module.

[0143] It can be seen from the above embodiments that the present invention provides a new method for constructing a diabetes prediction model, which can effectively improve the prediction performance of the prediction model by balancing sample data and integrating single classifiers, and has a good application prospect.

Claims

1. A method for constructing a diabetes prediction model based on ensemble learning, characterized in that: The steps include: Step 1: preprocess and feature screen the data set; Step 2, divide the training set and test set; Step 3: Find and extract positive samples in the training set, randomly generate positive samples that obey the normal distribution according to the mean vector and covariance matrix of the positive samples in the training set, and add the newly generated positive samples to the training set, and repeat until the ratio of negative and positive samples reaches a balance; Step 4: Use the processed training set machine learning model.

2. The method for constructing a diabetes prediction model according to claim 1, characterized in that: In step 1, the preprocessing step includes: missing value filling, outlier processing and data standardization operations.

3. The method for constructing a diabetes prediction model according to claim 2, characterized in that: The missing value filling method is: using the downward filling method in the adjacent value filling method, and deleting the samples that are still missing after the downward filling; The outlier processing adopts the box plot method principle, and the theoretical upper and lower bounds of each variable are calculated through the upper and lower quartiles, and outliers outside the theoretical upper and lower bounds are excluded.

4. The method for constructing a diabetes prediction model according to claim 1, characterized in that: In step 1, the feature screening method is selected from: principal component analysis, independent component analysis or a method based on correlation coefficient.

5. The method for constructing a diabetes prediction model according to claim 1, characterized in that: In step 2, the method of dividing the training set and the test set is stratified sampling and 5-fold crossover method.

6. The method for constructing a diabetes prediction model according to claim 1, characterized in that: In step 3, the formula for randomly generating positive samples that follow a normal distribution is: x=m+N(0,C) Where m and C are the mean vector and covariance matrix of the positive samples in the training set, respectively; N(0,C) is a multidimensional normal random variable with a mean vector of 0 and a covariance matrix of C; and x is a newly generated positive sample.

7. The method for constructing a diabetes prediction model according to claim 1, characterized in that: In step 4, the machine learning model is obtained by fusing a single classifier algorithm through a weighted voting rule, wherein the weighted voting rule includes: 1) Calculate a single classifier G m The misprediction error rate e m , the calculation formula is: Among them, I is the indicative function, N is the total number of prediction samples, and y i is the sample label value, is the model prediction value; 2) Calculate the weight ω of each single classifier m , the calculation formula is: 3) Normalize the weights of all single classifiers, and the calculation formula is: 4) Perform weighted voting on the classification results of the single classifier according to the normalized weights, and define the voting result as the prediction result of the integrated machine learning model.

8. The method for constructing a diabetes prediction model according to claim 7, characterized in that: The single classifier algorithm is selected from at least two of the following algorithms: K nearest neighbor, support vector machine, BP neural network, decision tree, random forest, Bagging + logistic regression, Bagging + artificial neural network, AdaBoost, Gradient Boost, LightGBM, CatBoost, Naive Bayes or XGBoost.

9. A diabetes prediction system based on ensemble learning, characterized in that: include: An input module, used for inputting characteristic data of patients; A prediction module, used to predict whether a patient has diabetes by using a machine learning model, wherein the machine learning model is obtained by using the diabetes prediction model construction method according to any one of claims 1 to 8; The output module is used to output the prediction results of the prediction module.

10. A computer-readable storage medium, characterized in that: Stored thereon is: a computer program for implementing the diabetes prediction model construction method described in any one of claims 1 to 8, or for implementing the diabetes prediction system described in claim 9.

Images