Double-index machine learning model precision discrimination method based on Kappa index and F1 score
The dual-index method using Kappa index and F1 score for machine learning model evaluation addresses biased assessments in imbalanced datasets, offering a comprehensive and robust evaluation for model optimization and reliability.
Patent Information
- Application Number
- CN202510326389.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-19
- Publication Date
- 2025-07-15
AI Technical Summary
When evaluating machine learning models, it is difficult for a single indicator to comprehensively measure the classification performance of the model, especially in the case of uneven category distribution, it is easy to overestimate the model accuracy, and lacks comprehensive considerations for the Kappa index and F1 score.
A two-index machine learning model accuracy discrimination method based on Kappa index and F1 score was constructed. By calculating Pearson correlation coefficient and removing parameters with high correlation, combining confusion matrix, accuracy, recall, F1 score and Kappa index, the model accuracy level was determined.
It provides a more comprehensive model evaluation system, improves the accuracy and applicability of evaluation in the case of uneven category distribution, is suitable for practical application scenarios, and improves the performance and reliability of the model.
Smart Images
Figure CN120316458A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of machine learning algorithms, and in particular relates to a dual-index machine learning model accuracy discrimination method based on Kappa index and F1 score. Background Art
[0002] With the rapid development of artificial intelligence and big data technologies, machine learning models are increasingly used in the field of civil engineering, such as structural damage prediction, structural image recognition, and intelligent classification of structural types. However, evaluating the performance of machine learning models is a crucial issue, which is directly related to the reliability and practicality of the models. In classification tasks, common evaluation indicators include accuracy, precision, recall, F1 score, and Kappa index. Among them, although accuracy is intuitive, it can easily lead to misleading evaluation results when the category distribution is uneven. Therefore, it is difficult for a single indicator to comprehensively measure the classification performance of the model, and an evaluation method that comprehensively considers the accuracy and robustness of the model is urgently needed.
[0003] Kappa index and F1 score are two widely used indicators in the evaluation of classification models. Kappa index is mainly used to measure the consistency between the classification results of the model and the random classification results, which can effectively reduce the impact caused by the imbalance of category distribution. The F1 score is the harmonic mean of precision and recall, which can better reflect the overall classification performance of the model on positive and negative samples. However, most current achievements and applications usually use Kappa index or F1 score alone for model evaluation, lacking comprehensive consideration of the two, which may lead to a one-sided interpretation of model performance. Summary of the invention
[0004] The core purpose of this invention is to construct a dual-index machine learning model accuracy discrimination method based on Kappa index and F1 score to solve the problem of overestimation of model accuracy in existing evaluation methods in scenarios with uneven variable category distribution.
[0005] On the one hand, the present invention provides a method for determining the accuracy of a dual-index machine learning model based on the Kappa index and the F1 score, comprising the following steps:
[0006] S1: Based on the obtained data variable database, calculate the Pearson correlation coefficient between any two variables, and eliminate any parameter in the parameter pair with a correlation coefficient greater than 0.8;
[0007] S2: Based on the machine model algorithm, the confusion matrix, model precision and recall are calculated for the predicted parameters;
[0008] S3: Calculate the F1 score and Kappa index, further obtain the minimum F1 value of each type of predicted sample, and determine the accuracy level of the machine learning model based on the classification criteria.
[0009] Preferably, the S1 specifically includes:
[0010] Based on the obtained data variable database, calculate the Pearson correlation coefficient ρ between any two variables (denoted as x m and y m , (m = 1, 2, 3,..., N)), and ρ is calculated by the following formula:
[0011]
[0012] In the formula, are the means of the random variables x m and y m respectively. The value range of the Pearson correlation coefficient ρ is between -1 and +1. x m represents the value of the m-th x, and similarly for y m . If ρ = -1, it indicates a perfect negative linear correlation between x m and y m ; if ρ = +1, it indicates a perfect positive linear correlation between x m and y m ; if ρ = 0, it indicates that there is no linear correlation between x m and y m .
[0013] Observe the calculated correlation coefficient ρ, find the parameter pairs with ρ greater than 0.8, and remove any one parameter from the parameter pairs.
[0014] Preferably, the S2 specifically includes:
[0015] Based on the selected machine model algorithm, calculate the confusion matrix, the precision (P) and recall rate (R c ) of the model for the predicted parameters. The precision (P) and recall rate (R c ) are calculated by the following formula:
[0016]
[0017]
[0018] In the formula, C ii represents the number of samples with the true class being i and being correctly predicted as i, and similarly for C ji ; n is the number of classes; P i represents the proportion of samples that are truly in the i-th class among those predicted as the i-th class; R ci represents the proportion of samples that are actually in the i-th class and are correctly predicted as the i-th class. The precision P is used to measure the accuracy of the classification model when predicting samples as a specific class; the recall rate R cUsed to measure the coverage of a classification model for samples of a specific class.
[0019] Preferably, the S3 specifically includes:
[0020] Based on the calculated precision (P) and recall (R c ), calculate the F1 score (F1) and Kappa index (κ) of each class of predicted samples. It is necessary to substitute P i and R ci into the following formula to obtain the F1 score (F1) and Kappa index (κ) of the predicted samples of the corresponding class. To avoid too many variables, F1 and κ in the following calculation formula are not marked with subscripts and are represented by generalized variables:
[0021]
[0022] In the formula, p0 is the degree of consistency between the actually predicted classification result and the true structure; p e is the degree of consistency between the predicted classification result and the true structure under random circumstances. p0 and p e are solved by the following formula:
[0023]
[0024] Based on the obtained F1 and κ, further calculate the minimum value (min(F1)) of the F1 value of each class of predicted samples, and determine the accuracy level of the machine learning model according to the following classification criteria:
[0025]
[0026] The method provided by the present invention has the following beneficial effects:
[0027] By combining the robustness of the Kappa index and the accuracy of the F1 score, this method constructs a more comprehensive evaluation system, improving the accuracy and applicability of model performance discrimination. This method is not only applicable to datasets with balanced class distributions, but also can provide a more fair evaluation in the case of unbalanced class distributions, providing a more scientific basis for the optimization and selection of machine learning models.
[0028] 1. Solve the problem of unbalanced class distribution: Traditional evaluation methods are prone to overestimating the model accuracy when the data class distribution is unbalanced. However, the present invention can effectively avoid this problem by combining the Kappa index and the F1 score. The Kappa index takes into account the influence of random consistency and is applicable to scenarios with unbalanced data distributions, while the F1 score further improves the evaluation accuracy by balancing precision and recall.
[0029] 2. Provide comprehensive and fair model evaluation: A single metric often only reflects one aspect of model performance and is difficult to comprehensively evaluate the advantages and disadvantages of a model. Through the combination of two metrics, the present invention can comprehensively evaluate model performance from different perspectives (such as the balance of classification performance), providing a more comprehensive and fair accuracy evaluation criterion.
[0030] 3. Applicable to actual application scenarios: In actual applications, the situation of unbalanced data category distribution is very common. The method of the present invention can effectively address this problem, providing a more scientific and practical evaluation tool for machine learning models in actual scenarios, thereby improving the performance and reliability of the model in actual applications.
[0031] In summary, the method provided by the present invention not only solves the problem of overestimation of traditional evaluation methods when the category distribution is unbalanced, but also realizes a more comprehensive and robust model performance evaluation through the combination of two metrics, providing strong support for model optimization and actual applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a flowchart of the two-metric machine learning model accuracy discrimination method based on the Kappa index and F1 score of the present invention;
[0033] Figure 2 is a Pearson correlation coefficient graph in an embodiment of the present invention;
[0034] Figure 3 is a confusion matrix of the prediction of site categories by three machine learning methods in an embodiment of the present invention;
[0035] In the figure, (a) is the AdaBoost method; (b) is the XgBoost method; (c) is the random forest method; DETAILED DESCRIPTION OF THE EMBODIMENTS
[0036] The following further describes in detail the specific embodiments of the present invention with reference to the drawings and embodiments. The following embodiments are used to illustrate the present invention but not to limit the scope of the present invention.
[0037] As Figure 1 shown, the two-metric machine learning model accuracy discrimination method based on the Kappa index and F1 score provided in this embodiment selects the AdaBoost, XgBoost, and random forest algorithms as examples, which can more comprehensively compare and evaluate the advantages and disadvantages of different machine learning algorithms.
[0038] Regarding the two-metric machine learning model accuracy discrimination method based on the Kappa index and F1 score, the specific steps of the method are as follows:
[0039] As Figure 1As shown, the dual-index machine learning model accuracy determination method based on Kappa index and F1 score of this embodiment includes:
[0040] Step S1, based on the obtained data variable database, calculate the Pearson correlation coefficient between any two variables, and eliminate any parameter in the parameter pair with a correlation coefficient greater than 0.8;
[0041] ρ is calculated using the following formula:
[0042]
[0043] In the formula, are random variables x m and m The value range of Pearson correlation coefficient ρ is between -1 and +1. m represents the value of the mth x, and similarly y m If ρ=-1, it means x m and m There is a completely negative linear correlation between them; if ρ=+1, it means x m and m There is a completely positive linear correlation between them; if ρ=0, it means x m and m There is no linear correlation between them.
[0044] Table 2 shows the 11 selected data variables
[0045] Table 2 Data variables
[0046]
[0047]
[0048] The calculated Pearson correlation coefficient ρ between any two parameters is as follows: Figure 2 As shown, since the correlation coefficients ρ in this embodiment are all less than 0.8, there is no need to eliminate the variables.
[0049] Step S2:
[0050] Based on the three selected machine model algorithms: AdaBoost, XgBoost and random forest algorithm, the site category is selected as the prediction parameter, and the confusion matrix (such as Figure 3 As shown in the figure), the model's precision (P) and recall (R c ). Precision (P) and recall (R c ) is calculated by the following formula, and the calculation results are shown in Table 3:
[0051]
[0052] In the formula, C ii represents the number of samples with the true class being i and being correctly predicted as i. Similarly, C ji ; n is the number of classes; P i represents the proportion of samples that are truly in the i-th class among the samples predicted to be in the i-th class; R ci represents the proportion of samples that are actually in the i-th class and are correctly predicted as being in the i-th class. The precision P is used to measure the accuracy of the classification model when predicting samples as a specific class; the recall R c is used to measure the coverage of the classification model for samples of a specific class.
[0053] Table 3 Classification results of different machine learning models for site categories
[0054]
[0055] Step S3: Calculate the F1 score and the Kappa index, and determine the accuracy level of the machine learning model according to the division criteria.
[0056] The above step S3 further includes the following steps: Based on the obtained F1 and κ, further calculate the minimum value of the F1 value of each class of predicted samples (min(F1)). The division criteria for determining the accuracy level of the machine learning model are shown in Table 4 below. The calculation results of the F1 score and the Kappa index and the determination results of the prediction model are shown in Table 5:
[0057] Table 4 Model accuracy division criteria based on [κ, min(F1)]
[0058]
[0059]
[0060] Table 5 Model accuracy based on [κ, min(F 1,v )]
[0061]
[0062] The above is only the preferred embodiment of the present invention and is not used to limit the present invention. It should be noted that for those of ordinary skill in the art of this technology, without departing from the technical principles of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.
Claims
1. A method for discriminating the accuracy of a dual-index machine learning model based on the Kappa index and the F1 score, characterized in that, The steps include: S1: Based on the obtained data variable database, calculate the Pearson correlation coefficient between any two variables, and eliminate any parameter in the parameter pair with a correlation coefficient greater than 0.8; S2: Based on the machine model algorithm, the confusion matrix, model precision and recall are calculated for the predicted parameters; S3: Calculate the F1 score and Kappa index, further obtain the minimum F1 value of each type of predicted sample, and determine the accuracy level of the machine learning model based on the classification criteria.
2. A method for discriminating the accuracy of a dual-index machine learning model based on the Kappa index and the F1 score according to claim 1, characterized in that The S1 specifically includes: Based on the obtained data variable database, calculate the Pearson correlation coefficient ρ between any two variables (denoted as x m and y m , (m = 1, 2, 3, …, N)), and ρ is calculated using the following formula: Wherein, are the mean values of the random variables x m and y m respectively. The value range of the Pearson correlation coefficient ρ is between -1 and +1; x m represents the value of the m-th x, and similarly y m . If ρ = -1, it indicates a perfect negative linear correlation between x m and y m ; if ρ = +1, it indicates a perfect positive linear correlation between x m and y m ; if ρ = 0, it indicates that there is no linear correlation between x m and y m . Observe the calculated correlation coefficient ρ, find parameter pairs with ρ greater than 0.8, and eliminate any parameter in the parameter pair.
3. A method for discriminating the accuracy of a dual-index machine learning model based on the Kappa index and the F1 score according to claim 1, characterized in that, The S2 specifically includes: Based on the selected machine model algorithm, calculate the confusion matrix, the precision (P) and recall rate (R) of the model for the predicted parameters c ), the precision (P) and recall rate (R c ) are calculated by the following formula: Wherein, C ii represents the number of samples with the true class being i and being correctly predicted as i. Similarly, C ji ; n is the number of classes; P i represents the proportion of samples that are actually in the i-th class among the samples predicted as the i-th class; R ci represents the proportion of samples that are actually in the i-th class and are correctly predicted as the i-th class; The precision P is used to measure the accuracy of the classification model when predicting samples as a certain specific class; The recall rate R c is used to measure the coverage of the classification model for samples of a certain specific class.
4. A method for discriminating the accuracy of a dual-index machine learning model based on the Kappa index and the F1 score according to claim 1, characterized in that, The S3 includes: Based on the calculated precision (P) and recall (R c ), calculate the F1 score (F1) and Kappa index (κ) of each class of predicted samples. It is necessary to substitute P i and R ci into the following formula to obtain the F1 score (F1) and Kappa index (κ) of the predicted samples of the corresponding class. To avoid too many variables, the F1 and κ in the following calculation formula are not marked with subscripts and are represented by generalized variables: Wherein, p0 is the degree of consistency between the actually predicted classification result and the true structure; p e is the degree of consistency between the classification result predicted under random circumstances and the true structure; p0 and p e are solved by the following formula: Based on the obtained F1 and κ, the minimum F1 value (min(F1)) of each type of predicted sample is further calculated, and the accuracy level of the machine learning model is determined according to the classification criteria in the following table: