Efficient comparison method for cost-sensitive binary classification model
By constructing a unified evaluation curve and optimization line, the problem of repetitive calculations in traditional methods is solved, enabling fast and visualized model selection. This method is applicable to dynamic adversarial domains and improves the efficiency and adaptability of model selection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CNGC INST NO 206 OF CHINA ARMS IND GRP
- Filing Date
- 2026-02-28
- Publication Date
- 2026-06-16
AI Technical Summary
Traditional cost-sensitive binary classification model evaluation methods require frequent calculation of the optimal threshold when facing dynamic adversarial domains, resulting in a large amount of repetitive calculation work and difficulty in quickly responding to changes in cost weights, which affects the efficiency and adaptability of model selection.
By constructing a unified evaluation curve and introducing an optimization line with a slope of b, the optimal model can be found by directly shifting the evaluation curve, avoiding repeated calculations and quickly adapting to changes in cost weights.
It significantly improves model selection efficiency, reduces computational complexity, provides intuitive visualization and evaluation capabilities, adapts to dynamic adversarial application scenarios, and has simple logic that is easy to integrate, making it suitable for a variety of mainstream algorithms.
Smart Images
Figure CN122221059A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this application relate to the field of intelligent algorithm model evaluation technology, and in particular to an efficient comparison method for cost-sensitive binary classification models. Background Technology
[0002] With the widespread application of cost-sensitive binary classification learning in the real world, it is increasingly integrated into fields such as anomaly detection, medical diagnosis, financial risk control, and public safety. These fields all exhibit cost asymmetry, meaning the cost of misclassification is far from equal. This makes traditional evaluation methods based on metrics such as accuracy, recall, and F1 score ineffective in closely aligning model decisions with real-world value, and thus failing to support the selection of the optimal model for specific scenarios. Therefore, it is necessary to research suitable comparison methods for cost-sensitive binary classification models.
[0003] The core of cost-sensitive binary classification model evaluation lies in minimizing the overall misclassification cost. The decision threshold, as the boundary between positive and negative classes, is a key factor affecting model performance. The decision threshold has two characteristics: first, under the same weight allocation, different models have different optimal decision thresholds; second, under different weight allocations, the optimal decision threshold for the same model is different. Traditional model selection methods require finding the optimal threshold for each model under given weights, calculating its overall misclassification cost, and then comparing them. Once the cost weights change, it is necessary to find the optimal threshold for each model again. For dynamic adversarial domains such as cybersecurity and stock trading, where cost weights fluctuate significantly, this involves a large amount of repetitive computation, posing a significant challenge to the selection and comparison of optimal models in time-varying environments. Summary of the Invention
[0004] To address the aforementioned technical issues, embodiments of this application propose an efficient comparison method for cost-sensitive binary classification models. This method can quickly determine the optimal model without calculating the optimal classification threshold for each model, thereby reducing the amount of repetitive computation in dynamic adversarial domains where cost weights fluctuate significantly.
[0005] To achieve the above objectives, embodiments of this application propose an efficient comparison method for cost-sensitive binary classification models, comprising: collecting sample data based on the current application scenario and dividing it into a training set and a test set; selecting multiple different binary classification algorithms and training each algorithm using the training set to obtain multiple binary classification models to be compared; wherein, there is a one-to-one correspondence between the binary classification algorithm and the binary classification model; traversing each binary classification model, predicting on the test set for the current binary classification model to obtain the probability that each test sample belongs to a positive example, and gradually decreasing from the most conservative decision threshold, sequentially calculating the false positive and false negative values corresponding to each decision threshold. The values of negative examples are calculated, and based on the values of all false positive examples and false negative examples, the evaluation curve corresponding to the current binary classification model is obtained. After traversing through all the binary classification models, the evaluation curves corresponding to each binary classification model are obtained and plotted on the same evaluation curve graph. Based on the current cost weight, an optimization line with a slope of b is plotted on the evaluation curve graph. The optimization line is located in the lower right corner of the evaluation curve graph and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. The optimization line is gradually shifted from the lower right corner to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model. This binary classification model is taken as the optimal binary classification model under the current cost weight.
[0006] To achieve the above objectives, embodiments of this application also propose an efficient comparison system for cost-sensitive binary classification models, comprising: a training set and test set construction module, a binary classification model training module, a unified evaluation curve plotting module, and a comparison optimization module; the training set and test set construction module is used to collect sample data based on the current application scenario and divide it into a training set and a test set; the binary classification model training module is used to select multiple different binary classification algorithms, train each binary classification algorithm using the training set, and obtain multiple binary classification models to be compared; wherein, the binary classification algorithm corresponds one-to-one with the binary classification model; the unified evaluation curve plotting module is used to traverse each binary classification model, predict the current binary classification model on the test set, obtain the probability that each test sample belongs to the positive example, and start from the most conservative decision The decision threshold is gradually reduced, and the values of false positives and false negatives corresponding to each decision threshold are calculated sequentially. Based on the calculated values of all false positives and false negatives, the evaluation curve corresponding to the current binary classification model is obtained. After traversal, the evaluation curves corresponding to each binary classification model are obtained and plotted on the same evaluation curve graph. The comparison optimization module is used to draw an optimization line with a slope of b on the evaluation curve graph based on the current cost weight. The optimization line is located in the lower right corner of the evaluation curve graph and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. The optimization line is gradually shifted from the lower right corner of the evaluation curve graph to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model in the evaluation curve graph. This binary classification model is taken as the optimal binary classification model under the current cost weight.
[0007] To achieve the above objectives, embodiments of this application also propose an electronic device, including: a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement an efficient comparison method for a cost-sensitive binary classification model as described above.
[0008] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables an efficient comparison method for a cost-sensitive binary classification model as described above.
[0009] Optionally, if the cost weights change, the method further includes: plotting a curve with a slope of [value missing] on the evaluation graph based on the latest cost weights. The latest optimization line is located in the lower right corner of the evaluation curve graph and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. The latest optimization line is gradually shifted from the lower right corner to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model. This binary classification model is then taken as the optimal binary classification model under the latest cost weight.
[0010] Optionally, let the set of selected binary classification algorithms be denoted as . , , For the selected number A binary classification algorithm, Let be the total number of binary classification algorithms, and let be the set of binary classification models obtained through training. , , To and The corresponding binary classification models; among them, the selected binary classification algorithms include LR algorithm, KNN algorithm, CT algorithm and NB algorithm, and the trained binary classification models include LR model, KNN model, CT model and NB model.
[0011] Optionally, the decision threshold can take values ranging from 1 to 2. The most conservative decision threshold, which is the largest decision threshold, is gradually reduced according to a preset step size until the most aggressive decision threshold, which is the smallest decision threshold, is reached.
[0012] Optionally, the most conservative decision threshold is 0.9, the most aggressive decision threshold is 0.1, and the preset step size is 0.1.
[0013] Optionally, the value of a false positive is recorded. The value of the false negative is Based on the calculated values of all false positives and false negatives, the evaluation curve corresponding to the current binary classification model is obtained, including: the calculated values of all... The coordinate points are marked in the coordinate system and connected in order to obtain the evaluation curve corresponding to the current binary classification model; The slope b of the optimization line is determined based on the current cost weight. , Indicates the current Misclassification cost weights Indicates the current Misclassification cost weight.
[0014] Optionally, if the optimization line is tangent to the evaluation curves of multiple binary classification models while gradually shifting from the lower right corner to the upper left corner, then all of these binary classification models are taken as the optimal binary classification model under the current cost weight, or a binary classification model is randomly selected from these binary classification models as the optimal binary classification model under the current cost weight.
[0015] This application proposes an efficient comparison method for cost-sensitive binary classification models, which has the following advantages compared with traditional model comparison (selection) methods.
[0016] First, it significantly improves model selection efficiency and reduces computational complexity. Traditional model comparison methods require recalculating the total misclassification cost for each candidate model at different thresholds when cost weights change, necessitating tedious comparisons to determine the optimal model. This application constructs a unified evaluation curve and introduces an optimization line, transforming the process of "finding the optimal model" into a geometric tangency problem. When cost weights change, there is no need to recalculate the prediction results of each model or traverse thresholds; simply adjusting the slope of the optimization line and shifting it is sufficient to quickly lock in the optimal model, greatly reducing redundant computations. This is particularly suitable for real-time systems requiring frequent model switching.
[0017] Second, it provides an intuitive and visual model evaluation capability. This application maps the performance of multiple binary classification models onto the same evaluation curve. This visualization method makes the advantages and disadvantages of different algorithms under different cost weights clear at a glance. Decision-makers can intuitively observe how model performance changes with the decision threshold, thereby assisting in more scientific model selection.
[0018] Third, it effectively adapts to dynamic adversarial application scenarios. In dynamic adversarial fields such as cybersecurity, stock trading, and financial risk control, the cost of misclassification (e.g., the ratio of missed loss to false positive loss) is often not statically fixed but fluctuates over time or with the environment. Traditional model comparison (selection) methods struggle to quickly respond to such dynamic changes in weights. This application can instantly match the optimal model under different weights through simple geometric calculations, significantly enhancing the robustness and practicality of the model selection method in time-varying environments.
[0019] Fourth, it has strong universality and low implementation cost. This application does not rely on the internal structure of a specific binary classification algorithm, but only requires the model to output the probability of positive examples. It can be widely used in a variety of mainstream algorithms. Its implementation logic is simple and easy to integrate into existing machine learning platforms or electronic devices, and it has high engineering application value. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.
[0021] Figure 1 This is a flowchart of an efficient comparison method for cost-sensitive binary classification models provided in one embodiment of this application; Figure 2 This is a schematic diagram provided in one embodiment of the present application, showing how the total cost of model misclassification changes with b under different decision thresholds; Figure 3 This is a schematic diagram provided in one embodiment of the present application, illustrating how the total cost of model misclassification varies with b and the decision threshold for different models; Figure 4 This is an evaluation curve diagram of four typical binary classification models provided in one embodiment of this application; Figure 5 This is a schematic diagram of the optimization line (b=0.6) provided in one embodiment of this application; Figure 6 This is a schematic diagram showing the tangency of the evaluation curves of four typical binary classification models to the optimization line (b=0.6) provided in one embodiment of this application. Figure 7 This is a schematic diagram of the optimization line (b=0.85) provided in one embodiment of this application; Figure 8This is a schematic diagram showing the tangency of the evaluation curves of four typical binary classification models to the optimization line (b=0.85) provided in one embodiment of this application. Figure 9 This is a schematic diagram of the optimal model corresponding to different b values provided in one embodiment of this application; Figure 10 This is a schematic diagram of the structure of an efficient comparison system for a cost-sensitive binary classification model provided in another embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.
[0023] One embodiment of this application proposes an efficient comparison method for cost-sensitive binary classification models. The implementation details of the efficient comparison method for cost-sensitive binary classification models proposed in this embodiment are described below. The following details are provided for ease of understanding and are not necessary for implementing this solution.
[0024] The specific process of the efficient comparison method for cost-sensitive binary classification models proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 11: Collect sample data based on the current application scenario and divide it into training set and test set.
[0025] In practical implementation, this embodiment can be applied to various scenarios such as network security and stock trading. The cost weight fluctuations differ in different scenarios, so different training and test sets need to be constructed for different application scenarios to better compare models. For the current application scenario, we need to collect a large amount of sample data based on the current application scenario and divide it into training and test sets according to a preset ratio.
[0026] In one example, we prepare the Stalog dataset according to the application scenario and randomly divide it into training and test sets in a 3:1 ratio.
[0027] Step 12: Select multiple different binary classification algorithms, and train each binary classification algorithm using the training set to obtain multiple binary classification models to be compared. The binary classification algorithms and binary classification models correspond one-to-one.
[0028] In the specific implementation, after constructing the training set and test set, multiple different binary classification algorithms can be selected (chosen according to actual needs). The training set is used to train each binary classification algorithm separately, resulting in multiple binary classification models to be compared. There is a one-to-one correspondence between the binary classification algorithm and the binary classification model.
[0029] Let the set of selected binary classification algorithms be denoted as . , , For the selected number A binary classification algorithm, Let be the total number of binary classification algorithms, and let be the set of binary classification models obtained through training. , , To and The corresponding binary classification model.
[0030] The selected binary classification algorithms include LR (linear regression), KNN (k-nearest neighbors), CT (decision tree), and NB (naive Bayes). The trained binary classification models include LR, KNN, CT, and NB models.
[0031] Step 13: Traverse each binary classification model, predict the probability of each test sample being a positive example on the test set for the current binary classification model, and gradually decrease the decision threshold from the most conservative one. Calculate the false positive and false negative values corresponding to each decision threshold in turn, and obtain the evaluation curve corresponding to the current binary classification model based on the calculated values of all false positive and false negative examples. After traversing the entire set of binary classification models, obtain the evaluation curves corresponding to each binary classification model and plot them on the same evaluation curve graph.
[0032] In the specific implementation, after training multiple binary classification models to be compared, the model comparison stage can be entered. Each binary classification model is traversed, and the current binary classification model is predicted on the test set to obtain the probability that each test sample belongs to a positive example. Then, starting from the most conservative decision threshold, the threshold is gradually lowered, and the false positive and false negative values corresponding to each decision threshold are calculated in turn. Based on the calculated values of all false positive and false negative examples, the evaluation curve corresponding to the current binary classification model is obtained. After the traversal is completed, the evaluation curves corresponding to each binary classification model can be obtained. At this time, the evaluation curves corresponding to each binary classification model need to be plotted on the same evaluation curve graph to prepare for model comparison.
[0033] In one example, the decision threshold ranges from [value missing]. The most conservative decision threshold, which is the largest decision threshold, is gradually reduced according to a preset step size until the most aggressive decision threshold, which is the smallest decision threshold, is reached.
[0034] In one example, the most conservative decision threshold is set to 0.9, the most aggressive decision threshold can be set to 0.1, and the preset step size is set to 0.1. Under this setting, there are a total of 9 different decision thresholds, namely 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2 and 0.1.
[0035] In one example, the value of a false positive is recorded. The value of the false negative is When plotting the evaluation curve for the current binary classification model, it is necessary to consider all the calculated values. The coordinate points are marked in the coordinate system and connected in order to obtain the evaluation curve corresponding to the current binary classification model.
[0036] Step 14: Based on the current cost weights, draw an optimization line with a slope of b on the evaluation curve graph. The optimization line is located in the lower right corner of the evaluation curve graph and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. Gradually shift the optimization line from the lower right corner to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model. Take this binary classification model as the optimal binary classification model under the current cost weights.
[0037] In the specific implementation, after obtaining the evaluation curve, it is necessary to draw an optimization line (also known as b-Line) with a slope of b on the evaluation curve based on the current cost weight. The optimization line is located in the lower right corner of the evaluation curve and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. The optimization line is gradually shifted from the lower right corner to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model. We take this binary classification model as the optimal binary classification model under the current cost weight.
[0038] It is important to note that the slope 'b' of the optimization line is determined based on the current cost weights; that is, 'b' changes as the current cost weights change. , Indicates the current Misclassification cost weights Indicates the current Misclassification cost weight.
[0039] In one example, if the optimization line is tangent to the evaluation curves of multiple binary classification models while gradually shifting from the lower right corner to the upper left corner, then all of these binary classification models can be used as the optimal binary classification model under the current cost weight, or a binary classification model can be randomly selected from these models as the optimal binary classification model under the current cost weight.
[0040] In one example, when the cost weights change, the old optimization line is no longer applicable. We need to plot a curve with a slope of [slope value missing] based on the latest cost weights. The latest optimization line is located in the lower right corner of the evaluation curve graph and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. The latest optimization line is gradually shifted from the lower right corner to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model. This binary classification model is then taken as the optimal binary classification model under the latest cost weight.
[0041] This embodiment proposes an efficient comparison method for cost-sensitive binary classification models, which has the following advantages compared with traditional model comparison (selection) methods.
[0042] First, it significantly improves model selection efficiency and reduces computational complexity. Traditional model comparison methods require recalculating the total misclassification cost for each candidate model at different thresholds when cost weights change, necessitating tedious comparisons to determine the optimal model. This embodiment transforms the process of "finding the optimal model" into a geometric tangency problem by constructing a unified evaluation curve and introducing an optimization line. When cost weights change, there is no need to recalculate the prediction results of each model or traverse thresholds; simply adjusting the slope of the optimization line and shifting it is sufficient to quickly lock in the optimal model, greatly reducing redundant calculations. This is particularly suitable for real-time systems requiring frequent model switching.
[0043] Second, it provides an intuitive and visual model evaluation capability. This embodiment maps the performance of multiple binary classification models onto the same evaluation curve. This visualization method makes the advantages and disadvantages of different algorithms under different cost weights clear at a glance. Decision-makers can intuitively observe how model performance changes with the decision threshold, thereby assisting in more scientific model selection.
[0044] Third, it effectively adapts to dynamic adversarial application scenarios. In dynamic adversarial fields such as cybersecurity, stock trading, and financial risk control, the cost of misclassification (such as the ratio of missed loss to false positive loss) is often not statically fixed, but fluctuates with time or environment. Traditional model comparison (selection) methods struggle to quickly respond to such dynamic changes in weights. This embodiment can instantly complete the optimal model matching under different weights through simple geometric calculations, significantly enhancing the robustness and practicality of the model selection method in time-varying environments.
[0045] Fourth, it has strong universality and low implementation cost. This embodiment does not rely on the internal structure of a specific binary classification algorithm, only requires the model to output the probability of positive examples, and can be widely used in a variety of mainstream algorithms. Its implementation logic is simple and easy to integrate into existing machine learning platforms or electronic devices, and it has high engineering application value.
[0046] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.
[0047] In one embodiment, to verify the effectiveness of the efficient comparison method for cost-sensitive binary classification models proposed in this application (hereinafter referred to as the method), we conducted relevant simulation experiments. Based on the Stalog dataset, we constructed a training set and a test set in a 3:1 ratio. We selected four typical binary classification algorithms: LR algorithm, KNN algorithm, CT algorithm, and NB algorithm. The decision threshold was gradually reduced from the most conservative 0.9 in steps of 0.1 until it reached the most aggressive 0.1.
[0048] Figure 2 and Figure 3 Based on the Stalog dataset, this paper demonstrates the relationship between different binary classification algorithms, decision thresholds, cost-weight ratios, and total cost of misclassification in two different ways. Figure 2 The diagram shows how the total misclassification cost of the LR, KNN, CT, and NB models varies with the slope b of the optimization line under different decision thresholds. The nine subplots represent nine different decision thresholds: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9. Figure 3 This shows how the total cost of misclassification varies with the slope b of the optimization line under different decision thresholds when different binary classification algorithms are applied. The four subgraphs represent the LR algorithm, KNN algorithm, CT algorithm, and NB algorithm, respectively.
[0049] from Figure 2 and Figure 3 As can be seen, for the same binary classification algorithm, the optimal decision threshold changes with the change of b, and for the same b, different binary classification algorithms correspond to different optimal decision thresholds.
[0050] Based on the above simulation conditions, the evaluation curve can be plotted as follows: Figure 4 As shown.
[0051] Figure 5 Given The optimization line is gradually shifted from the lower right corner to the upper left corner, first becoming tangent to the LR model (see [link]). Figure 6 Therefore, the LR model is The optimal binary classification model at that time.
[0052] Figure 7 Given The optimization line is gradually shifted from the lower right corner to the upper left corner, while simultaneously being tangent to both the LR and NB models (see [reference]). Figure 8 Therefore, both the LR model and the NB model are The optimal binary classification model at that time.
[0053] To verify whether the results of this method match the actual results, we presented the results through repeated calculations. The optimal decision threshold and the corresponding total misclassification cost for different values of b within the interval are calculated as follows: Figure 9 As shown. From Figure 9 It can be seen from this that, and The optimal decision result obtained at that time is completely consistent with the result obtained by this method.
[0054] Accordingly, another embodiment of this application proposes an efficient comparison system for cost-sensitive binary classification models. The details of this efficient comparison system for cost-sensitive binary classification models are described below. The following content is merely for ease of understanding and is not essential for implementing this solution. Figure 10 This is a schematic diagram of the structure of an efficient comparison system for cost-sensitive binary classification models proposed in this embodiment, including: a training set and test set construction module 21, a binary classification model training module 22, an evaluation curve unified plotting module 23, and a comparison optimization module 24.
[0055] The training set and test set construction module 21 is used to collect sample data based on the current application scenario and divide it into training set and test set.
[0056] The binary classification model training module 22 is used to select multiple different binary classification algorithms, train each binary classification algorithm using the training set, and obtain multiple binary classification models to be compared; wherein, the binary classification algorithm and the binary classification model correspond one-to-one.
[0057] The unified evaluation curve plotting module 23 is used to traverse each binary classification model, predict the current binary classification model on the test set, obtain the probability that each test sample belongs to the positive example, start from the most conservative decision threshold and gradually decrease it, calculate the false positive example value and false negative example value corresponding to each decision threshold in turn, and obtain the evaluation curve corresponding to the current binary classification model based on the calculated false positive example value and false negative example value. After traversal, the evaluation curves corresponding to each binary classification model are obtained and plotted on the same evaluation curve graph.
[0058] The comparison optimization module 24 is used to draw an optimization line with a slope of b on the evaluation curve graph based on the current cost weight. The optimization line is located in the lower right corner of the evaluation curve graph and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. The optimization line is gradually shifted from the lower right corner of the evaluation curve graph to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model in the evaluation curve graph. This binary classification model is then taken as the optimal binary classification model under the current cost weight.
[0059] It is worth noting that all modules involved in this embodiment are logical modules. In practical applications, a logical module can be a physical module, a part of a physical module, or an organic combination of multiple physical modules. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce modules that are not closely related to solving the technical problems proposed in this application. However, this does not mean that other modules are absent from this embodiment.
[0060] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.
[0061] Another embodiment of this application provides an electronic device, such as Figure 11 As shown, it includes a processor 31 and a memory 32. The memory 32 stores instructions that the processor 31 can execute. When the processor 31 is configured to execute the instructions, the electronic device can implement an efficient comparison method for a cost-sensitive binary classification model as described in the above method embodiment.
[0062] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.
[0063] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.
[0064] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables an efficient comparison method for a cost-sensitive binary classification model as described in the above method embodiments.
[0065] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.
[0066] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.
Claims
1. An efficient comparison method for cost-sensitive binary classification models, characterized in that, include: Sample data is collected based on the current application scenario and divided into training and test sets; Multiple different binary classification algorithms are selected, and each binary classification algorithm is trained using a training set to obtain multiple binary classification models to be compared; wherein, there is a one-to-one correspondence between the binary classification algorithm and the binary classification model. Iterate through each binary classification model, predict the probability of each test sample being a positive example on the test set, and gradually decrease the decision threshold from the most conservative one. Calculate the false positive and false negative values corresponding to each decision threshold in turn. Based on the calculated values of all false positive and false negative examples, obtain the evaluation curve corresponding to the current binary classification model. After the iteration is complete, obtain the evaluation curves corresponding to each binary classification model and plot them on the same evaluation curve graph. Based on the current cost weights, draw an optimization line with a slope of b on the evaluation curve graph. The optimization line is located in the lower right corner of the evaluation curve graph and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. Gradually shift the optimization line from the lower right corner to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model. Take this binary classification model as the optimal binary classification model under the current cost weights.
2. The efficient comparison method for cost-sensitive binary classification models according to claim 1, characterized in that, When the cost weights change, the method further includes: Based on the latest cost weights, a curve with a slope of [slope value missing] is plotted on the evaluation curve. The latest optimization line is located in the lower right corner of the evaluation curve graph and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. The latest optimization line is gradually shifted from the lower right corner to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model. This binary classification model is then taken as the optimal binary classification model under the latest cost weight.
3. The efficient comparison method for cost-sensitive binary classification models according to claim 1, characterized in that, Let the set of selected binary classification algorithms be denoted as . , , For the selected number A binary classification algorithm, Let be the total number of binary classification algorithms, and let be the set of binary classification models obtained through training. , , To and The corresponding binary classification model; The selected binary classification algorithms include LR, KNN, CT, and NB algorithms, and the trained binary classification models include LR, KNN, CT, and NB models.
4. The efficient comparison method for cost-sensitive binary classification models according to claim 1, characterized in that, The range of values for the decision threshold is: The most conservative decision threshold, which is the largest decision threshold, is gradually reduced according to a preset step size until the most aggressive decision threshold, which is the smallest decision threshold, is reached.
5. The efficient comparison method for cost-sensitive binary classification models according to claim 4, characterized in that, The most conservative decision threshold is 0.9, the most aggressive decision threshold is 0.1, and the preset step size is 0.
1.
6. The efficient comparison method for cost-sensitive binary classification models according to claim 1, characterized in that, The value of a false positive is... The value of the false negative is Based on the calculated values of all false positives and false negatives, the evaluation curve corresponding to the current binary classification model is obtained, including: All the calculations The coordinate points are marked in the coordinate system and connected in order to obtain the evaluation curve corresponding to the current binary classification model; The slope b of the optimization line is determined based on the current cost weight. , Indicates the current Misclassification cost weights Indicates the current Misclassification cost weight.
7. An efficient comparison method for cost-sensitive binary classification models according to any one of claims 1 to 6, characterized in that, If, during the process of gradually shifting the optimization line from the lower right corner to the upper left corner, it becomes tangent to the evaluation curves of multiple binary classification models, then all of these binary classification models are taken as the optimal binary classification model under the current cost weight, or a binary classification model is randomly selected from these models as the optimal binary classification model under the current cost weight.
8. An efficient comparison system for cost-sensitive binary classification models, characterized in that, include: The training and test set construction module is used to collect sample data based on the current application scenario and divide it into training and test sets; The binary classification model training module is used to select multiple different binary classification algorithms, train each binary classification algorithm using a training set, and obtain multiple binary classification models to be compared; wherein, there is a one-to-one correspondence between the binary classification algorithm and the binary classification model; The unified evaluation curve plotting module is used to traverse each binary classification model, predict the current binary classification model on the test set, obtain the probability that each test sample belongs to the positive example, start from the most conservative decision threshold and gradually decrease it, calculate the false positive and false negative values corresponding to each decision threshold in turn, and obtain the evaluation curve corresponding to the current binary classification model based on all the calculated false positive and false negative values. After traversing, the evaluation curves corresponding to each binary classification model are obtained and plotted on the same evaluation curve graph. The comparison optimization module is used to draw an optimization line with a slope of b on the evaluation curve graph based on the current cost weight. The optimization line is located in the lower right corner of the evaluation curve graph and does not intersect or tangent to the evaluation curve corresponding to any binary classification model. The optimization line is gradually shifted from the lower right corner of the evaluation curve graph to the upper left corner until it is tangent to the evaluation curve corresponding to any binary classification model in the evaluation curve graph. This binary classification model is then taken as the optimal binary classification model under the current cost weight.
9. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement an efficient comparison method for a cost-sensitive binary classification model as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement an efficient comparison method for cost-sensitive binary classification models as described in any one of claims 1 to 7.