Model performance evaluation method, device, equipment and storage medium

Abstract

The present invention discloses a model performance evaluation method, device, equipment and storage medium, which belongs to the field of computer information processing technology. The present invention obtains the first rejection rate of the new model of the experimental group and the ratio of the actual number of samples of the control group to the experimental group, calculates the restoration weight of the control group according to the first rejection rate and the ratio of the actual number of samples, obtains the number of bad samples in the control group, calculates the number of bad samples in the real total samples according to the restoration weight and the number of bad samples, and evaluates the model performance according to the real number of bad samples. By restoring the customers who have been rejected by the experimental group according to the restoration weight of the control group as the customer performance of the experimental group, it is possible to infer the quality of the rejected samples of the experimental group, and also restore the real bad sample ratio, so as to achieve an accurate evaluation of the performance of the new model.

Description

Technical Field

[0001] The present invention relates to the field of computer information processing technology, and in particular to a model performance evaluation method, device, equipment and storage medium. Background Art

[0002] In our daily risk management work, as our business evolves, we need to continuously iterate and optimize our strategies and models. Newly developed models often offer greater improvements to business performance, but these improvements are achieved through offline data analysis. Therefore, it's unwise to hastily replace an existing strategy model with a new one without conducting production testing. We need to test and compare the performance of the new and old strategy models to select the optimal one. A common technique used in this context is the champion / challenger experiment, also known as A / B testing.

[0003] When conducting A / B testing, existing technologies often use a control group of samples to evaluate the performance of a new model. However, using only the control group makes it difficult to obtain comprehensive and accurate evaluation results for the new model. Summary of the Invention

[0004] The main purpose of the present invention is to provide a model performance evaluation method, device, equipment and storage medium, aiming to solve the problem of inaccurate model performance evaluation in the prior art.

[0005] To achieve the above object, the present invention provides a model performance evaluation method, which includes the following steps:

[0006] Get the first rejection rate of the new model of the experimental group;

[0007] Obtaining the ratio of the actual number of samples of the control group to the number of samples of the experimental group;

[0008] Calculating the restoration weight of the control group according to the ratio of the first rejection rate to the actual number of samples;

[0009] Get the number of bad samples in the control group;

[0010] Calculating the true number of bad samples according to the restoration weight and the number of bad samples;

[0011] The model performance is evaluated based on the number of true bad samples.

[0012] Optionally, the step of calculating the restoration weight of the control group according to the first rejection rate and the ratio of the actual number of samples includes:

[0013] Calculating an original sample number ratio according to the first rejection rate and the actual sample number ratio;

[0014] The restored weight of the control group is obtained according to the original sample number ratio.

[0015] Optionally, before the step of obtaining the first rejection rate of the new model of the experimental group, the method further includes:

[0016] The experimental group and the control group were divided into two groups. The experimental group used the new model for online testing, and the control group directly went online for testing.

[0017] Optionally, the step of obtaining the ratio of the actual number of samples in the control group to the actual number of samples in the experimental group includes:

[0018] Obtain the expected number of control group samples;

[0019] Get the total number of samples and the sample pass rate;

[0020] The actual sample number ratio is calculated based on the expected control group sample number, the total sample number and the sample pass rate.

[0021] Optionally, the step of obtaining the sample pass rate includes:

[0022] Calculate the number of borrowers per day by collecting historical statistics on the number of daily business applications, the historical sample pass rate, and the historical approval rate;

[0023] The sample pass rate is calculated based on the number of borrowers on the historical day and the number of business applicants on the historical day.

[0024] Optionally, the step of obtaining the first rejection rate of the new model of the experimental group for customers includes:

[0025] Obtaining a preset second rejection rate;

[0026] Calculating the improvement of the new model after applying the second rejection rate;

[0027] Determining whether the lift exceeds a lift threshold;

[0028] If so, the second rejection rate is selected as the first rejection rate.

[0029] Optionally, the step of evaluating model performance according to the number of real bad samples includes:

[0030] Calculate the KS statistic according to the number of true bad samples;

[0031] According to the KS statistic, the degree of discrimination between good and bad samples of the new model is evaluated;

[0032] The performance evaluation result of the model is obtained based on the degree of differentiation between good and bad samples. In addition, to achieve the above purpose, the present invention also provides a model performance evaluation device, which includes: a first acquisition module for obtaining a first rejection rate of the new model of the experimental group;

[0033] The second acquisition module is used to obtain the ratio of the actual number of samples of the control group to the actual number of samples of the experimental group;

[0034] a first calculation module, configured to calculate a restoration weight of a control group according to the first rejection rate and the ratio of the actual number of samples;

[0035] The third acquisition module is used to obtain the number of bad samples in the control group;

[0036] A second calculation module is used to calculate the number of real bad samples according to the restoration weight and the number of bad samples in the control group;

[0037] An evaluation module is used to evaluate the model performance according to the number of real bad samples.

[0038] Optionally, the first calculation module is further configured to:

[0039] Calculating an original sample number ratio according to the first rejection rate and the actual sample number ratio;

[0040] The restored weight of the control group is obtained according to the original sample number ratio.

[0041] Optionally, the first acquisition module is further configured to:

[0042] The experimental group and the control group were divided into two groups. The experimental group used the new model for online testing, and the control group directly went online for testing.

[0043] Optionally, the second acquisition module is further configured to:

[0044] Obtain the expected number of control group samples;

[0045] Get the total number of samples and the sample pass rate;

[0046] The actual sample number ratio is calculated based on the expected control group sample number, the total sample number and the sample pass rate.

[0047] Optionally, the second acquisition module is further configured to:

[0048] Calculate the number of borrowers per day by collecting historical statistics on the number of daily business applications, the historical sample pass rate, and the historical approval rate;

[0049] The sample pass rate is calculated based on the number of borrowers on the historical day and the number of business applicants on the historical day.

[0050] Optionally, the first acquisition module is further configured to:

[0051] Obtaining a preset second rejection rate;

[0052] Calculating the improvement of the new model after applying the second rejection rate;

[0053] Determining whether the lift exceeds a lift threshold;

[0054] If so, the second rejection rate is selected as the first rejection rate.

[0055] Optionally, the evaluation module is further configured to:

[0056] Calculate the KS statistic according to the number of true bad samples;

[0057] According to the KS statistic, the degree of discrimination between good and bad samples of the new model is evaluated;

[0058] The performance evaluation result of the model is obtained based on the degree of differentiation between good and bad samples.

[0059] In addition, to achieve the above-mentioned purpose, the present invention also provides a model performance evaluation device, which includes: a memory, a processor, and a model performance evaluation program stored on the memory and executable on the processor, wherein the model performance evaluation program is configured to implement the steps of the model performance evaluation method described above.

[0060] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a model performance evaluation program is stored. When the model performance evaluation program is executed by a processor, the steps of the model performance evaluation method described above are implemented.

[0061] The embodiments of the present invention disclose a model performance evaluation method, apparatus, device, and storage medium. The method obtains the first rejection rate of the new model of the experimental group and the ratio of the actual number of samples of the control group to the experimental group, calculates the restoration weight of the control group based on the first rejection rate and the ratio of the actual number of samples, obtains the number of bad samples in the control group, calculates the number of bad samples in the actual total samples based on the restoration weight and the number of bad samples, and performs model performance evaluation based on the actual number of bad samples. By restoring the performance of customers who have been rejected by the experimental group as customers of the experimental group based on the restoration weight of the control group, it is possible to infer the quality of the rejected samples of the experimental group, restore the actual proportion of bad samples, and achieve accurate evaluation of the performance of the new model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of the structure of a model performance evaluation device for a hardware operating environment according to an embodiment of the present invention;

[0063] Figure 2 This is a flow chart of an embodiment of a method for evaluating model performance according to the present invention;

[0064] Figure 3 A schematic diagram of sample distribution in an embodiment of a model performance evaluation method of the present invention;

[0065] Figure 4 This is a detailed flowchart of step S20 in the second embodiment of the model performance evaluation method of the present invention;

[0066] Figure 5 1 is a flow chart of a third embodiment of the model performance evaluation method of the present invention;

[0067] Figure 6 Schematic diagram of the functional modules of an embodiment of the model performance evaluation method of the present invention.

[0068] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0069] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] Reference Figure 1 , Figure 1 Schematic diagram of the structure of a model performance evaluation device for a hardware operating environment according to an embodiment of the present invention.

[0071] like Figure 1 As shown, the model performance evaluation device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 may be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0072] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the model performance evaluation device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0073] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module and a model performance evaluation program.

[0074] exist Figure 1 In the model performance evaluation device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the asset level prediction device of the present invention can be set in the model performance evaluation device, and the model performance evaluation device calls the model performance evaluation program stored in the memory 1005 through the processor 1001, and executes the model performance evaluation method provided by the embodiment of the present invention.

[0075] The embodiment of the present invention provides a model performance evaluation method, referring to Figure 2 , Figure 2 Schematic diagram of the flow of a first embodiment of a model performance evaluation method of the present invention.

[0076] In this embodiment, the model performance evaluation method includes:

[0077] Step S10, obtaining the first rejection rate of the new model of the experimental group;

[0078] Step S20, obtaining the ratio of the actual number of samples of the control group to the actual number of samples of the experimental group;

[0079] Step S30, calculating the restoration weight of the control group according to the ratio of the first rejection rate to the actual number of samples;

[0080] Step S40, obtaining the number of bad samples in the control group;

[0081] Step S50, calculating the number of real bad samples according to the restoration weight and the number of bad samples in the control group;

[0082] Step S60: Perform model performance evaluation based on the number of real bad samples.

[0083] The model performance evaluation method of this embodiment is used for A / B testing to evaluate the performance of a new model. Typically, when iterating a model or strategy, an A / B test is performed between the new model or strategy and the old one. The differences (e.g., the bad sample rate) between the two groups of customers (the experimental group and the control group) are then compared. A significance test can also be performed to verify the effectiveness of the results, thus selecting the best strategy version for official release.

[0084] During our statistical analysis, we used a local sample instead of the overall sample, failing to consider whether the local sample itself was sufficiently representative. This biased our analysis of the overall population and led to erroneous conclusions. The risk control modeling used a sample from the approved population with post-loan performance, while the model needed to be applied to the entire population for launch. Because the approved and rejected populations have completely different distributions, determined by risk control strategies, the performance of the model derived from the approved population on the rejected population was completely unreliable. Consequently, the model's actual performance after launch fell far short of expectations. Therefore, its performance needed to be evaluated. During A / B testing, because the experimental group had already applied the new model to actual rejections, the sample distribution in the experimental group was skewed, requiring only the control group's results for evaluating the new model's performance. However, since the new model theoretically should outperform the old model, the experimental group was significantly larger than the control group during the A / B testing, resulting in a smaller control group. This not only prevented us from utilizing the full sample size for evaluation, wasting the experimental group's sample size, but also hindered accurate model evaluation due to the smaller control group sample size. Therefore, to accurately evaluate the performance of the new model, it was necessary to restore the true full sample distribution.

[0085] The following is a detailed description of each step:

[0086] Step S10, obtaining the first rejection rate of the new model;

[0087] In one embodiment, a first rejection rate for a new model is obtained. A new model is a newly developed model or strategy to be tested by R&D personnel, and is used for business risk management. The new model is configured with a rejection rate, which rejects clients based on the model's judgment criteria. The rejection rate is the ratio of rejected clients to the total number of clients. The rejection model identifies clients who are more likely to exhibit poor post-loan performance and rejects them, thereby reducing the number of bad clients among the client population.

[0088] Furthermore, in one embodiment, before the step of obtaining the first rejection rate of the new model of the experimental group, the method further includes:

[0089] Step S11, dividing the system into an experimental group and a control group, wherein the experimental group is tested online using the new model, and the control group is tested online directly.

[0090] In one embodiment, an experimental group and a control group are divided. It is understandable that when evaluating model performance, an A / B test is required. In the A / B test, the experimental group will use the new model for rejection, while the control group will not use the new model for rejection. Specifically, in the future, personnel who will handle business will be randomly divided into the experimental group or the control group. The new model will be used to evaluate the customers assigned to the experimental group to obtain a judgment result on whether to reject them. If they are not rejected, they are considered to have passed the test, and the samples of customers who passed the test are accumulated as the experimental group data. For customers assigned to the control group, they will not be rejected and will directly handle business according to the normal business process. The sample data of the control group will be accumulated until the number of samples reaches the expected requirement.

[0091] Step S20, obtaining the ratio of the actual number of samples of the control group to the actual number of samples of the experimental group;

[0092] In one embodiment, the ratio of the actual number of samples of the control group to the experimental group is obtained. The actual number of samples ratio is the ratio of the number of samples of the control group to the number of samples of the experimental group after the test. The actual number of samples ratio is preset by the business personnel based on experience, for example, it is set to 1:9 or 1:19. Because when a new model or strategy is launched, historical data is used for backtesting, and the effect of backtesting the new model on historical data is definitely better, but in the actual production process, it is not certain whether the model can have a better effect, so we need to make a control group to compare the effect. It is understandable that the purpose of launching a new model or strategy is to reduce the number of bad samples. Therefore, the number of samples will be tilted towards the experimental group, and the new model will be applied more to actually reject. In this implementation, it is necessary to obtain the actual number of samples ratio of the control group and the experimental group to confirm how much the number of samples of the control group and the experimental group differ.

[0093] Step S30, calculating the restoration weight of the control group according to the ratio of the first rejection rate to the actual number of samples;

[0094] In one embodiment, the restoration weight of the control group is calculated based on the first rejection rate and the ratio of the actual sample size of the control group to the experimental group. Specifically, the number of users rejected by the experimental group can be inferred based on the first rejection rate. This number of rejected users is then restored to obtain the original number of users in the experimental group without the new model applying the rejection model. The restoration weight of the control group is then calculated based on the ratio of the actual sample size of the experimental group to the control group after the rejections.

[0095] Furthermore, in one embodiment, the step of calculating the restoration weight of the control group based on the ratio of the first rejection rate to the actual number of samples includes:

[0096] Step S31, calculating the original sample number ratio according to the first rejection rate and the actual sample number ratio;

[0097] Step S32: Obtain the restoration weight of the control group according to the original sample number ratio:

[0098] In one embodiment, the ratio of the original number of people in the experimental group to the actual number of samples is used to infer the ratio of the original number of samples in the control group to the original experimental group. The control group's restoration weight is then derived based on the original number of samples. Specifically, assuming the number of people in the original experimental group is x and the rejection rate is a, the original number of people in the experimental group can be calculated based on the number of people in the current experimental group, b, using the following formula:

[0099]

[0100] Then, based on the original number of people in the experimental group and the actual number of people in the control group, the original sample number ratio is obtained.

[0101] Suppose that among all approved borrowers, 9,000 belong to the experimental group, where the new model partially rejects them. The remaining 1,000 belong to the control group, where the new model does not reject them. The new model has a 10% rejection rate for the experimental group. Because the control and experimental groups are randomly assigned, approximately 10% of the control group's samples hit the new model's rejection criteria. Based on the actual sample size ratio of the control group to the experimental group of 1:9, and the 10% more rejected samples in the experimental group, the original control group to experimental group sample size ratio should be 1,000:10,000, or 1:10. For every one sample in the control group, there are ten bad samples in the experimental group. The samples in the control group that hit the new model's rejection criteria are weighted 11 times as much.

[0102] Step S40, obtaining the number of bad samples in the control group;

[0103] In one embodiment, the number of bad samples in the control group is obtained. A / B testing uses data obtained during actual online use. After use, the performance of these customers can be determined, thereby determining whether the customers are good or bad samples. Specifically, through the approval strategy, a portion of the control group and a portion of the experimental group are approved. After a period of time, based on their repayment performance, labels of good and bad samples can be obtained, thereby obtaining the number of bad samples in the control group. For example, a threshold is set, and users with overdue rates exceeding the threshold are classified as bad samples, while those below the threshold are classified as good samples. The total number of good and bad samples is then counted separately.

[0104] Step S50, calculating the number of real bad samples according to the restoration weight and the number of bad samples in the control group;

[0105] In one embodiment, the true number of bad samples is calculated based on the restoration weight and the number of bad samples in the control group. The true number of bad samples is the number of bad samples in the total sample set, i.e., the number of bad samples in the total sample set when it is inferred that the experimental group was not rejected using the new model. Specifically, the true number of bad samples is obtained by multiplying the number of bad samples in the control group by the restoration weight.

[0106] Step S60: Perform model performance evaluation based on the number of real bad samples.

[0107] In one embodiment, the model performance is evaluated based on the number of real bad samples. It is understandable that in risk control business, whether the model performance is improved mainly depends on whether the number of bad samples can be reduced after applying the new model strategy. Therefore, after the real number of bad samples is restored according to steps S10 to S40, the corresponding number of real good samples can also be obtained, and the performance of the new model is evaluated using a preset model performance evaluation method, such as confusion matrix, ROC (Receiver Operating Characteristic, receiver operating characteristic), AUC: area under the ROC curve, etc.

[0108] Reference Figure 3 , Figure 3 This is a sample distribution diagram of an embodiment of the model performance evaluation method of the present invention, which explains the model performance evaluation method of the embodiment of the present invention:

[0109] like Figure 3 As shown in the figure, of all the approved borrowers, 9,000 were in the experimental group, where the new model partially rejected them. The remaining 1,000 were in the control group, where the new model did not reject them. Intuitively, the bad sample rate in the control group was 3%, while the bad sample rate in the experimental group was 1.1%. This indicates that the new model performs well, rejecting many bad customers, resulting in a lower bad sample rate in the experimental group. However, how can we quantitatively evaluate the actual performance of the new model using traditional performance evaluation methods (such as KS and AUC)? Using only the control group samples is too small to evaluate the performance of the new model. Therefore, it is necessary to use the control group samples, by restoring the weights, and then add the experimental group samples to restore the true sample distribution.

[0110] Assume that the customer rejection rate of the new model in the experimental group is 10%. Since the customers of the control group and the experimental group are randomly assigned, the number of people in the control group who hit the rejection rule of the new model is also about 10%. Assuming that there are 20 bad samples and 80 good samples among the people in the control group who hit the rejection rule of the new model, then the number of samples in the control group and the experimental group is 1:9, and the experimental group rejects 10% more customers than the control group. Therefore, the original ratio of the number of samples in the control group and the experimental group should be 1000:10000, that is, 1:10. The restoration weight of the samples in this part of the control group that hit the rejection rule of the new model is 11 times.

[0111] That is, if the new model rejection rule is not applied, 200 bad samples and 800 good samples should be added to the experimental group.

[0112] For samples that do not hit the new model's rejection rule, we simply maintain their original weights, which are set to 1. This way, we can estimate that the total number of passing samples in the simulation should be 330 bad samples and 10,670 good samples. By restoring the weights of samples in the control group that hit the new model's rejection rule, we can infer the quality of rejected samples in the experimental group and restore the true proportion of bad samples, allowing us to accurately evaluate the performance of the new model.

[0113] This embodiment obtains the first rejection rate of the new model, obtains the first sample number of the control group and the second sample number of the experimental group, calculates the actual sample number ratio, restores the number of people in the experimental group when the model was not used for rejection based on the first rejection rate, and then obtains the ratio of the original sample number of the control group to the original experimental group without rejection based on the actual sample number ratio of the control group and the experimental group. The restoration weight of the control group relative to the number of people in the experimental group is calculated, the number of bad samples in the control group is obtained, and the true number of bad samples is calculated based on the restoration weight. The model performance is then evaluated using the true number of bad samples. The weight restoration is equivalent to returning the experimental group to the state where the new model was not used for rejection. The model performance is then evaluated using the samples that were not rejected. This can restore the true bad sample rate of the experimental group and improve the accuracy of the model performance evaluation.

[0114] Reference Figure 4 , Figure 4 This is a detailed flowchart of step S20 in the first embodiment of the model performance evaluation method of the present invention. Further, based on the first embodiment of the model performance evaluation method of the present invention, a second embodiment of the model performance evaluation method of the present invention is proposed.

[0115] The difference between the second embodiment of the model performance evaluation method and the first embodiment of the model performance evaluation method is that the step of obtaining the ratio of the actual number of samples of the control group to the actual number of samples of the experimental group includes:

[0116] Step S21, obtaining the expected number of control group samples;

[0117] Step S22, obtaining the total number of samples and the sample pass rate;

[0118] Step S23, calculating the actual sample number ratio according to the expected control group sample number, the total sample number and the sample pass rate.

[0119] This embodiment uses the expected number of control group samples, the total number of samples, and the sample pass rate to obtain the actual sample number ratio. The following will explain each step:

[0120] Step S21, obtaining the expected number of control group samples;

[0121] In one embodiment, the customers of the control group and the experimental group are randomly selected, and the number of samples of the control group and the experimental group is divided according to a preset ratio. For example, if a total of 10,000 samples are expected to be collected, 5% of the people will usually be kept and 10% of the people will be placed in the control group. If the actual sample number ratio is selected as 10%, the expected number of control group samples is 1,000 and the expected number of experimental group samples is 9,000. It is understandable that if the number of people in the control group is relatively large, the bad sample rate will be relatively high, which will have a greater impact on the overall bad sample rate (overdue rate). The samples left in the control group are too few and difficult to evaluate. Because if the bad sample rate is relatively low and the number of samples is small, even if 500 samples are collected, no bad samples will appear. In this way, it will take a longer time to accumulate samples in the control group to meet the evaluation requirements. Therefore, the expected number of control group samples is first obtained. Specifically, the expected number of control group samples is set according to business needs.

[0122] Step S22, obtaining the total number of samples and the sample pass rate;

[0123] In one embodiment, the total number of samples and the sample pass rate are obtained. The sample pass rate is the ratio of those who successfully passed the application process to the total number of people who applied. The total number of samples is the total number of samples required to test the new model, including both the experimental and control groups. The total number of samples is set based on business needs, and the sample pass rate is set based on historical data.

[0124] Step S23, calculating the actual sample number ratio according to the expected control group sample number, the total sample number and the sample pass rate.

[0125] In one embodiment, the actual sample ratio is calculated based on the expected number of control group samples, the total number of samples, and the sample pass rate. This is inferred based on the total number of samples, the sample pass rate, and the number of samples that will ultimately be available for analysis in the control group. For example, if the total number of samples is 1,000 and the pass rate is 50%, the expected number of control group samples is 50, and the number of samples that pass is 500. The sample ratio of the control group to the experimental group is 50:450, resulting in an actual sample ratio of 1:9.

[0126] Furthermore, in one embodiment, the step of obtaining the sample pass rate includes:

[0127] Step S221: Count the number of people who applied for the service on a daily basis, the historical sample pass rate, and the historical approval rate to calculate the number of borrowers on a daily basis;

[0128] Step S222: Calculate the sample pass rate based on the historical daily number of borrowers and the historical daily number of business personnel.

[0129] In one embodiment, the number of historical daily business applications, historical sample pass rates, and historical approval rates are counted to calculate the number of historical daily borrowers. The sample pass rate is obtained by simulating past data. The number of daily applicants is multiplied by (1-rejection rate) and the approval rate to obtain the approximate number of daily borrowers. The number of daily borrowers is divided by the number of daily applicants to obtain the pass rate, which is used as the sample pass rate. The approval rate, in simple terms, is the proportion of individuals allowed into our sample under certain rules. A high proportion indicates a weak interception force under the rules, while a low proportion indicates a strong interception force.

[0130] This embodiment calculates the number of historical daily borrowing users by counting the number of historical daily business applicants, the historical sample pass rate and the historical approval rate, and then calculates the sample pass rate based on the historical number of daily borrowing users and the historical number of daily business applicants. Then, the expected number of control group samples and the total number of samples are obtained. Based on the expected number of control group samples, the total number of samples and the sample pass rate, the actual number of samples is calculated. This realizes a reasonable division of the sample size of the control group and the experimental group, so as to achieve a better model evaluation effect and improve the evaluation accuracy.

[0131] Reference Figure 5 , Figure 5 This is a flow chart of the third embodiment of the model performance evaluation method of the present invention. Further, based on the first and second embodiments of the model performance evaluation method of the present invention, the third embodiment of the model performance evaluation method of the present invention is proposed.

[0132] The third embodiment of the model performance evaluation method differs from the first and second embodiments of the model performance evaluation method in that the step of obtaining the first rejection rate of the new model for the experimental group for customers includes:

[0133] Step S12, obtaining a preset second rejection rate;

[0134] Step S13, calculating the improvement of the new model after applying the second rejection rate;

[0135] Step S14, determining whether the lift exceeds a lift threshold;

[0136] Step S15: If yes, select the second rejection rate as the first rejection rate.

[0137] This embodiment tests the improvement brought by the second rejection rate. When the selected second rejection rate exceeds the improvement threshold, the second rejection rate is selected as the first rejection rate.

[0138] The following describes each step in detail:

[0139] Step S12, obtaining a preset second rejection rate;

[0140] In one embodiment, a preset second rejection rate is obtained. The second rejection rate is set by business personnel based on actual needs, and different models or strategies have different rejection rates. The rejection rate is the ratio of the number of rejections in the model to the total number of business personnel.

[0141] Step S13, calculating the improvement of the new model after applying the second rejection rate;

[0142] In one embodiment, after the new model applies the preset second rejection rate, the lift of the new model is calculated. In model evaluation, we often use the Gain / Lift diagram to evaluate the model effect, where Lift is the ratio of the results obtained by "using the model" and "not using the model". Taking the scoring results of the credit score card model as an example, we usually sort the scored samples from low to high according to the scores, take 10 or 20 equal parts (there are cases where the same score corresponds to multiple observations, so the number of observations in each group may not be exactly equal), and count the number of observations in the group and the number of bad samples. The proportion of bad customers captured by the scoring card model can be calculated by dividing the number of bad samples in the group by the total number of bad samples;

[0143] Without this scorecard, the percentage of bad customers captured by random selection is equivalent to the ratio of the number of observations in this group to the total number of observations. The cumulative value of these two values ​​and their ratio yield the lift, which measures how many times better the scorecard's ability to capture bad customers is than random selection.

[0144] For example: the default rate of loan customers is 2%. You use data mining methods to build a model to predict the possibility of loan customers defaulting in the future. After using this model to score the customers, among the 10% customers with the highest default probability, 50% of the customers actually default. This means that the improvement rate of the model you built for identifying this part of customers is 5 (50% is 5 times that of 10%). As long as you do a good job of post-loan management of these 10% of customers, you can avoid the losses of the 50% of defaulting customers.

[0145] Step S14, determining whether the lift exceeds a lift threshold;

[0146] In one embodiment, a determination is made as to whether the lift exceeds a lift threshold. As is understandable, since lift is an important indicator for evaluating model quality, and rejection rate is a key parameter influencing model quality, the lift of the model after applying the second rejection rate is used to determine whether to apply the second rejection rate. The lift threshold is determined based on actual circumstances. For example, if the lift is required to be at least three times, then the rejection threshold is determined based on the lift.

[0147] Step S15: If yes, select the second rejection rate as the first rejection rate.

[0148] In one embodiment, if the calculated lift reaches the lift threshold after applying the second rejection rate to the new model, the second rejection rate is used as the first rejection rate for the new model's actual application. Therefore, the lift and rejection rate are weighed against each other based on the business's pass rate requirements and the lift of the new rejection strategy to determine the first rejection rate for the new model's actual application. Conversely, if the lift falls below the lift threshold, the second rejection rate is re-selected for testing.

[0149] Furthermore, in one embodiment, the step of evaluating the model performance according to the number of real bad samples includes:

[0150] Step S61, calculating the KS statistic according to the number of true bad samples;

[0151] Step S62, evaluating the degree of discrimination between good and bad samples of the new model based on the KS statistic;

[0152] Step S63: obtaining a performance evaluation result of the model based on the degree of differentiation between good and bad samples.

[0153] In one embodiment, after obtaining the number of true bad samples, model performance is evaluated based on the number of true bad samples. It is understood that after obtaining the number of true bad samples, the number of true good samples can be correspondingly obtained. The KS statistic is then calculated using the number of true good and bad samples. The KS (Kolmogorov-Smirnov) statistic is used to evaluate the model's risk discrimination ability. The metric measures the difference between the cumulative distribution of good and bad samples. The greater the cumulative difference between good and bad samples, the larger the KS metric, and thus the stronger the model's risk discrimination ability. The steps for calculating the KS statistic are as follows: 1. Bin the variables, selecting equal frequency, equal interval, or a custom distance. 2. Calculate the number of good accounts (goods) and bad accounts (bads) for each binning interval. 3. Calculate the ratio of the cumulative number of good accounts to the total number of good accounts (cum_good_rate) and the ratio of the cumulative number of bad accounts to the total number of bad accounts (cum_bad_rate) for each binning interval. 4. Calculate the absolute value of the difference between the cumulative bad account ratio and the cumulative good account ratio for each binning interval to obtain the KS curve. That is, ks = |cum_goodrate - cum_badrate|. 5. Take the maximum of these absolute values ​​to get the final KS value for this variable.

[0154] The KS value ranges from [0, 1] and is generally multiplied by 100%. Generally speaking, a larger KS value indicates better discrimination between positive and negative samples. The performance of the model is evaluated based on how well the new model distinguishes between good and bad samples. Specifically, a threshold can be set for judgment.

[0155] It should be noted that using the KS statistic to evaluate performance is only one of the evaluation methods, and other evaluation indicators can also be used for performance evaluation.

[0156] In addition, in this embodiment, the method for calculating the restoration weight of the control group according to the first rejection rate and the actual sample number ratio is:

[0157] Based on the ratio of the first rejection rate to the actual number of samples, an initial weight is obtained. Based on the experience of the business personnel, an empirical value is set. The initial weight is added to the empirical value to obtain the restored weight. For example, if the rejection rate is 10% and the actual sample number ratio of the control group to the experimental group is 1:9, the calculated initial weight is 11. Based on business experience, the initial weight is adjusted. The proportion of bad samples in the control group may be high or low due to policy reasons or at a certain time. Because the control group and the experimental group are usually not online for a long time, the proportion of bad samples may be abnormally high or low. Therefore, after the business personnel evaluate the proportion of bad samples in the control group, they set an empirical value and adjust the initial weight to obtain the restored weight. Specifically, the empirical value can be +0.1, +0.01, -0.2, -0.05, etc. The empirical value is not too large. The restored weight is then obtained by adding the initial weight to the empirical value. Because the problem to be solved in this embodiment is that the number of samples in the control group is too small, and the evaluation using the control group is not accurate, this embodiment uses the empirical value to adjust the initial weight to obtain the restored weight to obtain the actual number of bad samples in the experimental group.

[0158] This embodiment obtains a preset second rejection rate and calculates the lift after the new model applies the second rejection rate. It further determines whether the lift exceeds the lift threshold. If so, the second rejection rate is selected as the first rejection rate. By calculating the lift and setting the lift threshold, the performance of the new model is guaranteed. After obtaining the number of true bad samples, the KS statistic is calculated. Based on the KS statistic, the degree of discrimination between good and bad samples of the new model is evaluated. The performance evaluation result of the model is then obtained based on the degree of discrimination between good and bad samples. This method realizes the restoration of the true number of bad samples of the overall sample that are not rejected according to the restoration weights, and uses the KS statistic to evaluate the performance of the new model based on the true number of bad samples, thereby improving the effect of the model performance evaluation.

[0159] The present invention also provides a model performance evaluation device. Figure 6 As shown, Figure 6 Schematic diagram of the functional modules of an embodiment of the model performance evaluation method of the present invention.

[0160] The model performance evaluation device of the present invention comprises:

[0161] A first acquisition module, used to obtain the first rejection rate of the new model of the experimental group;

[0162] The second acquisition module is used to obtain the ratio of the actual number of samples of the control group to the actual number of samples of the experimental group;

[0163] a first calculation module, configured to calculate a restoration weight of a control group according to the first rejection rate and the ratio of the actual number of samples;

[0164] The third acquisition module is used to obtain the number of bad samples in the control group;

[0165] A second calculation module is used to calculate the actual number of bad samples according to the restoration weight and the number of bad samples;

[0166] An evaluation module is used to evaluate the model performance according to the number of real bad samples.

[0167] Optionally, the first calculation module is further configured to:

[0168] Calculating an original sample number ratio according to the first rejection rate and the actual sample number ratio;

[0169] The restored weight of the control group is obtained according to the original sample number ratio.

[0170] Optionally, the first acquisition module is further configured to:

[0171] The experimental group and the control group were divided into two groups. The experimental group used the new model for online testing, and the control group directly went online for testing.

[0172] Optionally, the second acquisition module is further configured to:

[0173] Obtain the expected number of control group samples;

[0174] Get the total number of samples and the sample pass rate;

[0175] The actual sample number ratio is calculated based on the expected control group sample number, the total sample number and the sample pass rate.

[0176] Optionally, the second acquisition module is further configured to:

[0177] Statistics are collected on the historical daily number of business users, historical sample pass rate, and historical approval rate to calculate the historical daily number of borrowers;

[0178] The sample pass rate is calculated based on the number of borrowing users and the number of business personnel on the historical day.

[0179] Optionally, the first acquisition module is further configured to:

[0180] Obtaining a preset second rejection rate;

[0181] Calculating the improvement of the new model after applying the second rejection rate;

[0182] Determining whether the lift exceeds a lift threshold;

[0183] If so, the second rejection rate is selected as the first rejection rate.

[0184] Optionally, the evaluation module is further configured to:

[0185] Calculate the KS statistic according to the number of true bad samples;

[0186] According to the KS statistic, the degree of discrimination between good and bad samples of the new model is evaluated;

[0187] The performance evaluation result of the model is obtained based on the degree of differentiation between good and bad samples.

[0188] The present invention also provides a storage medium.

[0189] The storage medium of the present invention stores a model performance evaluation program, which implements the steps of the model performance evaluation method described above when executed by a processor.

[0190] Among them, the method implemented when the model performance evaluation program running on the processor is executed can refer to the various embodiments of the model performance evaluation method of the present invention, and will not be repeated here.

[0191] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0192] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0193] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0194] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A performance evaluation method for a business risk identification model, characterized in that: Applied to a model performance evaluation device, the performance evaluation method of the business risk identification model includes the following steps: Obtaining the first rejection rate of the new model for the experimental group, wherein the rejection rate is the ratio of the number of rejected customers to the total number of customers. The new model is used for business risk identification; Obtaining the ratio of the actual number of samples of the control group to the number of samples of the experimental group; Calculating an original sample number ratio based on the first rejection rate and the actual sample number ratio; and obtaining a restoration weight of the control group based on the original sample number ratio; Get the number of bad samples in the control group; Calculating the number of real bad samples according to the restoration weight and the number of bad samples in the control group; Performing model performance evaluation based on the number of real bad samples; Before the step of obtaining the first rejection rate of the new model of the experimental group, the method further includes: Divide the system into an experimental group and a control group. The experimental group uses the new model for online testing, while the control group directly goes online for testing. In the future, people who handle business are randomly assigned to the experimental group or the control group. Customers assigned to the experimental group are evaluated using the new model to determine whether they are rejected. If they are not rejected, they are considered approved and the business is handled according to the normal business process. The samples of approved customers are accumulated as the experimental group data. Customers assigned to the control group are not rejected and the business is handled according to the normal business process. Sample data of the control group is accumulated. The step of obtaining the ratio of the actual number of samples of the control group to the actual number of samples of the experimental group includes: Obtain the expected number of control group samples; Get the total number of samples and the sample pass rate; The actual sample number ratio is calculated based on the expected control group sample number, the total sample number and the sample pass rate.

2. The performance evaluation method of the business risk identification model according to claim 1, characterized in that: The step of obtaining the sample pass rate includes: Calculate the number of borrowers per day by collecting historical statistics on the number of daily business applications, the historical sample pass rate, and the historical approval rate; The sample pass rate is calculated based on the number of borrowers on the historical day and the number of business applicants on the historical day.

3. The performance evaluation method of the business risk identification model according to claim 1, characterized in that: The step of obtaining the first rejection rate of the new model of the experimental group for the customer includes: Obtaining a preset second rejection rate; Calculating the improvement of the new model after applying the second rejection rate; Determining whether the lift exceeds a lift threshold; If so, the second rejection rate is selected as the first rejection rate.

4. The performance evaluation method of the business risk identification model according to claim 1, characterized in that: The step of performing performance evaluation of the business risk identification model according to the number of real bad samples includes: Calculate the KS statistic according to the number of true bad samples; According to the KS statistic, the degree of discrimination between good and bad samples of the new model is evaluated; The performance evaluation result of the model is obtained based on the degree of differentiation between good and bad samples.

5. A performance evaluation device for a business risk identification model, characterized in that: The device comprises: A first acquisition module is used to obtain a first rejection rate of the new model of the experimental group, wherein the rejection rate is the ratio of the number of rejected customers to the total number of customers. The new model is used for business risk identification; The second acquisition module is used to obtain the ratio of the actual number of samples of the control group to the actual number of samples of the experimental group; A first calculation module is configured to calculate an original sample number ratio based on the first rejection rate and the actual sample number ratio; and obtain a restoration weight of a control group based on the original sample number ratio; The third acquisition module is used to obtain the number of bad samples in the control group; A second calculation module is used to calculate the actual number of bad samples according to the restoration weight and the number of bad samples; An evaluation module, configured to evaluate model performance based on the number of real bad samples; The first acquisition module is further configured to: Divide the system into an experimental group and a control group. The experimental group uses the new model for online testing, while the control group directly goes online for testing. In the future, people who handle business are randomly assigned to the experimental group or the control group. Customers assigned to the experimental group are evaluated using the new model to determine whether they are rejected. If they are not rejected, they are considered approved and the business is handled according to the normal business process. The samples of approved customers are accumulated as the experimental group data. Customers assigned to the control group are not rejected and the business is handled according to the normal business process. Sample data of the control group is accumulated. The second acquisition module is further configured to: Obtain the expected number of control group samples; Get the total number of samples and the sample pass rate; The actual sample number ratio is calculated based on the expected control group sample number, the total sample number and the sample pass rate.

6. A model performance evaluation device, characterized in that: The device includes: a memory, a processor, and a performance evaluation program of a business risk identification model stored in the memory and executable on the processor, wherein the performance evaluation program of the business risk identification model is configured to implement the steps of the performance evaluation method of the business risk identification model according to any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium stores a performance evaluation program for a business risk identification model, and when the performance evaluation program for the business risk identification model is executed by the processor, the steps of the performance evaluation method for a business risk identification model according to any one of claims 1 to 4 are implemented.

Images