User classification method and apparatus

Abstract

The application discloses a user classification method and device, and relates to the technical field of artificial intelligence. A specific embodiment of the user classification method comprises the following steps: obtaining evaluation scores of a plurality of samples, dividing the samples into a plurality of sample sets according to the evaluation scores, and arranging the sample sets in a preset order of the evaluation scores; counting the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set; determining the positive sample rejection ratio factor and the positive sample proportion tuning item of each sample set according to the cumulative positive sample proportion of the sample set; determining the negative sample distribution correction factor and the negative sample proportion tuning item of each sample set based on the evaluation scores of the samples in the sample set; and determining the binary classification threshold for the evaluation scores according to the positive sample proportion tuning item and the negative sample proportion tuning item. The embodiment can determine a suitable classification threshold with a small amount of calculation without relying on artificial experience and preference.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a user classification method and device. BACKGROUND

[0002] It is very common to apply machine learning models in the field of bank credit, which is usually used to establish a classification model by using historical data to predict the risk level of a user, such as overdue risk and fraud risk. Such a machine learning model can output a probability value between 0 and 1, indicating the probability of a user being overdue or fraudulent. The higher the probability value, the higher the risk. This means that the probability value is positively correlated with the negative sample (representing a negative user), that is, the higher the probability value, the more likely it is to be a negative user. Conversely, if the evaluation score output by the machine learning model is higher, the risk of the user is lower, indicating that the evaluation score is positively correlated with the positive sample (representing a positive user).

[0003] When performing a classification task, such as a binary classification task, a threshold needs to be determined to label the user based on the threshold. Currently, the KS index method, the expert method with risk loss or pass rate as the target, or the profit method based on the profit equation are used to calculate the threshold. These three methods have the following defects: the KS index method is difficult to calculate a suitable threshold due to the lack of KS index calculation logic; the scientificity of the expert method is poor, mainly depending on the experience and preference of experts; the profit method has large calculation amount, long development cycle and high maintenance cost. SUMMARY

[0004] Therefore, the embodiments of the present application provide a user classification method and device, which can determine a suitable classification threshold and a reliable discrimination degree evaluation index of a machine learning model with small calculation amount without relying on artificial experience and preference.

[0005] To achieve the above-mentioned purpose, according to one aspect of the present application, a user classification method is provided.

[0006] The classification threshold determination method of the embodiments of the present application comprises: obtaining evaluation scores of a plurality of samples pre-labeled with positive sample labels or negative sample labels, dividing the samples into a plurality of sample sets according to the evaluation scores, and arranging the sample sets in a preset order of the evaluation scores; counting the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set; determining the positive sample rejection proportion factor and the positive sample proportion tuning item of each sample set according to the cumulative positive sample proportion of the sample set; determining the negative sample distribution correction factor and the negative sample proportion tuning item of each sample set based on the evaluation scores of the samples in the sample set; and determining a binary classification threshold for the evaluation scores according to the positive sample proportion tuning item and the negative sample proportion tuning item.

[0007] Optionally, the determining the positive sample rejection ratio factor and the positive sample ratio tuning item of each sample set according to the cumulative positive sample ratio of the sample set comprises: determining the positive sample rejection ratio factor of each sample set according to the cumulative positive sample ratio of the sample set, and forming the positive sample ratio tuning item of the sample set based on the cumulative positive sample ratio and the positive sample rejection ratio factor; and the determining the negative sample distribution correction factor and the negative sample ratio tuning item of each sample set based on the evaluation scores of the samples in the sample set comprises: determining the negative sample distribution correction factor of each sample set based on the evaluation scores of the samples in the sample set, and forming the negative sample ratio tuning item of the sample set based on the cumulative negative sample ratio of the sample set and the negative sample distribution correction factor.

[0008] Optionally, the determining the binary classification threshold for the evaluation score according to the positive sample ratio tuning item and the negative sample ratio tuning item comprises: determining the difference between the positive sample ratio tuning item and the negative sample ratio tuning item of each sample set, determining the sample set with the maximum value in the difference as a target sample set, and determining the binary classification threshold for the evaluation score by using the evaluation scores of the samples in the target sample set.

[0009] Optionally, the preset order is ascending order or descending order of the evaluation scores; and the dividing the samples into multiple sample sets according to the evaluation scores comprises: arranging the samples in the preset order, and dividing the first number of adjacent samples in the same sample set in sequence; wherein the same sample belongs to a unique sample set, and the first number is any number in a preset number interval; or arranging the samples in the preset order, equally spacing the value range of the evaluation scores of the samples into a preset number of non-intersecting score intervals, and dividing the samples with the evaluation scores in the same score interval into the same sample set; or determining each sample as a sample set.

[0010] Optionally, the cumulative positive sample ratio of each sample set is the proportion of the number of positive samples in the sample set and each sample set before the sample set in the total number of positive samples, and the cumulative negative sample ratio of each sample set is the proportion of the number of negative samples in the sample set and each sample set before the sample set in the total number of negative samples; and the statistics of the cumulative positive sample ratio and the cumulative negative sample ratio of each sample set comprises: sequentially counting the number of positive samples and the number of negative samples of each sample set according to the arrangement order of the sample set, and determining the cumulative positive sample ratio and the cumulative negative sample ratio of each sample set according to the number of positive samples and the number of negative samples.

[0011] Optionally, the determining the positive sample rejection ratio factor of each sample set according to the accumulated positive sample ratio of the sample set comprises: determining a difference between one and the accumulated positive sample ratio as the positive sample rejection ratio factor of the sample set; and the forming the positive sample ratio tuning term of each sample set based on the accumulated positive sample ratio and the positive sample rejection ratio factor comprises: determining a difference between the accumulated positive sample ratio and the positive sample rejection ratio factor of each sample set as the positive sample ratio tuning term of the sample set; or determining a product of a difference between the accumulated positive sample ratio and the positive sample rejection ratio factor of each sample set and a preset convergence coefficient as the positive sample ratio tuning term of the sample set.

[0012] Optionally, the evaluation score is a value between zero and one; the determining the negative sample distribution correction factor of each sample set based on the evaluation score of each sample in the sample set comprises: calculating an average value of the evaluation score of each sample in each sample set; in a case that the evaluation score is positively correlated with the positive sample, determining a quotient of the average value of the evaluation score and an opposite value of the average value of the evaluation score as the negative sample distribution correction factor of the sample set; wherein the opposite value of any value is a difference between one and the value; in a case that the evaluation score is positively correlated with the negative sample, determining a quotient of an opposite value of the average value of the evaluation score and the average value of the evaluation score as the negative sample distribution correction factor of the sample set; and the forming the negative sample ratio tuning term of each sample set based on the accumulated negative sample ratio of the sample set and the negative sample distribution correction factor comprises: multiplying the accumulated negative sample ratio and the negative sample distribution correction factor of each sample set to obtain the negative sample ratio tuning term of the sample set.

[0013] Optionally, the determining the binary classification threshold for the evaluation score based on the evaluation score of each sample in the target sample set comprises: determining an average value of the evaluation score of each sample in the target sample set as the binary classification threshold.

[0014] Optionally, the evaluation score is a value between zero and one; and in the case that the binary classification threshold is used to form the maximum value data, for any sample set: a positive sample rejection ratio factor of the sample set is a difference between one and an accumulated positive sample ratio of the sample set; a positive sample ratio tuning item of the sample set is a product of a difference between the accumulated positive sample ratio of the sample set and the positive sample rejection ratio factor and a preset convergence coefficient and a preset average interest rate; a negative sample distribution correction factor of the sample set is calculated by: calculating an evaluation score average of each sample in the sample set; in the case that the evaluation score is positively correlated with the positive sample, determining a quotient of the evaluation score average and an opposite number of the evaluation score average as the negative sample distribution correction factor of the sample set; wherein the opposite number of any value is a difference between one and the value; in the case that the evaluation score is positively correlated with the negative sample, determining a quotient of the opposite number of the evaluation score average and the evaluation score average as the negative sample distribution correction factor of the sample set; and a negative sample ratio tuning item of the sample set is a product of the accumulated negative sample ratio of the sample set and the negative sample distribution correction factor and a preset overdue loss rate.

[0015] To achieve the above object, according to another aspect of the present application, a machine learning model evaluation method is provided.

[0016] The machine learning model evaluation method of the embodiment of the present application comprises: obtaining evaluation scores output by a machine learning model to be evaluated for a plurality of samples pre-labeled with positive sample labels or negative sample labels, dividing the samples into a plurality of sample sets according to the evaluation scores, and arranging the sample sets in a preset order of the evaluation scores; counting an accumulated positive sample ratio and an accumulated negative sample ratio of each sample set; determining a positive sample rejection ratio factor and a positive sample ratio tuning item of each sample set according to the accumulated positive sample ratio of the sample set; determining a negative sample distribution correction factor and a negative sample ratio tuning item of each sample set based on the evaluation scores of the samples in the sample set; and determining a discrimination evaluation index of the machine learning model according to the positive sample ratio tuning item and the negative sample ratio tuning item.

[0017] Optionally, the determining the positive sample rejection ratio factor and the positive sample proportion tuning item of each sample set according to the cumulative positive sample proportion of the sample set comprises: determining the positive sample rejection ratio factor of each sample set according to the cumulative positive sample proportion of the sample set, and forming the positive sample proportion tuning item of the sample set based on the cumulative positive sample proportion and the positive sample rejection ratio factor; and the determining the negative sample distribution correction factor and the negative sample proportion tuning item of each sample set based on the evaluation scores of the samples in the sample set comprises: determining the negative sample distribution correction factor of each sample set based on the evaluation scores of the samples in the sample set, and forming the negative sample proportion tuning item of the sample set based on the cumulative negative sample proportion of the sample set and the negative sample distribution correction factor.

[0018] Optionally, the determining the discrimination evaluation index of the machine learning model according to the positive sample proportion tuning item and the negative sample proportion tuning item comprises: determining the difference between the positive sample proportion tuning item and the negative sample proportion tuning item of each sample set, and determining the maximum value in the difference as the discrimination evaluation index of the machine learning model.

[0019] Optionally, the preset order is ascending order or descending order of the evaluation scores, the cumulative positive sample proportion of each sample set is the proportion of the number of positive samples in the sample set and each sample set before the sample set in the total number of positive samples, and the cumulative negative sample proportion of each sample set is the proportion of the number of negative samples in the sample set and each sample set before the sample set in the total number of negative samples; and the dividing the samples into multiple sample sets according to the evaluation scores comprises: arranging the samples in the preset order, and dividing a first number of adjacent samples in the same sample set in sequence, wherein each sample belongs to a unique sample set, and the first number is any number in a preset number interval; or arranging the samples in the preset order, equally spacing the value range of the evaluation scores of the samples into a preset number of non-intersecting score intervals, and dividing the samples with the evaluation scores in the same score interval into the same sample set; or determining each sample as a sample set; and the counting the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set comprises: counting the number of positive samples and the number of negative samples of each sample set in sequence according to the arrangement order of the sample sets, and determining the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set according to the number of positive samples and the number of negative samples.

[0020] Optionally, the determining the positive sample rejection ratio factor of each sample set according to the accumulated positive sample ratio of the sample set comprises: determining a difference between one and the accumulated positive sample ratio as the positive sample rejection ratio factor of the sample set; and the forming the positive sample ratio adjustment term of each sample set based on the accumulated positive sample ratio and the positive sample rejection ratio factor comprises: determining a difference between the accumulated positive sample ratio and the positive sample rejection ratio factor of each sample set as the positive sample ratio adjustment term of the sample set; or determining a product of the difference between the accumulated positive sample ratio and the positive sample rejection ratio factor of each sample set and a preset convergence coefficient as the positive sample ratio adjustment term of the sample set.

[0021] Optionally, the evaluation score is a value between zero and one; the determining the negative sample distribution correction factor of each sample set based on the evaluation scores of the samples in the sample set comprises: calculating an average value of the evaluation scores of the samples in each sample set; in a case that the evaluation score is positively correlated with the positive sample, determining a quotient of the average value of the evaluation scores and an opposite value of the average value of the evaluation scores as the negative sample distribution correction factor of the sample set; wherein the opposite value of any value is a difference between one and the value; in a case that the evaluation score is positively correlated with the negative sample, determining a quotient of an opposite value of the average value of the evaluation scores and the average value of the evaluation scores as the negative sample distribution correction factor of the sample set; the forming the negative sample ratio adjustment term of each sample set based on the accumulated negative sample ratio and the negative sample distribution correction factor comprises: multiplying the accumulated negative sample ratio and the negative sample distribution correction factor of each sample set to obtain the negative sample ratio adjustment term of the sample set; and the machine learning model is a classification model, and the discrimination evaluation index is positively correlated with the discrimination ability of the machine learning model.

[0022] To achieve the above object, according to another aspect of the present application, a classification threshold determination device is provided.

[0023] The classification threshold determination device according to the embodiment of the present application comprises: a binning unit configured to obtain evaluation scores of a plurality of samples pre-labeled with positive sample labels or negative sample labels, divide the samples into a plurality of sample sets according to the evaluation scores, and arrange the sample sets in a preset order of the evaluation scores; an adjustment unit configured to count an accumulated positive sample ratio and an accumulated negative sample ratio of each sample set; determine a positive sample rejection ratio factor and a positive sample ratio adjustment term of each sample set according to the accumulated positive sample ratio of the sample set; determine a negative sample distribution correction factor and a negative sample ratio adjustment term of each sample set based on the evaluation scores of the samples in the sample set; and a threshold calculation unit configured to determine a binary classification threshold for the evaluation scores according to the positive sample ratio adjustment term and the negative sample ratio adjustment term.

[0024] To achieve the above object, according to another aspect of the present application, a machine learning model evaluation device is provided.

[0025] The machine learning model evaluation device of the embodiment of the present application comprises: a classification unit configured to obtain evaluation scores output by a machine learning model to be evaluated for a plurality of samples pre-labeled with positive sample labels or negative sample labels, divide the samples into a plurality of sample sets according to the evaluation scores, and arrange the sample sets in a preset order of the evaluation scores; an optimization unit configured to count a cumulative positive sample proportion and a cumulative negative sample proportion of each sample set; determine a positive sample rejection proportion factor and a positive sample proportion tuning item of the sample set according to the cumulative positive sample proportion of the sample set; determine a negative sample distribution correction factor and a negative sample proportion tuning item of the sample set based on the evaluation scores of the samples in the sample set; and an evaluation index calculation unit configured to determine a discrimination evaluation index of the machine learning model according to the positive sample proportion tuning item and the negative sample proportion tuning item.

[0026] To achieve the above object, according to another aspect of the present application, an electronic device is provided.

[0027] The electronic device of the present application comprises one or more processors, and a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the classification threshold determination or machine learning model evaluation method provided by the present application.

[0028] To achieve the above object, according to another aspect of the present application, an electronic device is provided.

[0029] The electronic device of the present application comprises one or more processors, and a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the classification threshold determination or machine learning model evaluation method provided by the present application.

[0030] To achieve the above object, according to another aspect of the present application, an electronic device is provided.

[0031] The electronic device of the present application comprises one or more processors, and a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the classification threshold determination or machine learning model evaluation method provided by the present application.

[0032] According to the technical solution of the present application, the embodiments of the above application have the following advantages or beneficial effects:

[0033] The positive sample rejection ratio factor is introduced as a penalty term for rejecting positive users, so that the proportion of positive samples in the KS index calculation logic is optimized; the negative sample distribution correction factor is introduced to correct the proportion of negative samples in the KS index calculation logic to the overall distribution of positive samples, so that the calculation dimension of the proportion of positive samples and the proportion of negative samples is consistent, and the explainability of the KS index calculation logic is improved. In this way, without relying on artificial experience and preferences, a suitable classification threshold that meets the business characteristics is determined with small calculation amount, and on this basis, the embodiment of the application provides a refined calculation logic based on interest rate and delinquency loss rate for the scene of pursuing value data. Finally, the embodiment of the application also provides a reliable discrimination evaluation index of the machine learning model by a similar method, which helps to accurately evaluate the discrimination ability of the machine learning model close to the business scene.

[0034] The further effects of the above-mentioned non-conventional optional mode will be described in the following in combination with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings are used to better understand the present application and do not constitute undue limitations on the present application. Among them:

[0036] Figure 1 is the main step schematic diagram of the classification threshold determination method in the embodiment of the application;

[0037] Figure 2 is the main step schematic diagram of the machine learning model evaluation method in the embodiment of the application;

[0038] Figure 3 is the component part schematic diagram of the classification threshold determination device in the embodiment of the application;

[0039] Figure 4 is the component part schematic diagram of the machine learning model evaluation device in the embodiment of the application;

[0040] Figure 5 is an exemplary system architecture diagram to which the embodiment of the application can be applied;

[0041] Figure 6 is an electronic device structure schematic diagram for implementing the classification threshold determination method or the machine learning model evaluation method in the embodiment of the application. DETAILED DESCRIPTION

[0042] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various specific details to facilitate understanding, and are to be considered in conjunction with the description. Thus, it should be apparent that the embodiments described herein can be practiced in a variety of embodiments other than those specifically described, without departing from the spirit and scope of the present application. Likewise, it should be apparent that features described for one embodiment are interchangeable with features described for other embodiments, so as to be employed in other embodiments within the scope of the embodiments.

[0043] It should be noted that the embodiments of the present application and the technical features in the embodiments can be combined with each other without conflict.

[0044] Figure 1 is a schematic diagram of main steps of the classification threshold determination method according to an embodiment of the present application.

[0045] As shown in Figure 1 , the classification threshold determination method according to an embodiment of the present application can be specifically executed according to the following steps:

[0046] Step S101: Obtain evaluation scores of a plurality of samples pre-labeled with positive sample labels or negative sample labels, divide the samples into a plurality of sample sets according to the evaluation scores, and arrange the sample sets according to a preset order of the evaluation scores.

[0047] It can be understood that the samples labeled with positive sample labels are positive samples, and the samples labeled with negative sample labels are negative samples. The positive samples and the negative samples can be pre-determined according to a business scenario. Generally, from a business perspective, the samples inclined to be approved are positive samples (for example, users with lower risks such as default or overdue), and the samples inclined to be rejected or resisted are negative samples (for example, users with higher risks such as default or overdue). In the embodiment of the present application, the sample sets can be divided in the following three ways. The first way is equal frequency binning, that is, the samples are arranged according to a preset order (which can be ascending or descending order of the evaluation scores), and the first number of adjacent samples are divided into the same sample set in turn. The same sample belongs to a unique sample set, and the first number is any number in a preset number interval. The span of the above number interval is generally small, for example, the number interval is [99, 101], and the sample number of each sample set is 99, 100 or 101. The second way is equal interval binning, that is, the samples are arranged according to a preset order, the value range of the evaluation scores of the samples is equally divided into a preset number of non-intersecting score intervals, and the samples with the evaluation scores in the same score interval are divided into the same sample set. It can be understood that the sample numbers of the sample sets thus divided are often unequal. The third way is non-binning, that is, each sample is determined as a sample set. Thereafter, the sample sets can be arranged according to the above preset order.

[0048] Step S102: statistics of cumulative positive sample proportion and cumulative negative sample proportion of each sample set; determination of positive sample rejection proportion factor and positive sample proportion tuning item of the sample set according to the cumulative positive sample proportion of the sample set; determination of negative sample distribution correction factor and negative sample proportion tuning item of the sample set based on the evaluation scores of samples in the sample set.

[0049] In the embodiment of the application, first, the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set are counted; then, the positive sample rejection proportion factor of the sample set is determined according to the cumulative positive sample proportion of the sample set, and the positive sample proportion tuning item of the sample set is formed based on the cumulative positive sample proportion and the positive sample rejection proportion factor; and the negative sample distribution correction factor of the sample set is determined based on the evaluation scores of samples in the sample set, and the negative sample proportion tuning item of the sample set is formed based on the cumulative negative sample proportion and the negative sample distribution correction factor of the sample set.

[0050] In this step, the current KS index calculation rule is optimized to obtain a classification threshold closer to the business scenario. Specifically, in the embodiment of the application, first, the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set are counted. The cumulative positive sample proportion of any sample set is the proportion of the number of positive samples in the sample set and each sample set before the sample set in the total number of positive samples, and the cumulative negative sample proportion of any sample set is the proportion of the number of negative samples in the sample set and each sample set before the sample set in the total number of negative samples. The above counting method can be that the number of positive samples and the number of negative samples of each sample set are counted in turn according to the arrangement order of the sample set, and the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set are determined according to the number of positive samples and the number of negative samples.

[0051] Then, the positive sample rejection proportion factor of the sample set can be determined according to the cumulative positive sample proportion of the sample set, and the positive sample proportion tuning item of the sample set is formed based on the cumulative positive sample proportion and the positive sample rejection proportion factor. The positive sample rejection proportion factor corresponds to the proportion of the original positive samples that are mistaken for negative samples and thus rejected, and the positive sample rejection proportion factor is added to the objective function as a penalty item of the proportion of rejected positive samples, encouraging the selection of sample sets with a relatively small proportion of rejected positive samples from each sample set. The cumulative positive sample proportion is optimized by the positive sample rejection proportion factor to obtain the positive sample proportion tuning item. The difference (which is a positive number) between the positive sample proportion tuning item and the negative sample proportion tuning item of the same sample set is the objective function L of the calculation logic of the embodiment of the application, and the corresponding formula is as follows:

[0052]

[0053]

[0054] wherein, This indicates the positive sample ratio optimization term. This indicates the negative sample proportion optimization term. Indicates the cumulative proportion of positive samples. denoted as the positive sample rejection ratio factor, and f is a function that forms the positive sample ratio optimization term.

[0055] In practical applications, the difference between the positive sample ratio and the cumulative positive sample ratio can be determined as the positive sample rejection ratio factor for the same sample set. This can be understood as the difference between the positive sample ratio and the cumulative positive sample ratio being used to determine the positive sample rejection ratio in the Bauhaus method. Subsequently, the difference between the cumulative positive sample ratio and the positive sample rejection ratio factor for any sample set can be determined as the positive sample ratio tuning term for that sample set, or the product of the difference between the cumulative positive sample ratio and the positive sample rejection ratio factor for any sample set and a preset convergence coefficient α can be determined as the positive sample ratio tuning term for that sample set. The corresponding formulas are as follows:

[0056]

[0057] or,

[0058] In this embodiment of the invention, a negative sample distribution correction factor for each sample set can be determined based on the evaluation scores of each sample in that set. A negative sample ratio optimization term is then formed based on the cumulative negative sample ratio and the negative sample distribution correction factor. In the current KS index calculation logic, the difference between the positive and negative sample ratios is used as the objective function. However, the overall distributions of these two ratios differ significantly. The calculation environment for the positive sample ratio is the overall positive sample population, while the calculation environment for the negative sample ratio is the overall negative sample population. This inconsistency in overall distribution leads to insufficient algorithm interpretability. To address these shortcomings, this embodiment of the invention proposes the aforementioned negative sample distribution correction factor to correct the overall distribution of negative samples to the overall distribution of positive samples, ensuring consistency in the computational dimensions of both, thereby improving algorithm interpretability and computational accuracy. The relevant formula is as follows:

[0059]

[0060] Where B represents the cumulative negative sample proportion. denoted as the negative sample distribution correction factor, and t represents the function for calculating the negative sample proportion optimization term.

[0061] In a specific application, the negative sample distribution correction factor can be calculated by the following steps, in which case the evaluation score is a value between zero and one. First, the average value of the evaluation score of each sample in any sample set is calculated; in the case of positive correlation between the evaluation score and the positive sample, the quotient of the average value of the evaluation score and the opposite of the average value of the evaluation score is determined as the negative sample distribution correction factor of the sample set; in the case of positive correlation between the evaluation score and the negative sample, the quotient of the opposite of the average value of the evaluation score and the average value of the evaluation score is determined as the negative sample distribution correction factor of the sample set. Wherein, the opposite of any value is the difference between one and the value. The above calculation principle is based on using the average value p of the evaluation score to represent the current overall number ratio of positive samples and negative samples, which helps to realize the correction from the negative sample population to the positive sample population, and the related formula is as follows:

[0062]

[0063]

[0064] Preferably, the cumulative negative sample proportion of any sample set can be multiplied by the negative sample distribution correction factor to obtain the negative sample proportion tuning item of the sample set, and the formula is as follows:

[0065]

[0066] Thus, the complete expression of the objective function is as follows:

[0067]

[0068] Step S103: determining the binary classification threshold for the evaluation score according to the positive sample proportion tuning item and the negative sample proportion tuning item.

[0069] In a specific application, the difference between the positive sample proportion tuning item and the negative sample proportion tuning item of each sample set can be determined first, then the sample set with the maximum value in the difference is determined as the target sample set, and the evaluation score of each sample in the target sample set is used to determine the binary classification threshold for the evaluation score.

[0070] In this step, the maximum value of the objective function can be taken as the optimization target to obtain the corresponding target sample set, and then the average value of the evaluation score of each sample in the target sample set is determined as the binary classification threshold. Based on the optimization of each item of the objective function, the above binary classification threshold can be more close to the business scenario, corresponding to fewer rejected positive samples, while the accuracy is higher.

[0071] Particularly, in the case that the binary classification threshold is used to form the maximum value data, i.e. the corresponding scene pays more attention to the profit rate, the above objective function can be further adjusted according to the average interest rate r and the loss rate of overdue LGD of the current scene, and the binary classification threshold obtained in this way can correspond to the maximum profit. The corresponding formula is as follows:

[0072]

[0073] That is, for any sample set: the positive sample rejection rate factor of the sample set is the difference between one and the cumulative positive sample proportion of the sample set; the positive sample proportion tuning item of the sample set is the difference between the cumulative positive sample proportion of the sample set and the positive sample rejection rate factor, multiplied by the preset convergence coefficient and the preset average interest rate; the negative sample distribution correction factor of the sample set is calculated by the following steps: calculating the evaluation score average of each sample in the sample set; in the case that the evaluation score is positively correlated with the positive sample, the quotient of the evaluation score average and the opposite number of the evaluation score average is determined as the negative sample distribution correction factor of the sample set; wherein the opposite number of any value is the difference between one and the value; in the case that the evaluation score is positively correlated with the negative sample, the quotient of the opposite number of the evaluation score average and the evaluation score average is determined as the negative sample distribution correction factor of the sample set; the negative sample proportion tuning item of the sample set is the product of the cumulative negative sample proportion of the sample set, the negative sample distribution correction factor and the preset loss rate of overdue.

[0074] The application further provides a machine learning model evaluation method, which refers to Figure 2 The machine learning model evaluation method of the embodiment of the application can include the following steps S201, S202 and S203, and the execution details of steps S201 and S202 are similar to the foregoing, and thus are not repeated here.

[0075] Step S201: obtaining evaluation scores output by a machine learning model to be evaluated for a plurality of samples pre-labeled with positive sample labels or negative sample labels, dividing the samples into a plurality of sample sets according to the evaluation scores, and arranging the sample sets in a preset order of the evaluation scores.

[0076] In the embodiment of the application, the machine learning model to be evaluated is a classification model, and the above evaluation scores are output results calculated by the machine learning model to be evaluated for input samples. The preset order is ascending order or descending order of the evaluation scores, the cumulative positive sample proportion of any sample set is the proportion of the number of positive samples in the sample set and each sample set before the sample set in the total number of positive samples, and the cumulative negative sample proportion of any sample set is the proportion of the number of negative samples in the sample set and each sample set before the sample set in the total number of negative samples.

[0077] Preferably, the corresponding step of the sample set division can be: arranging the samples according to the preset order, and sequentially dividing a first number of adjacent samples in the same sample set; wherein the same sample belongs to a unique sample set, and the first number is any number in a preset number interval; or arranging the samples according to the preset order, equally spacing the value range of the evaluation score of the samples into a preset number of disjoint score intervals, and dividing the samples with the same evaluation score in the same sample set; or determining each sample as a sample set.

[0078] Step S202: counting the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set; determining the positive sample rejection proportion factor and the positive sample proportion tuning item of the sample set according to the cumulative positive sample proportion of the sample set; and determining the negative sample distribution correction factor and the negative sample proportion tuning item of the sample set based on the evaluation score of each sample in the sample set.

[0079] In an actual scenario, the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set can be counted first; then the positive sample rejection proportion factor of the sample set is determined according to the cumulative positive sample proportion of the sample set, the positive sample proportion tuning item of the sample set is formed based on the cumulative positive sample proportion and the positive sample rejection proportion factor; and the negative sample distribution correction factor of the sample set is determined based on the evaluation score of each sample in the sample set, and the negative sample proportion tuning item of the sample set is formed based on the cumulative negative sample proportion of the sample set and the negative sample distribution correction factor.

[0080] Preferably, the calculation step of the cumulative positive sample proportion and the cumulative negative sample proportion can be: sequentially counting the positive sample number and the negative sample number of each sample set according to the arrangement order of the sample set, and determining the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set according to the positive sample number and the negative sample number.

[0081] As a preferred solution, the optimization step of the current KS index calculation logic is mainly as follows: determining the difference between one and the cumulative positive sample proportion of the sample set as the positive sample rejection proportion factor of the sample set, determining the difference between the cumulative positive sample proportion of the sample set and the positive sample rejection proportion factor as the positive sample proportion adjustment term of the sample set, or determining the product of the difference between the cumulative positive sample proportion of the sample set and the positive sample rejection proportion factor and a preset convergence coefficient as the positive sample proportion adjustment term of the sample set. In the case that the evaluation score is a value between zero and one; calculating the average value of the evaluation scores of the samples in the sample set; in the case that the evaluation score is positively correlated with the positive sample, determining the quotient of the average value of the evaluation scores and the opposite of the average value of the evaluation scores as the negative sample distribution correction factor of the sample set; wherein the opposite of any value is the difference between one and the value; in the case that the evaluation score is positively correlated with the negative sample, determining the quotient of the opposite of the average value of the evaluation scores and the average value of the evaluation scores as the negative sample distribution correction factor of the sample set. Multiplying the cumulative negative sample proportion of the sample set and the negative sample distribution correction factor to obtain the negative sample proportion adjustment term of the sample set.

[0082] Step S203: determining the discrimination evaluation index of the machine learning model according to the positive sample proportion adjustment term and the negative sample proportion adjustment term.

[0083] In this step, the difference between the positive sample proportion adjustment term and the negative sample proportion adjustment term of each sample set can be determined first, and then the maximum value in the difference is determined as the discrimination evaluation index of the machine learning model.

[0084] The discrimination evaluation index is positively correlated with the discrimination ability of the machine learning model. In this step, the maximum target function value can be calculated as the optimization target by taking the maximum value of the target function, that is, the maximum value in the difference between the positive sample proportion adjustment term and the negative sample proportion adjustment term, and this value can represent the classification ability of the model. The higher the value, the stronger the classification ability of the model. Due to the foregoing optimization measures of the target function, the discrimination evaluation index above can more accurately measure the classification ability of the model in the case of fitting the business scenario. It should be noted that the evaluation method should be consistent when evaluating multiple models.

[0085] The following describes a specific embodiment of the present application.

[0086] It is very common to apply machine learning models in the field of bank credit, which is often used to establish a classification model to predict the risk level of users, such as overdue risk and fraud risk, etc. Such machine learning models may output a probability value between 0 and 1, indicating the probability of user overdue or fraud, and the higher the probability value, the higher the risk. This means that the probability value is positively correlated with the negative sample, that is, the higher the probability value, the more likely it is a negative user. Conversely, if the evaluation score output by the machine learning model is higher, the user's risk is lower, indicating that the evaluation score is positively correlated with the positive sample.

[0087] When doing classification tasks, such as dividing users into positive and negative users in a binary classification task, a threshold needs to be determined to label users based on the threshold. Currently, KS index method, expert method with risk loss or pass rate as target, or profit method based on profit equation are used to calculate the threshold.

[0088] KS index is derived from Kolmogorov-Smirnov test and is widely used to evaluate the discrimination ability of classification models. The calculation process is relatively simple. The model score (i.e. evaluation score) is sorted from small to large, and the maximum difference between the cumulative proportion of positive samples and the cumulative proportion of negative samples is calculated in each interval. The larger the KS, the better the discrimination ability of the model. As shown in the following table, the model score in this table is a discrete value. When the threshold of the model score is equal to 0.6, the maximum KS is 53.4%.

[0089]

[0090] The disadvantage of KS index method is that the two items subtracted are the proportion of positive samples and the proportion of negative samples, which have different overall distributions, so they cannot directly reflect the optimal real number difference, and the problem of positive sample rejection is not considered in the algorithm, which is not close to the actual business scenario. Because in actual business, more attention is paid to screening more positive samples to avoid rejection, so the existing KS index calculation logic needs to be optimized.

[0091] The expert method generally targets risk loss or pass rate. For the former, through a series of data analysis, historical repayment performance is used in combination with other important indicators to stratify loan applications, and the corresponding model score threshold is deduced by a pre-set risk loss target. For the latter, the target pass rate is formulated according to business needs, and the threshold is deduced accordingly. The above methods can quickly adjust the threshold to achieve business goals. However, they completely rely on the experience and preferences of practitioners, which is less scientific and difficult to make a rational and quantitative trade-off between risk control intensity and economic benefits.

[0092] The profit method generally sets a return rate target of a loan application, maps risk loss by using a model, constructs a profit equation, and finally calculates the threshold of the model score. The profit equation can refer to the risk-adjusted capital return rate model RAROC. The above modeling process needs to consider the overall profitability, including various parameters of the product such as interest rate, fee, repayment method, period, risk loss, various costs such as customer acquisition cost, operation cost, and collection efficiency, so the calculation process is complex, the development cycle is long, and the maintenance cost is relatively high.

[0093] The embodiment of the present application optimizes the existing KS index method framework and proposes the following algorithm:

[0094]

[0095] When the business scene pursues the optimal profitability, the algorithm is as follows:

[0096]

[0097] The meanings of the parameters and the specific optimization details and effects of the algorithm have been described in the foregoing, and will not be repeated here.

[0098] Thereafter, the objective function is solved. Since the above optimization only makes a linear transformation to the traditional KS curve and does not change the concavity and convexity of the KS curve, there is a maximum value, and the model score corresponding to the maximum value is the optimal threshold.

[0099] In the above example, the optimal threshold changes from 0.6 to 0.3, that is, in order to achieve the optimal number of positive samples, the threshold should be determined at 0.3, so that a larger proportion of positive samples can be screened out, and the high rejection rate can be avoided. The optimal function value can be used as the discrimination index of the machine learning model.

[0100] In this way, by modifying the calculation logic of the traditional KS curve, the model evaluation index and the credit access threshold are closer to the actual business scene requirements, have higher scientificity and interpretability, and have lower calculation amount and maintenance cost and higher scalability.

[0101] It should be noted that in the technical solutions of the present application, the collection, analysis, use, transmission, storage, etc. of user personal information comply with relevant laws and regulations and are used for legal and reasonable purposes, are not shared, disclosed or sold outside these legal uses, and are subject to supervision and management by regulatory departments. Necessary measures should be taken to prevent illegal access to such personal information data and to ensure that personnel with access to personal information data comply with relevant laws and regulations and ensure the security of user personal information. Once these user personal information data are no longer needed, the risk should be minimized by limiting or even prohibiting data collection and / or deleting data. When used, including in certain related applications, user privacy is protected by de-identifying data, such as by removing specific identifiers (e.g. date of birth) when used, controlling the amount or specificity of stored data (e.g. collecting location data at the city level rather than at the specific address level), controlling how data is stored, and / or other methods of de-identification.

[0102] For the foregoing method embodiments, for ease of description, they are expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, and certain steps can actually be performed in other orders or simultaneously. In addition, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to implement the present application.

[0103] In order to better implement the above-mentioned scheme of the embodiments of the present application, the related device for implementing the above-mentioned scheme is also provided below.

[0104] Please refer to Figure 3 The classification threshold determination device 300 provided by the embodiments of the present application can include a binning unit 301, an optimization unit 302 and a threshold calculation unit 303.

[0105] The binning unit 301 is configured to obtain evaluation scores of a plurality of samples pre-labeled with positive sample labels or negative sample labels, divide the samples into a plurality of sample sets according to the evaluation scores, and arrange the sample sets in a preset order of the evaluation scores. The optimization unit 302 is configured to count cumulative positive sample proportions and cumulative negative sample proportions of each sample set, determine a positive sample rejection ratio factor and a positive sample proportion optimization term of the sample set according to the cumulative positive sample proportion of the sample set, and determine a negative sample distribution correction factor and a negative sample proportion optimization term of the sample set based on the evaluation scores of the samples in the sample set. The threshold calculation unit 303 is configured to determine a binary classification threshold for the evaluation scores according to the positive sample proportion optimization term and the negative sample proportion optimization term.

[0106] In actual applications, the tuning unit 302 can be further configured to: count the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set; determine a positive sample rejection proportion factor of the sample set according to the cumulative positive sample proportion of the sample set; form a positive sample proportion tuning item of the sample set based on the cumulative positive sample proportion and the positive sample rejection proportion factor; determine a negative sample distribution correction factor of the sample set based on the evaluation scores of the samples in the sample set; form a negative sample proportion tuning item of the sample set based on the cumulative negative sample proportion of the sample set and the negative sample distribution correction factor; and the threshold calculation unit 303 can be further configured to: determine the difference between the positive sample proportion tuning item and the negative sample proportion tuning item of each sample set, determine a target sample set having the maximum value in the difference, and determine the binary classification threshold for the evaluation score by using the evaluation scores of the samples in the target sample set.

[0107] In the embodiments of the present application, the preset order is ascending order or descending order of the evaluation scores; and the binning unit 301 can be further configured to: arrange the samples in the preset order, and divide a first number of adjacent samples in the same sample set in sequence; wherein the same sample belongs to a unique sample set, and the first number is any number in a preset number interval; or arrange the samples in the preset order, equally interval the value range of the evaluation scores of the samples into a preset number of non-intersecting score intervals, and divide the samples having the evaluation scores in the same score interval into the same sample set; or determine each sample as a sample set.

[0108] In specific applications, the cumulative positive sample proportion of any sample set is the proportion of the number of positive samples in the sample set and the sample sets before the sample set in the total number of positive samples, and the cumulative negative sample proportion of any sample set is the proportion of the number of negative samples in the sample set and the sample sets before the sample set in the total number of negative samples; and the tuning unit 302 can be further configured to: count the number of positive samples and the number of negative samples of each sample set in sequence according to the arrangement order of the sample sets, and determine the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set according to the number of positive samples and the number of negative samples.

[0109] In actual applications, the tuning unit 302 can be further configured to: determine a difference between the cumulative positive sample proportion and the positive sample rejection proportion factor as the positive sample rejection proportion factor of the same sample set; determine the difference between the cumulative positive sample proportion of any sample set and the positive sample rejection proportion factor as the positive sample proportion tuning item of the sample set; or determine the product of the difference between the cumulative positive sample proportion of any sample set and the positive sample rejection proportion factor and a preset convergence coefficient as the positive sample proportion tuning item of the sample set.

[0110] Preferably, the evaluation score is a value between 0 and 1; and the tuning unit 302 is further configured to: calculate an average value of the evaluation scores of the samples in any sample set; in a case that the evaluation score is positively correlated with the positive sample, determine a quotient of the average value of the evaluation scores and an opposite value of the average value of the evaluation scores as a negative sample distribution correction factor of the sample set; wherein the opposite value of any value is a difference between 1 and the value; in a case that the evaluation score is positively correlated with the negative sample, determine a quotient of the opposite value of the average value of the evaluation scores and the average value of the evaluation scores as the negative sample distribution correction factor of the sample set; and multiply the cumulative positive sample proportion and the negative sample distribution correction factor of any sample set to obtain a negative sample proportion tuning item of the sample set.

[0111] As a preferred solution, the threshold calculation unit 303 is further configured to: determine an average value of the evaluation scores of the samples in the target sample set as the binary classification threshold.

[0112] In addition, in the embodiments of the present application, the evaluation score is a value between 0 and 1; and in a case that the binary classification threshold is used to form the maximum value data, for any sample set: a positive sample rejection proportion factor of the sample set is a difference between 1 and a cumulative positive sample proportion of the sample set; a positive sample proportion tuning item of the sample set is a product of a difference between the cumulative positive sample proportion and the positive sample rejection proportion factor of the sample set, a preset convergence coefficient and a preset average interest rate; the tuning unit 302 is further configured to: calculate an average value of the evaluation scores of the samples in the sample set; in a case that the evaluation score is positively correlated with the positive sample, determine a quotient of the average value of the evaluation scores and an opposite value of the average value of the evaluation scores as a negative sample distribution correction factor of the sample set; wherein the opposite value of any value is a difference between 1 and the value; in a case that the evaluation score is positively correlated with the negative sample, determine a quotient of the opposite value of the average value of the evaluation scores and the average value of the evaluation scores as the negative sample distribution correction factor of the sample set; and a negative sample proportion tuning item of the sample set is a product of a cumulative negative sample proportion of the sample set, the negative sample distribution correction factor and a preset overdue loss rate.

[0113] Please refer to Figure 4 As shown in the figure, the machine learning model evaluation device 400 provided by the embodiments of the present application can include a classification unit 401, an optimization unit 402 and an evaluation index calculation unit 403.

[0114] The classification unit 401 can be configured to obtain evaluation scores of a machine learning model to be evaluated on a plurality of samples pre-labeled with positive sample labels or negative sample labels, divide the samples into a plurality of sample sets according to the evaluation scores, and arrange the sample sets in a preset order of the evaluation scores. The optimization unit 402 can be configured to count a cumulative positive sample proportion and a cumulative negative sample proportion of each sample set, determine a positive sample rejection proportion factor and a positive sample proportion tuning item of the sample set according to the cumulative positive sample proportion of the sample set, determine a negative sample distribution correction factor and a negative sample proportion tuning item of the sample set based on the evaluation scores of the samples in the sample set, and the evaluation index calculation unit 403 can be configured to determine a discrimination evaluation index of the machine learning model according to the positive sample proportion tuning item and the negative sample proportion tuning item.

[0115] In actual applications, the optimization unit 402 can be further configured to count a cumulative positive sample proportion and a cumulative negative sample proportion of each sample set, determine a positive sample rejection proportion factor of the sample set according to the cumulative positive sample proportion of the sample set, form a positive sample proportion tuning item of the sample set based on the cumulative positive sample proportion and the positive sample rejection proportion factor, determine a negative sample distribution correction factor of the sample set based on the evaluation scores of the samples in the sample set, form a negative sample proportion tuning item of the sample set based on the cumulative negative sample proportion of the sample set and the negative sample distribution correction factor, and the evaluation index calculation unit 403 can be further configured to determine a difference between the positive sample proportion tuning item and the negative sample proportion tuning item of each sample set, and determine a maximum value in the difference as the discrimination evaluation index of the machine learning model.

[0116] In an optional technical solution, the preset order is an ascending order or a descending order of the evaluation scores, the cumulative positive sample proportion of any sample set is a proportion of a number of positive samples in the sample set and each sample set before the sample set in a total number of positive samples, the cumulative negative sample proportion of any sample set is a proportion of a number of negative samples in the sample set and each sample set before the sample set in a total number of negative samples, and the classification unit 401 can be further configured to: arrange the samples according to the preset order, and divide a first number of adjacent samples in a same sample set in sequence, wherein each sample belongs to a unique sample set, and the first number is any number in a preset number interval; or arrange the samples according to the preset order, divide a value range of the evaluation scores of the samples into a preset number of disjoint score intervals at equal intervals, and divide samples with the same evaluation score in a same score interval into a same sample set; or determine each sample as a sample set. The optimization unit 402 can be further configured to: statistically determine the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set, including: statistically determining a number of positive samples and a number of negative samples of each sample set in sequence according to the arrangement order of the sample set, and determining the cumulative positive sample proportion and the cumulative negative sample proportion of each sample set according to the number of positive samples and the number of negative samples.

[0117] In an embodiment, the optimization unit 402 can be further configured to: determine a difference between the cumulative positive sample proportion and the cumulative negative sample proportion as a positive sample rejection proportion factor of a same sample set; determine a difference between the cumulative positive sample proportion and the positive sample rejection proportion factor of any sample set as a positive sample proportion tuning item of the sample set; or determine a product of a preset convergence coefficient and a difference between the cumulative positive sample proportion and the positive sample rejection proportion factor of any sample set as the positive sample proportion tuning item of the sample set.

[0118] In an optional implementation, the evaluation score is a value between zero and one, and the optimization unit 402 can be further configured to: calculate an average value of the evaluation scores of samples in any sample set; in a case where the evaluation score is positively correlated with the positive sample, determine a quotient of the average value of the evaluation scores and an opposite number of the average value of the evaluation scores as a negative sample distribution correction factor of the sample set; wherein the opposite number of any value is a difference between one and the value; in a case where the evaluation score is positively correlated with the negative sample, determine a quotient of the opposite number of the average value of the evaluation scores and the average value of the evaluation scores as the negative sample distribution correction factor of the sample set; and multiply the cumulative negative sample proportion and the negative sample distribution correction factor of any sample set to obtain a negative sample proportion tuning item of the sample set.

[0119] In an actual scenario, the machine learning model is a classification model, and the discrimination evaluation index is positively correlated with a discrimination ability of the machine learning model.

[0120] According to the technical scheme of the embodiment of the present application, the current KS index calculation logic is optimized to determine the optimal threshold close to the business scenario. The optimized calculation logic can also be used to determine the discrimination evaluation index of the machine learning model. Thus, the embodiment of the present application can determine the appropriate classification threshold and reliable classification ability evaluation index of the machine learning model in accordance with the business requirements without relying on artificial experience and preferences with a small amount of calculation.

[0121] Figure 5 An exemplary system architecture 500 to which the classification threshold determination or machine learning model evaluation method or device of the embodiments of the present application can be applied is shown.

[0122] As shown in Figure 5 The system architecture 500 can include terminal devices 501, 502, 503, a network 504, and a server 505 (this architecture is only an example, and the components included in the specific architecture can be adjusted according to the specific application). The network 504 is used to provide a communication link medium between the terminal devices 501, 502, 503 and the server 505. The network 504 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0123] The user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various client applications can be installed on the terminal devices 501, 502, 503, such as a threshold calculation application (only an example).

[0124] The terminal devices 501, 502, 503 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0125] The server 505 can be a server providing various services, such as a background server supporting the threshold calculation application operated by the user using the terminal devices 501, 502, 503 (only an example). The background server can process the received threshold calculation request and feed back the processing result (such as the calculated classification threshold - only an example) to the terminal devices 501, 502, 503.

[0126] It should be noted that the classification threshold determination or machine learning model evaluation method provided by the embodiments of the present application is generally executed by the server 505, and correspondingly, the classification threshold determination or machine learning model evaluation device is generally provided in the server 505.

[0127] It should be understood that Figure 5The number of terminal devices, networks and servers in the system is merely illustrative. Depending on the implementation needs, there can be any number of terminal devices, networks and servers.

[0128] The present application also provides an electronic device. The electronic device of the embodiments of the present application comprises: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the classification threshold determination or machine learning model evaluation method provided by the present application.

[0129] Reference is made below to Figure 6 which shows a structural diagram of a computer system 600 suitable for implementing the electronic device of the embodiments of the present application. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0130] As Figure 6 shown, the computer system 600 comprises a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage portion 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0131] The following components are connected to the I / O interface 605: an input portion 606 comprising a keyboard, a mouse, etc.; an output portion 607 comprising a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 comprising a hard disk, etc.; and a communication portion 609 comprising a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as needed, so that computer programs read therefrom are installed in the storage portion 608 as needed.

[0132] In particular, the process described by the above main steps diagram can be implemented as a computer software program according to the embodiments disclosed herein. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the method shown in the main steps diagram. In the above embodiments, the computer program can be downloaded and installed from a network by the communication section 609, and / or installed from the detachable medium 611. When the computer program is executed by the central processing unit 601, the above functions defined in the system of the present application are performed.

[0133] It should be noted that the computer readable medium shown in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium that can send, propagate or transmit the program for use by or in connection with an instruction execution system, device or apparatus. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0134] The computer program product of the present application can be a computer program implemented on one or more computers. The program itself can be stored on a computer-readable medium, such as a compact diskette, a floppy disk, a RAM of a computer, a read-only memory (ROM), or an erasable programmable read-only memory (EPROM). The computer program product can also be a computer program product comprising a computer via which the computer program is stored and executed, and a computer-readable medium on which the computer program is stored. The computer program product can also be a computer program product comprising a computer via which the computer program is stored and executed, and a computer-readable medium on which the computer program is stored. The computer program product can also be a computer program product comprising a computer via which the computer program is stored and executed, and a computer-readable medium on which the computer program is stored.

[0135] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware. The units described can also be arranged in a processor, for example, a processor can be described as including a binning unit, an optimization unit, and a threshold calculation unit. Among them, the name of these units does not constitute a limitation to the units themselves in some cases, for example, the binning unit can also be described as "a unit for providing a sample set to the optimization unit".

[0136] As another aspect, the application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to perform the steps including: obtaining evaluation scores of a plurality of samples pre-labeled with positive sample labels or negative sample labels, dividing the samples into a plurality of sample sets according to the evaluation scores, and arranging the sample sets in a preset order of the evaluation scores; counting cumulative positive sample proportions and cumulative negative sample proportions of each sample set; determining a positive sample rejection proportion factor and a positive sample proportion tuning item of the sample set according to the cumulative positive sample proportion of the sample set; determining a negative sample distribution correction factor and a negative sample proportion tuning item of the sample set based on the evaluation scores of the samples in the sample set; and determining a binary classification threshold for the evaluation scores according to the positive sample proportion tuning item and the negative sample proportion tuning item. Alternatively, the computer readable medium carries one or more programs, which, when executed by the device, cause the device to perform the steps including: obtaining evaluation scores output by a machine learning model to be evaluated for a plurality of samples pre-labeled with positive sample labels or negative sample labels, dividing the samples into a plurality of sample sets according to the evaluation scores, and arranging the sample sets in a preset order of the evaluation scores; counting cumulative positive sample proportions and cumulative negative sample proportions of each sample set; determining a positive sample rejection proportion factor and a positive sample proportion tuning item of the sample set according to the cumulative positive sample proportion of the sample set; determining a negative sample distribution correction factor and a negative sample proportion tuning item of the sample set based on the evaluation scores of the samples in the sample set; and determining a discrimination evaluation index of the machine learning model according to the positive sample proportion tuning item and the negative sample proportion tuning item.

[0137] The computer program product of the embodiment of the application comprises a computer program, which, when executed by a processor, implements the classification threshold determination or machine learning model evaluation method provided by the embodiment of the application.

[0138] In the technical solution of the embodiment of the application, the positive sample rejection proportion factor is introduced as a penalty item for rejecting positive users, so that the positive sample proportion in the KS index calculation logic is tuned; the negative sample distribution correction factor is introduced to correct the negative sample proportion in the KS index calculation logic to the overall distribution of positive samples, so that the calculation dimensions of the positive sample proportion and the negative sample proportion are consistent, and the interpretability of the KS index calculation logic is improved. In this way, a suitable classification threshold that meets business characteristics is determined with a small amount of calculation without relying on artificial experience and preferences, and on this basis, the embodiment of the application provides a refined calculation logic based on interest rates and delinquency loss rates for scenarios that pursue value data. Finally, the embodiment of the application also provides a reliable discrimination evaluation index of the machine learning model in a similar manner, which helps to accurately evaluate the discrimination ability of the machine learning model close to the business scenario.

[0139] The foregoing detailed description has set forth various embodiments of the devices and / or processes via the use of specific terminology. However, embodiments thereof can be practiced with the exact description not being set forth but with the same essence; the principles set forth herein can be practiced with plasticity in a manner leading to structurally equivalent devices and / or processes. Therefore, this description is not to be construed as limiting; the scope of the claims will be construed in the broadest context allowed by law.

Claims

1. A user classification method, executed by a computer device, characterized in that, The method comprises the following steps: obtaining evaluation scores output by a machine learning model for a plurality of sample users pre-labeled with positive or negative user labels, dividing the sample users into a plurality of user sets according to the evaluation scores, and arranging the user sets in a preset order of the evaluation scores; wherein the sample users labeled with positive user labels are users with a default or overdue risk score less than a preset threshold, and the sample users labeled with negative user labels are users with a default or overdue risk score greater than or equal to the preset threshold; statistically determining the proportion of positive users and the proportion of negative users in each user set, determining a positive sample rejection proportion factor of each user set according to the proportion of positive users in the user set, and determining a positive sample proportion tuning term of the user set based on the difference between the proportion of positive users and the positive sample rejection proportion factor, or determining the positive sample proportion tuning term of the user set based on the product of the difference between the proportion of positive users and the positive sample rejection proportion factor and a preset convergence coefficient; calculating the average value of the evaluation scores of each sample user in any user set; in the case that the evaluation score is positively correlated with the positive sample, determining the negative sample distribution correction factor of the user set as the quotient of the average value of the evaluation scores and the opposite of the average value of the evaluation scores; wherein the opposite of any value is the difference between one and the value; in the case that the evaluation score is negatively correlated with the negative sample, determining the negative sample distribution correction factor of the user set as the quotient of the opposite of the average value of the evaluation scores and the average value of the evaluation scores; and determining a negative sample proportion tuning term of the user set based on the proportion of negative users in the user set and the negative sample distribution correction factor, comprising: multiplying the proportion of negative users and the negative sample distribution correction factor of any user set to obtain the negative sample proportion tuning term of the user set; wherein the evaluation score is a value between zero and one; determining the difference between the positive sample proportion tuning term and the negative sample proportion tuning term of each user set, determining the user set with the maximum difference value as a target user set, and determining a binary classification threshold for the evaluation score output by the machine learning model using the evaluation scores of the sample users in the target user set to classify the to-be-classified users based on the binary classification threshold.

2. The method of claim 1, wherein, The determination of the positive sample rejection proportion factor and the positive sample proportion tuning term of each user set according to the proportion of positive users in the user set comprises: determining the positive sample rejection proportion factor of each user set according to the proportion of positive users in the user set, and determining the positive sample proportion tuning term of the user set based on the proportion of positive users and the positive sample rejection proportion factor; The determination of the negative sample distribution correction factor and the negative sample proportion tuning term of each user set based on the evaluation scores of the sample users in the user set comprises: determining the negative sample distribution correction factor of each user set based on the evaluation scores of the sample users in the user set, and determining the negative sample proportion tuning term of the user set based on the proportion of negative users in the user set and the negative sample distribution correction factor.

3. The method of claim 1, wherein, The preset order is ascending order or descending order of the evaluation scores; and the step of dividing the sample users into a plurality of user sets according to the evaluation scores comprises: arranging the sample users in the preset order, and dividing a first number of adjacent sample users in each user set in turn, wherein each sample user belongs to only one user set, and the first number is any number in a preset number interval; or arranging the sample users in the preset order, dividing the value range of the evaluation scores of the sample users into a preset number of non-intersecting score intervals at equal intervals, and dividing sample users with the same evaluation score in the same score interval into the same user set; or determining each sample user as a user set.

4. The method of claim 1, wherein, The positive user ratio of any user set is the proportion of the number of positive users in the user set and each user set before the user set in the total number of positive users, and the negative user ratio of any user set is the proportion of the number of negative users in the user set and each user set before the user set in the total number of negative users; and the step of calculating the positive user ratio and the negative user ratio of each user set comprises: calculating the number of positive users and the number of negative users of each user set in turn according to the arrangement order of the user sets, and determining the positive user ratio and the negative user ratio of each user set according to the number of positive users and the number of negative users.

5. The method of claim 1, wherein, The step of determining the positive sample rejection ratio factor of each user set according to the positive user ratio of the user set comprises: determining the difference between one and the positive user ratio as the positive sample rejection ratio factor of the same user set.

6. The method of claim 1, wherein, The step of determining the binary classification threshold of the evaluation score output by the machine learning model by using the evaluation scores of the sample users in the target user set comprises: determining the average value of the evaluation scores of the sample users in the target user set as the binary classification threshold.

7. The method of claim 1, wherein, The evaluation score is a value between zero and one; and in the case where the binary classification threshold is used to form the maximum value data, for any user set: the positive sample rejection ratio factor of the user set is the difference between one and the positive user ratio of the user set; the positive sample ratio adjustment term of the user set is the product of the difference between the positive user ratio of the user set and the positive sample rejection ratio factor, a preset convergence coefficient, and a preset average interest rate; the negative sample distribution correction factor of the user set is calculated by the following steps: calculating the average value of the evaluation scores of the sample users in the user set; in the case where the evaluation score is positively correlated with the positive sample, determining the quotient of the average value of the evaluation scores and the opposite of the average value of the evaluation scores as the negative sample distribution correction factor of the user set; wherein the opposite of any value is the difference between one and the value; in the case where the evaluation score is positively correlated with the negative sample, determining the quotient of the opposite of the average value of the evaluation scores and the average value of the evaluation scores as the negative sample distribution correction factor of the user set; and the negative sample ratio adjustment term of the user set is the product of the negative user ratio of the user set, the negative sample distribution correction factor, and a preset overdue loss rate.

8. A user classification apparatus characterized by comprising: ​ a binning unit configured to obtain evaluation scores output by a machine learning model for a plurality of sample users pre-labeled with positive or negative user labels, divide the sample users into a plurality of user sets according to the evaluation scores, and arrange the user sets in a preset order of the evaluation scores; wherein the sample users labeled with positive user labels are users with a default or overdue score less than a preset threshold, and the sample users labeled with negative user labels are users with a default or overdue score greater than or equal to the preset threshold; a tuning unit configured to calculate a positive user proportion and a negative user proportion in each user set, determine a positive sample rejection proportion factor of the user set according to the positive user proportion of the user set, and determine a positive sample proportion tuning term of the user set based on a difference between the positive user proportion and the positive sample rejection proportion factor, or a product of the difference and a preset convergence coefficient; calculate an average value of evaluation scores of sample users in any user set; in a case where the evaluation scores are positively correlated with positive samples, determine a negative sample distribution correction factor of the user set as a quotient of the average value of the evaluation scores and an opposite number of the average value of the evaluation scores; wherein the opposite number of any value is a difference between one and the value; in a case where the evaluation scores are negatively correlated with negative samples, determine the negative sample distribution correction factor of the user set as a quotient of an opposite number of the average value of the evaluation scores and the average value of the evaluation scores; and form a negative sample proportion tuning term of the user set based on the negative user proportion of the user set and the negative sample distribution correction factor, including multiplying the negative user proportion of any user set and the negative sample distribution correction factor to obtain the negative sample proportion tuning term of the user set; wherein the evaluation scores are values between zero and one; a threshold calculation unit configured to determine a difference between the positive sample proportion tuning term and the negative sample proportion tuning term of each user set, determine a target user set having a maximum value in the difference, and determine a binary classification threshold for the evaluation scores output by the machine learning model using the evaluation scores of sample users in the target user set to classify users to be classified based on the binary classification threshold.

9. An electronic device, comprising: comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-7.

Images