Method for automatically evaluating refresh air control model based on Swap Set analysis

Abstract

The invention relates to the field of risk control, in particular to a method for automatically evaluating a updating risk control model based on Swap Set analysis, and the method comprises the steps: deploying a new model and an old model on a decision engine; obtaining a pass threshold of each model, and forming a model performance decision according to a preset method based on the pass threshold, the model performance decision including passing of both new and old models, refusal of both new and old models, passing of the old model and refusal of the new model, and refusal of the old model and passing of the new model; based on the model performance decision, triggering to obtain an optimization target; and performing Swap Set analysis based on the optimization target to obtain an analysis result, and generating an analysis report. According to the method, the service performance is analyzed on the basis of model performance analysis, and the analysis report is obtained, so that the accuracy and intuition of new and old model replacement evaluation are improved.

Description

Technical Field

[0001] The present invention relates to the field of risk control, and in particular to a method for automatically evaluating and updating a risk control model based on Swap Set analysis. Background Art

[0002] In credit applications, after a new model is developed, it is necessary to comprehensively compare the new model with the old model and deeply evaluate the differences between the new and old models. If the difference between the new and old models is weak, it means that the new model has not brought significant improvement, and the optimization direction of the model needs to be reconsidered. This evaluation is time-consuming and laborious, and it is difficult to accurately quantify the degree of advantage of the new model over the old model.

[0003] At present, the industry usually uses methods such as KS curve and AUC-ROC curve to evaluate the performance of the model. AUC represents the area under the ROC curve, which is used to measure the model's ability to distinguish between positive and negative samples. The closer the AUC value is to 1, the better the classification effect of the model. The KS statistic is used to measure the model's ability to distinguish between good and bad customers. The larger the KS statistic, the stronger the model's ability to distinguish. However, these methods have certain limitations. They are not only time-consuming and labor-intensive, but also not very interpretable for non-professionals. Summary of the invention

[0004] The present invention aims to provide a method for automatically evaluating and updating a risk control model based on Swap Set analysis, analyzing business performance based on model performance analysis, and obtaining an analysis report to improve the accuracy and intuitiveness of the replacement evaluation of the old and new models.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] The method of automatically evaluating and updating the risk control model based on Swap Set analysis includes:

[0007] S1, deploy the new and old models on the decision engine;

[0008] S2, obtaining a passing threshold of each model, and forming a model performance decision based on the passing threshold according to a preset method, wherein the model performance decision includes: both the new and old models pass, both the new and old models reject, the old model passes and the new model rejects, and the old model rejects and the new model passes;

[0009] S3, triggering the acquisition of optimization targets based on model performance decisions;

[0010] S4, perform Swap Set analysis based on the optimization target, obtain analysis results, and generate an analysis report.

[0011] The principles and advantages of this solution are as follows: in actual application, different types of models correspond to different passing thresholds. After the new and old models are deployed to the decision engine, the passing threshold of the model is obtained, and decisions are made from the perspective of the model's own performance based on the passing threshold, so as to facilitate the screening of models whose performance meets the requirements and further apply them. When the model performance meets the requirements, the model needs to be iterated, that is, how to replace the old model with the new model. The replacement of the old and new models involves policy modifications, which will lead to business impacts. By obtaining the optimization target and analyzing it through Swap Set based on the optimization target, the business performance is analyzed on the basis of the model performance analysis, so as to maximize the business value of the new model, avoid the risks caused by policy changes, and provide more powerful data support for the formulation of risk control strategies.

[0012] Preferably, as an improvement, the preset method includes: evaluating the classification performance, model stability, interpretability, calibration ability, complexity and generalization ability, and model robustness of the new and old models respectively, obtaining the evaluation results, judging whether the evaluation results reach the passing threshold, and generating a model performance decision based on the judgment results.

[0013] Technical effect: By evaluating the performance of the model, the effectiveness and reliability of the model can be ensured to help financial institutions make credit decisions.

[0014] Preferably, as an improvement, the optimization objectives include approval rate, bad debt rate, response rate and profit.

[0015] Technical effect: It is convenient to conduct Swap Set analysis based on different swap targets.

[0016] Preferably, as an improvement, the S4 further includes:

[0017] S41, classifying the customer groups based on the optimization target, wherein the categories include the customer groups that are passed by the new model and rejected by the old model, the customer groups that are passed by the new model and rejected by the old model, and the customer groups that are passed or rejected by both the new and old models;

[0018] S42, analyze the scores of new and old customer groups based on Swap Set;

[0019] S43, judging whether the replacement standard is met based on the score, and triggering the model update when the replacement standard is met.

[0020] Technical effect: The analysis accuracy can be improved by optimizing the target. The impact of the model on business performance can be analyzed through Swap Set analysis to ensure the practicality of the updated model.

[0021] Preferably, as an improvement, the Swap Set analysis includes: taking the customer group rejected by the new model through the old model as a swap-in, taking the customer group rejected by the new model through the old model as a swap-out, obtaining the new customer group after the swap-in and swap-out, predicting the business impact indicators for the new customer group and the old customer group respectively, and comprehensively obtaining the scores of the new customer group and the old customer group.

[0022] Technical effect: It is convenient to replace good accounts and bad accounts when the model is updated, and the overall bad debt rate is reduced by replacing old bad accounts with more good accounts.

[0023] Preferably, as an improvement, the business impact indicators include: average number of credit items, overdue rate of amount, loan conversion rate and average customer unit revenue of the final product.

[0024] Technical effect: It is easy to compare and analyze the new and old models from the perspective of business impact, and help reduce the business impact risk caused by the replacement.

[0025] Preferably, as an improvement, when the replacement standard is not met, it also includes S44, performing customer conversion willingness scoring through a conversion willingness scoring model, obtaining customer groups with conversion willingness scores higher than a threshold in the replaced customer groups, and performing false rejection analysis.

[0026] Technical effect: False rejection analysis based on conversion intention can improve customer experience while retaining high-quality customers and re-analyzing the model.

[0027] Preferably, as an improvement, the conversion willingness scoring model includes:

[0028] y m =SR m *a+x m *b+ε

[0029] Among them, SR m is the savings rate, x m is brand loyalty, a and b are weight coefficients, and ε is the adjustment constant coefficient.

[0030] Technical effect: Facilitates analysis of false rejections. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The figure is a flowchart of a method for automatically evaluating and updating a risk control model based on Swap Set analysis. DETAILED DESCRIPTION

[0032] The following is further described in detail through specific implementation methods:

[0033] The embodiment is basically as shown in the attached Figure 1 As shown:

[0034] The method of automatically evaluating and updating the risk control model based on Swap Set analysis includes:

[0035] S1, deploy the old and new models on the decision engine to ensure that the models can smoothly transition and play their due role in the actual business environment. Specifically, conduct compliance checks on the models during the model preparation phase and write detailed model documents, including model training data, feature selection, evaluation indicators, model explanations and other information; after the model is prepared, deploy the new model to different business scenarios in stages according to business needs and model performance; set up a rollback mechanism to ensure that the old model can be quickly rolled back when problems occur, and conduct continuous monitoring.

[0036] S2, obtaining the passing threshold of each model, and forming a model performance decision based on the passing threshold according to a preset method, wherein the model performance decision includes passing, rejecting, the old model passing and the new model rejecting, and the old model rejecting and the new model passing; the preset method includes: respectively evaluating the classification performance, model stability, interpretability, calibration ability, complexity and generalization ability, and model robustness of the old and new models, obtaining the evaluation results, judging whether the evaluation results reach the passing threshold, and generating a model performance decision based on the judgment results.

[0037] The classification performance is evaluated by the F1 score, which is the harmonic mean of precision and recall; the model stability is evaluated by the stability index, which indicates the stability of the model in different time periods or different populations. A stability index close to 1 indicates that the model has good stability; the interpretability is evaluated by the feature importance, which indicates the ability to understand the importance of features to model prediction; the calibration ability is evaluated by the Brier Score, which indicates the accuracy of the model's prediction probability. The lower the Brier Score, the more accurate the model's prediction; the complexity and generalization ability are evaluated by cross-validation, which evaluates the generalization ability of the model by dividing the data set into a training set and a validation set; the model robustness is evaluated by sensitivity analysis, which indicates the sensitivity of the model to changes in input data. The evaluation result is obtained by assigning weights to the above evaluation. The passing threshold is obtained by querying the preset model type-passing threshold mapping table.

[0038] S3, triggering the acquisition of optimization targets based on model performance decisions, that is, when the model performance meets the passing threshold, further analysis is triggered, and when the model performance does not meet the passing threshold, the deployment of the new model is terminated. In this embodiment, the model performance decision is triggered when both the new and old models pass and the old model is rejected but the new model passes, which triggers the acquisition of optimization targets.

[0039] The optimization objectives include approval rate, bad debt rate, response rate and profit; for example, when the risk remains unchanged, the approval rate is increased (i.e., a larger proportion of high-quality accounts are replaced); when the approval rate remains unchanged, the bad debt rate is reduced (i.e., a larger proportion of bad accounts are replaced); when the approval rate and bad debt rate remain unchanged, the customer's response rate is increased or the loan amount is increased (i.e., accounts with higher response rates are replaced); the approval rate is increased, the risk is reduced, the response rate is increased and the profit is increased (i.e., accounts with higher LTV are replaced. Approval rate refers to the ratio of the number of credit applications approved by a financial institution to the total number of applications within a certain period of time; bad debt rate refers to the ratio of the loan balance that has not been repaid on time within a certain period of time to the total loan balance; response rate refers to the ratio of the number of people who respond to marketing information (such as applying for loans) in marketing activities to the total number of people who receive marketing information; profit refers to the remainder after deducting the total cost from the total income obtained by a financial institution within a certain period of time.

[0040] S4, performing Swap Set analysis based on the optimization target, obtaining the analysis results, and generating an analysis report. The S4 also includes: S41, classifying the customer groups based on the optimization target, the categories include the customer groups rejected by the new model through the old model, the customer groups passed by the new model through the old model, and the customer groups passed or rejected by both the new and old models; S42, analyzing the scores of the new customer groups and the old customer groups based on the Swap Set; taking the customer groups rejected by the new model through the old model as the swap-in, and taking the customer groups passed by the new model through the old model through the new model as the swap-out, obtaining the new customer groups after the swap-in and swap-out, and predicting the business impact indicators for the new customer groups and the old customer groups respectively, and the business impact indicators include: average credit items, overdue rate of amount, loan conversion rate, and average customer unit revenue of the final product. The old credit customer group is evaluated for business impact indicators through historical data. The new credit customer group includes the customer group that still maintains some credit and the customer group that is replaced. The customer group that still maintains some credit is evaluated for business impact indicators through historical data. If there is no historical data for the replaced customer group, it is sorted after prediction by the new model, and the historical data of the customer group with the same score segment is obtained to evaluate the business impact indicators. The scores of the new customer group and the old customer group are obtained by combining the business impact indicators through the scoring method. S43, based on the score, it is judged whether the replacement standard is met. The replacement standard is a constant threshold. When the replacement standard is met, the model update is triggered.

[0041] When the replacement standard is not met, the process further includes S44, wherein a customer group conversion willingness score is scored using a conversion willingness scoring model, a customer group whose conversion willingness score is higher than a threshold is obtained from the replaced customer group, and a false rejection analysis is performed; the conversion willingness scoring model includes:

[0042] y m =SR m *a+x m *b+ε

[0043] Among them, SR mis the savings rate, which is the ratio of savings to disposable income. A higher savings rate means more surplus income. m is brand loyalty, which is the possibility of customers choosing the brand when faced with alternative options, a and b are weight coefficients, and ε is the adjustment constant coefficient.

[0044] For customers whose conversion intention scores are higher than the threshold, there is a probability of false rejection due to adverse selection caused by unreasonable access measurement or unreasonable equity strategy. In order to reduce the impact of false rejection losses on the evaluation of the new model, false rejection analysis is required, and the customers determined to be falsely rejected are added to the new customer group and analyzed again.

[0045] The above is only an embodiment of the present invention, and the common knowledge such as the known specific technical solutions and / or characteristics in the solution is not described in detail here. It should be pointed out that for those skilled in the art, without departing from the technical solution of the present invention, several modifications and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for automatically evaluating and updating a risk control model based on Swap Set analysis, characterized in that: include: S1, deploy the new and old models on the decision engine; S2, obtaining a passing threshold of each model, and forming a model performance decision based on the passing threshold according to a preset method, wherein the model performance decision includes: both the new and old models pass, both the new and old models reject, the old model passes and the new model rejects, and the old model rejects and the new model passes; S3, triggering the acquisition of optimization targets based on model performance decisions; S4, perform Swap Set analysis based on the optimization target, obtain analysis results, and generate an analysis report.

2. The method for automatically evaluating and updating the risk control model based on Swap Set analysis according to claim 1 is characterized in that: The preset method includes: evaluating the classification performance, model stability, interpretability, calibration ability, complexity and generalization ability, and model robustness of the new and old models respectively, obtaining the evaluation results, judging whether the evaluation results reach the passing threshold, and generating a model performance decision based on the judgment results.

3. The method for automatically evaluating and updating the risk control model based on Swap Set analysis according to claim 1, characterized in that: The optimization objectives include approval rate, bad debt rate, response rate and profit.

4. The method for automatically evaluating and updating the risk control model based on Swap Set analysis according to claim 1 is characterized in that: The S4 further comprises: S41, classifying the customer groups based on the optimization target, wherein the categories include the customer groups that are passed by the new model and rejected by the old model, the customer groups that are passed by the new model and rejected by the old model, and the customer groups that are passed or rejected by both the new and old models; S42, analyze the scores of new and old customer groups based on Swap Set; S43, judging whether the replacement standard is met based on the score, and triggering the model update when the replacement standard is met.

5. The method for automatically evaluating and updating the risk control model based on Swap Set analysis according to claim 4 is characterized in that: The Swap Set analysis includes: using the customer groups rejected by the new model through the old model as swap-in, using the customer groups rejected by the new model through the old model as swap-out, obtaining new customer groups after the swap-in and swap-out, predicting business impact indicators for the new customer groups and the old customer groups respectively, and comprehensively obtaining scores for the new customer groups and the old customer groups.

6. The method for automatically evaluating and updating the risk control model based on Swap Set analysis according to claim 5 is characterized in that: The business impact indicators include: average number of credit items, overdue rate, loan conversion rate and average customer unit revenue of the final product.

7. The method for automatically evaluating and updating the risk control model based on Swap Set analysis according to claim 6 is characterized in that: When the replacement standard is not met, it also includes S44, using the conversion willingness scoring model to score the customer group conversion willingness, obtain the customer group whose conversion willingness score is higher than the threshold in the replaced customer group, and perform false rejection analysis.

8. The method for automatically evaluating and updating the risk control model based on Swap Set analysis according to claim 7 is characterized in that: The conversion intention scoring model includes: y m =SR m *a+x m *b+ε Among them, SR m is the savings rate, x m is brand loyalty, a and b are weight coefficients, and ε is the adjustment constant coefficient.

Images