Model monitoring system and method for quasi real-time calculation

[0050] Embodiment:

[0051] A model monitored by a model, including:

[0052] Model data layer: receive model data, and processes the model data set;

[0053] Model Information Processing Layer: Receive the model data set and data set of data marts from the model data layer and the data set of data markets, and integrates into model information data sets;

[0054] Model monitoring index calculation layer: Receive the model information data set from the model information processing layer real time, and calculate the monitoring index calculation based on the model information data set to obtain the results of the monitoring index;

[0055] Model monitoring formula calculation layer: According to the monitoring index calculation results, model monitoring formula calculations, and generate alarm content according to the model monitor formula calculation;

[0056] Model Hierarchy Warning Layer: Promoting a stratified warning based on the alarm content and the setup warning topic.

[0057] The model monitoring formula calculation layer downlink model analysis system, model analysis system underwent model system.

[0058] The model data layer is connected to the model system.

[0059] A method of quarantining models calculated in real time, including the following steps:

[0060] Step A: Turn the model system and model monitoring system;

[0061] Step B: Collect model data and data mart data, and process the model data and data market data as a data set;

[0062] Step c: The model index calculation is based on the data set, and the result of the index calculation is obtained;

[0063] Step D: Monitor formula calculation according to the indicator calculation results, and draw the results of the monitoring formula;

[0064] Step E: Generate alarm content and sends to the model analysis system based on the monitor formula.

[0065] Step f: Model Analysis System The model is offline according to the alarm content, and the feature is offline or model optimization.

[0066] The specific steps of the step c are:

[0067] Step C1, calculate the PSI index by the model monitoring index calculation layer by the model monitoring index calculation layer, PSI calculation formula: Divide the model results into n intervals, where A i Indicates the actual proportion of the I segment, equal to the number of actual samples in the i-segment, in addition to the actual total sample; i Indicates the expected proportion of the I segment interval, equal to the number of expected samples in the i-th segment except for the total sample number.

[0068] Step C2, calculate the KS indicator, KS calculation formula: ks = max (tp + fn), FPR = FP / (FP + TN), where TPR = FP / (FP + TN), where TPR is The real class, the FPR is a negative class, the TP is the correct affirmation number, FP is a false positive, no matched incorrect number, TN is a non-match number of correctly rejected, FN is a missing, did not find the number of correct match ;

[0069] Step C3, calculate the AUC index by the model monitoring index calculation layer, and approximate the approximation of the trapezoidal method and the ROC AuCH method.

[0070] The specific steps of the step d are:

[0071] Step D1, by the model monitor formula calculation layer monitoring the PSI index exception, PSI is in [0, 0.1), the model feature stability is good, and the model does not require monitoring subsequent changes; PSI is in [0.1, 0.25), the model is unstable, Indicates that the model changes, needs to continue to monitor subsequent changes; when PSI> 0.25, the model needs to be analyzed;

[0072] Step D2, by model monitoring formula calculation layer monitoring KS index exception, KS value is [0, 1], when the value of value <0.2, the model is not distinguished; the value of Ks is [0.2-0.75), and the model is distinguished The better the ability, and the KS is getting 0.75, the better the model is, the better the model; KS> 0.75, the model is abnormal;

[0073] Step D3, by model monitoring formula computing layer monitoring AUC index exception, AUC is [0.5, 1], AUC is [0.5-0.6), indicating that the model is free; AUC is [0.6-1) When the model is distinguished, the AUC index has near 1, the better the model of the model;

[0074] Step D4, through the model monitor formula computing layer through the alarm interface configuration specifies the alarm level, the alarm object, the alarm cycle, the alarm formula, and the alarm level is divided into one alarm, secondary alarm, three alarm, four Level alarm, five alarms; the alarm object group is the development group of the model; the alarm cycle includes 2 dimensions per hour, that is, 1 hour warning 1 and 1 day warning once.

[0075] In the above-described embodiment, the model monitoring system, the model system, and the data market are docked by the FTP file transmission mode and the MQ queue mode, where the model system receives the model data layer, the model data layer receives the model data, will model the model Data processing is a model data set, and the model data set is transferred to the model information processing layer. The model information processing layer receives the data set from the model data layer and the data market, with the data set of the model data layer including credit model, fraud model , Non-normal trading model, data sets of data markets include case-based data sets, non-normal transaction data sets, credit overdue data sets, black gray list data sets, model information processing layers to become model information data Set, and transfer to the model monitoring index calculation layer, the model monitoring index calculation layer is calculated based on the model information data set, including the KS indicator, PSI indicator, AUC index, where the KS indicator can express the distinguishing capacity of the model, and the PSI indicator can manifest The stability of the distribution of each fractional end and the modeling sample distribution, the AUC index can express the learning ability of the model, obtain the indicator calculation result after the calculation result is transferred to the model monitoring formula calculation layer, the model monitoring formula calculation layer calculates the index calculation As a result, write the monitoring formula, determine the stability of the model, and distinguish between the model, generate alarm content, and turn the alarm content into the model classification early warning layer, the model hierarchical early warning layer issues alarm information by setting the alarm group configuration When the model monitoring formula calculation layer generates alarm content, and when the model hierarchical early warning layer issues alarm information to the model analysis system, the model analysis system can optimize the model based on the alarm information content, the model offline, and the feature offline operation operation.