Model monitoring system and method for quasi real-time calculation

A model system and quasi-real-time technology, applied in the field of quasi-real-time computing model monitoring system, can solve the problems of inability to guarantee the timeliness of model monitoring, labor costs and time costs, to ensure the accuracy of calculation logic, reduce calculation complexity, and logic clear effect

Pending Publication Date: 2021-07-23
SICHUAN XW BANK CO LTD
0 Cites 0 Cited by

AI-Extracted Technical Summary

Problems solved by technology

[0004] Aiming at the problem that the timeliness of model monitoring cannot be guaranteed in the prior art, and the monitoring of the model requires a certain amount of manpower and time, the pre...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Abstract

The invention discloses a quasi real-time calculation model monitoring system and method, belongs to the field of artificial intelligence monitoring, and solves the problems that in the prior art, the timeliness of model monitoring cannot be guaranteed, and certain labor cost and time cost are needed for model monitoring, the quasi real-time calculation model monitoring system comprises a model data layer, and the model data layer is connected with a model system; the model information processing layer is connected with a data mart, a model monitoring index calculation layer and a model monitoring formula calculation layer, and the model monitoring formula calculation layer is connected with a model grading early warning system. The model grading early warning system sends the warning information to a model developer through a mail or a short message and the like, and the model developer optimizes or offline the model according to the information of the model early warning system, so that intelligent real-time monitoring of the model is realized, and the labor cost and the time cost are reduced.

Application Domain

Technology Topic

Data martText messaging +8

Image

  • Model monitoring system and method for quasi real-time calculation
  • Model monitoring system and method for quasi real-time calculation
  • Model monitoring system and method for quasi real-time calculation

Examples

  • Experimental program(1)

Example Embodiment

[0050] Embodiment:
[0051] A model monitored by a model, including:
[0052] Model data layer: receive model data, and processes the model data set;
[0053] Model Information Processing Layer: Receive the model data set and data set of data marts from the model data layer and the data set of data markets, and integrates into model information data sets;
[0054] Model monitoring index calculation layer: Receive the model information data set from the model information processing layer real time, and calculate the monitoring index calculation based on the model information data set to obtain the results of the monitoring index;
[0055] Model monitoring formula calculation layer: According to the monitoring index calculation results, model monitoring formula calculations, and generate alarm content according to the model monitor formula calculation;
[0056] Model Hierarchy Warning Layer: Promoting a stratified warning based on the alarm content and the setup warning topic.
[0057] The model monitoring formula calculation layer downlink model analysis system, model analysis system underwent model system.
[0058] The model data layer is connected to the model system.
[0059] A method of quarantining models calculated in real time, including the following steps:
[0060] Step A: Turn the model system and model monitoring system;
[0061] Step B: Collect model data and data mart data, and process the model data and data market data as a data set;
[0062] Step c: The model index calculation is based on the data set, and the result of the index calculation is obtained;
[0063] Step D: Monitor formula calculation according to the indicator calculation results, and draw the results of the monitoring formula;
[0064] Step E: Generate alarm content and sends to the model analysis system based on the monitor formula.
[0065] Step f: Model Analysis System The model is offline according to the alarm content, and the feature is offline or model optimization.
[0066] The specific steps of the step c are:
[0067] Step C1, calculate the PSI index by the model monitoring index calculation layer by the model monitoring index calculation layer, PSI calculation formula: Divide the model results into n intervals, where A i Indicates the actual proportion of the I segment, equal to the number of actual samples in the i-segment, in addition to the actual total sample; i Indicates the expected proportion of the I segment interval, equal to the number of expected samples in the i-th segment except for the total sample number.
[0068] Step C2, calculate the KS indicator, KS calculation formula: ks = max (tp + fn), FPR = FP / (FP + TN), where TPR = FP / (FP + TN), where TPR is The real class, the FPR is a negative class, the TP is the correct affirmation number, FP is a false positive, no matched incorrect number, TN is a non-match number of correctly rejected, FN is a missing, did not find the number of correct match ;
[0069] Step C3, calculate the AUC index by the model monitoring index calculation layer, and approximate the approximation of the trapezoidal method and the ROC AuCH method.
[0070] The specific steps of the step d are:
[0071] Step D1, by the model monitor formula calculation layer monitoring the PSI index exception, PSI is in [0, 0.1), the model feature stability is good, and the model does not require monitoring subsequent changes; PSI is in [0.1, 0.25), the model is unstable, Indicates that the model changes, needs to continue to monitor subsequent changes; when PSI> 0.25, the model needs to be analyzed;
[0072] Step D2, by model monitoring formula calculation layer monitoring KS index exception, KS value is [0, 1], when the value of value <0.2, the model is not distinguished; the value of Ks is [0.2-0.75), and the model is distinguished The better the ability, and the KS is getting 0.75, the better the model is, the better the model; KS> 0.75, the model is abnormal;
[0073] Step D3, by model monitoring formula computing layer monitoring AUC index exception, AUC is [0.5, 1], AUC is [0.5-0.6), indicating that the model is free; AUC is [0.6-1) When the model is distinguished, the AUC index has near 1, the better the model of the model;
[0074] Step D4, through the model monitor formula computing layer through the alarm interface configuration specifies the alarm level, the alarm object, the alarm cycle, the alarm formula, and the alarm level is divided into one alarm, secondary alarm, three alarm, four Level alarm, five alarms; the alarm object group is the development group of the model; the alarm cycle includes 2 dimensions per hour, that is, 1 hour warning 1 and 1 day warning once.
[0075] In the above-described embodiment, the model monitoring system, the model system, and the data market are docked by the FTP file transmission mode and the MQ queue mode, where the model system receives the model data layer, the model data layer receives the model data, will model the model Data processing is a model data set, and the model data set is transferred to the model information processing layer. The model information processing layer receives the data set from the model data layer and the data market, with the data set of the model data layer including credit model, fraud model , Non-normal trading model, data sets of data markets include case-based data sets, non-normal transaction data sets, credit overdue data sets, black gray list data sets, model information processing layers to become model information data Set, and transfer to the model monitoring index calculation layer, the model monitoring index calculation layer is calculated based on the model information data set, including the KS indicator, PSI indicator, AUC index, where the KS indicator can express the distinguishing capacity of the model, and the PSI indicator can manifest The stability of the distribution of each fractional end and the modeling sample distribution, the AUC index can express the learning ability of the model, obtain the indicator calculation result after the calculation result is transferred to the model monitoring formula calculation layer, the model monitoring formula calculation layer calculates the index calculation As a result, write the monitoring formula, determine the stability of the model, and distinguish between the model, generate alarm content, and turn the alarm content into the model classification early warning layer, the model hierarchical early warning layer issues alarm information by setting the alarm group configuration When the model monitoring formula calculation layer generates alarm content, and when the model hierarchical early warning layer issues alarm information to the model analysis system, the model analysis system can optimize the model based on the alarm information content, the model offline, and the feature offline operation operation.
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

no PUM

Description & Claims & Application Information

We can also present the details of the Description, Claims and Application information to help users get a comprehensive understanding of the technical details of the patent, such as background art, summary of invention, brief description of drawings, description of embodiments, and other original content. On the other hand, users can also determine the specific scope of protection of the technology through the list of claims; as well as understand the changes in the life cycle of the technology with the presentation of the patent timeline. Login to view more.
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Similar technology patents

Classification and recommendation of technical efficacy words

Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products