Model acceptance system and method

The model acceptance system automates and standardizes the model acceptance process, addressing inefficiencies and inconsistencies in manual reviews by using a structured approach with data preprocessing and automatic verification, enhancing efficiency and accuracy in financial risk control.

CN119089132APending Publication Date: 2024-12-06重庆富民银行股份有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411039547.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing manual review of risk control models in the financial industry is time-consuming, labor-intensive, and prone to subjective uncertainties, leading to inconsistent and inaccurate model acceptance results.

Method used

A model acceptance system and method that automates and standardizes the model acceptance process through modules for acquiring acceptance criteria, configuring verification tasks, preprocessing data, executing models, and automatically verifying outputs against predefined criteria, with features like real-time and historical data processing and early warnings for abnormal reports.

Benefits of technology

Enhances model acceptance efficiency and accuracy by reducing human influence, facilitating standardized and rapid deployment of models, and improving overall business performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089132A_ABST
    Figure CN119089132A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of finance, and discloses a model acceptance system and method, and the system comprises an acceptance rule obtaining module which is used for obtaining a pre-stored acceptance standard; the verification task configuration module is used for configuring a verification task and defining and configuring a verification task name and a verification input number, and the task name comprises a task ID, a product name, a scene name and a model name; the data preprocessing module is used for cleaning, converting and standardizing model input data; the model execution module is used for executing the to-be-accepted model and obtaining model output; the automatic verification module is used for outputting a standard result according to the acceptance standard automatic comparison model and generating an acceptance report; and the prompting module is used for giving out early warning when the acceptance report is abnormal. According to the invention, automation and standardization of model acceptance can be realized, and the efficiency and accuracy of model acceptance are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a model acceptance system and method. Background Art

[0002] In the financial industry, various prediction, risk assessment and decision support models are widely used. Especially in the whole life cycle of financial risk control, the pre-loan (the stage before and after successful credit approval) prediction model is used for pre-credit approval, the mid-loan (the stage after successful credit approval to before overdue) risk assessment model is used for mid-loan warning, and the post-loan (the stage after overdue) decision support model is used for collection management, etc. Different risk control stages are not independent, but closely linked. Therefore, the accuracy and stability of the risk control model are crucial to financial decision-making.

[0003] The continuous influence of risk control models throughout the entire life cycle of risk control means that the acceptance of risk control models has become a key step. At present, the acceptance of risk control models mainly relies on manual review, which is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in uncertainty and inconsistency in the acceptance results. Summary of the invention

[0004] The present invention aims to provide a model acceptance system and method to realize automation and standardization of model acceptance and improve the efficiency and accuracy of model acceptance.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] A model acceptance system, comprising:

[0007] An acceptance rule acquisition module is used to acquire pre-stored acceptance criteria;

[0008] Configuration verification task module, used to configure verification tasks, and define the configuration verification task name and the number of verification items. The task name includes task ID, product name, scenario name, and model name;

[0009] Data preprocessing module, used to clean, transform and standardize model input data;

[0010] Model execution module, used to execute the model to be accepted and obtain model output;

[0011] Automatic verification module, used to automatically compare the model output standard results according to the acceptance criteria and generate an acceptance report;

[0012] The prompt module is used to issue an early warning when the acceptance report is abnormal.

[0013] The principles and advantages of this solution are: in actual application, the acceptance criteria are obtained through the acceptance rule acquisition module; the acceptance rule library contains pre-defined acceptance criteria, and the pre-stored acceptance criteria are obtained based on the acceptance criteria to facilitate acceptance standardization; the configuration verification task module configures the verification task and defines the configuration verification task name and the number of verification items, and the verification task configuration facilitates subsequent management tracking; the data preprocessing module cleans, converts and standardizes the model input data to ensure data quality; the model execution module executes the model to be accepted after the data is input and obtains the model output; the automatic verification module automatically compares the model output standard results according to the acceptance criteria and generates an acceptance report for decision makers to refer to; the prompt module issues an early warning when the acceptance report is abnormal. This application realizes the automation and standardization of model acceptance, reduces the influence of human factors, improves the efficiency and accuracy of model acceptance, and helps financial institutions to quickly and accurately deploy and apply models and improve the overall business level.

[0014] Preferably, as an improvement, the configuration verification task module further includes:

[0015] The incoming file tracking submodule is used to track the incoming files of configuration verification tasks;

[0016] The scenario screening submodule is used to screen the incoming documents under the selected scenario from the tracked incoming documents;

[0017] The model association submodule is used to associate the selected model with the input quantity of the selected product and the selected scenario.

[0018] Technical effect: The above method realizes the tracking management of incoming documents.

[0019] Preferably, as an improvement, the model execution module further includes:

[0020] Online deployment submodule, used for online real-time data collection;

[0021] The variable processing submodule is used to process online real-time data to obtain online variables, and input the online variables into the model to be accepted.

[0022] Technical effect: Based on real-time data and online variables, it is convenient for model prediction.

[0023] Preferably, as an improvement, the model execution module further includes:

[0024] Offline deployment submodule, used to collect historical data and simulation data to obtain offline data sets;

[0025] The offline variable processing submodule is used to process the variables in the offline data set and make them consistent with the variables in the online environment.

[0026] Technical effect: It is easy to use historical data and simulated data as model predictions.

[0027] Preferably, as an improvement, the acceptance criteria include a systematic deviation acceptance criterion, the input data is screened from multiple dimensions to obtain several population subsets, the model output of the data subsets is obtained, and it is determined whether the output meets the consistency of the data set.

[0028] Technical effect: By obtaining data sets from different dimensions for verification, it is convenient to improve acceptance accuracy and avoid unfair predictions for special groups.

[0029] A model acceptance method, comprising:

[0030] S1, obtain the pre-stored acceptance criteria;

[0031] S2, configure the verification task, and define the configuration verification task name and the number of verification items;

[0032] S3, cleans, transforms and standardizes the model input data;

[0033] S4, execute the model to be accepted and obtain the model output;

[0034] S5, automatically compares the model output standard results according to the acceptance criteria and generates an acceptance report;

[0035] S6, issue an early warning when the acceptance report is abnormal.

[0036] Preferably, as an improvement, the S2 further includes:

[0037] S21, tracking the number of configuration verification task inputs;

[0038] S22, filtering the incoming application volume under the selected scenario from the tracked incoming application volume;

[0039] S23, associating the selected model with the input quantity of the selected product and the selected scenario.

[0040] Preferably, as an improvement, S4 includes:

[0041] S41, online real-time data collection;

[0042] S42, processing the online real-time data to obtain online variables, and inputting the online variables into the model to be accepted.

[0043] Preferably, as an improvement, the S4 further includes:

[0044] S43, collecting historical data and simulation data to obtain an offline data set;

[0045] S44, processes the variables in the offline dataset and makes them consistent with the variables in the online environment.

[0046] Preferably, as an improvement, the acceptance criteria include a systematic deviation acceptance criterion, the input data is screened from multiple dimensions to obtain several population subsets, the model output of the data subsets is obtained, and it is determined whether the output meets the consistency of the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The figure is a flow chart of a model acceptance system. DETAILED DESCRIPTION

[0048] The following is further described in detail through specific implementation methods:

[0049] The embodiment is basically as shown in the attached Figure 1 As shown:

[0050] A model acceptance system, comprising:

[0051] The acceptance rule acquisition module is used to obtain pre-stored acceptance criteria; the acceptance criteria include system deviation acceptance criteria, which screens the input data from multiple dimensions to obtain several group subsets, obtains the model output of the data subset, and determines whether the output meets the consistency of the data set. By obtaining data sets from different dimensions for verification, it is convenient to improve the accuracy of acceptance and avoid unfair predictions for special groups.

[0052] The configuration verification task module is used to configure the verification task and define the configuration verification task name and the number of verification submissions. The task name includes the task ID, product name, scenario name, and model name. The task name is to define a clear task name for this model verification task for subsequent management and tracking. The product name is to select a specific credit product from the product library to which the model will be applied. The scenario name is to filter according to the application scenario of the model, such as the personal credit application scenario or the small and micro credit application scenario. The number of verifications is to select according to the number of samples that need to be verified, and the number of verifications determines how many submissions will be used for model verification. The model name is to select the specific model that needs to be verified from the model library.

[0053] The configuration verification task module also includes: an application tracking submodule, which is used to track the application volume of the configuration verification task; for example, the application volume of the selected product, that is, the number of credit applications. A scenario screening submodule, which is used to screen the application volume under the selected scenario from the tracked application volume; and a model association submodule, which is used to associate the selected model with the application volume of the selected product and the selected scenario. The tracking management of the application is realized in the above manner.

[0054] The data preprocessing module is used to clean, transform and standardize the model input data to improve data usage efficiency. After the configuration verification task is completed, the associated model receives real-time data from the business environment, such as new applications, loans or transactions, and preprocesses and extracts features from the data to meet the model input requirements. The real-time data is then predicted through the associated risk control model to obtain the model output, that is, the prediction score or label.

[0055] The model execution module is used to execute the model to be accepted and obtain the model output; the model execution module also includes:

[0056] The online deployment submodule is used for online real-time data collection; the variable processing submodule is used to process online real-time data to obtain online variables, and input the online variables into the model to be accepted, and load the PMML model file package into the online environment; use real-time data and online variables to perform model prediction to obtain model output. The PMML model is a predictive model markup language used to represent and store machine learning models. It provides a cross-platform and cross-tool way to share and deploy predictive models, and is developed and maintained by data mining organizations. PMML is based on the XML standard and defines the elements and attributes used through XML Schema. The core parts include: data dictionary, data conversion, model definition and output. The data dictionary describes the structure and type of input data, and the data conversion defines how to convert the input data into the format required by the model. Each model type has its specific definition, including the parameters and structure of the model. Output is the output result and format of the specified model. The PMML prediction process conforms to the data mining analysis process to ensure that the model has consistent performance in different platforms and environments.

[0057] The model execution module also includes: an offline deployment submodule, which is used to collect historical data and simulation data to obtain an offline data set; an offline variable processing submodule, which is used to process the variables in the offline data set and make them consistent with the variables in the online environment, load the Python model file into the offline environment, use the offline data and offline variables to perform model prediction, and obtain the model output.

[0058] When the number of incoming cases reaches the selected number of verification cases, the system stops calculating. In this embodiment, after T+1 day, select the same number of samples for offline calculation; use the data from T+1 day as the offline calculation verification data set, perform preprocessing and feature engineering on the verification data set, extract information useful for model prediction, and calculate each incoming case offline according to the model logic to obtain the model output.

[0059] The automatic verification module is used to automatically compare the model output standard results according to the acceptance criteria and generate an acceptance report; the verification report includes data consistency, model consistency, result comparison, and threshold consistency; data consistency means verifying whether the online and offline data are consistent in feature selection, data preprocessing, and variable processing; model consistency means that the system verifies that the model files used online and offline are the same version, and the parameters and configuration of the model are consistent; result comparison means that the system verifies the prediction results of the online and offline models for the same data set, including model scores and other prediction indicators; threshold consistency means that the system verifies that the decision thresholds in the online and offline environments are consistent. If the verification is inconsistent, that is, when the acceptance report is abnormal, the prompt module will issue an early warning and mark the online and offline submission IDs in red.

[0060] This embodiment also includes a model acceptance method, including:

[0061] S1, obtaining pre-stored acceptance criteria; the acceptance criteria include system deviation acceptance criteria, filtering input data from multiple dimensions to obtain several population subsets, obtaining model outputs of the data subsets, and determining whether the outputs meet the consistency of the data set.

[0062] S2, configure the verification task, and define the configuration verification task name and the number of verification items; S2 also includes:

[0063] S21, tracking the number of configuration verification task inputs;

[0064] S22, filtering the incoming application volume under the selected scenario from the tracked incoming application volume;

[0065] S23, associating the selected model with the input quantity of the selected product and the selected scenario.

[0066] S3, cleans, transforms and standardizes the model input data;

[0067] S4, executing the model to be accepted and obtaining the model output; S4 includes:

[0068] S41, online real-time data collection;

[0069] S42, processing the online real-time data to obtain online variables, and inputting the online variables into the model to be accepted.

[0070] The S4 further comprises:

[0071] S43, collecting historical data and simulation data to obtain an offline data set;

[0072] S44, processes the variables in the offline dataset and makes them consistent with the variables in the online environment.

[0073] S5, automatically compares the model output standard results according to the acceptance criteria and generates an acceptance report;

[0074] S6, issue an early warning when the acceptance report is abnormal.

[0075] The above is only an embodiment of the present invention, and the common knowledge such as the known specific technical solutions and / or characteristics in the solution is not described in detail here. It should be pointed out that for those skilled in the art, without departing from the technical solution of the present invention, several modifications and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A model acceptance system, characterized in that: include: An acceptance rule acquisition module is used to acquire pre-stored acceptance criteria; Configuration verification task module, used to configure verification tasks, and define the configuration verification task name and the number of verification items. The task name includes task ID, product name, scenario name, and model name; Data preprocessing module, used to clean, transform and standardize model input data; Model execution module, used to execute the model to be accepted and obtain model output; Automatic verification module, used to automatically compare the model output standard results according to the acceptance criteria and generate an acceptance report; The prompt module is used to issue an early warning when the acceptance report is abnormal.

2. A model acceptance system according to claim 1, characterized in that: The configuration verification task module also includes: The incoming file tracking submodule is used to track the incoming files of configuration verification tasks; The scenario screening submodule is used to screen the incoming documents under the selected scenario from the tracked incoming documents; The model association submodule is used to associate the selected model with the input quantity of the selected product and the selected scenario.

3. A model acceptance system according to claim 2, characterized in that: The model execution module also includes: Online deployment submodule, used for online real-time data collection; The variable processing submodule is used to process online real-time data to obtain online variables, and input the online variables into the model to be accepted.

4. A model acceptance system according to claim 3, characterized in that: The model execution module also includes: Offline deployment submodule, used to collect historical data and simulation data to obtain offline data sets; The offline variable processing submodule is used to process the variables in the offline data set and make them consistent with the variables in the online environment.

5. A model acceptance system according to claim 4, characterized in that: The acceptance criteria include a systematic deviation acceptance criterion, which filters the input data from multiple dimensions to obtain several population subsets, obtains the model output of the data subsets, and determines whether the output meets the consistency of the data set.

6. A model acceptance method, characterized in that: include: S1, obtain the pre-stored acceptance criteria; S2, configure the verification task, and define the configuration verification task name and the number of verification items; S3, cleans, transforms and standardizes the model input data; S4, execute the model to be accepted and obtain the model output; S5, automatically compares the model output standard results according to the acceptance criteria and generates an acceptance report; S6, issue an early warning when the acceptance report is abnormal.

7. A model acceptance method according to claim 6, characterized in that: The S2 further includes: S21, tracking the number of configuration verification task inputs; S22, filtering the incoming application volume under the selected scenario from the tracked incoming application volume; S23, associating the selected model with the input quantity of the selected product and the selected scenario.

8. A model acceptance method according to claim 7, characterized in that: The S4 includes: S41, online real-time data collection; S42, processing the online real-time data to obtain online variables, and inputting the online variables into the model to be accepted.

9. A model acceptance method according to claim 8, characterized in that: The S4 further comprises: S43, collecting historical data and simulation data to obtain an offline data set; S44, processes the variables in the offline dataset and makes them consistent with the variables in the online environment.

10. A model acceptance method according to claim 9, characterized in that: The acceptance criteria include a systematic deviation acceptance criterion, which filters the input data from multiple dimensions to obtain several population subsets, obtains the model output of the data subsets, and determines whether the output meets the consistency of the data set.