Operation Monitoring Method and Device for Risk Control Model, Electronic Device, and Storage Medium

Abstract

The present invention discloses a method and device for running and monitoring a risk control model, an electronic device, and a storage medium, relating to the field of artificial intelligence. Among them, the monitoring method includes: obtaining model information and model monitoring information of a target risk control model, and based on the model information and model monitoring information, calling an automated configuration node. The automated configuration node determines a model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model, and uses a preset monitoring thread to analyze multiple model prediction results of the target risk control model within a preset time period to determine the index parameters of the target risk control model corresponding to each monitoring index. The present invention solves the technical problem in the related art that the risk control model cannot be automatically monitored, resulting in a waste of human resources.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular, to a method and device for monitoring the operation of a risk control model, an electronic device, and a storage medium. Background Art

[0002] Pre-loan risk control is an important link in the credit field. At present, before a risk control model is launched, offline model training needs to be carried out first. Among them, for each process in the offline model training, generally, a modeler needs to develop functions by himself. The modeler first performs feature selection and feature transformation, and then builds a model. After the offline part is completed, the modeler still needs to release and put the model into production, and still needs to develop functions such as model monitoring indicators.

[0003] In related technologies, the training and post-launch monitoring of risk control models are decentralized, and users need to assemble training steps by themselves for model training, resulting in users spending more time on model training steps and post-launch monitoring. Moreover, in the existing methods, manual monitoring of the risk control model is required, and different indicators are calculated by setting different parameters, resulting in a waste of human resources and low efficiency.

[0004] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of the present invention provide a method and device for monitoring the operation of a risk control model, an electronic device, and a storage medium, so as to at least solve the technical problem in related technologies that the risk control model cannot be automatically monitored, resulting in a waste of human resources.

[0006] According to one aspect of the embodiments of the present invention, a method for monitoring the operation of a risk control model is provided, including: obtaining model information and model monitoring information of a target risk control model; based on the model information and the model monitoring information, calling an automated configuration node, where the automated configuration node determines a model prediction result according to feature processing code during model training and a prediction strategy of the target risk control model, and the feature processing code refers to the processing code for each feature during training of the target risk control model; using a preset monitoring thread to analyze a plurality of the model prediction results of the target risk control model within a preset time period, and determining an index parameter corresponding to each monitoring index of the target risk control model, where the index parameter is used to indicate the operation state of the target risk control model.

[0007] Optionally, the steps of obtaining the model information and model monitoring information of the target risk control model include: receiving a model monitoring instruction transmitted by a client, where the model monitoring instruction carries at least: the model identifier of the specified target risk control model; reading the model information of the target risk control model based on the model identifier; receiving the model online parameters, online deployment method, and monitoring parameters transmitted by the client to obtain the model monitoring information.

[0008] Optionally, the steps of invoking the automated configuration node based on the model information and the model monitoring information include: obtaining model online information, where the model online information at least includes: the prediction strategy for the target risk control model to be put online; in the case where the prediction strategy is online model prediction, generating a first node invocation request based on the model information and the model monitoring information, where the first node invocation request is used to invoke the automated configuration node; in the case where the prediction strategy is offline batch prediction, generating a second node invocation request based on the model information, the model monitoring information, and the pre-configured prediction table identifier and result table identifier, where the second node invocation request is used to invoke the automated configuration node.

[0009] Optionally, after invoking the automated configuration node based on the model information and the model monitoring information, it further includes: in the case where the prediction strategy is online model prediction, the automated configuration node generates prediction code according to the feature processing code during model training and the prediction strategy of the target risk control model; using the prediction code to generate a model access request, where the model access request is used to invoke the remote access text protocol to access the online target risk control model to obtain the model prediction result.

[0010] Optionally, after invoking the automated configuration node based on the model information and the model monitoring information, it further includes: in the case where the prediction strategy is offline batch prediction, the automated configuration node generates a scheduling task according to the feature processing code during model training, the prediction strategy of the target risk control model, the target prediction table indicated by the prediction table identifier, and the target result table indicated by the result table identifier; using the scheduling task to execute the model prediction operation to obtain the model prediction result.

[0011] Optionally, before obtaining the model information and model monitoring information of the target risk control model, it further includes: obtaining sample sampling information, feature screening information, feature transformation information, model training algorithms, and model training evaluation metrics; configuring model sample parameters and model training parameters based on the sample sampling information, the feature screening information, the feature transformation information, the model training algorithms, and the model training evaluation metrics; training an initial model according to the model sample parameters and the model training parameters to obtain a target risk control model; and saving the trained target risk control model to a fixed access path.

[0012] Optionally, the sample sampling information includes one of the following: random sampling and stratified sampling; the feature screening information includes at least one of the following: missing rate, correlation; the feature transformation information includes one of the following: chi-square binning, equal-frequency binning, equal-distance binning, or custom transformation information, and the custom transformation information includes a code upload format and a transformation algorithm; the model training evaluation metrics include at least one of the following: the number of negative samples, the number of positive samples, the cumulative number of negative samples, the cumulative number of positive samples, capture rate, and negative sample ratio.

[0013] Optionally, after training the initial model according to the model sample parameters and the model training parameters to obtain a target risk control model, it further includes: generating a model metric evaluation file for the target risk control model according to the selected model training evaluation metric.

[0014] According to another aspect of the embodiments of the present invention, there is also provided an operation monitoring device for a risk control model, including: an acquisition unit for acquiring model information and model monitoring information of a target risk control model; a call unit for calling an automated configuration node based on the model information and the model monitoring information, where the automated configuration node determines a model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model, and the feature processing code refers to the processing code for each feature during the training of the target risk control model; and a determination unit for analyzing a plurality of the model prediction results of the target risk control model within a preset time period by using a preset monitoring thread to determine the metric parameters of the target risk control model corresponding to each monitoring metric, where the metric parameters are used to indicate the operation status of the target risk control model.

[0015] Optionally, the acquisition unit includes: a first receiving module for receiving a model monitoring instruction transmitted by a client, where the model monitoring instruction carries at least: the model identifier of the specified target risk control model; a first reading module for reading the model information of the target risk control model based on the model identifier; and a second receiving module for receiving the model online parameters, online deployment method, and monitoring parameters transmitted by the client to obtain the model monitoring information.

[0016] Optionally, the calling unit includes: a first obtaining module, configured to obtain model online information, where the model online information at least includes: a prediction strategy for the target risk control model to be put online; a first generating module, configured to generate a first node call request based on the model information and the model monitoring information when the prediction strategy is online model prediction, where the first node call request is used to call the automated configuration node; a second generating module, configured to generate a second node call request based on the model information, the model monitoring information, and pre-configured prediction table identifiers and result table identifiers when the prediction strategy is offline batch prediction, where the second node call request is used to call the automated configuration node.

[0017] Optionally, the monitoring device further includes: a third generating module, configured to, after calling the automated configuration node based on the model information and the model monitoring information, when the prediction strategy is online model prediction, the automated configuration node generates prediction code according to the feature processing code during model training and the prediction strategy of the target risk control model; a fourth generating module, configured to generate a model access request by using the prediction code, where the model access request is used to call the remote access text protocol to access the online target risk control model to obtain a model prediction result.

[0018] Optionally, the monitoring device further includes: a fifth generating module, configured to, after calling the automated configuration node based on the model information and the model monitoring information, when the prediction strategy is offline batch prediction, the automated configuration node generates a scheduling task according to the feature processing code during model training, the prediction strategy of the target risk control model, the target prediction table indicated by the prediction table identifier, and the target result table indicated by the result table identifier; a first execution module, configured to perform a model prediction operation by using the scheduling task to obtain a model prediction result.

[0019] Optionally, the monitoring device further includes: a second obtaining module, configured to obtain sample sampling information, feature screening information, feature conversion information, a model training algorithm, and a model training evaluation index before obtaining the model information and the model monitoring information of the target risk control model; a first configuration module, configured to configure model sample parameters and model training parameters based on the sample sampling information, the feature screening information, the feature conversion information, the model training algorithm, and the model training evaluation index; a first training module, configured to train an initial model according to the model sample parameters and the model training parameters to obtain the target risk control model; a first saving module, configured to save the trained target risk control model to a fixed access path.

[0020] Optionally, the sample sampling information includes one of the following: random sampling and stratified sampling; the feature screening information includes at least one of the following: missing rate, correlation; the feature transformation information includes one of the following: chi-square binning, equal-frequency binning, equal-width binning, or custom transformation information, and the custom transformation information includes a code upload format and a transformation algorithm; the model training evaluation metrics include at least one of the following: the number of negative samples, the number of positive samples, the cumulative number of negative samples, the cumulative number of positive samples, the capture rate, and the proportion of negative samples.

[0021] Optionally, the monitoring device further includes: a sixth generation module, configured to generate a model metric evaluation file of the target risk control model according to the selected model training evaluation metric after training the initial model according to the model sample parameters and the model training parameters to obtain the target risk control model.

[0022] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, where the computer-readable storage medium includes a stored computer program, and when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the above-mentioned risk control model operation monitoring method.

[0023] According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including one or more processors and a memory, where the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above-mentioned risk control model operation monitoring method.

[0024] In the present disclosure, the model information and model monitoring information of the target risk control model are obtained, and based on the model information and model monitoring information, an automated configuration node is called. The automated configuration node determines the model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model, and uses a preset monitoring thread to analyze multiple model prediction results of the target risk control model within a preset time period to determine the index parameters of the target risk control model corresponding to each monitoring index. In the present application, the model prediction result can be obtained based on the model information and model monitoring information, and then, through the preset monitoring thread for result analysis, the index parameters of the risk control model corresponding to each monitoring index can be obtained to determine the running state of the risk control model, which can not only efficiently complete the monitoring work, but also reduce the waste of human resources, thereby solving the technical problem in the related art that the risk control model cannot be automatically monitored, resulting in waste of human resources. Description of the Drawings

[0025] The accompanying drawings described herein are used to provide a further understanding of the present invention and form a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 is a flowchart of an optional method for monitoring the operation of a risk control model according to an embodiment of the present invention;

[0027] Figure 2 is a schematic diagram of an optional device for monitoring the operation of a risk control model according to an embodiment of the present invention;

[0028] Figure 3 is a hardware structure block diagram of an electronic device (or mobile device) for a method of monitoring the operation of a risk control model according to an embodiment of the present invention. Detailed implementation manners

[0029] In order to enable those skilled in the art of this technology to better understand the present invention solution, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above accompanying drawings are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should be noted that the method and device for monitoring the operation of the risk control model in the present disclosure can be used in the field of artificial intelligence when monitoring the operation of the risk control model, and can also be used in any field other than the field of artificial intelligence when monitoring the operation of the risk control model. The application fields of the method and device for monitoring the operation of the risk control model in the present disclosure are not limited.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data that have been authorized by the user or fully authorized by all parties.

[0033] The following embodiments of the present invention can be applied to various systems / applications / devices for running and monitoring risk control models. In the present invention, through parameterized configuration operations, users can quickly complete model training and post-launch model monitoring, greatly shortening the time for the model to go live. When the post-launch model monitoring indicators reach the set thresholds, the model responsible person will be notified to conduct cause analysis and model adjustment, which can improve the efficiency of model training and optimization. Users only need to have the ability of basic data analysis to complete the training and monitoring of risk control models. Moreover, the present invention can also encapsulate processes such as sample selection, feature screening, model generation, evaluation of the trained model, and post-launch model stability evaluation for model training, providing an integrated pipeline function, and can also support modelers to develop new algorithms, which can be shared and used by everyone, thereby improving the work efficiency of modelers, saving a lot of human resources, and being conducive to the rapid and efficient development of the business.

[0034] The present invention will be described in detail below in conjunction with each embodiment.

[0035] Embodiment 1

[0036] According to an embodiment of the present invention, there is provided an embodiment of a method for running and monitoring a risk control model. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0037] Figure 1 is a flowchart of an optional method for running and monitoring a risk control model according to an embodiment of the present invention. As Figure 1 shown, the method includes the following steps:

[0038] Step S101, obtain the model information and model monitoring information of the target risk control model.

[0039] Step S102, based on the model information and model monitoring information, call the automated configuration node. Among them, the automated configuration node determines the model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model. The feature processing code refers to the processing code for each feature during the training of the target risk control model.

[0040] Step S103: Analyze multiple model prediction results of the target risk control model within a preset time period using a preset monitoring thread, and determine the index parameters corresponding to each monitoring index of the target risk control model, where the index parameters are used to indicate the operating status of the target risk control model.

[0041] Through the above steps, the model information and model monitoring information of the target risk control model can be obtained. Based on the model information and model monitoring information, an automated configuration node is called. The automated configuration node determines the model prediction results according to the feature processing code during model training and the prediction strategy of the target risk control model, analyzes multiple model prediction results of the target risk control model within a preset time period using a preset monitoring thread, and determines the index parameters corresponding to each monitoring index of the target risk control model. In the embodiments of the present invention, the model prediction results can be obtained based on the model information and model monitoring information. Then, through the preset monitoring thread for result analysis, the index parameters corresponding to each monitoring index of the risk control model are obtained to determine the operating status of the risk control model, which not only can efficiently complete the monitoring work but also reduces the waste of human resources, thereby solving the technical problem in the related art that the risk control model cannot be automatically monitored, resulting in waste of human resources.

[0042] The embodiments of the present invention will be described in detail below in combination with the above steps.

[0043] In the embodiments of the present invention, before obtaining the model information and model monitoring information of the target risk control model, it further includes: obtaining sample sampling information, feature screening information, feature conversion information, model training algorithms, and model training evaluation indicators; configuring model sample parameters and model training parameters based on the sample sampling information, feature screening information, feature conversion information, model training algorithms, and model training evaluation indicators; training an initial model according to the model sample parameters and model training parameters to obtain the target risk control model; and saving the trained target risk control model to a fixed access path.

[0044] In the embodiments of the present invention, sample selection (including: random sampling and stratified sampling) can be performed first to obtain sampling information, and then feature screening (including: psi (population stability index), missing rate, IV value (information value), correlation and other features) can be performed to obtain feature screening information. After that, feature transformation can be performed (methods such as chi-square binning, equal-frequency binning, equal-distance binning, and custom transformation code can be used. Among them, for custom transformation code, the user needs to upload the code in a specified format and name the transformation algorithm, and the background will save the relevant code) to obtain feature transformation information. Then, a model training algorithm can be obtained for model training (for example, algorithms such as xgboost (extreme gradient boosting algorithm), LR (logistic regression algorithm), random forest, and custom model code can be used for training. Among them, for custom model code, the user needs to upload the code in a specified format and name the model algorithm, and the background will save the relevant code). After that, model training evaluation metrics can be obtained (including: ks (used to measure the difference between the cumulative distributions of positive and negative samples), the number of negative samples, the number of positive samples, the cumulative number of negative samples, the cumulative number of positive samples, capture rate, negative sample ratio, custom evaluation metrics, etc. Among them, for custom evaluation metrics, the user needs to upload the code and name the metric, and the background will save the relevant code), so as to configure model sample parameters and model training parameters. Among them, Table 1 is an exemplary description of model sample parameter configuration, and Table 2 is an exemplary description of model training parameter configuration.

[0045] Table 1 Model Sample Parameter Configuration

[0046]

[0047]

[0048] Table 2 Model Training Parameter Configuration

[0049]

[0050]

[0051]

[0052] After configuring the model sample parameters and model training parameters, the initial model can be trained according to the model sample parameters and model training parameters to obtain a target risk control model, and the trained target risk control model can be saved to a fixed access path.

[0053] Optionally, the sample sampling information includes one of the following: random sampling and stratified sampling; the feature screening information includes at least one of the following: missing rate, correlation; the feature transformation information includes one of the following: chi-square binning, equal-frequency binning, equal-width binning, or custom transformation information, and the custom transformation information includes the code upload format and transformation algorithm; the model training evaluation metrics include at least one of the following: the number of negative samples, the number of positive samples, the cumulative number of negative samples, the cumulative number of positive samples, the capture rate, and the proportion of negative samples.

[0054] Optionally, after training the initial model according to the model sample parameters and the model training parameters to obtain the target risk control model, it further includes: generating a model metric evaluation file of the target risk control model according to the selected model training evaluation metrics.

[0055] In the embodiment of the present invention, relevant parameter information (i.e., model sample parameters and model training parameters) can be read from the database according to the model_code (model code), and training can be performed according to the configured parameters (i.e., model sample parameters and model training parameters). After training is completed, a model metric evaluation file of the target risk control model can be generated according to the selected model training evaluation metrics. If there are multiple parameters and multiple models, multiple model evaluation metric files will be generated and saved to the database, and the trained model will be saved to a fixed path. The training task can be executed regularly, and only the latest version of the model and the metric evaluation file will be retained.

[0056] Step S101, obtain the model information and model monitoring information of the target risk control model.

[0057] Optionally, the step of obtaining the model information and model monitoring information of the target risk control model includes: receiving a model monitoring instruction transmitted by the client, where at least the model identifier of the specified target risk control model is carried in the model monitoring instruction; reading the model information of the target risk control model based on the model identifier; receiving the model online parameters, online deployment method, and monitoring parameters transmitted by the client to obtain the model monitoring information.

[0058] In the embodiment of the present invention, the model information of the target risk control model can be read according to the model identifier of the target risk control model carried in the model monitoring instruction transmitted by the client (the target risk control model is selected from the models that have been trained online in the fixed access path where the model is saved through the model identifier). After that, receive the model online parameters (including: online time, etc.), online deployment method, and monitoring parameters transmitted by the client to obtain the model monitoring information.

[0059] Step S102: Based on the model information and model monitoring information, call the automated configuration node. The automated configuration node determines the model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model. The feature processing code refers to the processing code for each feature during the training of the target risk control model.

[0060] In an embodiment of the present invention, based on the model information and model monitoring information, the automated configuration node can be called. The automated configuration node can determine the model prediction result according to the feature processing code during model training (the feature processing code refers to the processing code for each feature during the training of the target risk control model) and the prediction strategy of the target risk control model (including: online model prediction, offline batch prediction, etc.).

[0061] Optionally, the step of calling the automated configuration node based on the model information and model monitoring information includes: obtaining the model online information, where the model online information at least includes: the prediction strategy for the target risk control model to be launched; in the case where the prediction strategy is online model prediction, generating a first node call request based on the model information and model monitoring information, where the first node call request is used to call the automated configuration node; in the case where the prediction strategy is offline batch prediction, generating a second node call request based on the model information, model monitoring information, and pre-configured prediction table identifier and result table identifier, where the second node call request is used to call the automated configuration node.

[0062] In an embodiment of the present invention, different call requests can be generated according to different types of prediction strategies of the target risk control model to be launched in the model online information. Specifically, in the case where the prediction strategy is online model prediction, a first node call request can be generated based on the model information and model monitoring information (the first node call request can call the http service to access the online prediction model by calling the automated configuration node); in the case where the prediction strategy is offline batch prediction, a second node call request is generated based on the model information, model monitoring information, and pre-configured prediction table identifier and result table identifier (in this embodiment, the prediction table identifier and result table identifier for each day can be specified during offline batch prediction), where the second node call request can generate a scheduling task by calling the automated configuration node.

[0063] Optionally, after calling the automated configuration node based on the model information and model monitoring information, it further includes: in the case where the prediction strategy is online model prediction, the automated configuration node generates a prediction code according to the feature processing code during model training and the prediction strategy of the target risk control model; using the prediction code to generate a model access request, where the model access request is used to call the remote access text protocol to access the online target risk control model and obtain the model prediction result.

[0064] In an embodiment of the present invention, if the prediction strategy is online model prediction, the automated configuration node automatically generates prediction code according to the feature processing code during model training and the prediction strategy of the target risk control model, implements the model prediction action, and deploys the prediction code as an HTTP service (i.e., generates a model access request). The user can access the online prediction model by calling the HTTP service (i.e., can call the remote access text protocol through the model access request to access the online target risk control model) to obtain the model prediction result.

[0065] Optionally, after calling the automated configuration node based on the model information and model monitoring information, it further includes: in the case where the prediction strategy is offline batch prediction, the automated configuration node generates a scheduling task according to the feature processing code during model training, the prediction strategy of the target risk control model, the target prediction table indicated by the prediction table identifier, and the target result table indicated by the result table identifier; and performs the model prediction operation using the scheduling task to obtain the model prediction result.

[0066] In an embodiment of the present invention, if the prediction strategy is offline batch prediction, the automated configuration node automatically generates a batch scheduling task (this scheduling task is used to perform the prediction operation to obtain the model prediction result) at a specified time according to the feature processing code during model training, the prediction strategy of the target risk control model, the target prediction table indicated by the prediction table identifier, and the target result table indicated by the result table identifier, and stores the model prediction result in the database, encapsulates it as an HTTP service for the user to access the prediction result data.

[0067] Step S103, analyze multiple model prediction results of the target risk control model within a preset time period using a preset monitoring thread, and determine the index parameters corresponding to each monitoring index of the target risk control model, where the index parameters are used to indicate the running state of the target risk control model.

[0068] In an embodiment of the present invention, analyze multiple model prediction results of the target risk control model within a preset time period using a preset monitoring thread, and determine the index parameters corresponding to each monitoring index of the target risk control model (the index parameters are used to indicate the running state of the target risk control model). For example, start the monitoring thread. When the time interval between the current time and the online time is the set time interval (e.g., when the time interval between the current time and the online time is 1 month or an integer multiple of one month), read the current sample and the online sample from the user-defined sample table, calculate the PSI value (population stability index) of the current model on the two samples, and save it in a file. If the PSI value is greater than the preset threshold (e.g., 0.2), send a model instability warning message to the model responsible person.

[0069] The following will be described in detail in conjunction with another optional specific implementation manner.

[0070] The embodiments of the present invention can be divided into two links: training and online monitoring, and three modes can be provided to users: (1) only perform offline training of the model; (2) specify the trained model, go online at the specified online time and automatically perform monitoring; (3) go online immediately after training to monitor indicators. In this embodiment, the configuration generation can be started through a post request, and for the configuration of each link, the user can complete it through the front-end interface, as follows:

[0071] (1) Only perform offline training of the model

[0072] This embodiment can encapsulate sample selection, feature screening, feature transformation, model training algorithms, and model evaluation indicators. Users can select different algorithms for different links through parameter configuration. Among them,

[0073] The functions supported by sample selection include: random sampling, stratified sampling, etc.;

[0074] Feature screening can be performed using missing rate, IV value, correlation, etc.;

[0075] Feature transformation can be performed in ways such as chi-square binning, equal-frequency binning, equal-distance binning, custom transformation code, etc. Among them, for custom transformation code, the user needs to upload the code in the specified format and name the transformation algorithm, and the background will save the relevant code;

[0076] Model training can be performed through algorithms such as xgboost, LR, random forest, custom model code, etc. Among them, for custom model code, the user needs to upload the code in the specified format and name the model algorithm, and the background will save the relevant code.

[0077] The model training evaluation indicators can include: ks, the number of negative samples, the number of positive samples, the cumulative number of negative samples, the cumulative number of positive samples, capture rate, negative sample ratio, custom evaluation indicators, etc. Among them, for custom evaluation indicators, the user needs to upload the code and name the indicators, and the background will save the relevant code.

[0078] After that, the model training service can be called. In this embodiment, after the user sets the relevant parameters, the relevant configuration information will be saved in the database. Click training to send an http request to call the automation device service node, and the http request can be a post request.

[0079] In this embodiment, the automated device service interface receives a request, reads relevant parameter information from the database according to the model_code, performs training according to the configured parameters. After the training is completed, a model evaluation index file is generated. If there are multiple parameters and multiple models, there will be multiple evaluation index files, which are saved to the database. The trained model is saved to a fixed path. The training task is executed regularly (e.g., every day), and only the latest version of the model and the index evaluation file can be retained.

[0080] In this embodiment, the user can download the evaluation index file through the front-end interface. When the user clicks to download, the specified evaluation file is obtained from the database.

[0081] (2) Specify the trained model, go online at the specified online time and automatically monitor it

[0082] The application scenario of this mode is to specify a model and the online time in the completed models for online monitoring operations.

[0083] In this embodiment, the method of going online with the model can be online model prediction, batch model prediction, etc., and the user needs to specify the corresponding parameters; the model monitoring index can select the psi index, and the user needs to specify the monitoring time interval, which can be an integer multiple of a month. 1 represents one month, 2 represents two months, and so on.

[0084] In this embodiment, the front-end parameter selection can be performed first: select the name of the model to be put online. The model name list can read the models that have been trained online from the database, select the online time, select the online deployment method, select the monitoring time interval, etc. Then, select the method of going online with the model. If it is offline batch prediction, the name of the prediction table for each period and the definition name of the result table need to be specified. After that, click to confirm, generate an http request, and call the automated configuration node service. The automated configuration node receives the request and performs relevant operations according to the request parameters. Specifically:

[0085] Perform the online operation at the specified time. If it is online prediction, the automated configuration node automatically generates prediction code according to the feature processing code and model during training, and deploys the prediction code as an http service. The user can access the online prediction model by calling the http service.

[0086] If it is offline batch prediction, according to the pre-specified prediction table name and result table name, the automated configuration node automatically generates a batch scheduling task (this scheduling task is used to perform the prediction operation to obtain the prediction result) at the specified time according to the feature processing code and model during training, and stores the prediction result in the database, encapsulating it as an http service for the user to access the prediction result data.

[0087] After obtaining the prediction results, the monitoring thread can be started. When the current time is at the set time interval from the online time, for example, if the parameter is 1 (indicating that the time interval between the current time and the online time is 1 month), the current sample and the online sample are read from the user-defined sample table, the psi value of the current model on the two samples is calculated and saved in a file. If the psi value is greater than the preset threshold (for example, 0.2), a warning message indicating model instability is sent to the model responsible person.

[0088] (3) Immediately monitor the metrics after training and go online

[0089] At the same time, all the parameters in mode (1) and mode (2) need to be configured, and after operation (1) (i.e., training the model), operation (2) (i.e., online monitoring) is immediately executed.

[0090] In the embodiments of the present invention, users only need to configure relevant parameters to quickly complete model training and monitoring of the model after going online, greatly shortening the time for the model to go online. When the monitoring metrics of the model after going online reach the set threshold, the model responsible person will be notified to conduct cause analysis and model adjustment, which can improve the efficiency of model training and optimization. Users only need to have the ability of basic data analysis to complete the training and monitoring of the risk control model. Moreover, the present invention can also encapsulate processes such as sample selection, feature screening, model generation, evaluation of the trained model, and evaluation of model stability after going online for the model training, providing an integrated pipeline function. It can also support modelers to develop new algorithms, and the new algorithms can be shared by everyone, thereby improving the work efficiency of modelers, saving a lot of human resources, and being conducive to the rapid and efficient development of the business.

[0091] Embodiment 2

[0092] A risk control model operation monitoring device provided in this embodiment includes multiple implementation units, and each implementation unit corresponds to each implementation step in Embodiment 1 above.

[0093] Figure 2 is a schematic diagram of an optional risk control model operation monitoring device according to an embodiment of the present invention, as Figure 2 shown, the monitoring device may include: an acquisition unit 20, a call unit 21, a determination unit 22, where

[0094] The acquisition unit 20 is used to acquire the model information and model monitoring information of the target risk control model;

[0095] The call unit 21 is used to call an automated configuration node based on the model information and model monitoring information, where the automated configuration node determines the model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model, and the feature processing code refers to the processing code for each feature during the training of the target risk control model;

[0096] The determination unit 22 is used to use a preset monitoring thread to analyze multiple model prediction results of the target risk control model within a preset time period, and determine the indicator parameters of the target risk control model corresponding to each monitoring indicator, wherein the indicator parameters are used to indicate the operating status of the target risk control model.

[0097] The above-mentioned monitoring device can obtain the model information and model monitoring information of the target risk control model through the acquisition unit 20, and call the automatic configuration node based on the model information and model monitoring information through the calling unit 21, wherein the automatic configuration node determines the model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model, and determines the indicator parameters of the target risk control model corresponding to each monitoring indicator by analyzing multiple model prediction results of the target risk control model within a preset time period through the determination unit 22 using a preset monitoring thread. In an embodiment of the present invention, the model prediction result can be obtained based on the model information and the model monitoring information, and then the result analysis is performed through the preset monitoring thread to obtain the indicator parameters of the risk control model corresponding to each monitoring indicator to determine the operating status of the risk control model, which can not only efficiently complete the monitoring work, but also reduce the waste of human resources, thereby solving the technical problem that the risk control model cannot be automatically monitored in the related technology, resulting in waste of human resources.

[0098] Optionally, the acquisition unit includes: a first receiving module, used to receive model monitoring instructions transmitted by the client, wherein the model monitoring instructions carry at least: a model identifier of a specified target risk control model; a first reading module, used to read model information of the target risk control model based on the model identifier; a second receiving module, used to receive model online parameters, online deployment methods and monitoring parameters transmitted by the client to obtain model monitoring information.

[0099] Optionally, the calling unit includes: a first acquisition module, used to acquire model online information, wherein the model online information includes at least: a prediction strategy for the target risk control model to be launched; a first generation module, used to generate a first node call request based on model information and model monitoring information when the prediction strategy is online model prediction, wherein the first node call request is used to call the automation configuration node; a second generation module, used to generate a second node call request based on model information, model monitoring information and pre-configured prediction table identifier and result table identifier when the prediction strategy is offline batch prediction, wherein the second node call request is used to call the automation configuration node.

[0100] Optionally, the monitoring device further includes: a third generation module, configured to, after invoking the automated configuration node based on the model information and the model monitoring information, and when the prediction strategy is online model prediction, the automated configuration node generates prediction code according to the feature processing code during model training and the prediction strategy of the target risk control model; a fourth generation module, configured to generate a model access request by using the prediction code, where the model access request is used to invoke the remote access text protocol to access the online target risk control model and obtain a model prediction result.

[0101] Optionally, the monitoring device further includes: a fifth generation module, configured to, after invoking the automated configuration node based on the model information and the model monitoring information, and when the prediction strategy is offline batch prediction, the automated configuration node generates a scheduling task according to the feature processing code during model training, the prediction strategy of the target risk control model, the target prediction table indicated by the prediction table identifier, and the target result table indicated by the result table identifier; a first execution module, configured to perform a model prediction operation by using the scheduling task to obtain a model prediction result.

[0102] Optionally, the monitoring device further includes: a second acquisition module, configured to acquire sample sampling information, feature screening information, feature transformation information, a model training algorithm, and a model training evaluation metric before acquiring the model information and the model monitoring information of the target risk control model; a first configuration module, configured to configure model sample parameters and model training parameters based on the sample sampling information, the feature screening information, the feature transformation information, the model training algorithm, and the model training evaluation metric; a first training module, configured to train an initial model according to the model sample parameters and the model training parameters to obtain the target risk control model; a first saving module, configured to save the trained target risk control model to a fixed access path.

[0103] Optionally, the sample sampling information includes one of the following: random sampling and stratified sampling; the feature screening information includes at least one of the following: missing rate, correlation; the feature transformation information includes one of the following: chi-square binning, equal-frequency binning, equal-distance binning, or custom transformation information, and the custom transformation information includes a code upload format and a transformation algorithm; the model training evaluation metric includes at least one of the following: the number of negative samples, the number of positive samples, the cumulative number of negative samples, the cumulative number of positive samples, capture rate, negative sample ratio.

[0104] Optionally, the monitoring device further includes: a sixth generation module, configured to generate a model metric evaluation file of the target risk control model according to the selected model training evaluation metric after training the initial model according to the model sample parameters and the model training parameters to obtain the target risk control model.

[0105] The above monitoring device may further include a processor and a memory. The above obtaining unit 20, calling unit 21, determining unit 22, etc. are all stored in the memory as program units, and the processor executes the above program units stored in the memory to implement corresponding functions.

[0106] The above processor includes a kernel, and the kernel retrieves the corresponding program units from the memory. One or more kernels can be set, and the index parameters of the target risk control model corresponding to each monitoring index are determined by adjusting the kernel parameters.

[0107] The above memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash RAM. The memory includes at least one memory chip.

[0108] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program initialized with the following method steps: obtaining model information and model monitoring information of a target risk control model, and based on the model information and model monitoring information, calling an automated configuration node, wherein the automated configuration node determines a model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model, and uses a preset monitoring thread to analyze multiple model prediction results of the target risk control model within a preset time period to determine the index parameters of the target risk control model corresponding to each monitoring index.

[0109] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, which includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the above method for monitoring the operation of the risk control model.

[0110] According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including one or more processors and a memory, where the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above method for monitoring the operation of the risk control model.

[0111] Figure 3 is a hardware structure block diagram of an electronic device (or mobile device) for the method of monitoring the operation of a risk control model according to an embodiment of the present invention. As Figure 3As shown, the electronic device may include one or more processors 102 (illustrated as 102a, 102b, ……, 102n in the figure), where the processor 102 may include, but is not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA, and a memory 104 for storing data. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 3 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the electronic device may further include more or fewer components than those Figure 3 shown in, or have a different configuration from that Figure 3 shown.

[0112] The serial numbers of the above embodiments of the present invention are only for description and do not represent the superiority or inferiority of the embodiments.

[0113] In the above embodiments of the present invention, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0114] In several embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other can be through some interfaces, and the indirect couplings or communication connections of the units or modules can be in electrical or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0117] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs.

[0118] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for running and monitoring a risk control model, characterized in that, Including: Obtain the model information and model monitoring information of the target risk control model; Based on the model information and the model monitoring information, call the automated configuration node, wherein the automated configuration node determines the model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model, and the feature processing code refers to the processing code for each feature during the training of the target risk control model; When the prediction strategy is online model prediction, the automated configuration node generates a prediction code according to the feature processing code during model training and the prediction strategy of the target risk control model; use the prediction code to generate a model access request, wherein the model access request is used to call the remote access text protocol to access the online target risk control model to obtain the model prediction result; When the prediction strategy is offline batch prediction, the automated configuration node generates a scheduling task according to the feature processing code during model training, the prediction strategy of the target risk control model, the target prediction table indicated by the prediction table identifier, and the target result table indicated by the result table identifier; use the scheduling task to execute the model prediction operation to obtain the model prediction result; Use a preset monitoring thread to analyze multiple model prediction results of the target risk control model within a preset time period, and determine the index parameters corresponding to each monitoring index of the target risk control model, wherein the index parameters are used to indicate the running state of the target risk control model.

2. The operation monitoring method according to claim 1, characterized in that The steps of obtaining the model information and model monitoring information of the target risk control model include: Receive a model monitoring instruction transmitted by the client, wherein at least the model identifier of the specified target risk control model is carried in the model monitoring instruction; Based on the model identifier, read the model information of the target risk control model; Receive the model online parameters, online deployment method, and monitoring parameters transmitted by the client to obtain the model monitoring information.

3. The operation monitoring method according to claim 1, characterized in that The steps of calling the automated configuration node based on the model information and the model monitoring information include: Obtain the model online information, wherein at least the prediction strategy for the target risk control model to be online is included in the model online information; When the prediction strategy is online model prediction, generate a first node call request based on the model information and the model monitoring information, wherein the first node call request is used to call the automated configuration node; When the prediction strategy is offline batch prediction, generate a second node call request based on the model information, the model monitoring information, and the pre-configured prediction table identifier and result table identifier, wherein the second node call request is used to call the automated configuration node.

4. The operation monitoring method according to claim 1, characterized in that, Before obtaining the model information and model monitoring information of the target risk control model, it further includes: Obtain sample sampling information, feature screening information, feature conversion information, model training algorithm, and model training evaluation index; Based on the sample sampling information, the feature screening information, the feature conversion information, the model training algorithm, and the model training evaluation index, configure the model sample parameters and model training parameters; Train the initial model according to the model sample parameters and the model training parameters to obtain a target risk control model; Save the trained target risk control model to a fixed access path.

5. The operation monitoring method according to claim 4, characterized in that The sample sampling information includes one of the following: random sampling and stratified sampling; the feature screening information includes at least one of the following: missing rate, correlation; the feature transformation information includes one of the following: chi-square binning, equal-frequency binning, equal-width binning or custom transformation information, and the custom transformation information includes a code upload format and a transformation algorithm; the model training evaluation metrics include at least one of the following: the number of negative samples, the number of positive samples, the cumulative number of negative samples, the cumulative number of positive samples, the capture rate, and the negative sample ratio.

6. The operation monitoring method according to claim 4, wherein After training the initial model according to the model sample parameters and the model training parameters to obtain a target risk control model, it further includes: Generate a model metric evaluation file for the target risk control model according to the selected model training evaluation metric.

7. An operation monitoring device for a risk control model, characterized in that, It includes: An acquisition unit for acquiring model information and model monitoring information of the target risk control model; A calling unit for calling an automated configuration node based on the model information and the model monitoring information, where the automated configuration node determines a model prediction result according to the feature processing code during model training and the prediction strategy of the target risk control model, and the feature processing code refers to the processing code for each feature during the training of the target risk control model; The monitoring device further includes: a third generation module for generating a prediction code after calling the automated configuration node based on the model information and the model monitoring information, when the prediction strategy is online model prediction, the automated configuration node generates the prediction code according to the feature processing code during model training and the prediction strategy of the target risk control model; a fourth generation module for generating a model access request by using the prediction code, where the model access request is used to call a remote access text protocol to access the online target risk control model to obtain a model prediction result; A fifth generation module for generating a scheduling task after calling the automated configuration node based on the model information and the model monitoring information, when the prediction strategy is offline batch prediction, the automated configuration node generates the scheduling task according to the feature processing code during model training, the prediction strategy of the target risk control model, the target prediction table indicated by the prediction table identifier, and the target result table indicated by the result table identifier; a first execution module for performing a model prediction operation by using the scheduling task to obtain a model prediction result; A determination unit for analyzing multiple model prediction results of the target risk control model within a preset time period by using a preset monitoring thread to determine the metric parameters of the target risk control model corresponding to each monitoring metric, where the metric parameters are used to indicate the operating state of the target risk control model.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, where when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the method for monitoring the operation of the risk control model according to any one of claims 1 to 6.

9. An electronic device, characterized in that, Comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the operation monitoring method of the risk control model described in any one of claims 1 to 6.

Images