A model development system

Abstract

The present invention discloses a model development system, which specifically includes: multiple Dockers, which are used to execute multiple task steps in the model development process; a scheduling module, which is used to configure the startup sequence and startup parameters of each Docker, and start each Docker in sequence according to the startup sequence of each Docker, and execute the task steps corresponding to the Docker in the model development process until all task steps are completed to obtain a developed model; a training log generation module, which is used to record the relevant computing task information of each Docker when executing the corresponding task steps, and form a training log file; a model deployment module, which is used to deploy the developed model to an application system. In the solution provided by the embodiment of the present invention, the development process is unified, and the algorithm environment is managed in a unified standard; and monitoring and backtracking of the entire model development process, as well as multiple model deployment methods, are provided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model development system. Background Art

[0002] During the model development process, process management and effectiveness judgment of the model are required at multiple different stages, including algorithm preparation, modeling process, and modeling application, in order to minimize the development and management risks of models and algorithms.

[0003] Model development in different application fields has its own control requirements. However, current model development is mostly done by developers building their own algorithm environments and following their own development processes. There is a lack of a unified standard development process, which makes it difficult to conduct unified management.

[0004] In addition, due to security and compliance considerations, the network architecture and IT environment of model application scenarios are usually more complex. Different business systems are relatively independent and even isolated at the physical layer of the network. Therefore, the model may be deployed across systems or even across network environments, which leads to operational risks of migration when the model is deployed. Summary of the invention

[0005] In view of the above problems, the present invention provides a model development system that overcomes the above problems or at least partially solves the above problems. The technical solution is as follows:

[0006] A model development system, comprising:

[0007] Multiple application container engines Docker, used to execute multiple task steps in the model development process, each Docker executes the task step corresponding to it, and each task step is generated by independent computing task calculation;

[0008] A scheduling module is used to configure the startup sequence and startup parameters of each Docker, and start each Docker in sequence according to the startup sequence of each Docker, and execute the task steps corresponding to the Docker in the model development process until all the task steps are completed to obtain the developed model, and the startup sequence of each Docker corresponds to the serial order of each task step in the model development process;

[0009] A training log generation module is used to record the relevant computing task information of each Docker when executing the corresponding task steps to form a training log file;

[0010] The model deployment module is used to deploy the developed model to the application system.

[0011] In the above system, optionally, in the process of deploying the developed model to the application system, the model deployment module is further configured to:

[0012] If the network between the development environment in the model development process and the production environment of the application system is interconnected, the developed model is directly deployed to the application system;

[0013] If the network between the development environment in the model development process and the production environment of the application system is physically isolated, and the deployment system of the production environment and the development environment is the same system, the developed model is packaged as a Docker image file and deployed to the application system;

[0014] If there is a physical isolation between the development environment in the model development process and the production environment of the application system, and the deployment system of the production environment and the development environment is not the same system, a model file is generated according to the developed model, and the model file is provided to the application system to build a corresponding model in the application system.

[0015] In the above system, optionally, the model file includes the calculation logic and model parameters of the developed model.

[0016] In the above system, optionally, the multiple task steps in the model development process include a first step, and the Docker that executes the first step is specifically configured to: obtain raw data required for model development, and perform feature extraction on the raw data according to preset feature extraction rules to construct a sample data set required for the model development process;

[0017] In the process of executing the first step, the training log generating module is configured to record the acquisition time of the original data and the source information of the original data.

[0018] In the above system, optionally, the multiple task steps in the model development process further include a second step, and the Docker that executes the second step is specifically configured to: select a training algorithm required in the model development process, and apply the sample data set to train the training algorithm to obtain a trained first model;

[0019] During the execution of the second step, the training log generation module is configured to record the modeling script, software and hardware conditions, developer identity, verification information, start and end timestamps, and the algorithm name, algorithm type, parameters used by the algorithm, and round information of algorithm training iterations during the training process.

[0020] In the above system, optionally, the multiple task steps in the model development process further include a third step, and the Docker that executes the third step is specifically configured to: evaluate the first model to determine whether the first model meets the preset requirements; if the preset requirements are not met, re-perform the model training process and obtain a new first model, until the new first model meets the preset requirements, and determine the currently obtained new first model as the developed model;

[0021] In the process of executing the third step, the training log generation module is configured to record relevant evaluation information generated when evaluating the new first model obtained each time.

[0022] In the above system, optionally, the multiple task steps in the model development process further include a fourth step, and the Docker that executes the fourth step is specifically configured to: perform model interpretability analysis each time the model training is performed;

[0023] In the process of executing the fourth step, the training log generation module is configured to record the feature definition interpretability, feature abnormality interpretability and feature selection interpretability of each feature in the sample data set used for each model training, as well as the algorithm interpretability and algorithm parameter interpretability corresponding to the training algorithm selected for each model training.

[0024] The above-mentioned system may optionally further include a model monitoring module. After the developed model is deployed to the application system, the model monitoring module is configured to monitor the model effect of the developed model during actual use, and record relevant model effect monitoring information to form a model application monitoring report.

[0025] In the above system, optionally, the model monitoring module is further configured to: determine whether the developed model meets preset usage standards when actually used, and if it does not meet the preset usage standards, issue an alarm to redevelop the model.

[0026] The above system may optionally further include a display module, and the display module is used to display the training log file.

[0027] Compared with the prior art, the present invention has the following advantages: a model development system, specifically comprising: a plurality of application container engines Docker, used to execute a plurality of task steps in the model development process, each of the Dockers executes a task step corresponding to it, and each of the task steps is generated by independent computing task calculation; a scheduling module, used to configure the startup timing and startup parameters of each of the Dockers, and start each of the Dockers in sequence according to the startup timing of each of the Dockers, execute the task steps corresponding to the Docker in the model development process, until all the task steps are completed, and a developed model is obtained, and the startup timing of each of the Dockers corresponds one by one to the serial order of each task step in the model development process; a training log generation module, used to record the relevant computing task information of each of the Dockers when executing the corresponding task steps, to form a training log file; a model deployment module, used to deploy the developed model to an application system. In the solution provided by the embodiment of the present invention, the model development process is divided into multiple task steps, and the computing task of each task step is assigned to the corresponding Docker for independent execution. The model development is completed in combination with the serial scheduling design, the development process is unified, the algorithm environment is managed in a unified standard, and the operational risks in the model development process are reduced; the training log file formed in the model development process provides monitoring and backtracking of the entire model development process, and strengthens the supervision of the modeling process; a variety of deployment methods are provided for model deployment, and model deployment across systems and networks can also be achieved.

[0028] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0030] Figure 1 A structural block diagram of a model development system provided by an embodiment of the present invention;

[0031] Figure 2 Another structural block diagram of a model development system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0033] refer to Figure 1 FIG. 4 shows a structural block diagram of a model development system provided by an embodiment of the present invention. Figure 1 The connection structure shown is a feasible implementation scheme of a model development system provided by an embodiment of the present invention. The execution subject of the system is a server set in the system platform. The model development system specifically includes:

[0034] Multiple application container engines Docker, used to execute multiple task steps in the model development process, each Docker executes the task step corresponding to it, and each task step is generated by independent computing task calculation;

[0035] A scheduling module is used to configure the startup sequence and startup parameters of each Docker, and start each Docker in sequence according to the startup sequence of each Docker, and execute the task steps corresponding to the Docker in the model development process until all the task steps are completed to obtain the developed model, and the startup sequence of each Docker corresponds to the serial order of each task step in the model development process;

[0036] In the system provided by the embodiment of the present invention, the model development process mainly includes main steps such as modeling data preparation, model iterative development and model application. Each main step can be divided into one or more specific task steps. The division standard is that one task step is implemented by an independent computing task. Usually, the computing task of each task step needs to be completed using different algorithms or code tools. The system provided by the embodiment of the present invention starts an independent Docker to execute each task step in the modeling process at the bottom layer. Since each computing task is assigned a Docker environment to run independently, it realizes the minimized granularity isolation of computing resources and computing tasks, and provides basic support for iteration and monitoring requirements in the algorithm model development process.

[0037] In the system provided by an embodiment of the present invention, the scheduling module can adopt a K8S scheduler to configure the startup parameters of each Docker, including the input data of each Docker, the algorithm parameters used, etc.; the scheduling module is also used to start the Dockers in the corresponding order in sequence according to the serial order of each task step in the model development process to execute the corresponding task steps, thereby realizing the model development process.

[0038] The system provided by the embodiment of the present invention, in the use of underlying computing resources and the construction architecture of the development system environment, encapsulates each task step with the corresponding Docker, implements a separate computing task in a standard Docker virtualization manner to execute the corresponding task steps; and then combines the scheduling module to serialize the input data and output data of each computing task to form a model training workflow, thereby realizing that all computing tasks form a complete computing process in the form of a pipeline, and completing the complete training of the model development process. On the one hand, this can unify the management of the underlying resources of the entire system and allocate a consistent development virtual environment to developers, so that developers will use a unified image file when using the system platform. Through the management of image files, the consistency and security of the developed models and algorithms in the system environment and third-party open source algorithm code versions are guaranteed; on the other hand, by fine-grained segmentation of the model development process, developers only need to correct some calculation steps during the iteration of the model without modifying the complete modeling process code, and can reuse it between different model development tasks or different developers, thereby improving development efficiency; in addition, it can schedule and control computing resources, and at the same time, when resource demand expands, it can quickly expand or migrate horizontally, and provide the ability to quickly deploy, expand, and migrate the algorithm environment within the bank.

[0039] The system provided by the embodiment of the present invention also includes a training log generation module, which is used to record the relevant computing task information of each Docker when executing the corresponding task step to form a training log file. Since the model development process is modularly encapsulated, the computing task of each task step is executed by an independent Docker, so that the computing task log file can be formed more conveniently, which enables the system to monitor and trace the entire training process of the developer's model algorithm in detail, ensuring the traceability of each part of the model development process and strengthening the supervision of the modeling process.

[0040] In the system provided by the embodiment of the present invention, the training log file is used to provide backtracking and management of each step in the model development process, and contains a variety of reports on model management and evaluation and related documents on model development, including but not limited to model data reports, model evaluation reports, model operation reports, model development process logs, model online monitoring reports, explainability reports, etc.; these reports and logs are recorded and generated when the corresponding task steps in the model development process are executed, so that managers can easily backtrack the model development process, monitor and manage the online and application status of the model, improve the management process of the entire model development and application, and minimize the risks caused by algorithm model development and deployment.

[0041] In the system provided by the embodiment of the present invention, the model development process mainly includes the main steps of modeling data preparation, model iterative development and model application, wherein modeling data preparation is recorded as the first step, model training in model iterative development is recorded as the second step, model evaluation in model iterative development is recorded as the third step, and interpretability analysis in model iterative development is recorded as the fourth step. Each of the first to fourth steps can be implemented by one or more task steps, that is, each of the first to fourth steps can be executed by one or more Dockers to complete the corresponding one or more task steps. The number of Dockers configured for each step is determined by the number of task steps required to implement the step.

[0042] In the system provided by the embodiment of the present invention, the first step is modeling data preparation, including: obtaining the original data required for model development, and extracting features from the original data according to preset feature extraction rules to construct a sample data set required for the model development process, as a unified data source for subsequent algorithm training and iteration. The modeling data preparation stage includes data acquisition and data processing. Data acquisition uses a data interface to import raw data. The system provided by the embodiment of the present invention proposes two different ways to import raw data: one is to use middleware to connect multiple types of databases / data warehouses to extract raw data in a scripting manner; the other is to directly import raw data through local files or server files. After obtaining the raw data, data processing is performed, including feature screening, sample screening, feature engineering and other processing steps.

[0043] In the system provided by an embodiment of the present invention, when the corresponding Docker executes each task step in the first step, the training log generation module is configured to form a model data report, which includes the recorded acquisition time of the original data and the source information of the original data, and the source information includes the data storage medium and the data acquisition script; in the system provided by an embodiment of the present invention, the model data report also includes the feature extraction related information recorded by the training log generation module, including the data sampling method, the process and method of feature engineering; the sample data set can also use a signature or verification code to uniquely identify the sample data set.

[0044] In the system provided by the embodiment of the present invention, the second step is model training, including: selecting the training algorithm required in the model development process, often also needing to configure the parameters of the training algorithm, and then applying the sample data set to train the training algorithm to achieve the optimal model parameters, and obtain the first model that has been trained. The system provided by the embodiment of the present invention supports parallel training of multiple algorithms so that developers can compare the results of multiple algorithm performances. The system also supports the use of the champion challenger method to perform grid search on algorithm type, iteration rounds, and algorithm hyperparameters to select the optimal algorithm model performance.

[0045] In the system provided by the embodiment of the present invention, when the corresponding Docker executes each task step in the second step, the training log generation module is configured to generate a model development process log, which includes the modeling script, software and hardware conditions, developer identity, verification information, start and end timestamps, and the algorithm name, algorithm type, parameters used by the algorithm, and round information of algorithm training iterations recorded in the training process. In the system provided by the embodiment of the present invention, when a new branch of the model development process is formed, complete verification information such as (signature or verification code) is required to uniquely identify the model development process, and the model development process log is recorded if the verification is qualified.

[0046] In the system provided by the embodiment of the present invention, the third step is model evaluation. Since the first model obtained by one training often does not meet the preset requirements, the first model obtained by the training must be evaluated after each training to determine whether the first model meets the preset requirements; if the preset requirements are not met, the model training process is repeated and a new first model is obtained, until the new first model obtained meets the preset requirements, the new first model currently obtained is determined as the developed model. After multiple rounds of iterative optimization, the developed model is finally formed. Model evaluation can use methods including ROC curve, confusion matrix, V-fold cross validation, etc. to evaluate the performance of the first model, and use verification methods including cross validation, shuffle sampling, random sampling, etc. to detect the stability of the first model, so as to evaluate whether the effect of the model meets the preset requirements. If the first model obtained by training cannot meet the accuracy requirements, or the developer needs to modify and explore the algorithm or process of the model, it can return to the first step to select a new feature subset (i.e., the new features are screened in the modeling data preparation stage to form a sample data set) and / or return to the second step to select a new algorithm to iterate again to form a new branch of the model development process.

[0047] In the system provided by an embodiment of the present invention, when the corresponding Docker executes each task step in the third step, the training log generation module is configured to generate a model evaluation report, and the model evaluation report includes relevant evaluation information recorded when the new first model is evaluated each time.

[0048] In the system provided by the embodiment of the present invention, the fourth step is the interpretability analysis in the iterative development of the model, including: performing the model interpretability analysis each time the model is trained. In the process of executing the fourth step, the training log generation module is configured to generate an interpretability report, which includes the feature definition interpretability, feature abnormality interpretability and feature selection interpretability of each feature in the sample data set used for each model training, and the algorithm interpretability and algorithm parameter interpretability corresponding to the training algorithm selected for each model training. Specifically, the interpretability of the feature definition includes the relevant business logic and rules of the feature definition, the data location of the feature, and the detailed ETL (Extract-Transform-Load, that is, the process of extracting, transforming, and loading the data from the source end to the destination end) process. The interpretability of feature abnormality: the reasonable distribution of the feature should be ensured, and the missing values ​​and abnormal values ​​should be tested, and the rationality of the distribution should be explained; and the statistical distribution of the feature should be visualized. The interpretability of feature derivation: the rationality of the feature derivation method should be guaranteed, and the feature derivation process should be clearly recorded. Explanability of feature selection: The feature selection process should have certain quantitative statistical indicators as the basis for decision-making. It is necessary to record the IV value in the feature selection process, the feature importance method in the integrated algorithm GBDT and XGBOOSTING, the variable correlation, the variable significance test and other variable screening methods, and the developers are required to record the business logic and algorithm basis of feature selection. Explanability of the algorithm in the modeling process: The interpretability of the algorithm is explained through statistical indicators, the performance of sample features in the algorithm is visualized, and the attribution weights in the sample when the algorithm is used are explained in detail. Explanability of algorithm parameters: The scheme for setting algorithm parameters related to the business needs to be explained; if the automatic parameter adjustment method of grid search is used, the parameter adjustment process and code principles should be recorded. The validity of the model features used for each sample is visually reproduced through the interpretability report, which can assist developers in verifying and explaining the validity of the model application.

[0049] In the system provided by the embodiment of the present invention, the training log generation module is also configured to record the complete process in each computing node of the model development (such as algorithm performance data in different iteration rounds in the algorithm module, newly added feature sets in feature derivation, etc.), different performances of different versions of the model during model iteration, and records of manual adjustments made by developers to the model after the original model training (such as manual adjustments made by developers to variable binning in the scoring card model), etc., so that the system provided by the embodiment of the present invention supports version management and release of the developer's model achievements.

[0050] The system provided by the embodiment of the present invention also includes a model deployment module for deploying the developed model to the application system. When the iterative development process of the model is completed and the model effect of iterative training reaches the accuracy requirement, the model needs to be put into online application. The model deployment module can be deployed in a variety of deployment methods. In the system provided by the embodiment of the present invention, the model deployment module is also configured to deploy the developed model to the application system in the following three deployment methods to adapt to different network environments and system architectures.

[0051] The first deployment method: When the network between the development environment in the model development process and the production environment of the application system is interconnected, the developed model is directly deployed to the application system and provides services to the outside in the form of an interface. This method is to package the algorithm model in the form of an interface, and directly provide services to the outside in the form of an interface such as RESTFULL-API (RESTFUL-API is an interface design method, and its Chinese name is Representational State Transfer Interface Design). Other systems obtain the calculation results of the model by passing data into the interface service, and apply them to other application systems after returning, so as to achieve direct interaction with other business systems.

[0052] The second deployment method: When the development environment in the model development process is physically isolated from the production environment of the application system and cannot communicate, but the deployment system of the production environment and the development environment is the same system, the developed model is packaged as a Docker image file and deployed to the application system, and provides services to the outside in the form of interfaces such as RESTFULL-API. At this time, there is no need to deploy the model to other application systems, and it can be directly applied in the system of the production environment in the form of an interface, which is equivalent to the system of the production environment being the application system, and the application information after the model is deployed to other business systems or the model interface in the platform is recorded for management.

[0053] The third deployment method: When there is a physical isolation between the development environment in the model development process and the production environment of the application system, and the deployment system of the production environment and the development environment is not the same system, the algorithms and models developed by the system must be deployed through other business systems. A model file can be generated based on the developed model, and the model file can be provided to the application system to build a corresponding model in the application system.

[0054] In the system provided by the embodiment of the present invention, the model file includes the calculation logic and model parameters of the developed model, and the construction of the model corresponding to the model file can be realized by reproducing the calculation logic and model parameters of the model. For example, the models used in the financial field are mainly divided into two categories: linear regression model and model improvement method based on Boosting. For the linear regression model, it is essentially a type of discriminant formula composed of linear polynomials. The model file only needs to record the data conversion logic and the coefficients in the linear polynomial to reproduce the algorithm; for the model improvement method based on Boosting, its essence is a discriminant model composed of multiple decision trees. The model file records the logical judgment of the decision tree and the node parameters in the decision tree to reproduce the results of the model. The generation of existing model files can only generate files used in the model development language, but the system provided by the embodiment of the present invention is to go deep into the parameters and logic of the model itself, and automatically generate codes in related languages ​​for different model file types. For systems that do not support direct application of models, the model algorithm can be reproduced through calculation logic and parameters. This ensures that algorithms and models can be migrated and deployed stably and quickly in different network environments and system requirements, while also ensuring that algorithms and models will not be subject to operational risks such as leakage or tampering during the migration and deployment process.

[0055] In the system provided by the embodiment of the present invention, the model file supports most mainstream business system development environments, including stored procedures in database systems, JAR packages in JAVA environments, PMML files in PYTHON environments, DLL files in C++ environments, etc. Other business systems can use model files to directly load and use the trained model according to needs. The system provided by the embodiment of the present invention supports version management and release of the developer's model achievements, and can output the model results as a variety of model files, dock the model results with other business systems, and quickly go online and iterate, avoiding model leakage caused by model deployment and migration processes or operational risks in the model online process. The model file format provided by the system provided by the embodiment of the present invention covers most of the system requirements, including but not limited to: 1) Model files in the form of stored procedures, supporting Oracle, MySQL, Hive and other databases to directly use model files for model and algorithm batch running; 2) Storage files in PMML format, supporting PYTHON services, decision engines and other computing flow forms of business systems to use model files; 3) Output model files in the form of JAR packages, supporting business systems to directly reference system output model files, and directly calculate results in business processes.

[0056] refer to Figure 2Another structural block diagram of a model development system provided by an embodiment of the present invention is shown. The system provided by the embodiment of the present invention also includes a model monitoring module. After the developed model is deployed to the application system, the model monitoring module is configured to monitor the model effect of the developed model during actual use, and record relevant model effect monitoring information to form a model application monitoring report.

[0057] In the system provided by the embodiment of the present invention, the model monitoring module is also configured to: determine whether the developed model meets the preset usage standards when actually used, and if it does not meet the preset usage standards, issue an alarm to redevelop the model. The preset usage standards are specifically to determine whether the developed model meets the preset requirements when actually used, and to determine whether the model results generated by the developed model when actually used are consistent with the model results generated during the model development process. When it is determined that the developed model does not meet the preset requirements when actually used and / or it is determined that the model results generated by the developed model when actually used are inconsistent with the model results generated during the model development process, the model monitoring module issues an alarm to redevelop the model. In the system provided by an embodiment of the present invention, the model monitoring module supports running the model in batches using a verification data set or newly generated data through a script, testing multiple indicators of the model including stability and accuracy, continuously monitoring the effectiveness of the model algorithm, monitoring the consistency of model development and online applications (usually used to determine whether operational risks occur during model migration when deploying models across networks or systems), monitoring whether the performance of new data in the model remains stable over time, and determining whether the model needs to be reconstructed based on the monitoring results.

[0058] The system provided by the embodiment of the present invention further includes a display module, which is configured to display the training log file. In addition, the display module can also be configured to display the model application monitoring report generated by the model monitoring module and the alarm issued for redeveloping the model.

[0059] In order to more clearly illustrate the model development system of the present application, the embodiment of the present invention provides a specific example as follows:

[0060] Taking the development process of an independent model as an example, a series of serial tasks such as data cleaning, data feature screening, feature engineering, model training, model parameter tuning, model result verification and testing, and model interpretability analysis need to be completed to complete a round of model training and development. After the model development is completed, the model can also be deployed and the model application effect can be monitored. Each serial task in the model development process can be divided into one or more task steps, and the calculation tasks of each task step need to be completed using different algorithms or code tools.

[0061] Each task step in the model development process is encapsulated to form a separate computing task in a standard Docker virtualization manner. Each Docker is then started in sequence by a scheduling tool such as K8S. The input and output data of the computing tasks are serialized to form a model training workflow. All computing tasks are combined into a complete computing process in the form of a PIPLINE (pipeline) to complete the training of algorithms and models. During the model development process, the training log generation module can also automatically generate reports on model management and evaluation and related documents on model development to facilitate the backtracking of the model development process.

[0062] Due to the modular encapsulation of the training process and the independence of computing tasks in the Docker manner, on the one hand, it is easy to form logs and backtracking for each task step in the training process, strengthening the supervision of the modeling process; on the other hand, each step of the model development process is standardized, and the algorithm environment is managed in a unified standard, ensuring the consistency and security of model development and avoiding operational risks in the model algorithm development process.

[0063] The system provided by the embodiment of the present invention can be applied to various fields, especially to the banking and financial fields. When targeting financial modeling scenarios, the design and functions are mainly reflected in the following aspects:

[0064] First, in terms of algorithm support, it mainly focuses on linear regression and its derivative algorithms used in financial modeling and related algorithms based on the Boosting model improvement method. Through the system provided by the embodiment of the present invention, the corresponding components of algorithm evaluation, parameter automatic optimization, and installation and deployment of these two major types of algorithms used in financial modeling are improved.

[0065] Second, in the application of models in financial systems, since the quality of model effects will directly affect the loss of funds, in financial scenarios, the attitude towards models will be more cautious, and a more in-depth evaluation of the model effects is required. In the system provided in the embodiment of the present invention, in addition to the accuracy, recall rate, and confusion matrix methods in traditional model effect evaluation, the evaluation and monitoring of model effects can be increased by adding multi-dimensional statistical tests of models and data, stability tests, cross-period verification, improvement degree and other indicators, which is more in line with the process and method of model application in financial systems.

[0066] Third, in the system provided by the embodiment of the present invention, a cross-system and cross-network deployment solution is proposed for model deployment, which meets the design requirements of physical isolation of network environment in the bank's IT environment, and is more in line with the current incompatibility between the bank's current business system and algorithm system.

[0067] Fourth, the present invention can perform detailed backtracking, monitoring and management functions for the model development process, which complies with the risk management system for algorithm application in the financial system and the more stringent regulatory requirements in the financial system.

[0068] Fifth, the system provided by the embodiment of the present invention proposes an interpretable function design of the model. The goal is that when using relevant models based on the Boosting method, it is necessary to provide detailed descriptions of the model results, model parameters and risk control rule formulation plans for the bank's risk management process to meet the corresponding specifications of the model application.

[0069] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0070] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0071] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A model development system, characterized in that: include: Multiple application container engines Docker, used to execute multiple task steps in the model development process, each Docker executes the task step corresponding to it, and each task step is generated by independent computing task calculation; A scheduling module is used to configure the startup sequence and startup parameters of each Docker, and start each Docker in sequence according to the startup sequence of each Docker, and execute the task steps corresponding to the Docker in the model development process until all the task steps are completed to obtain the developed model, and the startup sequence of each Docker corresponds to the serial order of each task step in the model development process; A training log generation module is used to record the relevant computing task information of each Docker when executing the corresponding task steps to form a training log file; A model deployment module is used to deploy the developed model into an application system; The multiple task steps in the model development process include a first step, and the Docker that executes the first step is specifically configured to: obtain raw data required for model development, and extract features from the raw data according to preset feature extraction rules to construct a sample data set required for the model development process; In the process of executing the first step, the training log generation module is configured to record the acquisition time of the original data and the source information of the original data; The multiple task steps in the model development process also include a second step, and the Docker that executes the second step is specifically configured to: select a training algorithm required in the model development process, and apply the sample data set to train the training algorithm to obtain a first model that has been trained; in the process of executing the second step, the training log generation module is configured to record the modeling script, software and hardware conditions, developer identity, verification information, start and end timestamps, and the algorithm name, algorithm type, parameters used by the algorithm, and round information of algorithm training iterations during the training process; The multiple task steps in the model development process also include a third step, and the Docker that executes the third step is specifically configured to: evaluate the first model to determine whether the first model meets the preset requirements; If the preset requirement is not met, the model training process is repeated to obtain a new first model, until the new first model meets the preset requirement, the new first model currently obtained is determined as the developed model; in the process of executing the third step, the training log generation module is configured to record relevant evaluation information generated when evaluating the new first model obtained each time; The multiple task steps in the model development process also include a fourth step, and the Docker that executes the fourth step is specifically configured to: perform model interpretability analysis each time the model is trained; in the process of executing the fourth step, the training log generation module is configured to record the feature definition interpretability, feature abnormality interpretability and feature selection interpretability of each feature in the sample data set used for each model training, and record the algorithm interpretability and algorithm parameter interpretability corresponding to the training algorithm selected for each model training; In the process of deploying the developed model to the application system, the model deployment module is further configured to: If the network between the development environment in the model development process and the production environment of the application system is interconnected, the developed model is directly deployed to the application system; If the network between the development environment in the model development process and the production environment of the application system is physically isolated, and the deployment system of the production environment and the development environment is the same system, the developed model is packaged into a Docker image file and deployed to the application system; If there is a physical isolation between the development environment in the model development process and the production environment of the application system, and the deployment system of the production environment and the development environment is not the same system, a model file is generated based on the developed model, and the model file is provided to the application system to build a corresponding model in the application system.

2. The system according to claim 1, characterized in that The model file includes the calculation logic and model parameters of the developed model.

3. The system according to claim 1, characterized in that It also includes a model monitoring module. After the developed model is deployed to the application system, the model monitoring module is configured to monitor the model effect of the developed model during actual use, and record relevant model effect monitoring information to form a model application monitoring report.

4. The system according to claim 3, characterized in that The model monitoring module is also configured to determine whether the developed model meets preset usage standards when actually used, and if it does not meet the preset usage standards, issue an alarm to redevelop the model.

5. The system according to claim 1, characterized in that It also includes a display module, which is used to display the training log file.

Images