Enterprise-level model management system
Through the design of an enterprise-level model management system, the problem of inefficient model management in the existing technology is solved, centralized management of model assets and full life cycle monitoring are realized, and the efficiency and decision support of model management are improved.
Patent Information
- Application Number
- CN202411139661.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-13
AI Technical Summary
The existing model management system lacks unified model lifecycle management and model asset management functions, resulting in inefficient model management and failure to effectively accumulate model assets, limiting the value of its value.
An enterprise-level model management system was designed, including model development module, model monitoring module, model full life cycle management approval module and model asset management module, providing data analysis, model training, storage, monitoring, early warning, approval and asset management, etc. Function to realize the full life cycle management of the model and centralized asset management.
The centralized management of model assets is realized, and the status of model assets is displayed in real time, providing management with a broad market perspective, and timely grasping model effects and performance declines through diversified monitoring and early warning mechanisms, and timely taking disposal measures, which improves the efficiency of model management and Decision support.
Smart Images
Figure CN119130274A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of model management, and in particular is an enterprise-level model management system. Background Art
[0002] With the development of big data technology and artificial intelligence, commercial banks are faced with the need to use and manage massive models in their digital technology development plans. The construction of a model management system has become a must for banks to achieve digital transformation. In recent years, domestic regulators have also put forward requirements for the management of banking models, including full life cycle management of models, and establishing a full-process management mechanism covering model development, verification, deployment, monitoring, and exit.
[0003] As the number of model frameworks used increases and the differences between them increase, the efficiency of model lifecycle management becomes lower and lower. At the same time, the model-related documents, codes, and data scattered across various business departments lack unified management, and model assets are not accumulated, limiting their value. Some model management systems in the existing technology, although some framework models have development and deployment functions, lack unified model lifecycle management and model asset management functions. Summary of the invention
[0004] In view of the above problems, the present invention is proposed to provide an enterprise-level model management system that overcomes the above problems or at least partially solves the above problems.
[0005] To achieve the above object, the present invention adopts the following technical solutions:
[0006] An enterprise-level model management system, the system comprising:
[0007] Model development module, used to realize data analysis and exploration functions, model training functions, model storage functions and code storage functions;
[0008] The model monitoring module is used to realize the automated model monitoring and early warning functions. By setting model-level monitoring and early warning indicators, the business effect and degradation degree of the stock model can be timely grasped, and disposal measures can be taken in time according to the early warning rules;
[0009] Model full life cycle management and approval module, used to achieve systematic management of the model's full life cycle;
[0010] The model asset management module is used to manage and maintain model assets, establish a unified model asset list and view, and provide real-time feedback on model asset status.
[0011] Optional model development modules include:
[0012] The code modeling unit is used for initialization settings. After the initialization is completed, the code modeling resources are allocated to the user according to the input CPU and memory;
[0013] The visual modeling unit provides a drag-and-drop visual modeling interface. Users use built-in algorithm components to build learning flows and perform model training and debugging.
[0014] Intelligent Modeling Unit, which automates time-consuming, repetitive tasks in developing machine learning models;
[0015] The periodic task unit is used to provide users with retrain periodic tasks, custom periodic tasks, and pyspark periodic tasks;
[0016] Algorithm management unit, used to support users to upload custom algorithms;
[0017] The operation record unit is used to view the operation records and alarm records of learning flow tasks, operator tasks, and periodic tasks.
[0018] Optionally, the operation process of the visual modeling unit includes creating a project, creating a learning flow, dragging and dropping operators and configuring algorithm parameters, running the learning flow and saving the model file.
[0019] Optionally, the intelligent modeling unit includes AutoML template management, AutoML tasks and AutoML evaluation reports.
[0020] Optionally, the operation process of the intelligent modeling unit includes creating AutoML, selecting a module and creating an AutoML task, running the AutoML task, viewing the AutoML evaluation report, and saving the model file.
[0021] Optionally, the model monitoring module includes:
[0022] Personal dashboard unit for providing customized reports;
[0023] The model dashboard unit is used to understand the overall situation of the model through charts, display the global dashboard monitoring report of the model, and support refreshing, downloading, comparing and filtering the reports displayed in the dashboard;
[0024] Monitoring and reporting unit, used to set up monitoring for models, view reports, and support adding new reports and modifying reports;
[0025] The warning management unit is used to view, enable or disable warnings, edit warning rules, and delete warnings. It supports configuration of warning information and connects to the user warning system. When indicators reach the threshold, warnings are sent.
[0026] The binning management unit is used to discretize continuous variables in sections, merge multi-state discrete variables, and reduce the number of states of discrete variables.
[0027] Optional, model lifecycle management approval module includes:
[0028] My Model unit is used to display all models that the current user is responsible for and participates in managing, so as to achieve centralized management of models;
[0029] Model list unit, used to view model details;
[0030] Approval switch unit, including model operation review and file download review. Model operation review is used to control whether model operation needs approval, and file download review is used to control whether model file download needs approval.
[0031] Model asset information unit, used to report model asset information to the data asset management system;
[0032] The model asset folder directory development unit is used to display all created model templates, support query by name, and support adding, editing, and deleting operations on model detail templates.
[0033] Optionally, the operation process of the model asset information unit includes scheduling tasks, obtaining model asset information, writing to a specified working directory, GTP synchronization files and data asset management.
[0034] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0035] 1. The present invention centrally manages model assets in an enterprise-level model management system, including establishing a unified model asset list and asset view for the entire bank, displaying the status of the bank's model assets in real time, and providing a big picture perspective for management.
[0036] 2. The present invention sets up a variety of monitoring and early warning mechanisms to timely grasp the effects of the models currently in use, and timely track the degree of model performance degradation, the effect in decision-making, etc. according to the early warning rules. When a model alarm occurs, timely disposal measures can be taken. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A schematic diagram of an enterprise-level model management system provided in an embodiment of the present application;
[0038] Figure 2 This is a flowchart of the Notebook startup process;
[0039] Figure 3 Create a swimlane flow diagram for the Notebook. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0041] See also Figure 1 This embodiment provides an enterprise-level model management system, including:
[0042] The model development module is used to realize data analysis and exploration functions, model training functions, model storage functions and code storage functions. The model training function can be realized through code modeling, visual modeling, intelligent modeling and other methods.
[0043] The model development module includes:
[0044] The code modeling unit is used for initialization settings. After initialization, the code modeling resources are allocated to the user according to the input CPU and memory. The system integrates the Notebook development environment and provides users with a way to use Notebook for code modeling. Users obtain the allocated Notebook instance for code modeling development by calling the Jupyter management service (Turing). At the same time, this module integrates a code repository based on Gitlab to provide support for code version management and team collaborative development.
[0045] The notebook startup process is as follows Figure 2 shown.
[0046] Notebook creates swimlanes like Figure 3 shown.
[0047] Figure 3 Introduction to the core process nodes:
[0048] Turing application: a backend web management application for the machine learning platform.
[0049] enki-turing-k8s application: an encapsulation proxy for the k8s interface, which can be seen as a gateway between the turing application and the k8s cluster.
[0050] Notebook: A jupyter notebook started in a k8s cluster based on user-configured resources. One user corresponds to one deployment.
[0051] The visual modeling unit provides a drag-and-drop visual modeling interface. Users use built-in algorithm components to build learning flows, and perform model training and debugging, ultimately achieving zero-code modeling. It mainly includes three main modules: experimental learning flow, operator components, and engineering models.
[0052] Intelligent modeling units to automate time-consuming, repetitive tasks in the machine learning model development process.
[0053] The periodic task unit is used to provide users with retrain periodic tasks, custom periodic tasks, and pyspark periodic tasks; the retrain periodic task performs scheduled retraining of the learning stream based on automatic task scheduling, with the goal of training the best applicable model based on the continuously updated table data. The custom periodic task is similar to the retrain periodic task, but does not have the comparison, evaluation, and automatic saving of the learning stream model. It is only a scheduled periodic retraining of the learning stream. The pyspark periodic task mainly supports periodic scheduling and running of the pyspark code written by the user.
[0054] The algorithm management unit is used to support users to upload custom algorithms and use specified algorithms in Jupyter notebook.
[0055] The operation record unit is used to view the operation records and alarm records of learning flow tasks, operator tasks, and periodic tasks.
[0056] The operation process of the visual modeling unit includes creating a project, creating a learning flow, dragging and dropping operators and configuring algorithm parameters, running the learning flow and saving the model file.
[0057] The intelligent modeling unit includes AutoML template management, AutoML tasks, and AutoML evaluation reports.
[0058] The operation process of the intelligent modeling unit includes creating AutoML, selecting modules and creating AutoML tasks, running AutoML tasks, viewing AutoML evaluation reports, and saving model files.
[0059] The model monitoring module is used to realize automated model monitoring and early warning functions. By setting model-level monitoring and early warning indicators, the business performance and degradation degree of the existing models can be timely grasped, and disposal measures can be taken in time according to the early warning rules.
[0060] The model monitoring module includes:
[0061] Personal dashboard unit for providing customized reports.
[0062] The model dashboard unit is used to understand the overall situation of the model through charts, display the global dashboard monitoring report of the model, and support refreshing, downloading, comparing and filtering the reports displayed in the dashboard.
[0063] The monitoring and reporting unit is used to set up monitoring for the model, view reports, and support adding new reports and modifying reports.
[0064] The warning management unit is used to view, enable or disable warnings, edit warning rules, and delete warnings. It supports configuration of warning information and connects to the user warning system. When the indicator reaches the threshold, a warning is sent.
[0065] The binning management unit is used to discretize continuous variables in sections, merge multi-state discrete variables, and reduce the number of states of discrete variables. Binning includes equal interval, equal frequency, and by category. Models without binning information need to add binning information before setting the model monitoring report. After the model is launched, the monitoring is mainly for online models, and the operation status of the online model is displayed through the model dashboard.
[0066] The model life cycle management and approval module is used to achieve systematic management of the model's entire life cycle; by defining the requirements, development, verification, launch, monitoring and exit links in the model life cycle and setting approval permissions, systematic management of the model's entire life cycle is achieved. In order to meet the business needs of commercial banks with their own characteristics, this module has the ability to formulate different work processes based on the risk level of the model to achieve differentiated management of the model.
[0067] The model life cycle management approval module includes:
[0068] My Models unit is used to display all models that the current user is responsible for and participates in managing, so as to realize the centralized management of models. You can query models according to conditions, view model details, initiate workflows, set, delete, and register new models for the queried "My Models".
[0069] The model list unit is used to view model details; the model list includes all models and the two modules I applied for. All models support application permissions, and the list I applied for supports viewing the models for which I applied for permissions.
[0070] The approval switch unit includes model operation review and file download review. Model operation review is used to control whether model operation requires approval, and file download review is used to control whether model file download requires approval.
[0071] The model asset information unit is used to report model asset information to the data asset management system.
[0072] The model asset folder directory development unit is used to display all created model templates, support query by name, and support adding, editing, and deleting operations on model detail templates.
[0073] The operation process of the model asset information unit includes scheduling tasks, obtaining model asset information, writing to the specified working directory, GTP synchronization files and data asset management.
[0074] The model asset management module is used to manage and maintain model assets, establish a unified model asset list and view, and provide real-time feedback on the status of model assets to provide management with a broad perspective. It includes sub-modules such as My Model, Model List, Model Details, Approval Switch, and Model Asset Information.
[0075] This embodiment puts model assets into an enterprise-level model management system for centralized management, including establishing a unified model asset list and asset view for the entire bank, showing the status of the bank's model assets in real time, and providing a big picture perspective for management. The present invention can record processes including model requirement proposal, model development, model verification, model post-production monitoring, model optimization iteration, and model exit.
[0076] This embodiment sets up a variety of monitoring and early warning mechanisms to timely grasp the effects of the models currently in use, and timely track the degree of model performance degradation, the effect in decision-making, etc. according to the early warning rules. When a model alarm occurs, timely disposal measures can be taken.
[0077] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. An enterprise-level model management system, It is characterized in that The system comprises: Model development module, which is used to realize data analysis and exploration functions, model training functions, model storage functions, and code storage functions; The model monitoring module is used to realize the automated model monitoring and early warning functions. By setting model-level monitoring and early warning indicators, the business effect and degradation degree of the stock model can be timely grasped, and disposal measures can be taken in time according to the early warning rules; Model full life cycle management and approval module, used to achieve systematic management of the model's full life cycle; The model asset management module is used to manage and maintain model assets, establish a unified model asset list and view, and provide real-time feedback on model asset status.
2. An enterprise-level model management system as claimed in claim 1, It is characterized in that The model development module includes: The code modeling unit is used for initialization settings. After the initialization is completed, the code modeling resources are allocated to the user according to the input CPU and memory; The visual modeling unit provides a drag-and-drop visual modeling interface. Users use built-in algorithm components to build learning flows and perform model training and debugging. Intelligent Modeling Unit, which automates time-consuming, repetitive tasks in developing machine learning models; The periodic task unit is used to provide users with retrain periodic tasks, custom periodic tasks, and pyspark periodic tasks; Algorithm management unit, used to support users to upload custom algorithms; The operation record unit is used to view the operation records and alarm records of learning flow tasks, operator tasks, and periodic tasks.
3. An enterprise-level model management system as claimed in claim 2, It is characterized in that The operation process of the visual modeling unit includes creating a project, creating a learning flow, dragging and dropping operators and configuring algorithm parameters, running the learning flow and saving the model file.
4. An enterprise-level model management system as claimed in claim 2, It is characterized in that The intelligent modeling unit includes AutoML template management, AutoML tasks and AutoML evaluation reports.
5. An enterprise-level model management system as claimed in claim 2, It is characterized in that The operation process of the intelligent modeling unit includes creating AutoML, selecting modules and creating AutoML tasks, running AutoML tasks, viewing AutoML evaluation reports and saving model files.
6. An enterprise-level model management system as claimed in claim 1, It is characterized in that The model monitoring module includes: Personal dashboard unit for providing customized reports; The model dashboard unit is used to understand the overall situation of the model through charts, display the global dashboard monitoring report of the model, and support refreshing, downloading, comparing and filtering the reports displayed in the dashboard; Monitoring and reporting unit, used to set up monitoring for models, view reports, and support adding new reports and modifying reports; The warning management unit is used to view, enable or disable warnings, edit warning rules, and delete warnings. It supports configuration of warning information and connects to the user warning system. When indicators reach the threshold, warnings are sent. The binning management unit is used to discretize continuous variables in sections, merge multi-state discrete variables, and reduce the number of states of discrete variables.
7. An enterprise-level model management system as claimed in claim 1, It is characterized in that The model life cycle management approval module includes: My Model unit is used to display all models that the current user is responsible for and participates in managing, so as to achieve centralized management of models; Model list unit, used to view model details; Approval switch unit, including model operation review and file download review. Model operation review is used to control whether model operation needs approval, and file download review is used to control whether model file download needs approval. Model asset information unit, used to report model asset information to the data asset management system; The model asset folder directory development unit is used to display all created model templates, support query by name, and support adding, editing, and deleting operations on model detail templates.
8. An enterprise-level model management system as claimed in claim 7, It is characterized in that The operation process of the model asset information unit includes scheduling tasks, obtaining model asset information, writing into a specified working directory, gtp synchronization files and data asset management.