Model monitoring report generation method, related device and computer storage medium

Abstract

The application provides a model monitoring report generation method, related devices and a computer storage medium. The model monitoring report generation method comprises the following steps: firstly, obtaining model information of a target model; wherein the model information comprises dimension information and data source configuration information; then, updating a unified interface layer processing script according to the dimension information and the data source configuration information to obtain a target interface layer processing script; finally, processing an interface table by using the target interface layer processing script and a model monitoring index to obtain a monitoring report of the target model. Thus, the purpose of quickly generating a model monitoring report is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for generating model monitoring reports, related devices, and computer storage media. Background Technology

[0002] Currently, the traditional method for generating reports involves processing the interface table scripts required for the report on an ETL server, writing dimension tables that meet business needs into the database, generating a job by scheduling the scripts generated by the server, and then integrating the original tables, interface tables, and dimension tables in the database to generate a result table, which is then published to Congnos to generate the report.

[0003] However, when a new data source emerges, scripts need to be redeveloped for that data source before deployment, which is time-consuming and labor-intensive. During model monitoring report development, dozens of monitoring indicators are covered for different types of measurement models, generating dozens of monitoring reports. When a new measurement model is deployed in production, its results are stored in different data tables. If traditional report generation methods are used, technical personnel need to redevelop the underlying scripts for each new model, increasing the workload of report generation. When adding a new model, the dimension information table needs to be configured, which typically involves technical personnel directly performing CRUD operations on the dimension information table in the database, a time-consuming and labor-intensive process. Summary of the Invention

[0004] In view of this, this application provides a method, related apparatus, and computer storage medium for generating model monitoring reports, for quickly generating model monitoring reports.

[0005] The first aspect of this application provides a method for generating model monitoring reports, including:

[0006] Obtain the model information of the target model; wherein, the model information includes dimensional information and data source configuration information;

[0007] Based on the dimension information and the data source configuration information, update the unified interface layer processing script to obtain the target interface layer processing script;

[0008] The interface table is processed using the target interface layer processing script and model monitoring indicators to obtain a monitoring report of the target model.

[0009] Optionally, after processing the interface table using the target interface layer processing script and model monitoring indicators to obtain the target model monitoring report, the method further includes:

[0010] Generate the report information of the monitoring report of the target model; wherein, the report information of the monitoring report of the target model includes: the report name of the monitoring report of the target model, the storage location of the monitoring report of the target model, the name of the intermediate variable table of the target model, and the storage location of the intermediate variable table of the target model.

[0011] Optionally, after generating the report information of the monitoring report for the target model, the method further includes:

[0012] The report information of the monitoring report of the target model is written into the data source configuration information table of the database.

[0013] Optionally, after obtaining the model information of the target model, the method further includes:

[0014] The model information of the target model is written into the dimension information table of the database.

[0015] A second aspect of this application provides an apparatus for generating model monitoring reports, comprising:

[0016] An acquisition unit is used to acquire model information of the target model; wherein, the model information includes dimensional information and data source configuration information;

[0017] The update unit is used to update the unified interface layer processing script according to the dimension information and the data source configuration information to obtain the target interface layer processing script.

[0018] The first generation unit is used to process the interface table using the target interface layer processing script and model monitoring indicators to obtain a monitoring report of the target model.

[0019] Optionally, the device for generating the model monitoring report further includes:

[0020] The second generation unit is used to generate report information for the monitoring report of the target model; wherein, the report information for the monitoring report of the target model includes: the report name of the monitoring report of the target model, the storage location of the monitoring report of the target model, the name of the intermediate variable table of the target model, and the storage location of the intermediate variable table of the target model.

[0021] Optionally, the device for generating the model monitoring report further includes:

[0022] The first writing unit is used to write the report information of the monitoring report of the target model into the data source configuration information table of the database.

[0023] Optionally, the device for generating the model monitoring report further includes:

[0024] The second writing unit writes the model information of the target model into the dimension information table of the database.

[0025] A third aspect of this application provides a server, comprising:

[0026] One or more processors;

[0027] A storage device on which one or more programs are stored;

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating model monitoring reports as described in any one of the first aspects.

[0029] The fourth aspect of this application provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for generating model monitoring reports as described in any one of the first aspects.

[0030] As can be seen from the above scheme, the model monitoring report generation method, related device, and computer storage medium provided in this application include the following steps: First, obtaining model information of the target model; wherein, the model information includes dimension information and data source configuration information; then, updating the unified interface layer processing script according to the dimension information and the data source configuration information to obtain the target interface layer processing script; finally, processing the interface table using the target interface layer processing script and model monitoring indicators to obtain the monitoring report of the target model. This achieves the goal of quickly generating model monitoring reports. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a method for generating a model monitoring report, as provided in an embodiment of this application.

[0033] Figure 2 A detailed flowchart of a method for generating a model monitoring report, provided in another embodiment of this application;

[0034] Figure 3 A schematic diagram of a model monitoring report generation device provided in another embodiment of this application;

[0035] Figure 4 A schematic diagram of a model monitoring report generation device provided in another embodiment of this application;

[0036] Figure 5 This is a schematic diagram of a server for implementing a method for generating model monitoring reports, as provided in another embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0039] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0040] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0041] This application provides a method for generating model monitoring reports, such as... Figure 1 As shown, the specific steps include:

[0042] S101. Obtain the model information of the target model.

[0043] The model information includes dimensional information and data source configuration information.

[0044] It should be noted that dimensional information can include, but is not limited to, the model's deployment time, the evaluation metrics of the model's training samples, and the distribution information of the model's training samples; these are not limited here. The model's deployment time refers to the time the model is deployed to the production environment. It is understood that as the model is iterated and updated, older models may be optimized; in this case, the deployment time is the latest optimization time. The evaluation metrics of the model's training samples refer to the metrics used to judge the applicability of the model's training samples. This metric is submitted by the modeling team to the development team after model training, serving as an important standard for evaluating model performance. Evaluation metrics include industry-standard metrics such as recall, hit rate, and Kolmogorov-Smirnov metric values. By comparing the model's evaluation metrics on OOT (out-of-sample validation samples) with those on the model's training samples, the robustness of the model can be effectively identified. The distribution information of the model's training samples refers to the descriptive statistical distribution information of the model's training samples. This includes sample size, distribution shape, sample mean, variance, skewness, and kurtosis.

[0045] The data source configuration information can be, but is not limited to, the report name and storage location of the historical monitoring reports of the target model, and the name and storage location of the historical intermediate variable table of the target model. There are no restrictions here. The historical monitoring reports of the target model refer to the output files during model execution. This file stores the model's execution results, such as scores, probabilities, and ratings. The output results of the target model are database tables with complete primary keys and no redundant data. The model intermediate variable table refers to the file storing the model's input variables. This file stores the variable information input to the model, including the original values ​​of the input variables, the intermediate values ​​usable by the model after transformation (e.g., WOE), and the intermediate values ​​before the model outputs the final result (e.g., univariate scores). The model intermediate variable table and the model output result table are related, corresponding one-to-one through customer IDs. Records stored in the model output result table must have corresponding records in the model intermediate variable table; there should be no missing records.

[0046] Specifically, the user can enter the dimensional information and data source configuration information sequentially through the visual business front-end window after collecting them, or they can obtain it through querying in the system; there is no limitation here.

[0047] Optionally, in another embodiment of this application, the dimension information is written to the dimension information table of the database, and the data source configuration information is written to the data source configuration information table of the database.

[0048] S102. Based on the dimension information and data source configuration information, update the unified interface layer processing script to obtain the target interface layer processing script.

[0049] Specifically, a dynamic interface layer script generation program can be used in the ETL server. This program dynamically updates the unified interface layer processing script. The Pertl language is used to parse the data source configuration information table and dimension information table entered by the user (business personnel) in the visual business front-end window and stored in the database. These tables contain updatable model information. After parsing the updated model information, the dynamic interface layer generation program updates the unified interface layer processing script to obtain the target interface layer processing script.

[0050] S103. Process the interface table using the target interface layer processing script and model monitoring indicators to obtain the monitoring report of the target model.

[0051] The target interface layer processing script is a data processing program used to generate intermediate data tables and result tables for the final target model monitoring reports. In this solution, the intermediate data table mainly refers to the interface table. This program calls the data source configuration information table, dimension information table, and model monitoring indicator processing scripts stored in the database to generate the interface table and result table. This unified interface layer processing reflects the core business logic of the report monitoring requirements; therefore, script development must be tailored to these business needs. When the report business logic changes, technical personnel need to modify this script; when new reports are needed, technical personnel also need to develop new underlying scripts for the corresponding requirements. The program can be reused when the business logic and report scope remain unchanged. Most importantly, this script can be updated by the interface layer script dynamic generation program.

[0052] It should be noted that the model monitoring metrics are generated by the model monitoring metric processing script, which refers to the underlying data processing program of the model monitoring reports. Depending on business needs, the model monitoring reports focus on different business dimensions such as the model's discriminative power, stability, accuracy, prudence, and concentration; therefore, the script must be developed in accordance with business requirements. Generally, this program is relatively stable and can be deployed after testing. When the calculation method of the monitoring metrics changes, technical personnel need to modify the script; when new reports are needed, technical personnel also need to develop new underlying scripts for the corresponding requirements. The program can be reused when the business logic and report scope remain unchanged.

[0053] In practical applications, this application also includes a scheduling server, which works in conjunction with the aforementioned ETL server. The scheduling server includes a model monitoring indicator processing job, an interface table processing job, and a dynamic script generation job. The model monitoring indicator processing job corresponds to the model monitoring indicator processing script on the ETL server and is initiated according to the reporting period. This job depends on the interface table processing job. The interface table processing job corresponds to the unified interface layer processing script and the target interface layer processing script on the ETL server, and is triggered based on the source data table reporting period and the dependencies of preceding jobs. The dynamic script generation job corresponds to the interface layer script dynamic generation program on the ETL server. This job is triggered according to the job time and can also be manually adjusted by technical personnel. It should be noted that all jobs directly or indirectly depend on the dynamic script generation job.

[0054] As can be seen from the above scheme, the method for generating a model monitoring report provided in this application involves: first, obtaining the model information of the target model; wherein, the model information includes dimension information and data source configuration information; then, updating the unified interface layer processing script according to the dimension information and the data source configuration information to obtain the target interface layer processing script; finally, processing the interface table using the target interface layer processing script and model monitoring indicators to obtain the monitoring report of the target model. This achieves the goal of quickly generating model monitoring reports.

[0055] This application also provides a method for generating model monitoring reports, such as... Figure 2 As shown, the specific steps include:

[0056] S201. Obtain the model information of the target model.

[0057] S202. Based on the dimension information and the data source configuration information, update the unified interface layer processing script to obtain the target interface layer processing script.

[0058] S203. Process the interface table using the target interface layer processing script and model monitoring indicators to obtain the monitoring report of the target model.

[0059] The specific implementation methods of steps S201, S202 and S203 can be referred to steps S101, S102 and S103 in the above embodiments, respectively, and will not be repeated here.

[0060] S204. Generate monitoring report information for the target model.

[0061] The monitoring report information for the target model includes: the report name of the monitoring report for the target model, the storage location of the monitoring report for the target model, the name of the intermediate variable table of the target model, and the storage location of the intermediate variable table of the target model.

[0062] As can be seen from the above scheme, the model monitoring report generation method provided in this application involves: first, obtaining the model information of the target model; wherein, the model information includes dimension information and data source configuration information; then, updating the unified interface layer processing script according to the dimension information and the data source configuration information to obtain the target interface layer processing script; then, processing the interface table using the target interface layer processing script and model monitoring indicators to obtain the monitoring report of the target model. Finally, generating the report information of the target model monitoring report. The report information of the target model monitoring report includes: the report name of the target model monitoring report, the storage location of the target model monitoring report, the name of the target model intermediate variable table, and the storage location of the target model intermediate variable table. This achieves the goal of quickly generating model monitoring reports.

[0063] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0064] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0065] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Python, Java, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0066] Another embodiment of this application provides a device for generating model monitoring reports, such as... Figure 3 As shown, it includes:

[0067] The acquisition unit 301 is used to acquire model information of the target model.

[0068] The model information includes dimensional information and data source configuration information.

[0069] The update unit 302 is used to update the unified interface layer processing script according to the dimension information and the data source configuration information to obtain the target interface layer processing script.

[0070] The first generation unit 303 is used to process the interface table using the target interface layer processing script and model monitoring indicators to obtain the monitoring report of the target model.

[0071] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 1 As shown, it will not be elaborated further here.

[0072] Optionally, in another embodiment of this application, one implementation of the model monitoring report generation device further includes:

[0073] The first writing unit is used to write the report information of the monitoring report of the target model into the data source configuration information table of the database.

[0074] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, which will not be repeated here.

[0075] Optionally, in another embodiment of this application, one implementation of the model monitoring report generation device further includes:

[0076] The second writing unit writes the model information of the target model into the dimension information table of the database.

[0077] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, which will not be repeated here.

[0078] As can be seen from the above scheme, in the model monitoring report generation device provided in this application: First, the acquisition unit 301 acquires the model information of the target model; wherein, the model information includes dimension information and data source configuration information; then, the update unit 302 updates the unified interface layer processing script according to the dimension information and the data source configuration information to obtain the target interface layer processing script; finally, the first generation unit 303 processes the interface table using the target interface layer processing script and model monitoring indicators to obtain the monitoring report of the target model. This achieves the goal of quickly generating model monitoring reports.

[0079] Another embodiment of this application provides a device for generating model monitoring reports, such as... Figure 4 As shown, it includes:

[0080] Acquisition unit 401 is used to acquire model information of the target model.

[0081] The model information includes dimensional information and data source configuration information.

[0082] The update unit 402 is used to update the unified interface layer processing script according to the dimension information and the data source configuration information to obtain the target interface layer processing script.

[0083] The first generation unit 403 is used to process the interface table using the target interface layer processing script and model monitoring indicators to obtain the monitoring report of the target model.

[0084] The second generation unit 404 is used to generate report information for the monitoring report of the target model.

[0085] The monitoring report information for the target model includes: the report name of the monitoring report for the target model, the storage location of the monitoring report for the target model, the name of the intermediate variable table of the target model, and the storage location of the intermediate variable table of the target model.

[0086] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 2 As shown, it will not be elaborated further here.

[0087] As can be seen from the above scheme, in the model monitoring report generation device provided in this application: First, the acquisition unit 401 acquires the model information of the target model; wherein, the model information includes dimension information and data source configuration information; then, the update unit 402 updates the unified interface layer processing script according to the dimension information and the data source configuration information to obtain the target interface layer processing script; the first generation unit 403 processes the interface table using the target interface layer processing script and model monitoring indicators to obtain the monitoring report of the target model. Finally, the second generation unit 404 generates the report information of the monitoring report of the target model. The report information of the monitoring report of the target model includes: the report name of the monitoring report of the target model, the storage location of the monitoring report of the target model, the name of the intermediate variable table of the target model, and the storage location of the intermediate variable table of the target model. This achieves the purpose of quickly generating model monitoring reports.

[0088] The units described in the embodiments of this application can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0089] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0090] Another embodiment of this application provides a server, such as Figure 5 As shown, it includes:

[0091] One or more processors 501.

[0092] Storage device 502, on which one or more programs are stored.

[0093] When the one or more programs are executed by the one or more processors 501, the one or more processors 501 implement the model monitoring report generation method as described in any of the above embodiments.

[0094] Another embodiment of this application provides a computer storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the method for generating model monitoring reports as described in any of the above embodiments.

[0095] In the embodiments disclosed in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0096] Furthermore, the functional modules in the various embodiments of this disclosure can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a live streaming device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0097] Those skilled in the art will be able to implement or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating model monitoring reports, characterized in that, include: Obtain the model information of the target model; wherein, the model information includes dimensional information and data source configuration information; the data source configuration information includes: the report name of the historical monitoring report of the target model and the storage location of the historical monitoring report name, the name of the historical intermediate variable table of the target model and the storage location of the historical intermediate variable table of the target model; the dimensional information includes the model's launch time, the evaluation index value of the model training samples and the distribution information of the model training samples; Based on the dimensional information and the data source configuration information, the unified interface layer processing script is updated to obtain the target interface layer processing script; wherein, the target interface layer processing script is a data processing program used to generate the intermediate data table and result table of the monitoring report of the final target model; the intermediate data table includes at least the interface table; The interface table is processed using the target interface layer processing script and model monitoring indicators to obtain a monitoring report of the target model.

2. The generation method according to claim 1, characterized in that, After processing the interface table using the target interface layer processing script and model monitoring indicators to obtain the target model monitoring report, the process further includes: Generate the report information of the monitoring report of the target model; wherein, the report information of the monitoring report of the target model includes: the report name of the monitoring report of the target model, the storage location of the monitoring report of the target model, the name of the intermediate variable table of the target model, and the storage location of the intermediate variable table of the target model.

3. The generation method according to claim 2, characterized in that, After generating the monitoring report information for the target model, the method further includes: The report information of the monitoring report of the target model is written into the data source configuration information table of the database.

4. The generation method according to claim 1, characterized in that, After obtaining the model information of the target model, the process also includes: The model information of the target model is written into the dimension information table of the database.

5. A device for generating model monitoring reports, characterized in that, include: The acquisition unit is used to acquire model information of the target model; wherein, the model information includes dimensional information and data source configuration information; the data source configuration information includes: the report name of the historical monitoring report of the target model and the storage location of the historical monitoring report name, the name of the historical intermediate variable table of the target model and the storage location of the historical intermediate variable table of the target model; the dimensional information includes the model's launch time, the evaluation index value of the model training samples and the distribution information of the model training samples; The update unit is used to update the unified interface layer processing script according to the dimension information and the data source configuration information to obtain the target interface layer processing script; wherein, the target interface layer processing script is a data processing program used to generate the intermediate data table and the result table of the monitoring report of the final target model; the intermediate data table includes at least the interface table; The first generation unit is used to process the interface table using the target interface layer processing script and model monitoring indicators to obtain a monitoring report of the target model.

6. The generating apparatus according to claim 5, characterized in that, Also includes: The second generation unit is used to generate report information for the monitoring report of the target model; wherein, the report information for the monitoring report of the target model includes: the report name of the monitoring report of the target model, the storage location of the monitoring report of the target model, the name of the intermediate variable table of the target model, and the storage location of the intermediate variable table of the target model.

7. The generating apparatus according to claim 6, characterized in that, Also includes: The first writing unit is used to write the report information of the monitoring report of the target model into the data source configuration information table of the database.

8. The generating apparatus according to claim 5, characterized in that, Also includes: The second writing unit writes the model information of the target model into the dimension information table of the database.

9. A server, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating model monitoring reports as described in any one of claims 1 to 4.

10. A computer storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the method for generating model monitoring reports as described in any one of claims 1 to 4.

Images