A model verification report generation method, device, equipment, medium and product

Abstract

The application discloses a model verification report generation method, device, equipment, medium and product, and relates to the field of artificial intelligence, in particular to information technology related to the field of financial technology. The method comprises the following steps: obtaining a target template corresponding to the type of a credit risk measurement model, wherein the target template contains the union of the verification contents of the same type of model verification, and the verification contents are presented in the form of set labels; obtaining target data corresponding to the model group and the parameter group of the target template, executing the processing logic corresponding to the set labels according to the target data, and obtaining report contents; replacing the set labels according to the report contents, and outputting a model verification report. The technical scheme of the application realizes automatic filling of parameters and data in the report template, generates a complete model verification report, can improve the report compiling efficiency, reduce the operation risk of writing errors or data falsification, and improve the accuracy of report compiling.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to an information technology related to the financial technology field, specifically to a method, apparatus, device, medium and product for generating model verification reports. Background Technology

[0002] The model validation report is an important document that provides a comprehensive review of the model validation results.

[0003] Model validation analysis typically involves a significant workload, encompassing large volumes and diverse types of data. The validation conclusions must be objective, impartial, accurate, and timely, necessitating a high level of expertise from those writing the model validation reports. Currently, validation reports for industry-wide retail scoring models, retail pooling models, and non-retail rating models are all manually written. This process is time-consuming and carries the risk of errors or data falsification. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for generating model validation reports, which can solve problems related to report writing efficiency and operational risks.

[0005] In a first aspect, embodiments of the present invention provide a method for generating a model validation report, comprising:

[0006] Obtain a target template corresponding to the type of credit risk measurement model, wherein the target template contains the union of the verification content of this type of model, and the verification content is presented in the form of a set label;

[0007] Obtain the target data corresponding to the model group and parameter group of the target template, execute the processing logic corresponding to the set label according to the target data, and obtain the report content;

[0008] Replace the set labels according to the report content, and output the model validation report.

[0009] Secondly, embodiments of the present invention also provide an apparatus for generating a model validation report, the apparatus comprising:

[0010] The template acquisition module is used to acquire a target template corresponding to the type of credit risk measurement model. The target template contains the union of the verification content of this type of model, and the verification content is presented in the form of a set label.

[0011] The template execution module is used to obtain target data corresponding to the model group and parameter group of the target template, execute the processing logic corresponding to the set label according to the target data, and obtain the report content.

[0012] The report verification module is used to replace the set labels according to the report content and output a model verification report.

[0013] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating a model verification report as described in any of the embodiments of the present invention.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating a model verification report as described in any of the embodiments of the present invention.

[0015] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method for generating a model verification report as described in any of the embodiments of the present invention.

[0016] In this embodiment of the invention, by selecting a target template, target data corresponding to the model group and parameter group of the target template is obtained. The processing logic corresponding to the labels set in the target template is executed according to the target data to obtain report content. The set labels in the target template are then replaced with the report content, and a model validation report is output. This embodiment solves the problems of report writing efficiency and operational risks by automatically filling parameters and data into the report template, generating a complete model validation report. This significantly reduces the workload of text editing, improves report writing efficiency, reduces operational risks such as writing errors or data falsification, and improves the accuracy of report writing. Furthermore, the use of a standard report template greatly enhances the standardization of the model validation report. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a method for generating a model validation report according to an embodiment of the present invention;

[0019] Figure 2 A flowchart illustrating another method for generating a model validation report provided in an embodiment of the present invention;

[0020] Figure 3A flowchart illustrating another method for generating a model validation report provided in an embodiment of the present invention;

[0021] Figure 4 A flowchart illustrating another method for generating a model validation report provided in an embodiment of the present invention;

[0022] Figure 5 A structural block diagram of a model validation report generation device provided in an embodiment of the present invention;

[0023] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0025] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0026] First, the technical terms that may appear in the embodiments of the present invention will be explained.

[0027] General Template: This refers to a general template for the validation report after model deployment. The general template is a Word document, a standardized report template prepared according to the bank's model validation management requirements. Different general templates are used depending on the model type. Based on model type, they can be categorized as follows: general template for retail scoring models, general template for non-retail rating models, general template for retail PD pools, general template for retail LGD pools, general template for retail EAD pools, etc. Compared to the validation report, the general template has two key characteristics: First, the validation content in the general template is parameterized; model group names, validation time points, validation results, and validation conclusions are all presented using specially agreed-upon tags. Second, the general template is the union of validation content for this type of model, summarizing all validation content for this type of model (if the general template does not cover a particular area, this should be added to the corresponding section of the template). The template does not specifically distinguish between identical parts of the validation content for this type of model; however, it differentiates between different parts of the validation content for this type of model by matching the model number with the execution path.

[0028] Model monitoring reports: These reports are used to monitor the data generated after the model is deployed. Using specific evaluation indicators of the risk control model as the monitoring granularity, they form a multi-dimensional evaluation system horizontally and a multi-model performance comparison system under a single evaluation indicator vertically. They are the most important data source for model validation reports. During the model validation report generation phase, under certain parameter conditions, they are used to obtain the data required for the model validation report.

[0029] Model Groups: Model validation reports typically focus on a class of models. Each model within a model group is grouped together as a sub-model. This means combining one or more sub-models (monitoring reports monitor single models at a granular level) into a single model group. The model group list clearly specifies the number, name, and ID of the sub-models included in each model within the validation report.

[0030] Parameter Group: Outputs the model validation report. Required information includes: Model Name, Model Number, Sub-model Name, Sub-model Number, Model / Sub-model Reporting Period, and other custom parameters. General templates define various parameter tags and insert them into the template. Since the parameters referenced by different types of general templates may differ, the parameter group maintenance function first divides parameters into custom parameters and fixed parameters. Fixed parameter tags remain unchanged for each parameter group. Custom parameters can be defined by the validation personnel, generating specific tags for their use. The parameter group is set for the template to be configured, and parameter tags are generated.

[0031] Automation tools: These are post-production model validation automation tools that automatically acquire and populate validation data by establishing a mapping relationship between validation data in the model validation report and model monitoring report data. The workflow for using these automation tools can include: completing system configuration, template parameter configuration, and various tag configurations, followed by template editing. Users can create / select an editable template (create a new blank template, import a report sample (for modification into a template), or select an unpublished template), edit the report template, and then attempt to execute the template (setting parameters) to view detailed execution results and download the validation report. This verifies the completeness, correctness, and usability of the report template design. Once confirmed to be correct, the report template can be sent to the administrator for review and publication. Validation personnel set the model group name, validation time point (data validation, model validation, variable validation, etc., each corresponding to its own validation time point), and specific values ​​for custom parameters according to the specific requirements of this validation. The automation tool then automatically outputs the model validation report.

[0032] Term tags: These are verification term tags, generated in pairs by automated tools. One is a conclusive term tag, in the template format {Data Validation_Consistency_Conclusion}, and the other is a descriptive term tag, in the template format {Data Validation_Consistency_Description}. The judgment logic for both term tags is the same, but the content they replace differs. The content replaced by the conclusive term tag is "Normal / Attention / Should be Improved," while the content replaced by the descriptive term tag is a discussion elaborating on the above conclusion, such as "The proportion of some variable values ​​in the model exceeds the threshold; further refinement or replacement can be considered to avoid overly concentrated values ​​that fail to fully reflect risk characteristics." The main purpose of these terms is to automatically generate verification conclusions based on fixed verification thresholds, providing auxiliary references for verification personnel and further improving report writing speed and standardization.

[0033] Data retrieval labels: These are data retrieval logic labels. Data retrieval logic refers to the SQL statements automatically generated by automation tools, specified by the verification personnel, to extract certain fields from a monitoring report. Data retrieval labels represent this data retrieval logic. There are two label formats: one is [Retail Rating_4220_Data Consistency], placed where the template needs to display this data. The replaced content appears in the report as an Excel plugin, containing the report data extracted from the monitoring report according to this data retrieval logic; the other is <Retail Rating_4220_Data Consistency>, placed after the column name in the leftmost column of the table in the document. The replaced content appears in the report as a table, with the table header maintaining the template's style, and the extracted content appended to several rows after the header row.

[0034] Parameter labels: Parameter labels are divided into two categories. One category is labels for fixed parameters. Fixed parameters are required for each parameter group, so the labels for fixed parameters are the same for different parameter groups, including {parameter group name}, {number of sub-models}, {sub-model name}, {data validation time point 1}, {model validation time point 1}, etc. The other category is labels for custom parameters. Custom parameters are set freely by the validation personnel. After the automated tool generates the label {xxx}, the label is pasted into the corresponding position in the template.

[0035] Figure 1 This is a flowchart illustrating a method for generating a model validation report according to an embodiment of the present invention. This method is applicable to the automatic generation of validation reports for credit risk measurement models. The method can be executed by a model validation report generation device, which can be implemented in hardware and / or software and is typically housed in an electronic device. For example, the electronic device can be a server or a server cluster. Figure 1 As shown, the method includes:

[0036] S110. Obtain the target template corresponding to the type of credit risk measurement model.

[0037] The target template contains the union of the verification content of this type of model verification, and the verification content is presented in the form of set tags.

[0038] For example, based on the model type input by the user, a general template corresponding to that type of credit risk measurement model is obtained as the target template. Specifically, the target template is selected from the general templates based on the type of credit risk measurement model for which the validation report is to be compiled. For example, if a model validation report needs to be output for a retail scoring model, then the general template for retail scoring models is selected.

[0039] S120. Obtain target data corresponding to the model group and parameter group of the target template, and execute the processing logic corresponding to the set label according to the target data to obtain the report content.

[0040] The target data can be specific values ​​for a model group or a parameter group. For example, the target data for a model group may include one or more of the following: model type, model name, model number, and number of sub-models. The target data for a parameter group may include one or more of the following: model group name, number of sub-models, sub-model name, performance period, data validation point 1, data validation point 2, data validation point enumeration, model validation point 1, model validation point 2, and model validation point enumeration.

[0041] The tags can include one or more of the following: term tags, data retrieval tags, and parameter tags. That is, the general template for the model validation report can include term tags, data retrieval tags, and parameter tags, and tags can be added or removed as needed. Alternatively, other types of tags can be added based on the specific application.

[0042] In some embodiments, when the set label is a term label, for each sub-model in the model group, the verification time point in the target data is obtained, and a merged conclusion after merging the verification time points is generated based on the verification conclusions corresponding to the verification time points; the model group conclusion of the model group is determined according to the merged conclusions of individual sub-models according to the set rules, and the model group conclusion is used as the report content.

[0043] Specifically, the number of values ​​in the "Values" field of the term list in the term list interface is named x. x is divided by the total number of sub-models y, and the percentage is calculated as z. The calculated z is compared with a set threshold to obtain the overall conclusion of the model group, and this conclusion is used as the report content. For example, if the model group contains 5 sub-models, and 3 of them have a normal conclusion after time-point merging (i.e., x = 3), and the number of sub-models y = 5), then z = (x / y) * 100%. z is compared with a set threshold to obtain the overall conclusion of the model group, and this conclusion is used as the report content.

[0044] In some embodiments, when the set label is a data retrieval label, the sub-model number and verification time point in the target data are obtained; data retrieval logic is generated based on the data retrieval label, the sub-model number, and the verification time point; report data is obtained from the model monitoring report based on the data retrieval logic; and report content is determined based on the report data.

[0045] Specifically, data retrieval tags can be [Verification Type_Monitoring Report Number_Serial Number] or <Verification Type_Monitoring Report Number_Serial Number>, etc. For example, data retrieval tags can be [Retail Rating_4120_1] or <Retail Rating_4120_1>. When configuring data retrieval logic for each data retrieval tag, the first-level heading is the verification category, the second-level heading is the verification content under the verification category, the unique identifier is the table path identifier, the time type is usually cross-sectional or time-series, and the field names of the data retrieval logic include reporting period, model name, number of model indicators, and total number of indicator missing rates, etc. The data retrieval logic model will use the model number and verification time point parameters as data filtering conditions. If there are additional filtering conditions, they can be added through "Add Query Conditions". The "Data Retrieval Logic Preview" function of the automation tool can display the SQL of the data retrieval logic for preview.

[0046] For example, the data retrieval logic can be "select reporting period, model name, number of model indicators, number of customers, total number of missing indicators from 4120_1 where for (report model number = {model configuration list.submodel number} and (verification time point = (execution parameter table.verification time point)))".

[0047] In some embodiments, when the set label is a parameter label, the target data is used as the report content corresponding to the parameter label.

[0048] In this embodiment of the invention, parameter groups are set for the template to be configured, and parameter labels are generated. It should be noted that since the parameters referenced by different types of general templates may vary, the parameter group maintenance function divides parameters into custom parameters and fixed parameters. Fixed parameter labels remain unchanged for each parameter group, while custom parameters can be freely defined, generating specific labels for use. For example, a parameter group list can display the parameter group name, the number of custom parameters, and the number of fixed parameters. Fixed parameters are predefined parameters that are available in all report templates by default. If a required parameter is not present in the fixed parameters, a custom parameter can be added to meet the needs of different reports.

[0049] For parameter labels with fixed parameters: {Model Group Name}, {Number of Sub-models}, {Sub-model Name}, {Performance Period}, {Data Validation Point 1}, {Data Validation Point 2}, {Data Validation Point Enumeration}, {Model Validation Point 1}, {Model Validation Point 2}, and {Model Validation Point Enumeration}, input the corresponding target parameters: Model Group Name, Number of Sub-models, Sub-model Name, Performance Period, Data Validation Point 1, Data Validation Point 2, Data Validation Point Enumeration, Model Validation Point 1, Model Validation Point 2, and Model Validation Point Enumeration, as the report content.

[0050] S130. Replace the set labels according to the report content and output the model verification report.

[0051] Since the label is set as a placeholder representing variable content in the target template, the corresponding report content is used to replace the placeholder after the report template is instantiated.

[0052] In some embodiments, replacing the set tags according to the report content includes: replacing the term tags in the target template with the conclusions of the model group.

[0053] Assume the term tag in the target template is: "Based on the conclusions of both periods, the discrimination risk level is: {LGD pooling_model validation_discrimination_conclusion}". After the template is instantiated and a validation report is generated, the tag positions will be replaced by the actually calculated conclusions.

[0054] Replacement result: Combining the conclusions of the two phases, the risk level of discriminative power is: normal.

[0055] In some embodiments, replacing the set label according to the report content includes: determining the label replacement method based on the symbol of the data retrieval label; when the label replacement method is to insert a table, the report content is inserted into the position of the data retrieval label in the target template in the form of a table to replace the data retrieval label; when the label replacement method is to attach a report, the report content is attached to the target template in the form of an attachment, and the corresponding data retrieval label in the target template is replaced by a set statement.

[0056] The verification report will display and list the data generated during the verification process in the form of tables or attachments to support the verification conclusions. The data extraction label will be displayed after the report is instantiated, based on the data extracted from the model monitoring report.

[0057] Assuming the data retrieval label is: <Verification Type_Monitoring Report Number_Serial Number>, as shown in Table 1, the data will be populated into the table after instantiation.

[0058] Table 1 is a table diagram showing the positions of the data retrieval labels.

[0059]

[0060] Assuming the data retrieval label is: [Verification Type_Monitoring Report Number_Serial Number], as shown below, after instantiation, the label position will be replaced by the Excel attachment containing the data.

[0061] For example, the process for {model group name} was reproduced using the above rules and compared with the pooling results of the retail pooling system. The reproduction results showed no difference; detailed results are as follows:

[0062] Note: For detailed verification results, please refer to [LGD Pool_4340_Consistency].

[0063] The result after replacing the data tags is as follows:

[0064] The PD pooling process for micro and small enterprises was reproduced using the above rules, and compared with the pooling results of the retail pooling system. The reproduction results showed no difference. Detailed results are as follows:

[0065] Note: For detailed verification results, please see Attachment 01 - Data Consistency.

[0066] In some embodiments, replacing the set label according to the report content includes: replacing the parameter label in the target template with the report content.

[0067] The parameter labels can be: {model group name}, {data validation time point 1}, etc. After the report is instantiated, {model group name} is replaced with model group name, and {data validation time point 1} is replaced with data validation time point 1.

[0068] This invention provides a method for generating model validation reports. Using a universal template and pre-set configuration information, validation reports for different models can be instantiated, addressing issues related to report writing efficiency and operational risks. By automatically filling in parameters and data using the report template, a complete model validation report is generated, significantly reducing text editing workload, improving report writing efficiency, lowering operational risks such as writing errors or data falsification, and increasing report accuracy. Furthermore, the use of a standard report template greatly enhances the standardization of model validation reports.

[0069] Figure 2 This is a flowchart illustrating another method for generating a model validation report according to an embodiment of the present invention. This embodiment adds a configuration method for a general template based on the above embodiments. Figure 2 As shown, the method includes:

[0070] S201. Set up model groups according to model type, model name, model number, and number of sub-models.

[0071] In some implementations, model group configuration primarily involves combining one or more sub-models (with model monitoring reports monitoring at the single-model level) into a single model group. Typically, multiple sub-models are grouped together for validation after model deployment. New model groups can be added as needed. The sub-models can be filtered on the new model group page, and the required sub-models can be found by querying them. One or more sub-models can then be selected. Furthermore, defined model groups can be modified or deleted.

[0072] S202. Determine the table mapping relationship based on the model monitoring report number, model monitoring report name, and database table name of the model monitoring report.

[0073] For example, determining the table mapping relationship based on the model monitoring report number, model monitoring report name, and database table name of the model monitoring report can be achieved by maintaining a one-to-one correspondence between the model monitoring report number, model monitoring report name, and database table name. In the verification report template configuration, only the monitoring report number is used as the identifier. During backend logical judgment and data retrieval, the automation tool automatically matches the monitoring report number to the database table name. Furthermore, table mapping relationship maintenance can also include operations such as importing and manually maintaining table information.

[0074] S203. Determine the field mapping relationship based on the model monitoring report number, field name, and data field.

[0075] For example, determining field mapping relationships based on model monitoring report numbers, field names, and data fields can include maintaining a one-to-one correspondence between model monitoring report numbers, field names, and data fields. Furthermore, field mapping relationship maintenance can also perform manual maintenance such as adding, modifying, and deleting field relationships, as well as batch import operations.

[0076] S204. Set parameter groups according to fixed parameters and custom parameters, generate parameter labels according to the parameter groups, and insert the parameter labels into a general template.

[0077] The general template is a Word document, which refers to a standardized report template written in accordance with the requirements for model validation management.

[0078] Since the parameters referenced by different types of general templates may vary, the parameter group maintenance function categorizes parameters into custom parameters and fixed parameters. It sets parameter groups for the templates to be configured and generates parameter labels. During the template editing phase, the template content is written as required, and the generated parameter labels are filled into the corresponding positions in the general template.

[0079] S205. Set the judgment criteria for the model group conclusions to verify the content, generate term tags, and insert the term tags into the general template.

[0080] In some embodiments, the criteria for judging the model group conclusions are set for the verification content, and term tags are generated. Term tags are generated in pairs by the automated tool: one type is a conclusion-based term tag, and the other is a descriptive term tag. The judgment logic for both types of term tags is the same, but the replacement content differs. The criteria for judging the model group conclusions for the verification content can be set and term tags generated through the term page of the automated tool. "Add Term," "Modify Term," and "Delete Term Information" are for basic term maintenance; "View Term Logic" allows previewing term rules; and "Title Maintenance" maintains the first-level and second-level headings of the term.

[0081] The rules for generating term tags include: it consists of several parts: {term category_first-level heading_second-level heading_description / conclusion}. The term category is used to set the uniqueness of the term, the first and second-level headings are used to determine the position of the tag in the template, and the description or conclusion represents the attributes of the term.

[0082] The processing logic for the term tagging includes: the generation of terms and their association with validation time-point parameters. The validation conclusions of a single model at two validation time points will be used to generate the combined conclusion of the single model at both time points (hereinafter referred to as the two-phase conclusion) according to the rules. For cases where the model group has multiple sub-models, the two-phase conclusions of the multiple sub-models need to be merged into the validation conclusion of the model group. The merging of the conclusions of multiple sub-models generates the final model group conclusion according to the rules configured for the terms.

[0083] In some embodiments, the scorecard model validation section in the overall conclusion displays the following: Discrimination capability validation revealed that, for the model with xx, using the AR and KS indices for validation, the AR value of the model {model group name} is {model validation_AR value_description}, and the KS value is {model validation_KS value_description}, both exceeding the thresholds stipulated by validation management, indicating that the model {model group name} possesses discrimination capability {model validation_accuracy_conclusion}. After report instantiation, the corresponding validation content can replace the above term tags.

[0084] S206. Set up the processing logic for extracting data from the model monitoring report to verify the content, generate data extraction labels, and insert the data extraction labels into the general template.

[0085] For example, when configuring data retrieval logic, the first-level heading is the verification category, the second-level heading is the verification content under the verification category, the unique identifier is the table path identifier, and the time type is usually cross-sectional or time-series. You need to enter the logic group name, select the monitoring report number to be queried by the data retrieval logic, and select the first-level heading, second-level heading, time type, and unique identifier to generate corresponding labels for the data retrieval logic.

[0086] On the data retrieval logic configuration page, the "Add Field" function allows you to select the fields to be displayed in the report. You can also adjust the display order of fields in the form by moving them up or down, or delete unnecessary fields. The fields in the form correspond to the table headers in the general template, and may include the reporting period, model name, number of model indicators, total number of indicator missing rates, etc.

[0087] The verification report will display and list the data generated during the verification process in the form of tables or attachments to support the verification conclusions. The data extraction label will be displayed after the report is instantiated, based on the data extracted from the model monitoring report.

[0088] Data retrieval tags can include: <Verification Type_Model Monitoring Report Number_Serial Number> and [Verification Type_Model Monitoring Report Number_Serial Number], etc.

[0089] S207. Based on the correspondence between model number and execution path, distinguish the parts of the general template whose verification content is different.

[0090] In some embodiments, the general template is the union of validation content for a certain type of model, summarizing all validation content for this type of model (if the general template does not cover a particular case, it should be added to the corresponding position in the template). The general template does not specifically distinguish between the same validation content for this type of model; however, it distinguishes between different validation content based on the correspondence between model number and execution path. Specifically:

[0091] The general template includes the following: First, template content applicable to all situations, i.e., the intersection part A, which is considered general; second, parts B and C that differ due to different specific situations, which can include all possible situations in the general template, and select to execute a certain situation through logical judgment.

[0092] Specifically, the step of distinguishing the parts of the general template with different verification content based on the correspondence between model number and execution path includes: inserting a path selection mapping table and a set of verification content at the positions in the general template where the verification content is different, wherein the path selection mapping table contains model number and execution path.

[0093] For validation content of similar models, although the content is highly similar, it is not exactly the same. Path selection can be used to include the requirements of each model. A path selection mapping table can be inserted at the different locations of the validation content, as shown in Table 2.

[0094] Table 2 is a path selection mapping table.

[0095] A Model 1, Model 2, Model 3 B Model a, Model b, Model c

[0096] The path selection mapping table records the correspondence between model numbers and execution paths, thereby allowing different verification content to be selected for different models. The following content can be added to the general template:

[0097] Start marker: <start>

[0098] A Model 1, Model 2, Model 3 B Model a, Model b, Model c

[0099]

[0100] XXX……X

[0101]

[0102]

[0103] YYY……Y

[0104]

[0105] End marker: <end>

[0106] If the instantiated model is one of Model 1, Model 2, or Model 3, it will be displayed XXX……X The content within. For example, if the instantiated model is one of model a, model b, or model c, it will display... YYY……Y The content in [the document / article].

[0107] This invention provides a method for generating model validation reports. By interpreting and analyzing similar model validation reports, a universal template is formed, establishing comprehensive writing standards covering report format, data format, table format, standard wording, and chapter arrangement, significantly improving the standardization of model validation reports. Furthermore, different model validation reports can be instantiated using the universal template in conjunction with configuration information, and by setting tags, the universal template can be reused for different models and different validation time points.

[0108] Figure 3 This is a flowchart illustrating another method for generating a model validation report according to an embodiment of the present invention. This embodiment adds a step of selecting validation content from a set of validation content, based on the above embodiments. Figure 3 As shown, the method includes:

[0109] S301. Based on the model type input by the user, obtain the general template corresponding to the credit risk measurement model of the corresponding type as the target template, wherein the target template contains set labels.

[0110] The target template contains the union of the verification content of this type of model verification, and the verification content is presented in the form of set tags.

[0111] Compared to the validation report, the general template has two key characteristics: First, the validation content in the general template is parameterized. The model group name, validation time point, validation results, and validation conclusions are all presented in the form of specially agreed-upon tags, specifically the following types of tags:

[0112] Parameter labels: {model group name}, {data validation time point 1}, etc.

[0113] Entry tags: {Data Validation_Consistency_Conclusion}, {Data Validation_Consistency_Description}...;

[0114] Data tags: [Retail Rating_4120_1], <Retail Rating_4120_1>...

[0115] Second, the general template is the union of the verification content of this type of model, which summarizes all the verification content of this type of model. The template does not make specific distinctions for the parts of the verification content of this type of model. The parts of the verification content of this type of model are different according to the correspondence between the model number and the execution path.

[0116] S302. Obtain the target data corresponding to the model group and parameter group of the target template.

[0117] S303. When the set label is a term label, for each sub-model in the model group, obtain the verification time point in the target data, and generate a merged conclusion after merging the verification time points according to the verification conclusions corresponding to the verification time points.

[0118] S304. Determine the model group conclusion of the model group according to the set rules based on the merging conclusion of the individual sub-models, and use the model group conclusion as the report content.

[0119] S305. Replace the term tags in the target template with the conclusions of the model group, and execute S315.

[0120] S306. When the set label is a data retrieval label, obtain the sub-model number and verification time point in the target data.

[0121] S307. Generate data retrieval logic based on the data retrieval label, the sub-model number, and the verification time point; obtain report data from the model monitoring report based on the data retrieval logic; and determine the report content based on the report data.

[0122] S308. Determine the label replacement method based on the symbol of the data retrieval label.

[0123] S309. When the label replacement method is to insert a table, the report content is inserted in the form of a table into the position of the data retrieval label in the target template to replace the data retrieval label, and S315 is executed.

[0124] S310. When the label replacement method is a report attachment, the report content is attached to the target template as an attachment, and the corresponding data retrieval label in the target template is replaced by a setting statement, and S315 is executed.

[0125] S311. When the set label is a parameter label, the target data is used as the report content corresponding to the parameter label.

[0126] S312. Replace the parameter label in the target template with the report content, and execute S315.

[0127] S313. Query the path selection mapping table based on the model number in the target data to obtain the target execution path.

[0128] S314. Obtain target verification content from the verification content set according to the target execution path, and replace the path selection mapping table and the verification content set with the target verification content.

[0129] For example, insert the following content in different positions within the template's validation content:

[0130] <start>

[0131] A JRJG_X B Model b

[0132]

[0133] As mentioned above, the team has confirmed that...

[0134]

[0135]

[0136] Table 3 shows the accuracy of the model.

[0137]

[0138]

[0139] <end>

[0140] Assuming the identified model number is JRJG_X, the target execution path A is obtained by querying the path selection mapping table. Based on the target execution path A, the following steps are determined: As mentioned above, the team has confirmed that... The corresponding content is the target verification content, and only the phrase "As mentioned above, after team inquiry and confirmation,..." is retained in this position.

[0141] S315. If all the set labels in the target template have been replaced, output the model verification report.

[0142] This invention provides a method for generating model validation reports. By using a path selection mapping table to distinguish different validation contents in a general template, validation reports for different models and validation time points can be generated through configuration information. This effectively reduces the workload of writing code, editing text, and cross-checking during the report writing process, allowing validation personnel to focus more on the analysis of expert judgment, expert opinions, and key validation conclusions, thereby improving validation efficiency.

[0143] Figure 4 This is a flowchart illustrating another method for generating a model validation report according to an embodiment of the present invention. This embodiment presents a complete automatic model validation report generation process. Figure 4 As shown: The method includes:

[0144] S401, Begin.

[0145] S402, Model Group Configuration.

[0146] The main purpose is to combine one or more sub-models (the monitoring granularity of the monitoring report is a single model) into a model group.

[0147] S403, Template Parameter Maintenance.

[0148] Set parameter groups for the configured template and generate parameter labels.

[0149] S404, Table mapping relationship maintenance.

[0150] Maintain a one-to-one correspondence between the monitoring report number, the monitoring report name, and the name of the base table of the monitoring report.

[0151] S405, Field mapping relationship maintenance.

[0152] Maintain the one-to-one correspondence between monitoring report number, field name, and data field.

[0153] S406, Entry Configuration.

[0154] Set judgment statements for the model group conclusions to verify the content, and generate term tags.

[0155] S407, Data retrieval logic settings.

[0156] To verify the content, data is extracted from monitoring reports, and data tags are generated.

[0157] S408, Template Editing.

[0158] Write the template content as required, and insert the tags generated by the term configuration and data retrieval logic settings into the template's replacement positions.

[0159] S409. Select a template.

[0160] When generating a specific verification report, select the appropriate general template from the published general templates. These published general templates are those approved by the administrator, ensuring the quality of the report templates.

[0161] S410, Input template parameters.

[0162] After selecting the appropriate parameter group, enter the parameter data.

[0163] S411, Execute template.

[0164] Generate a post-deployment validation report for a specific model. For example, for a template uploaded to the template library, you can choose to instantiate it to generate a validation report for a specific model group. After selecting the template, select the model group and parameter group for which you want to generate the report. After selecting the parameter group, enter data such as "Validation Time Point" and "Performance Period". After saving the parameters, you can execute the report generation.

[0165] S412, Query report.

[0166] Template execution records can be queried by template or report name.

[0167] S413, Browse the report.

[0168] S414, Download Report.

[0169] You can download model validation reports and template execution logs. For example, you can view all reports that have completed the report generation step on the template management and execution page, and download the reports to your local computer and view the report logs.

[0170] This invention, through a universal template and pre-set configuration information, can instantiate validation reports for different models, shortening the time required to write model validation reports. Standardized templates and data filling effectively reduce the workload of writing code, editing text, and cross-checking during the report writing process, allowing validation personnel to focus more on expert judgment, expert opinions, and key validation conclusions, thereby improving validation efficiency. It also forms a closed-loop management system for model validation, enabling automatic generation of validation reports for online models and improving model risk management methods.

[0171] Figure 5 This is a structural block diagram of a model validation report generation apparatus provided in an embodiment of the present invention. The apparatus can be implemented by software and / or hardware and can execute the model validation report generation method provided in any embodiment of the present invention. This apparatus is typically configured in an electronic device. Figure 5 As shown, the device includes: a template acquisition module 510, a template execution module 520, and a report verification module 530.

[0172] The template acquisition module 510 is used to acquire a target template corresponding to the type of credit risk measurement model, wherein the target template contains the union of the verification content of the model of this type, and the verification content is presented in the form of a set label.

[0173] The template execution module 520 is used to obtain target data corresponding to the model group and parameter group of the target template, execute the processing logic corresponding to the set label according to the target data, and obtain the report content.

[0174] The report verification module 530 is used to replace the set labels according to the report content and output a model verification report.

[0175] Optionally, the template execution module 520 is specifically used for:

[0176] When the set tag is a term tag, for each sub-model in the model group, the verification time point in the target data is obtained, and a merged conclusion after merging the verification time points is generated based on the verification conclusions corresponding to the verification time points.

[0177] The model group conclusion is determined according to the set rules based on the merging conclusion of the individual sub-models, and the model group conclusion is used as the report content.

[0178] Furthermore, the report verification module 530 is specifically used for:

[0179] The conclusions of the model group are used to replace the term tags in the target template.

[0180] Optionally, the template execution module 520 is also specifically used for:

[0181] When the set label is a data retrieval label, obtain the sub-model number and verification time point in the target data;

[0182] Data retrieval logic is generated based on the data retrieval label, the sub-model number, and the verification time point. Report data is obtained from the model monitoring report based on the data retrieval logic, and the report content is determined based on the report data.

[0183] Furthermore, the report verification module 530 is also specifically used for:

[0184] The label replacement method is determined based on the symbol of the data retrieval label;

[0185] When the label replacement method is to insert a table, the report content is inserted in the form of a table into the position of the data retrieval label in the target template to replace the data retrieval label;

[0186] When the label replacement method is a report attachment, the report content is attached to the target template as an attachment, and the corresponding data retrieval labels in the target template are replaced using a setting statement.

[0187] Optionally, the template execution module 520 is also specifically used for:

[0188] When the set label is a parameter label, the target data is used as the report content corresponding to the parameter label.

[0189] Furthermore, the report verification module 530 is also specifically used for:

[0190] Replace the parameter labels in the target template with the report content.

[0191] Optionally, the template acquisition module 510 is specifically used for:

[0192] Based on the model type input by the user, obtain the general template corresponding to the credit risk measurement model of the corresponding type as the target template.

[0193] Optionally, it also includes a template configuration module, used to configure a general template in the following manner to obtain a general template corresponding to different types of credit risk measurement models:

[0194] Set up model groups based on model type, model name, model number, and number of sub-models;

[0195] The table mapping relationship is determined based on the model monitoring report number, model monitoring report name, and database table name of the model monitoring report;

[0196] Determine the field mapping relationship based on the model monitoring report number, field name, and data field;

[0197] Set parameter groups based on fixed parameters and custom parameters, generate parameter labels based on the parameter groups, and insert the parameter labels into a general template;

[0198] To verify the content, set the judgment criteria for the model group conclusion, generate term tags, and insert the term tags into the general template;

[0199] To verify the content, set up the processing logic for extracting data from the model monitoring report, generate data extraction tags, and insert the data extraction tags into the general template.

[0200] Furthermore, the template configuration module is also used for:

[0201] The different parts of the verification content of the general template are distinguished based on the correspondence between the model number and the execution path.

[0202] Furthermore, the distinction made based on the correspondence between model number and execution path to differentiate the parts of the general template whose verification content differs includes:

[0203] Insert a path selection mapping table and a set of verification contents at the locations where the verification content differs in the general template. The path selection mapping table includes the model number and the execution path.

[0204] Optionally, before outputting the model validation report, the following may also be included:

[0205] The path determination module is used to query the path selection mapping table based on the model number to obtain the target execution path;

[0206] The content replacement module is used to obtain target verification content from the verification content set according to the target execution path, and replace the path selection mapping table and the verification content set with the target verification content.

[0207] The model verification report generation apparatus provided in this embodiment of the invention can execute the model verification report generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0208] Figure 6 This is a structural block diagram of an electronic device provided for an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0209] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0210] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0211] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the method for generating model validation reports.

[0212] In some embodiments, the method for generating a model verification report may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for generating a model verification report described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the method for generating a model verification report by any other suitable means (e.g., by means of firmware).

[0213] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0214] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0215] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0216] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0217] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0218] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements a method for generating a model verification report as provided in any embodiment of this application.

[0219] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0220] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.< / end> < / start> < / end> < / start>

Claims

1. A method for generating a model validation report, characterized in that, include: Obtain the target template corresponding to the type of credit risk measurement model from the general template. The general template is a report template written based on model validation management requirements. The target template contains the union of validation content for this type of model validation, and the validation content is presented in the form of set tags. Obtain the target data corresponding to the model group and parameter group of the target template, execute the processing logic corresponding to the set label according to the target data, and obtain the report content; Replace the set labels according to the report content, and output a model validation report; The step of executing the processing logic corresponding to the set label based on the target data to obtain report content includes: When the set tag is a term tag, for each sub-model in the model group, the verification time point in the target data is obtained, and a merged conclusion after merging the verification time points is generated based on the verification conclusions corresponding to the verification time points. The model group conclusion is determined according to the established rules based on the merging conclusions of individual sub-models, and the model group conclusion is used as the report content; or, When the set label is a data retrieval label, obtain the sub-model number and verification time point in the target data; Data retrieval logic is generated based on the data retrieval label, the sub-model number, and the verification time point. Report data is obtained from the model monitoring report based on the data retrieval logic, and the report content is determined based on the report data.

2. The method according to claim 1, characterized in that, The step of replacing the set label according to the report content includes: The conclusions of the model group are used to replace the term tags in the target template.

3. The method according to claim 1, characterized in that, The step of replacing the set label according to the report content includes: The label replacement method is determined based on the symbol of the data retrieval label; When the label replacement method is to insert a table, the report content is inserted in the form of a table into the position of the data retrieval label in the target template to replace the data retrieval label; When the label replacement method is a report attachment, the report content is attached to the target template as an attachment, and the corresponding data retrieval labels in the target template are replaced using a setting statement.

4. The method according to claim 1, characterized in that, The step of executing the processing logic corresponding to the set label based on the target data to obtain report content includes: When the set label is a parameter label, the target data is used as the report content corresponding to the parameter label.

5. The method according to claim 4, characterized in that, The step of replacing the set label according to the report content includes: Replace the parameter labels in the target template with the report content.

6. The method according to claim 1, characterized in that, The acquisition of the target template corresponding to the type of credit risk measurement model includes: Based on the model type input by the user, obtain the general template corresponding to the credit risk measurement model of the corresponding type as the target template.

7. The method according to any one of claims 1-6, characterized in that, This also includes configuring a general template in the following way to obtain a general template corresponding to different types of credit risk measurement models: Set up model groups based on model type, model name, model number, and number of sub-models; The table mapping relationship is determined based on the model monitoring report number, model monitoring report name, and database table name of the model monitoring report; Determine the field mapping relationship based on the model monitoring report number, field name, and data field; Set parameter groups based on fixed parameters and custom parameters, generate parameter labels based on the parameter groups, and insert the parameter labels into a general template; To verify the content, set the judgment criteria for the model group conclusion, generate term tags, and insert the term tags into the general template; To verify the content, set up the processing logic for extracting data from the model monitoring report, generate data extraction tags, and insert the data extraction tags into the general template.

8. The method according to claim 7, characterized in that, Also includes: The different parts of the verification content of the general template are distinguished based on the correspondence between the model number and the execution path.

9. The method according to claim 8, characterized in that, The method of distinguishing the different verification content of the general template based on the correspondence between model number and execution path includes: Insert a path selection mapping table and a set of verification contents at the locations where the verification content differs in the general template. The path selection mapping table includes the model number and the execution path.

10. The method according to claim 9, characterized in that, Before outputting the model validation report, the following is also included: The target execution path is obtained by querying the path selection mapping table based on the model number. The target verification content is obtained from the verification content set according to the target execution path, and the target verification content is used to replace the path selection mapping table and the verification content set.

11. An apparatus for generating a model validation report, characterized in that, include: The template acquisition module is used to acquire a target template corresponding to the type of credit risk measurement model from a general template. The general template is a report template written based on model validation management requirements. The target template contains the union of validation content for this type of model validation, and the validation content is presented in the form of set tags. The template execution module is used to obtain target data corresponding to the model group and parameter group of the target template, execute the processing logic corresponding to the set label according to the target data, and obtain the report content. The report verification module is used to replace the set labels according to the report content and output a model verification report; The template execution module is specifically used for: When the set tag is a term tag, for each sub-model in the model group, the verification time point in the target data is obtained, and a merged conclusion after merging the verification time points is generated based on the verification conclusions corresponding to the verification time points. The model group conclusion is determined according to the set rules based on the merging conclusion of the individual sub-models, and the model group conclusion is used as the report content. The template execution module is also specifically used for: When the set label is a data retrieval label, obtain the sub-model number and verification time point in the target data; Data retrieval logic is generated based on the data retrieval label, the sub-model number, and the verification time point. Report data is obtained from the model monitoring report based on the data retrieval logic, and the report content is determined based on the report data.

12. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements a method for generating a model verification report as described in any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the method for generating a model validation report as described in any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for generating a model verification report as described in any one of claims 1 to 10.

Images