A model verification conclusion generation method and device, equipment, medium and product

Abstract

This invention discloses a method, apparatus, device, medium, and product for generating model validation conclusions. This invention relates to the fields of big data and artificial intelligence technologies. The method includes: obtaining validation indicators for a customer rating model; combining target validation indicators from the validation indicators according to the data format of training samples of a pre-trained classification model to obtain combined indicators; and inputting the remaining indicators and combined indicators from the validation indicators into a classification model to obtain model validation conclusions generated by the classification model. The technical solution of this invention, by combining target validation indicators from the validation indicators to obtain combined indicators and inputting the remaining indicators and combined indicators from the validation indicators into a classification model to obtain model validation conclusions generated by the classification model, can solve the problem of subjective influence on the generation of model validation conclusions due to human intervention in the prior art, and can generate model validation conclusions more objectively and accurately.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of big data and artificial intelligence technology, and in particular to a method, apparatus, device, medium and product for generating model verification conclusions. Background Technology

[0002] In customer rating models, it's necessary to monitor the model used. This monitoring process accumulates numerous model validation metrics. Different model validation metrics can be used to evaluate the stability and other capabilities of the model's predictive power. By evaluating the corresponding model capabilities based on different model validation metrics, we can obtain conclusions about the model under those specific metrics.

[0003] In existing technologies, determining the final conclusion of model validation based on multiple customer rating model validation indicators relies on personal experience. This manual judgment is highly subjective and leads to excessive workload when dealing with a large number of models. Furthermore, the periodic execution of data processing scripts can cause delays in the generation of model validation indicators, resulting in a lag in the determination of current model conclusions. Moreover, current calculation-based validation indicator values ​​can only determine the final conclusion of the model in the current period, lacking forward-looking prediction of model conclusions and the ability to anticipate future model risks, thus failing to provide effective guidance for business personnel.

[0004] Therefore, how to objectively and accurately generate model validation conclusions has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a method, apparatus, device, system, medium, and program product for generating model validation conclusions, in order to solve the problem that the subjective influence of human intervention on the generation of model validation conclusions in the prior art, and to generate model validation conclusions more objectively and accurately.

[0006] In a first aspect, embodiments of the present invention provide a method for generating model validation conclusions, comprising:

[0007] Obtain validation metrics for the customer rating model;

[0008] According to the data format of the training samples of the pre-trained classification model, the target verification indicators in the verification indicators are combined to obtain the combined indicators. The classification model is trained by the historical verification indicators in the training samples, and the training of the classification model is supervised by the historical model verification conclusions in the training samples.

[0009] The remaining indicators and the combined indicators from the validation metrics are input into the classification model to obtain the model validation conclusion generated by the classification model.

[0010] Secondly, embodiments of the present invention also provide a model validation conclusion generation apparatus, the apparatus comprising:

[0011] The metrics acquisition module is used to acquire validation metrics for the customer rating model.

[0012] The indicator combination module is used to combine the target verification indicators in the verification indicators according to the data format of the training samples of the pre-trained classification model to obtain the combined indicators. The classification model is trained by the historical verification indicators in the training samples, and the training of the classification model is supervised by the historical model verification conclusions in the training samples.

[0013] The conclusion acquisition module is used to input the remaining indicators and the combined indicators from the verification indicators into the classification model to obtain the model verification conclusion generated by the classification model.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program on the memory and executable on the processor, wherein the processor executes the program to implement a method for generating model verification conclusions as described in any of the embodiments of the present invention.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for generating model verification conclusions as described in any of the embodiments of the present invention.

[0016] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements a method for generating model verification conclusions as described in any of the embodiments of the present invention.

[0017] In this embodiment of the invention, a combined index is obtained by combining the target verification index in the verification index; the remaining index and the combined index in the verification index are input into the classification model to obtain the model verification conclusion generated by the classification model. This can solve the problem of the subjective influence of human intervention on the generated model verification conclusion in the prior art, and can generate model verification conclusion more objectively and accurately. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1A flowchart illustrating a method for generating model validation conclusions according to an embodiment of the present invention;

[0020] Figure 2 A flowchart illustrating another method for generating model verification conclusions provided in an embodiment of the present invention;

[0021] Figure 3 A flowchart illustrating another method for generating model verification conclusions provided in an embodiment of the present invention;

[0022] Figure 4 A flowchart illustrating another method for generating model verification conclusions provided in an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of a device for generating model verification conclusions provided in an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of an electronic device for generating model verification conclusions according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0027] Existing customer rating models require monitoring, particularly those used for large and medium-sized customers. This monitoring process accumulates numerous model validation metrics. For example, the KS (Kolmogorov-Smirnov, Lorenz curve) metric is used to assess the model's risk discrimination ability, the PSI (population stability index) measures the deviation between predicted and actual values, evaluating the model's predictive stability, and other metrics are used to assess other aspects of the model's capabilities. Each metric evaluates the corresponding model capability, leading to conclusions about the model under that specific metric.

[0028] The general steps for validating model conclusions based on multiple indicators are as follows:

[0029] 1) Obtain the calculation results of different model validation indicators and store each result data in a structured database;

[0030] 2) Establish the relationship between different validation metrics and model conclusions;

[0031] 3) Experienced business personnel synthesize the model conclusions based on the relationship between different verification indicators and the model.

[0032] However, the determination of the above model validation conclusions relies on personal experience, which is too subjective in the process of manual judgment, and the workload becomes too large when there are too many models. In addition, since the data processing scripts are run periodically, the model validation indicators may not be generated in a timely manner, causing a delay in the determination of the current model conclusions. Moreover, the current calculation-based validation indicator values ​​can only determine the final conclusion of the model in the current period, without forward-looking prediction of the model conclusions, making it impossible to predict future model risks in advance, and failing to provide good guidance for business personnel.

[0033] To address the aforementioned technical problems, this application proposes the following technical concept: Obtaining validation metrics for a customer rating model; combining the target validation metrics with the training samples of a pre-trained classification model to obtain combined metrics, wherein the classification model is trained using historical validation metrics from the training samples, and the training of the classification model is supervised by historical model validation conclusions from the training samples; inputting the remaining metrics and combined metrics from the validation metrics into the classification model to obtain the model validation conclusions generated by the classification model. This solves the problem of subjective influence on the generated model validation conclusions due to human intervention in existing technologies, enabling the generation of more objective and accurate model validation conclusions.

[0034] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0035] Figure 1 This is a flowchart illustrating a method for generating model validation conclusions according to an embodiment of the present invention. This embodiment is applicable to generating validation conclusions for customer rating models. The method can be executed by a model validation conclusion generation device, which can be implemented in hardware and / or software. This device can be configured in an electronic device capable of generating model validation conclusions. Figure 1 As shown, the method includes:

[0036] S110. Obtain validation metrics for the customer rating model.

[0037] Customer rating models can be a method for assessing and classifying the credit risk levels of corporate, institutional, or individual customers. This model can comprehensively assess a customer's credit risk by analyzing various indicators such as their financial situation, operational capabilities, industry prospects, market competitiveness, and repayment history, and classify customers into different levels to facilitate risk management and decision-making by banks, financial institutions, or other commercial entities.

[0038] Among them, the validation metrics for the customer rating model can be indicators used to evaluate the predictive accuracy and reliability of the model, so as to provide a quantitative measure of the model's performance.

[0039] In this embodiment, the obtained verification indicators can be determined based on specific business needs and risk management objectives.

[0040] For example, obtaining validation metrics for a customer rating model may include: running a statistical analysis based on the customer rating model using a script to obtain the validation metrics for the current period.

[0041] The current period validation metrics can be the results obtained by periodically running data from a pre-defined script retrieved from the database. For example, the current period validation metrics can be the current period validation metrics data obtained through a pre-defined script at a specified time. This embodiment does not specifically limit the specified time. For example, assuming that the validation metrics for the customer rating model are generated monthly by running a pre-defined script, then the validation metrics for the customer rating model generated by running the pre-defined script in the current month are the current period validation metrics.

[0042] Specifically, by running the customer rating model through the set script, the current validation metrics can be calculated. The current validation metrics can be AR (accuracy rate, used to characterize the model's discriminative ability), KS (Kolmogorov-Smirnov curve), CIER (Conditional Information Entropy Ratio, uncertainty measure), and TM (Transition Matrix), etc.

[0043] In this embodiment, when obtaining the validation metrics of the customer rating model, the validation metrics can be obtained based on the model ID of the customer rating model. Specifically, the model ID can be determined based on the internal number of the model; for example, the model ID could be M10. This embodiment does not impose any specific limitations on this.

[0044] S120. According to the data format of the training samples of the pre-trained classification model, the target verification indicators in the verification indicators are combined to obtain the combined indicators. Among them, the classification model is trained by the historical verification indicators in the training samples, and the training of the classification model is supervised by the historical model verification conclusions in the training samples.

[0045] Pre-training can involve using historical validation metrics from training samples to train a weak learner in a classification model, and then using historical model validation conclusions to supervise the model's prediction results.

[0046] The classification model can be a binary classification machine learning model, which is a commonly used model in machine learning that can classify data into different categories. Common classification models include, but are not limited to, logistic regression models, decision tree models, support vector machine models, Naive Bayes models, random forest models, and gradient boosting tree models.

[0047] Among them, the data form can be understood as the way the target verification indicators are organized in the verification indicators; the combined indicators can be understood as the combined indicators obtained by performing mathematical operations on the target verification indicators in the verification indicators.

[0048] For example, according to the data format of the training samples of the pre-trained classification model, the target verification indicators in the verification indicators are combined to obtain the combined indicators. This may include: obtaining the data organization strategy corresponding to the data format of the training samples of the classification model, wherein the data organization strategy is used to set and calculate the target historical verification indicators in the training samples; and setting and calculating the target verification indicators in the current verification indicators according to the data organization strategy to obtain the combined indicators.

[0049] Among them, the data organization strategy can be used to indicate the way indicators are organized, that is, the type of operation used for specific target verification indicators in the verification indicators.

[0050] Specifically, the target historical verification index participating in the setting operation can be at least one, and this embodiment does not specifically limit it.

[0051] In this embodiment, the set operation method can be mathematical operations such as addition, subtraction, and averaging, or logical operations such as AND, OR, and NOT. This embodiment does not specifically limit this.

[0052] For example, setting and calculating the target verification indicators in the current verification indicators according to the data organization strategy to obtain the combined indicators may include: determining the calculation method and the target verification indicators for specific calculations in the current verification indicators according to the data organization strategy; and calculating the target verification indicators according to the calculation method to obtain the combined indicators.

[0053] In this embodiment, the calculation method and target verification indicators can be determined according to the data organization strategy, and the target verification indicators can be combined according to the calculation method to obtain the combined indicators.

[0054] This embodiment ensures that the attribute characteristics of the input data and the training data are consistent. For example, if historical model validation metric 'a' is processed by A during training, then the current validation metric 'a' also needs to be processed by A. Specifically, if historical model validation metric 'b' and historical model validation metric 'c' are added together and then input into the model during model training, then the current model validation metric 'b' and current validation metric 'c' also need to be added together.

[0055] S130. Input the remaining indicators and combined indicators from the validation indicators into the classification model and obtain the model validation conclusion generated by the classification model.

[0056] The remaining indicators can be verification indicators other than the target verification indicators.

[0057] In this embodiment, the remaining indicators and combined indicators from the validation metrics are input into the classification model to obtain the predicted model validation conclusions.

[0058] For example, inputting the remaining indicators and combined indicators from the validation metrics into the classification model to obtain the model validation conclusion generated by the classification model may include: obtaining the remaining indicators other than the target validation indicator in the current validation metrics; inputting the remaining indicators and combined indicators into the classification model, dividing the remaining indicators and combined indicators into N datasets through the classification model, and inputting the N datasets into N weak learners respectively, and generating the current model validation conclusion based on the voting results of the N weak classifiers.

[0059] In this embodiment, the remaining indicators and combined indicators in the validation metrics can be input into the random forest model. The random forest model can be an ensemble learning method based on multiple decision trees, which can be applied to multi-classification problems and regression problems.

[0060] Specifically, the random forest model can aggregate decision tree models based on the random forest algorithm. The random forest model can divide the remaining indicators and combined indicators into N datasets, input the N datasets into N decision tree models, and combine the N decision tree models to form the final current model validation conclusion.

[0061] For example, the current model validation conclusion can be divided into two cases: normal and requiring improvement. Specifically, the model validation conclusion can be determined based on the probability of being normal and the probability of requiring improvement in the predicted model validation conclusion. If the probability of the model validation conclusion being normal is greater than a set first threshold, then the model validation conclusion is determined to be normal. The first threshold is a manually set threshold and can be dynamically adjusted according to the specific customer rating model. For example, different first thresholds can be set for individual customer rating models and enterprise customer rating models. Alternatively, different first thresholds can be set for different types of models within the individual customer rating model; this invention does not impose specific limitations. If the probability of the model validation conclusion requiring improvement is greater than a set second threshold, then the model validation conclusion is determined to require improvement. The setting of the second threshold is similar to the first threshold, both being manually set thresholds, and will not be elaborated further here.

[0062] In this embodiment of the invention, a combined index is obtained by combining the target verification index among the verification indices; the remaining indexes and the combined index are input into the classification model to obtain the model verification conclusion generated by the classification model. This can solve the problem of the subjective influence of human intervention on the generated model verification conclusion in the prior art, and can generate model verification conclusions more objectively and accurately.

[0063] Figure 2 The flowchart illustrates another method for generating model validation conclusions according to an embodiment of the present invention. Building upon the above embodiments, it further includes training a classification model in the following manner: obtaining a historical validation index dataset, wherein the historical validation index dataset includes historical validation indices for a set number of periods; obtaining historical model validation conclusions corresponding to each period's historical validation indices; and performing supervised training of the classification model based on the historical validation index dataset and the historical model validation conclusions. Terms identical to those in the above embodiments are not repeated here. Figure 2 As shown, the method for generating the model validation conclusions includes the following steps:

[0064] S210. Obtain the historical verification indicator dataset, which includes historical verification indicators for a set number of periods.

[0065] The historical verification indicator dataset may include at least one historical verification indicator, but this embodiment does not specifically limit it.

[0066] In this embodiment, when obtaining the historical verification indicator dataset, the historical verification indicator dataset can be obtained through the model ID of the customer rating model; wherein, the historical verification indicator dataset can be a set containing at least one historical verification indicator.

[0067] In this embodiment, the script performs periodic data runs based on a customer rating model, and the resulting historical verification indicators are stored in a database. Historical verification indicators for a set number of periods can be retrieved from the database. This embodiment allows for setting the data run cycle as needed. Specifically, for individual customers, the data run cycle can be set in a fine-grained manner, for example, once a week; for enterprise customers, the data run cycle can be set in a coarse-grained manner, for example, once a month.

[0068] For example, obtaining a historical validation indicator dataset may include: obtaining historical validation indicators for a set number of periods; for each period of historical validation indicators, obtaining a target historical validation indicator, and performing a set operation on the target historical validation indicator to obtain a historical combined indicator; and generating a historical validation indicator dataset based on the remaining historical validation indicators and historical combined indicators for each period.

[0069] For example, for each historical validation metric, obtaining the target historical validation metric and performing setting operations on the target historical validation metric to obtain the historical combined metric may include: obtaining the data organization strategy corresponding to the data format of the training samples of the classification model, wherein the data organization strategy is used to perform setting operations on the target historical validation metric in the training samples; and performing setting operations on the target historical validation metric in the historical validation metric according to the data organization strategy to obtain the historical combined metric.

[0070] For example, setting and calculating the target historical verification indicator in the historical verification indicators according to the data organization strategy to obtain the historical combined indicator may include: determining the calculation method and the target historical verification indicator for specific calculation in the historical verification indicators according to the data organization strategy; and calculating the target historical verification indicator according to the calculation method to obtain the historical combined indicator.

[0071] This embodiment ensures that the attribute characteristics of the input data are consistent with those of the training data. For example, if historical model validation metric 'a' is processed by A during training, then historical validation metric 'a' also needs to be processed by A. Specifically, if historical model validation metric 'b' and historical model validation metric 'c' are added together and then input into the model during model training, then historical model validation metric 'b' and historical validation metric 'c' also need to be added together.

[0072] S220. Obtain the historical model validation conclusions corresponding to the historical validation indicators for each period, and conduct supervised training of the classification model based on the historical validation indicator dataset and the historical model validation conclusions.

[0073] This embodiment can compare the predicted model validation results with the historical model validation results in the training samples, calculate the corresponding loss function based on the comparison results, and perform supervised training based on the loss function until the loss function meets the training termination condition, thereby completing the supervised training of the classification model.

[0074] For example, after obtaining the historical model validation conclusions corresponding to each historical indicator, N sample datasets can be generated based on the historical validation indicator dataset. The N sample datasets are then input into N weak classifiers. Each weak classifier generates a prediction result based on the corresponding sample dataset. The model validation prediction conclusion is generated by voting based on the prediction results of each weak classifier. The weak classifiers are then trained under supervision based on the historical model validation conclusions and the model validation prediction conclusions to obtain the classification model.

[0075] In this embodiment, after obtaining the historical model validation conclusions and the model validation prediction conclusions, it is necessary to construct a binary classification machine learning model for each model based on the historical model validation conclusions and the model validation prediction conclusions.

[0076] It is important to emphasize that a classification model can be trained for different model IDs.

[0077] In this implementation, the historical validation index dataset can be divided into N sample datasets, which are then input into N decision tree models. The N decision tree models are then combined to generate model validation prediction conclusions based on the validation results through voting. Supervised training of each decision tree model is performed based on the model prediction validation conclusions and the historical model validation conclusions to obtain a random forest model.

[0078] The embodiments of the present invention use historical validation index datasets and historical model validation conclusions to supervise the training of classification models, which can make the conclusions obtained by the classification models more objective when used for model validation.

[0079] Figure 3 This is a flowchart illustrating another method for generating model validation conclusions according to an embodiment of the present invention. Based on the above embodiments, it further defines the "obtaining validation indicators for the customer rating model." Terms identical to those in the above embodiments are not repeated here. Figure 3 As shown, the method for generating the model validation conclusions includes the following steps:

[0080] S310. Obtain the historical validation indicator dataset for the customer rating model.

[0081] S320. Generate time series datasets for each historical validation indicator based on the historical validation indicator dataset.

[0082] The time series dataset can be a time series dataset of historical validation metrics for each model, such as the AR dataset, KS dataset, CIER dataset, and TM dataset for model ID M10. This embodiment does not make specific limitations on this.

[0083] For example, generating time series datasets for each historical validation indicator based on the historical validation indicator dataset may include: sorting each historical validation indicator according to its indicator type and period number in the historical validation indicator dataset to obtain the time series datasets for each historical validation indicator.

[0084] In this embodiment, the historical verification indicators can be sorted according to their acquisition time order based on their indicator type and period number to obtain the time series dataset of each historical verification indicator.

[0085] S330. Based on the time series dataset of each historical validation indicator, generate the corresponding validation prediction indicator as the validation indicator for the customer rating model.

[0086] For example, generating a validation prediction index for a corresponding validation index based on the time series dataset of each historical validation index may include: using a time series algorithm to generate a validation prediction index for each historical validation index for the next m periods based on the time series dataset of each historical validation index.

[0087] In this embodiment, time series algorithms can be used to predict the validation prediction indicators for the next m periods based on the time series datasets of each historical validation indicator.

[0088] Specifically, based on the time series dataset of 108 historical validation indicators from 201501 to 202301, we can predict the data for three periods: 202302, 202303, and 202304. For each historical validation indicator, we can predict its validation prediction indicator for the next three periods.

[0089] S340. Obtain the data organization strategy corresponding to the data format of the training samples of the classification model, wherein the data organization strategy is used to set and calculate the target historical verification index in the training samples.

[0090] This embodiment can combine verification and prediction indicators according to data organization strategies.

[0091] S350. Based on the data organization strategy, the target verification indicators in the verification prediction indicators are set and calculated to obtain the combined indicators.

[0092] This embodiment can determine the calculation method and target verification indicators according to the data organization strategy, and combine the target verification indicators according to the calculation method.

[0093] For example, setting and calculating the target verification indicator in the verification prediction indicator according to the data organization strategy to obtain the combined indicator may include: determining the calculation method and the target verification indicator for specific calculation in the verification prediction indicator according to the data organization strategy; and calculating the target verification indicator according to the calculation method to obtain the combined indicator.

[0094] S360. For any period in the verification prediction indicators over the next m periods, obtain the remaining indicators in the verification prediction indicators excluding the target verification indicator.

[0095] The remaining indicators can be verification indicators other than the target verification indicators.

[0096] S370. Input the remaining indicators and combined indicators into the classification model. Divide the remaining indicators and combined indicators into N datasets through the classification model. Input the N datasets into N weak learners respectively. Generate future model validation conclusions based on the voting results of the N weak classifiers.

[0097] In this embodiment, the remaining indicators and combined indicators can be input into the random forest model. The random forest model can be an ensemble learning method based on multiple decision trees, which can be applied to multi-classification problems and regression problems.

[0098] Specifically, the random forest model can aggregate decision tree models based on the random forest algorithm. The random forest model can divide the remaining indicators and combined indicators into N datasets, input the N datasets into N decision tree models, and combine the N decision tree models to form future model validation conclusions.

[0099] Specifically, data from three periods—202302, 202303, and 202304—can be input into a trained random forest model, and the model's validation conclusions for the next three periods can be output.

[0100] This invention, through predicting future model validation results based on time series algorithms and classification models, can help business personnel understand the future state of the model, avoid model risks, and further improve the ability to prevent model risks.

[0101] Figure 4 This is a flowchart illustrating another method for generating model verification conclusions according to an embodiment of the present invention. This embodiment can specifically limit the above embodiments. For example... Figure 4 As shown, the method for generating the model validation conclusions also includes the following steps:

[0102] S410. Obtain the historical verification indicator dataset.

[0103] The historical validation metric dataset can be obtained from a database and may include historical validation data such as AR, KS, CIER, and TM.

[0104] S420. Obtain the historical model validation conclusions corresponding to the historical validation indicator dataset.

[0105] In a specific embodiment, the historical model verification conclusions can be determined according to Table 1 below.

[0106] Table 1. Conclusions of Historical Model Validation

[0107] time Model ID AR KS CIER TM Historical model validation conclusions 2015 / 01 M10 0.75 0.63 0.233 90.69％ ... normal 2015 / 02 M10 0.88 0.80 0.195 92.20％ ... Improvement is needed 2015 / 03 M10 0.70 0.58 0.453 90.31％ ... normal 2015 / 04 M10 0.66 0.52 0.275 89.10％ ... normal 2015 / 05 M10 0.90 0.94 0.384 96.19％ ... normal ... ... ... ... ... ... ...

[0108] The table above can be understood as obtaining data for each month of 2015 for model ID M10, setting and calculating the target historical verification indicators in the historical verification indicators according to the data organization strategy, obtaining historical combination indicators, and obtaining the verification conclusions of the historical model corresponding to each period's historical verification indicators.

[0109] S430. Train the random forest model based on the historical validation index dataset and the historical model validation conclusions.

[0110] In this embodiment, since the historical model's overall conclusion has two possibilities: normal and needing improvement, a binary classification machine learning model is required for training. This machine learning model can be a random forest model. After obtaining the historical model validation index dataset and the historical model's validation conclusion, a binary classification random forest model is constructed for each model based on the above data. The historical model's validation conclusion of "normal" is mapped to 1, and "needing improvement" is mapped to 0. The input to the classification model is the historical model validation index data such as AR, KS, CIER, and TM, and the output of the classification model is the probability (between 0 and 1) of the historical model's validation conclusion being judged as "normal". It should be noted that a binary classification machine learning model is trained for each different model ID.

[0111] S440. Obtain the current model validation index data and predict the model conclusions.

[0112] In this embodiment, the predicted model conclusion can be obtained by acquiring the current model validation index data, and combining the target validation index in the validation index according to the data format of the training samples of the pre-trained classification model to obtain the combined index. Then, the combined index and the remaining index are input into the random forest model. The remaining index and the combined index are divided into N datasets by the classification model, and the N datasets are input into N weak learners respectively. The current model validation conclusion is generated based on the voting results of the N weak classifiers.

[0113] The predictive model conclusions generated by the embodiments of this invention avoid the influence of subjectivity caused by human intervention, making the final conclusions of model validation more objective. Furthermore, when there are a large number of models, model conclusions can be generated quickly, reducing workload. A model conclusion prediction method based on historical model validation index data is also designed, proactively predicting model conclusions and helping business personnel to understand the model's status in advance, mitigate model risks, and further enhance model risk prevention capabilities.

[0114] Figure 5 This is a schematic diagram of a model validation conclusion generation device provided in an embodiment of the present invention. This device can execute the model validation conclusion generation method provided in this embodiment to generate more objective and accurate model validation conclusions. The device is configured in an electronic device, such as a server or server cluster. Figure 5 As shown, the device includes: an indicator acquisition module 510, an indicator combination module 520, and a conclusion acquisition module 530;

[0115] Among them, the indicator acquisition module 510 is used to acquire the validation indicators of the customer rating model;

[0116] The indicator combination module 520 is used to combine the target verification indicators in the verification indicators according to the data format of the training samples of the pre-trained classification model to obtain the combined indicators. The classification model is trained by the historical verification indicators in the training samples, and the training of the classification model is supervised by the historical model verification conclusions in the training samples.

[0117] The conclusion acquisition module 530 is used to input the remaining indicators and the combined indicators in the verification indicators into the classification model to obtain the model verification conclusion generated by the classification model.

[0118] The model validation conclusion generation device provided in this embodiment of the invention is configured to implement the model validation conclusion generation method. The implementation principle and technical effect of the model validation conclusion generation device are similar to those of the model validation conclusion generation method, and will not be repeated here.

[0119] Furthermore, the indicator acquisition module 510 is specifically used for:

[0120] By setting a script to run data based on the customer rating model, the current period's verification indicators can be obtained.

[0121] Furthermore, the indicator combination module 520 includes:

[0122] A data organization strategy acquisition unit is used to acquire the data organization strategy corresponding to the data format of the training samples of the classification model, wherein the data organization strategy is used to set and calculate the target historical verification index in the training samples.

[0123] The calculation unit is used to perform calculations on the target verification indicators in the current verification indicators according to the data organization strategy to obtain combined indicators.

[0124] Furthermore, the arithmetic unit is specifically used for:

[0125] Based on the data organization strategy, determine the calculation method and the target verification indicator for specific calculations in the current verification indicator;

[0126] The target verification indicators are calculated according to the aforementioned calculation method to obtain combined indicators.

[0127] Furthermore, the conclusion acquisition module 530 is specifically used for:

[0128] Obtain the remaining indicators from the current verification indicators, excluding the target verification indicator;

[0129] The remaining indicators and combined indicators are input into the classification model, which divides them into N datasets. The N datasets are then input into N weak learners, and the model validation conclusion for the current period is generated based on the voting results of the N weak classifiers.

[0130] Furthermore, based on the above embodiments, the classification model is also trained using the following modules:

[0131] The historical verification indicator acquisition module is used to acquire a historical verification indicator dataset, wherein the historical verification indicator dataset includes historical verification indicators for a set number of periods.

[0132] The training module is used to obtain the historical model validation conclusions corresponding to the historical validation indicators in each period, and to perform supervised training on the classification model based on the historical validation indicator dataset and the historical model validation conclusions.

[0133] Furthermore, the acquisition of historical verification indicators is specifically used for:

[0134] Obtain historical verification indicators for a set number of periods;

[0135] For each period's historical verification indicator, obtain the target historical verification indicator, and perform a setting calculation on the target historical verification indicator to obtain the historical combination indicator;

[0136] A historical validation index dataset is generated based on the remaining historical validation indices and historical combined indices for each period.

[0137] Furthermore, the indicator acquisition module 510 includes:

[0138] The acquisition unit is used to acquire the historical validation indicator dataset of the customer rating model.

[0139] The generation unit is used to generate time series datasets for each historical verification indicator based on the historical verification indicator dataset.

[0140] The prediction unit is used to generate corresponding verification prediction indicators based on the time series dataset of each historical verification indicator, which serve as the verification indicators for the customer rating model.

[0141] Furthermore, the generation unit is specifically used for:

[0142] Based on the indicator type and period number of each historical verification indicator in the historical verification indicator dataset, the historical verification indicators are sorted to obtain the time series dataset of each historical verification indicator.

[0143] Furthermore, the prediction unit is specifically used for:

[0144] Using time series algorithms, based on the time series datasets of each historical validation indicator, we generate validation prediction indicators for each historical validation indicator in the next m periods.

[0145] Furthermore, the indicator combination module is specifically used for:

[0146] The data organization strategy corresponding to the data format of the training samples of the classification model is obtained, wherein the data organization strategy is used to set and calculate the target historical verification index in the training samples.

[0147] Based on the data organization strategy, the target verification indicators in the verification prediction indicators are set and calculated to obtain the combined indicators.

[0148] Furthermore, the indicator combination module is specifically used for:

[0149] Based on the data organization strategy, determine the calculation method and the target verification index for specific calculations in the verification and prediction index;

[0150] The target verification indicators are calculated according to the aforementioned calculation method to obtain combined indicators.

[0151] Furthermore, the conclusion acquisition module is specifically used for:

[0152] For any period in the verification prediction index over the next m periods, obtain the remaining indicators in the verification prediction index excluding the target verification indicator.

[0153] The remaining indicators and combined indicators are input into the classification model, which divides them into N datasets. The N datasets are then input into N weak learners, and future model validation conclusions are generated based on the voting results of the N weak classifiers.

[0154] The model verification conclusion generation device provided in the embodiments of the present invention can execute the steps of the model verification conclusion generation method provided in the method embodiments of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0155] Figure 6 This is a schematic diagram of an electronic device provided for an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0156] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0157] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0158] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a method for generating model validation conclusions.

[0159] In some embodiments, a method for generating model verification conclusions may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for generating model verification conclusions described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform a method for generating model verification conclusions by any other suitable means (e.g., by means of firmware).

[0160] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0161] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other means of generating verification results for a programmable model, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0162] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0164] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0165] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0166] Optionally, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements a method for generating model verification conclusions as provided in any embodiment of this application.

[0167] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0169] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating model validation conclusions, characterized in that, include: Obtain validation metrics for the customer rating model; According to the data format of the training samples of the pre-trained classification model, the target verification indicators in the verification indicators are combined to obtain the combined indicators. The classification model is trained by the historical verification indicators in the training samples, and the training of the classification model is supervised by the historical model verification conclusions in the training samples. Input the remaining indicators and the combined indicators from the validation indicators into the classification model to obtain the model validation conclusion generated by the classification model. The step of combining the target validation metrics in the validation metrics according to the data format of the training samples of the pre-trained classification model to obtain the combined metrics includes: The data organization strategy corresponding to the data format of the training samples of the classification model is obtained, wherein the data organization strategy is used to set and calculate the target historical verification index in the training samples. Based on the data organization strategy, the target verification indicators in the current verification indicators are set and calculated to obtain combined indicators; The step of inputting the remaining indicators and the combined indicators from the validation indicators into the classification model to obtain the model validation conclusion generated by the classification model includes: Obtain the remaining indicators from the current verification indicators, excluding the target verification indicator; The remaining indicators and combined indicators are input into the classification model, which divides the remaining indicators and combined indicators into N datasets. The N datasets are then input into N weak learners, and the model validation conclusion for the current period is generated based on the voting results of the N weak learners. The method also includes training the classification model in the following manner: Obtain a historical validation indicator dataset, wherein the historical validation indicator dataset includes historical validation indicators for a set number of periods; Obtain the historical model validation conclusions corresponding to the historical validation indicators for each period, and perform supervised training on the classification model based on the historical validation indicator dataset and the historical model validation conclusions. The process of obtaining the historical verification indicator dataset includes: Obtain historical verification indicators for a set number of periods; For each period's historical verification indicator, obtain the target historical verification indicator, and perform a setting calculation on the target historical verification indicator to obtain the historical combination indicator; A historical validation index dataset is generated based on the remaining historical validation indices and historical combined indices for each period.

2. The method according to claim 1, characterized in that, The validation metrics for obtaining the customer rating model include: By setting a script to run data based on the customer rating model, the current period's verification indicators can be obtained.

3. The method according to claim 1, characterized in that, The step of setting and calculating the target verification indicators in the current verification indicators according to the data organization strategy to obtain combined indicators includes: Based on the data organization strategy, determine the calculation method and the target verification indicator for specific calculations in the current verification indicator; The target verification indicators are calculated according to the aforementioned calculation method to obtain combined indicators.

4. The method according to claim 1, characterized in that, The validation metrics for obtaining the customer rating model include: Obtain the historical validation index dataset of the customer rating model; Generate time series datasets for each historical validation indicator based on the historical validation indicator dataset; Based on the time series datasets of each historical validation indicator, validation prediction indicators are generated for the corresponding validation indicators, which serve as validation indicators for the customer rating model.

5. The method according to claim 4, characterized in that, The step of generating time series datasets for each historical validation indicator based on the historical validation indicator dataset includes: Based on the indicator type and period number of each historical verification indicator in the historical verification indicator dataset, the historical verification indicators are sorted to obtain the time series dataset of each historical verification indicator.

6. The method according to claim 4, characterized in that, The step of generating corresponding validation prediction metrics based on the time series datasets of each historical validation metric includes: Using time series algorithms, based on the time series datasets of each historical validation indicator, we generate validation prediction indicators for each historical validation indicator in the next m periods.

7. The method according to claim 6, characterized in that, The step of combining the target validation metrics in the validation metrics according to the data format of the training samples of the pre-trained classification model to obtain the combined metrics includes: The data organization strategy corresponding to the data format of the training samples of the classification model is obtained, wherein the data organization strategy is used to set and calculate the target historical verification index in the training samples. Based on the data organization strategy, the target verification indicators in the verification prediction indicators are set and calculated to obtain the combined indicators.

8. The method according to claim 7, characterized in that, The step of setting and calculating the target verification indicators in the verification prediction indicators according to the data organization strategy to obtain combined indicators includes: Based on the data organization strategy, determine the calculation method and the target verification index for specific calculations in the verification prediction index; The target verification indicators are calculated according to the aforementioned calculation method to obtain combined indicators.

9. The method according to claim 7 or 8, characterized in that, The step of inputting the remaining indicators and the combined indicators from the validation indicators into the classification model to obtain the model validation conclusion generated by the classification model includes: For any period in the verification prediction index over the next m periods, obtain the remaining indicators in the verification prediction index excluding the target verification indicator. The remaining indicators and combined indicators are input into the classification model, which divides them into N datasets. The N datasets are then input into N weak learners, and future model validation conclusions are generated based on the voting results of the N weak learners.

10. A device for generating model validation conclusions, characterized in that, include: The metrics acquisition module is used to acquire validation metrics for the customer rating model. The indicator combination module is used to combine the target verification indicators in the verification indicators according to the data format of the training samples of the pre-trained classification model to obtain the combined indicators. The classification model is trained by the historical verification indicators in the training samples, and the training of the classification model is supervised by the historical model verification conclusions in the training samples. The conclusion acquisition module is used to input the remaining indicators and the combined indicators from the verification indicators into the classification model to obtain the model verification conclusion generated by the classification model; The indicator combination module includes: A data organization strategy acquisition unit is used to acquire the data organization strategy corresponding to the data format of the training samples of the classification model, wherein the data organization strategy is used to set and calculate the target historical verification index in the training samples. The calculation unit is used to set and calculate the target verification indicators in the current verification indicators according to the data organization strategy to obtain the combined indicators; The conclusion acquisition module is specifically used for: Obtain the remaining indicators from the current verification indicators, excluding the target verification indicator; The remaining indicators and combined indicators are input into the classification model, which divides the remaining indicators and combined indicators into N datasets. The N datasets are then input into N weak learners, and the model validation conclusion for the current period is generated based on the voting results of the N weak learners. The device further includes: training the classification model using the following modules: The historical verification indicator acquisition module is used to acquire a historical verification indicator dataset, wherein the historical verification indicator dataset includes historical verification indicators for a set number of periods. The training module is used to obtain the historical model validation conclusions corresponding to the historical validation indicators in each period, and to perform supervised training on the classification model based on the historical validation indicator dataset and the historical model validation conclusions. The historical verification index acquisition module is specifically used for: Obtain historical verification indicators for a set number of periods; For each period's historical verification indicator, obtain the target historical verification indicator, and perform a setting calculation on the target historical verification indicator to obtain the historical combination indicator; A historical validation index dataset is generated based on the remaining historical validation indices and historical combined indices for each period.

11. An electronic device, characterized in that, The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements a method for generating model verification conclusions as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the method for generating model verification conclusions as described in any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for generating model verification conclusions as described in any one of claims 1-9.

Images