Model prediction capability test method, device and equipment and readable storage medium

Abstract

The invention discloses a model prediction capability test method. The method comprises the steps of obtaining preset model input parameter types and prediction result label types; assigning each model input parameter type, and screening prediction result label types to obtain each test sample; and testing the predictive ability of the to-be-tested model by using each test sample. By applying the model predictive ability testing method provided by the invention, the comprehensiveness of model predictive ability testing is greatly improved, and the testing effect is improved. The invention further discloses a model prediction capability testing device and equipment and a storage medium, which have corresponding technical effects.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and computer-readable storage medium for testing model prediction capabilities. Background Art

[0002] With the development of artificial intelligence technology, the use of prediction models to predict relevant indicators has become widely used. After the prediction model is trained, its predictive ability needs to be tested.

[0003] Currently, there are two main methods commonly used to predict model training capabilities: one is to test the model's predictive ability based on the predictive transmission relationship of indicator predictions; the other is to use statistical econometric models to test the model's predictive ability. However, both methods have certain shortcomings. First, testing the model's predictive ability based on the predictive transmission relationship of indicator predictions fails to account for the complexity of factors influencing the prediction results and may overlook some important factors influencing the prediction results. Second, using statistical econometric models to test the model's predictive ability requires high data quality and quantity. The model's assumptions and parameter estimates may contain certain errors, resulting in uncertainty in the model's prediction results.

[0004] In summary, how to effectively solve the problems of ignoring some important factors affecting the prediction results, the possible errors in the model's assumptions and parameter estimates, which lead to uncertainty in the model's prediction results and poor test results of the model's prediction ability, is an issue that technical personnel in this field urgently need to solve. Summary of the Invention

[0005] The purpose of this application is to provide a model prediction ability testing method, which greatly improves the comprehensiveness of the model prediction ability testing and improves the testing effect; another purpose of this application is to provide a model prediction ability testing device, equipment and computer-readable storage medium.

[0006] To solve the above technical problems, this application provides the following technical solutions:

[0007] A method for testing the predictive ability of a model, comprising:

[0008] Get the preset model input parameter types and prediction result label types;

[0009] Assign values ​​to each model input parameter type, and filter the prediction result label type to obtain each test sample;

[0010] The prediction ability of the model to be tested is tested using each test sample.

[0011] In a specific embodiment of the present application, each model input parameter type is assigned a value, and the prediction result label type is screened, including:

[0012] For each prediction result label type, randomly stratify and replace the original training sample set of the model to be tested to obtain a first sample set;

[0013] The input parameter type of each model is assigned according to the first sample set, and the prediction result label type of each first sample in the first sample set is determined as the prediction result label type of the corresponding sample.

[0014] In a specific embodiment of the present application, each model input parameter type is assigned a value, and the prediction result label type is screened, including:

[0015] Assigning model input parameters corresponding to each model input parameter type, randomly stratifying and replacing samples from the original training sample set of the model to be tested to obtain a second sample set;

[0016] The input parameter type of each model is assigned according to the second sample set, and the prediction result label type of each second sample in the second sample set is determined as the prediction result label type of the corresponding sample.

[0017] In a specific embodiment of the present application, each model input parameter type is assigned a value, and the prediction result label type is screened to obtain each test sample, including:

[0018] Performing a first modification on the model input parameter assignments and the prediction result label types of each model input parameter type corresponding to some original training samples in the original training sample set of the model to be tested, to obtain first modified samples;

[0019] The first modified samples and the samples in the original training sample set that have not been modified by the first modification are determined as test samples.

[0020] In a specific embodiment of the present application, each model input parameter type is assigned a value, and the prediction result label type is screened to obtain each test sample, including:

[0021] Performing a second modification on the prediction result label types corresponding to some original training samples in the original training sample set of the model to be tested to obtain second modified samples;

[0022] The second modified samples and the samples in the original training sample set that have not undergone the second modification are determined as test samples.

[0023] In a specific embodiment of the present application, the predictive ability test of the model to be tested is performed using each test sample, including:

[0024] Obtaining a preset model prediction indicator corresponding to the model to be tested;

[0025] The reference prediction index is calculated based on the prediction result label type corresponding to each test sample;

[0026] Utilize the model to be tested to predict the prediction result label type for each test sample, and calculate the actual prediction index according to the prediction result label type corresponding to each test sample obtained by prediction;

[0027] A model prediction capability test result of the model to be tested is determined based on the reference prediction index and the actual prediction index.

[0028] In a specific embodiment of the present application, after using each test sample to test the prediction ability of the model to be tested, the method further includes:

[0029] Obtain the model prediction ability test results;

[0030] The model parameters of the model to be tested are adjusted according to the model prediction ability test results.

[0031] A model prediction capability testing device, comprising:

[0032] The input parameter type and label type acquisition module is used to obtain the preset input parameter types of each model and the label types of each prediction result;

[0033] The test sample acquisition module is used to assign values ​​to the input parameter types of each model and filter the prediction result label types to obtain each test sample;

[0034] The prediction ability testing module is used to test the prediction ability of the model to be tested using various test samples.

[0035] A model prediction capability testing device, comprising:

[0036] memory for storing computer programs;

[0037] A processor is used to implement the steps of the model prediction ability testing method as described above when executing the computer program.

[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the model prediction capability testing method as described above.

[0039] The model prediction ability testing method provided in this application obtains preset model input parameter types and prediction result label types; assigns values ​​to each model input parameter type, and screens the prediction result label type to obtain each test sample; and uses each test sample to test the prediction ability of the model to be tested.

[0040] It can be seen from the above technical solution that there is no need to define a clear prediction transmission relationship. Traditional methods usually need to accurately estimate the prediction transmission relationship of the factors affecting the prediction results on the prediction indicators, which requires complex statistical model inference or personal subjective judgment and assumptions. This application only needs to assign values ​​to the model input types and screen the prediction result label types. It can cover a wider range in numerical processing and relies less on personal experience. The test scenario design focuses on the three dimensions of model input, sample, and prediction result label that affect the model prediction effect, and is comprehensive. Although this application can cover as many prediction ability test scenarios as possible, it is relatively simple at the operational level. There is no need to use complex statistical models to simulate and infer. The model prediction effect of each test scenario is statistically analyzed through model input assignment and prediction result label screening, which greatly improves the comprehensiveness of the model prediction ability test and improves the test effect.

[0041] Correspondingly, the present application also provides a model prediction capability testing device, equipment and computer-readable storage medium corresponding to the above-mentioned model prediction capability testing method, which has the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is a flowchart of an implementation method of the model prediction ability testing method in the embodiment of the present application;

[0044] Figure 2 This is another implementation flow chart of the model prediction ability testing method in the embodiment of the present application;

[0045] Figure 3 This is a structural block diagram of a model prediction capability testing device in an embodiment of the present application;

[0046] Figure 4 This is a structural block diagram of a model prediction capability testing device in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the specific structure of a model prediction capability testing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the embodiments described are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present application.

[0049] See also Figure 1 , Figure 1 This is a flowchart of an implementation method of the model prediction ability testing method in the embodiment of the present application. The method may include the following steps:

[0050] S101: Obtain the preset model input parameter types and prediction result label types.

[0051] During the training process of the model to be tested, the model input parameter types that affect the model prediction results are screened, and the original prediction model is trained using training samples containing model input parameters corresponding to each screened model input parameter type to obtain a trained prediction model, and the trained prediction model is determined as the model to be tested.

[0052] When the model prediction capability needs to be tested on the model to be tested, the preset model input parameter types and prediction result label types are obtained.

[0053] S102: Assign values ​​to the input parameter types of each model and filter the prediction result label types to obtain test samples.

[0054] After obtaining the preset model input parameter types and prediction result label types, we assign values ​​to each model input parameter type and filter the prediction result label type to obtain test samples. Compared to the training samples, the test samples can have only the model input parameter type values ​​changed, only the prediction result label type changed, or both the model input parameter type and the prediction result label type changed.

[0055] S103: Use each test sample to test the prediction ability of the model to be tested.

[0056] After obtaining each test sample, the predictive ability of the model to be tested is tested using each test sample. For example, a reference prediction index can be calculated based on the prediction result label type corresponding to each test sample. Each test sample is then input into the model to be tested, and the model to be tested predicts the prediction result label type for each test sample. Based on the predicted prediction result label type corresponding to each test sample, an actual prediction index is calculated. The predictive ability test results of the model to be tested are then obtained by comparing the actual prediction index with the reference prediction index.

[0057] It can be seen from the above technical solution that there is no need to define a clear prediction transmission relationship. Traditional methods usually need to accurately estimate the prediction transmission relationship of the factors affecting the prediction results on the prediction indicators, which requires complex statistical model inference or personal subjective judgment and assumptions. This application only needs to assign values ​​to the model input types and screen the prediction result label types. It can cover a wider range in numerical processing and relies less on personal experience. The test scenario design focuses on the three dimensions of model input, sample, and prediction result label that affect the model prediction effect, and is comprehensive. Although this application can cover as many prediction ability test scenarios as possible, it is relatively simple at the operational level. There is no need to use complex statistical models to simulate and infer. The model prediction effect of each test scenario is statistically analyzed through model input assignment and prediction result label screening, which greatly improves the comprehensiveness of the model prediction ability test and improves the test effect.

[0058] It should be noted that, based on the above embodiment, the present application also provides corresponding improved solutions. In subsequent embodiments, the same steps or corresponding steps as those in the above embodiment can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, and will not be described in detail in the following improved embodiments.

[0059] See also Figure 2 , Figure 2 This is another implementation flow chart of the model prediction ability testing method in the embodiment of the present application, which may include the following steps:

[0060] S201: Obtain the preset model input parameter types and prediction result label types.

[0061] S202: Assign values ​​to the input parameter types of each model and filter the prediction result label types to obtain test samples.

[0062] S203: Obtaining a preset model prediction indicator corresponding to the model to be tested.

[0063] The model to be tested may have different model prediction indicators depending on its training purpose. Obtain the preset model prediction indicators corresponding to the model to be tested.

[0064] S204: Calculate a reference prediction index based on the prediction result label type corresponding to each test sample.

[0065] After obtaining the preset model prediction indicators corresponding to the model to be tested and completing the construction of each test sample, the preset model prediction indicators are calculated according to the prediction result label types corresponding to each test sample to obtain the reference prediction indicators.

[0066] S205: Utilize the model to be tested to predict the prediction result label type for each test sample, and calculate the actual prediction index according to the prediction result label type corresponding to each test sample obtained by prediction.

[0067] After obtaining each test sample, the model to be tested is used to predict the label type of each test sample, and the corresponding label type of each test sample is obtained. The actual prediction index is calculated based on the predicted label type of each test sample.

[0068] S206: Determine a model prediction capability test result of the model to be tested based on the reference prediction index and the actual prediction index.

[0069] After the reference prediction index and the actual prediction index are calculated, the model prediction capability test result of the model to be tested is determined based on the reference prediction index and the actual prediction index. For example, the model prediction capability test result of the model to be tested is determined by comparing the difference between the reference prediction index and the actual prediction index.

[0070] S207: Obtain the model prediction capability test results.

[0071] After determining the model prediction ability test result of the model to be tested, the model prediction ability test result is obtained.

[0072] S208: Adjust model parameters of the model to be tested according to the model prediction ability test results.

[0073] After obtaining the model prediction ability test results, adjust the model parameters of the model to be tested based on the model prediction ability test results. For example, the training sample can be adjusted by optimizing the input parameters and correcting the samples. The model to be tested is then iteratively trained based on the adjusted training samples to achieve model parameter adjustment of the model to be tested.

[0074] In a specific embodiment of the present application, assigning values ​​to each model input parameter type and filtering the prediction result label type may include the following steps:

[0075] Step 1: For each prediction result label type, randomly stratify and replace the original training sample set of the test model to obtain the first sample set;

[0076] Step 2: Assign a value to each model input parameter type according to the first sample set, and determine the prediction result label type of each first sample in the first sample set as the prediction result label type of the corresponding sample.

[0077] For the convenience of description, the above two steps can be combined for explanation.

[0078] When constructing the test samples, for each prediction result label type, the original training sample set of the model to be tested is randomly stratified with replacement for sample extraction to obtain a first sample set. The input parameter type of each model is assigned according to the first sample set, and the prediction result label type of each first sample in the first sample set is determined as the prediction result label type of the corresponding sample. By extracting samples according to each prediction result label type, the distribution of the sample prediction result labels is changed. For example, the prediction result labels are divided into good samples and bad samples, and each test sample is determined by increasing the proportion of bad samples. Then, the prediction ability of the model to be tested is tested using each test sample after the proportion of bad samples is increased. By setting the distribution of the sample prediction result labels to be changed for test sample construction, the prediction ability of the model to be tested is tested using each test sample obtained by construction, thereby avoiding the influence of the sample prediction result label distribution of the training samples used in the training of the model to be tested on the prediction ability of the model to be tested.

[0079] In a specific embodiment of the present application, assigning values ​​to each model input parameter type and filtering the prediction result label type may include the following steps:

[0080] Step 1: Assign model input values ​​corresponding to each model input parameter type, and randomly stratify and replace samples from the original training sample set of the model to be tested to obtain the second sample set;

[0081] Step 2: Assign a value to each model input parameter type according to the second sample set, and determine the prediction result label type of each second sample in the second sample set as the prediction result label type of the corresponding sample.

[0082] For the convenience of description, the above two steps can be combined for explanation.

[0083] When constructing the test samples, the model input parameter values ​​corresponding to each model input parameter type are assigned, and the original training sample set of the model to be tested is randomly stratified with replacement to extract samples to obtain a second sample set. According to the second sample set, each model input parameter type is assigned, and the prediction result label type of each second sample in the second sample set is determined to be the prediction result label type of the corresponding sample. By extracting samples according to the model input parameter values ​​corresponding to each model input parameter type, the distribution of the model input parameter values ​​corresponding to each model input parameter type is changed, thereby diversifying the model input parameter values ​​corresponding to each model input parameter type, and using each test sample after changing the model input parameter values ​​corresponding to each model input parameter type to test the prediction ability of the model to be tested. By changing the distribution of the model input parameter values ​​corresponding to each model input parameter type to construct the test samples, and using each test sample obtained to test the prediction ability of the model to be tested, the influence of the distribution of the model input parameter values ​​corresponding to each model input parameter type in the training samples used when training the model to be tested on the prediction ability of the model to be tested is avoided.

[0084] In a specific embodiment of the present application, assigning values ​​to the input parameter types of each model and filtering the prediction result label types to obtain each test sample may include the following steps:

[0085] Step 1: Perform a first modification on the model input parameter assignment and the prediction result label type of each model input parameter type corresponding to some original training samples in the original training sample set of the test model to obtain first modified samples;

[0086] Step 2: Determine the first modified samples and the samples in the original training sample set that have not undergone the first modification as test samples.

[0087] For the convenience of description, the above two steps can be combined for explanation.

[0088] When constructing the test samples, the model input parameter assignments and the prediction result label types of each model input parameter type corresponding to some of the original training samples in the original training sample set of the model to be tested are first modified to obtain each first modified sample, and each first modified sample and each sample in the original training sample set that has not undergone the first modification are determined as each test sample. By modifying the model input parameter assignments and the prediction result label types of each model input parameter type corresponding to some of the original training samples in the original training sample set of the model to be tested to obtain each first modified sample, the prediction ability of the model to be tested is tested using each first modified sample and each sample in the original training sample set that has not undergone the first modification, thereby avoiding the influence of the sample prediction result label distribution of the training samples used in the training of the model to be tested on the prediction ability of the model to be tested, and at the same time avoiding the influence of the distribution of the model input parameter assignments corresponding to each model input parameter type on the prediction ability of the model to be tested.

[0089] In a specific embodiment of the present application, assigning values ​​to the input parameter types of each model and filtering the prediction result label types to obtain each test sample may include the following steps:

[0090] Step 1: Perform a second modification on the prediction result label types corresponding to some of the original training samples in the original training sample set of the test model to obtain second modified samples;

[0091] Step 2: Determine the second modified samples and the samples in the original training sample set that have not undergone the second modification as test samples.

[0092] For the convenience of description, the above two steps can be combined for explanation.

[0093] When constructing test samples, a second modification is performed on the prediction result label types corresponding to some of the original training samples in the original training sample set of the model to be tested, resulting in second modified samples. These second modified samples and the samples in the original training sample set that have not undergone the second modification are then determined as test samples. By modifying the prediction result label types corresponding to only some of the original training samples, the impact of emergencies on the model's prediction results is simulated. By using the second modified samples that only modified the prediction result label types corresponding to some of the original training samples and the samples in the original training sample set that have not undergone the second modification to perform model testing, the model's prediction capability under emergencies is tested.

[0094] To facilitate understanding of the technical solution provided by the embodiment of the present invention, the technical solution provided by the embodiment of the present invention is described in detail below using a specific scenario applied to a credit risk model stress test as an example.

[0095] It's important to note that banks are required to conduct stress testing to ensure they maintain sufficient capital to withstand risks in extreme scenarios, such as economic recession and financial market turmoil. These stress tests primarily focus on credit risk, market risk, and liquidity risk. Commercial banks should establish robust stress testing systems and utilize them as a key risk management tool to assess their capital adequacy and risk tolerance under various stress scenarios, ensuring they can effectively respond to various extreme risk events. Stress testing should be incorporated into banks' risk management processes and conducted regularly to inform their operational and capital management decisions. To this end, stress testing credit risk models to estimate potential credit losses under adverse scenarios is a key risk analysis method that complies with regulatory requirements and supports banks' long-term sustainable development goals.

[0096] Currently, banks use the following main methods for stress testing credit risk models: The first is indicator analysis. If credit risk has a clear stress transmission relationship, indicator analysis can be used directly for stress testing. This method is straightforward, easy to understand, and easy to use. However, it relies on the accurate identification and definition of credit risk transmission relationships and stress indicators. Inaccurate or incomplete identification and definition of these relationships and indicators can lead to biased test results. Furthermore, this relatively simple method fails to account for the complexity of credit risk and may overlook some important risk factors. The second method is based on statistical econometric models. When the relationship between stress indicators and stressed entities is complex, statistical econometric models are used for stress testing. This method assumes a quantitative relationship between credit risk factors and credit risk indicators. Based on a pre-defined stress scenario, model parameters are adjusted to predict credit risk under stress conditions. For example, a regression model is constructed using the probability of default as the dependent variable and risk factors such as macroeconomic variables as independent variables. The values ​​of the independent variables are adjusted based on the pre-defined macroeconomic stress scenario to predict changes in the probability of default. This method can handle complex nonlinear relationships and multi-factor interactions, and has a high degree of fit to data. However, it places high demands on data quality and quantity. Model assumptions and parameter estimates may contain certain errors, leading to uncertainty in the model's predictions. Other methods, such as the Portfolio Credit Risk model, are important tools for assessing the credit risk of financial institutions' credit portfolios and have become a common practice in macroeconomic stress testing. However, these methods require large amounts of accurate data to estimate model parameters, placing high demands on data and imposing limitations on model assumptions. For example, some assumptions, such as the assumption that asset values ​​follow a normal distribution, may not fully hold true in reality.

[0097] The data matching method provided in the embodiments of this application can be used to solve the above problems. Taking the stress test of the credit risk model as an example, the model prediction indicator corresponding to the credit risk model is generally the default rate, and the prediction result label types include default and non-default. The specific implementation process is as follows:

[0098] Obtain the macro model input parameter types and micro model input parameter types corresponding to the preset credit risk model, and obtain the default risk label and non-default risk label;

[0099] Assign values ​​to each macro model input parameter type and each micro model input parameter type, and screen for default risk labels and non-default risk labels to obtain test samples;

[0100] Use each test sample to perform stress testing on the credit risk model to be tested.

[0101] In a specific example application of the present application, the process of performing a credit risk model stress test may include:

[0102] Define the test objectives: For credit risk models, examine the impact of stress scenarios on the default rate predicted by the risk model and evaluate the impact on the model, such as the impact on the model's discrimination ability and model stability.

[0103] Identify risk factors:

[0104] (1) Determine the stress subject and stress indicators: The stress subject refers to the object to be stress tested, which is generally required to cover the main exposure of retail risk exposure or the main business line under it. In the embodiment of this application, it is the risk model of credit products. The stress indicator is a specific quantifiable and calculable indicator that reflects the stress subject. The main risk factor of stress testing is the credit risk parameters of the main asset portfolio. For credit risk models, it is mainly the model-predicted default rate.

[0105] (2) Identify stress factors and stress indicators: Stress factors refer to the reasons that cause fluctuations in the stress indicators of the stressed object. Generally speaking, stress factors come from three main sources: cyclical factors, special event factors, and concentration factors. Stress indicators are the specific manifestations of stress factors and are also called risk indicators. For example, cyclical factors can include macroeconomic factors, housing prices, interest rate fluctuations, etc.; special event factors include various event shocks; and concentration factors include the risk amplification caused by the excessive concentration of credit products among customers, industries, and / or regions.

[0106] Design stress scenarios:

[0107] The embodiments of this application are based on the three dimensions of model input, sample, and risk label. When designing scenarios, sensitivity analysis and scenario testing are comprehensively considered to cover as many stress items as possible. Sensitivity analysis examines the impact of changes in a certain risk factor on the risk model under the condition that other risk factors remain unchanged. Scenario testing assumes the impact of multiple risk factors changing simultaneously or certain extremely adverse events on the risk model.

[0108] (1) Assumptions: Stress events are correlated with economic recession and lack of liquidity. The root causes of risk are related to factors such as changes in the macro environment (financial market factors, industry, economy, regulation, politics, society, ecology, etc.), changes in counterparty qualifications (deterioration of the debtor's debt repayment ability and / or reduction in repayment willingness), etc. At the same time, the correlation between the risk influencing factors assumed in the model input is analyzed. For example, if the customer's credit debt limit in the model input is significantly increased compared to the model training time, it can indicate that the debt level, one of the risk factors, has increased; for example, if the number of customers' recent defaults increases, it can indicate that the risk factors of repayment behavior or repayment ability have increased.

[0109] (2) Scenario design:

[0110] First, sensitivity testing is used. This involves assuming changes to a single key risk factor or a few closely related factors, and testing the impact on the model's predictive capabilities. Taking the factors influencing retail credit risk as an example, the main risk factors can be summarized as funding needs, debt levels, repayment capacity, and repayment behavior through analysis of assumptions and model inputs. Secondly, scenario testing is used. This involves assuming that multiple key risk factors simultaneously affect customer quality, but with varying degrees of impact on the overall customer base. Based on the impact of each stress factor, testing can be conducted using stress scenarios designed according to Table 1. Table 1 shows a stress scenario design test table.

[0111] Table 1

[0112]

[0113] As shown in Table 1, stress items involved in interest rate impact risk factors may include changes in deposit and loan interest rates and changes in bond yields; stress items involved in liquidity tightness risk factors may include increased financing demand, blocked financing channels leading to increased financing costs, and decreased asset liquidity; stress items involved in asset quality decline risk factors may include industry risks such as the decline in the value of real estate and collateral; stress items involved in personal credit risk factors may include decreased income and increased debt.

[0114] Conduct testing: On an operational level, we will use the models related to the stress items to enter different affected scenarios and measure the model's ability to distinguish between the new scenarios. If some customer groups are affected by specific risk factors, we will adopt a sampling method that only measures the specific customer group. Specifically, there are several situations:

[0115] First, due to the combined influence of risk factors, the overall quality of the customer base has declined. A random stratified sampling method with replacement is used to measure changes in model input parameters and sample labels. Specifically, a random stratified sampling method with replacement is used to sample good and bad customers. If the original sample set has a bad customer ratio of 5%, assuming that the ratio after sampling is 1%-20%, the model performance is evaluated on the new sample set.

[0116] Second, for the impact of some stressed projects, we employ random stratified sampling with replacement for the changed model inputs. Specifically, if certain model inputs change, such as interest rates, we conduct stratified sampling with replacement for interest-related inputs (such as loan pricing and funding costs) and evaluate the model's performance on the new sample set.

[0117] Third, under the corresponding stress items, the model inputs are designed to be assigned values ​​that change in a negative direction based on the degree of impact. At the same time, the labels of some samples that were originally good but whose inputs have changed to bad are changed to bad labels. Specifically, if some model inputs change, for example, if the value of a house is a model input, and the house price drops, the overall valuation parameter is assigned the original value minus a portion of the value. At the same time, the labels of customers who were originally good customers but may become bad customers due to falling house prices are assigned to bad, simulating the model's capabilities under a falling market.

[0118] Fourth, consider the situation where some samples become bad while the model input parameters remain unchanged. This means considering samples that have been approved by the model but have not yet reached their performance period, and how the model performs when external factors influence increased defaults. In other words, the model input parameters remain unchanged, but some labels change from good to bad. If the customer's qualifications remain unchanged, but there are external influences (such as economic losses caused by an earthquake), a certain proportion of the good samples will be labeled bad (in this case, stratified sampling can also be performed based on the input parameters, for example, the samples can be divided into large, medium, and small amounts, and a certain proportion of the labels after sampling in each stratum will be changed). The model's performance is then evaluated on the new sample set.

[0119] Pay attention to changes in pressure indicators:

[0120] After setting a stress scenario, it's common to monitor changes in stress indicators over time. After stress testing, the predicted default rate under the stress scenario is calculated, and the change multiple relative to the predicted default rate after a period of no stress is calculated. This is done by multiplying the predicted default rate by the change multiple caused by the corresponding stress scenario, and setting an upper limit for the predicted default rate. The upper limit for the predicted default rate is 0.99 (less than 1). Potential risk points worth noting can be selected and their impact levels assessed, as shown in Table 2. Table 2 shows the changes in the predicted default rate under three stress scenarios (mild, moderate, and severe).

[0121] Table 2

[0122]

[0123] As shown in Table 2, assuming no stress test, that is, assuming that the new sample distribution is consistent with the distribution during model training, the model score prediction value after a period of time is calculated as Pa. In the case of stress testing, the new model score prediction value Pb is calculated after testing according to the above four methods; the coefficient is calculated based on the value of Pb / Pa. The larger the coefficient value, the more changes there are, but the model score prediction value is always between [0, 1]. All new probability values ​​are set to the minimum value of the predicted default rate * coefficient and 0.99.

[0124] Potential risks and emergency measures:

[0125] Analyze stress test results and identify potential risk points and vulnerabilities. Under certain extreme values, a decrease or failure in the model's predictive ability is within reasonable limits. The results should be reported according to internal procedures, and remedial measures should be formulated based on the results. If potential risks lead to increased default risk, the risk points can be highlighted based on the stress test scenario. If model optimization is required, adjustments can be made directly based on the corresponding input parameters or samples. If, according to the stress test results, the model's predictive ability decreases or fails under certain extreme values, this is within reasonable limits and should be reported as required, along with measures to be formulated. For example, if economic losses from an earthquake increase the default rate, this is not directly predictable or can be adjusted. Model adjustments can also include optimizing parameters and revising samples. For example, in an economic recession scenario, features such as industry indices can be added for input optimization. For example, if previously good samples deteriorate under the stress scenario and actually default, the proportion of these samples in the training set can be increased or their labels can be corrected to achieve sample correction.

[0126] By taking the predicted default rate as a stress indicator, and designing stress projects that cover as much as possible based on model input, samples, and risk labels, the design of stress scenarios is not limited to changes in the economic environment. Unlike traditional stress transmission models that first infer the macroeconomic environment to design stress scenarios, the embodiment of this application assumes that changes in internal and external conditions are reflected in the data distribution. By designing stress projects and varying degrees of influence on affected customer groups, changes in stress indicators are comprehensively observed. Risk points are identified based on stress test results, and attention is paid to situations in which the model fails or is inapplicable in the test results. Potential risk factors are identified in order to propose emergency measures, thereby achieving a comprehensive assessment of changes in the predictive ability of the credit risk model under changes in the internal and external environment.

[0127] Corresponding to the above method embodiment, the present application also provides a model prediction ability testing device. The model prediction ability testing device described below and the model prediction ability testing method described above can refer to each other.

[0128] See also Figure 3 , Figure 3 This is a structural block diagram of a model prediction capability testing device in an embodiment of the present application. The device may include:

[0129] Input parameter type and label type acquisition module 31, used to obtain the preset input parameter type of each model and the label type of each prediction result;

[0130] The test sample acquisition module 32 is used to assign values ​​to the input parameter types of each model and filter the prediction result label types to obtain each test sample;

[0131] The prediction capability test module 33 is used to test the prediction capability of the model to be tested using each test sample.

[0132] It can be seen from the above technical solution that there is no need to define a clear prediction transmission relationship. Traditional methods usually need to accurately estimate the prediction transmission relationship of the factors affecting the prediction results on the prediction indicators, which requires complex statistical model inference or personal subjective judgment and assumptions. This application only needs to assign values ​​to the model input types and screen the prediction result label types. It can cover a wider range in numerical processing and relies less on personal experience. The test scenario design focuses on the three dimensions of model input, sample, and prediction result label that affect the model prediction effect, and is comprehensive. Although this application can cover as many prediction ability test scenarios as possible, it is relatively simple at the operational level. There is no need to use complex statistical models to simulate and infer. The model prediction effect of each test scenario is statistically analyzed through model input assignment and prediction result label screening, which greatly improves the comprehensiveness of the model prediction ability test and improves the test effect.

[0133] In a specific embodiment of the present application, the test sample obtaining module 32 may include:

[0134] The first sample set acquisition submodule is used to randomly stratify and replace samples from the original training sample set of the test model for each prediction result label type to obtain the first sample set;

[0135] The first input parameter assignment and label type determination submodule is used to assign the input parameter type of each model according to the first sample set, and determine the prediction result label type of each first sample in the first sample set as the prediction result label type of the corresponding sample.

[0136] In a specific embodiment of the present application, the test sample obtaining module 32 may include:

[0137] The second sample set acquisition submodule is used to assign model input parameters corresponding to each model input parameter type, and randomly stratify and replace the original training sample set of the test model to obtain the second sample set;

[0138] The second input parameter assignment and label type determination submodule is used to assign the input parameter type of each model according to the second sample set, and determine the prediction result label type of each second sample in the second sample set as the prediction result label type of the corresponding sample.

[0139] In a specific embodiment of the present application, the test sample obtaining module 32 may include:

[0140] A first modified sample obtaining submodule is used to perform a first modification on the model input parameter assignment and the prediction result label type of each model input parameter type corresponding to some original training samples in the original training sample set of the model to be tested, to obtain each first modified sample;

[0141] The first test sample determination submodule is configured to determine the first modified samples and the samples in the original training sample set that have not undergone the first modification as test samples.

[0142] In a specific embodiment of the present application, the test sample obtaining module 32 may include:

[0143] A second modified sample obtaining submodule is used to perform a second modification on the prediction result label types corresponding to some original training samples in the original training sample set of the model to be tested, to obtain second modified samples;

[0144] The second test sample determination submodule is configured to determine the second modified samples and the samples in the original training sample set that have not undergone the second modification as test samples.

[0145] In a specific embodiment of the present application, the predictive ability testing module 33 may include:

[0146] The prediction indicator acquisition submodule is used to obtain the preset model prediction indicators corresponding to the model to be tested;

[0147] The reference prediction index calculation submodule is used to use the model to be tested to predict the prediction result label type of each test sample, and calculate the actual prediction index according to the prediction result label type corresponding to each test sample;

[0148] The actual prediction index calculation submodule is used to obtain a second prediction value obtained by using the model to be tested to predict the preset model prediction index according to each test sample;

[0149] The test result determination submodule is used to determine the model prediction ability test result of the model to be tested based on the reference prediction index and the actual prediction index.

[0150] In a specific embodiment of the present application, the device may further include:

[0151] A test result acquisition module is used to obtain the model prediction ability test result after using each test sample to test the prediction ability of the model to be tested;

[0152] The model parameter adjustment module is used to adjust the model parameters of the model to be tested according to the model prediction ability test results.

[0153] Corresponding to the above method embodiment, see Figure 4 , Figure 4 This is a schematic diagram of the model prediction ability testing device provided in this application, which may include:

[0154] Memory 332, for storing computer programs;

[0155] The processor 322 is configured to implement the steps of the model prediction capability testing method of the above method embodiment when executing the computer program.

[0156] For details, please refer to Figure 5 , Figure 5 This is a schematic diagram of the specific structure of a model prediction capability testing device provided in this embodiment. The model prediction capability testing device may vary significantly due to different configurations or performance. It may include a processor (central processing unit, CPU) 322 (for example, one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary storage or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), each of which may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332 to execute the series of instruction operations in the memory 332 on the model prediction capability testing device 301.

[0157] The model prediction capability testing device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0158] The steps in the model prediction capability testing method described above can be implemented by the structure of the model prediction capability testing device.

[0159] Corresponding to the above method embodiment, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0160] Obtain the preset model input parameter types and prediction result label types; assign values ​​to each model input parameter type and filter the prediction result label type to obtain each test sample; use each test sample to test the prediction ability of the model to be tested.

[0161] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0162] For an introduction to the computer-readable storage medium provided in this application, please refer to the above method embodiment, and this application will not go into details here.

[0163] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. The devices, apparatuses, and computer-readable storage media disclosed in the embodiments are described briefly because they correspond to the methods disclosed in the embodiments. For relevant details, refer to the description of the methods.

[0164] Specific examples are used herein to illustrate the principles and implementation methods of this application. The description of the above embodiments is only intended to help understand the technical solution and core ideas of this application. It should be noted that, for those skilled in the art, without departing from the principles of this application, various improvements and modifications may be made to this application, and such improvements and modifications also fall within the scope of protection of this application.

Claims

1. A method for testing model prediction ability, characterized in that: include: Get the preset model input parameter types and prediction result label types; Assign values ​​to each model input parameter type, and filter the prediction result label type to obtain each test sample; The prediction ability of the model to be tested is tested using each test sample.

2. The model prediction ability testing method according to claim 1, characterized in that: Assign values ​​to each model input parameter type and filter the prediction result label type, including: For each prediction result label type, randomly stratify and replace the original training sample set of the model to be tested to obtain a first sample set; The input parameter type of each model is assigned according to the first sample set, and the prediction result label type of each first sample in the first sample set is determined as the prediction result label type of the corresponding sample.

3. The model prediction ability testing method according to claim 1, characterized in that: Assign values ​​to each model input parameter type and filter the prediction result label type, including: Assigning model input parameters corresponding to each model input parameter type, randomly stratifying and replacing samples from the original training sample set of the model to be tested to obtain a second sample set; The input parameter type of each model is assigned according to the second sample set, and the prediction result label type of each second sample in the second sample set is determined as the prediction result label type of the corresponding sample.

4. The model prediction ability testing method according to claim 1, characterized in that: Assign values ​​to each model input parameter type and filter the prediction result label type to obtain various test samples, including: Performing a first modification on the model input parameter assignments and the prediction result label types of each model input parameter type corresponding to some original training samples in the original training sample set of the model to be tested, to obtain first modified samples; The first modified samples and the samples in the original training sample set that have not been modified by the first modification are determined as test samples.

5. The model prediction ability testing method according to claim 1, characterized in that: Assign values ​​to each model input parameter type and filter the prediction result label type to obtain various test samples, including: Performing a second modification on the prediction result label types corresponding to some original training samples in the original training sample set of the model to be tested to obtain second modified samples; The second modified samples and the samples in the original training sample set that have not undergone the second modification are determined as test samples.

6. The model prediction ability testing method according to any one of claims 1 to 5, characterized in that: Use each test sample to test the predictive ability of the model to be tested, including: Obtaining a preset model prediction indicator corresponding to the model to be tested; The reference prediction index is calculated based on the prediction result label type corresponding to each test sample; Utilize the model to be tested to predict the prediction result label type for each test sample, and calculate the actual prediction index according to the prediction result label type corresponding to each test sample obtained by prediction; A model prediction capability test result of the model to be tested is determined based on the reference prediction index and the actual prediction index.

7. The model prediction ability testing method according to claim 1, characterized in that: After using each test sample to test the predictive ability of the model to be tested, it also includes: Obtain the model prediction ability test results; The model parameters of the model to be tested are adjusted according to the model prediction ability test results.

8. A model prediction capability testing device, characterized in that: include: The input parameter type and label type acquisition module is used to obtain the preset input parameter types of each model and the label types of each prediction result; The test sample acquisition module is used to assign values ​​to the input parameter types of each model and filter the prediction result label types to obtain each test sample; The prediction ability testing module is used to test the prediction ability of the model to be tested using various test samples.

9. A model prediction capability testing device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the model prediction capability testing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the model prediction capability testing method according to any one of claims 1 to 7.

Images