Credit risk prediction method and system, storage medium and terminal

Abstract

The invention discloses a credit risk prediction method and system, a storage medium and a terminal, and belongs to the field of big data, and the method mainly comprises the steps: carrying out the optimization of credit data through an ATD algorithm; the ResNet and the TabNet are combined, and feature enhancement is carried out on the optimized credit data; and constructing a credit risk prediction model, training the credit risk prediction model by using the extracted features, and performing hyper-parameter optimization on the credit risk prediction model by using a TuRBO trust region-based Bayesian optimization algorithm. According to the method, the problem of insufficient data optimization of the credit risk prediction model is solved through the ATD algorithm, the ResNet and the TabNet are combined, the original data features are reserved, important information in the high-dimensional features is obtained, the original features are enhanced, the model prediction accuracy and generalization ability are improved, and the interpretability of the model is greatly reserved; and meanwhile, the parameter adjustment difficulty of the model is reduced through a TuRBO optimization algorithm.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a credit risk prediction method, system, storage medium and terminal. Background Art

[0002] With the current vigorous development of the national economy and the improvement of consumption concepts, the scale of credit business continues to expand and the complexity continues to increase. How to accurately and quickly predict the credit risk of credit customers based on existing credit data has become an important problem that needs to be solved in credit business practice. Credit risk prediction requires extracting important features from the basic information of customers to predict whether the loan business is risky. In the context of big data, if only relying on approval personnel to analyze and predict, it is costly and difficult to expand. Therefore, researchers try to solve existing problems by learning similarities in massive financial data through machine learning models. Credit risk prediction technology can use the information provided by customers to predict their loan risks in advance during the financial business process, and can help approval personnel accurately and efficiently identify whether customers have the ability to repay the loan, thereby reducing the work pressure of loan approval personnel to a certain extent.

[0003] In the field of credit forecasting research, there are generally two main categories of forecasting methods. The earliest is based on statistical methods, which use statistical properties to analyze data distribution and patterns and build models. Common statistical methods include logistic regression and linear discriminant analysis. With the rise of machine learning, some scholars have begun to introduce machine learning methods into credit forecasting research. This method is different from traditional statistical methods. It does not need to explicitly look for patterns in the data, but focuses on improving forecasting accuracy by learning data features.

[0004] A large number of existing studies have shown that compared with a single machine learning model, the ensemble learning model has higher prediction accuracy and stronger applicability in credit risk prediction. There are generally three ways to use ensemble learning. The first is the Boosting architecture, which uses a serial method between base learners to construct a strong learner, such as Random Forest. The second is the Bagging architecture, which constructs multiple independent models and then constructs a strong learner by election or weighting. The mainstream algorithms of the Boosting architecture commonly used in the credit field mainly include Extreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM) and Categorical Boosting (CatBoost). Among them, the PSO algorithm has been used to optimize the hyperparameters of the LightGBM model for credit risk assessment, and the synthetic minority oversampling (SMOTE) has been used to optimize the original data. At the same time, the improved Q-learning algorithm based on reinforcement learning is used to optimize the XGBoost hyperparameters and verify its effectiveness. Another mainstream integrated learning method, Stacking, combines Boosting and Bagging, uses multiple base learners to learn the original data, and then passes the data learned by these base learners to the second-layer meta-learner model for fitting. Due to the unique multi-layer architecture and combination mechanism of this method, the classification effect of the combined model can often be significantly improved. Some studies in the credit field have achieved significant improvements compared to single models through stacking combinations of algorithms such as logistic regression, decision tree, SVM, XGBoost, and random forest.

[0005] With the development of deep learning models, researchers have applied deep learning models to credit default risk prediction. For example, the existing technology uses CNN models to extract features and combines machine learning methods to predict credit risk, and has achieved good prediction results. Some people use the hidden layer of deep neural networks as feature extractors to convert raw variables into high-level abstract features for input into integrated model training, which significantly improves the training effect of Boosting models. Compared with general machine learning models, deep learning models have more advantages in feature extraction.

[0006] However, there are still some problems in the existing research: First, most data optimization methods are limited to dealing with data imbalance through undersampling and oversampling techniques. In the undersampling method, it is possible that the instances required in the training process are removed, affecting the performance of the classifier. In the oversampling method, the noise data may also be copied repeatedly, thus affecting the prediction results; second, most existing machine learning models have relatively simple feature representations during the training process, and some studies only use raw data to train the model. In order to make the model learn high-dimensional features, some studies use deep learning as a "feature extractor" and input it into the machine learning model training, but ignore the importance of the original features and the interpretability of the model; third, when training the machine learning model, the numerous hyperparameters of the classifier bring great difficulties to the parameter adjustment work, and a large workload is required in the process of finding the optimal solution. Summary of the invention

[0007] The purpose of the present invention is to overcome the technical problems existing in the prior art and to provide a credit risk prediction method, system, storage medium and terminal.

[0008] The objective of the present invention is achieved through the following technical solutions:

[0009] In a first aspect, a credit risk prediction method is provided, the method comprising the following steps:

[0010] S1. Collect credit data;

[0011] S2. Optimizing the credit data by using the ATD algorithm;

[0012] S3. Combine ResNet and TabNet to enhance the features of the optimized credit data;

[0013] S4. Build a credit risk prediction model and use the enhanced features to train the credit risk prediction model, wherein the TuRBO trust region-based Bayesian optimization algorithm is used to optimize the hyperparameters of the credit risk prediction model;

[0014] S5. Use the trained credit risk prediction model to perform credit risk prediction.

[0015] In some embodiments, the credit risk prediction model is an XGBoost-Stacking model.

[0016] In some embodiments, collecting credit data includes:

[0017] Process missing values ​​in credit data.

[0018] In some embodiments, the optimizing the credit data by using the ATD algorithm includes:

[0019] Generate synthetic samples through ADASYN oversampling algorithm;

[0020] The TBallNN error detection technique is used to identify noise samples in synthetic samples;

[0021] The noise samples are optimized by using the MTFRDE multi-objective feature-preserving differential evolution algorithm.

[0022] In some embodiments, the combination of ResNet and TabNet to enhance the features of the optimized credit data includes:

[0023] ResNet and TabNet are used to extract features of credit data respectively;

[0024] The features extracted by ResNet and TabNet are combined with their respective input feature residuals to obtain multi-dimensional feature representation.

[0025] In some embodiments, the credit risk prediction model is optimized using a TuRBO trust region-based Bayesian optimization algorithm, including:

[0026] Based on a series of real samples, multiple local Gaussian process models are constructed in parallel, and trust regions are constructed through the local Gaussian process models;

[0027] According to the multiple optimal candidate points returned by the multiple trust regions, multiple true values ​​of the optimization function are obtained and the global optimal value is selected;

[0028] By comparing the true value and the Gaussian model value, choose to expand or reduce the trust region, and add the optimal true value and the optimal candidate point to the training sample;

[0029] Repeat the above steps until the maximum number of iterations is reached.

[0030] In some embodiments, the method further comprises:

[0031] The credit risk prediction model is evaluated using accuracy, precision, recall and F1-score.

[0032] In a second aspect, a credit risk prediction system is provided, the system comprising:

[0033] a data collection module configured to collect credit data;

[0034] a credit data optimization module configured to optimize the credit data using an ATD algorithm;

[0035] A feature enhancement module, configured to combine ResNet and TabNet to perform feature enhancement on the optimized credit data;

[0036] A credit risk prediction model building module is configured to build a credit risk prediction model and train the credit risk prediction model using enhanced features, wherein the credit risk prediction model is hyperparameter optimized using a TuRBO trust region-based Bayesian optimization algorithm;

[0037] The credit risk prediction module is configured to use the trained credit risk prediction model to perform credit risk prediction.

[0038] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the credit risk prediction methods is implemented.

[0039] In a fourth aspect, a terminal is provided, comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor executes any one of the credit risk prediction methods described above when executing the computer instructions.

[0040] It should be further explained that the technical features corresponding to the above options can be combined or replaced with each other to form a new technical solution if there is no conflict.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] (1) The present invention solves the problem of insufficient data optimization of the credit risk prediction model through the ATD algorithm, wherein the data set is balanced through ADASYN, and then the noise in the data is located through TBallNN, and the noise in the data is optimized through the differential evolution algorithm of MTFRDE multi-objective feature preservation, and there is a noise location and optimization process; by combining ResNet and TabNet, the original data features are retained while obtaining important information in high-dimensional features, the original features are enhanced, the model prediction accuracy and generalization ability are improved, and the interpretability of the model is greatly retained; at the same time, the difficulty of model parameter adjustment is reduced through the TuRBO optimization algorithm. The data quality is improved, the hyperparameters of the model are optimized, and the prediction accuracy is significantly improved.

[0043] (2) In one example, the credit risk prediction model was evaluated using accuracy, precision, recall and F1-score. The prediction results of the model of the present invention were improved compared with the existing methods in terms of accuracy, precision, recall and F1-score. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a credit risk prediction method shown in an embodiment of the present invention;

[0045] Figure 2The overall framework diagram of ATDTS shown in the embodiment of the present invention;

[0046] Figure 3 This is a diagram showing the internal structure of a residual block in a ResNet model according to an embodiment of the present invention;

[0047] Figure 4 The basic architecture diagram of the TabNet model shown in the embodiment of the present invention;

[0048] Figure 5 The internal structure diagram of the Feature transformer shown in the embodiment of the present invention;

[0049] Figure 6 The internal structure diagram of the Attentive Transformer shown in the embodiment of the present invention;

[0050] Figure 7 The figure is a schematic diagram of the XGBoost-Stacking model process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various configurations. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0052] It should be noted that the defects existing in the solutions in the above-mentioned prior art are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above-mentioned problems and the solutions proposed in the embodiments of the present application for the above-mentioned problems below should all be the contributions made by the inventor to the present application in the process of invention and creation, and should not be understood as technical contents known to technical personnel in this field.

[0053] In view of the technical problems pointed out in the background technology, the embodiments provided by the present invention are as follows:

[0054] Example 1

[0055] In an exemplary embodiment, a credit risk prediction method is provided, referring to Figure 1 , the method comprises the following steps:

[0056] S1. Collect credit data;

[0057] S2. Optimizing the credit data by using the ATD algorithm;

[0058] S3. Combine ResNet and TabNet to enhance the features of the optimized credit data;

[0059] S4. Build a credit risk prediction model and use the enhanced features to train the credit risk prediction model, wherein the TuRBO trust region-based Bayesian optimization algorithm is used to optimize the hyperparameters of the credit risk prediction model;

[0060] S5. Use the trained credit risk prediction model to perform credit risk prediction.

[0061] Specifically, in the current big data environment, the scale of credit business is large and the data is complex, so the credit risk prediction model has become a research hotspot in credit finance. Aiming at the problems of insufficient data optimization, single feature representation and difficult model parameter adjustment in the existing credit risk prediction model, this method is based on the massive high-quality credit data of LendingClub. First, the data is optimized by the ATD algorithm to improve the data quality. Then, ResNet and TabNet are used to extract features, and the composite feature representation is optimized by combining with the residuals of the respective input features. While retaining the original data features, important information in the high-dimensional features is obtained, and the original features are enhanced, which improves the model prediction accuracy and generalization ability, and greatly retains the interpretability of the model. Finally, the XGBoost-Stacking model after the hyperparameters are optimized by the TuRBO algorithm is input for training, which reduces the difficulty of model parameter adjustment, optimizes the hyperparameters of the model, and significantly improves the prediction accuracy.

[0062] Example 2

[0063] Based on the inventive concept of Example 1, a TuRBO adaptive integrated model credit risk prediction method ATDTS based on ATD data optimization and composite feature representation optimization is proposed. This example introduces the specific process of the credit risk prediction method ATDTS in detail.

[0064] The credit risk prediction method ATDTS proposed in this embodiment is mainly aimed at personal credit risk prediction, in which each credit data includes the basic information of the borrower and the corresponding prediction label. ATDTS mainly includes three modules: data processing optimization, composite feature representation optimization, and adaptive credit risk prediction model. The overall architecture of this method is as follows Figure 2 shown.

[0065] Data processing optimization is as follows:

[0066] The data of this invention comes from the public data of the loan platform. In order to ensure the timeliness of the research, this invention selects data with a time span close to the present. At the same time, it ensures that the characteristics of the samples will not produce large distribution differences due to time changes, thereby not affecting the fitting quality of the model. After comprehensive consideration, this invention intercepts 887,379 relatively new data for studying the credit prediction model. The original data set contains 887,379 samples and 74 features.

[0067] In the data preprocessing stage, the present invention first counted the missing values ​​of the features of the data set, and only retained the features with less than 20% missing values. After the missing value processing, the present invention still retained 52 valid features. According to the definition of the performance and default status of LendingClub in the prior art, the present invention selected borrowers with a loan status of Fully Paid as performance customers and borrowers with a loan status of Charged Off as default customers, and used them as target variables. Subsequently, the present invention used three methods, random forest, CatBoost and XGBoost, to sort the feature importance and selected the top 30 common variables. By removing factors such as a large number of missing values ​​and meaningless post-loan information for the evaluation of the applicant borrower information, the present invention finally retained 21 original features, and the number of valid data samples was 252,971. Then, the present invention performed feature abstraction on the data and converted some data into features that the algorithm can understand. At the same time, the present invention corrected the features with incorrect data types and processed multi-valued ordered data and multi-valued unordered data.

[0068] After the above preliminary processing of the original data, the feature number is 36. At this time, the present invention performs statistical analysis on the two categories of fully paid and charged off and finds that the ratio between the two is 82:18, and the data set is seriously unbalanced.

[0069] To this end, the present invention proposes a new oversampling method ATD (ADASYN-TBallNN-MTFR DE) to optimize the data set. First, the original data set is divided into majority samples and minority samples. Then, the present invention applies a method based on adaptive synthetic sampling (ADASYN) to generate synthetic samples, thereby increasing the scale of the original data. In the next second step, the present invention uses TBallNN as an error detection technology to identify noise samples (unreliable samples). Once the unreliable samples and normal samples are clearly defined, the present invention applies the MTFRDE method (differential evolution improvement: differential evolution algorithm with multi-objective feature preservation) to optimize the noise samples. The result of this process is that the data set contains an optimized set from noise and normal samples.

[0070] The TBallNN rule is improved based on the existing two-layer nearest neighbor (TLNN) rule. The Ball tree is used instead of the Kd tree in TLNN for data search. It is more suitable for data sets with large data scale and high dimension. TBallNN is used for k-nearest neighbor (kNN) classifier. This method considers the neighborhood of the query and the neighborhood of all selected training instances in this neighborhood at the same time. Then, the two-layer nearest neighbors of the query are determined based on the distance, distribution relationship and reverse nearest neighbor relationship between the query and all selected training instances in the above neighborhood. ATD uses this method to query unreliable samples. Unreliable samples and normal samples are defined as follows.

[0071] Unreliable samples: If a sample Si belongs to the unreliable sample set (US), then the natural neighbors of this sample have different class labels.

[0072]

[0073] Where l(Si) and l(Sj) are class labels. NN(Si) = {Sj, ...} is the natural neighbor set of sample Si.

[0074] Normal samples: Assuming that sample di belongs to the normal sample set (NS), then the natural neighbors of this sample have only the same class labels.

[0075]

[0076] Where l(Si) and l(Sj) are class labels. NN(Si) = {Sj, ...} is the natural neighbor set of sample Si.

[0077] The DE effective variant based on multi-objective feature retention (MTFRDE) is a simple and efficient algorithm. In this step, MTFRDE is used to optimize the features of unreliable samples. In this method, mutation and crossover operations of multi-objective updates and original sample data feature retention operations are performed. The improved method based on differential evolution DE / rand / 2 mutation strategy brings population diversity. Each particle can explore a larger search space during the evolution process and retains part of the weight of the original features. While ensuring that the data sample is optimized, some of the original data characteristics are retained, ensuring data diversity. First, initialize the unreliable sample U G The optimized initial population is X i,G , i = 1, 2, ...., NP, where the dimension of X is D and NP is the population size. The initial population is selected from the natural neighbor set of unreliable samples, and the samples with the same number of labels are selected from the natural neighbor set containing positive and negative labels to form a set as the optimized initial population. Thereafter, the mutation, crossover, and then selection strategies of the differential evolution algorithm are applied. This process is repeated to generate optimized data.

[0078] In this method, a mutation strategy for improving DE with multi-objective feature preservation is used: DE / rand / 2, as shown below.

[0079] V G+1 =K*U G +X r1,G +F*(X r2,G -X r3,G )+P*(X r4,G -X r5,G ) (3)

[0080] Among them, V G+1 is the data sample after mutation optimization, r1, r2, r3, r4 and r5 are random numbers in the range [1, NP]. NP represents the population size, K represents the weight, F and P represent the scaling factors, and G represents the generation update.

[0081] In order to increase the population diversity of the target sample vectors, a crossover process is applied. The strategy used is as follows:

[0082]

[0083] Among them, C j,G+1 is the crossover data sample, CR is the crossover rate for generating new test vectors, rand is a random number generated in the range [0,1]. Where j = 1, 2, ..., D = 36 is the dimension of the data.

[0084] At the end of the MTFRDE method, since the purpose of this method is to optimize the data sample to a normal sample, the differential evolution algorithm is not used to select the optimal sample. Instead, C G+1 Whether the sample is already a normal sample, if so, add it to the normal sample set NS. If the test vector is still judged as an unreliable sample, repeat the above steps until the sample is a normal sample or the maximum number of iterations is reached.

[0085] After the above data processing, statistical analysis was performed again on the two categories of Fully Paid and Charged Off, and it was found that the ratio between the two was 3:1, the data became more balanced, and most of the unreliable samples were optimized and can be used to train the model. The present invention divides the data into a training set and a test set. The ratio between the two is 8:2.

[0086] The ATD data optimization algorithm process can be described as follows:

[0087] (a) Given an initial data sample set S, the maximum number of iterations max.

[0088] (b) ADASYN synthesized samples, increasing the size of the original sample set and balancing the sample set.

[0089] (c) TBallNN is used for error detection.

[0090] (d)IfSn∈US(unreliable sample set):

[0091] (e) Return the unreliable sample and its natural neighbor set.

[0092] (f) Optimize unreliable samples using the MTFRDE optimization method.

[0093] (g) Compare the number of natural neighbor labels to select the initial optimized population Xi,G, i = 1, 2, ...., NP.

[0094] (h) Use the improved DE / rand / 2 mutation strategy to optimize the target data sample UG, as shown in formula (3).

[0095] (i) Use the crossover strategy on the optimized data sample VG+1 and the target data sample UG to obtain CG+1, as shown in formula (4).

[0096] (j)IfCG+1∈US&&does not reach the maximum number of iterations max:

[0097] (k) Return to step h.

[0098] (l)Else add CG+1 to the optimized data set.

[0099] (m)ElseAdd the data to the optimized dataset.

[0100] (n) Return the optimized data set and the algorithm ends.

[0101] In summary, the ATD algorithm in this application balances the data set through ADASYN, then locates the noise in the data through TBallNN, and optimizes the noise in the data through MTFRDE. There is a noise location and optimization process. ATD has strong independent innovation.

[0102] Furthermore, the composite feature representation is optimized as follows:

[0103] The present invention uses ResNet and TabNet to extract features respectively, and combines them with the residuals of the respective input features to obtain a multi-dimensional feature representation. The present invention converts the processed credit data into a grayscale image, and uses ResNet to extract features from the grayscale image. ResNet is usually used to process image data and can capture local and global features in the image; while TabNet is specially designed for tabular data and can capture complex feature interactions and hierarchical information; the residual combination method controllably retains the information of the original features, which is beneficial to the interpretability of the model. Even in the complex feature extraction and combination process, the characteristics of the original features can still be traced and explained. TabNet itself has good interpretability because it uses the attention mechanism to clarify which features contribute most to the output of the model, which can further improve the interpretability of the entire model. While retaining the original data features, the model can learn multi-dimensional information from different feature spaces, aiming to enhance the original features and improve the prediction accuracy and generalization ability of the model.

[0104] First, the data features after optimization processing in the present invention are 36, all of which are converted into 6*6 pixel grayscale images, and ResNet is used to extract features from the grayscale images. ResNet introduces residual blocks and skip connections, making it easier for the network to learn identity mappings. This helps the model to be robust to the effects of noise, because skip connections allow the network to skip some layers when necessary, reducing excessive sensitivity to noise. Skip connections improve the propagation of gradients in deep networks and help resist the problem of gradient vanishing, which may be more advantageous when processing noisy data.

[0105] In the ResNet model, the local features of the grayscale image data are first captured through the convolution layer, which contains 32 convolution kernels, each of which has a size of (3,3) and no pooling layer. Subsequently, three residual blocks are introduced, each of which contains two convolution layers. The internal structure of each residual block is realized through a combination of convolution layers, batch normalization, and ReLU activation functions. Figure 3 . At the end of the residual block, the input and output are added through a residual connection and then activated by an activation function. These residual blocks are designed to effectively transfer gradients and alleviate the vanishing gradient problem when training deep networks, while providing the ability to model more complex features of tabular data. Finally, the model flattens the output of the convolution part through a Flatten layer, and then adds a fully connected layer with 32 neurons, using ReLU as the activation function. The introduction of ResNet helps solve the vanishing gradient problem, reduce the noise impact caused by data visualization, and enable the final prediction model to learn the high-dimensional features of the data.

[0106] Secondly, the present invention adopts the TabNet model to receive the residual fusion features of the original features and the ResNet extracted features. The TabNet model retains the end-to-end and representation learning characteristics of DNN, and also has the advantages of the interpretability and sparse feature selection of the tree model. TabNet uses the attention mechanism in the Transformer architecture, which enables the model to adaptively focus on the important relationships between input features. This attention mechanism enables TabNet to effectively handle the complex interactions between features, thereby improving the performance of the model. And it introduces a sparsity constraint mechanism, through which the model can automatically select important features for prediction, thereby improving the generalization ability and interpretability of the model and reducing the risk of overfitting. Through this step, the present invention can extract more informative credit risk features from the original data. The TabNet model is as follows Figure 4 , where the BN layer is the batchnormalization layer, the Feature transformer layer is used for feature calculation, and the Split layer cuts the vector output by the Feature transformer layer into two parts, one for calculating the final output of the model and the other for calculating the Mask layer of the next step. The Attentive transformer layer is used to calculate the Mask layer of the current step based on the result of the previous step.

[0107] The Feature transformer layer consists of two parts, such as Figure 5 The parameters of the first half of the layers are shared, that is, they are trained together in all steps; while the parameters of the second half are not shared and are trained separately in each step. This is done considering that the same features are input for each step. Therefore, the present invention can first use the same layer to do the common part of feature calculation, and then use different layers to do the characteristic part of each step. In addition, it can be seen that residual connections are used in the layer, and multiplication by √0.5 is to ensure the stability of the network.

[0108] The internal structure of the Attentive transformer layer is as follows Figure 6 , where the Sparsemax layer can be understood as a sparse version of Softmax. However, unlike the smooth transformation of Softmax, Sparsemax achieves sparsification by directly projecting the vector into a simplex. According to the structure of the Attentive transformer layer, its calculation formula can be written as formula (5).

[0109] M[i] = sparsemax(P[i-1]·h i (a[i-1])) (5)

[0110] Where a[i-1] is the value divided by the Split layer in the previous step, and h i represents the FC+BN layer, and P[i] is the Priorscales item, which is calculated as:

[0111]

[0112] It is used to indicate the degree of use of a feature in the previous step. According to normal logic, if a feature has been used many times in the previous step, it should not be selected by the model again. Therefore, the model uses this Prior scales item to reduce the weight of such features. From the formula, we can see that if γ = 1, each feature can only be used once, and when γ increases, this constraint will become "softer".

[0113] In order to enhance the model's ability to select sparse features, an additional regularization term is introduced into the loss function. The specific form is as follows:

[0114]

[0115] This regularization term calculates an average entropy, where B is the batch size, D is the feature dimension, and M[i] represents the mask matrix of the i-th step. The purpose is to make the distribution of M[i] as close to 0 or 1 as possible. Then L sparse It reflects the sparsity of M[i], L sparse The smaller it is, the sparser M[i] is.

[0116] Finally, the present invention combines the fused feature residuals of TabNet output and TabNet input, and passes them as input features to the XGBoost-Stacking model for training. This integration process aims to extract rich features of the data from different dimensions, so that the model can have a more comprehensive understanding of the nature and potential risks of the data. The input and output are combined in a simulated residual network manner, which not only retains the important features of the original data but also allows the model to learn multi-model and multi-dimensional feature representations. Through such a process, the present invention hopes to build a more powerful and more interpretable credit risk prediction model, which comprehensively utilizes the advantages of deep learning and machine learning and improves the model's ability to model complex financial data. The specific process is as follows: Figure 3 .

[0117] Specifically, the present invention uses the features extracted by ResNet and TabNet combined with the residuals of the respective input features to obtain multi-dimensional feature representation, while retaining the original data features, obtaining important information in high-dimensional features, enhancing the original features, and the residual combination method controllably retains the information of the original features, which is beneficial to the interpretability of the model. Even in the complex feature extraction and combination process, the characteristics of the original features can still be traced and explained. This method improves the prediction accuracy and generalization ability of the model, and greatly retains the interpretability of the model.

[0118] Furthermore, the optimization of the adaptive credit risk prediction model is as follows:

[0119] The present invention uses TuRBO algorithm to optimize model hyperparameters. The full name of TuRBO algorithm is The Trust Region Bayesian Optimization Algorithm, TuRBO, that is, Bayesian optimization algorithm based on trust region. The algorithm first constructs multiple local Gaussian process models in parallel based on a series of real samples, and constructs trust regions through local Gaussian process models. Then, multiple true values ​​of optimization functions are obtained based on multiple optimal candidate points returned by multiple trust regions, and the global optimal value is selected. By comparing the true value and the Gaussian model value, the trust region is expanded or reduced, and the obtained optimal true value and the optimal candidate point are added to the training sample, and the above steps are repeated until the maximum number of iterations is reached. The algorithm has both global search capability and adaptability, so that it has better efficiency and performance.

[0120] The algorithm adopts an implicit multi-armed bandit strategy to distribute samples among multiple local models. In this way, the algorithm can avoid excessive exploration in the global scope, while making full use of the second-order convergence of the trust region method in the local scope to perform efficient solution. This method has shown excellent results in problems such as hyperparameter optimization of machine learning. The role of the implicit multi-armed bandit in this case is to help the present invention to effectively distribute samples among multiple local optimization models to achieve global optimization of large-scale high-dimensional problems. The synergy of strategies such as local Gaussian process models, implicit multi-armed bandit sampling strategies, and local proxy models improves the efficiency and accuracy of search. These characteristics make the TuRBO algorithm highly competitive in the field of machine learning hyperparameter optimization, and can find the global optimal hyperparameter combination more quickly, thereby improving model performance.

[0121] During the optimization process, the trust region method adjusts the size of the trust region according to the effect of each iteration. Specifically, the present invention defines the ratio:

[0122]

[0123] Among them, f(xk) is the value of the objective function at the current iteration point, f(xk+p k ) is the value of the objective function at the new iteration point, mk(0) and mk(p k ) are the values ​​of the GP model function at the current iteration point and the new iteration point respectively. k When it is close to 1, it indicates that the model function and the objective function are close. At this time, the present invention considers that the current iteration step is effective and the current trust region radius can be expanded. k When it is close to 0, it indicates that the model function and the objective function are far apart. In this case, the invention needs to reduce the radius of the trust region in the next iteration. k When it is between 0.25 and 0.75, it means that after this step is taken, it is between "reliable" and "unreliable", and the current trust region radius can be maintained and the maximum number of iterations can be repeated. In this way, by dynamically adjusting the size of the trust region, the trust region method can find a good balance between global optimization and local search, thereby improving the efficiency and accuracy of optimization.

[0124] The specific process of the TuRBO trust region-based Bayesian optimization algorithm is as follows:

[0125] (a) Given an initial training sample x1…x n .

[0126] (b) Construct N local Gaussian process models based on the training samples.

[0127] (c) Construct N trust regions through the local Gaussian process model.

[0128] (d) Given the initial trust region scale parameter δ, the maximum scale δmax, the initial point x n .

[0129] (e) while the convergence criterion is not reached do: / / The criterion is the maximum number of iterations of the TuRBO algorithm and δ is between δmax and δmin.

[0130] (f) Calculate the ratio p of the true value to the Gaussian model value by the formula k .

[0131] (g) Update the trust region scale parameter δ:

[0132]

[0133] (h) Update the initial point x n is the local optimal point and returns the local optimal point.

[0134] (i) end while

[0135] (j) Select the best points among N local Gaussian process models and add them to training samples.

[0136] (k) Repeat steps b to j until the optimal solution is obtained or the maximum number of iterations is reached.

[0137] (l) Return to the optimal point and the algorithm ends.

[0138] The TuRBO algorithm has parallelization capabilities and can evaluate multiple candidate points simultaneously in each iteration step, thereby increasing the speed of hyperparameter search and effectively processing high-dimensional parameter spaces. By adaptively adjusting the size of the trust region, the TuRBO algorithm can search more effectively in different hyperparameter spaces, enabling it to find the global optimal solution faster.

[0139] The final prediction model uses the XGBoost-Stacking model. Initially, the data set is trained in XGBoost, CatBoost, Random Forest, and logistic regression, and it is found that XGBoost has the best training effect. Therefore, CatBoost, Random Forest, and logistic regression are used as the base learners of the stacking model, XGBoost is used as the meta-learner of the stacking model, and logistic regression is integrated into the stacking model as the base classifier in order to increase the diversity of the model, improve robustness and generalization ability, balance the interpretability and prediction performance of the model, and improve the performance of the overall model in certain cases. The specific process is as follows Figure 7 .

[0140] XGBoost is an efficient and scalable machine learning algorithm, especially suitable for regression and classification problems. It is based on the Gradient Boosting DecisionTree (GBDT) algorithm, and its performance is improved through optimization and improvement. XGBoost mainly consists of two parts: loss function and regularization term. Its loss function represents the fit of the model, and the regularization term is used to control the complexity of the model. XGBoost solves the optimal model parameters by optimizing the objective function. Its objective function consists of the loss function and the regularization term, and the general form is as follows:

[0141]

[0142] Where l represents the loss function, which measures the difference between the predicted value and the true value. k ) represents the regularization term, which is used to control the complexity of the model. CatBoost and XGBoost are both variants of gradient boosting trees, but they can automatically adjust the learning rate and dynamically update it during training to better fit the data.

[0143] Random Forest is an ensemble learning method that is based on decision trees and uses multiple decision trees to make predictions or classifications. The outputs of these decision trees are integrated by voting or averaging, and the result of the entire random forest is finally output. The core idea is to reduce overfitting and improve the generalization ability of the model through random feature selection and voting of multiple trees. For classification problems, the prediction model of random forest can be expressed as:

[0144]

[0145] Where Ntrees is the number of trees in the forest, f i (X) is the predicted output of the i-th tree, I is the indicator function, and c is the classification category. Random forest improves the prediction ability of the model by constructing multiple decision trees and integrating their prediction results, and has good robustness and generalization ability.

[0146] Stacking is an ensemble learning technique that uses the prediction results of multiple different base classifiers as new features to input into the meta classifier to further improve the performance of the overall model. First, in the first layer, multiple base classifiers of different types are trained to model and predict the data. The training set is used for cross-validation, and the prediction results generated by each base classifier are retained as new features instead of being used directly for the final prediction. Based on the new features generated in the first layer, the meta classifier XGBoost is used to combine these prediction results. The output of each base classifier in the first layer is used as the input feature of the meta classifier, and then the meta classifier is used to make an overall prediction. In the test phase, the trained base classifiers are used to generate new features, and then the meta classifier is used to predict these features to obtain the final integrated model output. Stacking can combine the advantages of multiple different models, improve the prediction performance and generalization ability of the model through the integration of meta classifiers, reduce the risk of overfitting, and increase the robustness of the model. In the XGBoost-Stacking model of the present invention, each base model uses 4-fold cross validation, only k-1 folds are trained each time, and then the prediction values ​​of the remaining 1 fold are used as new data. Each model obtains the prediction value in this way, and then merges them by column.

[0147] Example 3

[0148] Based on Example 2, the method further includes:

[0149] The credit risk prediction model is evaluated using accuracy, precision, recall and F1-score. The following is a specific experiment. It includes the following parts.

[0150] 1. Experimental environment

[0151] The platform used in this experiment is a server composed of an Intel Xeon E5-2603 v4 1.7GHz processor, a Tesla K80 GPU, and 32.0GB of RAM. JDK 1.8 and Python 3 are used in combination with the Pytorch framework as the operating environment for data screening and cleaning as well as credit risk prediction.

[0152] 2. Experimental data

[0153] The data of this paper comes from the public data of LendingClub from 2007 to 2020. In order to ensure the timeliness of the research, this paper selects data with a time span close to the present. After comprehensive consideration, 887,379 relatively new data were intercepted for studying the credit prediction model. The original data set contains 887,379 samples and 74 features, and the processed data set contains 252,971 samples and 36 features. It is divided into training set and test set in a ratio of 8:2.

[0154] 3. Evaluation Metrics

[0155] Confusion matrix: In the confusion matrix, the true classification and predicted classification can be analyzed, and the number of observations judged to be correctly classified and the number of observations judged to be incorrectly classified can be counted.

[0156]

[0157] This experiment selected accuracy, precision, recall and F1-score as evaluation indicators. In the prediction results of the model, the ratio of the number of true bad customers and true good customers to the number of all samples in the prediction results is called accuracy, that is:

[0158]

[0159] Precision represents the proportion of correct results in the model's predictions, that is, the proportion of correct results in the model's predictions of the positive class; recall represents the proportion of correct results in the model's predictions to all actual positive results, that is, the proportion of correct predictions by the model in all actual positive results. The calculation of precision and recall is as follows:

[0160]

[0161] When the two indicators of precision and recall are contradictory and no decision can be made, an indicator is usually needed to comprehensively consider the relevant indicators. This comprehensive indicator is expressed as F-score (F1 value), which is regarded as the weighted average of the model's accuracy and recall. Its maximum value is 1 and its minimum value is 0. F-score is calculated as follows:

[0162]

[0163] 4. Experimental results

[0164] In order to test the ATD data optimization performance of ATDST, the most commonly used SOMTE algorithm, ADASYN algorithm and BorderlineSMOTE algorithm were selected for comparison with the ATD algorithm of the present invention. The algorithm was combined with XGBoost to reflect the data optimization effect by comparing the model performance. The results are shown in Table 1.

[0165] Table 1 Comparison results of model ATD data optimization

[0166]

[0167] Compared with traditional methods, the ATD method does not simply delete noise and boundary samples, but generates synthetic samples through ADASYN, uses TBallNN technology to identify unreliable samples, and then uses the MTFRDE method to optimize these samples. This processing method helps to retain key information and improve the model's understanding of noise and boundary data, thereby improving model performance. The ATD+XGBoost method shows good performance in accuracy, precision, and F1 score. Compared with other sampling methods combined with XGBoost, the ATD method shows higher performance. This confirms the effectiveness of the ATD method on this dataset. The process of the ATD method is specially designed for unbalanced datasets and noisy datasets. By processing minority class samples, identifying and optimizing noise and boundary samples, the dataset can be better balanced. Therefore, when dealing with data optimization problems, the ATD method may be more advantageous and applicable.

[0168] In order to test the optimization performance of TuRBO algorithm in ATDST, the most commonly used model hyperparameter optimization algorithms in the field of credit risk prediction, namely Bayesian Optimization algorithm, GridSearch algorithm and GeneticAlgorithm algorithm, were selected for comparison with TuRBO algorithm of the present invention, and the algorithm was combined with XGBoost to reflect the data optimization effect by comparing the model performance. The results are shown in Table 2.

[0169] Table 2 Comparison results of model hyperparameter optimization

[0170]

[0171] From Table 2, we can see that based on LendingClub data, TuRBO achieved an accuracy of 0.955 on the XGBoost model, which is slightly higher than the accuracy of other algorithms. TuRBO completed the optimization in a relatively short time, taking only 880 seconds, which is much less than GridSearch and GeneticAlgorithm. This shows that TuRBO achieves higher performance in a shorter time. TuRBO searches based on multiple local Gaussian process models, which helps to conduct a global search in the hyperparameter space and avoid falling into the local optimal solution. By dynamically adjusting the trust region and using previous sampling information, it shows adaptability and high efficiency, which helps to find a better hyperparameter combination within a relatively small number of iterations. By comparing the true value and the Gaussian model value to adjust the trust region size, this strategy makes the optimization function converge faster. In general, TuRBO has shown certain advantages in this specific hyperparameter optimization task, with high accuracy and relatively short optimization time.

[0172] In order to test the optimization performance of the ATDST model, the benchmark models XGBoost, CatBoost, RandomForest and Logistic regression models as well as PSO-LightGBM and Log-CNN models were selected for comparison on the LendingClub dataset. The algorithm was combined with XGBoost to reflect the model optimization effect by comparing the model performance.

[0173] Table 3. Comparison results of model performance

[0174]

[0175] On the LendingClub dataset, the ATDST model showed the highest accuracy and overall performance. The optimization process of the ATDST model combines the advantages of deep learning and machine learning, and uses multi-dimensional feature representation to improve the model's ability to model complex financial data. The design of the XGBoost-Stacking model gives the model high robustness, generalization ability, and predictive performance, while improving the overall model performance under certain circumstances. By comprehensively utilizing various models and algorithms, the ATDST model has shown excellent results in credit risk prediction, providing an effective method for solving practical problems.

[0176] In summary, the ATDTS model proposed in the present invention improves data quality, optimizes the model's hyperparameters, and significantly improves prediction accuracy. The prediction results are improved in accuracy, precision, recall, and F1-score compared with existing methods.

[0177] Example 4

[0178] Based on the same inventive concept as Example 1, a credit risk prediction system is provided, the system comprising:

[0179] a data collection module configured to collect credit data;

[0180] a credit data optimization module configured to optimize the credit data using an ATD algorithm;

[0181] A feature enhancement module, configured to combine ResNet and TabNet to perform feature enhancement on the optimized credit data;

[0182] A credit risk prediction model building module is configured to build a credit risk prediction model and train the credit risk prediction model using enhanced features, wherein the credit risk prediction model is hyperparameter optimized using a TuRBO trust region-based Bayesian optimization algorithm;

[0183] The credit risk prediction module is configured to use the trained credit risk prediction model to perform credit risk prediction.

[0184] Example 5

[0185] Based on the same inventive concept as that of Example 1, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the credit risk prediction method provided by the embodiment of the present invention is implemented. Based on such an understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0186] Example 6

[0187] Based on the same inventive concept as that of Example 1, a terminal is provided, including a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor executes the credit risk prediction method provided by the embodiment of the present invention when executing the computer instructions.

[0188] The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0189] Embodiments of the subject matter and functional operations described in this specification may be implemented in: tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode and transmit information to a suitable receiver device for execution by the data processing device.

[0190] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0191] Processors suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, just to name a few.

[0192] It should be understood that each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A credit risk prediction method, characterized in that: The method comprises the following steps: S1. Collect credit data; S2. Optimizing the credit data by using the ATD algorithm; S3. Combine ResNet and TabNet to enhance the features of the optimized credit data; S4. Build a credit risk prediction model and use the enhanced features to train the credit risk prediction model, wherein the TuRBO trust region-based Bayesian optimization algorithm is used to optimize the hyperparameters of the credit risk prediction model; S5. Use the trained credit risk prediction model to perform credit risk prediction.

2. A credit risk prediction method according to claim 1, characterized in that: The credit risk prediction model is the XGBoost-Stacking model.

3. A credit risk prediction method according to claim 1, characterized in that: The collection of credit data includes: Process missing values ​​in credit data.

4. A credit risk prediction method according to claim 1, characterized in that: The optimizing the credit data by using the ATD algorithm includes: Generate synthetic samples through ADASYN oversampling algorithm; The TBallNN error detection technique is used to identify noise samples in synthetic samples; The noise samples are optimized by using the MTFRDE multi-objective feature-preserving differential evolution algorithm.

5. A credit risk prediction method according to claim 1, characterized in that: The above-mentioned combination of ResNet and TabNet is used to enhance the features of the optimized credit data, including: ResNet and TabNet are used to extract features of credit data respectively; The features extracted by ResNet and TabNet are combined with their respective input feature residuals to obtain multi-dimensional feature representation.

6. A credit risk prediction method according to claim 1, characterized in that: TuRBO trust region-based Bayesian optimization algorithm is used to optimize the hyperparameters of the credit risk prediction model, including: Based on a series of real samples, multiple local Gaussian process models are constructed in parallel, and trust regions are constructed through the local Gaussian process models; According to the multiple optimal candidate points returned by the multiple trust regions, multiple true values ​​of the optimization function are obtained and the global optimal value is selected; By comparing the true value and the Gaussian model value, choose to expand or reduce the trust region, and add the optimal true value and the optimal candidate point to the training sample; Repeat the above steps until the maximum number of iterations is reached.

7. A credit risk prediction method according to claim 1, characterized in that: The method further comprises: The credit risk prediction model is evaluated using accuracy, precision, recall and F1-score.

8. A credit risk prediction system, characterized in that: The system comprises: a data collection module configured to collect credit data; a credit data optimization module configured to optimize the credit data using an ATD algorithm; A feature enhancement module, configured to combine ResNet and TabNet to perform feature enhancement on the optimized credit data; A credit risk prediction model building module, configured to build a credit risk prediction model and train the credit risk prediction model using enhanced features, wherein the credit risk prediction model is hyperparameter optimized using a TuRBO trust region-based Bayesian optimization algorithm; The credit risk prediction module is configured to use the trained credit risk prediction model to perform credit risk prediction.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the credit risk prediction method described in any one of claims 1 to 7 is implemented.

10. A terminal comprising a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, characterized in that: When the processor runs the computer instructions, it executes the credit risk prediction method described in any one of claims 1-7.

Images