Unbalanced credit default prediction method and device based on self-attention mechanism

Abstract

The invention provides an unbalanced credit default prediction method and device based on a self-attention mechanism, the method comprises a data preprocessing step, a model training step and a credit default prediction step, a deep neural network model based on the self-attention mechanism is trained, and the model is used for credit default prediction. According to the model, an advanced self-attention mechanism is utilized, key information in sparse and complex financial credit data can be effectively extracted, and a robust prediction mechanism is deduced by learning internal modes in the key information. Furthermore, in order to deal with the imbalance of the financial credit data, an f1 score is introduced in the model training step to serve as a training evaluation index of the model to assist model training, so that the optimal model state is found, default users are identified more accurately, effective learning of the unbalanced financial credit data is realized, and the user experience is improved. And the accuracy of credit default prediction is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of credit default prediction, and in particular relates to an unbalanced credit default prediction method and device based on a self-attention mechanism. Background Art

[0002] Credit default prediction is the core task of credit risk management, which mainly predicts whether a borrower will default. Traditional credit default prediction mainly uses statistical methods, such as Logistic regression, decision tree, support vector machine, etc., which have been widely used in practice. However, these traditional methods often have limited performance when facing complex and nonlinear credit data. In addition, compared with conventional tabular data, financial credit data usually presents the characteristics of low correlation between features, high sparsity and class imbalance. These traditional methods cannot handle the problem of data imbalance well, that is, the difference in the number of positive samples (default) and negative samples (non-default) is too large, which will affect the prediction performance of the model.

[0003] In order to overcome these problems, researchers began to try to use deep learning methods for credit default prediction. Deep learning can automatically learn and extract features, and has good processing capabilities for complex and nonlinear data. For example, artificial neural networks, convolutional neural networks, etc. have been applied in credit default prediction. Nevertheless, deep learning methods also have some problems that need to be solved. First, credit data is complex, high-dimensional and sparse, so effective information is difficult to be effectively learned, resulting in poor performance. At the same time, for unbalanced data sets, neural networks tend to ignore minority samples.

[0004] A common method for dealing with imbalanced data is to use resampling techniques, such as random oversampling, random undersampling, SMOTE, etc. However, such methods are prone to overfitting or underfitting. In addition, methods have been proposed to adjust the punishment of positive and negative samples, such as weighted cross entropy, but this solution has limited performance. Recently, some researchers have begun to explore the use of generative adversarial networks to solve the problem of data imbalance, but the model is difficult to train and exhibits high complexity.

[0005] In general, credit default prediction is a challenging problem that requires processing complex, nonlinear, and unbalanced data. Although many methods have been proposed, there are still many problems to be solved. In order to solve the problem of unbalanced credit data in credit default prediction and improve the accuracy of credit default prediction, a new credit default prediction method is needed. Summary of the invention

[0006] The present invention is made to solve the above problems, and aims to provide a credit default prediction method and device that can effectively solve the problem of financial credit data imbalance and improve credit default prediction. The present invention adopts the following technical solutions:

[0007] The present invention provides an unbalanced credit default prediction method based on a self-attention mechanism, which is characterized in that it includes the following steps: a data preprocessing step: preprocessing the financial credit data used for model training to obtain a training set; a model training step: training a deep neural network model based on a self-attention mechanism based on the training set and an early stopping strategy to obtain the trained deep neural network model, wherein the deep neural network includes a first fully connected layer, a batch normalization layer, a self-attention layer and a second fully connected layer connected in sequence, and during the training process, the learning rate of the deep neural network model is optimized and adjusted using the f1 score; a credit default prediction step: inputting the financial credit data to be predicted into the trained deep neural network model to obtain a prediction result of credit default.

[0008] The unbalanced credit default prediction method based on the self-attention mechanism provided by the present invention may also have such a technical feature, wherein the model training step includes the following sub-steps: initializing the parameters used for model training, including setting the initial value of the learning rate of the deep neural network model and the initial value of the best f1 score; training the deep neural network model based on the training set and the early stopping strategy until a predetermined number of training rounds is reached and the accuracy of the model is no longer improved, thereby obtaining the trained deep neural network model, wherein the training rounds are reset when the f1 score is less than or equal to the best f1 score, and when the f1 score is greater than the best f1 score and reaches the predetermined number of training rounds, the learning rate of the deep neural network model is updated.

[0009] The unbalanced credit default prediction method based on the self-attention mechanism provided by the present invention may also have such a technical feature, wherein the initialized parameters also include the training round, the weight, bias and learning rate of the deep neural network model, the model training is performed based on the Adam optimizer, and the initialized parameters also include the learning rate, first-order moment estimation and second-order moment estimation in the Adam optimizer, and the step of training the SA-DNN model based on the training set and the early stopping strategy includes: inputting a set of training sets into the deep neural network model, and calculating its gradient based on the weight and bias of the deep neural network model; updating the first-order moment estimation and the second-order moment estimation of the momentum based on the gradient; correcting the deviation of the first-order moment estimation and the second-order moment estimation; and updating the gradient based on the correction. The second-order moment estimate after correction is used to update the weights and biases of the deep neural network model; based on the prediction result of the deep neural network model, the f1 score at this time is calculated; it is determined whether the f1 score is greater than the optimal f1 score, and when the judgment is yes, the optimal f1 score is updated to the f1 score, and when the judgment is no, the training round is reset; it is determined whether the training round reaches a first predetermined value, and when the judgment is yes, the learning rate of the deep neural network model is updated, and the learning rate in the Adam optimizer is updated; it is determined whether the training round reaches a second predetermined value and the f1 score of this iteration is not improved compared with the f1 score of the previous iteration, and when the judgment is yes, the model training is stopped and the trained deep neural network model is output.

[0010] The unbalanced credit default prediction method based on the self-attention mechanism provided by the present invention may also have such a technical feature, wherein the calculation formula of the f1 score is: f1=2*(Precision*Recall) / (Precision+Recall), wherein Precision is the precision rate, and its calculation formula is Precision=TP / (TP+FP), Recall is the recall rate, and its calculation formula is Recall=TP / (TP+FN), TP and FP are indicators of the real default users predicted as default and non-default prediction labels, respectively, and FN is a non-default sample with a default user prediction label.

[0011] The unbalanced credit default prediction method based on the self-attention mechanism provided by the present invention may also have such a technical feature, wherein, in the model training step, the gradient of the deep neural network model is calculated based on the weight and bias, and the expression is: Wherein, w and b are the weight and bias of the deep neural network model respectively, and the first-order moment estimation based on the gradient update momentum is expressed as: w =β1·m w +(1-β1)·g w , mb =β1·m b +(1-β1)·g b , where m w is the first moment estimate of w, m b is the first-order moment estimate of b, β1 is the exponential decay rate of the first-order moment estimate, and the second-order moment estimate of the momentum is updated based on the gradient, expressed as: In the formula, v w is the second moment estimate of w, and v b is the second-order moment estimate of b, β2 is the exponential decay rate of the second-order moment estimate, correcting the deviation between the first-order moment estimate and the second-order moment estimate, and the expression is: In the formula, is β1 to the power of t, is β2 to the power of t, where t is the current number of iterations, is the corrected first-order moment estimate of w, is the corrected first-order moment estimate of b, is the corrected second moment estimate of w, is the corrected second-order moment estimate of b. Based on the corrected second-order moment estimate, the weights and biases of the deep neural network model are updated. The expression is: Where η is the learning rate and ∈ is a predetermined value used to prevent the denominator from being zero.

[0012] The unbalanced credit default prediction method based on the self-attention mechanism provided by the present invention may also have such a technical feature, wherein, in the model training step, the initial value of the best f1 score is 0, the first predetermined value for determining the training round is 10, and the expression for updating the learning rate of the deep neural network model is: ξ i+1 =ξ i *0.1, the second predetermined value is 20.

[0013] The unbalanced credit default prediction method based on the self-attention mechanism provided by the present invention may also have such a technical feature, wherein, in the deep neural network model, the first fully connected layer is a fully connected layer based on the Relu activation function, and the second fully connected layer is a fully connected layer based on the Sigmoid activation function.

[0014] The unbalanced credit default prediction method based on the self-attention mechanism provided by the present invention may also have such a technical feature, wherein, in the model training step, the pre-processed financial credit data is input into the deep neural network model, and the output of the first fully connected layer is: X 1=max(0,W1*X+b1), where W1 and b1 are the weight and bias of the first fully connected layer, respectively. The batch normalization layer is used to perform batch normalization on the output of the first fully connected layer, and its output is: In the formula, μ B and σ B are the mean and standard deviation of the batch, ∈ is a predetermined value used to prevent the denominator from being 0, γ and β are learnable parameters, and the self-attention layer first linearly maps the output of the batch normalization layer to the query matrix Q, the key matrix K, and the value matrix V: Where W Q , W K and W V are the corresponding weights, b Q 、b K and b K are the corresponding bias terms respectively. The self-attention layer calculates the attention value α of each two input vectors of the query matrix Q and the key matrix K, thereby obtaining the attention matrix: Attention=(α) (i,j) =softmax(Q*K), the self-attention layer calculates the output vector corresponding to each two input vectors: X attention =Attention*V, the second fully connected layer activates the output of the self-attention layer: y predict =sigmoid(W4*X attention +b4), where W4 and b4 are the weight and bias of the second fully connected layer respectively.

[0015] The unbalanced credit default prediction method based on the self-attention mechanism provided by the present invention may also have such a technical feature, wherein, in the data preprocessing step, the financial credit data is normalized, and the expression is as follows: X=(X-min(X)) / (max(X)-min(X)), where X is the feature of the financial credit data.

[0016] The present invention provides an unbalanced credit default prediction device based on a self-attention mechanism, which is characterized by comprising: a data preprocessing unit, which preprocesses financial credit data used for model training to obtain a training set; a model training unit, which trains a deep neural network model based on a self-attention mechanism based on the training set and an early stopping strategy to obtain the trained deep neural network model, wherein the deep neural network includes a first fully connected layer, a batch normalization layer, a self-attention layer and a second fully connected layer connected in sequence, and during the training process, the learning rate of the deep neural network model is optimized and adjusted using the f1 score; and a credit default prediction unit, which is used to input the financial credit data to be predicted into the trained deep neural network model to obtain a prediction result of credit default.

[0017] Function and Effect of the Invention

[0018] According to the unbalanced credit default prediction method and device based on the self-attention mechanism of the present invention, the method includes a data preprocessing step, a model training step and a credit default prediction step, wherein a deep neural network model based on the self-attention mechanism is trained and the model is used to predict credit default. The model uses an advanced self-attention mechanism to effectively extract key information from sparse and complex financial credit data, and derives a robust prediction mechanism by learning the inherent patterns in these key information. Further, in order to cope with the imbalance of financial credit data, the f1 score is introduced in the model training step as a training evaluation indicator of the model to assist model training, the purpose of which is to find the optimal model state to more accurately identify default users, thereby achieving effective learning of unbalanced financial credit data and improving the accuracy of credit default prediction. In addition, the f1 score can assist the model to quickly converge to the state of optimally distinguishing default samples, which can reduce the time required for model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of an unbalanced credit default prediction method based on a self-attention mechanism in an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of the structure of a deep neural network model based on a self-attention mechanism in an embodiment of the present invention;

[0021] Figure 3 is a flow chart of the model training step S3 in an embodiment of the present invention;

[0022] Figure 4 is a flow chart of step S3-2 in an embodiment of the present invention;

[0023] Figure 5 It is a block diagram of an unbalanced credit default prediction device based on a self-attention mechanism in an embodiment of the present invention.

[0024] Reference numerals:

[0025] Unbalanced credit default prediction device 100; prediction model storage unit 101; training data acquisition unit 102; data preprocessing unit 103; prediction model training unit 104; credit default prediction unit 105; credit default prediction result output unit 106; credit default prediction control unit 107. DETAILED DESCRIPTION

[0026] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the unbalanced credit default prediction method and device based on the self-attention mechanism of the present invention are specifically described below in combination with embodiments and drawings.

[0027] <Example>

[0028] Figure 1 is a flow chart of the unbalanced credit default prediction method based on the self-attention mechanism in this embodiment.

[0029] like Figure 1 As shown in Figure 2, the unbalanced credit default prediction method based on the self-attention mechanism includes the following steps:

[0030] Data preprocessing step S1: Preprocess the financial credit data used for model training to obtain a training set.

[0031] Model construction step S2: Construct a deep neural network (SA-DNN) model based on the self-attention mechanism.

[0032] Model training step S3: Use the training set to train the SA-DNN model to obtain a trained SA-DNN model, wherein the learning rate of the SA-DNN model is optimized and adjusted using the f1 score.

[0033] Credit default prediction step S4: input the financial credit data to be predicted into the trained SA-DNN model to obtain the prediction result of credit default.

[0034] The above steps will be described in detail below.

[0035] Data preprocessing step S1: Preprocess the financial credit data used for model training to obtain a training set.

[0036] Among them, for financial credit data (credit default data) {X, y}, where X = {x1, x2, ..., x n},

[0037] First, the financial credit data is normalized to improve the convergence efficiency of the model in the subsequent model training process. The normalization expression is as follows:

[0038] X=(X-min(X)) / (max(X)-min(X)), (1)

[0039] Where X is the input feature, where x1,x2,...,x n are the 1st, 2nd, ...nth features respectively; y is the credit default label, which takes the value of 0 or 1.

[0040] Model construction step S2: Construct a deep neural network (SA-DNN) model based on the self-attention mechanism.

[0041] Compared with conventional tabular data, financial credit data usually exhibits the significant characteristics of low correlation between features, high sparsity, and class imbalance, which increase the difficulty of accurately identifying defaulting users. To solve this problem, this embodiment provides a SA-DNN model designed specifically for mining and learning effective information in financial credit data.

[0042] Figure 2 It is a structural diagram of the deep neural network model based on the self-attention mechanism in this embodiment.

[0043] like Figure 2 As shown, the SA-DNN model includes a first fully connected layer ( Figure 2 FC (20)), batch normalization layer (Norm), self-attention layer (Self-attention), the second fully connected layer ( Figure 2 FC(1) in the figure). The first fully connected layer is a fully connected layer based on the Relu activation function; the second fully connected layer is a fully connected layer based on the Sigmoid activation function.

[0044] Model training step S3: Use the training set to train the SA-DNN model to obtain a trained SA-DNN model, wherein the learning rate of the SA-DNN model is optimized and adjusted using the f1 score.

[0045] During the model training process, the normalized financial credit data X is input into the SA-DNN model, and the first fully connected layer obtains the financial credit data X and performs the following operations:

[0046] X 1 =max(0,W1*X+b1). (2)

[0047] Where W1 and b1 are the weight and bias of the first fully connected layer, respectively, and * represents matrix multiplication.

[0048] The batch normalization layer is applied to the output of the first fully connected layer X 1Batch normalization is performed to accelerate model convergence during training. The expression is as follows:

[0049]

[0050] In the formula, μ B and σ B are the mean and standard deviation of the batch, ∈ is a predetermined minimum number to prevent the denominator from being 0, and γ and β are learnable parameters.

[0051] The calculation method of the self-attention layer includes the calculation of weights and context vectors. Specifically, the self-attention layer first converts the output of the batch normalization layer Linear mapping to query matrix Q, key matrix K and value matrix V:

[0052]

[0053]

[0054]

[0055] Where W Q , W K and W V are the corresponding weights, b Q 、b K and b K are the corresponding bias terms respectively.

[0056] Then, the self-attention layer calculates the attention value α of every two input vectors of the query matrix Q and the key matrix K, thereby obtaining the attention matrix, which is expressed as follows:

[0057] Attention=(α) (i,j) =softmax(Q*K). (7)

[0058] Then, the self-attention layer calculates the output vector of the self-attention layer corresponding to each attention vector α based on the attention matrix Attention. The expression is as follows:

[0059] X attention =Attention*V, (8)

[0060] Among them, X attention Each element of The associated information of other elements in the image is used to obtain the key information extracted by the neural network.

[0061] The second fully connected layer is used to learn the mapping relationship between features and true labels, and its expression is:

[0062] y predict=sigmoid(W4*X attention +b4), (9)

[0063] Where W4 and b4 are the weight and bias of the second fully connected layer, respectively.

[0064] In this embodiment, the f1 score is used to optimize and adjust the learning rate of the SA-DNN to assist the model to quickly converge to a state that optimally distinguishes default samples. The f1 score is the harmonic mean of Precision and Recall.

[0065] Figure 3 It is a flow chart of the model training step S3 in this embodiment.

[0066] like Figure 3 As shown, step S3 specifically includes the following sub-steps:

[0067] Step S3-1: Initialize parameters for model training.

[0068] In this embodiment, the initialized training parameters include the weight w, bias b and learning rate ξ0 of the SA-DNN model, and the best f1 score f1 best =0, training round epoch count =0; for the Adam optimizer, initialize its parameter m w =0,v w =0,m b =0,v b =0,t=0, where t represents the number of iterations, m w represents the first-order moment estimate of weight w, m b represents the first-order moment estimate of the bias b, v w represents the second-order moment estimate of weight w, m b Represents the second-order moment estimate of the bias b.

[0069] Step S3-2: Train the SA-DNN model based on the training set and the early stopping strategy until a predetermined number of training rounds is reached and the accuracy of the model no longer improves, wherein the f1 score is used to optimize and adjust the learning rate of the SA-DNN.

[0070] Figure 4 This is a flowchart of step S3-2 in this embodiment.

[0071] like Figure 4 As shown, step S3-2 specifically includes the following sub-steps:

[0072] Step S3-2-1: Input a training set into the SA-DNN model and calculate its gradient based on the weights and biases of the SA-DNN model:

[0073]

[0074] Step S3-2-2: Update the first-order moment estimate of momentum based on the gradient of the model:

[0075] m w =β1·m w +(1-β1)·g w , (11)

[0076] m b =β1·m b +(1-β1)·g b , (12)

[0077] Where β1 is the exponential decay rate of the first-order moment estimate.

[0078] Step S3-2-3: Update the second-order moment estimate of momentum based on the gradient of the model:

[0079]

[0080]

[0081] Where β2 is the exponential decay rate of the second-order moment estimate.

[0082] Step S3-2-4: Correct the deviation of the first-order moment estimation and the second-order moment estimation of momentum:

[0083]

[0084]

[0085] In the formula, is β1 to the power of t, It is β2 raised to the power of t, where t is the current iteration number.

[0086] Step S3-2-5: Update the weights and biases of the SA-DNN model based on the corrected second-order moment estimate:

[0087]

[0088]

[0089] Step S3-2-6: Calculate the output y of the SA-DNN model at this time predict The f1 score is:

[0090] f1=2*(Precision*Recall) / (Precision+Recall), (19)

[0091] Among them, precision Precision = TP / (TP+FP), recall Recall = TP / (TP+FN), TP and FP are indicators of the real defaulting users being predicted as default and non-default prediction labels, respectively, and FN is the non-defaulting sample with the defaulting user prediction label.

[0092] In the learning process of traditional neural networks, it is easy to ignore minority samples, resulting in limited prediction performance for unbalanced data sets. Therefore, in this embodiment, the f1 score is used to tune the learning rate during model training to improve the prediction accuracy of the SA-DNN model for minority samples. It can be seen that the f1 score, as the harmonic mean of Precision and Recall, fully considers the prediction accuracy of default samples.

[0093] Step S3-2-7: Determine whether f1>f1 best .

[0094] Step S3-2-8: If the judgment in step S3-2-7 is yes, set f1 best =f1, and set epoch count =epoch count +1.

[0095] Step S3-2-9: If the judgment in step S3-2-7 is no, set the epoch count =0.

[0096] Step S3-2-10: Determine epoch count Whether the accumulation reaches 10, that is, whether there is an epoch count =10.

[0097] Step S3-2-11: When the judgment in step S3-2-10 is yes, update the learning rate of the SA-DNN model:

[0098] ξ i+1 =ξ i *0.1, (20)

[0099] Step S3-2-12: Update the learning rate ξ in the Adam optimizer i+1 .

[0100] Step S3-2-13: Determine epoch count Whether the number has reached 20, that is, whether there is an epoch count =20.

[0101] Step S3-2-14: When the judgment in step S3-2-13 is yes, further determine whether the f1 score of this iteration is greater than the f1 score of the previous iteration.

[0102] Step S3-2-15: When the judgment in step S3-2-14 is no, that is, the f1 of the model has not been further improved after 20 rounds, the model training is stopped to obtain the trained SA-DNN model.

[0103] Step S3-2-16: When step S3-2-10 is judged as no, or step S3-2-13 is judged as no, or step S3-2-15 is judged as yes, return to step S3-2-1 and perform iterative training again.

[0104] Step S3-3: Output the trained SA-DNN model.

[0105] Credit default prediction step S4: input the financial credit data to be predicted into the trained SA-DNN model to obtain the prediction result of credit default.

[0106] After inputting the financial credit data to be predicted, the processing process of the data by each layer of the SA-DNN model is similar to that described in step S3, so it will not be repeated.

[0107] Figure 5 is a block diagram of an unbalanced credit default prediction device based on a self-attention mechanism in this embodiment.

[0108] like Figure 5 As shown, this embodiment also provides an unbalanced credit default prediction device 100 based on the self-attention mechanism corresponding to the above method, which includes a prediction model storage unit 101, a training data acquisition unit 102, a data preprocessing unit 103, a prediction model training unit 104, a credit default prediction unit 105, a credit default result output unit 106, and a credit default prediction control unit 107.

[0109] Among them, the prediction model storage unit 101 stores the above-mentioned SA-DNN model. The training data acquisition unit 102 is used to obtain financial credit data for model training from other devices. The data preprocessing unit 103 preprocesses the acquired financial credit data according to the method of the above-mentioned step S1 to form a training set. The prediction model training unit 104 uses the training set to train the stored SA-DNN model according to the method of the above-mentioned step S3, and stores the trained SA-DNN model in the prediction model storage unit 101. The credit default prediction unit 105 inputs the financial credit data to be predicted into the trained SA-DNN model according to the method of the above-mentioned step S4 to predict and obtain the prediction result of credit default. The credit default prediction unit 105 outputs the prediction result to a display device or other device. The credit default prediction control unit 107 is used to control the work of the above-mentioned functional units.

[0110] In this embodiment, parts not described in detail are well-known technologies in the art.

[0111] Example Function and Effect

[0112] According to the unbalanced credit default prediction method and device based on the self-attention mechanism provided by this embodiment, the method includes a data preprocessing step, a model training step and a credit default prediction step, wherein a deep neural network model based on the self-attention mechanism is trained and the model is used to predict credit default. The model uses an advanced self-attention mechanism to effectively extract key information from sparse and complex financial credit data, and derives a robust prediction mechanism by learning the inherent patterns in these key information. Further, in order to cope with the imbalance of financial credit data, the f1 score is introduced as a training evaluation indicator of the model in the model training step to assist model training, the purpose of which is to find the optimal model state to more accurately identify default users, thereby achieving effective learning of unbalanced financial credit data and improving the accuracy of credit default prediction. In addition, the f1 score can assist the model to quickly converge to the state of optimally distinguishing default samples, which can reduce the time required for model training.

[0113] In the embodiment, model training is performed based on the early stopping strategy and the Adam optimizer, wherein the early stopping strategy can improve the efficiency of model training and effectively avoid overfitting. The Adam optimizer is simple to implement and computationally efficient, and its hyperparameters usually do not need to be adjusted or only require very little adjustment. It can also improve the efficiency of model training and is suitable for scenarios with large-scale data and parameters.

[0114] <Comparative Example>

[0115] This comparative example provides several existing models for credit default prediction, which are used as comparison models to compare the effects with the SA-DNN model of the embodiment.

[0116] In this comparative example, the comparison models include DNN in the prior art, four algorithm models based on ensemble learning: extreme gradient boosting (XGBoost), adaptive boosting (AdaBoost), gradient boosting decision tree (GBDT), random forest (RF), and two machine learning models: decision tree (DT) and logistic regression (LR), that is, a total of 7 comparison models. The SA-DNN model of the embodiment and the above 7 comparison models are respectively tested and evaluated for performance.

[0117] Among them, for the performance test, the test set is the Zhongyuan Bank credit data set, which comes from the 2021 CCF competition. It contains 10,000 records and covers 34 features. The main data revolves around personal credit default records. The data in this data set mainly includes information such as each user's current loan record, historical loan record, occupation, age, and marital status. It is worth noting that the ratio of default samples to non-default samples in this data set is approximately 1:4.94, which means that there is an obvious sample imbalance problem in the data set. In this comparative example, the test set is used to test the SA-DNN model of the embodiment and the above-mentioned 7 comparison models respectively.

[0118] For performance evaluation, this comparative example conducts overall performance evaluation and category prediction performance evaluation on each model.

[0119] Specifically, the indicators of overall performance evaluation include accuracy, AUC (Area Under the ROC Curve) and KS (Kolmogorov-Smirnov). Accuracy is used to measure the correctness of the model's prediction, that is, the ratio between the number of samples correctly classified by the model and the total number of samples. AUC calculates the area under the ROC curve, which is not affected by the problem of sample imbalance. KS is a statistic based on the cumulative distribution function, which is used to measure the degree of difference between positive and negative samples in the classification model.

[0120] In the category prediction performance evaluation, for the default samples with label 1 and the non-default samples with label 0, the Precision class , Recall class , and f1 class .

[0121] Table 1 below shows the comparative results of the performance evaluation of the above-mentioned various models.

[0122] Table 1 Performance comparison of SA-DNN model and comparison model

[0123]

[0124] From the data comparison in Table 1, it can be seen that the SA-DNN model has a significant advantage in predicting default users and can recall more default users. Compared with the best Recall1=0.414 in the comparison model, the SA-DNN model's Recall1=0.485 achieved a 17.15% performance improvement. Similarly, compared with the best f11=0.438 in the comparison model, the SA-DNN model's f11=0.479 increased by 9.3%. At the same time, for the prediction performance of non-default samples, the SA-DNN model showed excellent performance in indicators such as Precision0, Recall0 and f10.

[0125] It is worth noting that although the LR model achieved 97.2% in recalling non-defaulting users, its Precision0 was the lowest, which reflects that the LR model tends to misclassify defaulting users as non-defaulting users. In contrast, from the perspective of global performance, the SA-DNN model has no significant decline compared with other models, and even achieved significant improvements in some key indicators.

[0126] In summary, the SA-DNN model provided in the embodiment performs well in identifying defaulting users, while also maintaining excellent performance in predicting the accuracy of non-defaulting and overall data.

[0127] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. An unbalanced credit default prediction method based on self-attention mechanism, characterized in that: The following steps are involved: Data preprocessing step: preprocess the financial credit data used for model training to obtain a training set; Model training step: training a deep neural network model based on a self-attention mechanism based on the training set and an early stopping strategy to obtain the trained deep neural network model, wherein the deep neural network includes a first fully connected layer, a batch normalization layer, a self-attention layer, and a second fully connected layer connected in sequence, and during the training process, the learning rate of the deep neural network model is optimized and adjusted using the f1 score; Credit default prediction step: input the financial credit data to be predicted into the trained deep neural network model to obtain the prediction result of credit default.

2. The unbalanced credit default prediction method based on self-attention mechanism according to claim 1, characterized in that: in, The model training step includes the following sub-steps: Initializing parameters for model training, including setting an initial value of a learning rate of the deep neural network model and an initial value of an optimal f1 score; The deep neural network model is trained based on the training set and the early stopping strategy until a predetermined number of training rounds is reached and the accuracy of the model no longer improves, thereby obtaining the trained deep neural network model, wherein the training rounds are reset when the f1 score is less than or equal to the optimal f1 score, and when the f1 score is greater than the optimal f1 score and a predetermined number of training rounds is reached, the learning rate of the deep neural network model is updated.

3. The unbalanced credit default prediction method based on self-attention mechanism according to claim 2, Features: in, The initialized parameters also include training rounds, weights, biases and learning rates of the deep neural network model, Model training is performed based on the Adam optimizer, and the initialized parameters also include the learning rate, first-order moment estimation and second-order moment estimation in the Adam optimizer, The steps of training the deep neural network model based on the training set and the early stopping strategy include: Input a set of training sets into the deep neural network model, and calculate its gradient based on the weights and biases of the deep neural network model; updating the first-order moment estimate and the second-order moment estimate of momentum based on the gradient; Correcting the deviation of the first-order moment estimate and the second-order moment estimate; Based on the corrected second-order moment estimate, updating weights and biases of the deep neural network model; Based on the prediction result of the deep neural network model, calculate the f1 score at this time; Determine whether the f1 score is greater than the best f1 score, and if so, update the best f1 score to the f1 score; if not, reset the training round; Determine whether the training rounds reach a first predetermined value, and if so, update the learning rate of the deep neural network model and update the learning rate in the Adam optimizer; Determine whether the training round reaches a second predetermined value and the f1 score of this iteration has not improved compared to the f1 score of the previous iteration. If the judgment is yes, stop model training and output the trained deep neural network model.

4. The unbalanced credit default prediction method based on self-attention mechanism according to claim 3, characterized in that: in, The calculation formula of the f1 score is: f1=2*(Precision*Recall) / (Precision+Recall) Where Precision is the precision, which is calculated as Precision = TP / (TP+FP), Recall is the recall, which is calculated as Recall = TP / (TP+FN), TP and FP are indicators of the real defaulting users being predicted as defaulting and non-defaulting prediction labels, respectively, and FN is the non-defaulting sample with the defaulting user prediction label.

5. The unbalanced credit default prediction method based on self-attention mechanism according to claim 3, characterized in that: in, In the model training step, the gradient of the deep neural network model is calculated based on its weights and biases, and the expression is: Where w and b are the weight and bias of the deep neural network model, respectively. The first-order moment estimate of the momentum based on the gradient update is expressed as: m w =β1·m w +(1-β1)·g w m b =β1·m b +(1-β1)·g b In the formula, m w is the first moment estimate of w, m b is the first-order moment estimate of b, β1 is the exponential decay rate of the first-order moment estimate, The second-order moment estimate of the momentum based on the gradient update is expressed as: In the formula, v w is the second moment estimate of w, v b is the second-order moment estimate of b, β2 is the exponential decay rate of the second-order moment estimate, Correct the deviation of the first-order moment estimate and the second-order moment estimate, the expression is: In the formula, is β1 to the power of t, is β2 raised to the power of t, where t is the current iteration number, is the corrected first-order moment estimate of w, is the corrected first-order moment estimate of b, is the corrected second moment estimate of w, is the corrected second-order moment estimate of b, Based on the corrected second-order moment estimate, the weights and biases of the deep neural network model are updated, and the expression is: Where η is the learning rate and ∈ is a predetermined value used to prevent the denominator from being zero.

6. The unbalanced credit default prediction method based on self-attention mechanism according to claim 3, characterized in that: in, In the model training step, the initial value of the best f1 score is 0, The first predetermined value for determining the number of training rounds is 10, and the expression for updating the learning rate of the deep neural network model is: x i+1 =ξ i *0·1 The second predetermined value is 20.

7. The unbalanced credit default prediction method based on self-attention mechanism according to claim 1, characterized in that: in, In the deep neural network model, the first fully connected layer is a fully connected layer based on the Relu activation function, and the second fully connected layer is a fully connected layer based on the Sigmoid activation function.

8. The unbalanced credit default prediction method based on self-attention mechanism according to claim 7, characterized in that: in, In the model training step, the preprocessed financial credit data is input into the deep neural network model, and the output of the first fully connected layer is: X 1 =max(0,W1*X+b1) Where W1 and b1 are the weight and bias of the first fully connected layer, The batch normalization layer is used to perform batch normalization on the output of the first fully connected layer, and its output is: In the formula, μ B and σ B are the mean and standard deviation of the batch, ∈ is a predetermined value to prevent the denominator from being 0, γ and β are learnable parameters, The self-attention layer first linearly maps the output of the batch normalization layer to the query matrix Q, key matrix K and value matrix V: Where W Q , W K and W V are the corresponding weights, b Q , b K and b K are the corresponding bias terms, The self-attention layer calculates the attention value α of every two input vectors of the query matrix Q and the key matrix K, thereby obtaining the attention matrix: Attention=(a) (i，j) =softmax(Q*K) The self-attention layer calculates the output vector corresponding to each two input vectors: X attention =Attention*V The second fully connected layer activates the output of the self-attention layer: y predict =sigmoid(W4*X attention +b4) Where W4 and b4 are the weight and bias of the second fully connected layer, respectively.

9. The unbalanced credit default prediction method based on self-attention mechanism according to claim 1, characterized in that: in, In the data preprocessing step, the financial credit data is normalized, and the expression is as follows: x=(X-min(X)) / (max(X)-min(X)) Wherein, X is the characteristic of the financial credit data.

10. An unbalanced credit default prediction device based on self-attention mechanism, characterized in that: include: The data preprocessing department preprocesses the financial credit data used for model training to obtain the training set; A model training unit trains a deep neural network model based on a self-attention mechanism based on the training set and an early stopping strategy to obtain the trained deep neural network model, wherein the deep neural network includes a first fully connected layer, a batch normalization layer, a self-attention layer, and a second fully connected layer connected in sequence, and during the training process, the learning rate of the deep neural network model is optimized and adjusted using the f1 score; as well as The credit default prediction unit is used to input the financial credit data to be predicted into the trained deep neural network model to obtain the prediction result of credit default.

Images