Credit default prediction method and device based on self-attention integrated automatic multi-view

Abstract

The invention provides a credit default prediction method and device based on self-attention integrated automatic multi-view, and the method comprises the steps: automatically dividing the features of financial data for model training into four feature groups based on discrete connectivity and positive and negative correlation, saving a time-consuming manual marking link, and improving the accuracy of credit default prediction. The division mode enables the same or similar features to be gathered and enables different features to be distinguished, so that higher efficiency is realized on the aspect of information organization; a stack network model based on a self-attention mechanism is adopted, the model applies an advanced self-attention mechanism, so that a neural network can more accurately identify and understand internal association between data, further, through stack generalization learning based on the self-attention mechanism, information from different views can be effectively integrated, and the accuracy of the stack network model is improved. And an integration mechanism is learned from the basis neural network model, and the integration mechanism can combine predictions made by different basis neural network models, so that the accuracy and robustness of a prediction result are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of credit default prediction methods, and in particular relates to a credit default prediction method and device based on self-attention integrated automatic multi-view. Background Art

[0002] Credit risk is an important aspect of financial risk, and credit default is its main manifestation. Credit default has a significant impact on the stability of financial markets. Due to slowing economic growth, escalating geopolitical risks and trade frictions, global default rates are rising. These defaults have far-reaching consequences, including weak economic growth, declining investor confidence, and significant losses to financial institutions. Therefore, it is imperative to develop effective credit risk management strategies to mitigate the adverse effects of credit defaults on financial markets and the economy.

[0003] However, credit default prediction is a challenging task due to the complex and high sparsity of data. In recent years, recent research works have focused on statistical learning and machine learning techniques to reduce the risk of credit default.

[0004] Statistical learning is mainly based on credit rating and logistic regression (LR) for credit analysis and prediction. However, LR is also sensitive to outliers and missing data, and it is difficult to train LR models on large data sets. Despite these limitations, LR is a widely used credit default prediction method due to its good interpretability. However, the global financial crisis revealed some shortcomings of LR methods, such as their slow adaptability to changing economic conditions and their limited ability to model complex nonlinear interactions between economic, financial, and credit variables.

[0005] Machine learning has excellent adaptive capabilities to adapt to changes in data and accommodate large feature sets. Decision trees (DTs) can capture interactions and nonlinear relationships between variables, providing good distinction between default and non-default cases. However, DT's repeated attribute segmentation is prone to overfitting. Ensemble learning solves this problem by growing multiple trees and calculating their average. For example, the Gradient Boosting DecisionTree (GBDT) learns a series of weak learners to predict the output, where the weak learner is the DT. Chen et al. demonstrated that GBDT outperforms other methods in tabular data applications. Several GBDT algorithms, such as XGBoost, LightGBM, and CatBoost, have been developed to successfully predict credit risk. Although these ensemble algorithms exhibit differences, their performance in various tasks is generally similar.

[0006] Deep learning has become a promising method for credit default prediction due to its ability to capture complex patterns and nonlinear relationships between variables. Tan et al. and Yang et al. proposed improved deep neural networks (DNNs) for better credit default prediction than machine learning and CNN. Luo et al. developed a credit risk assessment model using a deep belief network, which is good at learning effective feature representations and capturing potential patterns in data. Fu et al. combined DNNs and bidirectional long short-term memory to accurately identify potential credit risks. These studies show that deep learning techniques can significantly improve the predictive performance of credit default models, making them a valuable tool in credit risk management.

[0007] In addition, some new methods improve model performance based on multi-view feature organization. Song et al. designed multi-view feature sampling to optimize imbalanced learning. Tan et al. proposed that multi-view feature learning based on discrete, continuous and related symbol differences is an effective strategy to improve prediction accuracy.

[0008] Existing works have shown that GBDT, deep learning, and multi-view integration can significantly improve model performance. However, there is limited exploration on how to combine these optimization strategies to obtain better solutions. Summary of the invention

[0009] The present invention is made to solve the above problems, and aims to provide a credit default prediction method and device with higher accuracy. The present invention adopts the following technical solutions:

[0010] The present invention provides a credit default prediction method based on self-attention integrated automatic multi-view, characterized in that it includes the following steps: a feature group division step: automatically dividing the features of financial data used for model training into four feature groups based on discrete connectivity and positive and negative correlation to obtain a training set; a model training step: training a stacked network model based on a self-attention mechanism based on the training set to obtain the trained stacked network model, wherein the model includes four base neural network models and a self-attention layer, the four base neural network models are respectively used to learn the knowledge of the four feature groups, and the self-attention layer is used to stack the knowledge of the four feature groups and learn the integration mechanism therefrom; a credit default prediction step: inputting the financial data to be predicted into the trained stacked network model to obtain a prediction result of credit default.

[0011] The credit default prediction method based on self-attention integration automatic multi-view provided by the present invention may also have such a technical feature, wherein the feature group division step includes the following sub-steps: calculating the Pearson correlation coefficient between the features and labels of the financial data; screening out the features whose Pearson correlation coefficient is greater than a predetermined value as a positively correlated feature group, and screening out the features whose Pearson correlation coefficient is less than or equal to the predetermined value as a negatively correlated feature group; calculating the number of class values ​​of each feature; calculating the quantile of all the class value numbers; screening out the features whose class value number is greater than the quantile as a discrete feature group, and screening out the features whose class value number is less than or equal to the quantile as a continuous feature group; screening out the features that exist in both the positively correlated feature group and the discrete feature group as a positively correlated discrete group, screening out the features that exist in both the negatively correlated feature group and the discrete feature group as a negatively correlated discrete group, screening out the features that exist in both the positively correlated feature group and the continuous feature group as a positively correlated continuous group, and screening out the features that exist in both the negatively correlated feature group and the continuous feature group as a negatively correlated continuous group.

[0012] The credit default prediction method based on self-attention integrated automatic multi-view provided by the present invention may also have the following technical features, wherein each of the base neural network models includes a first fully connected layer, a batch normalization layer, a second fully connected layer, and a third fully connected layer connected in sequence, the batch normalization layer is used to regularize the output of the first fully connected layer, the stacked network model also includes a splicing layer and an output fully connected layer, the outputs of multiple base neural network models are all connected to the splicing layer, the splicing layer, the self-attention layer and the output fully connected layer are connected in sequence, the splicing layer is used to splice the outputs of each of the base neural network models, the first fully connected layer and the second fully connected layer are both fully connected layers based on the Relu activation function, the third fully connected layer and the output fully connected layer are both fully connected layers based on the Sigmoid activation function.

[0013] The credit default prediction method based on self-attention integration automatic multi-view provided by the present invention may also have such a technical feature, wherein the batch normalization layer regularizes the output of the first fully connected layer to obtain the regularized feature, which is expressed as: X 1 =Norm(Relu(W1*X+b1)), where W1 and b1 are the weight and bias of the first fully connected layer, respectively. The second fully connected layer further abstracts the regularized features, expressed as: 2 =Relu(W2*X 1+b2), where W2 and b2 are the weight and bias of the second fully connected layer, respectively. The third fully connected layer activates the output of the second fully connected layer of each feature group, and the expression is: Wherein, W3 and b3 are the weight and bias of the second fully connected layer respectively. The concatenation layer concatenates the outputs of each of the third fully connected layers to obtain the concatenated features, which are expressed as follows: The self-attention layer linearly maps the concatenated features to the query matrix Q, the key matrix K and the value matrix V, expressed as: Q = W Q *X * +b Q , K=W K *X * +b K , V=W V *X * +b K , where W Q , W K and W V is the corresponding weight, b Q 、b K and b K The self-attention layer calculates the attention value α between each two input vectors of the query matrix Q and the key matrix K, thereby obtaining the attention matrix A, which is expressed as: A = (α) (i,j) =softmax(Q*K), the self-attention layer calculates the output vector corresponding to every two input vectors, expressed as: attention =A*V, the fourth fully connected layer activates the output of the self-attention layer, and the expression is: predict =sigmoid(W4*X attention +b4), where W4 and b4 are the weight and bias of the fourth fully connected layer respectively.

[0014] The credit default prediction method based on self-attention integrated automatic multi-view provided by the present invention may also have such a technical feature, wherein the model training step includes the following sub-steps: initializing the parameters used to train the stacked network model; training the stacked network model based on the training set and the early stopping strategy until a predetermined number of training rounds is reached and the accuracy of the model no longer improves, thereby obtaining the trained stacked network model.

[0015] The credit default prediction method based on self-attention integration automatic multi-view provided by the present invention may also have such a technical feature, wherein, model training is performed based on the Adam optimizer, the initialized parameters include the weights and biases of the stacked network model, the learning rate of the Adam optimizer, the first-order moment estimate and the second-order moment estimate, and the step of training the stacked network model using the training set includes: inputting a group of the training sets into the stacked network model, and calculating the gradient of the stacked network model based on the weights and the biases; updating the first-order moment estimate and the second-order moment estimate of the momentum based on the gradient; correcting the deviation of the first-order moment estimate and the second-order moment estimate; updating the weights and bias of the stacked network model based on the corrected second-order moment estimate; calculating the accuracy score of the prediction result of the stacked network model; judging whether the predetermined first training round has been reached, and updating the learning rate when the judgment is yes; judging whether the predetermined second training round has been reached and the accuracy score of this iteration has not been improved compared with the accuracy score of the previous iteration, and outputting the trained stacked network model when the judgment is yes.

[0016] The credit default prediction method based on self-attention integration and automatic multi-view provided by the present invention may also have such a technical feature, wherein the gradient of the stacked network model is calculated based on the weight and the bias, and the expression is: Wherein, w and b are the weight and bias of the stacked network model respectively, and the first-order moment estimation based on the gradient update momentum is expressed as: w =β1·m w +(1-β1)·g w , m b =β1·m b +(1-β1)·g b , where m w is the first moment estimate of w, m b is the first-order moment estimate of b, β1 is the exponential decay rate of the first-order moment estimate, and the second-order moment estimate of the momentum is updated based on the gradient, expressed as: In the formula, v w is the second-order moment estimate of w, v b is the second-order moment estimate of b, β2 is the exponential decay rate of the second-order moment estimate, correcting the deviation between the first-order moment estimate and the second-order moment estimate, and the expression is: In the formula, is β1 to the power of t, is the t-th power of β2, where t is the current number of iterations, is the corrected first-order moment estimate of w, is the corrected first-order moment estimate of b, is the corrected second-order moment estimate of w, are respectively the corrected second-order moment estimates of b. Based on the corrected second-order moment estimates, the weights and biases of the stacked network model are updated, and the expression is: Where η is the learning rate and ∈ is a predetermined value used to prevent the denominator from being zero.

[0017] The credit default prediction method based on self-attention integrated automatic multi-view provided by the present invention may also have such a technical feature, wherein the accuracy score of the prediction result of the stacked network model is calculated, and the expression is: acc = (TP + TN) / (TP + FP + TN + FN), wherein TP and FP are indicators of the real default users predicted as default and non-default prediction labels, respectively, TN and FN are samples of the non-default users predicted as non-default and default prediction labels, respectively, the first training round is 10, and the expression for updating the learning rate is: ξ i+1 =ξ i *0.1, the second training round is 20.

[0018] The present invention provides a credit default prediction device based on self-attention integrated automatic multi-view, characterized in that it includes: a feature group division unit, which automatically divides the features of financial data used for model training into four feature groups based on discrete connectivity and positive and negative correlation to obtain a training set; a model training unit, which trains a stacked network model based on a self-attention mechanism based on the training set to obtain the trained stacked network model, wherein the model includes four base neural network models and a self-attention layer, the four base neural network models are respectively used to learn the knowledge of the four feature groups, and the self-attention layer is used to stack the knowledge of the four feature groups and learn the integration mechanism therefrom; and a credit default prediction unit, which is used to input the financial data to be predicted into the trained stacked network model to obtain a prediction result of credit default.

[0019] Function and Effect of the Invention

[0020] According to the credit default prediction method and device based on self-attention integration automatic multi-view of the present invention, the method includes a feature group division step, a model training step and a credit default prediction step, wherein in the feature group division step, the features of the financial data used for model training are automatically divided into four feature groups based on discrete connectivity and positive and negative correlation, thereby eliminating the time-consuming and labor-intensive corresponding manual labeling process, and this division method enables the same or similar features to be aggregated, and different features to be distinguished, thereby achieving higher efficiency at the information organization level and providing the possibility for the model to achieve better performance; in the model training step, the stacked network model based on the self-attention mechanism is trained using the training set of the divided feature groups. The model uses an advanced self-attention mechanism to enable the neural network to more accurately identify and understand the intrinsic correlation between data. Further, through the stacked generalization learning based on the self-attention mechanism, information from different feature groups (different views) can be effectively integrated, and the integration mechanism can be learned therefrom. The integration mechanism can combine the predictions made by different base neural network models, more accurately identify and process the key information in the predictions from different base neural network models, thereby improving the model's prediction ability for credit default, so that the accuracy and robustness of the prediction results can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a credit default prediction method based on self-attention integration automatic multi-view in an embodiment of the present invention;

[0022] Figure 2 is a flow chart of a credit default prediction method based on self-attention integration automatic multi-view in an embodiment of the present invention;

[0023] Figure 3 is a flow chart of dividing the features of financial data into four feature groups in an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of the structure of a stacked network model based on a self-attention mechanism in an embodiment of the present invention;

[0025] Figure 5 is a flowchart of a stacked network model training process based on a self-attention mechanism in an embodiment of the present invention;

[0026] Figure 6 is a flow chart of each training epoch in an embodiment of the present invention;

[0027] Figure 7 It is a block diagram of a credit default prediction device based on self-attention integration automatic multi-view in an embodiment of the present invention.

[0028] Reference numerals:

[0029] The credit default prediction device 100 includes a prediction model storage unit 101 , a training data acquisition unit 102 , a feature group classification unit 103 , a model training unit 104 , a credit default prediction unit 105 , a credit default prediction result output unit 106 , and a credit default prediction control unit 107 . DETAILED DESCRIPTION

[0030] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following is a detailed description of the credit default prediction method and device based on self-attention integration automatic multi-view of the present invention in combination with embodiments and drawings.

[0031] <Example>

[0032] Figure 1 4 is a flowchart of the credit default prediction method based on self-attention integration automatic multi-view in this embodiment.

[0033] Figure 2 4 is a flow chart of the credit default prediction method based on self-attention integration and automatic multi-view in this embodiment.

[0034] like Figure 1 and Figure 2 As shown, the credit default prediction method based on self-attention integration automatic multi-view includes the following steps:

[0035] Feature group division step S1: The features of the financial data used for model training are automatically divided into four feature groups based on discrete continuity and positive and negative correlation to obtain a training set.

[0036] Model construction step S2: Construct a stacked network (SA-SN) model based on the self-attention mechanism.

[0037] Model training step S3: Train the SA-SN model based on the training set and the early stopping strategy to obtain a trained SA-SN model.

[0038] Credit default prediction step S4: The financial data to be predicted is input into the trained SA-SN model, and the model outputs the prediction result of credit default.

[0039] The above steps will be described in detail below.

[0040] Feature group division step S1: The features of the financial data used for model training are automatically divided into four feature groups based on discrete continuity and positive and negative correlation to obtain a training set.

[0041] Figure 3 It is a flowchart of dividing the features of financial data into four feature groups in this embodiment.

[0042] like Figure 3As shown, step S1 specifically includes the following sub-steps:

[0043] Step S1-1: Calculate the Pearson correlation coefficient between the features and labels of the financial data.

[0044] The calculation is performed according to the following formula:

[0045]

[0046] In the formula, x i is the value of the feature, y i is the value of the label, is the mean value of the feature, is the average value of the label. ∑ is the summation symbol and sqrt is the square root symbol.

[0047] Step S1-2: Screen out features with a Pearson correlation coefficient greater than 0 (ie, r>0), and regard these features as a positive correlation feature group; and screen out features with a Pearson correlation coefficient less than or equal to 0 (r≤0), and regard these features as a negative correlation feature group.

[0048] Step S1-3: Calculate the class value of each feature. For the i-th feature, calculate its class value N i , that is, the number of eigenvalues.

[0049] Step S1-4: Calculate the number of all class values ​​N i The quantile M of .

[0050] Step S1-5: Filter out the classes with values ​​greater than the binary quantile (N i >M), treat these features as a discrete feature group, and filter out the features with class values ​​less than or equal to the quantile (N i ≤M), and regard these features as a continuous feature group.

[0051] Step S1-6: Screen out the features that exist in both the positive correlation feature group and the discrete feature group, and regard these features as the positive correlation discrete group; screen out the features that exist in both the negative correlation feature group and the discrete feature group, and regard these features as the negative correlation discrete group; screen out the features that exist in both the positive correlation feature group and the continuous feature group, and regard these features as the positive correlation continuous group; screen out the features that exist in both the negative correlation feature group and the continuous feature group, and regard these features as the negative correlation continuous group.

[0052] Among them, corresponding labels are automatically set for the features of each feature group.

[0053] Through step S1, four feature groups are automatically divided, that is, all features are automatically divided into four feature subspaces. This division method allows the same or similar features to be clustered and different features to be distinguished, thereby achieving higher efficiency at the level of information organization. In model training, these four feature groups can provide four different perspectives for the model.

[0054] Model construction step S2: Construct a stacked network model based on the self-attention mechanism (Self-Attentionbased Stacked Network, SA-SN) to stack the knowledge of four feature groups.

[0055] Figure 4 It is a structural diagram of the stacked network model based on the self-attention mechanism in this embodiment.

[0056] like Figure 4 As shown, the input of the stacked network model based on the self-attention mechanism is financial data, and the output is the prediction result of credit default. In this embodiment, the financial data is divided into four feature groups, namely, a positive correlation discrete group, a negative correlation discrete group, a positive correlation continuous group, and a negative correlation continuous group. The stacked network model improves the prediction accuracy by stacking knowledge from different feature groups.

[0057] The model consists of four base neural network models, a concatenation layer, a self-attention layer, and an output fully connected layer ( Figure 4 FC(1) in the middle and lower part). The outputs of the four base neural network models are all connected to the concatenation layer, and the concatenation layer, self-attention layer, and output are connected in sequence using a fully connected layer. The four base neural network models are used to learn the knowledge of the above four feature groups respectively, and the self-attention layer is used to integrate the knowledge from different feature groups. This integration mechanism can combine the predictions made by different base neural network models, thereby improving the accuracy and robustness of the prediction results.

[0058] Each base neural network includes a first fully connected layer ( Figure 4 FC(20) in the Figure 4 Norm in), the second fully connected layer ( Figure 4 FC(10) in), the third fully connected layer ( Figure 4 The upper FC(1) in , forms a multi-layer perceptron (MLP).

[0059] In this embodiment, the first fully connected layer and the second fully connected layer are both fully connected layers based on the Relu activation function, and the third fully connected layer and the output fully connected layer are both fully connected layers based on the Sigmoid activation function.

[0060] Model training step S3: Train the SA-SN model based on the training set and the early stopping strategy to obtain a trained SA-SN model.

[0061] During the model training process, when the training data of each feature group is input, the corresponding base neural network first obtains the input through its first fully connected layer. Then, in order to reduce the risk of overfitting, the output of the first fully connected layer is regularized through the batch normalization layer, as shown in the following formula:

[0062] X 1 =Norm(Relu(W1*X+b1)). (2)

[0063] Where W1 and b1 are the weight and bias of the first fully connected layer respectively. Relu(W1*X+b1) is the output result of the first fully connected layer.

[0064] Then, in order to further abstract the features, the output of the Batch Norm layer is input into the second fully connected layer, as shown in the following formula:

[0065] X 2 =Relu(W2*X 1 +b2). ​​(3)

[0066] Where W2 and b2 are the weight and bias of the second fully connected layer, respectively.

[0067] Then, in order to obtain the knowledge of the feature group, the third fully connected layer obtains the output of the batch normalization layer of the corresponding feature group and activates the output using the Sigmoid activation function, as shown in the following formula:

[0068]

[0069] Where W3 and b3 are the weight and bias of the third fully connected layer, respectively.

[0070] Through the third fully connected layer, the true label and , to learn exclusive feature group knowledge from local views.

[0071] Then, the concatenation layer concatenates the outputs of the third fully connected layer corresponding to different feature groups. Perform splicing to obtain the spliced ​​multi-view features, as shown in the following formula:

[0072]

[0073] Then, the importance of different elements in the spliced ​​multi-view features is learned through the self-attention layer. The self-attention layer converts the above X *Linear mapping to query matrix Q, key matrix K and value matrix V, the calculation formulas are:

[0074] Q=W Q *X * +b Q , (6)

[0075] K=W K *X * +b K , (7)

[0076] V=W V *X * +b K , (8)

[0077] Where W Q , W K and W V is the corresponding weight, b Q 、b K and b K is the corresponding bias term.

[0078] Then, the self-attention layer calculates the attention value α between each two input vectors of the query matrix Q and the key matrix K, thus obtaining the attention matrix A:

[0079] A=(α) (i,j) =softmax(Q*K). (9)

[0080] The self-attention layer calculates the output vector of the self-attention layer corresponding to each two input vectors based on the attention matrix A:

[0081] X attention =A*V, (10)

[0082] Among them, X attention Each element of contains X * The association information of other elements in the model can capture a wider range of combination mechanisms.

[0083] Finally, the mapping relationship between features and true labels is learned through the output of the fully connected layer, and the prediction results are output as follows:

[0084] y predict =sigmoid(W4*X attention +b4), (11)

[0085] Where W4 and b4 are the weight and bias of the fourth fully connected layer, respectively.

[0086] Figure 5 4 is a flow chart of the SA-SN model training process in this embodiment.

[0087] like Figure 5 As shown, in this embodiment, model training is performed based on the early stopping strategy, and step S3 specifically includes the following sub-steps:

[0088] Step S3-1: Initialize parameters for training the model.

[0089] In this embodiment, the following parameters for training the model are initialized: weight w, bias b, learning rate ξ0, and optimal accuracy acc of the SA-SN model. best =0, number of training rounds epoch count =0; use Adam optimizer, for Adam optimizer, initialize its parameter m w =0,v w =0,m b =0,v b =0,t=0, where t represents the number of iterations, m w represents the first-order moment estimate of weight w, v w represents the second-order moment estimate of weight w, m b represents the first-order moment estimate of the bias b, v b Represents the second-order moment estimate of the bias b.

[0090] Step S3-2: Train the SA-SN model based on the training set and the early stopping strategy until a predetermined number of training rounds is reached and the accuracy of the model no longer improves.

[0091] Figure 6 It is a flowchart of each training epoch in this embodiment.

[0092] like Figure 6 As shown, step S3-2 specifically includes the following sub-steps:

[0093] Step S3-2-1: Set the number of iterations t=1, i.e. start the first round of iterations.

[0094] Step S3-2-2: Input a training set into the SA-SN model and calculate its gradient based on the weights and biases of the SA-SN model:

[0095]

[0096] Step S3-2-3: Based on the gradient, update the first-order moment estimate of momentum:

[0097] m w =β1·m w +(1-β1)·g w , (13)

[0098] m b =β1·m b+(1-β1)·g b , (14)

[0099] Where β1 is the exponential decay rate of the first-order moment estimate.

[0100] Step S3-2-4: Based on the gradient, update the second-order moment estimate of momentum:

[0101]

[0102]

[0103] Where β2 is the exponential decay rate of the second-order moment estimate.

[0104] Step S3-2-5: Correct the deviation of the first-order moment estimation and the second-order moment estimation of momentum:

[0105]

[0106]

[0107] In the formula, is β1 to the power of t, It is β2 raised to the power of t, where t is the current iteration number.

[0108] Step S3-2-6: Update the weights and biases of the SA-SN model based on the corrected second-order moment estimate:

[0109]

[0110]

[0111] Where η is the learning rate and ∈ is a predetermined minimum number to prevent the denominator from being zero.

[0112] Step S3-2-7: Calculate the prediction result y output by the SA-SN model at this time predict The accuracy score acc:

[0113] acc=(TP+TN) / (TP+FP+TN+FN), (21)

[0114] Where TP and FP are the indicators of the real default users being predicted as default and non-default prediction labels, respectively, and TN and FN are the samples of the non-default users being predicted as non-default and default prediction labels, respectively.

[0115] Step S3-2-8: Determine whether acc>acc best .

[0116] Step S3-2-9: If the judgment in step S3-2-8 is yes, set accbest =acc, and epoch count =epoch count +1.

[0117] Step S3-2-10: If the judgment in step S3-2-8 is no, set the epoch count =0.

[0118] Step S3-2-11: Determine whether there is an epoch count =10.

[0119] Step S3-2-12: When step S3-2-11 is judged as yes, update the learning rate ξ in the Adam optimizer i+1 :

[0120] ξ i+1 =ξ i *0.1, (22)

[0121] Step S3-2-13: Further determine whether there is an epoch count =20.

[0122] Step S3-2-15: When the judgment in step S3-2-13 is yes, further determine whether the acc of this iteration is higher than the acc of the previous iteration.

[0123] Step S3-2-15: When the judgment in step S3-2-15 is no, that is, the acc of the model has not been further improved after 20 rounds, the model training is stopped.

[0124] Step S3-2-16: When step S3-2-11 is judged as no, or when step S3-2-13 is judged as no, or when step S3-2-15 is judged as yes, set t=t+1, that is, the number of iterations is increased by one, and return to step S3-2-2 to perform iterative training again.

[0125] Step S3-3: Output the trained SA-SN model.

[0126] Credit default prediction step S4: The financial data to be predicted is input into the trained stacked network based on the self-attention mechanism, and the model outputs the prediction result of credit default.

[0127] Among them, the financial data to be predicted is tabular data, which needs to be preprocessed and multiple feature groups generated before being input into the stacked network. The preprocessing method can refer to the above step S1. After the preprocessed financial data is input, the processing process of the features of the financial data by each layer of the SA-SN model is similar to that in step S2, so it will not be repeated.

[0128] Figure 7It is a block diagram of the credit default prediction device based on self-attention integration automatic multi-view in this embodiment.

[0129] like Figure 7 As shown, this embodiment also provides a credit default prediction device 100 based on self-attention integration automatic multi-view corresponding to the above method, and the device includes a prediction model storage unit 101, a training data acquisition unit 102, a feature group division unit 103, a model training unit 104, a credit default prediction unit 105, a credit default prediction result output unit 106, and a credit default prediction control unit 107.

[0130] Among them, the prediction model storage unit 101 stores the above-mentioned SA-SN model. The training data acquisition unit 102 is used to obtain training data for model training from other devices. The feature group division unit 103 automatically divides the financial data used as training data into four feature groups according to the method of the above-mentioned step S1 to generate a training set. The model training unit 104 trains the stored SA-SN model according to the method of the above-mentioned step S3, and stores the trained SA-SN model in the prediction model storage unit 101 again. The credit default prediction unit 105 inputs the feature group into the trained SA-SN model for prediction according to the method of the above-mentioned step S4. The credit default prediction result output unit 106 outputs the prediction result to a display device or other device. The credit default prediction control unit 107 is used to control the operation of the above-mentioned functional units.

[0131] In this embodiment, parts not described in detail are well-known technologies in the art.

[0132] Example Function and Effect

[0133] According to the method and device for predicting credit default based on automatic multi-view based on self-attention integration provided by the present embodiment, the method includes a feature group division step, a model training step and a credit default prediction step, wherein in the feature group division step, the features of the financial data used for model training are automatically divided into four feature groups based on discrete connectivity and positive and negative correlation, thereby eliminating the time-consuming and labor-intensive corresponding manual labeling process, and this division method enables the same or similar features to be aggregated, and enables different features to be distinguished, thereby achieving higher efficiency at the information organization level and providing the possibility for the model to achieve better performance; in the model training step, the stacked network model based on the self-attention mechanism is trained using the training set of the divided feature groups. The model uses an advanced self-attention mechanism to enable the neural network to more accurately identify and understand the intrinsic correlation between data. Furthermore, through stacked generalization learning based on the self-attention mechanism, information from different feature groups (different views) can be effectively integrated, and an integration mechanism can be learned therefrom. The integration mechanism can combine the predictions made by different base neural network models, more accurately identify and process the key information in the predictions from different base neural network models, thereby improving the model's prediction ability for credit default, and greatly improving the accuracy and robustness of the prediction results.

[0134] In the embodiment, model training is performed based on the early stopping strategy and the Adam optimizer, wherein the early stopping strategy can improve the efficiency of model training and effectively avoid overfitting. The Adam optimizer is simple to implement and computationally efficient, and its hyperparameters usually do not need to be adjusted or only require very little adjustment. It can also improve the efficiency of model training and is suitable for scenarios with large-scale data and parameters.

[0135] <Comparative Example>

[0136] This comparative example provides several existing models for credit default prediction, which are used as comparative models to compare the effects of the SA-SN model of the embodiment.

[0137] In this comparative example, the comparison models include DNN in the prior art, four algorithm models based on ensemble learning: XGBoost, AdaBoost, GBDT, RF, and two machine learning models: DT and LR, that is, a total of 7 comparison models. The SA-SN model of the embodiment and the above 7 comparison models are respectively tested and evaluated for performance.

[0138] Among them, for the performance test, the test set is the Zhongyuan Bank credit data set, which comes from the 2021 CCF competition. It contains 10,000 records and 34 features, focusing on personal credit default records. The data in this data set mainly includes information such as the current loan record, historical loan record, occupation, age, and marital status of each user. The SA-SN model of the embodiment and the above 7 comparison models are tested using this test set.

[0139] For performance evaluation, this comparative example conducts overall performance evaluation and category prediction performance evaluation on each model.

[0140] The indicators for overall performance evaluation include accuracy, AUC (Area Under the ROC Curve) and KS (Kolmogorov-Smirnov). Accuracy is used to measure the correctness of the model's prediction, that is, the ratio between the number of samples correctly classified by the model and the total number of samples. AUC calculates the area under the ROC curve, which is not affected by the problem of sample imbalance. KS is a statistic based on the cumulative distribution function, which is used to measure the degree of difference between positive and negative samples in the classification model.

[0141] In the category prediction performance evaluation, for the default samples with label 1 and the non-default samples with label 0, the Precision class , Recall class , and f1 class .

[0142] Table 1 below shows the comparative results of the performance evaluation.

[0143] Table 1 Performance comparison of SA-SN model and comparison model

[0144]

[0145]

[0146] From the data comparison in Table 1, we can see that in terms of the prediction performance of default samples, the SA-SN model can obviously recall more defaulting users compared with the comparison model. Compared with the best Recall1 in the comparison model, the Recall1 of the SA-SN model has increased by 11.12%. At the same time, compared with the best f11 in the comparison model, the f11 of the SA-SN model has increased by 13.24%. In contrast, in terms of the prediction performance of non-default samples, the SA-SN model has achieved significantly better performance in Precision0, Recall0, and f10 indicators.

[0147] In addition, some performance indicators of the comparison model are significantly worse. Taking LR as an example, it can be seen that although LR recalls 97.2% of non-defaulting users, its Precision0 is the worst, which shows that LR is more likely to mistakenly identify defaulting users as non-defaulting users. In contrast, in terms of overall performance, the SA-SN model achieves comparable results to the comparison model, with no significantly worse indicators.

[0148] As described above, compared with other credit default prediction models in the prior art, the SA-SN model provided in the embodiment can more accurately identify defaulting users, while maintaining better performance for the prediction of non-defaulting users and overall data.

[0149] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A credit default prediction method based on self-attention integration automatic multi-view, characterized in that: The following steps are involved: Feature group division step: The features of the financial data used for model training are automatically divided into four feature groups based on discrete connectivity and positive and negative correlation to obtain a training set; Model training step: training a stacked network model based on a self-attention mechanism based on the training set to obtain the trained stacked network model, wherein the model includes four base neural network models and a self-attention layer, the four base neural network models are used to learn the knowledge of the four feature groups respectively, and the self-attention layer is used to stack the knowledge of the four feature groups and learn the integration mechanism therefrom; Credit default prediction step: input the financial data to be predicted into the trained stacked network model to obtain the prediction result of credit default.

2. The credit default prediction method based on self-attention integration automatic multi-view according to claim 1, Features: The step of dividing the feature groups includes the following sub-steps: Calculating the Pearson correlation coefficient between the features and labels of the financial data; Screening out the features whose Pearson correlation coefficient is greater than a predetermined value as a positive correlation feature group, and screening out the features whose Pearson correlation coefficient is less than or equal to the predetermined value as a negative correlation feature group; Calculate the class value of each of the features; Calculate the quantiles of all the class values; Screening out the features whose class values ​​are greater than the quantile as a discrete feature group, and screening out the features whose class values ​​are less than or equal to the quantile as a continuous feature group; The features that exist in both the positively correlated feature group and the discrete feature group are screened out as the positively correlated discrete group, the features that exist in both the negatively correlated feature group and the discrete feature group are screened out as the negatively correlated discrete group, the features that exist in both the positively correlated feature group and the continuous feature group are screened out as the positively correlated continuous group, and the features that exist in both the negatively correlated feature group and the continuous feature group are screened out as the negatively correlated continuous group.

3. The credit default prediction method based on self-attention integration automatic multi-view according to claim 1, characterized in that: in, Each of the base neural network models comprises a first fully connected layer, a batch normalization layer, a second fully connected layer, and a third fully connected layer connected in sequence, wherein the batch normalization layer is used to regularize the output of the first fully connected layer. The stacked network model further includes a splicing layer and an output fully connected layer, the outputs of the plurality of base neural network models are all connected to the splicing layer, the splicing layer, the self-attention layer and the output fully connected layer are sequentially connected, and the splicing layer is used to splice the outputs of the respective base neural network models. The first fully connected layer and the second fully connected layer are both fully connected layers based on the Relu activation function. The third fully connected layer and the output fully connected layer are both fully connected layers based on the Sigmoid activation function.

4. The credit default prediction method based on self-attention integration automatic multi-view according to claim 3, characterized in that: in, The batch normalization layer regularizes the output of the first fully connected layer to obtain the regularized features, which are expressed as: X 1 =Norm(Relu(W1*X+b1)) Where W1 and b1 are the weight and bias of the first fully connected layer, The second fully connected layer further abstracts the regularized features, and the expression is: X 2 =Relu(W2*X 1 +b2) Where W2 and b2 are the weight and bias of the second fully connected layer, The third fully connected layer activates the output of the second fully connected layer of each feature group, and the expression is: Where W3 and b3 are the weight and bias of the second fully connected layer, The concatenation layer concatenates the outputs of each of the third fully connected layers to obtain concatenated features, which are expressed as: The self-attention layer linearly maps the concatenated features to the query matrix Q, the key matrix K, and the value matrix V, expressed as: Q=W Q *X * +b Q K=W K *X * +b K V=W V *X * +b K Where W Q , W K and W V is the corresponding weight, b Q 、b K and b K is the corresponding bias term, The self-attention layer calculates the attention value α between each two input vectors of the query matrix Q and the key matrix K, thereby obtaining the attention matrix A, which is expressed as: A=(α) (i,j) =softmax(Q*K) The self-attention layer calculates the output vector corresponding to every two input vectors, and the expression is: X attention =A*V The fourth fully connected layer activates the output of the self-attention layer, and the expression is: y predict =sigmoid(W4*X attention +b4) Where W4 and b4 are the weight and bias of the fourth fully connected layer, respectively.

5. The credit default prediction method based on self-attention integration automatic multi-view according to claim 1, characterized in that: in, The model training step includes the following sub-steps: Initializing parameters for training the stacked network model; The stacked network model is trained based on the training set and the early stopping strategy until a predetermined number of training rounds is reached and the accuracy of the model is no longer improved, thereby obtaining the trained stacked network model.

6. The credit default prediction method based on self-attention integration automatic multi-view according to claim 5, Features: Wherein, the model training is performed based on the Adam optimizer, and the initialized parameters include the weight and bias of the stacked network model, the learning rate, the first-order moment estimation and the second-order moment estimation of the Adam optimizer, The step of training the stacked network model using the training set includes: Inputting a set of the training sets into the stacked network model, and calculating the gradient of the stacked network model based on the weights and the biases; updating the first-order moment estimate and the second-order moment estimate of momentum based on the gradient; Correcting the deviation of the first-order moment estimate and the second-order moment estimate; Based on the corrected second-order moment estimate, updating weights and biases of the stacked network model; Calculating the accuracy score of the prediction result of the stacked network model; Determine whether a predetermined first training round has been reached, and if so, update the learning rate; Determine whether a predetermined second training round has been reached and the accuracy score of this iteration has not improved relative to the accuracy score of the previous iteration, and if the judgment is yes, output the trained stacked network model.

7. The credit default prediction method based on self-attention integration automatic multi-view according to claim 6, characterized in that: in, The gradient of the stacked network model is calculated based on the weight and the bias, and the expression is: Where w and b are the weight and bias of the stacked network model, respectively. The first-order moment estimate of the momentum based on the gradient update is expressed as: m w =β1·m w +(1-β1)·g w m b =β1·m b +(1-β1)·g b In the formula, m w is the first moment estimate of w, m b is the first-order moment estimate of b, β1 is the exponential decay rate of the first-order moment estimate, The second-order moment estimate of the momentum based on the gradient update is expressed as: In the formula, v w is the second moment estimate of w, v b is the second-order moment estimate of b, β2 is the exponential decay rate of the second-order moment estimate, Correct the deviation of the first-order moment estimate and the second-order moment estimate, the expression is: In the formula, is β1 to the power of t, is the t-th power of β2, where t is the current number of iterations, is the corrected first-order moment estimate of w, is the corrected first-order moment estimate of b, is the corrected second moment estimate of w, are the corrected second-order moment estimates of b, Based on the corrected second-order moment estimate, the weights and biases of the stacked network model are updated, and the expression is: Where η is the learning rate and ∈ is a predetermined value used to prevent the denominator from being zero.

8. The credit default prediction method based on self-attention integration automatic multi-view according to claim 7, characterized in that: in, The accuracy score of the prediction result of the stacked network model is calculated as follows: acc=(TP+TN) / (TP+FP+TN+FN) In the formula, TP and FP are the indicators of the real default users being predicted as default and non-default prediction labels, respectively, TN and FN are the samples of the non-default users being predicted as non-default and default prediction labels, respectively. The first training round is 10, and the expression for updating the learning rate is: x i+1 =ξ i *0.1 The second training round is 20.

9. A credit default prediction device based on self-attention integration automatic multi-view, characterized in that: include: The feature group division unit automatically divides the features of the financial data used for model training into four feature groups based on discrete connectivity and positive and negative correlation to obtain a training set; A model training unit, which trains a stacked network model based on a self-attention mechanism based on the training set to obtain the trained stacked network model, wherein the model includes four base neural network models and a self-attention layer, the four base neural network models are respectively used to learn the knowledge of the four feature groups, and the self-attention layer is used to stack the knowledge of the four feature groups and learn the integration mechanism therefrom; as well as The credit default prediction unit is used to input the financial data to be predicted into the trained stacked network model to obtain the prediction result of credit default.

Images