FT-Transform-based bank credit prediction method and system

Abstract

The invention relates to a bank credit prediction method and system based on FT-Transform, and the method comprises the steps: collecting a bank customer credit data set which comprises numerical features and category features, and carrying out the preprocessing of the data set; the method comprises the following steps: respectively carrying out feature processing on a bank customer credit data set according to the data types of the bank customer credit data set through a shared embedded vector mechanism to obtain a first feature embedded matrix, the data types of the bank customer credit data set being divided into numeric features and category features; combining the learnable one-dimensional embedding vector and the first feature embedding matrix into a second feature embedding matrix; inputting the second feature embedding matrix into a multilayer Transform encoder stacking unit to extract a global token vector; and inputting the global token vector into a prediction module, and carrying out classification decision on the obtained prediction value to obtain a final bank credit prediction result. Compared with the prior art, the method has the advantages that the feature interaction efficiency is improved, the robustness is high, and the accuracy is high.

Description

Technical Field

[0001] This invention relates to the field of bank credit assessment technology, and in particular to a bank credit forecasting method and system based on FT-Transformer. Background Technology

[0002] In the financial sector, particularly in banking, credit forecasting is a crucial element of risk management and decision-making. Credit forecasting aims to assess the creditworthiness of individuals or businesses and predict their future debt repayment ability, which is vital for loan approval, credit line allocation, interest rate setting, and risk control. In today's digital financial era, data-driven decision-making is gradually becoming the mainstream trend in the financial industry. Among these methods, tabular data, as an important carrier of information on the creditworthiness of individuals or businesses, is widely used in credit assessment.

[0003] However, with the increase in data dimensions and the improvement in data complexity, traditional credit prediction methods, such as decision trees and category embedding methods, have gradually revealed their inherent limitations when dealing with the tabular data on which bank credit assessments rely.

[0004] First, traditional credit forecasting methods struggle to effectively handle high-dimensional, mixed-type data. Tabular data related to bank credit assessment typically contains a large number of features, including numerical data such as age, account balance, and service duration, as well as categorical data such as gender, occupation, region code, channel, product type, and activity level. Faced with this high-dimensional and mixed-type data structure, traditional methods encounter significant challenges in feature engineering and model training, struggling to efficiently integrate and utilize all the information. Furthermore, traditional credit forecasting methods fail to capture complex nonlinear feature relationships. Creditworthiness is influenced by numerous factors, and features often exhibit complex, nonlinear interactions. For example, the impact of age and account balance on creditworthiness may differ drastically for customers with different occupations in different regions. Traditional methods such as decision trees and categorical embeddings, due to limitations in their model structure or representational capabilities, struggle to fully mine and capture these deep, nonlinear feature relationships, resulting in insufficient learning of the inherent patterns in the data. These limitations make it difficult for traditional credit forecasting methods to meet the growing demands for risk management and sophisticated decision-making when dealing with the massive, complex, and high-dimensional tabular data in the modern financial environment.

[0005] Chinese patent application CN116308754A discloses a bank credit risk early warning system and method, which, based on deep learning and artificial intelligence natural language recognition technology, accurately characterizes and deeply semantically understands the hidden relationships about customer background information contained in application materials. However, this application processes unstructured text and does not combine it with structured tabular data such as numerical values. It fails to integrate with tabular data, which is the main basis for judgment in bank credit assessment scenarios, making it difficult to solve the problem of mixed-type features and limiting its application. Therefore, how to effectively process high-dimensional mixed-type tabular data and deeply mine the complex relationships between features to improve the accuracy and reliability of bank credit prediction is a technical problem that needs to be solved. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a bank credit prediction method and system based on FT-Transformer. By directly adding the shared embedding vector and the feature embedding vector, the total number of parameters is reduced while strengthening the feature representation, capturing the complex relationship between features, and improving the accuracy of converting the global token into the final predicted category through a multilayer perceptron and prediction module.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] According to one aspect of the present invention, a bank credit forecasting method based on FT-Transformer is provided, the specific steps of which include:

[0009] S1. Collect bank customer credit datasets including numerical and categorical features, and preprocess them;

[0010] S2. Through the shared embedding vector mechanism, feature processing is performed on the bank customer credit dataset according to the data type of the bank customer credit dataset to obtain the first feature embedding matrix. The data type of the bank customer credit dataset is divided into numerical features and categorical features.

[0011] S3. Combine the learnable one-dimensional embedding vector and the first feature embedding matrix into a second feature embedding matrix; input the second feature embedding matrix into the stacked unit of the multi-layer Transformer encoder to extract the global token vector;

[0012] S4. Input the global token vector into the prediction module, and make a classification decision on the obtained prediction values ​​to obtain the final bank credit prediction result.

[0013] Furthermore, the bank customer credit dataset includes a unique customer identifier, gender, age, region code, occupation, channel code, service months, credit product type, average account balance over the past 12 months, activity status over the past 3 months, and credit rating.

[0014] Furthermore, in the feature processing of S2, the data processing of numerical features involves multiplying the normalized original scalar value of the numerical feature with the corresponding trainable weight matrix, and then combining it with the shared embedding vector to generate a numerical feature embedding vector; the data processing of categorical features involves converting the categorical feature into a numerical representation using a one-hot encoding vector, multiplying the one-hot encoding vector with the corresponding trainable weight matrix, and then combining it with the shared embedding vector to generate a categorical feature embedding vector.

[0015] Furthermore, the expression for the second feature embedding matrix in S3 is:

[0016] T0 = ​​stack[[CLS],T]T i =F i (T i-1 ),

[0017] Where T0 is the second feature embedding matrix, CLS is the learnable one-dimensional embedding vector, and T is the first feature embedding matrix. i =F i (T i-1 ) represents the update performed on the second feature embedding matrix during training.

[0018] Furthermore, the operations performed by the multi-layer Transformer encoder stack unit in S3 include mapping the second feature embedding matrix to a query vector, a key vector, and a value vector; calculating the relationship between features through a multi-head self-attention mechanism; performing a nonlinear transformation using a feedforward neural network; and outputting a global token representing the output vector.

[0019] Furthermore, the prediction module in S4 includes a first linear layer, a first ReLU activation function layer, a Dropout layer, a layer normalization layer, a second ReLU activation function layer, and a second linear layer.

[0020] The operations performed by the prediction module in S4 include: mapping the global token vector to the hidden layer dimension d1 in the first linear layer; retaining non-negative values ​​and suppressing negative values ​​in the first ReLU activation function layer; randomly zeroing some neurons in the Dropout layer; performing standardization in the layer normalization layer; retaining non-negative values ​​and suppressing negative values ​​for the second time in the second ReLU activation function layer; and mapping the d1-dimensional vector to the output space in the second linear layer.

[0021] According to another aspect of the present invention, a bank credit prediction system based on FT-Transformer is provided, the system comprising a feature extraction module, a Transformer module and a prediction module;

[0022] The feature extraction module includes a data acquisition unit, a numerical processing unit, a categorical processing unit, and a stacking unit. The numerical processing unit processes numerical feature data by multiplying the normalized original scalar value of the numerical feature with the corresponding trainable weight matrix and then combining it with a shared embedding vector to generate a numerical feature embedding vector. The categorical processing unit processes categorical feature data by converting the categorical feature into a numerical representation using a one-hot encoding vector, multiplying the one-hot encoding vector with the corresponding trainable weight matrix, and then combining it with a shared embedding vector to generate a categorical feature embedding vector. The stacking unit stacks the numerical feature embedding vector and the categorical feature embedding vector into a matrix and embeds a learnable one-dimensional embedding vector.

[0023] The Transformer module includes a multi-layer Transformer encoder stacking unit and a CLS extraction unit. The multi-layer Transformer encoder stacking unit is used to map the second feature embedding matrix into a query vector, a key vector, and a value vector through a multi-head self-attention mechanism and calculate attention weights, and perform nonlinear transformation through a feedforward neural network. The CLS extraction unit is used to extract the global token vector.

[0024] The prediction module includes a first linear layer, a first ReLU activation function layer, a Dropout layer, a layer normalization layer, a second ReLU activation function layer, and a second linear layer, which are used to obtain predicted values ​​and make classification decisions.

[0025] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0026] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] (1) Shared embedding mechanism enhances feature interaction and compresses model size: By introducing the same shared embedding vector for all numerical and categorical features, replacing the independent bias terms of each feature in the traditional model, different features are forced to interact in a unified embedding space, enhancing cross-feature common learning ability, reducing the total number of model parameters, and reducing the demand for computing resources. In bank credit assessment, it effectively captures the potential correlation between multi-source heterogeneous features such as occupation type, geographical distribution, and account activity, and improves the recognition accuracy and generalization ability of complex credit risk patterns.

[0029] (2) Deep structured prediction layer improves the robustness of feature representation: The global token vector is processed by operation layers in a specific order, namely linear transformation, nonlinear activation, random masking, data standardization, second nonlinear activation and final linear mapping. The inherent laws of features are extracted through double nonlinear activation, and overfitting is suppressed by random masking and standardization to ensure training stability. The core discrimination information is preserved by dimensional compression, which enhances the anti-interference ability of abnormal customer data, such as temporary large transactions, reduces the bank's credit misjudgment rate and improves the reliability of risk decision-making.

[0030] (3) End-to-end heterogeneous feature fusion architecture improves the efficiency of tabular data processing: By constructing a complete processing pipeline, numerical features are transformed by linear scaling combined with shared embedding vectors, and categorical features are transformed by embedding lookup tables combined with shared embedding vectors; the complex dependencies of the transformed features are analyzed through a multi-layer self-attention mechanism, and mixed-type features such as age, occupation, and account balance are processed in a unified manner, eliminating manual feature engineering. The deep model automatically mines nonlinear feature interactions, solving the problem of limitations in processing high-dimensional bank tabular data, accurately quantifying the cross-influence of multiple factors, such as the synergistic effect of occupational stability and consumption activity, and improving the accuracy of credit prediction. Attached Figure Description

[0031] Figure 1 This is a data flow diagram for a bank credit forecasting method based on FT-Transformer.

[0032] Figure 2 This is a structural diagram of a bank credit forecasting system based on FT-Transformer.

[0033] Figure 3 This is a comparison chart of the accuracy of this embodiment with decision trees and category embedding;

[0034] Figure 4 The figures show the ablation experiment results under different structures. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] like Figure 1 The image shows a bank credit forecasting method based on FT-Transformer. The specific steps include:

[0037] S1. Collect bank customer credit datasets including numerical and categorical features, and preprocess them;

[0038] S2. Through the shared embedding vector mechanism, feature processing is performed on the bank customer credit dataset according to the data type of the bank customer credit dataset to obtain the first feature embedding matrix. The data types of the bank customer credit dataset are divided into numerical features and categorical features.

[0039] S3. Combine the learnable one-dimensional embedding vector and the first feature embedding matrix into a second feature embedding matrix; input the second feature embedding matrix into the stacked unit of the multi-layer Transformer encoder to extract the global token vector;

[0040] S4. Input the global token vector into the prediction module, and make a classification decision on the obtained prediction values ​​to obtain the final bank credit prediction result.

[0041] The bank customer credit dataset includes the customer's unique identifier, gender, age, region code, occupation, channel code, service months, credit product type, average account balance over the past 12 months, activity status over the past 3 months, and credit rating.

[0042] Based on bank credit prediction data, the data is divided into training, testing, and validation sets, with data in each set possessing the same type of attributes. In feature processing in S2, numerical features are processed by multiplying the normalized original scalar values ​​of the numerical features by the corresponding trainable weight matrix, and then combining this with a shared embedding vector to generate a numerical feature embedding vector. Categorical features are processed by converting categorical features into numerical representations using one-hot encoding vectors, multiplying the one-hot encoding vectors by the corresponding trainable weight matrix, and then combining this with a shared embedding vector to generate a categorical feature embedding vector.

[0043] Generate an embedding vector T of dimension d for the j-th feature. j The expression is:

[0044]

[0045] Where S is the shared embedding vector across features; f j (x j ) for feature x j The mapping function.

[0046] For data with numerical features, generate numerical feature embedding vectors. The expression is:

[0047]

[0048] in, For the j-th numerical feature, W j (num) This is the corresponding trainable weight matrix.

[0049] For data with categorical features, generate categorical feature embedding vector T. j (num) The expression is:

[0050]

[0051] in, W is the one-hot encoding vector of the j-th categorical feature. j (cat) This is the corresponding trainable weight matrix.

[0052] The final total embedding, i.e. the first feature embedding matrix, is:

[0053]

[0054] Where, k (num) k represents the number of numerical features. (cat) This represents the number of categorical features.

[0055] A global representation of the entire input sequence is learned through a learnable one-dimensional embedding vector. During training, this one-dimensional embedding vector is updated along with the embedding matrix. The learnable one-dimensional embedding vector and the first feature embedding matrix are combined to form the second feature embedding matrix T0, expressed as:

[0056] T0 = ​​stack[[CLS],T]T i =F i (T i-1 ),

[0057] Where T0 is the second feature embedding matrix, CLS is the learnable one-dimensional embedding vector, and T is the first feature embedding matrix. i =F i (T i-1) represents the update performed on the second feature embedding matrix during training.

[0058] The second feature embedding matrix is ​​input into the multi-layer Transformer encoder stack unit to extract the global token vector. The operations performed by the multi-layer Transformer encoder stack unit in S3 include mapping the second feature embedding matrix into a query vector, a key vector, and a value vector; calculating the relationship between features through a multi-head self-attention mechanism; performing a non-linear transformation using a feedforward neural network; and outputting a global token representing the output vector after passing through 6 encoder blocks.

[0059] The global token output by the Transformer layer is further learned using a linear layer incorporating a single hidden layer. The hidden layer consists of a linear layer, a dropout layer, and a batch normalization layer. The hidden layer is followed by a linear layer that maps the high-dimensional representation to the output space, classifying it. The activation function after the linear layer is the ReLU function. The prediction module consists of a first linear layer, a first ReLU activation function layer, a dropout layer, a batch normalization layer, a second ReLU activation function layer, and a second linear layer. The overall expression is:

[0060]

[0061] in, The final output of the Transformer module is the global token. The operations performed include: mapping the global token vector to a hidden layer dimension d1 in the first linear layer; performing element-wise non-negative retention in the first ReLU activation function layer to introduce non-linearity and enhance feature representation; the output length of this layer is d1, with elements being 0 or originally positive, exhibiting sparse non-linear features; randomly zeroing some neurons in the Dropout layer; and performing standardization in the LayerNorm normalization layer, including calculating the mean and standard deviation, subtracting the mean, dividing by the standard deviation, and combining this with learnable scaling / offset parameters to stabilize the data distribution and accelerate convergence. The normalized vector output from the current layer has a length of d1, with a more regular element distribution, a mean close to 0, and a variance close to 1. In the second ReLU activation function layer, non-negative values ​​are retained element-wise for the second time, suppressing negative values ​​and further exploring complex feature relationships. The current layer output vector has a length of d1, with elements being 0 or normalized positive values, resulting in richer non-linear features. In the second linear layer, the d1-dimensional vector is mapped to the output space. Through a final linear transformation, the weight matrix dimension dou×d1 and the bias length dou can be learned. Based on the task requirements and the number of categories corresponding to the classification task, this embodiment uses binary classification for experiments, mapping features to an output of dimension dou. In classification tasks, the subsequent Softmax transformation converts the data into class probabilities, completing the process from global tokens to prediction results.

[0062] In existing technologies, such as the original FT-Transformer or traditional embedding methods, feature embedding typically involves setting independent biases or independent embedding vectors for each feature, leading to parameter redundancy and limited information interaction between features. This embodiment strengthens feature representation by directly adding a shared embedding vector to the embedding vector for each class during training. The shared embedding vector reduces the total number of parameters and shrinks the model size. This added shared embedding vector promotes feature interaction and helps the model capture complex relationships between features, improving model accuracy while reducing model size. In existing technologies, such as the global token CLS vector output by the Transformer, it is often directly mapped to the prediction result through a simple linear layer, making it difficult to fully utilize the global features contained in the CLS. In this invention, when using the Transformer to generate CLS for prediction, an additional multilayer perceptron with 32 units is added to the head, and a hidden layer containing a linear layer, a dropout layer, and a batch normalization layer is introduced. The dropout layer reduces the risk of overfitting and improves the accuracy of converting the CLS into the final predicted class.

[0063] This embodiment also provides a system for a bank credit forecasting method based on FT-Transformer. For example... Figure 2 As shown, the system includes a feature extraction module, a Transformer module, and a prediction module. In the figure, x is the input vector, i.e., the bank customer credit dataset; the Feature Tokenizer is the structure in the feature extraction module; T is the first feature embedding matrix; and T0 is the second feature embedding matrix. This is the final prediction result.

[0064] The feature extraction module includes a data acquisition unit, a numerical processing unit, a categorical processing unit, and a stacking unit. The numerical processing unit processes numerical features by multiplying the normalized original scalar values ​​of the numerical features with the corresponding trainable weight matrix and then combining them with a shared embedding vector to generate a numerical feature embedding vector. The categorical processing unit processes categorical features by converting categorical features into numerical representations using one-hot encoding vectors, multiplying the one-hot encoding vectors with the corresponding trainable weight matrix, and then combining them with a shared embedding vector to generate a categorical feature embedding vector. The stacking unit stacks the numerical and categorical feature embedding vectors into a matrix and embeds a learnable one-dimensional embedding vector.

[0065] The Transformer module includes a multi-layer Transformer encoder stack unit and a CLS extraction unit. The multi-layer Transformer encoder stack unit is used to map the second feature embedding matrix into query vector, key vector and value vector through a multi-head self-attention mechanism and calculate attention weights, and perform nonlinear transformation through a feedforward neural network. The CLS extraction unit is used to extract the global token vector.

[0066] The prediction module includes a first linear layer, a first ReLU activation function layer, a Dropout layer, a layer normalization layer, a second ReLU activation function layer, and a second linear layer, which are used to obtain predicted values ​​and make classification decisions.

[0067] like Figure 3 The image shows a comparison of the accuracy of this embodiment with decision trees and category embedding. The test used a bank customer credit dataset containing both numerical and categorical features. The classification accuracy was calculated using an FT-Transformer-based credit prediction scheme, traditional decision trees, and category embedding methods. Figure 4 As shown in the figure, the ablation experiment results under different structures were tested using the complete structure including shared embedding vectors and a single hidden layer prediction module, the structure with shared embedding vectors removed and only the single hidden layer prediction module retained, and the structure with the single hidden layer removed and only the shared embedding vectors retained. As can be seen from the figure, the shared embedding vectors improve training stability, while the single hidden layer structure accelerates convergence and suppresses overfitting. The quantitative results of the ablation experiment are shown in Table 1.

[0068] Table 1 Quantitative results of the ablation experiment

[0069] Model Accuracy Embedded vector addition + hidden layer 0.860074 Hidden layers only 0.859066 Only embedding vector addition 0.859989

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0071] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0072] Multiple components in the device are connected to an I / O interface, including: input units such as a keyboard, mouse, etc.; output units such as various types of displays, speakers, etc.; storage units such as disks, optical disks, etc.; and communication units such as network interface cards, modems, wireless transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention may be implemented as a computer software program tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or the communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method of the present invention described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute the method of the present invention by any other suitable means (e.g., by means of firmware).

[0073] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0074] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0075] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0076] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A bank credit prediction method based on FT-Transformer, characterized in that, The specific steps include: S1, collecting a bank customer credit data set including numerical features and categorical features, and preprocessing; S2, through a shared embedding vector mechanism, according to the data type of the bank customer credit data set, respectively processing the bank customer credit data set to obtain a first feature embedding matrix, the data type of the bank customer credit data set is divided into numerical features and categorical features; S3, combining the learnable one-dimensional embedding vector and the first feature embedding matrix into a second feature embedding matrix; inputting the second feature embedding matrix into a multi-layer Transformer encoder stacking unit to extract a global token vector; S4, inputting the global token vector into a prediction module, and classifying the obtained prediction value to obtain a final bank credit prediction result.

2. The FT-Transformer-based bank credit prediction method of claim 1, wherein, The bank customer credit data set includes customer unique identification, gender, age, region code, occupation, channel code, service month, credit product type, average account balance in the last 12 months, active state in the last 3 months, and credit level mark.

3. The FT-Transformer-based bank credit prediction method of claim 1, wherein, In the feature processing in S2, the data processing of the numerical features is to multiply the normalized original scalar value of the numerical features with the corresponding trainable weight matrix, and then combine the shared embedding vector to generate the numerical feature embedding vector; the data processing of the categorical features is to convert the categorical features into numerical representation by using a one-hot encoding vector, multiply the one-hot encoding vector with the corresponding trainable weight matrix, and then combine the shared embedding vector to generate the categorical feature embedding vector.

4. The FT-Transformer-based bank credit prediction method of claim 1, wherein, The expression of the second feature embedding matrix in S3 is: T0 = stack[[CLS], T]T i = F i (T i-1 ), wherein T0 is a second feature embedding matrix, CLS is a learnable one-dimensional embedding vector, T is a first feature embedding matrix, T i = F i (T i-1 ) is an update of the second feature embedding matrix made during training.

5. The FT-Transformer-based bank credit prediction method of claim 1, wherein, The operations performed by the multi-layer Transformer encoder stacking unit in S3 include mapping the second feature embedding matrix into a query vector, a key vector and a value vector; calculating the relationship between features through a multi-head self-attention mechanism; Using a feedforward neural network for nonlinear transformation, outputting a global token representing an output vector.

6. The FT-Transformer-based bank credit prediction method of claim 1, wherein, The prediction module in S4 includes a first linear layer, a first ReLU activation function layer, a Dropout layer, a layer normalization layer, a second ReLU activation function layer, and a second linear layer.

7. The FT-Transformer-based bank credit prediction method of claim 6, wherein, The operations performed by the prediction module in S4 include mapping the global token vector to a hidden layer dimension d1 in the first linear layer; retaining non-negative values and suppressing negative values in the first ReLU activation function layer; randomly setting some neurons to zero in the Dropout layer; performing standardization processing in the layer normalization layer; Retaining non-negative values and suppressing negative values for the second time in the second ReLU activation function layer; mapping the d1-dimensional vector to the output space in the second linear layer.

8. A system for the FT-Transformer-based bank credit prediction method according to any one of claims 1 to 7, characterized in that, The system includes a feature extraction module, a Transformer module and a prediction module; The feature extraction module comprises a data acquisition unit, a numerical processing unit, a category processing unit and a stacking unit. The numerical processing unit is used for data processing of numerical features, multiplies normalized original scalar values of numerical features with corresponding trainable weight matrices, and then combines shared embedding vectors to generate numerical feature embedding vectors. The category processing unit is used for data processing of category features, converts category features into numerical representation by using one-hot encoding vectors, multiplies one-hot encoding vectors with corresponding trainable weight matrices, and then combines shared embedding vectors to generate category feature embedding vectors. The stacking unit is used for stacking numerical feature embedding vectors and category feature embedding vectors into a matrix, and embedding a learnable one-dimensional embedding vector. The Transformer module comprises a multi-layer Transformer encoder stacking unit and a CLS extraction unit. The multi-layer Transformer encoder stacking unit is used for mapping the second feature embedding matrix into query vectors, key vectors and value vectors by a multi-head self-attention mechanism and calculating attention weights, and performing nonlinear transformation by a feedforward neural network. The CLS extraction unit is used for extracting a global token vector. The prediction module comprises a first linear layer, a first ReLU activation function layer, a Dropout layer, a layer normalization layer, a second ReLU activation function layer and a second linear layer, which are used to obtain a prediction value and make a classification decision.

9. An electronic device comprising a memory and a processor, said memory having stored thereon a computer program, characterized in that, The processor executes the program to implement the method of any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-8.

Images