A hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree

Abstract

This invention discloses a hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree, relating to the field of financial risk control. It addresses the technical problem that existing hybrid modeling methods cannot extract high-order interaction information while retaining the advantages of the original numerical data. The invention includes constructing a dual-channel data processing stream: the first channel retains all numerical features to obtain the original numerical features, and the second channel discretizes all numerical features to obtain standardized numerical features. A business domain embedding mechanism is introduced to construct a business domain-aware feature representation based on the standardized numerical features. The feature representation is processed by a high-order feature extraction module to output latent vectors, which are then grouped and concatenated according to the business domain to obtain high-order feature vectors. The original numerical features and the high-order feature vectors are concatenated and input into a gradient boosting decision tree to output a risk score. This invention utilizes discretization and embedding techniques to enable the neural network to adapt to high-missing data while retaining the original numerical data to supply the tree model, ensuring the model's robustness under data quality fluctuations.

Description

Technical Field

[0001] This invention relates to the field of financial risk control, specifically to a hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree. Background Technology

[0002] In the field of financial risk control, data generated from real-world business scenarios primarily exists in the form of structured tabular data. This type of data has the following significant characteristics:

[0003] 1. High dimensionality and high sparsity: It contains hundreds of features, and due to business collection reasons, a large number of features (such as third-party data and historical behavior records) have a high proportion of missing values;

[0004] 2. Clear business grouping attributes: Features do not exist independently, but belong to different business categories (such as user basic information domain, multiple lending domain, App installation list domain, etc.), and there are often strong internal relationships between similar features.

[0005] Currently, gradient boosting decision trees have long been the mainstream modeling solution due to their ability to naturally handle missing values ​​and maintain the monotonicity of numerical features (i.e., ensuring that the risk score changes monotonically as a specific feature value increases). However, traditional tree models often rely on tedious manual feature engineering when dealing with high-order nonlinear interactions across business domains, and they are difficult to automatically capture deep implicit patterns.

[0006] On the other hand, while deep neural networks based on attention mechanisms are good at capturing complex interactions, they have obvious drawbacks when directly applied to such high-missing, high-dimensional tabular data: First, after continuous numerical features are mapped by embedding, the monotonicity and boundary information of the original values ​​are lost, making the model sensitive to outliers; Second, existing network structures often ignore the business grouping attributes of features, treating all features as homogeneous nodes for interactive calculation, introducing a large amount of cross-domain noise.

[0007] Therefore, there is an urgent need for a hybrid modeling method that can both utilize tree models to preserve the monotonicity and missing value of the original numerical data and utilize neural networks to capture higher-order interactive information of different domains. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree, which solves the technical problem that existing hybrid modeling methods cannot extract high-order interaction information while retaining the advantages of the original numerical values.

[0009] Hybrid risk control modeling methods based on self-attention mechanisms and gradient boosting decision trees include:

[0010] S01: Obtain table-type data generated during business processes, construct a dual-channel data processing stream. The first channel retains all numerical features to obtain the original numerical features, and the second channel discretizes all numerical features to obtain standardized numerical features.

[0011] S02: Introduce a business domain embedding mechanism to construct a business domain-aware feature representation based on the standardized numerical features generated by the second channel;

[0012] S03: The feature representation is processed by the high-order feature extraction module to output the latent vector, and then the latent vector is grouped and concatenated according to the business domain to obtain the high-order feature vector.

[0013] S04: Original numerical features X raw With the extracted high-order feature vector X emb Horizontal concatenation is performed to construct an augmented feature space input gradient boosting decision tree, which outputs a risk score.

[0014] Furthermore, the discretization process in step S01 includes:

[0015] The quantile discretization method is used to map the numerical features in the table to discrete IDs, with missing values ​​mapped to specific IDs, and non-missing values ​​divided into K bins according to their distribution. The process of determining the value of K is as follows:

[0016] (1) Fit the bin boundaries using only the non-missing values ​​of the feature on the training set;

[0017] (2) The actual number of boxes ≤ K;

[0018] (3) The actual number of bins ≤ the number of valid samples for this feature;

[0019] (4) To avoid the instability of values ​​due to excessively dense bin boundaries, the maximum reasonable number of bins is calculated based on the range of the effective value of this feature and the minimum allowable bin width. The actual number of bins shall not exceed this value.

[0020] (5) If the number of valid samples is too small or the standard deviation of the valid values ​​is too small, the actual number of bins shall be further limited to no more than max.

[0021] Further, step S02 includes:

[0022] Based on business logic, standardized numerical features are divided into M business domains, and a learnable domain vector is initialized for each business domain. The business domains include user application class, user multi-head class, APP class, user basic information class, relationship network class, user payment class, and call record class. Before the features enter the interaction layer, the value of each standardized numerical feature is embedded and fused with the domain vector of its respective domain, so that the model can perceive the business category to which the feature belongs, and obtain the feature representation perceived by the fused business domain.

[0023] Further, step S03 includes: grouping the latent vectors according to their respective business domains, performing weighted average pooling within each domain to obtain M domain vectors; mapping each domain vector to the output dimension through a dedicated two-layer MLP for that domain to obtain an output vector; concatenating the output vectors of the M domains in the last dimension to obtain a high-order feature vector.

[0024] Furthermore, the high-order feature extraction module is implemented by cascading multiple multi-head self-attention modules. Each multi-head self-attention module performs LayerNorm on the input, and then performs a linear transformation to obtain Q, K, and V. These are then split into H parts by head, and scaled dot product attention is performed within each head. The outputs of each multi-head self-attention module are concatenated and added to the input of that layer, i.e., the residual connection within the layer is: the output of that layer = the input of that layer + the self-attention result of that layer.

[0025] The beneficial effects of this invention include:

[0026] 1. This method addresses the challenge of modeling highly missing tabular data by employing a dual-channel design. Two representations are constructed in parallel using the same sample and labels, and then explicitly fused at the decision level to balance numerical physical meaning with higher-order interaction capabilities. Channel A retains the original numerical features without imputation or standardization, directly inputting them into a gradient boosting decision tree. This leverages the tree model's inherent ability to handle missing values ​​while preserving the monotonicity and boundary information of features such as amount and frequency, meeting the interpretability and monotonicity requirements of financial risk control. Channel B discretizes the same set of feature quantiles and inputs them into a self-attention network. This network learns higher-order nonlinear interactions in the discrete space, generating refined features and avoiding the loss of monotonicity and outlier sensitivity issues that arise from directly processing continuous values ​​in neural networks. The two channels are concatenated along the feature dimension, with the same set of tree models performing unified splitting and scoring in the augmented feature space, achieving input-level synergy rather than the fusion of two independent models. This method utilizes discretization and embedding techniques to adapt the neural network to highly missing data while preserving the original numerical values ​​for the tree model, ensuring the model's robustness under data quality fluctuations.

[0027] 2. Enhance the business semantics of feature interactions by introducing a business domain embedding mechanism. Addressing the characteristics of financial tabular data—divided by business segments and with strong correlations among similar features—a business domain partitioning mechanism is introduced into the attention mechanism: features are divided into M domains based on business logic, and a learnable domain vector is set for each domain. Features are fused before entering self-attention, ensuring that features within the same domain share domain biases while different domains are distinguishable in their representation space. Without altering the attention calculation method, this provides the model with prior information about the business category to which the features belong, suppressing cross-domain noise interactions and improving the interpretability and effectiveness of interactive features in risk control scenarios.

[0028] 3. To achieve structured representation of feature information, a domain-based aggregation output strategy is proposed. Addressing the issue of global pooling easily obscuring strong local signals, global pooling is not performed after attention output. Instead, latent vectors are grouped according to the same business domain, and weighted pooling is performed within each domain. Then, a domain-specific MLP is used to obtain M domain vectors, which are concatenated into Xemb. This ensures that risk signals from each business subdomain occupy independent dimensions in the vector space, facilitating the tree model's use of features and importance analysis according to business dimensions. Attached Figure Description

[0029] Figure 1 This is a flowchart of a hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree involved in the embodiments of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0031] Example 1

[0032] The following is in conjunction with the appendix Figure 1 Specific embodiments of the present invention will be described in detail;

[0033] The hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree includes the following steps:

[0034] S01: Construct a dual-channel data processing stream. For the original financial tabular dataset, two independent data streams are constructed, one retaining the physical meaning of the data and the other mining interactive information. The original financial tabular dataset refers to tabular data generated during the actual business operations of real financial companies, containing detailed characteristic records of borrowers for each lending transaction. The purpose of the dataset is to filter out high-risk transactions; the total data volume is approximately 4 million records with over 600 columns of features, making it high-dimensional tabular data. Due to privacy requirements, semantic features of borrowers cannot be included; each column of features corresponds to a numerical feature. Furthermore, due to business collection methods, the tabular dataset naturally has a high missing rate and clear business groupings and relationships between groups.

[0035] Channel A - Original Numerical Feature Channel: Retains the original values ​​of all numerical features, including missing values, without imputation or standardization. It is directly used as the input X to the subsequent gradient boosting decision tree model. raw The tree model's splitting algorithm directly handles missing values ​​while fully preserving the monotonicity boundary information of features such as amount and frequency;

[0036] Channel B - Discretized Feature Channel: A quantile discretization method is used to map the numerical features in the table to discrete IDs. Specifically, missing values ​​(NaN) are mapped to specific IDs (e.g., 0), and non-missing values ​​are divided into K bins according to their distribution. This step generates the data X. tokens This method is used as input to deep neural networks, effectively avoiding the problem of neural networks being sensitive to outliers and missing values. The specific process is as follows:

[0037] 1. Binning Strategy and Encoding: A quantile strategy is adopted, with quantiles calculated linearly. Missing values ​​are mapped individually to discrete 0, while non-missing values ​​are mapped to 1 to the actual number of bins based on the binning result; the vocabulary size for each feature is the actual number of bins + 1 (including missing classes).

[0038] 2. Default number of boxes: The number of boxes K is set as a hyperparameter. In this embodiment, the default K=10 (i.e., =10);

[0039] 3. Dynamically determine the actual number of bins based on features: Discretize each feature individually using quantiles. The actual number of bins used can be less than or equal to K, determined by the following constraints:

[0040] (1) Fit the bin boundaries using only the non-missing values ​​of the feature on the training set;

[0041] (2) The actual number of boxes ≤ K;

[0042] (3) The actual number of bins ≤ the number of valid samples (non-missing samples) for this feature;

[0043] (4) To avoid the instability of values ​​due to excessively dense bin boundaries, the maximum reasonable number of bins is calculated based on the range of the effective value of this feature and the minimum allowable bin width (e.g., 1e-6). The actual number of bins shall not exceed this value.

[0044] (5) If the number of valid samples is too small (e.g., less than 10) or the standard deviation of the valid values ​​is too small (e.g., less than 1e-8), then the actual number of bins will be further limited to no more than max(2, number of valid samples / / 2).

[0045] S02: Construct a business domain-aware feature representation. For the grouping characteristics of tabular data, introduce a business domain embedding mechanism:

[0046] Based on the business logic, all features are divided into M business domains, and a learnable domain vector is initialized for each business domain; in this embodiment, M=7, namely user application class, user multi-head class, APP class, user basic information, relationship network class, user payment class, and call record class.

[0047] Before the features enter the interaction layer, the value of each feature is embedded and fused with the domain vector of its respective domain, so that the model can perceive the business category to which the feature belongs and obtain the feature representation of the fused domain information, which serves as the input of the subsequent higher-order feature extraction module.

[0048] Specifically, each business domain corresponds to a learnable domain vector. The dimension is consistent with the feature embedding dimension, denoted as d=32; the domain vector is stored in an embedding table of shape (M,d); a small-range uniform initialization [-0.01, 0.01] is used, and a scaling factor α=0.1 is used during fusion to control the initial contribution of domain information and avoid masking the feature embedding signal;

[0049] Embed the value of the i-th feature ( The domain vectors of the domain to which the vector belongs are merged according to the following formula: This yields a feature representation that incorporates domain information (with a shape of batch × ...). ×d), which serves as the input to the higher-order feature extraction module.

[0050] S03: High-order feature extraction and domain-specific aggregation output. While preserving risk signals from different business dimensions, structured high-order features are obtained, and the following steps are performed sequentially:

[0051] The feature representation output by S02 is fed into the high-order feature extraction module. This module consists of multiple layers of multi-head self-attention cascaded, with residual connections within each layer (the output of this layer = the input of this layer + the self-attention result of this layer). Layers are sequentially connected, and high-order feature interactions are gradually refined through multi-layer stacking to output richer latent vectors.

[0052] The higher-order feature extraction module is used to further refine the feature representation output by feature fusion, outputting richer latent vectors (the shape remains batch×). ×d), for subsequent domain aggregation. This module is implemented by a multi-layer multi-head self-attention cascade; a total of 3 layers of multi-head self-attention modules are cascaded, with sequential connections between layers, and the output of the previous layer serving as the input of the next layer; the number of heads in a single layer of multi-head self-attention module is H=4, the embedding dimension is d=32, and the dimension of each head is d_k = d / H = 8. After performing LayerNorm on the input, each layer of multi-head self-attention module performs a linear transformation to obtain Q, K, and V, which are then split into H parts by head, and scaled dot product attention (scale factor is ) is performed within each head. The outputs of each multi-head self-attention module are concatenated and then added to the input of that layer as residuals, i.e., intra-layer residual connections: output of that layer = input of that layer + result of self-attention of that layer. Gradient flow is maintained and higher-order interactions are gradually refined through residual connections.

[0053] The latent vectors output by the high-order feature extraction module are grouped according to their respective business domains. Weighted average pooling is then performed within each domain (weights are generated by a small attention network within the domain and normalized by softmax) to obtain M domain vectors, each with a dimension of d=32.

[0054] Each domain vector is mapped to the output dimension via a two-layer MLP (containing ReLU and Dropout) dedicated to that domain. =16; Concatenate the output vectors of the M domains in the last dimension to obtain the higher-order feature vector. The dimension is M× =7×16=112.

[0055] higher-order feature vectors It is composed of M=7 consecutive blocks stitched together along the feature dimension, each consecutive block having a dimension of =16, therefore Total dimensions M× =112. The m-th block (m=0,…,M-1) corresponds one-to-one with the m-th business domain. For example, block 0 corresponds to "User Application Class", block 1 corresponds to "User Multi-head Class", etc. In terms of source, the value of the m-th block is only the latent vector corresponding to the features belonging to the m-th business domain (i.e., the representation of the features within that domain output by the high-order feature extraction module). After grouping by business domain, a weighted average pooling is performed within that domain, and then mapped through a two-layer MLP dedicated to that domain to... In this context, the weighted average pooling is generated by a small attention network within the domain and then normalized using softmax. Therefore, the m-th block is determined solely by features within the m-th business domain and their higher-order interactions within that domain, without mixing with features from other domains during the aggregation phase. This results in a vector space carrying the aggregated risk-related representation of the business sub-domain, i.e., the risk signal of that business sub-domain.

[0056] The training of the high-order feature extraction module uses binary cross-entropy loss, that is, applying a sigmoid function to the logits of the model output and then calculating the binary cross-entropy. It is pre-trained on labeled data to learn the high-order feature representation of the data.

[0057] First, during the training phase, labeled data is used to train the self-attention feature extraction network with binary cross-entropy loss, so that the representation of each domain block has discriminative power for risk labels; after freezing the network, the inferences obtained from the full dataset are... Therefore, it is statistically associated with the risk status of the business sub-domain. Then, the tree model learns the splitting rules and leaf weights on each dimension of X_final; thus... Each dimension, as well as each dimension within each block, can be used as a splitting feature or participate in leaf score calculation, establishing an explicit, dimension-by-dimensional traceable mapping relationship with the final risk score.

[0058] The training hyperparameters of the higher-order feature extraction module include: =32, number of attention layers L=3, number of heads H=4, dropout=0.2, output dimension per domain =16, number of business domains M=7, domain embedding scaling factor α=0.1; during training =100, =0.0001, weight decay ( ) = 1e-4, =256, early stop patience value=10; learning rate scheduling uses ReduceLROnPlateau (monitoring validation set AUC), factor=0.5, patience=5. =1e-6.

[0059] S04: Fusion and Decision-Making of Explicit and Latent Feature Spaces

[0060] The original numerical feature X from step S01 raw With the extracted high-order feature vector X emb Perform horizontal stitching to construct an augmented feature space:

[0061] Will The input gradient boosting decision tree is used for final training. The tree model utilizes its split structure, on the one hand, to utilize X... raw Maintain monotonicity constraints on core indicators, while utilizing X emb The introduced high-level interactive information is finely adjusted to output the final risk score.

[0062] Gradient boosting decision trees are trained using a binary classification objective function, i.e., log loss, to augment features. Conduct supervised training; set the parameters according to the ratio of positive to negative samples in the training set. To alleviate category imbalance.

[0063] The hyperparameters of a gradient boosting decision tree include: The ratio of positive to negative samples in the training set is 14.

[0064] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. A hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree, characterized in that, Includes the following steps: S01: Obtain table-type data generated during business processes, construct a dual-channel data processing stream. The first channel retains all numerical features to obtain the original numerical features, and the second channel discretizes all numerical features to obtain standardized numerical features. S02: Introduce a business domain embedding mechanism to construct a business domain-aware feature representation based on the standardized numerical features generated by the second channel; S03: The feature representation is processed by the high-order feature extraction module to output the latent vector, and then the latent vector is grouped and concatenated according to the business domain to obtain the high-order feature vector. S04: original numerical features X raw with the extracted high-order feature vector X emb perform horizontal concatenation, construct an augmented feature space input gradient boosting decision tree, and output a risk score.

2. The hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree as described in claim 1, characterized in that, The discretization process in step S01 includes: The quantile discretization method is used to map the numerical features in the table to discrete IDs, with missing values ​​mapped to specific IDs, and non-missing values ​​divided into K bins according to their distribution. The process of determining the value of K is as follows: (1) Fit the bin boundaries using only the non-missing values ​​of the feature on the training set; (2) The actual number of boxes ≤ K; (3) The actual number of bins ≤ the number of valid samples for this feature; (4) To avoid the instability of values ​​due to excessively dense bin boundaries, the maximum reasonable number of bins is calculated based on the range of the effective value of this feature and the minimum allowable bin width. The actual number of bins shall not exceed this value. (5) If the number of valid samples is too small or the standard deviation of the valid values ​​is too small, the actual number of bins shall be further limited to no more than max.

3. The hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree as described in claim 1, characterized in that, Step S02 includes: Based on business logic, standardized numerical features are divided into M business domains, and a learnable domain vector is initialized for each business domain. The business domains include user application class, user multi-head class, APP class, user basic information class, relationship network class, user payment class, and call record class. Before the features enter the interaction layer, the value of each standardized numerical feature is embedded and fused with the domain vector of its respective domain, so that the model can perceive the business category to which the feature belongs, and obtain the feature representation perceived by the fused business domain.

4. The hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree as described in claim 1, characterized in that, Step S03 includes: grouping the latent vectors according to their respective business domains, performing weighted average pooling within each domain to obtain M domain vectors; mapping each domain vector to the output dimension through a dedicated two-layer MLP for that domain to obtain an output vector; concatenating the output vectors of the M domains in the last dimension to obtain a high-order feature vector.

5. The hybrid risk control modeling method based on self-attention mechanism and gradient boosting decision tree as described in claim 1, characterized in that, The high-order feature extraction module is implemented by cascading multiple multi-head self-attention modules. Each multi-head self-attention module performs LayerNorm on the input and then performs a linear transformation to obtain Q, K, and V, which are split into H parts by head. Scaling dot product attention is performed within each head. The outputs of each multi-head self-attention module are concatenated and added to the input of that layer, i.e., the residual connection within the layer is: the output of that layer = the input of that layer + the self-attention result of that layer.

Images