GBDT-based data drift analysis method, apparatus and device, and storage medium
Through the GBDT-based data drift analysis method, data path flow is predicted and compared, data drift is calculated, and data drift is attributed to specific features, the problem of lack of quantification and factor interpretation of data drift in the prior art is solved, and the analysis accuracy of data drift and strategy formulation effect is improved.
Patent Information
- Application Number
- CN202411767773.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-03-04
AI Technical Summary
The existing data drift detection methods lack the fine and quantitative description of key information such as drift degree, speed, and direction, and fail to deeply analyze the specific data factors that cause drift, which limits the effectiveness of strategy formulation.
Using the GBDT-based data drift analysis method, the GBDT model predicts the standard path flow of the preset decision tree in the target data set through the pre-trained gradient enhancement decision tree, and compares it with the current path flow, calculates the data drift, and attributes the data drift to specific features through an exemplary function.
It realizes the accuracy of deep quantification of data drift and factor-level interpretation, provides interpretability of model predictions, and improves the strategy formulation and execution effect of data drift.
Smart Images
Figure CN119557777A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data analysis technology, which is applied to the field of financial technology business, and in particular to a data drift analysis method, device, equipment and storage medium based on GBDT. Background Art
[0002] With the rapid development of artificial intelligence, the application of artificial intelligence in the field of data analysis has also become an important method of data analysis. Data interpretability is an important consideration in the field of modern data analysis and modeling, especially in scenarios involving high-risk decision-making, regulatory compliance, and user trust building. The explanation of data drift requires not only the identification of the occurrence of drift, but also the revelation of the specific nature and degree of drift and the influencing factors behind it, providing a clear and operational basis for model adjustment, data update, and decision optimization. However, existing data drift detection methods often focus on the qualitative description and overall perception of drift, lack of in-depth analysis and quantitative evaluation of drift, and the ability to attribute it to specific data factors. This undoubtedly limits the formulation and implementation of strategies in response to data drift. Specifically, the main defects in current data drift research are manifested in the following aspects:
[0003] Most existing drift detection technologies indirectly reflect the existence of data drift through statistical indicators or changes in model performance, but often lack detailed and quantitative descriptions of key information such as the degree, speed, and direction of drift. This non-quantitative processing method makes it difficult to accurately assess the actual impact of drift on model performance, and it is impossible to set clear thresholds or trigger conditions for drift response strategies.
[0004] Most data drift research focuses on changes in overall data distribution or model output, and fails to deeply analyze the specific data factors that cause drift (such as feature variables, correlations, etc.). Although this macro perspective helps to quickly locate the occurrence of drift, it cannot reveal the internal mechanism of drift, that is, which factors have undergone what changes and what form of drift has been caused. The lack of accurate grasp of the root causes of drift will hinder the targeted adoption of measures such as data preprocessing, feature engineering, and model updates to effectively alleviate or eliminate the negative impact of drift. Therefore, how to improve the accuracy of the deep quantification of data drift and the factor-level explanation has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] The present application provides a data drift analysis method, device, equipment and storage medium based on GBDT to improve the accuracy of the deep quantification of data drift and the factor-level interpretation.
[0006] In a first aspect, the present application provides a data drift analysis method based on GBDT, the method comprising:
[0007] Predict the standard path flow of the preset decision tree in the target data set based on the pre-trained gradient boosting decision tree GBDT model;
[0008] Obtaining the current path flow of the preset decision tree in the target data set, and calculating all data drifts between the current path flow and the standard path flow;
[0009] The total data drift amount is attributed to the target feature in the decision tree through a preset indicative function to generate data drift analysis information.
[0010] In a second aspect, the present application also provides a data drift analysis device based on GBDT, the device comprising:
[0011] The standard path traffic prediction module is used to predict the standard path traffic of the preset decision tree in the target data set based on the pre-trained gradient boosting decision tree GBDT model;
[0012] A total data drift calculation module, used for obtaining the current path flow of the preset decision tree in the target data set, and calculating the total data drift between the current path flow and the standard path flow;
[0013] The data drift analysis information generation module is used to attribute all the data drift amounts to the target features in the decision tree through a preset indicative function to generate data drift analysis information.
[0014] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the data drift analysis method based on GBDT as described above when executing the computer program.
[0015] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the data drift analysis method based on GBDT as described above.
[0016] The present application discloses a data drift analysis method, device, equipment and storage medium based on GBDT. The data drift analysis method based on GBDT includes predicting the standard path flow of a preset decision tree in a target data set according to a pre-trained gradient boosting decision tree GBDT model; obtaining the current path flow of the preset decision tree in the target data set, calculating the total data drift between the current path flow and the standard path flow; attributing the total data drift to the target feature in the decision tree through a preset characteristic function, and generating data drift analysis information. In the above manner, the present application can timely discover changes in data distribution by predicting the standard path flow of a preset decision tree in a target data set and comparing it with the current path flow. By attributing the data drift to specific features through an characteristic function, it is possible to clearly understand which features cause data drift, providing the interpretability of model predictions, and thereby improving the accuracy of the deep quantification of data drift and the factor-level interpretation. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 is a schematic flow chart of a data drift analysis method based on GBDT provided in the first embodiment of the present application;
[0019] Figure 2 It is a GBDT training schematic diagram of a GBDT-based data drift analysis method provided by the first embodiment of the present application;
[0020] Figure 3 is a schematic flow chart of a data drift analysis method based on GBDT provided in the second embodiment of the present application;
[0021] Figure 4 is a schematic flow chart of a data drift analysis method based on GBDT provided in the third embodiment of the present application;
[0022] Figure 5 A schematic block diagram of a data drift analysis device based on GBDT provided in an embodiment of the present application;
[0023] Figure 6 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0026] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0027] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0028] The embodiments of the present application provide a data drift analysis method, device, equipment and storage medium based on GBDT. Among them, the data drift analysis method based on GBDT can be applied to the server. By predicting the standard path flow of the preset decision tree in the target data set and comparing it with the current path flow, the change of data distribution can be discovered in time. By attributing the data drift amount to specific features through the indicative function, it can be clearly understood which features cause data drift, providing the interpretability of model prediction, thereby improving the accuracy of the deep quantification of data drift and the factor-level interpretation. Among them, the server can be an independent server or a server cluster.
[0029] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0030] See also Figure 1 , Figure 1It is a schematic flow chart of a data drift analysis method based on GBDT provided by the first embodiment of the present application. The data drift analysis method based on GBDT can be applied to the server, and is used to predict the standard path traffic of the preset decision tree in the target data set and compare it with the current path traffic, so as to timely discover the changes in data distribution, attribute the data drift amount to specific features through the indicative function, and clearly understand which features cause data drift, thereby providing the interpretability of model prediction, thereby improving the accuracy of the deep quantification of data drift and the factor-level interpretation.
[0031] like Figure 1 As shown, the data drift analysis method based on GBDT specifically includes steps S10 to S30.
[0032] Step S10, predicting the standard path flow of the preset decision tree in the target data set according to the pre-trained gradient boosting decision tree GBDT model;
[0033] Specifically, in the GBDT (Gradient Boosting Decision Tree) model, each decision tree consists of a series of internal nodes (feature splitting points) and leaf nodes. Each leaf node represents a path from the root node to the node, and the feature splitting points on the path determine the flow direction of the data instance. The path flow is defined as the proportion of data instances following the path to the total data. Suppose the decision tree t has N leaf nodes, and the path flow corresponding to the i-th leaf node is w i Then we have:
[0034]
[0035] This means that the sum of all path flows is equal to 100%, that is, all data instances are assigned to a leaf node. The calculation of path flow depends on the segmentation of data by feature branches. For a feature split point, if the feature value meets the splitting condition, the data instance continues to move along the current branch; otherwise, it switches to another branch.
[0036] This process can be regarded as the reduction of path flow by feature branches, where the flow of the root node is 1. This process can be expressed as:
[0037]
[0038] That is, on the data set X, the flow of the ith path of the tth decision tree in the gbdt model is the sum of the flow reduction ΔD brought by the feature k branch. Figure 2 As shown, Figure 2This is a GBDT training diagram of a data drift analysis method based on GBDT provided by the first embodiment of the present application. The prediction result of decision tree t on data set X can be expressed based on traffic:
[0039]
[0040] Where P t is the number of paths on decision tree t, f t,i is the predicted value of the leaf node of the i-th path in decision tree t.
[0041] Step S20, obtaining the current path flow of the preset decision tree in the target data set, and calculating the total data drift between the current path flow and the standard path flow;
[0042] Step S30: attributing all data drift amounts to target features in the decision tree through a preset indicative function to generate data drift analysis information.
[0043] In one embodiment, based on the idea of traffic flow, if drift occurs, it must be because the traffic through the node has changed on different data sets, that is, At the same time, the prediction drift of decision tree t can also be expressed as ΔFX t =FX t (X 2 )-FX t (X 1 ). Since GBDT is an additive tree model, the drift of the decision tree can be extended to the entire GBDT model, that is, ΔMX = ∑ΔFX t , and then put it into the previous formula to get:
[0044]
[0045] In order to attribute the features, we can add an indicative function, X j Representing the jth feature, we can get the drift caused by feature j:
[0046]
[0047] Additivity is satisfied here, and the sum of the drift of all features is the total drift of the model:
[0048]
[0049] The present embodiment discloses a data drift analysis method, apparatus, device and storage medium based on GBDT, wherein the data drift analysis method based on GBDT includes predicting the standard path flow of a preset decision tree in a target data set according to a pre-trained gradient boosting decision tree GBDT model; obtaining the current path flow of the preset decision tree in the target data set, calculating all data drift amounts between the current path flow and the standard path flow; attributing all data drift amounts to target features in the decision tree through a preset characteristic function, and generating data drift analysis information. In the above manner, the present application can timely discover changes in data distribution by predicting the standard path flow of a preset decision tree in a target data set and comparing it with the current path flow, and can clearly understand which features cause data drift by attributing the data drift amount to specific features through an characteristic function, thereby providing the interpretability of model predictions, and thereby improving the accuracy of the deep quantification of data drift and the factor-level interpretation.
[0050] See also Figure 3 , Figure 3 It is a schematic flow chart of a data drift analysis method based on GBDT provided in the second embodiment of the present application. The data drift analysis method based on GBDT can be applied to the server, and is used to predict the standard path flow of the preset decision tree in the target data set and compare it with the current path flow, so as to timely discover the changes in data distribution, attribute the data drift amount to specific features through the indicative function, and clearly understand which features cause data drift, thereby providing the interpretability of model prediction, thereby improving the accuracy of the deep quantification of data drift and the factor-level interpretation.
[0051] based on Figure 1 The embodiment shown, this embodiment Figure 3 As shown, step S20 includes steps S11 and S12.
[0052] Step S11, judging whether data drift occurs in the target data set according to the current path flow and the standard path flow;
[0053] Step S12: When the difference between the current path flow and the standard path flow is greater than a preset threshold, determining that data drift occurs in the preset decision tree in the target data set.
[0054] In one embodiment, a GBDT model is trained using a historical data set, and the model includes multiple decision trees. For a preset decision tree, the path flow is defined as the proportion of data instances that follow the path to the total data. The training data set is used to calculate the standard path flow of the preset decision tree, and the pre-trained GBDT model is used to predict the target data set to obtain the predicted path for each data instance.
[0055] According to the predicted path in the target data set, the current path flow of the preset decision tree is calculated, the current path flow in the target data set is compared with the standard path flow in the training data set, and the difference value between the current path flow and the standard path flow is calculated.
[0056] A preset threshold is determined to determine whether the data drift is significant. If the difference between the current path flow and the standard path flow is greater than the preset threshold, it is determined that the preset decision tree has data drift in the target data set.
[0057] The present embodiment discloses a data drift analysis method, apparatus, device and storage medium based on GBDT. The data drift analysis method based on GBDT includes predicting the standard path flow of a preset decision tree in a target data set according to a pre-trained gradient boosting decision tree GBDT model; judging whether data drift occurs in the preset decision tree in the target data set according to the current path flow and the standard path flow; determining that data drift occurs in the preset decision tree in the target data set when the difference value between the current path flow and the standard path flow is greater than a preset threshold; obtaining the current path flow of the preset decision tree in the target data set, and calculating the total data drift amount between the current path flow and the standard path flow; attributing the total data drift amount to the target feature in the decision tree through a preset indicative function, and generating data drift analysis information. Through the above method, the present application predicts the standard path flow of the preset decision tree in the target data set and compares it with the current path flow, so as to timely discover the changes in data distribution. By attributing the data drift to specific features through the indicative function, it can clearly understand which features cause the data drift, provide the interpretability of the model prediction, and thus improve the accuracy of the deep quantification of data drift and the factor-level interpretation.
[0058] In a more preferred embodiment, step S20 includes:
[0059] Calculate the current node flow and standard node flow of each subnode in the preset decision tree;
[0060] Calculating the sub-node flow difference between the current node flow and the standard node flow at each of the sub-nodes respectively;
[0061] The flow differences of the sub-nodes are summed to generate a flow difference sum, and the flow difference sum is determined as the total data drift amount.
[0062] In one embodiment, for a preset decision tree, the node flow of each child node is defined as the proportion of data instances reaching the child node to the total data. The standard node flow of each child node in the preset decision tree is calculated using the training data set. The target data set is predicted using the pre-trained GBDT model to obtain the predicted path of each data instance, and the current node flow of each child node in the preset decision tree is calculated based on the predicted path in the target data set.
[0063] For each child node, the difference between the current node flow and the standard node flow is calculated, the flow differences of all child nodes are summed up to generate the sum of flow differences, and the sum of flow differences is determined as the total data drift.
[0064] The present embodiment discloses a data drift analysis method, device, equipment and storage medium based on GBDT. The data drift analysis method based on GBDT includes predicting the standard path flow of a preset decision tree in a target data set according to a pre-trained gradient boosting decision tree GBDT model; judging whether data drift occurs in the preset decision tree in the target data set according to the current path flow and the standard path flow; determining that data drift occurs in the preset decision tree in the target data set when the difference value between the current path flow and the standard path flow is greater than a preset threshold; obtaining the current path flow of the preset decision tree in the target data set, calculating the current node flow and the standard node flow of each subnode in the preset decision tree; respectively calculating the subnode flow difference between the current node flow and the standard node flow at each subnode; summing the subnode flow differences to generate a flow difference sum, and determining the flow difference sum as the total data drift amount; attributing the total data drift amount to the target feature in the decision tree through a preset indicative function to generate data drift analysis information. Through the above method, the present application can accurately detect data drift by comparing the current path traffic and the standard path traffic. By setting a threshold to judge data drift, it can take quick action when the data distribution changes significantly, reducing the risk of model performance degradation. The generated data drift analysis information provides strong support, thereby improving the accuracy of the deep quantification of data drift and the factor-level interpretation.
[0065] See also Figure 4 , Figure 4It is a schematic flow chart of a data drift analysis method based on GBDT provided by the third embodiment of the present application. The data drift analysis method based on GBDT can be applied to the server, and is used to more accurately calculate the path flow of each leaf node by accurately allocating data instances to each leaf node of the prediction decision tree, thereby improving the accuracy of data drift analysis. Calculating the path flow makes it possible to quantify the data distribution on each decision path, enhancing the understanding of how data flows in the decision tree. Accurate path flow calculation helps to identify deviations in model predictions, thereby improving the prediction accuracy of the model, thereby improving the accuracy of the deep quantification of data drift and the factor-level interpretation.
[0066] based on Figure 1 The embodiment shown, this embodiment Figure 4 As shown, step S10 includes step S101 to step S103.
[0067] Step S101, obtaining the total number of data instances in the target data set, and assigning each data instance to each leaf node in the prediction decision tree;
[0068] Step S102, calculating the number of the data instances in each of the leaf nodes through the pre-trained GBDT model;
[0069] Step S103: Based on the ratio between the number of the data instances in each leaf node and the total number of the data instances, the ratio is defined as the path flow.
[0070] In one embodiment, the total number of data instances in the target data set is calculated, and each data instance in the target data set is predicted using a pre-trained GBDT model, and each instance is assigned to a corresponding leaf node of a decision tree. For each leaf node in the decision tree, the number of data instances falling on the node is calculated, and for each leaf node, the path flow is calculated, that is, the ratio between the number of data instances in the leaf node and the total number of data instances, and the change in the path flow is analyzed to identify potential changes in data distribution or data drift.
[0071] The present embodiment discloses a data drift analysis method, apparatus, device and storage medium based on GBDT. The data drift analysis method based on GBDT includes obtaining the total number of data instances in the target data set, and assigning each data instance to each leaf node in the prediction decision tree; calculating the number of the data instances in each leaf node by using the pre-trained GBDT model; defining the ratio as the path flow according to the ratio between the number of the data instances in each leaf node and the total number of the data instances; obtaining the current path flow of the preset decision tree in the target data set, and calculating all data drift amounts between the current path flow and the standard path flow; attributing all data drift amounts to the target features in the decision tree by using a preset indicative function, and generating data drift analysis information. Through the above method, the present application can more accurately calculate the path flow of each leaf node by accurately allocating data instances to each leaf node of the prediction decision tree, thereby improving the accuracy of data drift analysis. Calculating the path flow makes it possible to quantify the data distribution on each decision path, enhancing the understanding of how data flows in the decision tree. Accurate path flow calculation helps to identify deviations in model predictions, thereby improving the prediction accuracy of the model, thereby improving the accuracy of the deep quantification of data drift and the factor-level interpretation.
[0072] based on Figure 1 In the embodiment shown, in this embodiment, before step S30, the following steps are included:
[0073] Identify at least one preset feature in the preset decision tree, and define a node diversion condition based on each of the preset features;
[0074] The preset indicative function is determined according to the node diversion conditions and the initial function.
[0075] Specifically, identify at least one key feature from the preset decision tree that has a significant impact on the model's prediction results. For each preset feature, define one or more conditions that will determine the diversion path of the data instance in the decision tree. For example, if the feature is "age", the condition can be "age>30". Create an initial function for each feature and condition. This function can be a simple logical expression that is used to evaluate whether the data instance meets a specific diversion condition. Based on the node diversion condition and the initial function, determine the indicative function. The indicative function is usually a function that returns 0 or 1, indicating whether the data instance meets a specific diversion condition.
[0076] In a more preferred embodiment, step S30 includes:
[0077] Calculating the single-node data drift amount corresponding to each preset feature in the total data drift amount;
[0078] Calculating the drift ratio of each of the single node data drift amounts to the total data drift amount;
[0079] The preset feature whose drift amount ratio is greater than the preset ratio is determined as the target feature, and the data drift analysis information is generated according to the target feature and the single-node data drift amount corresponding to the target feature.
[0080] Specifically, for each preset feature, the difference between its current node flow in the target data set and the standard node flow in the training data set is calculated, and this difference is the single-node data drift of the feature. For each preset feature, the ratio of its single-node data drift to the total data drift is calculated, and this ratio is the drift ratio of the feature. The preset feature whose drift ratio is greater than the preset ratio is determined as the target feature. The preset ratio can be set according to business needs and experience. Data drift analysis information is generated based on the target feature and the single-node data drift corresponding to the target feature. This may include information such as the name of each target feature, the single-node data drift, and the drift ratio.
[0081] Based on any of the above embodiments, in this embodiment, before step S10, the following steps are included:
[0082] Acquire a training data set, and perform data preprocessing on the training data set;
[0083] Dividing the preprocessed training data set into a feature set and a variable set;
[0084] Training the initial GBDT model by using the feature set and the variable set, and determining the target parameters of the initial GBDT model by using a preset random search technique and / or a grid search technique;
[0085] The pre-trained GBDT model is determined according to the target parameters and the initial GBDT model.
[0086] Specifically, determine the prediction target (variable set) of the model, that is, the label or response variable of the model. Select the features (feature set) for prediction, that is, the input variables of the model, use the feature set and variable set to train an initial GBDT model, and apply random search technology or grid search technology to optimize the model parameters. Define the parameter search space, including learning rate, number of trees, tree depth, minimum number of samples required for splitting, etc., evaluate the performance of different parameter combinations through cross-validation, and select the best parameter combination.
[0087] It should be noted that the non-Company software tools or components appearing in the embodiments of the present application are merely examples and do not represent actual use.
[0088] See also Figure 5 , Figure 5 The embodiment of the present application provides a schematic block diagram of a data drift analysis device based on GBDT, and the data drift analysis device based on GBDT is used to execute the aforementioned data drift analysis method based on GBDT. The data drift analysis device based on GBDT can be configured on a server.
[0089] like Figure 5 As shown, the data drift analysis device 400 based on GBDT includes:
[0090] A standard path traffic prediction module 410 is used to predict the standard path traffic of a preset decision tree in a target data set according to a pre-trained gradient boosting decision tree GBDT model;
[0091] The total data drift calculation module 420 is used to obtain the current path flow of the preset decision tree in the target data set, and calculate the total data drift between the current path flow and the standard path flow;
[0092] The data drift analysis information generating module 430 is used to attribute all the data drift amounts to the target features in the decision tree through a preset indicative function to generate data drift analysis information.
[0093] Furthermore, the data drift analysis device 400 based on GBDT further includes:
[0094] A data drift judgment module, used for judging whether data drift occurs in the preset decision tree in the target data set according to the current path flow and the standard path flow;
[0095] The data drift determination module is used to determine that data drift occurs in the target data set in the preset decision tree when the difference value between the current path flow and the standard path flow is greater than a preset threshold.
[0096] Furthermore, the total data drift calculation module 420 includes:
[0097] A node flow calculation unit, used to calculate the current node flow and standard node flow of each sub-node in the preset decision tree;
[0098] A subnode flow difference calculation unit, used for respectively calculating the subnode flow difference between the current node flow and the standard node flow at each subnode;
[0099] The total data drift amount determining unit is used to sum the flow differences of each of the sub-nodes to generate a flow difference sum, and determine the flow difference sum as the total data drift amount.
[0100] Furthermore, the standard path traffic prediction module 410 includes:
[0101] A data instance allocation unit, used to obtain the total number of data instances in the target data set and allocate each data instance to each leaf node in the prediction decision tree;
[0102] A leaf node instance quantity calculation unit, used to calculate the number of the data instances in each leaf node through the pre-trained GBDT model;
[0103] The path flow definition unit is used to define the ratio between the number of the data instances in each leaf node and the total number of the data instances as the path flow.
[0104] Furthermore, the data drift analysis device 400 based on GBDT further includes:
[0105] A diversion condition definition module, used to identify at least one preset feature in the preset decision tree, and define a node diversion condition based on each of the preset features;
[0106] The preset indicative function determination module is used to determine the preset indicative function according to each of the node diversion conditions and the initial function.
[0107] Furthermore, the data drift analysis information generating module 430 includes:
[0108] A single-node data drift amount calculation unit, used to calculate the single-node data drift amount corresponding to each preset feature in the total data drift amount;
[0109] A drift ratio calculation unit, used for calculating the drift ratio of each single node data drift to the total data drift;
[0110] The data drift analysis information generating unit is used to determine the preset feature whose drift amount ratio is greater than the preset ratio as the target feature, and generate the data drift analysis information according to the target feature and the single-node data drift amount corresponding to the target feature.
[0111] Furthermore, the data drift analysis device 400 based on GBDT further includes:
[0112] A preprocessing module, used to obtain a training data set and perform data preprocessing on the training data set;
[0113] A training data set division module, used for dividing the preprocessed training data set into a feature set and a variable set;
[0114] A target parameter determination module, used for training an initial GBDT model through the feature set and the variable set, and determining target parameters of the initial GBDT model through a preset random search technique and / or a grid search technique;
[0115] The pre-trained GBDT model determination module is used to determine the pre-trained GBDT model according to the target parameters and the initial GBDT model.
[0116] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0117] The above-mentioned device can be implemented in the form of a computer program. Figure 6 Runs on the computer device shown.
[0118] See also Figure 6 , Figure 6 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.
[0119] See also Figure 6 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0120] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any data drift analysis method based on GBDT.
[0121] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0122] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any data drift analysis method based on GBDT.
[0123] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0124] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0125] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0126] Predict the standard path flow of the preset decision tree in the target data set based on the pre-trained gradient boosting decision tree GBDT model;
[0127] Obtaining the current path flow of the preset decision tree in the target data set, and calculating all data drifts between the current path flow and the standard path flow;
[0128] The total data drift amount is attributed to the target feature in the decision tree through a preset indicative function to generate data drift analysis information.
[0129] In one embodiment, before obtaining the current path flow of the preset decision tree in the target data set and calculating all data drift amounts between the current path flow and the standard path flow, it is used to achieve:
[0130] Determining whether data drift occurs in the target data set according to the current path flow and the standard path flow;
[0131] When the difference between the current path flow and the standard path flow is greater than a preset threshold, it is determined that data drift occurs in the preset decision tree in the target data set.
[0132] In one embodiment, all data drifts between the current path flow and the standard path flow are calculated to achieve:
[0133] Calculate the current node flow and standard node flow of each subnode in the preset decision tree;
[0134] Calculating the sub-node flow difference between the current node flow and the standard node flow at each of the sub-nodes respectively;
[0135] The flow differences of the sub-nodes are summed to generate a flow difference sum, and the flow difference sum is determined as the total data drift amount.
[0136] In one embodiment, the standard path flow of a preset decision tree in a target data set is predicted based on a pre-trained gradient boosting decision tree GBDT model to achieve:
[0137] Obtaining the total number of data instances in the target data set, and assigning each data instance to each leaf node in the prediction decision tree;
[0138] Calculate the number of data instances in each leaf node through the pre-trained GBDT model;
[0139] According to the ratio between the number of the data instances in each of the leaf nodes and the total number of the data instances, the ratio is defined as the path flow.
[0140] In one embodiment, the total data drift amount is attributed to the target feature in the decision tree by a preset indicative function, before generating the data drift analysis information, for achieving:
[0141] Identify at least one preset feature in the preset decision tree, and define a node diversion condition based on each of the preset features;
[0142] The preset indicative function is determined according to the node diversion conditions and the initial function.
[0143] In one embodiment, all data drift amounts are attributed to target features in the decision tree through a preset indicative function to generate data drift analysis information for achieving:
[0144] Calculating the single-node data drift amount corresponding to each preset feature in the total data drift amount;
[0145] Calculating the drift ratio of each of the single node data drift amounts to the total data drift amount;
[0146] The preset feature whose drift amount ratio is greater than the preset ratio is determined as the target feature, and the data drift analysis information is generated according to the target feature and the single-node data drift amount corresponding to the target feature.
[0147] In one embodiment, a pre-trained gradient boosted decision tree (GBDT) model is used to predict the standard path flow in the target data set before the preset decision tree is used to achieve:
[0148] Acquire a training data set, and perform data preprocessing on the training data set;
[0149] Dividing the preprocessed training data set into a feature set and a variable set;
[0150] Training the initial GBDT model by using the feature set and the variable set, and determining the target parameters of the initial GBDT model by using a preset random search technique and / or a grid search technique;
[0151] The pre-trained GBDT model is determined according to the target parameters and the initial GBDT model.
[0152] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the GBDT-based data drift analysis methods provided in the embodiments of the present application.
[0153] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Medi a Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card), etc., equipped on the computer device.
[0154] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A data drift analysis method based on GBDT, characterized in that: include: Predict the standard path flow of the preset decision tree in the target data set based on the pre-trained gradient boosting decision tree GBDT model; Obtaining the current path flow of the preset decision tree in the target data set, and calculating all data drifts between the current path flow and the standard path flow; The total data drift amount is attributed to the target feature in the decision tree through a preset indicative function to generate data drift analysis information.
2. The data drift analysis method based on GBDT according to claim 1, characterized in that: The obtaining of the current path flow of the preset decision tree in the target data set and before calculating all data drift amounts between the current path flow and the standard path flow include: Determining whether data drift occurs in the target data set according to the current path flow and the standard path flow; When the difference between the current path flow and the standard path flow is greater than a preset threshold, it is determined that data drift occurs in the preset decision tree in the target data set.
3. The data drift analysis method based on GBDT according to claim 2 is characterized in that: The calculating of the total data drift between the current path flow and the standard path flow includes: Calculate the current node flow and standard node flow of each subnode in the preset decision tree; Calculating the sub-node flow difference between the current node flow and the standard node flow at each of the sub-nodes respectively; The flow differences of the sub-nodes are summed to generate a flow difference sum, and the flow difference sum is determined as the total data drift amount.
4. The data drift analysis method based on GBDT according to claim 1, characterized in that: The method of predicting the standard path flow of a preset decision tree in a target data set according to the pre-trained gradient boosting decision tree GBDT model includes: Obtaining the total number of data instances in the target data set, and assigning each data instance to each leaf node in the prediction decision tree; Calculate the number of data instances in each leaf node through the pre-trained GBDT model; According to the ratio between the number of the data instances in each of the leaf nodes and the total number of the data instances, the ratio is defined as the path flow.
5. The data drift analysis method based on GBDT according to claim 1, characterized in that: Before attributing all the data drift amounts to the target features in the decision tree by using a preset indicative function and generating data drift analysis information, the method includes: Identify at least one preset feature in the preset decision tree, and define a node diversion condition based on each of the preset features; The preset indicative function is determined according to the node diversion conditions and the initial function.
6. The data drift analysis method based on GBDT according to claim 5 is characterized in that: The method of attributing all data drift amounts to target features in the decision tree by using a preset indicative function to generate data drift analysis information includes: Calculating the single-node data drift amount corresponding to each preset feature in the total data drift amount; Calculating the drift ratio of each of the single node data drift amounts to the total data drift amount; The preset feature whose drift amount ratio is greater than the preset ratio is determined as the target feature, and the data drift analysis information is generated according to the target feature and the single-node data drift amount corresponding to the target feature.
7. The data drift analysis method based on GBDT according to any one of claims 1 to 6, characterized in that: The method of predicting the standard path flow of the preset decision tree in the target data set according to the pre-trained gradient boosting decision tree GBDT model includes: Acquire a training data set, and perform data preprocessing on the training data set; Dividing the preprocessed training data set into a feature set and a variable set; Training the initial GBDT model by using the feature set and the variable set, and determining the target parameters of the initial GBDT model by using a preset random search technique and / or a grid search technique; The pre-trained GBDT model is determined according to the target parameters and the initial GBDT model.
8. A data drift analysis device based on GBDT, characterized in that: include: The standard path traffic prediction module is used to predict the standard path traffic of the preset decision tree in the target data set based on the pre-trained gradient boosting decision tree GBDT model; A total data drift calculation module, used for obtaining the current path flow of the preset decision tree in the target data set, and calculating the total data drift between the current path flow and the standard path flow; The data drift analysis information generation module is used to attribute all the data drift amounts to the target features in the decision tree through a preset indicative function to generate data drift analysis information.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the GBDT-based data drift analysis method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the GBDT-based data drift analysis method according to any one of claims 1 to 7.