A GBDT classification method based on skewed regression trees

By optimizing the node splitting process of the skewed decision tree through pre-sorting and local approximation strategies, the problems of high computational time complexity and low classification accuracy are solved, achieving more efficient classification results and lower model complexity.

CN117436000BActive Publication Date: 2026-11-17XIDIAN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311443174.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-01
Publication Date
2026-11-17
Estimated Expiration
2043-11-01

AI Technical Summary

Technical Problem

In existing technologies, skewed decision trees require repeated sorting of all attribute sets when splitting nodes, which leads to a sharp increase in computational time complexity. Furthermore, single decision tree models have poor generalization ability and low classification accuracy.

Method used

The algorithm employs a pre-sorted tree generation algorithm and a local approximation strategy algorithm. By saving the splitting result of the previous node as the initial sample set of the next node, the algorithm reduces the number of sortings of the attribute queue. It also randomly selects sampling coefficients in the local region for perturbation and jumping, thereby reducing the amount of computation and improving the computational efficiency.

Benefits of technology

It reduces computation time, improves classification efficiency and accuracy, reduces model complexity, and enhances the model's generalization ability and classification speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117436000B_ABST
    Figure CN117436000B_ABST
Patent Text Reader

Abstract

This invention discloses a GBDT classification method based on skewed regression trees. The method includes an improved skewed decision tree regression tree generation algorithm. Firstly, this improved algorithm performs node splitting based on a pre-sorted tree generation algorithm, using the sorted sample results of the left and right subsets after the previous node split as the initial sample set for the next node, thus improving the speed of single-node splitting. Secondly, this improved algorithm uses a local approximation strategy to obtain extraction coefficients and perturbs and jumps each node of the axis-parallel tree, reducing the number of decision tree layers, lowering model complexity, and improving the actual classification speed while maintaining accuracy. Furthermore, by iteratively training the residuals fitted by the ensemble model of multiple regression trees, a multi-class ensemble model is obtained, and data classification is performed based on the prediction results of this multi-class ensemble model to improve classification accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of machine learning methods, specifically relating to a GBDT classification method based on skewed regression trees. Background Technology

[0002] In recent years, artificial intelligence (AI) technology has been increasingly widely applied across various fields. Machine learning is a crucial foundation of AI technology and a research hotspot in both academia and industry. Decision trees are a classic machine learning method and a base learner for boosting ensemble learning methods. Decision trees have been successfully applied in fields such as computer vision and information retrieval. When a single machine learning model is used for classification or regression tasks, its performance often falls short of requirements due to limitations in model complexity or training data. Ensemble learning, by combining existing machine learning models, can further improve performance. Gradient Boosting Decision Tree (GBDT) is an additive model based on the boosting ensemble learning concept. Its high prediction accuracy and strong interpretability have made it popular among researchers in various fields, attracting widespread attention and research.

[0003] In his 2020 paper "Ensembles of Oblique Decision Trees" published in *Computer Science*, Torsha Majumder disclosed a skewed decision tree classification method. This method first iterates through each attribute value and calculates the Gini coefficient to find the optimal axis-parallel split. Then, it perturbs each coefficient of the hyperplane until the optimal value is found. Finally, it uses randomized jumps to escape local minima to find the optimal split. Compared to traditional Classification and Regression Tree (CART) decision tree classification algorithms, this method is less affected by noise and achieves higher classification accuracy.

[0004] However, in existing technologies, the base learner for skewed decision trees requires repeated sorting of all attribute sets when splitting nodes. Furthermore, each perturbation necessitates a new round of traversal and Gini value calculation. During this traversal, a greedy strategy is used to select the hyperplane coefficients, repeatedly calculating Gini values ​​under different splitting conditions. While this is the best approach to finding the optimal split, the computation time increases dramatically with the number of samples, increasing computational complexity and leading to increased training time costs. In addition, the single decision tree model in the algorithm exhibits poor generalization ability, resulting in low classification accuracy. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, the present invention is achieved through the following technical solution:

[0006] This invention provides a GBDT classification method based on skewed regression trees, the method comprising:

[0007] Step 1: Complete the one-hot encoding of the signal sample labels, generate a K×n dimensional label set matrix, initialize the iteration number o1, let o1=1, where K is an integer and K≥1, K represents the label category of the signal sample, n is an integer and n≥1, n represents the number of samples corresponding to each label category, o1≤o, o≥1, o represents the preset total number of iterations;

[0008] Step 2: Initialize the sample data of the k-th label category in the K×n-dimensional label set matrix to obtain the initialization model of the k-th label category, where k = 1, 2, ..., K;

[0009] Step 3: Train the initialization model of the k-th label category using the improved skewed decision tree regression tree generation algorithm to obtain the skewed regression tree of the k-th category. Calculate the predicted probability of classifying the input sample into the k-th category based on the skewed regression tree of the k-th category. The improved skewed decision tree regression tree generation algorithm includes a pre-sorted tree generation algorithm, which uses the sorted sample results of the left and right subsets after the previous node split as the initial sample set for the next node.

[0010] Step 4: Calculate the negative gradient and leaf node fit values ​​of the skew regression tree for the k-th category;

[0011] Step 5: Obtain the first fitted regression tree in the o1th iteration based on the leaf node fitting values;

[0012] Step 6: Repeat steps 2 to 5 to train the remaining K-1 trees to obtain an ensemble model composed of K fitted regression trees. Based on the output of each fitted regression tree, obtain the new predicted probability, and recalculate the new negative gradient to form the fitting residual.

[0013] Step 7: Let o1 = o1 + 1. When o1 ≤ o, repeat steps 2 to 7 based on the residuals of the ensemble model fitting in step 6 until all o iterations are completed to obtain the multi-class ensemble model.

[0014] Step 8: Classify the data based on the prediction results of the multi-class ensemble model.

[0015] In one embodiment of the present invention, the initialization model of the k-th category is represented as: f k,0 f(x), where x represents the feature vector of the input sample, f k,0(x) represents the tree model of the k-th category in the 0th iteration, initialized with the prior probability corresponding to the category, where the prior probability is expressed as: D k This represents the number of sample data for the k-th label category, and D represents the total sample set.

[0016] In one embodiment of the present invention, the improved skewed decision tree regression tree generation algorithm further includes:

[0017] Step S31: Create nodes using the current total sample set;

[0018] Step S32: Based on the pre-sorted tree generation algorithm, obtain the optimal splitting attribute of each node according to the principle of minimizing the mean square error. The optimal splitting value x ok For each node, the optimal splitting value x is determined by its optimal splitting attribute. ok Perform splitting to obtain the axis-parallel tree of the k-th category;

[0019] Step S33: Perturb and jump each node of the axis parallel tree of the k-th category. The perturbation and jump process includes:

[0020] Extraction coefficients are obtained based on a local approximation strategy algorithm, and the optimal splitting attribute of the current node is determined according to these extraction coefficients. The hyperplane H is perturbed to update the optimal splitting attribute of the current node. a m and b, where a m Let represent the slope of hyperplane H, and b represent the perturbation of hyperplane H, where hyperplane H is the optimal splitting attribute of the current node. hyperplane;

[0021] Substituting the attribute values ​​from the extraction coefficients into the hyperplane H for perturbation, the optimal splitting attribute is obtained when the purity of the extraction coefficients is optimal. The slope of the hyperplane H and the hyperplane perturbation will affect a. m The optimal value is identified as When the purity after this disturbance is the same as the previous one When the corresponding purities are equal, the stagnation coefficient of the hyperplane H determines whether to update the hyperplane slope of the hyperplane H.

[0022] The hyperplane H is along a random vector R = (r1,...,r) m ,r m+1 Directional perturbation, the magnitude of the perturbation is α, r m Represents the optimal splitting attribute The corresponding directional perturbation involves substituting all sample attribute values ​​from the extracted coefficients into the hyperplane H perturbation process to obtain all constraints on α, thereby obtaining a new hyperplane by jumping different distances along the R direction. The purity after partitioning according to the new hyperplane is then calculated to obtain the optimal perturbation amount α. ok If the purity is improved by random jumps in this iteration compared to the previous iteration, then the optimal perturbation α is used. ok Update the parameters of the hyperplane H;

[0023] Step S34: Repeat step S33 until the preset number of iterations is reached to obtain the skewed regression tree of the k-th category;

[0024] Step S35: Based on the skewed regression tree of the kth category, calculate the predicted probability of classifying the input sample into the kth category.

[0025] In one embodiment of the present invention, the pre-sorted tree generation algorithm includes:

[0026] Step S321: Transfer the attribute x of the current node sample data m All attribute values ​​are sorted, and a one-to-one mapping between index number and corresponding sample is established to directly index the label sample of the current node by index number, where x m Let m represent the m-th attribute, where m = 1, 2, ..., M, and M represents the total number of attributes of the current node.

[0027] Step S322, according to attribute x m The attribute value divides the label samples of the current node into left and right subsets one by one, and calculates the attribute value according to the attribute x. m The mean squared error of the attribute value is used to divide the left and right subsets, and the attribute x is found when the mean squared error is minimized. m The optimal attribute splitting value;

[0028] Step S323: Determine whether the current node has stopped splitting;

[0029] Specifically, the current node is evaluated, and if the current node exhibits at least one of the preset constraints, the current node stops splitting. The preset constraints include the following three conditions:

[0030] In the first case, the samples in the current node have the same label category;

[0031] The second scenario is that the sample attribute set of the current node is empty, or all attributes of the sample have the same value.

[0032] In the third case, a subset after the current node is partitioned is an empty set;

[0033] Step S324: Transfer attribute x mReplace the M attributes with other attributes from the list, and repeat steps S322 to S323 until the optimal attribute splitting value has been calculated for all M attributes. Then, select the optimal splitting attribute from all M attributes of the current node based on the principle of minimizing the mean square error. and the optimal splitting attribute The optimal splitting value x ok ;

[0034] Step S325: Distribute the label sample data of the current node according to the optimal splitting attribute. The optimal splitting value x ok Perform a split, and label the resulting left and right subsets as the first identifier and the second identifier, respectively.

[0035] Step S326: Use the sorting results of the samples in the left and right subsets corresponding to the first and second identifiers as the initial sample set for the next node, and repeat steps S321 to S325 until the training of a single decision tree is completed, and obtain the axis parallel tree of the kth category.

[0036] In one embodiment of the present invention, both the first identifier and the second identifier are of byte type. The child node performs a Boolean judgment on the first identifier and the second identifier passed by its parent node in order to use the overall sorting result before splitting passed by its parent node.

[0037] In one embodiment of the present invention, the local approximation strategy algorithm includes: taking the optimal splitting attribute of the current node as... The preset candidate attribute values ​​are divided into multiple local regions according to preset intervals. The candidate values ​​are randomly selected from each local region and stored in an array to obtain the extraction coefficient.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. The GBDT classification method provided by this invention is based on a pre-sorted tree generation algorithm. It saves the total sorting result before the previous node splits and passes it to the next node for direct use in a specified way. This speeds up the time spent on single node splitting, avoids multiple sorting of attribute queue subsets, reduces computation time, and improves computational efficiency and actual classification efficiency. It solves the problem of increased training time cost caused by the sharp increase in computation time and computational complexity as the number of samples increases in the prior art.

[0040] 2. The GBDT classification method provided by this invention uses a local approximation strategy algorithm to obtain sampling coefficients to perturb and jump nodes. This local approximation strategy algorithm randomly extracts a sampling value within a specific region. When selecting sampling coefficients, instead of using an exhaustive method to traverse every possible result, it divides the local region by a set threshold and randomly extracts within the local region. This significantly reduces the computational load of the algorithm while ensuring accuracy, making it more practically valuable. It also reduces the number of tree branches and greatly improves the speed of node perturbation and jumping while ensuring accuracy.

[0041] 3. This invention uses an improved skewed decision tree as the base learner of the GBDT ensemble model, which makes the classification effect more ideal when the samples are polygonally segmented in the attribute space. At the same time, it reduces the number of decision tree layers, reduces model complexity, improves the model's generalization ability, and thus improves the classification effect and the actual classification speed. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the iterative process of the GBDT classification method based on skewed regression tree provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the process for obtaining the optimal hyperplane based on axis parallel segmentation in the improved tilted decision tree regression tree generation algorithm provided in this embodiment of the invention.

[0044] Figure 3 This is a flowchart illustrating the pre-sorted tree generation algorithm provided in an embodiment of the present invention.

[0045] Figure 4 This is a schematic diagram of obtaining extraction coefficients based on a local approximation strategy algorithm provided in an embodiment of the present invention. Detailed Implementation

[0046] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the iterative process of the GBDT classification method based on skewed regression trees provided in an embodiment of the present invention. The GBDT classification method includes the following steps:

[0047] Step 1: Complete the one-hot encoding of the signal sample labels, generate a K×n dimensional label set matrix, initialize the iteration number o1, let o1=1, where K is an integer and K≥1, K represents the label category of the signal sample, n is an integer and n≥1, n represents the number of samples corresponding to each label category, o1≤o, o≥1, o represents the preset total number of iterations;

[0048] Specifically, a label set is reconstructed for each category of input data. A K×n-dimensional label set matrix is ​​established based on the input label information and the number of samples. Each dimension of the label set matrix corresponds to only one category. The label of the corresponding category is encoded as 1, and other label categories are encoded as 0, thus realizing one-hot encoding of signal sample labels.

[0049] Step 2: Initialize the sample data of the k-th label category in the K×n-dimensional label set matrix to obtain the initialization model of the k-th label category, where k = 1, 2, ..., K;

[0050] Specifically, the initialization model for the k-th category is represented as: f k,0 f(x), where x represents the feature vector of the input sample, f k,0 (x) represents the tree model of the k-th category in the 0th iteration, initialized with the prior probability corresponding to the category, where the prior probability is expressed as: D k This represents the number of sample data for the k-th label category, and D represents the total sample set.

[0051] Step 3: Train the initial model of the k-th label category using the improved skewed decision tree regression tree generation algorithm to obtain the regression tree of the k-th category. Calculate the predicted probability of classifying the input sample into the k-th category based on the regression tree of the k-th category.

[0052] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the process of obtaining the optimal hyperplane based on axis-parallel partitioning in the improved tilted decision tree regression tree generation algorithm provided in this embodiment of the invention. The improved tilted decision tree regression tree generation algorithm includes: First, a pre-sorting tree generation algorithm uses the sorted results of the samples after the left and right subsets of the previous node's split as the initial sample set for the next node, to avoid repeated sorting of the initial sample set for each node, and generates an axis-parallel tree for the k-th category; then, a local approximation strategy algorithm divides the candidate attribute values ​​according to a preset interval, where the candidate attribute value is the attribute value of the optimal splitting attribute in the samples of each node; next, randomly extracting the candidate attribute values ​​within the divided interval to generate extraction coefficients, and performing hyperplane parameter perturbation and randomized jumps on each node in the axis-parallel tree of the k-th category according to these extraction coefficients to obtain the optimal hyperplane, generating the tilted regression tree for the k-th category. Specific steps are detailed in steps S31 to S35 below.

[0053] Furthermore, the sample set of the k-th label category is axially parallelized to generate an axially parallel tree for the k-th category, specifically including the following steps S31 to S32:

[0054] Step S31: Create nodes using the current total sample set;

[0055] Specifically, the root node of the tree is created using the initialization model of the sample set of the k-th label category.

[0056] Step S32: Based on the pre-sorted tree generation algorithm, obtain the optimal splitting attribute of each node according to the principle of minimizing the mean square error (MSE) value. The optimal splitting value x ok For each node, according to its optimal splitting attribute The optimal splitting value x ok Perform a split to obtain an axis-parallel tree for the k-th category, where x m Let m represent the m-th attribute, where m = 1, 2, ..., M, and M represent the total number of attributes in the current node. The optimal splitting attribute is... This indicates the optimal splitting attribute among all M attributes calculated according to the principle of minimizing the mean squared error value in the current node.

[0057] Further, please refer to Figure 3 , Figure 3 This is a flowchart illustrating a pre-sorted tree generation algorithm provided in an embodiment of the present invention. The pre-sorted tree generation algorithm includes:

[0058] Step S321: Add a sample index, and add the attribute x of the current node sample data. m All attribute values ​​are sorted, and the current node label sample is directly indexed by index number through a one-to-one mapping relationship between index number and corresponding sample.

[0059] Step S322, according to attribute x m The attribute value divides the label samples of the current node into left and right subsets one by one, and calculates the attribute value according to the attribute x. m The mean squared error of the attribute value is used to divide the left and right subsets, and the attribute x is found when the mean squared error is minimized. m The optimal attribute splitting value;

[0060] Specifically, sort the values ​​of each attribute in the current node's sample set, using attribute x as the sorting order. m For example, regarding the attribute x of the sample set m Divide the sample set into left and right subsets based on the median value, and calculate the minimum mean square error of the left and right subsets, including: the minimum mean square error of the attribute x of the sample set. m The attribute value queue is sorted in ascending order to obtain the sequence {x}. i}, i = 1, 2, ..., n, then take the sequence {x i The midpoint between two adjacent values, where, Unless otherwise specified, the value of i is i = 1, 2, ..., n), x i ′(i=2,3,...,n) represents the sequence {x i The midpoint of two adjacent values ​​(i = 1, 2, ..., n) is then used as the midpoint x. i 'Transfer attribute x' m The sequence {x i The attribute value x is divided into two subsets. i The corresponding i-th sample Calculate the minimum mean square error of the left and right subsets within the same node, and assign weights |D to the left and right branch nodes respectively. L | / |D|、|D R | / |D| indicates the proportion, where D L Indicates the intermediate value x i 'Transfer the sequence {x} i The number of samples in the left subset of the partitioned dataset, D R This indicates that the intermediate value x i 'Transfer the sequence {x} i The formula for calculating the minimum mean square error of the left and right subsets within the same node is as follows: (The number of samples in the right subset of the partitioned node is given by the formula.)

[0061]

[0062] in, m represents the mean of the sample set. L m represents the number of samples contained in the left subset of the same node. R Let x represent the number of samples contained in the right subset of the same node, D represent the total number of samples, and x represent the number of samples in the right subset of the same node. i Represents attribute x m The i-th attribute value, y i Represents attribute x m The predicted value of the i-th sample, This represents the sample mean of the left subset within the same node. This represents the sample mean of the right subset within the same node.

[0063] Step S323: Determine whether the current node has stopped splitting;

[0064] Specifically, the current node is evaluated, and if the current node exhibits at least one of the preset constraints, the current node stops splitting. The preset constraints include the following three conditions:

[0065] In the first case, the samples in the current node have the same label category;

[0066] The second scenario is that the sample attribute set of the current node is empty, or all attributes of the sample have the same value.

[0067] In the third case, a subset after the current node is partitioned is an empty set.

[0068] Step S324: Transfer attribute x m Replace the M attributes with other attributes from the list, and repeat steps S322 to S323 until the optimal attribute splitting value has been calculated for all M attributes. Then, select the optimal splitting attribute from all M attributes of the current node based on the principle of minimizing the mean square error. and optimal splitting properties The optimal splitting value x ok ;

[0069] Step S325: Distribute the label sample data of the current node according to the optimal splitting attribute. The optimal splitting value x ok Perform a split, and label the resulting left and right subsets as the first identifier and the second identifier, respectively.

[0070] Furthermore, the first identifier and the second identifier can be Boolean indexes, strings, or numbers; the present invention does not limit this.

[0071] Step S326: Use the sorting results of the samples in the left and right subsets corresponding to the first and second identifiers as the initial sample set for the next node. Repeat steps S321 to S325 until the training of a single decision tree is completed, and obtain the axial parallel tree of the kth category, ensuring that the node splitting rate is better than the axial parallel segmentation method in the prior art.

[0072] In this embodiment, both the first and second identifiers are byte types. Specifically, the first identifier is 1, indicating that the sample set belonging to the left subtree is identified as 1, and the second identifier is 0, indicating that the sample sequence number belonging to the right subtree is identified as 0. This embodiment sets the first and second identifiers to byte types. Child nodes perform Boolean judgments on the first and second identifiers passed to their parent nodes to use the overall sorting result before splitting passed by the parent node. This saves storage space and improves computational efficiency. The left and right subsets corresponding to the first and second identifiers are iteratively updated for each node until the training of a single decision tree is complete. In other embodiments, the first and second identifiers are of other data types. This embodiment avoids searching for the optimal splitting attribute for different nodes by setting the first and second identifiers and updating the sample data of the left and right subsets corresponding to the first and second identifiers in real time. and the optimal splitting value x okMultiple sorting of the time attribute queue subset improves the splitting efficiency of the algorithm and reduces its time complexity. The pre-sorted tree generation algorithm provided in this embodiment improves the splitting efficiency of the algorithm by optimizing the splitting attributes of the previous node. and the optimal splitting value x ok The left and right subsets after the split are identified and passed to the next node as sample data, that is, the parent node is processed through its optimal splitting attribute. The optimal splitting value x ok The left and right subsets after splitting are identified and passed to the child nodes of the parent node as sample data, which reduces the computation time and improves the computation efficiency, making the node splitting effect better than the axis parallel partitioning method.

[0073] The GBDT classification method provided in this embodiment is based on a pre-sorted tree generation algorithm. It saves the total sorting result before the previous node splits and passes it to the next node for direct use in a specified way. This speeds up the time spent splitting a single node, avoids multiple sorting of the attribute queue subset, reduces computation time, and improves computational efficiency and actual classification efficiency. It solves the problem of increased training time cost caused by the sharp increase in computation time and computational complexity as the number of samples increases in the prior art.

[0074] Furthermore, the parameters of the perturbation hyperplane, such as Figure 3 As shown, see step S33 for details.

[0075] Step S33: Perturb and jump each node of the axis parallel tree of the k-th label category. The perturbation and jump process specifically includes steps S331, S332 and S333.

[0076] Furthermore, step S331 specifically includes the following steps:

[0077] Step S331: Obtain the extraction coefficients based on the local approximation strategy algorithm, and determine the optimal splitting attribute of the current node based on the extraction coefficients. The hyperplane H is perturbed to update the optimal splitting attribute of the current node. a m and b, where a m Let 'b' represent the slope of hyperplane H, 'b' represent the perturbation of hyperplane H, and hyperplane H represent the optimal splitting attribute of the current node. The specific steps for constructing the hyperplane are as follows.

[0078] Step S3311: After axis-parallel splitting, when the optimal splitting properties are... Obtain the optimal split value x for the current node. ok When b = -x ok The segmentation formula a is obtained. m x m+b≤0,a m =1, and all other attribute coefficients are 0, forming a coefficient vector [a1,a2,...,a...]. m ,...,a M The initial hyperplane H of the current node is denoted as:

[0079]

[0080] Step S3312: Analyze the attribute x of the j-th (j = 1, 2, 3, ..., n) sample in all k-th labels. m attribute value x j′ Substituting into the hyperplane H, we obtain the value V j′ The sign indicates the position of the sample relative to the hyperplane H;

[0081] Step S3313: Obtain the extraction coefficients using a local approximation strategy algorithm;

[0082] Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the extraction coefficients based on a local approximation strategy algorithm provided in an embodiment of the present invention. The local approximation strategy algorithm specifically includes: [The following is a separate, unrelated section:] ...the optimal splitting attribute of the current node... The preset candidate attribute values ​​are divided into multiple local regions according to a preset interval. In this embodiment, the interval is d in the figure. A candidate value is randomly selected from each local region and stored sequentially in an array to obtain the extraction coefficient. This extraction coefficient is identified as extraction coefficient a. s This reduces the splitting time of nodes and improves the efficiency of tree model generation.

[0083] Step S332: Extract the coefficient a s Substituting the attribute values ​​into the hyperplane H for perturbation, the optimal splitting attribute is obtained when the purity is optimal in the extracted coefficient set. The slope of the hyperplane H and the hyperplane perturbation amount are used to obtain a. m The optimal value of b, and a m The optimal value is identified as When the purity after this disturbance is the same as the previous one When the corresponding purities are equal, P stag Determine whether to update the hyperplane slope of hyperplane H, P stag P is the stagnation coefficient of the hyperplane H, which determines whether to update the slope of the hyperplane H. Specifically, P stag The initial value of P is 1, and as the preset number of perturbations m' increases, P... stag Gradually decrease, parameter a m The update is represented as:

[0084]

[0085] After determining the parameters of the perturbation hyperplane, perform randomized jumps. Please refer to [link / reference needed]. Figure 3 Furthermore, the randomized jumping process specifically includes step S333.

[0086] Step S333: Move the hyperplane H along the random vector R = (r1,...,r) m ,r m+1 Directional perturbation, the magnitude of the perturbation is α, r m Represents the optimal splitting attribute The corresponding directional perturbation is set as α, where the optimal perturbation amount for the corresponding sample in the extraction coefficient is α. ok′ Substituting all sample attribute values ​​from the extraction coefficients into the hyperplane H perturbation process yields all constraints for α, resulting in new hyperplanes obtained by jumping different distances along the R direction. The purity after partitioning according to the new hyperplane is then calculated to obtain the optimal perturbation amount α. ok If the purity is improved by random jumps in this iteration compared to the previous iteration, then the optimal perturbation α is used. ok Update the parameters of the hyperplane H.

[0087] Furthermore, this perturbation process can be represented as:

[0088]

[0089] Where H1 represents the hyperplane after perturbation, and H represents the hyperplane before perturbation, let... r m Represents the optimal splitting attribute The corresponding directional perturbation, R q This represents the value obtained after substituting the attribute value of the q-th sample. Let α represent the m-th attribute value of the q-th sample, and let α represent the sample perturbation. Treating parameter α as a unique variable, each sample corresponds to a different α, and the optimal perturbation is set to α. ok Substituting all sample attribute values ​​into the constraints for α, sorting them and taking the median value, yields a new hyperplane obtained by jumping different distances along the R direction. The purity after partitioning by the new hyperplane is then calculated, and the sample purity is compared to obtain the optimal perturbation α. ok If this random jump increases purity, then according to the optimal perturbation α ok Update the parameters of the hyperplane H.

[0090] The GBDT classification method provided in this embodiment uses a local approximation strategy algorithm to obtain sampling coefficients for node perturbation and jumping. This local approximation strategy algorithm randomly extracts a sampling value within a specific region. When selecting sampling coefficients, instead of using an exhaustive method to traverse every possible result, it divides the local region by a set threshold and randomly extracts within the local region. This significantly reduces the computational load of the algorithm while ensuring accuracy, making it more practically valuable. It also reduces the number of tree branches, allowing the algorithm to greatly improve the speed of node perturbation and jumping while maintaining accuracy.

[0091] Step S34: Repeat step S33 until the preset number of iterations is reached to obtain the skewed regression tree of the k-th category;

[0092] Specifically, within a preset number of iterations, when the purity of the partitioning result in step S33 is higher than the purity calculated from the parameters in the previous iteration, a multi-local search is performed again, repeating the parameter perturbation process of step S33, and the best hyperplane parameter a is updated for the hyperplane H. m and b until the preset number of perturbations is reached.

[0093] Step S35: Calculate the predicted probability of classifying the input sample into the kth category based on the skewed regression tree of the kth category;

[0094] Specifically, the regression tree predicts the probability of classifying the input sample into the k-th class as follows:

[0095]

[0096] Where f1(x) represents the first tree in the same training round, h represents the index of the skew regression tree for the k-th class, h = 1, 2, ..., K, and x represents the feature vector of the input sample. Let P represent the probability of the k-th category, and Y represent the probability of each predicted outcome. f represents the selected label category. h (x) represents the h-th tree in the same round of training.

[0097] Step 4: Calculate the negative gradient and leaf node fit value of the skew regression tree for the k-th category;

[0098] Furthermore, through Obtain the negative gradient r h , where y h This represents the predicted probability of the k-th class in the h-th tree. Let represent the true probability of the k-th class in the h-th tree. The regression tree structure is obtained by fitting the pseudo-residuals. To satisfy the loss minimization principle, the fitted value of the j-th leaf node is obtained using the Newton-Raphson method as follows:

[0099]

[0100] Among them, c j D represents the fitted value of the j-th leaf node. j Let r represent the set of leaf samples. h Let x represent the negative gradient of the h-th tree. i This represents the attribute value of the input sample.

[0101] Step 5: Obtain the first fitted regression tree in the o1th iteration based on the leaf node fitting values;

[0102] The first fitted regression tree is represented as:

[0103]

[0104] Among them, f 1,0 (x) represents the initial prediction function of the first fitted regression tree, J1 represents the number of leaves in this fitted regression tree, and c 1,j D represents the fitted value of the j-th leaf of the first fitted regression tree. 1,j This represents the sample set of the j-th leaf, where j represents the leaf index.

[0105] Step 6: Repeat steps 2 to 5 to train the remaining K-1 fitted regression trees to obtain an ensemble model composed of K fitted regression trees. Based on the output of each fitted regression tree, obtain the new predicted probability, and recalculate the new negative gradient to form the fitting residual.

[0106] Specifically, each dimension of the one-hot encoded label is taken sequentially, and the above steps are repeated to train the remaining K-1 fitted regression trees. After this round of training is completed, the ensemble model composed of K fitted regression trees is obtained. At the same time, new predicted probabilities are obtained based on the output of each fitted regression tree, and new negative gradients are calculated again to form the residuals to be fitted in the next round of training.

[0107] Step 7: Let o1 = o1 + 1. When o1 ≤ o, repeat steps 2 to 7 based on the residuals of the ensemble model fitting in step 6 until all o iterations are completed to obtain the multi-class ensemble model.

[0108] Step 8: Classify the data based on the prediction results of the multi-class ensemble model.

[0109] Specifically, after o rounds of training, a total of o×K decision trees are generated, each containing K binary classification ensemble models, and each having an output result. These K output results are then substituted into...

[0110]

[0111] The probability of each category is calculated, and the category with the highest probability after dividing by feature value is taken as the final classification result. In this embodiment, the preset total number of iterations is 3, i.e., o = 3. The weighted sum of the skew regression tree models obtained from each round of learning regression is used to obtain the final model. In other embodiments, the preset total number of iterations can be 4, 5, etc., and this invention does not limit it.

[0112] When two categories can be distinguished by a single feature value, axial parallel segmentation works well. However, in actual classification, the objects being classified are often affected by noise, causing the segmentation boundary to be slanted. Using the traditional CART decision tree as the base learner of the GBDT ensemble model results in unsatisfactory classification performance. This embodiment uses an improved slanted decision tree as the base learner of the GBDT ensemble model, which makes the classification performance more ideal when the samples are polygonally segmented in the attribute space. At the same time, it reduces the number of decision tree layers, lowers model complexity, improves the model's generalization ability, and thus improves the classification performance and increases the actual classification speed.

Claims

1. A GBDT classification method based on tilted regression trees, characterized in that, include: Step 1: Complete the one-hot encoding of the signal sample labels, generate a K×n dimensional label set matrix, initialize the iteration number o1, let o1=1, where K is an integer and K≥1, K represents the label category of the signal sample, n is an integer and n≥1, n represents the number of samples corresponding to each label category, o1≤o, o≥1, o represents the preset total number of iterations; Step 2: Initialize the sample data of the k-th label category in the K×n dimensional label set matrix to obtain the initial model of the k-th label category, where k = 1, 2, ..., K; Step 3: Train the initialization model of the k-th label category using the improved skewed decision tree regression tree generation algorithm to obtain the skewed regression tree of the k-th category. Calculate the predicted probability of classifying the input sample into the k-th category based on the skewed regression tree of the k-th category. The improved skewed decision tree regression tree generation algorithm includes a pre-sorted tree generation algorithm, which uses the sorted sample results of the left and right subsets after the previous node split as the initial sample set for the next node. Step 4: Calculate the negative gradient and leaf node fit value of the skew regression tree for the k-th category; Step 5: Obtain the first fitted regression tree in the o1th iteration based on the fitted values ​​of the leaf nodes; Step 6: Repeat steps 2 to 5 to train the remaining K-1 trees to obtain an ensemble model composed of K fitted regression trees. Based on the output of each fitted regression tree, obtain the new predicted probability, and recalculate the new negative gradient to form the fitting residual. Step 7: Let o1 = o1 + 1. When o1 ≤ o, repeat steps 2 to 7 based on the residuals of the ensemble model fitting in step 6 until all o iterations are completed to obtain the multi-class ensemble model. Step 8: Classify the data based on the prediction results of the multi-class ensemble model.

2. The GBDT classification method based on skewed regression trees according to claim 1, characterized in that, The initialized model of the kth category is represented as: f k,0 (x), x represents a feature vector of an input sample, f k,0 (x) represents a tree model of the kth category in the 0th iteration, and the initialized parameter is a prior probability corresponding to the category, which is represented as D k represents the number of sample data of the kth label category, and D represents a total set of samples. 3.The GBDT classification method based on the tilted regression tree according to claim 2, characterized in that, The improved skewed decision tree regression tree generation algorithm further includes: Step S31: Create nodes using the current total sample set; Step S32, based on the pre-ordered tree generation algorithm, the optimal split attribute of each node is obtained according to the principle of minimum mean square error value The optimal split value x of the optimal split attribute of each node ok Split according to the optimal split value x of the optimal split attribute of each node ok To obtain the axis-parallel tree of the kth class Step S33: Perturb and jump each node of the axis parallel tree of the k-th category. The perturbation and jump process includes: Extraction coefficients are obtained based on a local approximation strategy algorithm, and the optimal splitting attribute of the current node is determined according to these extraction coefficients. The hyperplane H is perturbed to update the optimal splitting attribute of the current node. a m and b, where a m Let represent the slope of hyperplane H, and b represent the perturbation of hyperplane H, where hyperplane H is the optimal splitting attribute of the current node. hyperplane; Substituting the attribute values ​​from the extraction coefficients into the hyperplane H for perturbation, the optimal splitting attribute is obtained when the purity of the extraction coefficients is optimal. The slope of the hyperplane H and the hyperplane perturbation will affect a. m The optimal value is identified as When the purity after this disturbance is the same as the previous one When the corresponding purities are equal, the stagnation coefficient of the hyperplane H determines whether to update the hyperplane slope of the hyperplane H. The hyperplane H is along a random vector R = (r1,...,r) m ,r m+1 Directional perturbation, the magnitude of the perturbation is α, r m Represents the optimal splitting attribute The corresponding directional perturbation involves substituting all sample attribute values ​​from the extracted coefficients into the hyperplane H perturbation process to obtain all constraints on α, thereby obtaining a new hyperplane by jumping different distances along the R direction. The purity after partitioning according to the new hyperplane is then calculated to obtain the optimal perturbation amount α. ok If the purity is improved by random jumps in this iteration compared to the previous iteration, then the optimal perturbation α is used. ok Update the parameters of the hyperplane H; Step S34: Repeat step S33 until the preset number of iterations is reached to obtain the skewed regression tree of the k-th category; Step S35: Based on the skewed regression tree of the kth category, calculate the predicted probability of classifying the input sample into the kth category.

4. The GBDT classification method based on skewed regression trees according to claim 3, characterized in that, The pre-sorted tree generation algorithm includes: Step S321: Transfer the attribute x of the current node sample data m All attribute values ​​are sorted, and a one-to-one mapping between index number and corresponding sample is established to directly index the label sample of the current node by index number, where x m Let m represent the m-th attribute, where m = 1, 2, ..., M, and M represents the total number of attributes of the current node. Step S322, divide the label samples of the current node into left and right subsets according to the attribute value of attribute x m , calculate the mean square error value of dividing the left and right subsets according to the attribute value of attribute x m , and find the optimal attribute split value of attribute x m when the mean square error value is the smallest. Step S323: Determine whether the current node has stopped splitting; Specifically, the current node is evaluated, and if the current node exhibits at least one of the preset constraints, the current node stops splitting. The preset constraints include the following three conditions: In the first case, the samples in the current node have the same label category; The second scenario is that the sample attribute set of the current node is empty, or all attributes of the sample have the same value. In the third case, a subset after the current node is partitioned is an empty set; Step S324: Transfer attribute x m Replace the M attributes with other attributes from the list, and repeat steps S322 to S323 until the optimal attribute splitting value has been calculated for all M attributes. Then, select the optimal splitting attribute from all M attributes of the current node based on the principle of minimizing the mean square error. and the optimal splitting attribute The optimal splitting value x ok ; Step S325: Distribute the label sample data of the current node according to the optimal splitting attribute. The optimal splitting value x ok Perform a split, and label the resulting left and right subsets as the first identifier and the second identifier, respectively. Step S326: Use the sorting results of the samples in the left and right subsets corresponding to the first and second identifiers as the initial sample set for the next node, and repeat steps S321 to S325 until the training of a single decision tree is completed, and obtain the axis parallel tree of the kth category.

5. The GBDT classification method based on skewed regression trees according to claim 4, characterized in that, Both the first identifier and the second identifier are of byte type. The child node performs a Boolean judgment on the first identifier and the second identifier passed by its parent node in order to use the overall sorting result before splitting passed by its parent node.

6. The GBDT classification method based on skewed regression trees according to claim 5, characterized in that, The local approximation strategy algorithm includes: using the optimal splitting attribute of the current node. The preset candidate attribute values ​​are divided into multiple local regions according to preset intervals. The candidate values ​​are randomly selected from each local region and stored in an array to obtain the extraction coefficient.

Citation Information

Patent Citations

  • XGBoost algorithm-based heat tracing band device service life state prediction method

    CN114595624A

  • Method and apparatus for decision tree based search result ranking

    US20160162793A1