Improved multivariate decision tree construction method and system for multi-classification task

Abstract

This invention discloses an improved multivariate decision tree construction method and system for multi-class classification tasks. The method includes: acquiring a training sample set with class labels; at the current split node, using an improved G-means clustering algorithm to cluster the sample subset, adaptively determining the number of clusters; if the number of clusters is 0, then directly split the current node into 1 child node; if the number of clusters is 0, then switch to binary split mode, using K-means with k=2 to split the sample subset into two child nodes; recursively execute until the stopping condition is met, generating leaf nodes and storing the corresponding class labels; and predicting the class of the samples to be classified based on the constructed decision tree. This invention achieves adaptive multiple splitting through improved G-means, solving the problem of excessively deep tree structures caused by existing multivariate decision trees relying on binary splitting, while avoiding the class imbalance problem introduced by One-vs-All decomposition.

Description

Technical Field

[0001] This invention relates to the field of machine learning and pattern recognition technology, specifically to an improved multivariate decision tree construction method and system for multi-classification tasks, which is particularly suitable for data processing with complex distribution structures in multi-classification tasks. Background Technology

[0002] Decision trees are widely used in classification tasks due to their intuitive structure, high computational efficiency, and strong interpretability. Decision trees can be divided into univariate decision trees and multivariate decision trees.

[0003] 1. Classification and shortcomings of existing technologies (1) Univariate decision tree Univariate decision trees (such as C4.5 and CART) split based on a single feature at each internal node, generating axis-parallel decision boundaries. When there are complex interactions between data features, axis-parallel boundaries can easily lead to overfitting or underfitting of the model, resulting in limited generalization ability.

[0004] (2) Multivariate decision tree Multivariate decision trees construct decision boundaries by splitting internal nodes based on multiple features, enabling them to better fit the true distribution of data. However, existing multivariate decision trees suffer from the following technical problems: Question 1: Limitations of binary splitting Existing methods (such as OC1, STree, BDTKS, etc.) almost all employ a binary splitting strategy (each node splits into two child nodes). In multi-class classification tasks, binary splitting often fails to effectively divide multi-class samples in one go, resulting in an overly deep tree structure, model redundancy, and susceptibility to overfitting.

[0005] Question 2: Dependency Decomposition Scheme Some existing methods typically require decomposing multi-class classification problems into multiple binary classification problems (such as One-vs-All or One-vs-One). The One-vs-All (OvA) approach introduces a severe class imbalance problem (a significant disparity in the number of positive and negative samples), while the One-vs-One (OvO) approach alleviates the imbalance but requires training. Each classifier has a large computational cost.

[0006] Question 3: Insufficient exploration of multiple splitting mechanisms A few methods that attempt multiple splits (such as CRUISE and BoostTree) have the following limitations: CRUISE relies on linear discriminant analysis for splitting and uses a fixed, predetermined number of splits; BoostTree requires pre-training multiple regression models, which is computationally complex. Neither of them has achieved adaptive multiple splitting—that is, automatically determining the number of splits based on the data's own distribution characteristics—while ensuring computational efficiency.

[0007] 2. Limitations of the G-means clustering algorithm G-means is a clustering algorithm that can automatically determine the number of clusters. Its core idea is to recursively split non-Gaussian clusters until all clusters pass the normality test. However, the original G-means algorithm has the following technical drawbacks: Defect 1: Lack of data preprocessing The original G-means algorithm operates directly on the raw data, without considering the impact of differences in the dimensions of features on distance calculation. When there is overlap between clusters, these differences in dimensions can distort the distance measurement, resulting in clustering results that do not reflect reality.

[0008] Defect 2: Excessive fragmentation G-means will continue to split until all clusters pass the normality test. In areas where clusters overlap, samples do not conform to a normal distribution, and the algorithm will continuously split to produce a large number of small, fragmented clusters, resulting in excessive fragmentation that does not conform to the true clustering structure.

[0009] The aforementioned limitations restrict the effectiveness of G-means in splitting decision tree nodes.

[0010] 3. Summary of existing technical problems In summary, the existing technology has the following unresolved technical problems: (1) The binary splitting strategy of multivariate decision trees leads to excessively deep tree structures; (2) Multi-class classification problems rely on OvA / OvO decomposition schemes, which may introduce class imbalance or computational overhead; (3) The existing G-means algorithm results in distorted clustering in overlapping clustering scenarios and is prone to excessive fragmentation. Summary of the Invention

[0011] To address the problems existing in the prior art, the present invention aims to provide an improved multivariate decision tree construction method and system for multi-classification tasks. Its core concept lies in: standardizing and... Two constraints are imposed, and the improved G-means is applied to the splitting of decision tree nodes. The multi-split or binary splitting mode is automatically selected based on the number of clusters, thereby improving multi-classification performance while maintaining a compact tree structure.

[0012] To achieve the above objectives, the present invention adopts the following technical solution: An improved multivariate decision tree construction method for multi-classification tasks includes the following steps: Step A: Obtain training data Obtain the training sample set with category labels. ,in for Sample vectors with 3D features For category labels, This represents the total number of categories.

[0013] Step B: Determine the stopping condition for the node For the sample subset corresponding to the current node Determine if the stopping condition is met: Stop condition 1: All samples belong to the same category; Stop condition 2: (Value range 2 to 20).

[0014] If any stopping condition is met, the current node is marked as a leaf node, and the category label with the most samples in that node is stored, and the splitting stops.

[0015] If the stopping condition is not met, execute the splitting process.

[0016] Step C: Improved G-means clustering This invention proposes two key improvements to the original G-means algorithm and applies them to node splitting.

[0017] Improvement 1: Z-score normalization preprocessing Before performing G-means, Z-score standardization is performed on the samples in the dataset: in: The original sample vector; This is the original sample mean vector; Let be the standard deviation vector, where ; This indicates element-wise division.

[0018] Z-score standardization transforms the features of each dimension into a standard distribution with a mean of 0 and a standard deviation of 1, eliminating dimensional differences and making the clustering results more reflective of the true similarity structure of the samples.

[0019] Improvement 2: Maximum Cluster Number Constraint ( ) In the G-means clustering process, a user-specifiable maximum number of clusters constraint parameter is introduced. When the current number of clusters reaches Even if some clusters fail the normality test, the splitting process is forcibly stopped.

[0020] In the decision tree construction scenario of this invention Preferably, it is set to the total number of categories in the training sample set. ,Right now This is because: at any node in a decision tree, the number of clusters cannot exceed the total number of categories. It retains the adaptability of G-means while effectively suppressing excessive splitting in overlapping regions.

[0021] Improve the complete execution flow of G-means: C1. The samples in the dataset are Z-score standardized. C2. Initialize the number of clusters to... That is, all samples belong to one cluster; C3. Perform Anderson-Darling normality test on each cluster (significance level). (Preferred range 0.01 to 0.15) If a cluster passes the test (conforms to a Gaussian distribution), then that cluster is retained. If a cluster fails the test (does not conform to a Gaussian distribution), it is split into two sub-clusters (by using K-meanssk=2). Increase by 1, then re-execute K-means to update all cluster centers; C4. Repeat step C3 until all clusters pass the normality test or the number of clusters reaches the target value. .

[0022] Let the final number of clusters be . The set of cluster centers is (Located in a standardized space).

[0023] C5. Restore the cluster centers to the original data space using an inverse transformation: Step D: Adaptive selection of splitting mode Based on the number of clusters Select split mode: Multiple splitting modes ( ): Split the current node directly into Each child node corresponds to a cluster, i.e., the nth child node. The child nodes contain all nodes belonging to the first child node. Samples from each cluster.

[0024] Two-part splitting model ( ):when When the condition is met, it indicates that the sample subset of the current node generally conforms to a single Gaussian distribution but contains multiple categories (i.e., a multi-class Gaussian cluster). At this point, the improved G-means cannot split further, so it switches to the binary split mode and uses K-means (k=2) to split the node into two child nodes.

[0025] Initialization optimization for binary splitting: To avoid the sensitivity of K-means to initial centers, this invention uses PCA projection and median splitting to determine the initial cluster centers. Principal component analysis (PCA) is performed on the standardized samples to obtain the direction vector of the first principal component. ; Project the sample onto the direction of the first principal component: ; Calculate the median m of the projected values; Calculate the mean of the two groups of samples whose projected values ​​are greater than m and less than or equal to m, respectively, and use them as the two initial cluster centers; Perform K-means (k=2) with the initial centers mentioned above.

[0026] Step E: Store split information Stores the split information of the current node, including: Cluster center set (Original space, in which) (This represents the actual number of child nodes).

[0027] Step F: Recursive Construction Recursively execute steps B through E for each child node until all nodes become leaf nodes.

[0028] Step G: Classification Prediction Based on the constructed decision tree, the samples to be classified Make a prediction: G1. Start from the root node; G2. Calculation Euclidean distances to the cluster centers stored in the current node: ; G3. Select the child node corresponding to the nearest cluster center: ; G4. Enter child node Repeat G2-G3; G5. When a leaf node is reached, output the category label stored in that leaf node as the prediction result.

[0029] Compared with the prior art, the present invention has the following beneficial effects: (1) The tree structure is more compact The G-means clustering algorithm's ability to split multiple nodes enables the early identification and one-time separation of pure Gaussian clusters, significantly reducing tree depth and the number of split nodes.

[0030] (2) Avoid class imbalance problem This invention directly processes multi-class samples at multiple split nodes, eliminating the need for One-vs-All or One-vs-One decomposition schemes, thus fundamentally avoiding the class imbalance problem.

[0031] (3) Improve the clustering accuracy in overlapping clustering scenarios Z-score standardization eliminates the distortion of distance metrics caused by differences in units, making G-means clustering results more consistent with the true distribution of the data.

[0032] (4) Suppress excessive fragmentation constraint( This avoids meaningless, fragmented splitting in overlapping areas.

[0033] (5) Maintain computational efficiency standardization, Both the inspection and binary split initialization are lightweight operations. The overall decision tree construction efficiency is comparable to existing multivariate decision tree methods and superior to existing univariate decision tree methods. It maintains a high prediction efficiency comparable to existing multivariate decision tree methods.

[0034] (6) Good generalization performance The compact tree structure reduces the risk of overfitting and provides better generalization performance. Attached Figure Description

[0035] To more clearly illustrate the technical implementation schemes in the embodiments of this application, related drawings are provided. It should be noted that the drawings described in this section are only some embodiments of this application. Those skilled in the art can obtain other drawings based on this invention and the aforementioned drawings without any creative effort.

[0036] Figure 1 This is the overall flowchart of the GMDT method of the present invention.

[0037] Figure 2 This is a schematic diagram of adaptive splitting mode selection in this invention, including (a) the transformation relationship between cluster types obtained from the actual category distribution, and (b) multiple splitting ( (c) binary splitting There are two scenarios.

[0038] Figure 3The figures show a comparison of the results of the original G-means and the improved G-means on the overlapping clustering dataset. (a) shows the true clustering distribution of the 10-class dataset, (b) shows the clustering results of the original G-means, and (c) shows the clustering results of the improved G-means in this invention.

[0039] Figure 4 The diagram shows a comparison of the classification results of GMDT and BDTKS using the method of this invention. (a) shows the training samples and class labels; (b) shows the test samples and class labels; (c) shows the misclassified sample points of BDTKS; and (d) shows the misclassified sample points of GMDT. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0041] like Figure 1 As shown, this invention provides an improved multivariate decision tree construction method for multi-classification tasks, comprising the following steps: Step A: Obtain a training sample set with category labels, wherein each sample in the training sample set has multidimensional features and a corresponding category label; Step B: At the current split node, there are 6 possible actual distribution patterns of the categories corresponding to the sample subsets reaching this node, such as... Figure 2 As shown in (a), an improved G-means clustering algorithm is applied to it to adaptively determine the number of clusters. ,in The improved G-means clustering algorithm includes: performing Z-score standardization on the sample subset before clustering, and setting a maximum cluster number constraint parameter. When the current number of clusters reaches The splitting stops at that time; Step C: Based on the number of clusters Select split mode: like Then proceed as follows Figure 2 The multi-split method shown in (b) directly splits the current node into... Each child node corresponds to a cluster; like Then switch to Figure 2 The binary splitting pattern shown in (c) uses K-means with k=2 to split the sample subset into two child nodes; Step D: Store the splitting information of the current node, including the cluster center corresponding to each child node; Step E: Recursively execute steps B through D for each child node until the stopping condition is met; Step F: Mark the nodes that meet the stopping condition as leaf nodes, store the category label that appears most frequently in the node, and complete the construction of the decision tree.

[0042] Example 1: Validation of the improved clustering effect of G-means Experimental objective: To verify the two improvements proposed in this invention (Z-score standardization + Constraints improve the performance of G-means clustering.

[0043] Datasets: Construct two synthetic datasets containing 1000 two-dimensional sample points, one containing 6 real clusters and the other containing 10 real clusters, with overlap between the clusters.

[0044] Experimental setup: The original G-means dataset had neither Z-score standardization nor... Constraints; This invention improves G-means by incorporating Z-score standardization and setting... =True number of clusters; Parameters are set uniformly. Table 1. Comparison between the original G-means and the improved G-means of this invention. Experimental Results: The Adjusted Rand Index (ARI) was used to measure clustering performance, and the predicted number of clusters was recorded. Table 1 shows the comparison results of the evaluation metrics between the original G-means and the improved G-means of this invention. Figure 3 Figures (a), (b), and (c) show the actual clustering distribution, the original G-means clustering results, and the improved G-means clustering results for 10 clusters with overlapping classes, respectively.

[0045] Conclusion: This invention improves G-means, enhancing the realism and compactness of clustering in overlapping clustering scenarios.

[0046] Example 2: Decision Tree Construction and Validation on Synthetic Datasets Dataset: A synthetic dataset of 2400 two-dimensional sample points was constructed, containing 12 classes with 200 sample points per class. Ten classes follow a Gaussian distribution, one a heavy-tailed distribution, and one a Copula distribution. There is varying degrees of overlap between the classes. Random stratified sampling was used to divide the samples into 80% and 20% sets, serving as the training and test sets, respectively.

[0047] Parameter settings: In GMDT, significance level , , .

[0048] Construction process: The root node contains all 1920 training sample points. Multiple splits are performed using improved G-means clustering to obtain k=12 sub-clusters. One sub-cluster is a high-purity cluster and directly becomes a leaf node. The remaining 11 mixed sub-clusters continue to split recursively. The final tree depth is 6, containing 51 split nodes.

[0049] Comparison: Using the same dataset, with parameter λ=1, The constructed BDTKS decision tree (two-way split) has a depth of 10 and contains 83 split nodes.

[0050] Prediction results: The GMDT classification accuracy of the method of this invention is 0.8417, and the macro F1 score is 0.8417; the BDTKS accuracy is 0.8208, and the macro F1 score is 0.8219. Figure 4 This paper presents a comparison of the classification results of GMDT and BDTKS using the method of this invention. The numbers in the figure represent the centroids of each class in the training samples. Figure 4 (c)-(d) in the figure show the sample points misclassified by BDTKS and GMDT. It can be seen that GMDT is cleaner around the true class label "3" than BDTKS, indicating that fewer sample points were misclassified.

[0051] Performance analysis: This invention improves classification performance and generates a more compact tree structure.

[0052] Example 3: Ablation experiment of GMDT using the method of the present invention Experimental objective: To verify Z-score standardization and The contribution of constraints to the performance of decision trees.

[0053] Dataset: 12 types of synthetic datasets.

[0054] Experimental results: Keep parameter settings , , The configurations remain unchanged, with the following transformations: original unmodified G-means, +standardization only, and +standardization only. Decision trees were constructed based on the constraints, and the resulting decision trees and their classification performance are compared with those of the present invention in Table 2.

[0055] Table 2 Ablation Experiment Results of GMDT Analysis: Standardization improves clustering quality, enabling multi-splitting to separate high-purity clusters earlier, thereby reducing the number of splitting nodes; Constraints prevent excessive splitting in overlapping regions, improving the classification generalization performance of the decision tree. The two work synergistically to generate a compact decision tree with better generalization ability.

[0056] This invention can be applied to the following scenarios: General multi-classification tasks: applicable to any scenario requiring multi-classification, including but not limited to image recognition, text classification, medical diagnosis, fault detection, financial risk control, etc.

[0057] Imbalanced data classification: Since it does not require OvA / OvO decomposition, this invention is naturally applicable to imbalanced datasets.

[0058] Scenarios with high interpretability requirements: The transparent structure of decision trees makes the classification process traceable and verifiable, making them suitable for scenarios such as healthcare and finance that require decision interpretation.

[0059] The method of this invention can be deployed on computing servers, edge computing devices or cloud platforms, is compatible with existing machine learning systems, and has good prospects for industrialization.

Claims

1. An improved multivariate decision tree construction method for multi-classification tasks, characterized in that, Includes the following steps: Step A: Obtain a training sample set with category labels, wherein each sample in the training sample set has multidimensional features and a corresponding category label; Step B: At the current split node, perform an improved G-means clustering algorithm on the subset of samples that have reached that node, adaptively determining the number of clusters. ,in The improved G-means clustering algorithm includes: performing Z-score standardization on the sample subset before clustering, and setting a maximum cluster number constraint parameter. When the current number of clusters reaches the maximum number of clusters constraint parameter The splitting stops at that time; Step C: Based on the number of clusters Select split mode: like Then the current node will be directly split into Each child node corresponds to a cluster; like Then switch to binary split mode, using K-means with k=2 to split the sample subset into two child nodes; Step D: Store the splitting information of the current node, including the cluster center corresponding to each child node; Step E: Recursively execute steps B through D for each child node until the stopping condition is met; Step F: Mark the nodes that meet the stopping condition as leaf nodes, store the category label that appears most frequently in the leaf node, and complete the construction of the decision tree.

2. The method according to claim 1, characterized in that, It also includes the step of predicting the class of the samples to be classified based on the constructed decision tree: Starting from the root node, calculate the distance between the sample to be classified and each cluster center stored in the current node, and select the child node corresponding to the nearest cluster center to continue the routing. Repeat the above process until a leaf node is reached; Output the category label stored in the leaf node as the prediction result.

3. The method according to claim 1, characterized in that, In step C, when When the current node's sample subset conforms to a single Gaussian distribution but contains multiple classes, the system switches to binary split mode; when The time indicates that the sample subset of the current node can be naturally divided into multiple Gaussian subclusters, and the multi-splitting mode is directly adopted.

4. The method according to claim 1, characterized in that, The formula for Z-score standardization is: in The standardized sample vector, The original sample vector, This is the original sample mean vector. The standard deviation vector, This represents element-wise division; After obtaining the cluster centers in the standardized space, the original sample space is restored through an inverse transformation. The formula for the inverse transformation is: in For the first sample space Cluster centers, For the standardized sample space, the first Cluster centers.

5. The method according to claim 1, characterized in that, The maximum clustering number constraint parameter Set to the total number of categories c in the training sample set.

6. The method according to claim 1, characterized in that, The initial cluster centers in the K-means k=2 binary splitting pattern are determined in the following way: Principal component analysis was performed on the standardized samples to obtain the direction vector of the first principal component. w ; The sample is projected onto the direction of the first principal component to obtain the projected value. ; Calculate the median m of the projected values; Calculate the mean values ​​of the two groups of samples whose projected values ​​are greater than m and less than or equal to m, respectively, and use them as the two initial cluster centers.

7. The method according to claim 1, characterized in that, The stopping conditions include: (a) All samples in the current node belong to the same category; or (b) The number of samples in the current node is less than the preset threshold min_samples_split, where the value of min_samples_split ranges from 2 to 20.

8. The method according to claim 1, characterized in that, The improved G-means clustering algorithm uses the Anderson-Darling statistical test to determine whether each cluster conforms to a Gaussian distribution, with a significance level of [missing information]. The value range is from 0.01 to 0.

15.

9. An improved multivariate decision tree construction system for multi-classification tasks implementing the method of any one of claims 1 to 8, characterized in that, include: The sample acquisition module is used to acquire a training sample set with category labels; An improved G-means clustering module is used to perform G-means clustering after Z-score standardization on the subset of samples reaching the current split node, adaptively determining the number of clusters. And set the maximum cluster number constraint parameter. ; The split mode selection module is used to select the split mode based on... Value selection split mode: If Then switch to binary splitting mode and use K-means with k=2 for splitting. Then execute multiple splits; The recursive building block is used to recursively perform clustering and splitting until the stopping condition is met, generating a decision tree; The storage module is used to store the cluster centers and category labels of the leaf nodes for each node.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

Images