An image classification method based on tree synthesis diversity deep forest

Abstract

The present application relates to a kind of image classification methods based on tree comprehensive diversity depth forest, belong to the technical field of application in multiclass classification in data mining.The present application is directed to the problem that the time, memory and storage cost required by original depth forest model are high, according to the characteristics of decision tree model, a kind of diversity measurement method based on the morphological structure of decision tree model is designed, used to calculate the diversity between decision tree model;Then by comprehensively weighing the diversity and accuracy of decision tree two aspects, a kind of pruning strategy for optimizing random forest in cascade layer is designed;Finally, through pruning, the efficient depth forest model is obtained, the computational complexity of the model is reduced, the time, memory and storage overhead can be greatly reduced, the prediction performance and efficiency of image classification task are effectively improved.

Description

Technical Field

[0001] This invention relates to an image classification method based on tree-integrated diversity deep forest, belonging to the field of multi-class classification application technology in data mining. Background Technology

[0002] Deep Forest, a deep learning method based on non-differentiable modules, is a relatively recent development. Compared to deep neural networks, it has fewer hyperparameters, and its complexity can be adaptively determined based on the complexity of the dataset, making it easier to train. Since its inception, Deep Forest has received widespread attention from academia and industry, and has been widely applied in various fields, such as financial applications, disease classification, and computer vision. Numerous experimental results demonstrate that Deep Forest outperforms deep neural networks in many classification tasks.

[0003] Deep forests primarily consist of two modules: multi-granularity scanning and cascaded forests. Multi-granularity scanning is often used to handle datasets with high dimensionality and correlations between features. It scans data features using sliding windows of different scales, then inputs the data into a random forest or a fully random forest, finally obtaining the output features to achieve feature transformation. Cascaded forests are the core module of deep forests. They consist of multiple cascaded layers, each containing one or more independent random and fully random forests. The random and fully random forests are composed of multiple decision trees. Each cascaded layer generates a corresponding class probability vector, which is then concatenated with the original data as new training data input to the next layer. Subsequent cascaded layers are continuously trained based on new training data until a pre-defined termination condition is met, thus constituting the deep learning paradigm.

[0004] Deep forests are ensemble models based on a large number of decision trees, achieving performance far superior to single decision trees. The feature representation of a deep forest consists of predicted class probability vectors. This prediction-based representation requires storing a large number of forest models during training for use during testing. This leads to low training efficiency and high time, memory, and storage overhead. Training a deep forest model on large datasets can require tens of gigabytes of runtime memory and tens of hours of runtime, significantly hindering its application in real-world scenarios. Summary of the Invention

[0005] This invention proposes an image classification method based on tree-integrated diversity deep forest. It uses pruning to solve the problems of low training efficiency and high time, memory and storage overhead in the original deep forest algorithm, and applies it to image classification to improve the efficiency of image classification.

[0006] The technical solution of this invention is as follows: First, based on the characteristics of decision tree models, a diversity metric is designed that only targets the morphological structure of tree models to calculate the diversity among tree models; then, to comprehensively consider the relationship between the diversity and accuracy among tree models, a tree comprehensive diversity ranking and pruning algorithm is proposed and applied to deep forests to optimize the random forest in cascaded layers; finally, based on the simplified and efficient forest model after pruning, image sample classification is completed. The specific steps of the method are as follows:

[0007] Step 1: Obtain images to form a training set;

[0008] Step 2: Input the features of the training set into the cascaded forest and train the cascaded layers, which include random forest and fully random forest.

[0009] Step 3: Extract multiple decision rules from each decision tree in the trained forest;

[0010] Given a classification dataset with D features and C classes, a decision tree forest model is constructed. For each decision tree in the forest, R decision rules can be parsed. Only the splitting features used by the splitting nodes of the decision trees are used to describe them. Starting from the root node of the tree model, a depth-first traversal is performed, and the splitting features on the decision path are obtained during the traversal. A simple form of a decision rule containing d splitting features and corresponding class labels c is shown below:

[0011] Rule r :if feature1&feature2,...,&feature d |then response class c

[0012] The entire decision rule is divided into two parts: the antecedent and the consequent. The antecedent is the splitting feature in the first half and the corresponding category in the second half. A single decision tree will generate R decision rules containing different features. Taking the intersection of the R decision rules will yield a sequence containing all the splitting features used by a single decision tree. For a forest model containing N decision trees, N rule sequences will be generated.

[0013] Step 4: Measure the antecedent and consequent of the decision rule respectively to obtain the symmetric Jaccard similarity matrix P and Cosine similarity matrix Q.

[0014] For the antecedents of the decision rule, a bag-of-words model is used to transform each sequence into a fixed-length feature vector. The rule feature vector is a binary vector; if an element in the vector has a value of 1, it means the decision rule used the corresponding feature; a value of 0 means it did not. By vectorizing the decision rules, comparing the similarity between decision rules can be transformed into comparing the similarity between feature vectors. To estimate the Jaccard similarity between vectors of arbitrary length using relatively little memory and in linear time, a minimum hash algorithm is used to calculate an N×N symmetric Jaccard similarity matrix P.

[0015] For the consequent of the decision rule, count the number of decision rules corresponding to each category in a single decision tree, and generate the corresponding rule number vector S. n =[s n0 ,s n1 ,...,s nc ], s nc This represents the number of rules belonging to category c in the nth decision tree. (Rule count vector s) nc The dimension of the vector is dynamically changing, varying with the number of categories in different datasets. After obtaining N regularity vectors, cosine similarity is used for measurement. Assuming there are two vectors A and B, the cosine similarity between them is defined as follows:

[0016]

[0017] Where c is the vector dimension, A i and B i These are the elements of vectors A and B in the i-th dimension, respectively. By calculation, a symmetric Cosine similarity measure matrix Q of size N×N can be obtained.

[0018] Step 5: Simultaneously weigh the two similarity matrices regarding the antecedent and consequent of the decision rule, and calculate the similarity measure of the tree model in terms of morphological structure.

[0019] The two obtained symmetric similarity measurement matrices P and Q correspond to the antecedent and consequent of the decision rule, respectively, i.e., the splitting node and leaf node in the tree model structure. The similarity values ​​in both matrices range from [0,1], with larger values ​​indicating greater similarity. The similarity matrix T regarding the morphological structure of the decision tree is calculated using the following formula.

[0020] T = αP + (1-α)Q, α∈[0,1]

[0021] α is a tradeoff parameter, ranging from [0,1]. A value closer to 0 emphasizes the number of rules, while a value closer to 1 emphasizes the characteristics of the decision rules. This metric is independent of the specific sample data and only represents the differences in the morphological structure between paired decision trees in the random forest.

[0022] Step 6: Taking into account the similarity of the internal morphological structure and accuracy of the tree model, select the decision tree with high accuracy and high diversity.

[0023] Suppose an ensemble with N base classifiers Where a single classifier h i The diversity is defined as the average of all paired measures other than itself, as follows:

[0024]

[0025] In the formula, similarity(t) i ,t j ) is the value in the i-th row and j-th column of the structural similarity matrix T. Since the accuracy and diversity values ​​of different datasets fall within different ranges, direct weighted calculation is inappropriate. After calculating the accuracy and diversity of each base classifier in the ensemble model, the Min-Max scaling method is used to scale the data, mapping the accuracy and diversity values ​​to the range [0,1]. The specific scaling formula is as follows:

[0026]

[0027] The variable x' represents the accuracy or diversity value of a base classifier after interval scaling; a larger value indicates higher accuracy or greater diversity. Finally, the comprehensive structural diversity div(h) is given. i ) and model accuracy acc(h i The evaluation criteria for pruning methods are defined as follows:

[0028]

[0029] Calculate the value of tree comprehensive diversity by giving an appropriate trade-off parameter ρ, and select from the ranked list of tree comprehensive diversity measures. The base classifiers with the highest position constitute the ensemble trimmed subset.

[0030] Step 7: Classify the images to be classified based on the pruned forest model.

[0031] Furthermore, the minimum hash algorithm used to calculate Jaccard similarity in Step 3 specifically includes the following steps:

[0032] (1) First, define K independent and uniform random hash functions. For the i-th regular feature vector, it is necessary to iterate through each element in the vector and select the minimum hash value of that element in the k-th hash function as the signature. Finally, a K-dimensional signature vector M composed of K minimum hash values ​​can be obtained. i =[m i0 ,m i1 ,...,m ik ].

[0033] (2) For N regular feature vectors, repeat step (1) to obtain a MinHash signature matrix M of size N×K, where each element is the minimum hash value of the corresponding hash function. This signature matrix can be used to calculate the Jaccard similarity estimate between the N vectors.

[0034] (3) Assume that there are two vectors M in the signature matrix M. i and M j The similarity estimate between them is calculated using the following formula, where I(.) is an indicator function used to count the number of elements with the same hash value in the two vectors. This count divided by the length of the signature vector is an estimate of the Jaccard similarity between the two vectors. The pairwise similarity calculation results between all vectors are stored in a symmetric similarity matrix P of size N×N, where a larger value indicates a greater similarity.

[0035]

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] This invention designs a diversity metric for the morphological structure of tree models, which measures the splitting nodes and leaf nodes of the decision tree separately, enabling a comprehensive and accurate measurement of the morphological diversity among tree models. This invention comprehensively considers the relationship between diversity and accuracy among tree models, and prunes the random forest in a deep forest using a tree ensemble diversity ranking and pruning algorithm, effectively reducing the time, memory, and storage overhead in cascaded layers. Compared to the original deep forest model, the pruning operation can select a subset with high diversity and accuracy from the original ensemble model, effectively improving the prediction performance and efficiency of image classification tasks. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the training process of the present invention;

[0039] Figure 2 This is an exemplary decision tree structure in an embodiment of this application;

[0040] Figure 3This is a schematic diagram of the pruning process in the tree integrated diversity deep forest model of the present invention;

[0041] Figure 4 This is a flowchart illustrating the testing process of the present invention. Detailed Implementation

[0042] To more clearly illustrate the implementation process of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0043] Example 1:

[0044] This embodiment uses a tree-based multi-diversity deep forest image classification method for the MNIST handwritten image data classification task. The specific process is as follows: Figure 1 As shown:

[0045] Step 1: Obtain the MNIST handwritten digit image dataset, which contains 60,000 training images and 10,000 test images. The images contain handwritten digits, and the dataset has 10 categories corresponding to the digits 0-9. Transform each 28×28 pixel image in the MNIST dataset into 784 feature columns.

[0046] Step 2: Input the features of the training set into the cascaded forest and train one cascaded layer. The cascaded layer contains random forests and fully random forests. In this embodiment, one random forest and one fully random forest are used, and the number of base classifiers in each forest is set to 100.

[0047] If the current cascaded layer is the first cascaded layer, it is trained based on the original training data. If the current cascaded layer is not the first cascaded layer, it is trained based on new features obtained by concatenating the original training data and the predicted class vector output by the previous cascaded layer.

[0048] Figure 2 An exemplary decision tree structure according to an embodiment of this application is shown. The decision tree structure in both random forests and fully random forests employs the following... Figure 2 The decision tree data structure is shown below. Figure 3 The image shows the pruning process.

[0049] Step 3: Extract multiple decision rules from each decision tree in the trained forest. These are described using the splitting features of the splitting nodes. A depth-first traversal is performed starting from the root node of the tree model, capturing the splitting features along the decision path during the traversal. A simplified form of a decision rule containing multiple splitting features and their corresponding categories is shown below:

[0050] rule:feature1&feature3&feature4&,...,&feature 782 &feature 784 |thenresponse class 2

[0051] The entire decision rule can be divided into two parts: the antecedent and the consequent, namely the splitting features in the first half and the response category in the second half. Next, we will measure them separately.

[0052] Step 4: Measure the decision rules and take the intersection of multiple decision rules to obtain a sequence containing all the splitting features used by a single decision tree. For the forest model with 100 decision trees in this embodiment, this will generate 100 rule sequences. Then, the bag-of-words model is used to convert each sequence into a fixed-length feature vector. Each sequence can be expressed as a 784-dimensional rule feature vector, whose dimension exactly matches the total number of features used in the entire forest. Following this method, 100 rule feature vectors in the forest can be obtained sequentially.

[0053] Feature vector 1: [1,0,1,1,...,1,0,1]

[0054] Feature vector 2: [1,1,0,0,...,0,1,0] ...

[0056] Regular feature vector 100: [0,1,0,1,...,1,1,1]

[0057] By using the minimum hash algorithm to estimate the Jaccard similarity between vectors of arbitrary length, a symmetric Jaccard similarity matrix P of size 100×100 can be obtained.

[0058]

[0059] Step 5: Next, consider the information in the consequent of the decision rule, count the number of decision rules corresponding to each category in a single decision tree, and generate the corresponding rule number vector S. n =[s n0 ,s n1 ,...,s nc ], s nc The number of rules belonging to category c in the nth decision tree is represented by the number of rules. Therefore, the dimension of the rule number vector in this embodiment is 10.

[0060] Rule number vector 1: [6578,5571,...,7453,6368,7214]

[0061] Rule number vector 2: [6698,5336,...,6153,5968,6753] ...

[0063] Rule number vector 100: [6912,5324,...,7658,4369,5631]

[0064] Similarly, after obtaining 100 such regularity vectors, a simple and easy-to-understand cosine similarity metric is used. This metric yields a symmetric cosine similarity metric matrix Q of size 100×100.

[0065]

[0066] Step 6: Apply the following formula to the two obtained symmetric similarity measurement matrices P and Q to calculate the final structural similarity matrix T.

[0067] T = αP + (1-α)Q, α∈[0,1]

[0068] The formula contains a trade-off parameter α, whose value ranges from [0,1]. In this embodiment, a value of 0.5 is used for calculation.

[0069]

[0070] Step 7: Consider both the structural diversity and accuracy of the decision tree model. Obtain the diversity and accuracy of each base classifier in the ensemble model. The individual classifier h... i The diversity of a metric is calculated as the average of all paired measures excluding itself, as follows:

[0071]

[0072] In the formula, similarity(t) i ,t j ) is the value in the i-th row and j-th column of the structural similarity matrix T, and N is the number of base classifiers in the forest, which is set to 100 in this embodiment. Since the accuracy and diversity values ​​of different datasets fall within different ranges, direct weighted calculation is inappropriate. Therefore, the Min-Max scaling method is used to scale the data, mapping the accuracy and diversity values ​​to the range [0,1]. The specific scaling formula is as follows:

[0073]

[0074] The variable x' represents the accuracy or diversity value of a base classifier after interval scaling. The larger the value, the higher the accuracy or the greater the diversity.

[0075] Step 8: Provide the weighted structural diversity div(h) i ) and model accuracy acc(h i The evaluation criteria for pruning methods are defined as follows:

[0076]

[0077] Next, the final tree comprehensive diversity measure value is calculated using the formula. Finally, from the ranking list using the comprehensive diversity measure, [the following trees are selected]. The classifiers with the highest rank constitute the ensemble subset after pruning. In this embodiment, the value of ρ is 0.5, and the retention rate is 10%, which means that the 10 base classifiers with the highest tree diversity metric are finally selected.

[0078] List of base classifier diversity values ​​in the forest: [0.81, 0.87, 0.64, ..., 0.72, 0.93]

[0079] List of accuracy rates for each base classifier in the forest: [0.77, 0.91, 0.72, ..., 0.84, 0.89]

[0080] List of comprehensive diversity measures for each base classifier tree in the forest: [0.79, 0.89, 0.68, ..., 0.78, 0.91]

[0081] Step 9: The training samples generate predicted class vectors using the pruned forest model. In each layer, the average of all predicted class vectors generated by the forest is used as the predicted class vector for that layer. The values ​​in the predicted class vectors are summed to 1, where the dimension of the predicted class vector is the number of classes.

[0082] Step 10: Loop through the layers. Concatenate the predicted class vector from this cascaded layer with the original feature vector, and use them together as new features to input into the next cascaded layer. Repeat steps 2 through 10. The condition for stopping the growth of layers is: the loop will stop when the accuracy of a certain layer is greater than the accuracy of the layer below it.

[0083] Step 11: Testing, such as Figure 4 As shown, after training, a deep forest model based on tree-based diversity pruning is established. Test set samples are then input into the model for prediction. The 10,000 images in the test set are fed into each layer of the forest, generating predicted class vectors. These vectors are then concatenated with the original features and fed into the next layer for prediction, until the final layer outputs the predicted class for each test image. These predicted classes are then compared with the true labels to calculate the model's accuracy.

[0084] Pruning the random forest within a deep forest using a tree-based diversity ranking pruning algorithm effectively improves training efficiency and significantly reduces time, memory, and storage overhead in cascaded structures. Compared to the original deep forest model, pruning selects a more diverse and accurate subset from the original ensemble model, effectively enhancing its predictive performance. The model presented in this invention exhibits significant advantages over the original deep forest model in image classification tasks.

Claims

1. An image classification method based on tree-integrated diversity deep forest, characterized in that, A tree diversity ranking and pruning strategy is applied to a deep forest to optimize the random forest in the cascaded layers. Based on the pruned forest model, image sample classification is performed. The specific steps are as follows: Step 1: Obtain images to form a training set, and convert each image pixel into a feature column; Step 2: Input the feature columns of the training set into the cascaded forest and train the cascaded layers, which include random forest and fully random forest. Step 3: Extract multiple decision rules from each decision tree in the trained forest; Step 4: Measure the antecedent and consequent of the decision rule respectively to obtain the symmetric Jaccard similarity matrix P and Cosine similarity matrix Q. Step 5: Simultaneously weigh the two similarity matrices regarding the antecedent and consequent of the decision rule, and calculate the similarity measure of the tree model in terms of morphological structure. Step 6: Taking into account the similarity of the tree model's morphological structure and its accuracy, select decision trees with high accuracy and high diversity. Step 7: Classify the images to be classified based on the pruned forest model; The specific steps of Step 3 are as follows: Given D One characteristic, C A decision tree forest model is constructed using a classification dataset of 10 categories. For each decision tree in the forest, a solution can be obtained. R Each decision rule uses only the splitting features used by the splitting nodes of the decision tree. A depth-first traversal is performed starting from the root node of the tree model, acquiring the splitting features along the decision path during the traversal. Each rule contains... d The simplified form of the decision rule for each splitting feature and its corresponding class label c is shown below: ； The entire decision rule is divided into two parts: the antecedent and the consequent. The antecedent consists of the splitting features, and the consequent consists of the corresponding categories. A single decision tree will produce... R The decision rules contain different splitting characteristics, for R The intersection of the decision rules yields a sequence containing all the splitting features used by a single decision tree, where each sequence is expressed as a rule feature vector with a dimension matching the total number of features used by the entire forest. N A forest model with decision trees will produce N A sequence of rules; The specific steps of Step 4 are as follows: For the antecedent of the decision rule, the bag-of-words model is used to transform each sequence into a fixed-length feature vector, and the Jaccard similarity between the vectors is calculated using the minimum hash algorithm, resulting in a feature vector of size [value missing]. The symmetric Jaccard similarity matrix P; For the consequent of the decision rule, count the number of decision rules corresponding to each category in a single decision tree, and generate the corresponding rule number vector. , Representing the n In a decision tree, belonging to category c The number of rules is measured using cosine similarity. Assuming there are two vectors A and B, the cosine similarity between them is defined as follows: ； in b It is the vector dimension. and These are the elements of vectors A and B in their i-th dimension, respectively. Their size can be calculated to be... The symmetric Cosine similarity measure matrix Q.

2. The image classification method based on tree-integrated diversity deep forest according to claim 1, characterized in that, The specific steps of Step 5 are as follows: The similarity matrix T of the decision tree morphology structure is calculated using the following formula. ； The parameter is used for weighting and its value ranges from [0,1].

3. The image classification method based on tree-integrated diversity deep forest according to claim 2, characterized in that, The specific steps of Step 6 are as follows: Suppose a has N Ensemble of base classifiers , where a single classifier The diversity is defined as the average of all paired measures other than itself, as follows: ； in the formula It is the th element in the structural similarity matrix T i Okay, number j After calculating the accuracy and diversity of each base classifier in the ensemble model, the data is scaled using the Min-Max scaling method, mapping the accuracy and diversity values ​​to the range [0,1]. The specific scaling formula is as follows: ； variable x’ This represents the accuracy or diversity value of a base classifier after interval scaling, and finally gives the overall structural diversity. and model accuracy The evaluation criteria for pruning methods are defined as follows: ； By giving appropriate trade-off parameters Calculate the value of tree comprehensive diversity by selecting from the sorted list of tree comprehensive diversity measures. The base classifiers with the highest position constitute the ensemble trimmed subset.

4. The method according to claim 1, characterized in that, The minimum hash algorithm used to calculate Jaccard similarity in Step 4 specifically includes the following steps: (1) First, define K independent and uniform random hash functions. For the i-th regular feature vector, it is necessary to iterate through each element in the vector and select the minimum hash value of that element in the k-th hash function as the signature. Finally, we can obtain K Composed of the smallest hash values K dimensional signature vector ; (2) For N Given a set of regular feature vectors, repeat step (1) to obtain a MinHash signature matrix of size N×K. M Each element in the matrix is ​​the minimum hash value of the corresponding hash function; (3) Assuming a signature matrix M Two vectors in and The similarity between them is estimated using the following formula: ； in, It is an indicator function used to count the number of elements with the same hash value in two vectors. The pairwise similarity calculation results between all vectors are stored in a file of size . In the symmetric Jaccard similarity matrix P, The larger the value, the greater the similarity.

Images