Multi-label classification method based on depth fuzzy system

Through the improved Wang-Mendel method and stacking generalization principle, the deep model is constructed, combined with the random grouping averaging strategy and the training method that does not depend on gradient calculation, the data requirements and complexity problems of deep neural networks in multi-label classification are solved, and the data requirements and complexity problems of deep neural networks in multi-label classification are realized. Efficient and explainable multi-label classification effect.

CN119025970APending Publication Date: 2024-11-26DALIAN NATIONALITIES UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411144240.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-06-13
Filing Date
2024-08-20
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing multi-label classification algorithms have problems such as huge data demand, long training time, high model complexity, and poor model interpretability when processing deep neural network models. The deep fuzzy system has not been studied in multi-label classification applications.

Method used

The improved Wang-Mendel method is used to generate fuzzy rules, and a hierarchical depth model is constructed through the principle of stacking generalization, and a random grouping average feature transformation strategy is used in the intermediate layer. Combined with a model training method that does not rely on gradient calculation, the classification effect and training efficiency are optimized. .

Benefits of technology

It realizes efficient processing of multi-label classification problems and interpretability of results, improves the generalization ability and stability of the model, and is especially suitable for high-dimensional feature data sets, significantly improving the performance of multi-label classification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025970A_ABST
    Figure CN119025970A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-label classification method based on a depth fuzzy system, and belongs to the technical field of multi-label image classification algorithms. The method comprises the following steps: generating a fuzzy rule by adopting a WM method, and associating the rule with a feature and a label set; a layered depth model is constructed through a stacking generalization principle, and layer-by-layer stacking of the multi-label WM fuzzy system is achieved; in the training process, the fuzzy set membership degree is calculated to determine the rule weight, and the rule is optimized to process conflicts and redundancy; a downsampling strategy based on random grouping is used in the middle layer to reduce feature dimensions, and training efficiency is improved. Experimental results show that the method has excellent classification accuracy on high-dimensional features and label data sets; in addition, the model training process does not depend on gradient calculation, and is suitable for various data types.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of multi-label image classification algorithms, and in particular to a multi-label classification method based on a deep fuzzy system. Background Art

[0002] Multi-label classification is an important research field in machine learning. Unlike traditional single-label classification problems, multi-label classification allows an instance to belong to multiple categories at the same time. This feature makes multi-label classification more closely reflect the complexity of real-world situations, such as text classification, image recognition, bioinformatics, music and video classification, medical diagnosis and other applications. Existing multi-label classification algorithms can be mainly divided into three categories: problem transformation methods, algorithm adaptation methods and integration methods.

[0003] Problem transformation methods involve converting a multi-label problem into one or more single-label problems for solution, such as binary association, label power set, classifier chain, etc. These methods can be applied directly, but may not fully exploit the relationship between labels. Algorithm adaptation methods directly model multi-label data without converting the problem into a single-label problem, such as k-nearest neighbor-based methods, decision tree-based methods, and neural network-based methods. These methods do a better job of capturing the dependencies between labels, but they are more complex and computationally intensive. With the rapid development of data and applications, the requirements for adaptive algorithms in modeling the relationship between features and labels are also increasing.

[0004] Due to their excellent data representation capabilities, deep neural network models are widely used to learn new feature spaces and input them into multi-label classifiers, while using the network architecture itself to explore label correlations. However, deep neural network models have problems such as huge data requirements, high training time, large model complexity, and weak model interpretability. The success of deep neural network models lies in layer-by-layer processing, feature transformation within the model, and sufficient model complexity. As long as these requirements are met, deep models can be built based on any components.

[0005] So far, there is no literature report on how to apply classification systems based on deep fuzzy systems to multi-label classification. Summary of the invention

[0006] In order to solve the problems existing in the prior art, the present invention provides a multi-label classification method based on a multi-label deep fuzzy rule classification system, which has important technical value and application prospects. This method will combine the advantages of multi-label learning and deep fuzzy systems, and achieve efficient processing of multi-label classification problems and interpretability of results through deep learning of fuzzy rules and multi-level feature interaction.

[0007] S1. The improved Wang-Mendel (WM) method is used to generate fuzzy rules that can capture the relationship between features and label sets. Each rule is determined by calculating the fuzzy set membership of the feature.

[0008] S2. A hierarchical deep model is constructed by stacking the generalization principle, and the multi-label WM fuzzy system is stacked layer by layer to form a deep model structure. The structure is divided into input layer, intermediate layer and output layer.

[0009] S3. Use feature transformation strategies such as random grouping and averaging in the middle layer to optimize the classification effect and reduce the feature dimension. This strategy can effectively improve the training efficiency, especially for high-dimensional feature data sets.

[0010] S4. Input the optimized fuzzy rules into the deep model and use a model training method that does not rely on gradient calculation. This method can better handle complex multi-label data in reality and provides an efficient and stable solution for model training.

[0011] Further: a multi-label classification method based on a deep fuzzy system, comprising the following steps: S1. using the Wang-Mendel method to generate fuzzy rules to capture the relationship between features and label sets, each rule is determined by calculating the fuzzy set membership of the feature;

[0012] Generate initial fuzzy rules based on input features. If fuzzy sets are defined in each feature, the feature space will be divided into a series of fuzzy region cells; let S = {s t |t∈[1,z]} is the set of cells, is the number of cells;

[0013] The corresponding fuzzy rules are as follows:

[0014]

[0015] where x i is any instance that falls in the tth cell, p t,j Represents x i With y i,j The correlation between t =(κ t,1 ,κ t,2 ,…,κ t,d ) represents the vector consisting of the activated fuzzy set indices, and t and κ t The mapping relationship between them can be determined by the following formula:

[0016]

[0017] Let xt represents the sample set falling in the tth cell, then p t It can be obtained from the following formula:

[0018]

[0019] The parameters of the cells covered by the training samples are learned through the above method;

[0020] For cells where no samples fall, the neighborhood mean strategy is used to estimate. During the traversal of S, assuming that s t is a cell that is not covered by any training sample, find s t Adjacent cell; if for any Both So called s t The neighbor set of In the calculation, the corresponding values ​​of the cells covered by non-zero samples are averaged to estimate p t ; S2. A hierarchical deep model is constructed by stacking generalization principle, and the multi-label WM fuzzy system is stacked layer by layer to form a deep model structure; the structure is divided into input layer, middle layer and output layer; because the deep model is designed by stacking generalization principle, the output of the previous layer is used as the input of the next layer;

[0021] For the first layer x i =(x i,1 ,x i,2 ,...,x i,d ) represents the original input; after N r When resampling with replacement, randomly select N s indexes to generate feature subsets; the original d-dimensional feature space is divided into N s Various subspaces of dimensionality for training WM fuzzy systems where γ∈[1,N r ] and φ represent the layer index; Output express;

[0022] N was performed on the WM fuzzy system index set. g sampling without replacement, each sampling randomly generates a m WM fuzzy system outputs; then to the nth group, its elements are Average; the average results of all groups are concatenated as input to the next layer;

[0023] The output layer processes the average output of the WM fuzzy system group: λ η,δrepresents the weight associated with the δth label of the ηth group output; at the end of the output layer, λ is calculated by the following formula η,δ :

[0024]

[0025] Among them, N l is the number of layers;

[0026] S3. Use random group averaging feature transformation strategy in the middle layer;

[0027] This strategy can effectively improve training efficiency, especially for high-dimensional feature data sets. The advantage of the random grouping average strategy is that it can effectively reduce the risk of overfitting during model training and improve the generalization ability of the model. Through random grouping, different sample combinations are obtained each time training, thereby increasing the model's diverse learning of data. This can effectively prevent the model from over-relying on a specific data distribution or sample order, thereby improving the model's adaptability to unknown data. In addition, the random grouping average strategy can also reduce the local minimum problem during training, making the model more stable and easier to converge to the global optimal solution. Therefore, the random grouping average strategy plays an important role in improving model robustness, generalization ability and stability.

[0028] S4. Input the optimized fuzzy rules into the deep model and use a model training method that does not rely on gradient calculation;

[0029] The optimized fuzzy rules are input into the deep model. Given the training dataset D and parameters, at each layer In the example, different feature combinations are generated according to the given number and dimension of features, each of which contains N s features, and the combined information is stored in middle;

[0030] According to the generated feature combination, the input data is reorganized into As the input data set; then for each feature combination, use the reorganized data And the given number of fuzzy sets n constructs the WM fuzzy system Then, the model parameters of the WM fuzzy system are obtained according to the training data set The missing model parameters of the WM fuzzy system are estimated by the neighborhood averaging strategy. Using restructured data WM Fuzzy System Perform inference and obtain output The output of the WM fuzzy system is divided into N g groups, each containing N moutputs, and the grouping information is stored in V; and the output of each group is averaged to obtain If the current layer is the last layer, the weights of different WM fuzzy system groups related to each label are calculated according to the specified formula; otherwise, the average results of all groups are concatenated as the input data of the next layer; after all layers are trained, the training results are returned.

[0031] In the step S1, before generating the initial fuzzy rules, the input features are preprocessed, including feature standardization and label processing.

[0032] The parameters in step S4 include: the number of fuzzy sets defined in each feature, n, the number of times the sampling with replacement is performed, N r , the dimension of the feature subspace N r , the number of times N sampling without replacement is performed g , the number of WM fuzzy systems in each group N m , and the number of layers N l .

[0033] Compared with the prior art, the present invention has the following significant advantages: (1) This method adopts a hierarchical stacking strategy to build a deep model through hierarchical Wang-Mendel (WM) components, which enhances the generalization ability of the model. (2) The WM method is used to effectively model uncertain data, and the association between features and categories is characterized by fuzzy rules, which is particularly suitable for processing data sets with high-dimensional features or labels. (3) In order to reduce the dimension of the output of the intermediate layer, this method introduces a strategy of random grouping combined with average aggregation, which not only improves the training efficiency, but also comprehensively captures the intrinsic feature information of the data. (4) This technology does not rely on gradient calculation during the training process, which improves its applicability when processing data sets with mixed types of features. (5) Experimental results show that ML-DFRBCS performs well in multi-label classification performance, especially on high-dimensional data sets, and is competitive compared with other methods.

[0034] Fuzzy systems are seen as a promising approach to developing more structured, interpretable, and causal models. This is due to the advantages of fuzzy systems in knowledge representation and processing of uncertain and imprecise data. Inspired by these facts, a deep fuzzy system-based classification system (DFRBCS) was proposed and demonstrated high classification accuracy and post hoc interpretability in solving multi-class classification tasks. The system designs a hierarchical model structure through the principle of superposition generalization, which makes it have strong classification accuracy and makes it easier to explore its working mechanism, prediction process, and decision-making basis. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an example diagram of a cell when d=2 and n=5.

[0036] Figure 2 It is a schematic diagram of the multi-label deep fuzzy system model. DETAILED DESCRIPTION

[0037] A multi-label classification method based on a deep fuzzy system, comprising:

[0038] The first step is to generate initial fuzzy rules based on the input features. If fuzzy sets are defined in each feature, the feature space will be divided into a series of fuzzy regions, which are named cells. Let S = {s t |t∈[1,z]} is the set of cells, The number of cells. The corresponding fuzzy rules are as follows:

[0039]

[0040] where x i is any instance that falls in the tth cell, p t,j Represents x i With y i,j The correlation between t =(κ t,1 ,κ t,2 ,…,κ t,d ) represents the vector consisting of the activated fuzzy set indices, and t and κ t The mapping relationship between them can be determined by the following formula:

[0041]

[0042] Let x t represents the sample set falling in the tth cell, then p t It can be obtained from the following formula:

[0043]

[0044] Through the above method, we can learn the parameters of the cells covered by the training samples.

[0045] However, for cells where no samples fall, their parameters cannot be directly derived from the training data. The neighborhood mean strategy is used to infer these cells. This process is implemented in an iterative manner. During the traversal of S, assume that s t is a cell that is not covered by any training sample. We first t Find the adjacent cell. If for any Both So called s t The neighbor set of . Then, by By averaging the corresponding values ​​of the cells covered by non-zero samples, we can estimate p t .

[0046] In the second step, the deep model is designed by adopting the stack generalization principle, and the output of the previous layer is used as the input of the next layer. i =(x i,1 ,x i,2 ,...,x i,d ) represents the original input. r When resampling with replacement, randomly select N s Generate feature subsets based on the indices.

[0047] In this way, the original d-dimensional feature space is divided into N s These reorganized low-dimensional datasets are used to train the WM fuzzy system where γ∈[1,N r ] and φ represent the layer index. Output express.

[0048] If we directly connect the outputs of all WM fuzzy systems, the input dimension of the next layer will increase significantly when l is large. In order to achieve effective dimensionality reduction, we perform N on the WM fuzzy system index set. g Each sampling randomly generates a m A group consisting of the outputs of a WM fuzzy system.

[0049] Then to the nth group, its elements are Average. The average results of all groups are connected as the input of the next layer. The difference between the first layer and the middle layer is the source of their input. The former is the original training data, while the latter is the output of the previous layer. For the output layer, unlike the other two types of layers, the average output of the WM fuzzy system group is processed. Let λ η,δ represents the weight associated with the δth label of the ηth group output. At the end of the output layer, λ can be calculated by the following formula: η,δ °:

[0050]

[0051] Among them, N l It is the number of layers.

[0052] The third step is to use feature transformation strategies such as random grouping and averaging in the middle layer to optimize the classification effect and reduce the feature dimension. This strategy can effectively improve the training efficiency, especially for high-dimensional feature data sets. The advantage of the random grouping and averaging strategy is that it can effectively reduce the risk of overfitting during model training and improve the generalization ability of the model. Through random grouping, different sample combinations are obtained each time training, thereby increasing the model's diverse learning of data. This can effectively prevent the model from over-relying on a specific data distribution or sample order, thereby improving the model's adaptability to unknown data. In addition, the random grouping and averaging strategy can also reduce the local minimum problem during training, making the model more stable and easier to converge to the global optimal solution. Therefore, the random grouping and averaging strategy plays an important role in improving the robustness, generalization ability and stability of the model.

[0053] The ML-DFRBCS model combines the random sampling strategy with replacement and the random group averaging strategy to give full play to their respective advantages and improve the performance and generalization ability of the model. These two strategies complement each other in data processing and feature extraction. The random sampling strategy with replacement maintains the constancy and consistency of data when processing large-scale data sets, ensuring the efficiency and credibility of the model training and prediction process. The random group averaging strategy effectively reduces the risk of overfitting in the model training process by randomly grouping the data and performing averaging operations within each group, thereby improving the generalization ability and stability of the model.

[0054] The fourth step is to input the optimized fuzzy rules into the deep model when training the model, using a model training method that does not rely on gradient calculation. The optimized fuzzy rules are input into the deep model, given the training data set D, and other parameters, such as the number of fuzzy sets defined in each feature n, the number of times N sampling with replacement is performed r , the dimension of the feature subspace N r , the number of times N sampling without replacement is performed g , the number of WM fuzzy systems in each group N m , and the number of layers N l .

[0055] Then, on each layer In the first step, different feature combinations are generated according to the given number and dimension of features, each of which contains N s The combined information is stored in Then, based on the generated feature combinations, the input data is reorganized into This is the input dataset for this layer.

[0056] For each feature combination, use the reorganized data And the given number of fuzzy sets n constructs the WM fuzzy system Then, the model parameters of the WM fuzzy system are obtained according to the training data set The missing model parameters of the WM fuzzy system are estimated by the neighborhood averaging strategy. Next, using the reorganized data WM Fuzzy System Perform inference and obtain output Then the output of the WM fuzzy system is divided into N g groups, each containing N m outputs. The grouping information is stored in V. Next, the outputs of each group are averaged to obtain If the current layer is the last layer, the weights of different WM fuzzy system groups related to each label are calculated according to the specified formula; otherwise, the average results of all groups are concatenated as the input data of the next layer. After all layers are trained, the training results are returned.

[0057] Experimental Results

[0058] The advantages of the ML-DFRBCS model are demonstrated through more detailed experiments from various aspects, the role of the random grouping strategy is verified, and the number of fuzzy sets (n) and the deep architecture (number of sampling with replacement N) of deep fuzzy system building blocks are studied. r and system layer number N l ) The impact of the parameters involved in the proposed method on the performance. Extensive experimental analysis has been performed in this paper to fully evaluate the proposed ML-DFRBCS model, including the classification results, parameter settings, etc.

[0059] The experiments were conducted in Python 3.8.3 environment, and the computer configuration was as follows: Intel(R) Xeon(R) E-2224CPU (3.40GHz) processor, 32GB memory, and 64-bit Windows 11 system.

[0060] In order to illustrate the effectiveness of the method of the present invention and the improvement of the classification effect, tests were conducted on multiple data sets. Taking the Flags data set as an example, the test results obtained by the present invention were compared with the currently commonly used methods. The evaluation results are shown in Table 1.

[0061] method Average Precision Standard Deviation ML-DFRBCS 0.84 0.18 C2AE 0.74 0.36 ML-KNN 0.80 0.24 CC 0.80 0.23 BR 0.81 0.22 JBNN 0.80 0.23

[0062] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-label classification method based on a deep fuzzy system, characterized in that: The following steps are involved: S1. The Wang-Mendel method is used to generate fuzzy rules to capture the relationship between features and label sets. Each rule is determined by calculating the fuzzy set membership of the feature; Generate initial fuzzy rules based on input features. If fuzzy sets are defined in each feature, the feature space will be divided into a series of fuzzy region cells; let S = {s t |t∈[1,z]} is the set of cells, is the number of cells; The corresponding fuzzy rules are as follows: where x i is any instance that falls in the tth cell, p t,j Represents x i With y i,j The correlation between them, k t =(k t,1 ,k t,3 ,…,k t,d ) represents the vector consisting of the activated fuzzy set indices, and t and k t The mapping relationship between them can be determined by the following formula: Let x t represents the sample set falling in the tth cell, then p t It can be obtained from the following formula: The parameters of the cells covered by the training samples are learned through the above method; For cells where no samples fall, the neighborhood mean strategy is used for inference. During the traversal of S, it is assumed that s t is a cell that is not covered by any training sample, find s t Adjacent cell; if for any Both So called s t The neighbor set of In the above example, the corresponding values ​​of the cells covered by non-zero samples are averaged to estimate pt; S2. Construct a hierarchical deep model through the stacking generalization principle, stack the multi-label WM fuzzy system layer by layer to form a deep model structure; The structure is divided into input layer, intermediate layer and output layer; because the deep model is designed using the stack generalization principle, the output of the previous layer is used as the input of the next layer; For the first layer x i =(x i,1 , x i,2 , ..., x i,d ) represents the original input; after N r When resampling with replacement, randomly select N s indexes to generate feature subsets; the original d-dimensional feature space is divided into N s Various subspaces of dimensionality for training WM fuzzy systems where γ∈[1,N r ] and φ represent the layer index; Output express; N was performed on the WM fuzzy system index set. g sampling without replacement, each sampling randomly generates a m WM fuzzy system outputs; then to the nth group, its elements are Average; the average results of all groups are concatenated as input to the next layer; The output layer processes the average output of the WM fuzzy system group: λη,δ represents the weight associated with the ηth group output and the δth label; at the end of the output layer, λη,δ is calculated by the following formula: Among them, N l is the number of layers; S3. Use random group averaging feature transformation strategy in the middle layer; S4. Input the optimized fuzzy rules into the deep model and use a model training method that does not rely on gradient calculation; The optimized fuzzy rules are input into the deep model. Given the training dataset D and parameters, at each layer In the example, different feature combinations are generated according to the given number and dimension of features, each of which contains N s features, and the combined information is stored in middle; According to the generated feature combination, the input data is reorganized into As the input data set; then for each feature combination, use the reorganized data And the given number of fuzzy sets n constructs the WM fuzzy system Then, the model parameters of the WM fuzzy system are obtained according to the training data set The missing model parameters of the WM fuzzy system are estimated by the neighborhood averaging strategy. Using restructured data WM Fuzzy System Perform inference and obtain output The output of the WM fuzzy system is divided into N g groups, each containing N m outputs, and the grouping information is stored in V; and the output of each group is averaged to obtain If the current layer is the last layer, the weights of different WM fuzzy system groups related to each label are calculated according to the specified formula; otherwise, the average results of all groups are concatenated as the input data of the next layer; after all layers are trained, the training results are returned.

2. The multi-label classification method based on deep fuzzy system according to claim 1, characterized in that: In the step S1, before generating the initial fuzzy rules, the input features are preprocessed, including feature standardization and label processing.

3. The multi-label classification method based on deep fuzzy system according to claim 1, characterized in that: The parameters in step S4 include: the number of fuzzy sets defined in each feature, n, the number of times the sampling with replacement is performed, N r , the dimension of the feature subspace N r , the number of times N sampling without replacement is performed g , the number of WM fuzzy systems in each group N m , and the number of layers N l .