Fuzzy multi-label classification method for long-tail distribution

Abstract

The invention relates to the field of machine learning and data mining, in particular to a long-tail multi-label classification method based on a fuzzy rule. The method comprises a training stage and a using stage. The training stage comprises feature transformation based on the fuzzy rule, mark balance learning based on the fuzzy rule and mark coupling learning based on the fuzzy rule. According to the feature transformation based on the fuzzy rule, original features of multiple marks are mapped to a high-dimensional fuzzy feature space, and rich implicit knowledge is provided for subsequently constructing the mapping relation between the fuzzy features and the marks (especially tail marks). When a mapping function (namely a loss function) is constructed for a fuzzy feature space and a mark space, mark balance learning based on a fuzzy rule can adaptively weight learning processes of different marks so as to reduce decision deviation of a tail mark. The fuzzy rule-based label coupling learning analyzes the co-occurrence relationship between any two labels by constructing coupling for the two labels.

Description

Technical Field

[0001] The present invention relates to the fields of machine learning and data mining, and in particular to a long-tail multi-label classification method based on fuzzy rules. Technical Background

[0002] The sparsity of tags usually makes multi-label datasets obey long-tail distribution. That is, the frequency distribution obtained by arranging the tags in descending order of frequency of occurrence is long-tailed, in which a large number of tags are only related to very few samples (also called tail tags), and only a small number of tags are related to a large number of samples (also called head tags). In many practical application scenarios, the problem of long-tail distribution is particularly prominent. For example: in the Internet of Vehicles, the tags of traffic conditions, accident types and road conditions often have long-tail distribution characteristics. In most cases, there are more samples of common traffic conditions and accident types (head tags), while rare traffic conditions and special accident types have few samples (tail tags). In bioinformatics, some rare diseases or gene mutations are also typical examples of long-tail distribution. In the field of emotion perception, some subtle or uncommon emotional states also have similar problems.

[0003] Compared with the head tag, the tail tag has insufficient relevant samples, which causes the traditional model to have a large decision deviation for the tail tag. At present, the methods that can handle the long-tail distribution problem in the multi-label scenario can be roughly summarized into three directions: namely, appropriate sampling strategies, knowledge transfer, and the construction of complex (synthetic) objective functions. For the appropriate sampling strategy, the current solution in this direction needs to rely on the traditional single-label-based sampling strategy. For example, the long-tail multi-label classification task is split into multiple unbalanced binary classification tasks. In each binary classification task, the method performs multiple under-sampling learning on irrelevant samples and generates multiple decision boundaries, and then uses the fusion strategy to integrate multiple strategy boundaries into a comprehensive boundary. For knowledge transfer, this direction can currently be summarized into two ideas: one is to achieve intra-label transfer. That is, the multi-label long-tail distribution problem is split into multiple independent single-label category imbalance problems to achieve knowledge transfer from large categories to small categories. The other is to achieve inter-label transfer. This idea is specially designed for the multi-label long-tail distribution problem, and uses knowledge transfer to transfer the knowledge of the head tag to the tail tag. For constructing complex (synthetic) objective functions, this direction directly constructs suitable objective functions for multi-label long-tail distribution problems.

[0004] The above three directions can solve the long-tail distribution problem in multi-label scenarios, but there are almost no methods that can focus on the following three problems at the same time: 1) How to enrich the learnable knowledge of tail labels. 2) How to reduce the decision bias of tail labels. 3) How to use the relationship between labels to improve prediction performance. Therefore, it is urgent to develop an effective long-tail multi-label learning method to take into account the above three challenges. Summary of the invention

[0005] The purpose of the present invention is to provide a long-tail multi-label classification method based on fuzzy rules applicable to various fields, aiming to solve the problems of scarce learnable knowledge, large decision-making bias and underutilized label relationships caused by the scarcity of tail labels in multi-label data sets in different fields. By introducing fuzzy rules and related learning mechanisms, the present invention can effectively mine potential information in data, reduce decision-making bias caused by long-tail distribution, and improve prediction performance by utilizing co-occurrence relationships between labels. This method has broad application prospects in practical application scenarios such as bioinformatics and emotion perception.

[0006] The technical solution of the present invention is as follows:

[0007] A fuzzy multi-label classification method for long-tail distribution, the steps are as follows:

[0008] The steps of the training phase are:

[0009] Step 1: Feature transformation based on fuzzy rules. The specific process is as follows:

[0010] For a multi-output Takagi-Sugeno-Kang fuzzy system with K inference rules, its kth inference rule can be expressed as follows:

[0011]

[0012] where x j (1≤j≤D) represents the jth feature of instance x, D represents the feature dimension, represents the antecedent fuzzy set of the j-th feature of instance x in the k-th rule, is the consequent parameter of the jth feature corresponding to the lth output in the kth rule, is the lth output of instance x on the kth rule.

[0013] The antecedent fuzzy set in formula (1) It can be described by the Gaussian membership function, namely:

[0014]

[0015] in and It can be obtained by Fuzzy C-Means (FCM) clustering method:

[0016]

[0017] where u ik It represents the membership value of the i-th instance on the k-th inference rule, which can be obtained by FCM calculation.

[0018] ∈ is a hyperparameter used to adjust

[0019] According to the above analysis, the ignition level of instance x on the kth rule can be expressed as:

[0020]

[0021] Wherein formula (6) is the normalized form of formula (5). Finally, the l-th output of instance x in the multi-output TSK FS is expressed as:

[0022]

[0023] Furthermore, formula (7) can be re-expressed as a linear model of instance x in the fuzzy feature space:

[0024]

[0025] in,

[0026]

[0027] Then, for the i-th instance x in the multi-label data i (1≤i≤N), under the premise that the number of fuzzy rules is set to K, its corresponding fuzzy feature vector in the fuzzy feature space is The specific mapping process can refer to formulas (2)-(6) and (9)-(11). In other words,

[0028] φ1(x i )=x g,i (12)

[0029] Correspondingly, for the input matrix X of the multi-label dataset, N ], it can get the following fuzzy feature matrix through φ1:

[0030]

[0031] Compared with the original features, the fuzzy features in the high-dimensional fuzzy feature space have an inherent advantage. That is, the fuzzy features have stronger expressive power for the instances. This means that the tail tag can learn richer implicit knowledge from the fuzzy features.

[0032] Step 2: Label balance learning based on fuzzy rules. The specific process is as follows:

[0033] For the jth label vector (i.e., the j-th row element in Y), its corresponding output in the multi-output Takagi-Sugeno-Kang fuzzy system is linearly expressed as:

[0034]

[0035] in corresponds to the consequent parameter vector of the jth label in the multi-output Takagi-Sugeno-Kang fuzzy system. Then, c j The optimal solution of can be obtained by the least squares method, that is:

[0036]

[0037] Formula (15) is the loss function of linear regression between the feature and the jth label.

[0038] Weighting the learning process of the jth tag in formula (15) is the key to balancing the decision bias of the tail tag. The present invention adopts a gentle and flexible way to perform weighting. That is, the learning weight q of the jth tag is obtained without artificially separating the head tag and the tail tag. j (1≤j≤L). Specifically,

[0039]

[0040] in,

[0041] p j =o j / m (17)

[0042] o j =|{n|y nj =1,1≤n≤N}| (18)

[0043]

[0044] Among them j represents the frequency of occurrence of the jth tag (i.e., the relevant sample size). Formulas (16)-(19) show that the smaller the frequency of occurrence of the jth tag, the larger the corresponding weight q j (q j >0). Therefore, formula (15) can be improved as follows:

[0045]

[0046] Compared with formula (15), formula (20) is more suitable for multi-label learning scenarios with long-tail distribution. The main reason is that q j The introduction of enables the present invention to alleviate to a certain extent the problem that tail marking cannot be fully learned due to the small amount of relevant samples, which is conducive to reducing the decision bias of tail marking.

[0047] Based on the above analysis, the label balance learning based on fuzzy rules can be expressed as:

[0048]

[0049]

[0050] in and Furthermore, we add a regularization term to formula (21), that is,

[0051]

[0052] where β>0 is a hyperparameter used to balance the impact of fuzzy rule-based labeling and model complexity.

[0053] Step 3: Label coupling learning based on fuzzy rules. The specific process is as follows:

[0054] First, the present invention uses the Pearson correlation coefficient to measure and The co-occurrence relationship between ij . Generally, -1≤u ij ≤1. When the marker vector and When there is a positive correlation between ij >0, otherwise, u ij < 0. Mark vector and The stronger the correlation between ij The larger the absolute value of .

[0055] Secondly, in the multi-output Takagi-Sugeno-Kang fuzzy system, the fuzzy consequent vector c i and c j Can and Screening discriminant features. This means that c i and c j The correlation between them should be the same as the label vector and The coupling relationship between them is consistent.

[0056] Combined with the above analysis, the labeled coupling learning based on fuzzy rules can further optimize the consequent parameters of the multi-output Takagi-Sugeno-Kang fuzzy system. The specific formula is defined as:

[0057]

[0058] in And a ij =1-u ij .

[0059] Step 4: By integrating the above three steps, a complete long-tail multi-label learning objective function based on fuzzy rules is realized. In the application scenario where multiple labels present long-tail distribution characteristics, multiple label results can still be effectively predicted.

[0060] The usage phase steps are:

[0061] Step 5: Use the antecedent part of the multi-output Takagi-Sugeno-Kang fuzzy system to convert the multi-label test data with long-tail distribution characteristics into fuzzy feature data.

[0062] Step 6: Perform linear regression on the fuzzy feature data and the optimized multi-output Takagi-Sugeno-Kang fuzzy system consequent parameters, and implement long-tail multi-label prediction in the application scenario based on the threshold function.

[0063] The advantages of the present invention include the following points:

[0064] 1) Through feature transformation based on fuzzy rules, the present invention uses the antecedent part of the fuzzy rules to map the original features into the fuzzy feature space, thereby mining rich potential knowledge for the learning of tail tags.

[0065] 2) Through label balance learning based on fuzzy rules, the present invention introduces adaptive weighting when using the consequent part of the fuzzy rule to construct a mapping relationship (i.e., loss function) between the fuzzy feature space and the label space, thereby alleviating the problem of large deviation in tail label decision.

[0066] 3) Through fuzzy rule-based label coupling learning, the present invention establishes coupling for any two labels in the label space to analyze their co-occurrence relationship, thereby utilizing the label co-occurrence relationship to further improve the prediction performance of long-tail multi-label learning.

[0067] 4) A large number of experimental analyses have demonstrated the effectiveness of the present invention in dealing with long-tail multi-label learning problems in various fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is the overall framework diagram of the algorithm of the present invention;

[0069] FIG2( a ) is a Bibtex dataset analysis diagram of the effectiveness of labeled balanced learning of the present invention;

[0070] FIG2( b ) is a diagram of the analysis of the CAL500 dataset for the effectiveness of labeled balanced learning of the present invention;

[0071] FIG2( c ) is a Corel16k1 dataset analysis diagram of the effectiveness of label-balanced learning of the present invention;

[0072] FIG2( d ) is an analysis diagram of the Emotions dataset of the effectiveness of labeled balanced learning of the present invention;

[0073] FIG2( e ) is a diagram of the analysis of the Flags dataset of the effectiveness of the labeled balanced learning of the present invention;

[0074] FIG2( f ) is a Mirflickr dataset analysis diagram of the effectiveness of label-balanced learning of the present invention;

[0075] FIG2( g ) is an analysis diagram of the Rcv1s1 dataset for the effectiveness of label-balanced learning of the present invention;

[0076] FIG2(h) is a Yeast dataset analysis diagram of the effectiveness of labeled balanced learning of the present invention;

[0077] Figure 3 This is a fuzzy rule-based label coupling learning effectiveness analysis diagram of the present invention. DETAILED DESCRIPTION

[0078] The present invention is described in detail below with reference to the accompanying drawings and embodiments:

[0079] Example 1

[0080] According to the implementation method of the training phase, the present invention uses eight data sets from different fields to verify the effectiveness of the present invention for long-tail multi-label learning. The specific information of the eight data sets is shown in Table 1.

[0081] Table 1: Detailed information of multi-label datasets in different fields

[0082] Dataset field Number of samples Feature Dimension Label Dimensions Bibtex text 7395 1836 159 CAL500 music 502 68 174 Corel16k1 image 13766 500 153 Emotions music 593 72 6 Flags image 194 19 7 Mirflickr image 25000 1000 38 Rcv1s1 text 6000 944 101 Yeast biology 2417 103 14

[0083] The present invention uses four commonly used multi-label learning indicators: average precision (AP), Hamming loss (HL) and ranking loss (RL). The present invention is compared with six methods on eight data sets, and each method adopts a five-fold cross-validation strategy. The experimental results are shown in Tables 2-9.

[0084] As can be seen from the table, compared with the six comparison methods, the overall performance of the present invention is excellent on eight data sets. The main reason is that when modeling long-tail multi-label learning, the present invention not only uses fuzzy rules to effectively mine the hidden information in long-tail multi-label data, but also reduces the decision bias of the model for long-tail labels through label balance learning, and deeply analyzes the co-occurrence between different labels through label coupling learning, thereby further improving the prediction performance of the model.

[0085] Table 2: Experimental results of different methods on the Bibtex dataset (AP-HL-RL):

[0086]

[0087] Table 3: Experimental results of different methods on the CAL500 dataset (AP-HL-RL):

[0088]

[0089]

[0090] Table 4: Experimental results of different methods on the Corel16k1 dataset (AP-HL-RL):

[0091]

[0092] Table 5: Experimental results of different methods on the Emotions dataset (AP-HL-RL):

[0093]

[0094] Table 6: Experimental results of different methods on the Flags dataset (AP-HL-RL):

[0095]

[0096] Table 7: Experimental results of different methods on the Mirflickr dataset (AP-HL-RL):

[0097]

[0098] Table 8: Experimental results of different methods on the Rcv1s1 dataset (AP-HL-RL):

[0099]

[0100] Table 9: Experimental results of different methods on the Yeast dataset (AP-HL-RL):

[0101]

[0102] Example 2

[0103] There are various weight definitions that can be used for labeling balance learning in the present invention. In addition to the definition in formula (16) In addition to the type, it can also be defined as other function types, such as q j =1-p j +ε(ε is a minimum value, ensuring q j >0), or The present invention mainly analyzes the performance differences caused by the above four weight definition methods. In addition, in order to analyze the effectiveness of label balance learning, the present invention also adds qj = 1, that is, the learning weights of all tags are 1. Figure 2 shows the experimental results, in which the best performance is bolded. The experimental results in Figure 2 show that j =1, the labeling balance learning based on fuzzy rules can improve the learning performance of the model; and overall, The performance is better.

[0104] Example 3

[0105] In order to verify the effectiveness of the label coupling learning mechanism based on fuzzy rules, the present invention conducts two groups of comparative experiments using the RL indicator as an example. The smaller the value of the indicator, the better the classification performance of the model. Specifically, the parameter α can adjust the impact of the label coupling learning based on fuzzy rules on the present invention. Group A sets α to a suitable positive value, so that the present invention can obtain the best performance after parameter optimization. Group B sets α to 0, and the other parameters are the same as those of Group A. In other words, the label coupling learning based on fuzzy rules is effective in Group A and ineffective in Group B. Figure 3 The comparison results of Group A and Group B on 8 data sets are given. The experimental results prove that the label coupling learning based on fuzzy rules can effectively improve the prediction performance of the present invention.

Claims

1. A fuzzy multi-label classification method for long-tail distribution, characterized in that: Here are the steps: The steps of the training phase are: Step 1: Feature transformation based on fuzzy rules; construct a multi-output Takagi-Sugeno-Kang fuzzy system with K rules and calculate the feature set The membership degree of K fuzzy rules is obtained to obtain fuzzy feature data Step 2: Marking balance learning based on fuzzy rules; A linear regression loss function is established between features and tags; and the learning weight q of the jth tag is obtained without artificially separating the head tag and the tail tag. j (1≤j≤L); the smaller the frequency of the jth tag, the corresponding weight q j (q j >0), the larger the value, the opposite is true; Step 3: Label coupling learning based on fuzzy rules; The Pearson correlation coefficient is used to measure the label space and Mark The co-occurrence relationship between them is maintained, and the fuzzy rule consequent parameter c to be optimized is maintained. i and c j The relationship between the labeled vector and The coupling relationship between them is consistent; Step 4: By integrating the above three steps, a complete long-tail multi-label learning objective function based on fuzzy rules is realized; in application scenarios where multiple labels present long-tail distribution characteristics, multiple label results can still be effectively predicted.

2. A fuzzy multi-label classification method for long-tail distribution as claimed in claim 1, characterized in that: The second step is specifically performed as follows: For the jth label vector The corresponding output in the multi-output Takagi-Sugeno-Kang fuzzy system is linearly expressed as: in The consequent parameter vector corresponding to the jth label in the multi-output Takagi-Sugeno-Kang fuzzy system; c j The optimal solution of is obtained by the least squares method, namely: Formula (15) is the loss function of linear regression between the feature and the jth tag; Get the learning weight q of the jth tag without manually separating the head tag and the tail tag j (1≤j≤L); specifically, in, p j =o j / m (17) either j =|{n|y nj =1,1≤n≤N}| (18) Among them j represents the frequency of occurrence of the jth tag. Formulas (16)-(19) show that the smaller the frequency of occurrence of the jth tag, the corresponding weight q j (q j >0) is larger; formula (15) is improved as follows: Based on the above analysis, the labeling balance learning based on fuzzy rules is expressed as: in and Add a regularization term to formula (21), that is, where β>0 is a hyperparameter used to balance the impact of fuzzy rule-based labeling and model complexity.

3. A fuzzy multi-label classification method for long-tail distribution according to claim 1 or 2, characterized in that: The third step is specifically performed as follows: The Pearson correlation coefficient is used to measure the and The co-occurrence relationship between ij ; -1≤u ij ≤1; When the marker vector and When there is a positive correlation between ij >0, otherwise, u ij <0; label vector and The stronger the correlation between ij The larger the absolute value of Secondly, in the multi-output Takagi-Sugeno-Kang fuzzy system, the fuzzy consequent vector c i and c j Can and Screening discriminant features; Optimize the consequent parameters of the multi-output Takagi-Sugeno-Kang fuzzy system. The specific formula is defined as: in And a ij =1-u ij .

4. A fuzzy multi-label classification method for long-tail distribution as claimed in claim 1 or 2, characterized in that: The usage phase steps are: Step 5: Use the antecedent part of the multi-output Takagi-Sugeno-Kang fuzzy system to convert the multi-label test data with long-tail distribution characteristics into fuzzy feature data; Step 6: Perform linear regression on the fuzzy feature data and the optimized multi-output Takagi-Sugeno-Kang fuzzy system consequent parameters, and implement long-tail multi-label prediction in the application scenario based on the threshold function.

5. A fuzzy multi-label classification method for long-tail distribution as claimed in claim 3, characterized in that: The usage phase steps are: Step 5: Use the antecedent part of the multi-output Takagi-Sugeno-Kang fuzzy system to convert the multi-label test data with long-tail distribution characteristics into fuzzy feature data; Step 6: Perform linear regression on the fuzzy feature data and the optimized multi-output Takagi-Sugeno-Kang fuzzy system consequent parameters, and implement long-tail multi-label prediction in the application scenario based on the threshold function.

Images