A multi-label feature selection method based on sparse learning coupled mutual information

Abstract

The application discloses a multi-label feature selection method based on sparse learning coupled mutual information, and comprises the following steps: inputting a feature matrix X, a label matrix Y and hyperparameters alpha, beta, gamma and delta, selecting a feature number k, and initializing a label correlation matrix Z and a feature correlation matrix W; calculating a diagonal matrix A and a similarity matrix S according to the feature matrix X, and calculating a graph Laplacian similarity matrix L of the feature matrix X x ; updating the label correlation matrix Z and the feature correlation matrix W through a target function, iterating n times, and obtaining an updated label correlation matrix Z n and an updated feature correlation matrix W n after the target function reaches a convergence condition; and obtaining the selected k features according to the 2-norm of W n . The application adopts the multi-label feature selection method based on sparse learning coupled mutual information, and the effectiveness of the method is proved by experiments; and the method simultaneously solves the suboptimal solution problem caused by a random initialization strategy in the sparse learning method.

Description

Technical Field

[0001] This invention relates to the field of machine learning and pattern recognition technology, and in particular to a multi-label feature selection method based on sparse learning coupled with mutual information. Background Technology

[0002] Multi-label learning is widely used in fields such as text classification, image analysis, emotion detection, and gene selection. Similar to problems in other machine learning areas, multi-label learning is also plagued by the "curse of dimensionality." Multi-label feature selection methods can eliminate irrelevant and redundant features while retaining informative features for classification, effectively addressing this problem. Among multi-label feature selection methods, sparse learning and information theory-based methods have shown significant effectiveness in multi-label applications.

[0003] However, there are three problems with both frameworks:

[0004] (I) Information theory-based methods do not fully utilize the correlations of global variables. Information theory-based methods learn labels by calculating the correlations between variables, such as the mutual information between features and each label, or the conditional mutual information between features and labels. However, these methods often only focus on low-dimensional correlations, such as the correlation between labels, labels and features, or features. In fact, high-dimensional correlations can also affect the results: for example, a set of features may be closely related to some labels, but each feature in the set may be independent of these labels. Information theory-based methods tend to eliminate features, which results in the loss of a lot of information.

[0005] (ii) Suboptimal problems caused by random initialization in sparse learning methods. Although sparse learning methods can utilize the correlation of global variables, these methods require random initialization of the given variable correlation matrix. However, different random initialization schemes not only reduce the robustness of the method but also lead to suboptimal solutions.

[0006] (iii) In sparse learning methods, the label settings are difficult to reflect the importance of the labels. The labels in multi-label datasets are logical labels and cannot reflect the importance of each label.

[0007] Previous sparse learning methods assumed that each label was of equal importance, which affected the accuracy of further evaluation of the correlation between features and labels. Therefore, such methods may not be able to obtain an optimal subset of features. Summary of the Invention

[0008] The purpose of this invention is to provide a multi-label feature selection method based on sparse learning coupled with mutual information, while partially solving the suboptimal solution problem that is commonly caused by random initialization strategies in sparse learning methods, and introducing information theory methods into sparse learning to obtain accurate feature correlation evaluation.

[0009] To achieve the above objectives, this invention provides a multi-label feature selection method based on sparse learning coupled mutual information, comprising the following steps:

[0010] S1. Input the feature matrix X, label matrix Y, and hyperparameters α, β, γ, δ. Select the number of features k. Initialize the label correlation matrix Z and feature correlation matrix W based on the input feature matrix X and label matrix Y.

[0011] S2. Calculate the diagonal matrix A and the similarity matrix S based on the characteristic matrix X, and calculate the graph Laplacian similarity matrix Lx of the characteristic matrix X;

[0012] S3. Update the label correlation matrix Z and feature correlation matrix W through the objective function. Iterate n times. After the objective function reaches the convergence condition, the updated label correlation matrix Zn and the updated feature correlation matrix Wn are obtained.

[0013] S4. Obtain the k selected features based on the 2-norm of Wn.

[0014] Preferably, in step S1, the method for initializing the label correlation matrix Z and the feature correlation matrix W based on the input feature matrix X and label matrix Y is as follows: calculate the initial label correlation matrix C and the initial feature correlation matrix D, and set Z = C and W = D, where,

[0015] C i,j =I(l i ;l j )

[0016] D i,j =I(f i ;l j )

[0017] C i，j Let D represent the element in the i-th row and j-th element of matrix C. i，j Let I represent the i-th row and j-th element in matrix D, where I is the mutual information and l is the number of elements in matrix D. i f represents the i-th label. i Let l represent the i-th feature. j f represents the j-th label. j This represents the j-th feature.

[0018] Preferably, in step S2, the formula for calculating the similarity matrix S is as follows:

[0019]

[0020] S i，j Let represent the i-th row and j-th element in matrix S, e be the natural logarithm, σ be the hyperparameter, and Np(XWj.) represent the set of p vectors in matrix XW that are closest to the j-th vector.

[0021] The formula for calculating the diagonal matrix A is:

[0022]

[0023] Where Aii is the i-th row and the i-th element;

[0024] The formula for calculating the graph Laplace similarity matrix Lx is as follows:

[0025] L x =AS.

[0026] Preferably, in step S3, updating matrices Z and W using the objective function includes:

[0027] The objective function is:

[0028]

[0029] in, All are Phoenician norms, ||W|| 2，1 It is a 2-1 norm, and Tr represents the trace of the matrix;

[0030] By differentiating the objective function under the KKT conditions, the update formula for the label correlation matrix Z is as follows:

[0031]

[0032] The update formula for the feature correlation matrix W is as follows:

[0033]

[0034] Where t represents the result of the t-th iteration, and XT, YT are the transposes of matrices X and Y.

[0035] Preferably, in step S3, after n iterations, the objective function reaches the convergence condition by: calculating the value Fn of the objective function based on Wn and Zn during the iteration process, and calculating Cver through Fn.

[0036] Cver=|Fn-Fn-1| / Fn-1

[0037] The objective function converges when Cver is less than 0.001.

[0038] Preferably, in step S4, obtaining the selected k features based on the 2-norm of Wn includes: calculating the 2-norm of Wn ||W ni. ||2, where i is the i-th row of the updated feature correlation matrix Wn, and ||W is selected. ni. The k largest features.

[0039] Therefore, the present invention employs the aforementioned multi-label feature selection method based on sparse learning coupled mutual information, and its technical effects are as follows:

[0040] (1) This invention partially solves the suboptimal solution problem that is commonly caused by random initialization strategy in sparse learning methods, and verifies the reliability of the method through ablation experiments.

[0041] (2) This invention introduces information theory methods into sparse learning to obtain accurate feature relevance evaluation: features and label matrices are reconstructed through relevance, and then the label relevance matrix and feature relevance matrix are integrated into a sparse learning framework that considers the relevance strategy of global variables.

[0042] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0043] Figure 1 This is a graph showing the Macro-F1 classification performance of the SVM classifier of this invention for each dataset;

[0044] Figure 2 This is a graph showing the Micro-F1 classification performance of the SVM classifier of this invention for each dataset;

[0045] Figure 3 The convergence curves of the three methods are shown on four benchmark datasets.

[0046] Figure 4 This is a parameter sensitivity analysis diagram of the Macro-F1 classifier of the SVM classifier of this invention on the sentiment of the dataset. Detailed Implementation

[0047] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0049] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0050] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that can be understood by those skilled in the art. These other embodiments are also covered within the scope of protection of this invention.

[0051] It should also be understood that the specific embodiments described above are only used to explain the present invention, and the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0052] Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0053] All prior art documents cited in this specification are incorporated herein by reference in their entirety and are therefore part of the disclosure of this invention.

[0054] Example 1

[0055] This invention provides a multi-label feature selection method based on sparse learning coupled mutual information, comprising the following steps:

[0056] S1. Input the feature matrix X, label matrix Y, and hyperparameters α, β, γ, δ. Select the number of features k. Initialize the label correlation matrix Z and feature correlation matrix W based on the input feature matrix X and label matrix Y.

[0057] The method for initializing the label correlation matrix Z and feature correlation matrix W based on the input feature matrix X and label matrix Y is as follows: Calculate the initial label correlation matrix C and the initial feature correlation matrix D, and set Z = C and W = D, where,

[0058] C i，j=I(l i ;l j )

[0059] D i，j =I(f i ;l j )

[0060] C i，j Let D represent the element in the i-th row and j-th element of matrix C. i，j Let I represent the i-th row and j-th element in matrix D, where I is the mutual information and l is the number of elements in matrix D. i f represents the i-th label. i Let l represent the i-th feature. j f represents the j-th label. j This represents the j-th feature.

[0061] S2. Calculate the diagonal matrix A and the similarity matrix S based on the characteristic matrix X, and calculate the graph Laplacian similarity matrix L of the characteristic matrix X. x ;

[0062] The formula for calculating the similarity matrix S is as follows:

[0063]

[0064] S i，j Let N represent the element in the i-th row and j-th column of matrix S, where e is the natural logarithm, σ is the hyperparameter, and N is the number of elements in the matrix S. p (XW j. () represents the set of p vectors in matrix XW that are closest to the j-th vector;

[0065] The formula for calculating the diagonal matrix A is:

[0066]

[0067] Among them, A ii This refers to the i-th row and the i-th element.

[0068] The Thulaplatz similarity matrix L x The calculation formula is as follows:

[0069] L x =AS.

[0070] S3. Update the label correlation matrix Z and feature correlation matrix W using the objective function. Iterate n times until the objective function reaches the convergence condition, then obtain the updated label correlation matrix Z. n and the updated feature correlation matrix W n ;

[0071] The objective function is:

[0072]

[0073] in, All are Phoenician norms, ||W|| 2，1 It is a 2-1 norm, and Tr represents the trace of the matrix;

[0074] By differentiating the objective function under the KKT conditions, the update formula for the label correlation matrix Z is as follows:

[0075]

[0076] The update formula for the feature correlation matrix W is as follows:

[0077]

[0078] Where t represents the result of the t-th iteration, X T Y T Let X be the transpose of matrices X and Y.

[0079] After n iterations, the objective function reaches convergence under the following conditions: based on W during the iteration process. n Z n Calculate the value of the objective function F n via F n Calculate Cver:

[0080] Cver=|Fn-Fn-1| / Fn-1

[0081] The objective function converges when Cver is less than 0.001.

[0082] S4, according to W n The 2-norm yields the k selected features.

[0083] Calculate the 2-norm ||Wn ni. ||2, where i is the updated feature correlation matrix W n In the i-th row, select ||W ni. The k largest features.

[0084] The method proposed in this invention will be verified through four sets of experiments.

[0085] The data from the four sets of experiments were from the Emotions dataset, Yeast dataset, Flags dataset, and Medical dataset.

[0086] The Emotions dataset comes from the music field, containing 72 musical features from 593 songs, categorized into one or more of six emotion types. The Yeast dataset comes from the biology field, consisting of microarray expression data and phylogenetic spectra; the training set contains 1500 genes, and the test set contains 917 genes. The input dimension is 103. The Flags dataset comes from image regions; the original data can be found in the UCI database. Finally, the Medical dataset was extracted from a large number of suicide letters obtained after natural language processing of clinical free text. Information on all datasets is summarized in Table 1.

[0087] Table 1 Information about the experimental dataset

[0088] Datasets Training Test Instances Features Labels Domains Flags 129 65 194 19 7 Images Emotions 391 202 593 72 Music Medical 333 645 978 1449 45 Text Yeast 1500 917 2417 103 14 Biology

[0089] SLcMI is a sparse learning method coupled with information theory, primarily investigating the suboptimal problem of the objective function of sparse learning methods caused by variable-related stochastic initialization strategies. Therefore, we compare SLcMI with two representative sparse learning methods (RALM-FS and SSFS) and three information theory-based methods (D2F, SCLS, and FSSL) on two popular evaluation metrics, macro-F1 and micro-F1. Both are performed on linear SVM classifiers. Furthermore, convergence analysis and parameter sensitivity studies are conducted.

[0090] In the ablation experiments, the variables for SLcMI were randomly initialized five times. The parameters of the SLcMI method and other sparse learning methods (RALM-FS and SSFS) were tuned in a grid of [0.01, 0.01.1, 0.3, 0.5, 0.5, 0.7, 0.9, 1.0].

[0091] First, the average results of SLcMI are presented, and compared with the aforementioned method on four datasets in Tables 2 and 3. The top 20% of all features were selected from Emotions and Yeast, while 17% was retained from the Medical dataset because it has more features. The Flags dataset has only 19 features; therefore, all classification results for Flags were calculated.

[0092] As shown in Tables 2 and 3, the SLcMI method achieves the best performance on all four datasets. Furthermore, it demonstrates a significant advantage in average results compared to other methods. Additionally, according to Tables 2 and 3, the average classification results of the three information theory-based methods outperform the sparse learning method RALM-FS, and they outperform other sparse learning methods such as SSFS. This indicates that both information theory-based and sparse learning frameworks have their own advantages, thus necessitating the design of methods based on both sparse learning and information theory frameworks.

[0093] Table 2 shows the Macro-F1 results of the six methods on the SVM classifier.

[0094] Datasets SLcMI D2F SCLS RALM-FS FSSL SSFS Flags 0.5904±0.0808 0.5444±0.0528 0.5388±0.0431 0.4872±0.0448 0.506±0.0525 0.4987±0.0494 Emotions 0.355±0.0841 0.0882±0.0677 0.2562±0.1672 0.0051±0.0191 0.0346±0.0288 0.2521±0.0726 Medical 0.3067±0.0934 0.1912±0.0547 0.0793±0.0128 0.1288±0.0631 0.2269±0.086 0.2306±0.065 Yeast 0.2523±0.0433 0.2368±0.0497 0.2031±0.0273 0.2397±0.0426 0.1328±0.0156 0.2084±0.0449 Average 0.3761 0.26515 0.26935 0.2152 0.225075 0.29745

[0095] Note: The last line represents the average classification result of the four datasets.

[0096] Table 3. Results of six methods on the Micro-F1 classifier using SVM.

[0097] Datasets SLcMI D2F SCLS RALM-FS FSSL SSFS Flags 0.7279±0.0417 0.6787±0.0413 0.6746±0.0319 0.6315±0.0494 0.6477±0.0338 0.652±0.0349 Emotions 0.4253±0.0911 0.1437±0.1267 0.3119±0.2027 0.0082 ± 0.0307 0.0627±0.0533 0.4215±0.1213 Medical 0.714±0.093 0.6293±0.0704 0.3697±0.0095 0.3632±0.1477 0.6609±0.1684 0.6936±0.0779 Yeast 0.5627±0.0329 0.5525±0.0356 0.5331±0.0194 0.559±0.0295 0.4849±0.0099 0.5395±0.0341 Average 0.607475 0.50105 0.472325 0.390475 0.46405 0.57665

[0098] Note: The last line represents the average classification result of the four datasets.

[0099] Furthermore, in order to clearly demonstrate the classification performance of the proposed method and compare it with other methods, Figure 1 and Figure 2 The classification performance of all methods on Macro-F1 and Micro-F1 is shown. The x-axis represents the number of selected features, and the y-axis represents the value of the evaluation criterion. Figure 1 In the datasets Emotions, Medical, and Yeast, all methods showed improved classification performance, except for the "flag" feature. This is because the Flags dataset only has 19 features, and when the number of features is smaller, each feature has a greater impact. Figure 2 A similar situation has occurred in China.

[0100] Ablation experiment:

[0101] In this section, we conduct ablation experiments to verify the importance of the correlations of the initialization variables, namely W and Z. The ablation experiments were implemented on the same four datasets collected by Yahoo (Emotions, Flags, Medical, and Yeast) and two other large datasets (Society and Social). Both large datasets consist of 5000 instances, with 2000 instances for training and 3000 instances for testing. The Society dataset has 636 features and 27 labels, while Social has 1047 features and 39 labels. Table 4 shows a comparison between our proposed method and one based on a random initialization strategy. Our strategy achieves state-of-the-art performance on all datasets. These results further demonstrate the rationale for the proposed method.

[0102] Table 4 shows the macro-f1 ablation experiments on the SVM classifier (the baseline represents the variable-related random initialization strategy).

[0103] Datasets SLcMI Baseline1 Baseline2 Baseline3 Baseline4 Baseline5 Flags 0.5904 0.5841 0.5858 0.5869 0.5722 0.5834 Emotions 0.3550 0.3374 0.3505 0.3479 0.3359 0.3453 Medical 0.3067 0.2969 0.2354 0.2061 0.2069 0.2090 Yeast 0.2523 0.2026 0.2090 0.2055 0.2226 0.2042 Society 0.0634 0.0554 0.0651 0.0208 0.0202 0.0618 Social 0.0886 0.0829 0.0705 0.0646 0.0673 0.0786

[0104] Convergence and Time Complexity Analysis

[0105] Furthermore, the convergence experiments on the four datasets (Flags, Emotions, Medical, and Yeast) are shown in the figure. Figure 3 (a), (b), and (c) show the convergence curves of the three sparse learning methods, respectively:

[0106] SLcMI converges immediately on the Flags, Emotions, and Medical datasets, while converging after approximately 7 iterations on the Yeast dataset. In comparison, RALM-FS fluctuates until convergence on the Yeast and Emotions datasets, while converging quickly on the Flags and Medical datasets. In the SSFS convergence curve, it converges after approximately 22 iterations on the Flags dataset, while converging initially on the other datasets. Therefore, SLcMI converges significantly faster than the other two sparse learning methods.

[0107] Then, the time complexity of methods such as SLcMI (D2F, SCLS, RALM-FS, FSSL, and SSFS) was calculated. Let 'a' represent the number of selected features, 'd' represent the number of features, 'n' represent the number of instances, and 'l' represent the number of labels. The time complexities of these six methods are shown in Table 5. SCLS has the lowest time complexity because its main purpose is to reduce runtime; however, its classification performance lags far behind the method of this invention. The time complexities of the other five methods are all on the same order of magnitude.

[0108] Table 5 Time Complexity of Six Methods

[0109] SLcMI <![CDATA[O(d 2 l+l 2 n)]]> SSFS <![CDATA[O(nda+n 2 a)]]> SCLS O(dl + ad) RALM-FS <![CDATA[O(d 2 n+n 2 d)]]> FSSL O(adl) D2F O(ndc + adnl)

[0110] Parameter sensitivity study

[0111] The SLcMI method employs four parameters: α, β, γ, and δ. α and β control the contributions of the label correlation matrix and feature correlation matrix, respectively. γ controls the constraints regarding the instance-level manifold, while δ controls the sparsity of the SLcMI model. Figure 4 The Macro-F1 score represents the parameter sensitivity of SLcMI to the sentiment of the dataset. For example... Figure 4 As shown, classification performance is not sensitive to these four parameters. Therefore, the parameters can be safely chosen within a wide range.

[0112] Therefore, this invention adopts the above-mentioned multi-label feature selection method based on sparse learning coupled mutual information. Experiments have proven the effectiveness of the method. This method also partially solves the suboptimal solution problem that is commonly caused by random initialization strategies in sparse learning methods, and the reliability of the method is verified by ablation experiments. This method introduces information theory into sparse learning to obtain accurate feature relevance evaluation: features and label matrices are reconstructed through relevance, and then the label relevance matrix and feature relevance matrix are integrated into a sparse learning framework that considers the relevance strategy of global variables.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-label feature selection method based on sparse learning coupled mutual information, applied to image analysis, characterized in that, Comprising the following steps: S1, input a feature matrix X, a label matrix Y and a hyperparameter , select a feature number k, initialize a label correlation matrix Z and a feature correlation matrix W according to the input feature matrix X and the label matrix Y; S2. computing a diagonal matrix A and a similarity matrix S from the feature matrix X, and computing a graph Laplacian similarity matrix L of the feature matrix X x ; S3, updating the label correlation matrix Z and the feature correlation matrix W through an objective function, iterating n times, and obtaining the updated label correlation matrix Z and the updated feature correlation matrix W when the objective function reaches a convergence condition n . n ; S4. According to W n The 2-norm of the selected k features is obtained. S5, the above method is verified by experiment, and the test data is derived from the Flags data set, the Flags data set is from an image region, and the original data can be found in the UCI database; In step S1, the method for initializing the label correlation matrix Z and the feature correlation matrix W according to the input feature matrix X and the label matrix Y is: calculating an initial label correlation matrix C and an initial feature correlation matrix D, and making Z=C and W=D, wherein, ； ； Represents the first element in matrix C. line, number One element, Represents the first in matrix D line, number One element, For mutual information, Indicates the first One tag, Indicates the first One characteristic, Indicates the first One tag, Indicates the first One feature; In step S2, the calculation formula of the similarity matrix S is as follows: ； representing matrix S the middle row, the middle element, is the natural logarithm, is a hyperparameter, N p (XW j. ) represents a set of the N closest vectors to the middle vector in the matrix XW; and p vector in the matrix XW. The calculation formula of the diagonal matrix A is as follows: ； wherein is the first i row, i-th element; The Laplacian similarity matrix L x The calculation formula is as follows: ； In step S3, updating the matrix Z and W through the objective function includes: The objective function is: ； wherein , , are the Frobenius norms, is the 2-1 norm and Tr denotes the trace of a matrix; By deriving the objective function under the KKT condition, the update formula of the label correlation matrix Z is as follows: ； The update formula of the feature correlation matrix W is as follows: ； where t represents the result of the tth iteration, X T , Y T is the transpose of the matrix X, Y. In step S3, after n iterations, the objective function reaches the convergence condition by including: based on W during the iteration process. n Z n Calculate the value of the objective function F n via F n Calculate Cver: ； When Cver is less than 0.001, the objective function converges.

2. The multi-label feature selection method based on sparse learning coupled mutual information according to claim 1, characterized in that, In step S4, the selected features are obtained according to the 2-norm of W n The k features are selected including: calculating the 2-norm of W n wherein, is the i-th row of the updated feature correlation matrix W n the largest k features.​​​​

Images