Hierarchical feature selection method and system based on label correlation and instance correlation

Abstract

The invention provides a hierarchical feature selection method and system based on label correlation and instance correlation, a feature selection task is decomposed into a plurality of subtasks, and a feature subset is selected for each subtask to improve classification accuracy, and the method specifically comprises the following steps: S1, inputting a hierarchical training data set; s2, using a hierarchical tree structure to describe a semantic relationship between categories in a hierarchical training data set, wherein each internal node corresponds to a subtask; s3, using a loss item and regularization items including sparse learning, label correlation and instance correlation to construct a target function of subtasks corresponding to the internal nodes, optimizing a weight matrix of each internal node according to the target function, sorting the weight matrixes of the subtasks in a descending order, selecting features with large weight values, and selecting features with large weight values; and completing a hierarchical feature selection task. According to the method, not only is the hierarchical relationship between the categories considered, but also the similarity between the instances is utilized, so that the features can be selected more effectively, and the classification performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning and data processing, and in particular to a hierarchical feature selection method and system based on label correlation and instance correlation. Background Art

[0002] In the era of big data, the number of categories in classification tasks is growing rapidly, from a few or dozens of categories in the past to tens of thousands of categories today. These categories are organized into hierarchical structures, such as tree structures and directed acyclic graphs, based on the semantic information between categories to solve classification learning problems with a large number of categories. Compared with traditional classification learning that divides categories into flat sets, hierarchical classification learning can decompose classification tasks into multiple subtasks. First, the abstract coarse-grained categories of instances are identified, and then they are refined layer by layer to specific fine-grained categories. This approach significantly reduces the difficulty of classification.

[0003] In recent years, hierarchical classification learning has been widely used in many research fields, including image classification, text classification, disease diagnosis, and gene identification. In hierarchical classification learning, as the number of categories increases dramatically, the number of features describing the classification task also grows rapidly, which leads to the problem of dimensionality curse. Feature selection has received widespread attention as an effective dimensionality reduction method. However, traditional feature selection methods usually select a common feature subset to distinguish all categories, resulting in a large subset of selected features, and some features may only be useful for distinguishing individual categories. These methods assume that the categories are independent, but ignore the possible hierarchical structure between them.

[0004] Inspired by the divide-and-conquer strategy, the researchers proposed a new hierarchical feature selection method: selecting respective feature subsets for different subtasks to effectively reduce the number of features. In addition, between categories in the hierarchy, such as "mobile phones" and "tablets" under "electronic products", there is a high degree of correlation between them and they share many features. In medical image data, the same disease may exhibit a variety of different features and pathological morphologies, which are divided into different clusters in the feature space. By considering instance correlation, the medical image classification system can more accurately classify image instances with similar features into the same category, thereby improving classification accuracy. Therefore, the correlation in the label space and the similarity in the instance space can be used as auxiliary information, enabling the hierarchical feature selection method to achieve better results in improving classification performance.

[0005] Although the current research on label and instance correlation has made some progress in hierarchical feature selection, there are still several significant deficiencies. In the label space, existing hierarchical feature selection methods mainly focus on the connection between parent and child classes and sibling classes. Although these methods have exploited the correlation between categories to a certain extent, they ignore the connection between broader categories. They fail to fully utilize the rich semantic information embedded in the hierarchy to consider the correlation of each category with other categories to varying degrees. At the same time, it is also particularly important to pay attention to the correlation between features and labels under local subtasks. On the one hand, this reduces the complexity of the model. On the other hand, it can highlight the differences between different categories within each subtask and explore the small but critical features in the instance for the classification subtask in more depth, thereby obtaining a compact and powerful classification model. However, existing hierarchical feature selection methods lack sufficient consideration of both label correlation and instance correlation. In addition, existing methods mainly evaluate instance correlation through mechanisms such as cosine similarity, which cannot effectively handle noisy data and redundant features in the instance space.

[0006] To solve the above problems, we propose a hierarchical feature selection method based on label correlation and instance correlation (HFS-LCIC) and system; the information of hierarchical structure and instance distribution is transformed into an effective regularization term to improve classification performance. Summary of the invention

[0007] The purpose of the present invention is to propose a hierarchical feature selection method and system based on label relevance and instance relevance; under the global structure, the label relevance constraint is constructed by using the path distance between internal categories in the hierarchical tree structure; under the local subtask, the instance relevance constraint is constructed by the k-nearest neighbor mechanism optimized by the Gaussian kernel function; not only the hierarchical relationship between categories is considered, but also the similarity between instances is utilized, so that features can be selected more effectively and the classification performance can be improved.

[0008] To achieve the above object, the technical solution of the present invention is as follows: Based on the hierarchical feature selection method of label relevance and instance relevance, the feature selection task is decomposed into several subtasks, and a feature subset is selected for each subtask to improve the classification accuracy. The specific steps include: S1, input hierarchical training data set, including an instance matrix containing n instances and m features , and a label matrix containing n instances and a label , and convert the single-label matrix into a multi-label matrix; S2, use a hierarchical tree structure to describe the semantic relationship between categories in the hierarchical training dataset. The number of internal nodes in the hierarchical tree structure is set to N+1; each internal node corresponds to a subtask; S3. Construct loss terms for internal nodes, and use sparse learning, label relevance, and instance relevance as regularization terms. Use loss terms and regularization terms including sparse learning, label relevance, and instance relevance to construct the objective function of the subtask corresponding to the internal node. Optimize the weight matrix of each internal node according to the objective function, sort the weight matrix of each subtask in descending order, select features with large weight values, and complete the hierarchical feature selection task.

[0009] Preferably, the conversion of the single label matrix into a multi-label matrix is ​​specifically as follows: Convert to a multi-label matrix via binary association ,in Representation instance j Tags, d Indicates the number of categories.

[0010] Preferably, the use of a hierarchical tree structure to describe the semantic relationship between categories in a hierarchical training data set is specifically: the instance matrix X According to the internal nodes, it is divided into ,in Represents and internal nodes i The corresponding instance matrix, n i Represents and internal nodes i The number of corresponding instances; X 0 is the instance matrix corresponding to the root node; express The label matrix of Is with the internal node i Corresponding label matrix, internal nodes i Examples j Tags , d max Indicates the maximum number of categories in the internal node; Used to indicate the i The weight matrix of the internal nodes.

[0011] Preferably, the loss term constructed for the internal node is specifically to use the least squares loss as the loss function of the hierarchical feature selection: (1) Represents the Frobenius norm of a matrix.

[0012] Optimally, the regularization term for sparse learning is chosen Norm.

[0013] Preferably, i The label correlation regularization term of the internal nodes is specifically: (4) In the formula, tr represents the trace of the matrix, that is, the sum of the elements on the main diagonal of the matrix; Represents an internal node i With internal nodes l The path distance between: (3) Where, 0≤ i ≤N,0≤ l ≤N, i =0 or l =0 represents the root node in the hierarchical tree structure, Indicates i The internal nodes and l The lowest common ancestor node of the internal nodes.

[0014] Preferably, i The instance correlation regularization term of the internal nodes is specifically: (7) In the formula, and Respectively represent i Subtask instance p and examples q , represents the instance correlation matrix, express of The Laplace matrix of Indicates i Subtask instance p and examples q The correlation between: (6) In the formula, the parameters , Represents the Euclidean distance of k nearest neighbor instances.

[0015] Preferably, the objective function of the subtask corresponding to the internal node constructed using the loss term and the regularization terms including sparse learning, label correlation and instance correlation is specifically: (8) Where λ, α, and β are all non-negative constants, controlling sparsity, label relevance, and instance relevance, respectively.

[0016] Preferably, the weight matrix of each internal node is optimized according to the objective function, specifically: calculate about The derivative of : (9) In the formula, is a diagonal matrix, r The diagonal elements are , according to formula (9), replace formula (8) with: (10) Formula (10) is about The derivative of is: (11) Let (11) be equal to 0, and we obtain The expression is as follows: (12) According to the maximum number of categories in the internal node d max Initialize weights , ; Iterate through formula (12) until convergence to obtain the optimized weight matrix.

[0017] A hierarchical feature selection system based on label correlation and instance correlation includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically executes the steps in the above-mentioned hierarchical feature selection method based on label correlation and instance correlation.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention realizes effective feature selection and classification of large-scale hierarchical data by optimizing an objective function that comprehensively considers multiple factors. The least squares loss ensures the accuracy of prediction, and sparse learning promotes the effective selection of features. At the same time, the label correlation and instance correlation regularization terms respectively utilize the category relationship in the hierarchy and the similarity between instances, further improving the accuracy and robustness of classification. This makes it possible to select features more effectively and improve classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a model diagram of the hierarchical feature selection method based on label relevance and instance relevance of the present invention.

[0020] Figure 2 It is a schematic diagram of the algorithm of the hierarchical feature selection method based on label correlation and instance correlation of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0022] The present invention proposes a hierarchical feature selection method based on label correlation and instance correlation (HFS-LCIC). The feature selection task is decomposed into several subtasks, and a feature subset is selected for each subtask to improve the classification accuracy. Specifically, the following steps are included: S1, input hierarchical training data set, including an instance matrix containing n instances and m features , and a label matrix containing n instances and a label , and convert the single-label matrix into a multi-label matrix; S2. A hierarchical tree structure is used to describe the semantic relationship between categories in the hierarchical training dataset. The number of internal nodes (non-leaf nodes) in the hierarchical tree structure is set to N+1. Each internal node in the hierarchical tree structure is used to recursively model the problem, and each internal node corresponds to a subtask. S3. Construct loss terms for internal nodes , while using sparse learning, label relevance and instance relevance as regularization terms , using loss terms and regularization terms including sparse learning, label correlation, and instance correlation to construct the objective function of the subtask corresponding to the internal node , optimize the weight matrix of each internal node according to the objective function, sort the weight matrix of each subtask in descending order, select features with large weight values, and complete the hierarchical feature selection task.

[0023] The HFS-LCIC method of the present invention consists of four main components: Figure 1 As shown in the figure; (1) Least squares loss: This loss term aims to minimize the error between the predicted label and the actual label; (2) Sparse learning: The sparse regularization term constrains the feature coefficients so that the optimal shared features can be quickly selected; (3) Label correlation: This regularization term uses the rich semantic information in the hierarchical tree structure to analyze the different degrees of association between each category and other categories from a global perspective; (4) Instance correlation: This regularization term uses instance distribution information to analyze the similarity between instances under subtasks to further distinguish fine-grained categories.

[0024] In this embodiment, the conversion of a single label matrix into a multi-label matrix is ​​specifically as follows: Convert to a multi-label matrix via binary association ,in Representation instance j Tags, d Indicates the number of categories.

[0025] In this embodiment, the semantic relationship between categories in the hierarchical training data set described using the hierarchical tree structure is specifically: the instance matrix X According to the internal nodes, it is divided into ,in Represents and internal nodes i The corresponding instance matrix, n i Represents and internal nodes i The number of corresponding instances; X 0 is the instance matrix corresponding to the root node; express The label matrix of Is with the internal node i Corresponding label matrix, internal nodes i Examples j Tags , d max Indicates the maximum number of categories in the internal node; Used to indicate the i The weight matrix of the internal nodes.

[0026] In this embodiment, the loss term constructed for the internal node is specifically to use the least squares loss as the loss function of the hierarchical feature selection: (1) Represents the Frobenius norm of the matrix; the least squares loss uses the Euclidean distance as the similarity metric, which is simple and efficient to calculate.

[0027] In this embodiment, the regularization term of sparse learning is selected norm, As a convex function that meets the feature sparsity requirement, the norm is easier to achieve global optimization. The objective function of the basic level feature selection method is: (2) in λ Represents a non-negative constant.

[0028] In this embodiment, the semantic relationship embedded in the hierarchical structure as prior knowledge helps to capture the connection between categories, thereby assisting the design of the feature selection method; the smaller the path distance between internal nodes (categories) in the hierarchical tree structure, the closer the relationship between the categories, and the more similar their corresponding weight matrices; i Internal nodes and other internal nodes l The path distance between The calculation formula is: (3) Where, 0≤ i ≤N,0≤ l ≤N, i =0 or l =0 represents the root node in the hierarchical tree structure, Indicates i The internal nodes and l The lowest common ancestor node of the internal nodes. According to the path distance between the internal nodes , we can get the corresponding label correlation matrix P ,in Therefore, the i The label correlation regularization term of the internal nodes is defined as: (4) After adding the regularization term, formula (2) can be rewritten as: (5) Where λ and α are non-negative constants controlling sparsity and label correlation, respectively.

[0029] In this embodiment, in the hierarchical tree structure, instances under the same internal class have higher similarity; the more similar the instances are, the more similar their corresponding labels are; instance correlation is a factor that needs to be considered in subtasks that focus on the association between features and labels in local instances; this increases the gap between different categories within each subtask and distinguishes fine-grained categories more deeply; this leads to a more accurate selection of an effective feature subset for each subtask, thereby producing a compact and robust classification model; however, there are still noisy data and redundant features in the feature space of the subtask, and the commonly used cosine similarity is not sufficient to accurately evaluate the similarity between instances; therefore, based on the k-nearest neighbor mechanism in the subtask, the present invention uses a Gaussian kernel function to assign weights to instances of different distances to obtain an instance correlation matrix consisting of instance similarities between each pair of instances ;and Indicates i Subtask instance p and examples q The correlation between: (6) In the formula, the parameters , Represents the Euclidean distance of k nearest neighbor instances, and Respectively representi Subtask instance p and examples q ;No. i The instance correlation regularization term of the internal nodes is specifically: (7) In the formula, express of The Laplace matrix of ; combined with this regularization term, the final objective function is rewritten as: (8) Where λ, α, and β are all non-negative constants, controlling sparsity, label relevance, and instance relevance, respectively.

[0030] In this embodiment, the weight matrix of each internal node is optimized according to the objective function, specifically: calculate about The derivative of : (9) In the formula, is a diagonal matrix, r The diagonal elements are , according to formula (9), replace formula (8) with: (10) Formula (10) is about The derivative of is: (11) Let (11) be equal to 0, and we obtain The expression is as follows: (12) The whole process of HFS-LCIC method is as follows Figure 2 As shown, according to the maximum number of categories in the internal node d max Initialize weights , ; Iterate through equation (12) until convergence to obtain the optimized weight matrix. Then, sort the weight matrix of each subtask in descending order, and finally select the features with larger weight values ​​to complete the hierarchical feature selection task.

[0031] The present invention also proposes a hierarchical feature selection system based on label correlation and instance correlation, comprising a processor, a memory and a computer program stored in the memory. When the processor executes the computer program, it specifically executes the steps in the above-mentioned hierarchical feature selection method based on label correlation and instance correlation.

[0032] In summary, firstly, the present invention introduces the least squares loss and the sparse regularization term. Secondly, when considering the global hierarchy, the present invention takes into account the different degrees of association between categories. The correlation matrix is ​​constructed by calculating the path distance between internal categories, the regularization term is used to limit the label correlation, and the supervision information is provided for each subtask. Finally, under the local subtask, the present invention constructs a regularization term using the k-nearest neighbor mechanism optimized by the Gaussian kernel function to constrain the instance correlation. This helps to widen the gap between different categories under each subtask. Thereby, features can be selected more effectively and the classification performance can be improved.

[0033] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A hierarchical feature selection method based on label correlation and instance correlation, characterized in that: The feature selection task is decomposed into several subtasks, and a feature subset is selected for each subtask to improve classification accuracy. The specific steps include: S1, input hierarchical training data set, including an instance matrix containing n instances and m features , and a label matrix containing n instances and a label , and convert the single-label matrix into a multi-label matrix; S2, use a hierarchical tree structure to describe the semantic relationship between categories in the hierarchical training dataset. The number of internal nodes in the hierarchical tree structure is set to N+1; each internal node corresponds to a subtask; S3. Construct loss terms for internal nodes, and use sparse learning, label relevance, and instance relevance as regularization terms. Use loss terms and regularization terms including sparse learning, label relevance, and instance relevance to construct the objective function of the subtask corresponding to the internal node. Optimize the weight matrix of each internal node according to the objective function, sort the weight matrix of each subtask in descending order, select features with large weight values, and complete the hierarchical feature selection task.

2. The hierarchical feature selection method based on label correlation and instance correlation according to claim 1, characterized in that: The conversion of a single label matrix into a multi-label matrix is ​​specifically as follows: Convert to a multi-label matrix via binary association ,in Representation instance j Tags, d Indicates the number of categories.

3. The hierarchical feature selection method based on label correlation and instance correlation according to claim 1, characterized in that: The use of a hierarchical tree structure to describe the semantic relationship between categories in a hierarchical training data set is specifically: X According to the internal nodes, they are divided into ,in Represents and internal nodes i The corresponding instance matrix, n i Represents and internal nodes i The number of corresponding instances; X 0 is the instance matrix corresponding to the root node; express The label matrix of Is with the internal node i The corresponding label matrix, internal nodes i Examples j Tags , d max Indicates the maximum number of categories in the internal node; Used to indicate the i The weight matrix of the internal nodes.

4. The hierarchical feature selection method based on label correlation and instance correlation according to claim 3 is characterized in that: Specifically, the loss term for the internal node is constructed by using the least squares loss as the loss function for hierarchical feature selection: （1） Represents the Frobenius norm of a matrix.

5. The hierarchical feature selection method based on label correlation and instance correlation according to claim 4 is characterized in that: Regularization term selection for sparse learning Norm.

6. The hierarchical feature selection method based on label correlation and instance correlation according to claim 5, characterized in that: No. i The label correlation regularization term of the internal nodes is specifically: （4） In the formula, tr represents the trace of the matrix, that is, the sum of the elements on the main diagonal of the matrix; Represents an internal node i With internal nodes l The path distance between: （3） Where, 0≤ i ≤N,0≤ l ≤N, i =0 or l =0 represents the root node in the hierarchical tree structure, Indicates i The internal nodes and l The lowest common ancestor node of the internal nodes.

7. The hierarchical feature selection method based on label correlation and instance correlation according to claim 6, characterized in that: No. i The instance correlation regularization term of the internal nodes is specifically: （7） In the formula, and Respectively represent i Subtask instance p and examples q , represents the instance correlation matrix, express of The Laplace matrix of Indicates i Subtask instance p and examples q The correlation between: （6） In the formula, the parameters , Represents the Euclidean distance of k nearest neighbor instances.

8. The hierarchical feature selection method based on label correlation and instance correlation according to claim 7, characterized in that: The objective function of the subtask corresponding to the internal node constructed using the loss term and the regularization terms including sparse learning, label correlation and instance correlation is specifically: （8） Where λ, α, and β are all non-negative constants, controlling sparsity, label relevance, and instance relevance, respectively.

9. The hierarchical feature selection method based on label correlation and instance correlation according to claim 8, characterized in that: The weight matrix of each internal node is optimized according to the objective function, specifically: calculate about The derivative of : （9） In the formula, is a diagonal matrix, r The diagonal elements are , according to formula (9), replace formula (8) with: （10） Formula (10) is about The derivative of is: （11） Let (11) be equal to 0, and we obtain The expression is as follows: （12） According to the maximum number of categories in the internal node d max Initialize weights , ; Iterate through formula (12) until convergence to obtain the optimized weight matrix.

10. A hierarchical feature selection system based on label correlation and instance correlation, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory. When the processor executes the computer program, the method specifically performs the steps in the hierarchical feature selection method based on label correlation and instance correlation as described in any one of claims 1 to 9.

Images