Hierarchical structure-based feature selection method, terminal equipment and storage medium

Abstract

The invention relates to a feature selection method based on a hierarchical structure, a terminal device and a storage medium, in the method, after irrelevant features are removed by using a sparse regular term, a father-son relationship in the hierarchical structure is combined with a reconstruction relationship between samples in a feature space, and correlation of samples of each category under the same subtree is learned. According to the method, the accuracy of hierarchical structure data mark prediction can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data analysis, and in particular to a feature selection method based on a hierarchy, a terminal device and a storage medium. Background Art

[0002] In the era of big data, the number of data samples and features has increased dramatically, and the number of category labels contained in the data space is huge. For example, there are hundreds of categories in stellar spectra, and the categories of ImageNet image data and web page data have reached tens of thousands. It is difficult for traditional methods to find a unified and compact global feature subset to distinguish numerous categories. Therefore, inspired by the divide-and-conquer strategy, the numerous categories in the classification task are managed with a hierarchical structure. This classification task for the hierarchy is called hierarchical classification.

[0003] As we all know, high-dimensional data space usually contains a large number of redundant and irrelevant features, which not only greatly increases the storage and computing costs of data analysis, but also reduces the efficiency of learning algorithms. Feature selection is an important data preprocessing technology and an effective dimensionality reduction method. By retaining relevant features and removing redundant features, the learning model is simplified and the training time of the model is reduced. Traditional feature selection methods assume that all categories are independent of each other and do not take into account the hierarchical structure between categories.

[0004] In order to solve the challenges brought by hierarchical data in large-scale classification learning, many hierarchical feature selection methods have been proposed. Although these feature selection methods make full use of the hierarchical structure between categories and effectively improve the performance of the classifier, they do not consider issues such as the relationship between samples. In recent years, more and more scholars have introduced sample correlation into the research of feature selection algorithms. This method considers the relationship between samples and achieves better performance than most multi-label feature selection algorithms that do not consider the correlation between samples. However, in hierarchical data sets, samples have only a single category, so the above methods cannot solve the feature selection problem of hierarchical data sets. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a feature selection method based on a hierarchical structure, a terminal device and a storage medium.

[0006] The specific plan is as follows:

[0007] A hierarchical feature selection method includes the following steps:

[0008] S1: Receive a hierarchical data set X, and extract the sample matrix X of each internal node i in the hierarchical data set X based on hierarchical sparse learning i , then X={X0,X1,...,Xi ,...,X N}, where i=0,1,...,i,...,N represents the serial number of the internal node, and N+1 represents the total number of internal nodes; set the label of internal node i to Y i , then based on the labels of all internal nodes, we get the label matrix Y = {Y0, Y1, ..., Y i ,...Y N};

[0009] S2: Initialization parameters: The value b of the positive sequence b at the 0th iteration obtained by the accelerated proximal gradient method (0) and the value b at the first iteration (1) are all 1, the value of the sample weight matrix W at the 0th iteration is W (0) and the value W at the first iteration (1) All are (X T X+γI) -1 X T Y, where γ represents the regularization term balance factor, T represents the transpose of the matrix, and the number of iterations t is initially set to 2;

[0010] S3: The weight matrix W at the tth iteration is calculated by the following formula (t) :

[0011]

[0012] S4: based on the weight matrix W at the tth iteration (t) , calculate Z (t) :

[0013]

[0014] Among them, Z (t) Represents the weight matrix W at the tth iteration (t) The result of executing the soft threshold function on each element in , Lip represents the Lipschitz constant, α and β both represent the balance factor, I represents the unit matrix, S represents the reconstruction coefficient matrix, and the subscript F represents the F norm. Represents the weight matrix of the parent node of each node at the tth iteration;

[0015] S5: Calculate the optimal solution of the weight matrix by soft threshold method:

[0016]

[0017] in, represents the optimal weight value of the jth feature of the ith internal node in the weight matrix at the t+1th iteration of the prediction, Represents Z (t)The value of the element corresponding to the jth feature of the i-th internal node in , λ represents the balance factor;

[0018] S6: Determine whether the number of iterations t is less than the iteration number threshold. If so, proceed to S7; otherwise, terminate the iteration, output the optimal solution of the weight matrix, and proceed to S9;

[0019] S7: Calculate b at the t+1th iteration by the following formula (t+1) :

[0020]

[0021] S8: increment the number of iterations t by 1 and return to S3;

[0022] S9: Complete feature selection for each internal node according to the size of the weight corresponding to each feature of each internal node in the optimal solution of the weight matrix.

[0023] Furthermore, the calculation formula of Lipschitz constant Lip is:

[0024]

[0025] Furthermore, after each iteration, all diagonal elements in the reconstruction coefficient matrix S are set to 0.

[0026] A hierarchical feature selection terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above in the embodiment of the present invention are implemented.

[0027] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above in an embodiment of the present invention are implemented.

[0028] The present invention adopts the above technical solution to effectively improve the accuracy of labeling prediction of hierarchical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Shown is a flow chart of Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0030] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0031] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0032] Embodiment 1:

[0033] The embodiment of the present invention provides a feature selection method based on a hierarchical structure, and the related theories required are as follows:

[0034] (1) Category-based hierarchy

[0035] Hierarchical structures are mainly divided into tree structures and directed acyclic graph structures. The "subordinate" relationship in the tree structure has three characteristics: irreversibility, anti-reflexivity and transitivity. The hierarchical structure is defined by (Y, <), where Y represents a set of tags and "<" represents a "subordinate" relationship.

[0036] (1) Irreversibility: If t i <t j , Then t j <t i

[0037] (2) Anti-reflexivity: There is i <t i

[0038] (3) Transitivity: If t i <t k And t k <t j ,right Then t i <t j .

[0039] (2) Hierarchical-based sparse learning

[0040] Let X∈R n×d is the sample matrix, n and d represent the number of samples and features respectively. Let the number of all non-leaf nodes in the category hierarchy be N+1, then the sample X can be divided into X0, X1, ..., X N , where the sample matrix of internal node i can be expressed as Let the class mark (i.e. label) matrix be Y0,Y1,...Y N ,in and where d max Indicates the maximum number of categories in an internal node.

[0041] In supervised feature selection, sparse learning has been shown to be an effective technique due to its good interpretability. The purpose of this method is to minimize the fitting error and the sparse regularization term. The general form of the model based on sparse learning is as follows:

[0042]

[0043] Wherein, L(.) is a loss function. Commonly used loss functions include least squares loss, hinge loss, and logistic loss. In this embodiment, the least squares loss widely used by most sparse learning-based methods is adopted. Therefore, the loss function is defined as:

[0044]

[0045] Γ(W) is the sparse regularization term on W, and the subscript F represents the F-norm.

[0046] In sparse learning methods, 2,1 Norm regularization can add structured sparsity to feature selection so that irrelevant features can be easily removed. 2,1 Norm combination can reformulate formula (1) as:

[0047]

[0048] in, represents the weight matrix of internal node i, W=[W0,W1,...,W N ] represents the weight matrix of the sample.

[0049] (3) Hierarchical learning based on parent-child relationships

[0050] There are two main types of relationships in a hierarchical tree structure: parent-child relationships and sibling relationships. Generally speaking, categories from the same subtree share more neighborhood information than categories from different subtrees. Parent-child relationships are the most adjacent nodes in a category tree hierarchy and share certain features. Therefore, the regularization term that introduces parent-child relationships into hierarchical feature selection is as follows:

[0051]

[0052] Among them, p i Represents the parent node of node i.

[0053] Based on the above theoretical foundation, this embodiment proposes a sample correlation model, and on this basis, defines the objective function of the model. Finally, the optimization method of the model is introduced.

[0054] (1) Sample correlation model

[0055] Due to the presence of noise and redundant features in the feature space, the traditional cosine similarity evaluation of sample correlation may not be accurate enough and cannot reflect the complex relationship between samples. Therefore, this embodiment proposes to use the reconstruction relationship between samples in the feature space to model the relationship between a sample and other samples to obtain the reconstruction coefficient matrix S∈R n ×n , where S ij represents the reconstruction contribution of the jth sample to the ith sample, S i =(s i1 ,s i2 ,...,s in )∈R n×n represents the i-th row of the reconstruction coefficient matrix S, S ii = 0. The reconstruction coefficient matrix S can be obtained by solving the following optimization problem:

[0056]

[0057] The first term is the linear reconstruction error of all samples, and the second term is the l1 norm that controls the sparsity of the reconstruction coefficient of each sample. The parameter γ represents the regularization term balance factor, which is used to balance the relative importance of the regularization term. Where diag(S) = 0 means that any sample does not contribute to its own reconstruction.

[0058] In order to keep the reconstruction relationship between training samples in the feature space in the label space, for each sample, it can be reconstructed by other samples according to the reconstruction coefficient in S, and a regularization term that measures the sample correlation is given to make the measurement error of the sample correlation as small as possible. The regularization term is as follows:

[0059]

[0060] Then, the final objective function is defined as

[0061]

[0062] In formula (7), the first term is the loss function, which is used to measure the distance between the predicted label and the true label. The second term is l 2,1 The regularization term is used to obtain a sparse solution to meet the requirement of feature sparsity. The third term considers the parent-child relationship between nodes. The fourth term is used to maintain the reconstruction relationship between training samples in the feature space. α, β, and λ are all balancing factors.

[0063] (2) Model optimization method

[0064] For convenience, the objective function in formula (7) can be expressed as:

[0065]

[0066] Where I∈R n×n is the identity matrix.

[0067] Because l 2,1 The non-smoothness of the regularization term, in this embodiment, the accelerated proximal gradient method is used to solve the minimization problem of the function T(W). The accelerated proximal gradient method is usually used to solve the optimization problem of non-smooth and convex functions. It can be expressed in a general optimization framework as follows:

[0068]

[0069] Among them, f(W) is Lipschitz continuous, that is, f(W) satisfies the following conditions:

[0070]

[0071] Where 1≤i≤q,ΔW=W i '-W i , Lip is the Lipschitz constant, W i ' indicates W i The derivative of .

[0072] From formula (8) and formula (9), f(W) and g(W) can be expressed as:

[0073]

[0074] g(W)=λ||W|| 2,1

[0075] Therefore, the gradient of f(W) with respect to W can be obtained as:

[0076]

[0077] Among them, E=(IS)X.

[0078] Then, from formula (10) and formula (12), we can get:

[0079]

[0080] Therefore, the Lipschitz constant Lip can be calculated by the following formula:

[0081]

[0082] Consider the second-order Taylor series of f(W) with respect to the parameter vector W (t) Current estimates:

[0083]

[0084] where const is a constant that is independent of W, and <·,·> represents the inner product. (t+1) The minimum value of formula (15) can be obtained:

[0085]

[0086] After that, the optimal solution of W is obtained iteratively in the following way:

[0087]

[0088] in,

[0089] Set k, for sequence b t Satisfy (b t ) 2 -b t ≤(b t-1 ) 2 , W (t) is the result of W at the tth iteration.

[0090] The closed-form solution of formula (17) can be calculated by the soft threshold method, which is defined as follows:

[0091]

[0092] in, represents the optimal weight value of the jth feature of the ith internal node in the weight matrix at the t+1th iteration of the prediction, Represents Z (t) The value of the element corresponding to the jth feature of the i-th internal node in .

[0093] The accelerated proximal gradient method using formulas (9) to (18) can also solve the optimization problem of the reconstruction matrix S in formula (5). In order to satisfy the constraint condition diag(S)=0, this embodiment sets all diagonal elements in S to 0 after each iterative update.

[0094] According to the final objective function of formula (7) and the optimization process of formula (9) to formula (18), the optimal solution of the weight matrix can be obtained, as follows: Figure 1 As shown, the specific steps are as follows:

[0095] S1: Receive a hierarchical data set X, and extract the sample matrix X of each internal node i in the hierarchical data set X based on hierarchical sparse learning i , then X={X0,X1,...,X i ,...,X N}, where i=0,1,...,i,...,N represents the serial number of the internal node, and N+1 represents the total number of internal nodes; set the label of internal node i to Y i , then based on the labels of all internal nodes, we get the label matrix Y = {Y0, Y1, ..., Y i ,...Y N};

[0096] S2: Initialization parameters: The value b of the positive sequence b at the 0th iteration obtained by the accelerated proximal gradient method (0) and the value b at the first iteration (1) are all 1, the value of the sample weight matrix W at the 0th iteration is W (0) and the value W at the first iteration (1) All are (X T X+γI) -1 X T Y, where γ represents the regularization term balance factor, T represents the transpose of the matrix, and the number of iterations t is initially set to 2;

[0097] S3: The weight matrix W at the tth iteration is calculated by the following formula (t) :

[0098]

[0099] S4: based on the weight matrix W at the tth iteration (t) , calculate Z (t) :

[0100]

[0101] Among them, Z (t) Represents the weight matrix W at the tth iteration (t) The result of executing the soft threshold function on each element in , Lip represents the Lipschitz constant, α and β both represent the balance factor, I represents the unit matrix, S represents the reconstruction coefficient matrix, and the subscript F represents the F norm. Represents the weight matrix of the parent node of each node at the tth iteration;

[0102] S5: Calculate the optimal solution of the weight matrix by soft threshold method:

[0103]

[0104] in, represents the optimal value of the element in the pth row and qth column of the weight matrix at the t+1th iteration of the prediction, Represents Z (t) The value of the element in the pth row and qth column of , λ represents the balance factor;

[0105] S6: Determine whether the number of iterations t is less than the iteration number threshold. If so, proceed to S7; otherwise, terminate the iteration, output the optimal solution of the weight matrix, and proceed to S9;

[0106] S7: Calculate b at the t+1th iteration by the following formula (t+1) :

[0107]

[0108] S8: increment the number of iterations t by 1 and return to S3;

[0109] S9: According to the weight corresponding to each feature of each internal node in the optimal solution of the weight matrix, the feature selection of each internal node is completed. That is, the weight values ​​in the optimal solution of the weight matrix are arranged in descending order, and the feature corresponding to the larger weight value is selected.

[0110] The method of this embodiment can be applied to any dataset with a hierarchical structure, such as a protein dataset, an image dataset, etc.

[0111] The method of the embodiment of the present invention utilizes the recursive regularization term to learn the parent-child relationship between category labels in the hierarchical structure and the correlation between samples within each category label, selects label-specific features for the corresponding labels, and obtains the sample correlation in the feature space through iterative optimization of the accelerated proximal gradient method, ultimately achieving a better classification effect.

[0112] The embodiments of the present invention have the following beneficial effects:

[0113] (1) Combine sample correlation and category hierarchy. Not only the parent-child relationship between categories is considered, but also the correlation between samples within each category is fully considered.

[0114] (2) When measuring sample correlation, the reconstruction coefficient matrix is ​​integrated into the training model, and l is introduced 2,1 The norm selects the most important features.

[0115] (3) The accelerated proximal gradient method is used to solve the optimization problem of the proposed model, and the superiority of the proposed algorithm is evaluated under multiple evaluation indicators.

[0116] Embodiment 2:

[0117] The present invention also provides a hierarchical feature selection terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method embodiment of the first embodiment of the present invention when executing the computer program.

[0118] Further, as an executable solution, the hierarchical structure-based feature selection terminal device can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The hierarchical structure-based feature selection terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the composition structure of the above-mentioned hierarchical structure-based feature selection terminal device is only an example of a hierarchical structure-based feature selection terminal device, and does not constitute a limitation on the hierarchical structure-based feature selection terminal device. It may include more or less components than the above, or a combination of certain components, or different components. For example, the hierarchical structure-based feature selection terminal device may also include input and output devices, network access devices, buses, etc., and the embodiments of the present invention do not limit this.

[0119] Further, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the hierarchical feature selection terminal device, and uses various interfaces and lines to connect the various parts of the entire hierarchical feature selection terminal device.

[0120] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the hierarchical feature selection terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0121] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0122] If the module / unit integrated in the hierarchical feature selection terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc.

[0123] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A hierarchical feature selection method, characterized in that: The following steps are involved: S1: Receive a hierarchical data set X, and extract the sample matrix X of each internal node i in the hierarchical data set X based on hierarchical sparse learning i , then X={X0,X1,...,X i ,...,X N }, where i=0,1,...,i,...,N represents the serial number of the internal node, and N+1 represents the total number of internal nodes; set the label of internal node i to Y i , then based on the labels of all internal nodes, we get the label matrix Y = {Y0, Y1, ..., Y i ,...Y N }; S2: Initialization parameters: The value b of the positive sequence b at the 0th iteration obtained by the accelerated proximal gradient method (0) and the value b at the first iteration (1) are all 1, the value of the sample weight matrix W at the 0th iteration is W (0) and the value W at the first iteration (1) All are (X T X+γI) -1 X T Y, where γ represents the regularization term balance factor, T represents the transpose of the matrix, and the number of iterations t is initially set to 2; S3: The weight matrix W at the tth iteration is calculated by the following formula (t) : S4: based on the weight matrix W at the tth iteration (t) , calculate Z (t) : Among them, Z (t) Represents the weight matrix W at the tth iteration (t) The result of executing the soft threshold function on each element in , Lip represents the Lipschitz constant, α and β both represent the balance factor, I represents the unit matrix, S represents the reconstruction coefficient matrix, and the subscript F represents the F norm. Represents the weight matrix of the parent node of each node at the tth iteration; S5: Calculate the optimal solution of the weight matrix by soft threshold method: in, represents the optimal weight value of the jth feature of the ith internal node in the weight matrix at the t+1th iteration of the prediction, Represents Z (t) The value of the element corresponding to the jth feature of the i-th internal node in , λ represents the balance factor; S6: Determine whether the number of iterations t is less than the iteration number threshold. If so, proceed to S7; otherwise, terminate the iteration, output the optimal solution of the weight matrix, and proceed to S9; S7: Calculate b at the t+1th iteration by the following formula (t+1) : S8: increment the number of iterations t by 1 and return to S3; S9: Complete feature selection for each internal node according to the size of the weight corresponding to each feature of each internal node in the optimal solution of the weight matrix.

2. The hierarchical feature selection method according to claim 1, characterized in that: The calculation formula of Lipschitz constant Lip is:

3. The hierarchical feature selection method according to claim 1, characterized in that: After each iteration, all diagonal elements in the reconstruction coefficient matrix S are set to 0.

4. A hierarchical feature selection terminal device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 3 when executing the computer program.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images