Feature selection method suitable for unmeasurable medical data

Abstract

The invention discloses a feature selection method suitable for unmeasurable medical data. The method is suitable for assisting processing and classification of unmeasurable data in the related fields of medical diagnosis and pattern recognition. According to the method, redundant information in the unmeasurable data can be quickly and accurately eliminated on the premise of not introducing noise, so that relevant important features are reserved. The method mainly comprises the following steps: (1) preprocessing data; (2) generating a prior probability of an existing sample; (3) establishing an identification model of the target sample; (4) giving a target function corresponding to the recognition model; (5) representing a target function by using an equivalent optimization problem; (6) solving an optimization problem; (7) carrying out model interpretability analysis; and (8) model effect and evaluation. According to the method, feature selection is performed by utilizing probability information of the data, so that additional error information is prevented from being manually added to the data, and important features can be selected from large-scale unmeasurable data.

Description

Technical Field

[0001] The present invention relates to a key factor identification technology in non-measurable data features in machine learning and related fields, and specifically to a disease risk factor discovery method in the field of auxiliary medical diagnosis. Background Art

[0002] There is a large proportion of immeasurable non-numerical data in current medical data. At the same time, in auxiliary medical diagnosis, it is usually necessary to find the most important factors from a large number of suspected pathogenic factors.

[0003] Feature selection is an important method to select the most effective feature subset from the original features, while eliminating redundant information and improving the performance of machine learning algorithms. It is a key data preprocessing step in pattern recognition. Existing feature selection methods are mainly suitable for processing numerical data. However, in practical applications, there are a large number of non-numerical and immeasurable data. The main feature of immeasurable data is that the data cannot be compared, such as genetic phenotypes ('AA', 'AB', 'BB'), different blood types (A, B, AB, O), etc.

[0004] If the commonly used feature selection methods involve this type of non-numeric data that cannot be measured, a common method is to use dummy variables to convert such non-numeric variables into multiple numeric variables. This conversion increases the scale of the problem and introduces additional noise. Another method is to directly force the data to be converted into numeric data, such as (A, B, AB, O) to (1, 2, 3, 4). This forced conversion makes it possible to compare the size of data that were originally incomparable. In essence, it artificially adds information to the data that does not exist in the data itself. Existing methods use this information for feature selection, and the result is that the correct data features are usually not obtained. Summary of the invention

[0005] This method mainly uses the probability of data to achieve feature selection. Probability is the inherent intrinsic information of the data (including numerical data and non-numerical data) itself. By using probability for feature selection, no additional error information is artificially added to the data, thereby achieving feature selection for numerical data and non-numerical data.

[0006] This feature selection method takes the processing of hypertension risk factor data as an example to describe the main content of the present invention. The method includes the following eight steps:

[0007] 1) Data preprocessing: According to the existing hypertension diagnostic criteria, healthy people and hypertensive patients were screened from the existing samples, and samples with missing data features were supplemented (smaller proportion missing) or directly discarded (larger proportion missing);

[0008] 2) Generate prior probabilities: Calculate the prior probabilities of different feature values ​​in healthy population samples and hypertensive patient samples. The probability of numerical features can be calculated using the probability density function, and the probability of non-measurable features can be obtained based on the frequency of occurrence of different values;

[0009] 3) Build the model: According to Bayes' theorem, the probability that the new sample of the unknown population belongs to the healthy population sample or the hypertensive patient sample can be calculated based on the prior probability obtained in step 2), that is, the posterior probability. Whether the unknown population sample is a healthy population or a hypertensive patient depends on the class with the largest posterior probability. This method has improved this process. First, the left and right logarithms of the calculation formula of the posterior probability are taken, and the multiplication operation is converted into an addition operation. Then, each item of the prior probability is multiplied by a weight parameter. At this time, each feature corresponds to a parameter, and the weight of the constant term is 1;

[0010] 4) Establishing the objective function: This method identifies samples of unknown populations, and its purpose is to improve the accuracy of identification as much as possible. That is, during the test, the original healthy samples must be identified as healthy people, and the hypertensive samples must be identified as hypertensive patients. According to this goal, rewriting the formula obtained in step 3) can obtain an optimization objective function for the weight parameter;

[0011] 5) Optimize the model: The model obtained by solving the objective function obtained in step 4) cannot realize the feature selection function, and this model may also cause overfitting. This method adds a 1-norm regularization term to the objective function obtained in step 4), which can reduce overfitting and improve the model's ability to recognize new population samples. The main function of the model with a 1-norm regularization term is to realize the feature selection function;

[0012] 6) Model solution: The objective function in step 5) is a convex function, and the whole problem is a constrained optimization problem. To solve it, we can first determine the upper bound of the regularization parameter, then convert the constrained optimization problem into an unconstrained optimization problem, and finally use the conjugate gradient method to solve it, and we can get an optimal solution for the weight parameter;

[0013] 7) Model interpretability analysis: After obtaining the optimal solution in step 6), the weights with very small absolute values ​​are set to 0, which is equivalent to eliminating the factors corresponding to these weights, and the remaining factors are hypertension risk factors, thus achieving the goal of identifying hypertension risk factors through feature selection;

[0014] 8) Model Effect and Evaluation: The experimental results show that most irrelevant factors can be eliminated by using this method, while the recognition rate of the target population is improved. Model evaluation is based on ten-fold cross-validation and independent testing, and the area under the ROC curve (AUC) is specifically used for performance measurement. The final performance is compared with the benchmark algorithms WNB-G, L1 logistic regression, and SVM-RFE, and the t-test is used to ensure the validity of the results. Effect of the Invention

[0015] This method combines the characteristics of Bayesian for handling immeasurable categorical data and the ability of L1 regularization to handle high-dimensional data, realizing feature selection for high-dimensional immeasurable categorical data. The feature selection results for hypertension show that this method can effectively select relevant factors for hypertension and improve the accuracy of hypertension diagnosis at the same time. The comparative experiment with the benchmark method shows that the effect of this method is significantly better than the currently commonly used benchmark algorithms WNB-G, L1-LR, and SVM-RFE. Specific Embodiment

[0016] The following further elaborates on the content of the present invention in combination with the table and embodiments, but it is not a limitation to the present invention.

[0017] Embodiment: A feature selection method applicable to immeasurable medical data, and the specific implementation steps are as follows.

[0018] For ease of description, we first make the following regulations: For each sample (healthy person or hypertensive patient), there are n possible pathogenic factors to be considered. For a new sample X, we can represent it as X = (x1, x2,..., x n ), where x i represents the value corresponding to the i-th factor of this sample. And we stipulate that the class label of the healthy population is -1, and the class label of the hypertensive patient is +1. Therefore, the class label y ∈ {-1, +1}. According to Bayes' theorem, if a sample X is a healthy person, then the conditional probability P(-1|X) > P(+1|X) (i.e., P(-1|x1, x2,..., x n ) > P(+1|x1, x2,..., x n ), otherwise P(-1|X) < P(+1|X) (i.e., P(-1|x1, x2,..., x n ) < P(+1|x1, x2,..., x n ).

[0019] 1) Data preprocessing: First, we will remove the data features with more than 90% missing values. Then, according to the diagnostic criteria for hypertension, we get the hypertension data samples that meet the conditions. Finally, we fill in the missing values. We use two strategies to handle missing values. One is to fill with the mean value, and the other is to fill with the default value, which depends on the characteristics of each field. For example, the missing values ​​of the 'height' field are filled with the mean value, but the default value '0' is used for the 'amount of liquor consumed'.

[0020] 2) Generate prior probabilities: Calculate the prior probabilities of different characteristic values ​​in healthy population samples and hypertensive patient samples based on the numerical characteristics of each factor. Numerical factors can obtain a probability density function and use it to calculate the probability of a certain value. Unmeasurable factors can count the frequency of occurrence of different values ​​to obtain the corresponding probability. It is possible that a certain value of some sample characteristics is not in the current sample, but this does not mean that the probability of this happening is 0. When estimating the probability, smoothing is required, and we use the Laplace correction. The specific calculation method is as follows:

[0021] Assume that the frequency of a certain value in a certain feature is Dc, the total number of samples is D, and the number of sample categories is N (in this example, N=2, because the samples are divided into two categories: healthy people and hypertensive people), then the prior probability Pc of this value is Pc=(Dc+1) / (D+N)

[0022] 3) Build a model: According to Bayes' theorem, we can calculate the probability that a sample X belongs to a healthy person. The formula is:

[0023] P(-1|X)=P(X|-1)P(-1) / P(X)

[0024] According to the conditional independence assumption and X=(x1,x2,…,x n ), we can rewrite the above formula as:

[0025] P(-1|X)=[P(x1|-1)P(x2|-1)…P(x n |-1)P(-1)] / P(X)

[0026] Similarly, the formula for the probability that sample X belongs to a patient with hypertension is:

[0027] P(+1|X)=[P(x1|+1)P(x2|+1)…P(x n |+1)P(+1)] / P(X)

[0028] Comparing the magnitudes of the above two expressions can determine the status of this sample. Since the denominators of the above two expressions are both P(X), it is only necessary to compare the numerators. If P(-1|X) > P(+1|X), it indicates that the sample is a healthy person; conversely, if P(-1|X) < P(+1|X), it indicates that the sample is a hypertensive patient. That is:

[0029] P(x1|-1)P(x2|-1)…P(x n |-1)P(-1) > P(x1|+1)P(x2|+1)…P(x n |+1)P(+1) (y = -1) (1)

[0030] P(x1|-1)P(x2|-1)…P(x n |-1)P(-1) < P(x1|+1)P(x2|+1)…P(x n |+1)P(+1) (y = +1) (2)

[0031] Taking the logarithms of formulas (1) and (2) respectively, and assigning a weight parameter w to the logP(x i |-1) term and the logP(x i |+1) term, then formulas (1) and (2) can be rewritten in the following form:

[0032] w1logP(x1|-1) + w2logP(x2|-1) + … + w n logP(x n |-1) + logP(-1) > w1logP(x1|+1) + w2logP(x2|+1) + … + w n logP(x n |+1) + logP(+1)(y = -1) (3)

[0033] w1logP(x1|-1) + w2logP(x2|-1) + … + w n logP(x n |-1) + logP(-1) < w1logP(x1|+1) + w2logP(x2|+1) + … + w n logP(x n |+1) + logP(+1)(y = +1) (4)

[0034] Rearranging formulas (3) and (4) gives:

[0035] w1[logP(x1|+1) - logP(x1|-1)] + w2[logP(x2|+1) - logP(x2|-1)] + … + w​​n [logP(x n |+1)-logP(x n |-1)]+[logP(+1)-logP(-1)]<0(y=-1) (5)

[0036] w1[logP(x1|+1)-logP(x1|-1)]+w2[logP(x2|+1)-logP(x2|-1)]+…+w n [logP(x n |+1)-logP(x n |-1)]+[logP(+1)-logP(-1)]>0(y=+1) (6)

[0037] Considering the value of the class label, equations (5) and (6) can be combined into a unified formula as follows:

[0038] y{w1[logP(x1|+1)-logP(x1|-1)]+w2[logP(x2|+1)-logP(x2|-1)]+…+w n [logP(x n |+1)-logP(x n |-1)]+[logP(+1)-logP(-1)]}>0 (7)

[0039] Assume that sample X i =(x i1 , x i2 ,…,x in ) is the i-th sample, and the corresponding class label is y i , we let a ij =logP(x ij |+1)-logP(x ij |-1), b=logP(+1)-logP(-1), w=[w1,w2,…w n ] T , and A (i) =[a i1 ,a i2 ,…,a in ] T , so formula (7) can be written as follows:

[0040] y i (w T A (i) +b)>0(i=1,2,…,m.y i ∈{-1,+1}) (8)

[0041] Our ultimate goal is to learn a weight vector w based on the existing hypertension data, which can make as many of the m inequalities in (8) as possible hold true. In this way, when the Bayesian classifier using this weight vector w is used to classify the hypertension sample data, the classification error rate can be minimized.

[0042] 4) Establish the objective function: In order to obtain the weight vector w that meets the conditions in step 3), we define an optimization problem as follows:

[0043] min f(w)=1 / m∑ i＝1..m L(y i (w T A (i) +b)) swt j >0,(j=1,2,…,m.)y i ∈{-1,+1} (I)

[0044] Here, the L function satisfies L(t)=0(t>0); L(t)=1(t≤0), which is usually called the loss function. Here, L(t) is the gold standard loss function, sometimes also called 0-1 loss. w=[w1,w2,…w n ] T ∈R n , A (i) ∈R n , b is a constant value.

[0045] By solving the optimization problem (I), we can obtain the maximum likelihood model for the hypertension classification (identification) problem. Unfortunately, this type of solution usually leads to overfitting problems; on the other hand, the 0-1 loss function is not a convex function, which makes it difficult to solve.

[0046] 5) Optimization model: In order to solve these problems in optimization problem (I), we use an approximate loss function of 0-1 loss, which has a convex relaxation of logistic loss, and apply L1 regularization to it, so that optimization problem (I) can be written as the following optimization problem (II):

[0047] min f(w)=1 / m∑ i＝1..m C(y i (w T A (i) +b))+λ||w||1 swt j >0,(j=1,2,…,m.)y i ∈{-1,+1} (II)

[0048] Here the loss function C(t) = log(1+e -t ), w=[w1,w2,…w n ] T ∈R n , A (i) ∈R n , b is a constant value, and λ is the regularization parameter.

[0049] Optimization problem (II) is a type of L1 regularized learning problem. After solving this optimization problem, we can obtain a sparse global optimal solution w, which contains only N (N<<n) non-zero values ​​at most. This means that we can only consider these N features, thereby greatly reducing the complexity of the model.

[0050] 6) Model solution: In the optimization problem (II), we first determine the upper bound λ of the regularization parameter λ max , then for each determined λ, let w j =(v j ) 2 , thus transforming the optimization problem (II) into an unconstrained convex optimization problem. The final optimization problem can be solved by the conjugate gradient method.

[0051] 7) Model interpretability analysis: (usually the threshold is 0.01 of the maximum absolute value weight) A weight parameter of 0 indicates that the corresponding feature can be eliminated in the classification model. Resetting the weight with a very small absolute value to 0 indicates that the corresponding feature has little effect on the target class recognition and can be ignored.

[0052] 8) Model effect and evaluation:

[0053] Model effect: This method performs feature selection on hypertension risk data, identifies the risk factors of hypertension, and identifies which samples are hypertensive people and which are healthy people. There are 1775 hypertension data samples in this embodiment, including 342 samples of hypertensive patients and 1433 samples of healthy people. There are 68 features involved, including gender, age, education, occupation, blood sugar, height, weight and other physiological, living habits and environmental factors.

[0054] This feature selection method finally identified 9 features from 68 features as risk factors for hypertension, as shown in Table 1 below:

[0055] Table 1: Screening results of risk factors for hypertension Risk factors for hypertension Weight age 0.495217 Years of drinking 0.848864 Daily meat consumption 0.739663 height 0.272691 weight 0.476119 waistline 0.286072 Hips 0.386034 pulse 0.697943

[0056] Table 1 continued: Screening results of risk factors for hypertension Risk factors for hypertension Weight Fasting blood sugar 0.777147

[0057] When the original 68 features are used to identify people with hypertension, the accuracy rate is 78.69%; when the 9 features obtained by the feature selection method are used to identify people with hypertension, the accuracy rate is 86.04%.

[0058] Model evaluation: The model evaluation of this method is not based on the number of features selected. The reason is twofold. First, the purpose of feature selection is to eliminate redundant information and improve the recognition rate of the model. Second, the method that cannot handle unmeasurable features transforms the features instead of using the original features. In the model evaluation, the recognition performance of this method is compared with the benchmark algorithms WNB-G, L1 logistic regression, and SVM-RFE. To ensure the effectiveness of the experiment of this method, ten-fold cross validation and independent testing are used, and the area under the ROC curve is used to measure the performance of the algorithm.

[0059] All the data in the experiment come from the UCI machine learning database, including Chess, Votes, Mushroom, SPECT heart, and Lung cancer. The information of these five datasets is shown in Table 2.

[0060] Table 2: Dataset information Dataset Number of samples The proportion of samples in two categories Number of attributes Chess 3196 1527:1669 36 Votes 435 168:267 16 Mushroom 8124 3916:4208 22 SPECT Heart 267 55:212 22 Lung Cancer 32 9:23 56

[0061] In order to eliminate the influence of random factors, we run the algorithm 100 times for each data set and take the mean and variance of AUC. The results are shown in Table 3.

[0062] Table 3: AUC values ​​of this example, WNB-G, L1Reg, and SVM-RFE after running 100 times Dataset This example WNB-G L1Reg SVM-RFE Chess 0.9942±0.0019 0.9795±0.0057 0.9377±0.0125 0.9621±0.0069 Votes 0.9898±0.0083 0.9806±0.0104 0.9474±0.0247 0.9617±0.0115 Mushroom 0.9998±0.0002 0.9986±0.0004 0.9991±0.0004 0.9992±0.0004 SPECT Heart 0.8431±0.0576 0.8369±0.0624 0.7899±0.0739 0.7256±0.0632 Lung Cancer 0.8950±0.1869 0.7525±0.2117 0.6175±0.2711 0.7950±0.1293

[0063] Note: The largest mean value in the table is marked in bold.

[0064] We use t-test to test the significance of the final results obtained by comparing this example with WNB-G, this example with L1 regression, and this example with SVM-RFE. The significance level is 0.05. The test results are shown in Table 4.

[0065] Table 4: P values ​​of t-tests comparing WNB-G, L1Reg, and SVM-RFE in this example Dataset vs WNB-G vs L1Reg vs SVM-RFE Chess <![CDATA[4.36×10 -62 ]]> <![CDATA[2.34×10 -105 ]]> <![CDATA[8.18×10 -106 ]]> Votes <![CDATA[5.33×10 -11 ]]> <![CDATA[2.40×10 -38 ]]> <![CDATA[8.61×10 -49 ]]> Mushroom <![CDATA[1.17×10 -67 ]]> <![CDATA[1.89×10 -35 ]]> <![CDATA[6.40×10 -27 ]]> SPECT Heart 0.465240907 <![CDATA[4.83×10 -8 ]]> <![CDATA[1.25×10 -30 ]]> Lung Cancer <![CDATA[1.02×10 -6 ]]> <![CDATA[7.22×10 -15 ]]> <![CDATA[1.78×10 -5 ]]>

[0066] Note: Bold indicates that the results are significant at the significance level p = 0.05

[0067] From Table 3, we can see that this example outperforms the other three algorithms on all five datasets, and from Table 4, we can see that the t-test results show that except for one (SPECT Heart, vs WNB-G), the other results are significantly different. This shows that this method is significantly better than the three benchmark algorithms.

Claims

1. A feature selection method for non-measurable medical data, characterized in that: The Bayesian method is combined with weight parameters to achieve feature selection of non-measurable data.

2. A feature selection method applicable to non-measurable medical data according to claim 1, characterized in that: The model used in this method corresponds to the following optimization problem: min f(w)=1 / m∑ i＝1..m C(y i (In T AND (i) +b))+λ||w||1 s.t.w j >0,(j=1,2,…,m.)y i ∈{-1,+1} Here the loss function C(t) = log(1+e -t ), w=[w1,w2,…w n ] T ∈R n , A (i) ∈R n , b is a constant value, and λ is the regularization parameter.