Over-sampling algorithm based on natural neighborhood weighting

Abstract

The invention discloses an oversampling algorithm based on natural neighborhood weighting, and aims to solve the problem that a large number of synthetic samples which are not obvious in minority class sample classification effect and even have noise may be generated due to the fact that a traditional oversampling method has large randomness in a sample synthesis process. Firstly, a natural neighborhood algorithm is utilized to identify and delete abnormal points in minority classes so as to reduce interference of noise samples. Then, the number of new samples generated by each minority class sample is reasonably distributed by calculating a variable coefficient between the minority class points and natural neighbors around the minority class points; and finally, based on the distance relationship between the samples, distributing the number of generated new samples, and generating more new samples from the samples which are far away from each other, thereby better covering the whole feature space. According to the method, the representativeness of minority class samples can be effectively improved, the generated synthetic samples can better maintain the distribution characteristics of original data, and the problem that the randomness is large in a traditional method is solved.

Description

Technical field:

[0001] The present invention relates to an oversampling algorithm based on natural neighborhood weighting, which can ensure that the newly generated samples are reasonably distributed in space, while better retaining the local features of minority class samples, thereby improving the classification performance of the model. Background technology:

[0002] As time goes by, the characteristics of data often change, which causes the data itself to become unbalanced. In particular, in many applications, the collected data classes often present an unbalanced distribution. The main manifestation of class imbalance is that the number of instances in different classes is unequal. In order to solve this problem, researchers at home and abroad have mainly dealt with imbalanced data in two aspects, namely the data level and the algorithm level.

[0003] At the data level, the idea of ​​dealing with imbalanced data is to adjust the number of samples in the data set to make it more balanced by artificially deleting majority class instances or increasing minority class instances according to certain rules. Common strategies include undersampling, oversampling, and mixed sampling. Undersampling balances the data by reducing majority class samples, while oversampling achieves balance by increasing minority class samples. These methods aim to alleviate the imbalance between positive and negative class samples by adjusting the distribution of samples.

[0004] At the algorithm level, the main solutions include ensemble learning and cost-sensitive learning. Ensemble learning is a common classification method that improves overall performance by combining the prediction results of multiple classifiers. Ensemble learning has good adaptability to imbalanced data because it can effectively improve the accuracy of classification by adjusting the weights of each classifier so that the minority class gets more attention. Summary of the invention:

[0005] In order to solve the problems of noise propagation, boundary effect and uneven sample distribution encountered by the traditional SMOTE algorithm when generating new points, the present invention discloses an oversampling algorithm based on natural neighborhood weighting.

[0006] To this end, the present invention provides the following technical solutions:

[0007] 1. An oversampling algorithm based on natural neighbor weighting, characterized in that the method comprises the following steps:

[0008] Step 1: Use the natural neighbor algorithm to identify outliers in the minority class samples and remove them.

[0009] Step 1-1 sets the number of search rounds r to 1, that is, the minimum number of neighbors. Then, search for the 1 nearest neighbor of each sample, that is, the sample with the highest similarity to it. Next, determine whether these 1 nearest neighbors meet the condition: each sample has at least one mutual 1 nearest neighbor. If the condition is met, that is, each sample has at least one real neighbor, then a natural neighborhood is formed. If the condition is not met, it means that there are some samples that do not have mutual 1 nearest neighbors, then increase the value of r and enter the next round of search. In each round of search, find the r nearest neighbors of each sample, and again determine whether the condition that each sample has at least one mutual r nearest neighbor is met. When an r value is found so that each sample meets the condition, that is, each sample has at least one real neighbor, it is considered that a natural neighborhood has been formed, where λ is the eigenvalue of the natural neighborhood, and the specific formula is as follows:

[0010]

[0011] Among them, r is the number of search rounds, NN λ (x) represents the λ nearest neighbors of data object x.

[0012] Step 1-2 Find the sample x i If a point has a different category label from its natural neighbors, then the point is an outlier. The specific formula is as follows:

[0013]

[0014] Among them, AE is the set of outliers, l(x i ) represents x i The category label of the minority class is represented by 1, and the majority class sample is represented by 0. j ∈NaN(x i ) represents the sample x j is the sample x i natural neighbors.

[0015] Step 2: Determine the weight of each newly generated point by calculating the coefficient of variation between the minority class samples and their neighborhood points. The specific formula is as follows:

[0016]

[0017] Among them, c i For the minority class sample x i The coefficient of variation, l(x i ) represents x i The category label of the minority class sample is represented by 1, and the majority class sample is represented by 0. λ is the characteristic value of the natural neighborhood. The larger the coefficient of variation, the more dispersed the data is, so the larger the weight assigned is, and vice versa. N i(k) Represents the sample x iThe index of the kth nearest neighbor, Represents the sample point x i The average of the features of all minority class samples among its λ nearest neighbors.

[0018] Step 3: Calculate the total number of new samples that should be generated for all minority class samples. Set the balance ratio of the synthesized dataset to 1:1, and calculate the total number of new samples that need to be synthesized for the entire dataset. The specific formula is as follows:

[0019] T=T MAX -T min

[0020] Among them, T is the total number of new samples synthesized, T MAX is the number of majority class samples in the dataset, T min is the number of minority class samples in the dataset.

[0021] Step 4: Calculate the number of new samples that need to be synthesized for each minority class point based on the coefficient of variation. The specific formula is as follows:

[0022]

[0023] g i =T*p i

[0024] Among them, p i is the variation factor, c i is the minority class sample point x i The coefficient of variation, m is the total number of minority class sample points, g is i is the minority class sample point x i The total number of new samples generated, T is the total number of newly generated samples.

[0025] Step 5: Through step 4, we get the total number of new samples generated by each minority class sample. On this basis, further sample allocation is performed. The specific method is to calculate the distance between each minority class sample and other minority class samples around it, and then allocate the number of new samples generated according to these distances. This ensures that the generated new samples are reasonably distributed in space, while better maintaining the local characteristics of minority class samples and improving the classification effect of the model. The specific steps are as follows.

[0026] Calculate any minority class object x in the sample space D i Other minority class objects x in its natural neighborhood j (where j∈[1,λ]) is the number of newly generated points r i j , the calculation formula is as follows:

[0027]

[0028] Among them, d(x i , x j ) represents x i Neighbors with Nature j The distance between them, l(x i ) represents x i The category label of the minority class sample is represented by 1, and the majority class sample is represented by 0. i is the minority class sample point x i The total number of new samples generated, N i(r) Represents the sample x i The index of the nearest r-th neighbor.

[0029] Step 6: Use the improved SMOTE algorithm to generate new sample points. The specific steps are as follows.

[0030] Step 6-1 Interpolate and divide r equally according to the number of newly generated points i j +1 value, get the minority class object x i The difference factor diff i , the calculation formula is as follows:

[0031]

[0032] in,

[0033] r i j Represents point x i and x j The number of points generated between.

[0034] Step 6-2 Generate new points based on the interpolation factors obtained in the above steps. The generation formula is as follows:

[0035] x new =x i +|x i -x j |×k*diff ij

[0036] where k∈[1, r i j ],r i j Represents point x i and x j This means that the coordinates of each new point are obtained by i and x j The generated objects are generated by equidistant interpolation between them. This uniform generation method ensures that the created synthetic objects are i and xj Evenly distribute them to avoid being too close, too far or stacked.

[0037] Beneficial effects:

[0038] 1. The present invention is an oversampling algorithm based on natural neighborhood weighting. In order to solve the problems of noise propagation, boundary effect, and uneven sample distribution encountered by the traditional SMOTE algorithm when generating new points, the natural neighbor algorithm is used to delete noise points before generating new points, and the coefficient of variation is calculated to control the distribution of newly generated samples. The points generated in dense areas are fewer, and the points generated in sparse areas are more. On this basis, sample allocation is further performed. The specific method is to calculate the distance between each minority class sample and other minority class samples around it, and then allocate the number of new samples generated according to these distances. In this way, it can ensure that the generated new samples are reasonably distributed in space, while better maintaining the local characteristics of minority class samples and improving the classification effect of the model. Description of the drawings:

[0039] Figure 1 This is a schematic diagram of the improved SMOTE algorithm in an implementation manner of the present invention.

[0040] Figure 2 The figure is a flow chart of an oversampling algorithm based on natural neighbor weighting in an embodiment of the present invention. Specific implementation method:

[0041] In order to clearly and completely describe the technical solutions in the embodiments of the present invention, the present invention is further described in detail below in conjunction with the drawings in the embodiments.

[0042] The present invention implements an oversampling algorithm based on natural neighborhood weighting, comprising the following steps:

[0043] Step 1: Use the natural neighbor algorithm to detect outliers. First, set the number of search rounds, and set an initial search round number r = 1, which means that only one nearest neighbor of each sample is considered initially. For each sample, find its nearest neighbor and determine the mutual neighbor condition.

[0044] Step 2: Construct natural neighbors for sample x i and sample x j , determine whether they are neighbors. That is: if x i is x j neighbors, then x i It should also be x j If the mutual neighbor condition is not met, increase the number of search rounds r and recheck the neighbor relationship until a mutual neighbor that meets the conditions is found.

[0045] Step 3: Identify outliers. If there are samples with different class labels in the neighbors of a minority class sample, the sample is considered an outlier, and the set of outliers is AE.

[0046]

[0047] Among them, r is the number of search rounds, NN λ (x) represents the λ nearest neighbors of data object x.

[0048] Step 4: Remove outliers.

[0049] Step 5: Calculate the coefficient of variation and determine the weight of the new sample. The coefficient of variation reflects the dispersion of minority class samples in their neighborhood. The formula is obtained by calculating the difference between each minority class sample and the samples in its neighborhood. The larger the coefficient of variation, the more dispersed the sample is in space, and thus a higher weight is assigned. Conversely, a lower weight is assigned. The formula for calculating the coefficient of variation is as follows.

[0050]

[0051] Among them, c i For the minority class sample x i The coefficient of variation, l(x i ) represents x i The category label of the minority class is represented by 1, and the majority class is represented by 0. λ is the characteristic value of the natural neighborhood. The larger the coefficient of variation, the more dispersed the data is, so the greater the weight assigned, and vice versa. i(k) Represents the sample x i The index of the kth nearest neighbor, Represents the sample point x i The average of the features of all minority class samples among its λ nearest neighbors.

[0052] Step 6: Calculate the number of new samples that need to be synthesized and set the balance ratio of the data set to 1:1, that is, the number of minority class samples is equal to the number of majority class samples.

[0053] Step 7: Allocate the generated new samples according to the coefficient of variation. The larger the coefficient of variation, the more new samples will be synthesized. The specific calculation formula is as follows.

[0054]

[0055] g i =T*p i

[0056] Among them, p i is the variation factor, c i is the minority class sample point x iThe coefficient of variation, m is the total number of minority class sample points, g is i is the minority class sample point x i The total number of new samples generated, T is the total number of newly generated samples.

[0057] Step 8: Through step 7, we get the total number of new samples generated by each minority class sample. In order to make the distribution of the newly generated sample points more even, on this basis, further sample allocation is performed. The specific method is to calculate the distance between each minority class sample and other minority class samples around it, and then allocate the number of new samples generated according to these distances. This ensures that the generated new samples are reasonably distributed in space, rather than concentrated in a certain area, and improves the classification effect of the model. The specific calculation formula is as follows.

[0058]

[0059] Among them, d(x i , x j ) represents x i Neighbors with Nature j The distance between them, l(x i ) represents x i The category label of the minority class sample is represented by 1, and the majority class sample is represented by 0. i is the minority class sample point x i The total number of new samples generated, N i(r) Represents the sample x i The most recent r-th index.

[0060] Step 9: Use the improved SMOTE algorithm to generate new sample points.

[0061] Step 9-1: According to each pair of samples (x i ) and (x j ) to generate new synthetic samples. New sample points are generated by interpolation (linear interpolation), and the difference factor diff i , the calculation formula is as follows:

[0062]

[0063] Among them, r i j Represents point x i and x j The number of points generated between.

[0064] Step 9-2 Through r i j The value of determines the number of points generated by this point and other points, and generates new points according to the interpolation factor obtained in step 9-1. The generation formula is as follows:

[0065] x new =x i +|x i -x j |×k*diff ij

[0066] where k∈[1, r i j ],r i j Represents point x i and x j This means that the position of each new point is obtained by i and x j The generated values ​​are interpolated in an equidistant manner. This uniform generation method ensures that the created synthetic object is i and x j Evenly distribute them to avoid being too close, too far or stacked.

[0067] In order to make the experimental results more accurate, the oversampling algorithm based on natural neighbor weighting is compared with the traditional SMOTE algorithm, and a classification effect confusion matrix is ​​introduced, where the value of TP is the number of samples that are actually in the minority class but predicted to be in the minority class, the value of FN refers to the number of samples that are actually in the minority class but predicted to be in the majority class, FP refers to the number of samples that are actually in the majority class but predicted to be in the minority class, and TN refers to the number of samples that are actually in the majority class but predicted to be in the majority class.

[0068] According to the confusion matrix, the ratio of correctly classified minority class samples to all predicted minority class samples can be obtained:

[0069] Precision:

[0070] The percentage of samples that are actually predicted to be correct for minority samples:

[0071] Recall rate:

[0072] The percentage of samples that are actually the majority class but are predicted incorrectly:

[0073] False Positive Rate:

[0074] The percentage of samples that are actually the majority class that are predicted correctly:

[0075] True negative rate:

[0076] F-value and geometric mean accuracy G-mean take into account the correct accuracy and recall rate. The formulas are defined as follows:

[0077]

[0078] The ROC curve is a curve formed by connecting the points formed by the coordinates formed by FRP as the horizontal axis and recall as the vertical axis. AUC represents the area value from the ROC curve to the coordinate axis. The larger the area, the higher the classification accuracy and the better the algorithm effect of the experiment.

[0079] The above is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, and the specific implementation methods herein are only used to help understand the method of the present invention. For those of ordinary skill in the art, according to the idea of ​​the present invention, changes and modifications can be made in the specific implementation methods and application scopes, so the present invention should not be understood as limiting the present invention.

Claims

1. An oversampling algorithm based on natural neighbor weighting, characterized in that: The method comprises the following steps: Step 1: Use the natural neighbor algorithm to identify outliers in the minority class samples and remove them. Step 2: Determine the weight of each newly generated point by calculating the coefficient of variation between the minority class samples and their neighborhood points. Step 3: Calculate the total number of new samples that should be generated for all minority class samples. Step 4: Calculate the number of new samples needed to synthesize each minority class sample based on the coefficient of variation. Step 5: Through step 4, we get the total number of new samples generated by each minority class sample. On this basis, further sample allocation is performed. The specific method is to calculate the distance between each minority class sample and other minority class samples around it, and then allocate the number of new samples generated according to these distances. This ensures that the generated new samples are reasonably distributed in space, while better maintaining the local characteristics of minority class samples and improving the classification effect of the model. Step 6: Use the improved SMOTE algorithm to generate new sample points.

2. The oversampling algorithm based on natural neighbor weighting according to claim 1, characterized in that: In step 1, natural neighborhood detection is performed on all samples, and the specific steps are as follows: Step 1-1 sets the number of search rounds r to 1, that is, the minimum number of neighbors. Then, search for the 1 nearest neighbor of each sample, that is, the sample with the highest similarity to it. Next, determine whether these 1 nearest neighbors meet the condition: each sample has at least one mutual 1 nearest neighbor. If the condition is met, that is, each sample has at least one real neighbor, then a natural neighborhood is formed. If the condition is not met, it means that there are some samples that do not have mutual 1 nearest neighbors, then increase the value of r and enter the next round of search. In each round of search, find the r nearest neighbors of each sample, and again determine whether the condition that each sample has at least one mutual r nearest neighbor is met. When an r value is found so that each sample meets the condition, that is, each sample has at least one real neighbor, it is considered that a natural neighborhood has been formed, where λ is the eigenvalue of the natural neighborhood, and the specific formula is as follows: Among them, r is the number of search rounds, NN λ (x) represents the λ nearest neighbors of data object x. Step 1-2 Find the sample x i If a point has a different category label from its natural neighbors, then the point is an outlier. The specific formula is as follows: Among them, AE is the set of outliers, l(x i ) represents x i The category label of the minority class is represented by 1, and the majority class is represented by 0. j ∈NaN(x i ) represents the sample x j is the sample x i natural neighbors.

3. The oversampling algorithm based on natural neighbor weighting according to claim 1, characterized in that: In step 2, the coefficient of variation of the minority class samples and their neighborhood points is calculated to determine the weight of each newly generated point. The specific formula is as follows: Among them, c i For the minority class sample x i The coefficient of variation, l(x i ) represents x i The category label, the minority class is represented by 1, the majority class is represented by 0, λ is the eigenvalue of the natural neighborhood, Ni (k) Represents the sample x i The index of the nearest k-th natural neighbor, Represents the sample point x i Among its λ nearest neighbors, the average value of the features of all minority class samples, the larger the coefficient of variation, the more dispersed the data is, so the larger the assigned weight is, and vice versa.

4. The oversampling algorithm based on natural neighbor weighting according to claim 1, characterized in that: In step 3, the balance ratio of the synthesized data set is set to 1:1, and the total number of new samples that need to be synthesized in the entire data set is calculated. The specific formula is as follows: T=T MAX -T min Among them, T is the total number of new samples synthesized, T MAX is the number of majority class samples in the dataset, T min is the number of minority class samples in the dataset.

5. The oversampling algorithm based on natural neighbor weighting according to claim 1, characterized in that: In step 4, the number of new class samples that need to be synthesized for each minority class point is calculated according to the coefficient of variation. The specific formula is as follows: g i =T*p i Among them, p i is the variation factor, c i is the minority class sample point x i The coefficient of variation, m is the total number of minority class sample points, g is i is the minority class sample point x i The total number of new samples generated, T is the total number of newly generated samples.

6. The oversampling algorithm based on natural neighbor weighting according to claim 1, characterized in that: In step 5, the total number of new samples generated by each minority class point is further distributed, and the specific steps are as follows: Calculate any minority class object x in the sample space D i Other minority class objects x in its natural neighborhood j (where j∈[1,λ]) is the number of newly generated points r i j , the calculation formula is as follows: Among them, d(x i ,x j ) represents x i Neighbors with Nature j The distance between them, l(x i ) represents x i The category label of the minority class is represented by 1, and the majority class is represented by 0. i is the minority class sample point x i The total number of new samples generated, N i(r) Represents the sample x i The index of the nearest r-th natural neighbor.

7. The oversampling algorithm based on natural neighbor weighting according to claim 1, characterized in that: In step 6, the improved SMOTE method is used to synthesize new minority class samples. The specific steps are: Step 6-1 Interpolate and divide r equally according to the number of newly generated points i j +1 value, get the minority class object x i The difference factor diff i , the calculation formula is as follows: Among them, r i j Represents point x i and x j The number of points generated between. Step 6-2 Generate new points based on the interpolation factors obtained in the above steps. The generation formula is as follows: x new =x i +|x i -x j |×k*diff ij where k∈[1,r i j ],r i j Represents point x i and x j This means that the coordinates of each new point are obtained by i and x j The generated values ​​are interpolated in an equidistant manner. This uniform generation method ensures that the created synthetic object is i and x j Evenly distribute them to avoid being too close, too far or stacked.

Images