Unbalanced data oversampling method based on sample importance

Through the imbalanced data oversampling method based on the importance of samples, the degree of influence is calculated and the new samples are synthesized, which solves the problem that a few types of samples are ignored in imbalanced data processing in traditional technology, and improves the generalization ability and classification of the model ability, reduces overfitting.

CN119089197APending Publication Date: 2024-12-06HAINAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411053685.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

When processing unbalanced data in the prior art, traditional learning algorithms tend to ignore the importance of a few types of samples, resulting in a decrease in model generalization and performance capabilities, and it is easy to overfit, making it difficult to effectively improve recognition performance.

Method used

Using an imbalanced data oversampling method based on sample importance, the influence of samples is calculated through neural network models, high-quality samples are selected for oversampling, and new samples are synthesized using interpolation method, combined with K nearest neighbor algorithm and random interpolation, and inserted into a few Class sample set, filtering synthetic samples to reduce noise effects.

Benefits of technology

The quality of a few types of sample data is improved, the overfitting phenomenon of the model is reduced, and the generalization and classification capabilities of the model are enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089197A_ABST
    Figure CN119089197A_ABST
Patent Text Reader

Abstract

The invention discloses an unbalanced data oversampling method based on sample importance, which is characterized by comprising a process of determining an original data set, a process of determining samples needing to be oversampled in a minority of sample sets, and a process of synthesizing new samples and inserting the new samples into a sample set to which the samples needing to be oversampled belong. According to the method, the influence of noise data can be reduced, the quality of minority class sample data is improved, and the over-fitting phenomenon of the model is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Invention Name

[0002] The present invention relates to the technical field of data processing, and in particular to an unbalanced data oversampling method based on sample importance. Background Art

[0003] Machine learning and artificial intelligence have made rapid progress in computer vision tasks such as image recognition and classification, video analysis, and target detection. These developments require the availability of data sets. For example, CI FAR10 and CI FAR100 data sets have sufficient and similar number of samples in each category. However, data sets in the real world are unbalanced, so data imbalance has become a research focus for computer vision tasks.

[0004] Data imbalance refers to the existence of multiple data categories in a data set, and the number of each category is very different. When the number of majority class samples far exceeds that of minority class samples, an imbalance problem will occur. Data imbalance can be divided into long-tail imbalance and step imbalance. Long-tail imbalance refers to the number of each data category decreasing from large class samples to small class samples. As the number of sample categories increases, the number of minority class samples will decrease sharply, and in many cases the number of minority class samples is in the single digit. Step imbalance is not common in the real world. It divides the data set into multiple groups, each group contains multiple categories, and the number of each data category in the group is very small. The number of categories between different groups is different and there is a certain degree of multiple relationship. In our real life, data sets usually exist in a long-tail imbalanced way. Studying long-tail imbalance is an important research point of data imbalance.

[0005] In machine learning, traditional learning algorithms use overall accuracy as the learning goal, which ignores the importance of minority class samples and makes the training results oriented to majority class samples. However, learning and training without processing the data set and learning algorithm will not only reduce the generalization ability of the model, but also reduce the performance in minority class samples. In actual application scenarios, in many cases the value of minority class samples is higher than that of majority class samples. In applications such as medical diagnosis, software defect detection and fraud detection, network intrusion monitoring, and fault diagnosis, the important value of minority class samples is reflected. Using imbalanced data to improve recognition performance poses a huge challenge to modern deep learning methods. Therefore, studying multi-category long-tail imbalanced data classification has important practical significance.

[0006] To solve the problem of learning and classification of imbalanced data, existing research programs are divided into two technical levels: (1) Algorithm level: Algorithm level methods focus on minority class samples by modifying classifiers, setting weights, improving loss functions, etc. Or adjust the model training process through a penalty mechanism to improve the training strategy of small class samples. However, these methods lack flexibility and are prone to overfitting of the model. (2) Data level: The main methods are undersampling, oversampling, and hybrid sampling that combines these two sampling methods. This type of method mainly starts from the perspective of data and uses various methods to achieve theoretical balance between various types of data, thereby solving the problem of imbalanced data. This method is not only easy to understand and efficient, but also can retain the original feature information of the data. It is considered to be a method that can solve the problem of data imbalance. Summary of the invention

[0007] To solve the above problems, the present invention provides an unbalanced data oversampling method based on sample importance.

[0008] The present invention relates to the field where imbalanced data processing, classifier training and model optimization are required. Imbalanced data classification has a wide range of applications in scenarios such as disease monitoring, vulnerability detection, video analysis and target detection. The present invention can reduce the impact of noise data, improve the quality of minority class sample data, and reduce the overfitting phenomenon of the model.

[0009] The method comprises:

[0010] Determine the process of the original data set; the original data set includes a minority sample set and a majority sample set. The original data set includes several samples and the true labels and weights corresponding to each sample;

[0011] A process for determining samples that need to be oversampled in a minority sample set; This process is used to identify samples that need to be oversampled from a minority sample set based on sample importance;

[0012] The process of synthesizing new samples and inserting them into the sample set to which the samples that need to be oversampled belong; the process is used to synthesize new samples by interpolation and insert them into the sample set to which the samples that need to be oversampled belong.

[0013] Furthermore, the process of determining the samples that need to be oversampled in the minority sample set includes the following steps:

[0014] Step 2.1: Pre-select the neural network model function f(x,w), and use the samples in the minority sample set, their weights, and true labels as its input;

[0015] Step 2.2: Calculate the relationship between the predicted label and the actual label of each sample;

[0016] Let the sample to be calculated be x i, the relationship is calculated as follows:

[0017] L(y i ,f(x,w))=-∑ t y t,i logf t,i (1)

[0018] Among them, L is the true label y of the sample i The relationship function between the labels predicted by the neural network model function f(x,w) is calculated as the loss value vector, y t,i Represents the input sample x i The category predicted by the neural network function is t, f t,i is the neural network model function predicting sample x i The probability of belonging to category t;

[0019] Step 2.3: Use the differential function to calculate the gradient of each sample and each dimension with respect to the loss value vector. The calculation formula is as follows:

[0020]

[0021] Formula (2) can be used to obtain sample x i The gradient in dimension j;

[0022] Step 2.4: Accumulate the product of the gradient of each dimension of the sample and the weight vector of that dimension to obtain the total gradient value of the sample. The calculation formula is as follows:

[0023]

[0024] in, Represents sample x i The partial derivative in the jth feature dimension, p is the number of dimensions of the sample, w i,j Represents the weight of the j-th feature dimension;

[0025] Step 2.5: Compare the total gradient value of each sample with the influence threshold, and the samples with a value greater than the influence threshold are used as the initial samples for oversampling.

[0026] Furthermore, the process of synthesizing new samples and inserting them into the sample set to which the samples that need to be oversampled belong is as follows:

[0027] Step 3.1: Select a sample and use the K nearest neighbor algorithm to evaluate the similarity between samples. The calculation formula is as follows:

[0028]

[0029] Among them, a and b represent any two samples in the same sample category, n here represents the feature dimension of the sample, ai 、b i is the i-th dimension of a and b;

[0030] Step 3.2: Get the k neighboring samples closest to the selected sample; k is a predetermined integer;

[0031] Step 3.3: Use random interpolation to synthesize new samples between the selected sample and its k neighboring samples;

[0032] Assume that the selected sample is x i , for one of the nearest neighbor samples x j , the formula for synthesizing new samples is:

[0033]

[0034] in, is a randomly generated weight, xs ij Indicates that in sample x i and x j New samples are generated by interpolation.

[0035] Furthermore, the value of k is 3 or 5.

[0036] Furthermore, after the process of synthesizing new samples and inserting them into the sample set to which the samples that need to be oversampled belong is executed, it is also necessary to execute the process of filtering the synthesized samples.

[0037] Furthermore, the specific method of filtering the synthetic samples is:

[0038] Step 3.1: Calculate the relationship between the predicted label and the actual label of each synthetic sample;

[0039] Step 3.2: Calculate the gradient of each dimension of each synthetic sample with respect to the loss value vector;

[0040] Step 3.3: Accumulate the gradient of each dimension of each synthetic sample and multiply it by the weight vector of the synthetic sample to obtain the total gradient value of the synthetic sample;

[0041] Step 3.4: Compare the total gradient value of each synthetic sample with the influence threshold, and filter out the synthetic samples whose total gradient value does not meet the predetermined requirement.

[0042] The technical effects of the present invention are:

[0043] 1. The influence function calculation model is used to realize the selection of initial samples and the filtering of synthetic samples; the sample oversampling algorithm is used to oversample the minority class samples to increase the number of minority class sample data.

[0044] 2. It provides a new influence calculation model, which filters samples by calculating the loss and gradient of the model and combining the sample weights to reduce the impact of noise samples and overfitting, and improve the generalization and classification capabilities of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the present invention.

[0046] Figure 2 Select a schematic diagram for the sample to be improved.

[0047] Figure 3 Schematic diagram of sample synthesis. DETAILED DESCRIPTION

[0048] The present invention can be divided into four processes: the first is the process of determining the original data set; the second is the process of determining the samples that need to be oversampled in the minority sample set; the third is the process of synthesizing new samples and inserting them into the sample set to which the samples that need to be oversampled belong; and the fourth is the process of filtering the synthesized samples.

[0049] Combine the following Figure 1-3 Describe each process.

[0050] 1. Process of determining the original data set

[0051] In general, a data set contains multiple categories, and the number of samples in each category is also different. The categories with fewer samples are called minority classes. The imbalance of the data set is measured by the IR value, IR = the number of samples in the majority class with the largest number / the number of samples in the minority class with the smallest number. Generally, if IR is greater than or equal to 10, the data set is considered to be unbalanced. To determine whether a category of data is a minority class, calculate IR = the number of samples in the majority class with the largest number / the number of samples in the category to be determined. If IR is greater than 10, the category is considered to be a minority class sample. The specific identification criteria are not limited to the above and can be set as needed.

[0052] Let S(X,Y) be the original data set, which is the sum of the minority sample set and the majority sample set, and has n training samples (x1,y1),…,(x i ,y i ),…,(x n ,y n ). X represents the sample set, x i is the sample in set X, Y is the true label set of the sample, y i Represents sample x i The true label of each y i There is an x i Correspondingly, suppose the original data set has a total of m sample categories, and the j-th sample category has nj samples, then there are

[0053] In addition, each sample has a weight vector w i ,The weight represents the importance of the sample, which can be generated by the algorithm or set manually.

[0054] Sample x i It has multiple features and is distributed in different dimensions. In order to more comprehensively reflect the importance of the sample to the output of the overall neural network model, it is necessary to comprehensively consider the multiple dimensions of the sample. i Distributed in p different dimensions, where sample x i The feature representation under dimension j is x i,j Each dimension of the sample has a corresponding weight, and the weight under dimension j is represented by w i,j .

[0055] 2. Process of determining samples that need to be oversampled in the minority sample set

[0056] This process is used to identify samples that need to be oversampled from a small number of sample sets based on sample importance. The purpose is to use high-quality samples as initial samples to improve the quality of synthetic samples.

[0057] By inputting the sample into the neural network, the predicted label of the sample can be obtained. The loss value between the predicted label and the actual label is calculated based on the loss function, so that the gradient of each dimension of the sample can be calculated. The influence of the sample can be calculated based on the sample gradient and weight. It can guide the algorithm to synthesize high-quality samples, thereby improving the performance of the model in minority class samples. The importance of each sample is measured by the influence, which is calculated by the sample weight and gradient. The initial sample selection process is as follows: Figure 1 shown.

[0058] The specific method for selecting samples that need to be oversampled is as follows.

[0059] Step 2.1: Pre-select the neural network model function, and use the samples in the minority sample set, their weights, and true labels as its input.

[0060] This step uses the neural network model function in the prior art and is represented by f(x,w). The input parameter w is the sample weight and x is the input sample. Assume that one of the minority class data sets numbered i is M(x,y i ), where x is the minority class data set, y i is the true label of the minority class (samples of the same class use the same label).

[0061] Step 2.2: Calculate the relationship between the predicted label and the actual label of each sample.

[0062] The loss function uses softmax to calculate the loss between the neural network model function prediction and the true value, and calculates the relationship between the predicted label and the actual label corresponding to each sample, as shown in formula (1).

[0063] L(y i ,f(x,w))=-∑ t y t,i logf t,i (1)

[0064] Among them, L is the true label y of the sample i The relationship function between the labels predicted by the neural network function f(x,w) is calculated as the loss value vector, y t,i Represents the input sample x i The predicted label of category t predicted by the neural network function, f t,i is the neural network model function predicting sample x i,j The probability of belonging to category t, where i may be equal to t or not. t is the predicted sample category.

[0065] Step 2.3: Calculate the gradient of the loss value for each dimension of each sample using the differential function. The calculation formula is as follows:

[0066]

[0067] Step 2.4: Accumulate the product of the gradient of each dimension of the sample and the weight vector of that dimension to obtain the total gradient value of the sample. The calculation formula is as follows:

[0068]

[0069] in, Represents sample x i The partial derivative in the jth feature dimension, p is the number of dimensions of the sample, w i,j Represents the weight of the j-th feature dimension.

[0070] Step 2.5: Compare the total gradient value of each sample with the influence threshold, and the samples with a value greater than the influence threshold are used as the initial samples for oversampling.

[0071] The IB calculated by formula 3 is a numerical value, based on which we can further judge whether the sample is important. The influence threshold δ is set in advance, such as selecting samples with an IB value greater than δ as the initial sample. Of course, we can also select synthetic samples with an IB value equal to δ as needed.

[0072] The influence threshold δ can be set according to actual needs. For example, when there are very few minority samples, a smaller value can be set to synthesize more new samples.

[0073] 3. The process of synthesizing new samples and inserting them into the sample set to which the samples that need to be oversampled belong

[0074] This process is an important method to balance the minority class samples. The technical idea is to synthesize new samples through interpolation to fill in the minority class samples.

[0075] The principle of synthesizing new samples is: take a selected minority class sample that needs to be oversampled as a point, find a minority class sample in its K nearest neighbors as another point, and then randomly synthesize new samples on the line connecting the two points. The sample synthesis process is as follows: Figure 2 shown.

[0076] For each selected nearest neighbor sample, the present invention generates a new synthetic sample by random interpolation. The interpolation weight is controlled by a random number to ensure that the generated synthetic sample is smooth in the sample space.

[0077] This process includes the following steps:

[0078] Step 3.1: Select a sample and use the K nearest neighbor algorithm to evaluate the similarity between samples.

[0079] The K nearest neighbor algorithm is used to evaluate the similarity between samples and guide the synthesis of samples. The choice of K value will affect the quality of the synthesized samples to ensure that the synthesized samples are flexible. It is ensured that the synthesized samples can retain some characteristics of the original samples. For each minority class sample, the K nearest neighbor is used to calculate the Euclidean distance between it and the samples of the same class, and its nearest k samples can be obtained. These samples are used in the subsequent synthesis sample generation process. The Euclidean distance is calculated as follows:

[0080]

[0081] Among them, a and b represent any two samples in the same sample category, n here represents the feature dimension of the sample, a i 、b i is the i-th dimension of a and b. It should be noted that the specific number of dimensions can be determined according to actual conditions.

[0082] Step 3.2: Get the k neighboring samples closest to the selected sample; k is a predetermined integer.

[0083] For samples of minority categories, samples with greater influence can be obtained according to the initial sample selection process and used as initial samples. Then, their k nearest neighbor samples are selected. The value of k can be selected as needed. Generally speaking, the larger the value of k, the more synthetic samples are generated, but it may cause overfitting; on the contrary, the smaller the value of k, the fewer synthetic samples are generated, but it may cause underfitting. A smaller k value is preferred, such as 3 or 5, and more preferably, k=5.

[0084] Step 3.3: Use random interpolation to synthesize new samples between the selected sample and its k neighboring samples;

[0085] Assume that the selected sample is x i According to formula (4), we can get the distance sample x i The nearest k samples, k is set to 5 in this embodiment, and new samples are generated by using random interpolation in these samples. Random interpolation can make the synthesized samples more flexible and help synthesize diverse samples. For each nearest neighbor sample x j , synthesize new samples through the following formula.

[0086]

[0087] Among them, x j is a neighbor sample selected by step 3.2, is a randomly generated weight with a value range of Used to control the degree of interpolation, xs ij Indicates that in sample x i and x j The new samples are interpolated between i 、x j 、xs ij belong to the same minority class.

[0088] Preferably, the present application also includes a process for filtering synthetic samples.

[0089] The newly synthesized samples are random and diverse. Since the synthesized samples may contain low-quality samples and noise samples, it is necessary to filter the samples. The principle of filtering the newly synthesized samples is that there are noise samples and samples with less information in the newly synthesized samples. To address this problem, the present invention further filters the newly synthesized samples. This process is achieved by calculating the influence of the newly synthesized samples, which can ultimately greatly improve the quality of the generated samples.

[0090] The purpose of this process is: (1) Improve the quality of generated samples. By calculating the influence of synthetic samples, filtering and deleting low-quality samples, the quality of synthetic samples can be improved. (2) Filtering noisy samples: Some synthetic samples have a negative impact on the model, and they need to be filtered out to reduce the negative impact on the model and avoid introducing unnecessary noise, thereby improving the stability and generalization ability of the model.

[0091] This embodiment uses the same method as the above sample influence calculation method to calculate the gradient calculation result of the new synthetic sample. ij , respectively identify the sample features in each dimension, and then input them into the neural network model function f(x,w), multiply the gradient of each dimension by the weight (the weight of the synthetic sample is randomly generated by the random generation function) and sum them up, and finally get the influence IB (xs ij ). The product of the gradient of each dimension of the synthetic sample and the weight vector of the dimension is accumulated to obtain the total gradient value of the synthetic sample; the total gradient value of each synthetic sample is compared with the influence threshold, and the synthetic samples whose total gradient value does not meet the predetermined requirements are filtered out. The filtered synthetic sample set is shown in Formula 6.

[0092] DF={(xs ij ,ys ij )∣IB(xs ij )>δ} (6)

[0093] Where IB(xs ij ) represents the sample xs i The influence degree of , DF is the filtered synthetic sample set, ys xj Represents the synthetic sample xs xj δ is the influence threshold, which is used to filter the synthetic samples and only retain the synthetic samples whose sample IB value is greater than δ. Of course, you can also choose to retain the synthetic samples equal to δ as needed. The influence threshold here can be set as needed and is not necessarily equal to the aforementioned influence threshold.

[0094] After filtering the synthetic samples, a new sample set with improved imbalance can be obtained.

Claims

1. A method for oversampling unbalanced data based on sample importance, characterized in that: The method comprises: Determine the process of the original data set; the original data set includes a minority sample set and a majority sample set. The original data set includes several samples and the true labels and weights corresponding to each sample; A process for determining samples that need to be oversampled in a minority sample set; This process is used to identify samples that need to be oversampled from a minority sample set based on sample importance; The process of synthesizing new samples and inserting them into the sample set to which the samples that need to be oversampled belong; the process is used to synthesize new samples by interpolation and insert them into the sample set to which the samples that need to be oversampled belong.

2. The unbalanced data oversampling method based on sample importance according to claim 1, characterized in that: The steps involved in the process of determining the samples that need to be oversampled in the minority sample set are: Step 2.1: Pre-select the neural network model function f(x,w), and use the samples in the minority sample set, their weights, and true labels as its input; Step 2.2: Calculate the relationship between the predicted label and the actual label of each sample; Let the sample to be calculated be x i , the relationship is calculated as follows: L(y i ,f(x,w))=-∑ t y t,i logf t,i (1) Among them, L is the true label y of the sample i The relationship function between the labels predicted by the neural network model function f(x,w), the result of which is the loss value vector; y t,i Represents the input sample x i The predicted label of category t predicted by the neural network function; f t,i is the neural network model function predicting sample x i The probability of belonging to category t; t is the predicted sample category; Step 2.3: Use the differential function to calculate the gradient of each sample and each dimension with respect to the loss value vector. The calculation formula is as follows: Formula (2) can be used to obtain sample x i The gradient in dimension j; Step 2.4: Accumulate the product of the gradient of each dimension of the sample and the weight vector of that dimension to obtain the total gradient value of the sample. The calculation formula is as follows: in, Represents sample x i The partial derivative in the jth feature dimension, p is the number of dimensions of the sample, w i,j Represents the weight of the j-th feature dimension; Step 2.5: Compare the total gradient value of each sample with the influence threshold, and the samples that are greater than the influence threshold are used as initial samples that need to be oversampled.

3. The unbalanced data oversampling method based on sample importance according to claim 2, characterized in that: The specific process of synthesizing new samples and inserting them into the sample set to which the samples that need to be oversampled belong is as follows: Step 3.1: Select a sample and use the K nearest neighbor algorithm to evaluate the similarity between samples. The calculation formula is as follows: Among them, a and b represent any two samples in the same sample category, n here represents the number of feature dimensions of the sample, a i , b i is the i-th dimension of a and b; Step 3.2: Get the k neighboring samples closest to the selected sample; k is a predetermined integer; Step 3.3: Use random interpolation to synthesize new samples between the selected sample and its k neighboring samples; Assume that the selected sample is x i , for one of the nearest neighbor samples x j , the formula for synthesizing new samples is: in, is a randomly generated weight, xs ij Indicates that in x i and x j New samples are generated by interpolation.

4. The unbalanced data oversampling method based on sample importance as claimed in claim 3, characterized in that: The value of k is 3 or 5.

5. The unbalanced data oversampling method based on sample importance according to any one of claims 1 to 4, characterized in that: After the process of synthesizing new samples and inserting them into the sample set to which the samples that need to be oversampled belong is executed, the process of filtering the synthesized samples needs to be executed.

6. The unbalanced data oversampling method based on sample importance according to claim 5, characterized in that: The specific method for filtering synthetic samples is: Step 3.1: Calculate the relationship between the predicted label and the actual label of each synthetic sample; Step 3.2: Calculate the gradient of each dimension of each synthetic sample with respect to the loss value vector; Step 3.3: Accumulate the product of the gradient of each dimension of the synthetic sample and the weight vector of the dimension to obtain the total gradient value of the synthetic sample; Step 3.4: Compare the total gradient value of each synthetic sample with the influence threshold, and filter out the synthetic samples whose total gradient value does not meet the predetermined requirement.