Optimization algorithm method and system based on K nearest neighbor
By performing Kmeans division and pseudo-test set generation on large-scale data sets, combining weight distance calculation and K value adjustment, the K nearest neighbor algorithm is optimized, the prediction deviation and noise impact caused by excessive data scale is solved, and the model is improved Accuracy and computing efficiency.
Patent Information
- Application Number
- CN202411306547.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2025-01-03
AI Technical Summary
In K nearest neighbor algorithm, as the scale of training sample data increases, the prediction deviation increases, resulting in a decrease in accuracy, and traditional K-Means algorithms have difficulties in initial cluster center selection and noise point processing.
By Kmeans partitioning the large-scale data set, decompose it into a subset that meets the scale standard, and introduce weight calculation distance, generate a pseudo-test set for post-processing, adjusting the K value to optimize model complexity and overfitting problems.
It improves the availability of K nearest neighbor algorithms under large-scale data, reduces the influence of noise points, avoids the neglect of irrelevant data, and enhances the accuracy and computing speed of the model.
Smart Images
Figure CN119249185A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of large-scale training sets, and in particular to an optimization algorithm method and system based on K nearest neighbors. Background Art
[0002] The K-Nearest Neighbors (KNN) algorithm is an intuitive and powerful tool in the field of machine learning. It can not only be applied to classification tasks, but also effectively handle regression problems. The core idea of the KNN algorithm is that "the closer you are to the red, the more you are to the black", that is, the category (or value) of a sample should be the same or similar to the category (or average value) of its nearest K neighbors in the feature space.
[0003] In the KNN algorithm, first, we need to determine a suitable K value, which represents the number of nearest neighbors we will consider when making predictions. Next, for a given test sample, the algorithm calculates the distance between the sample and all samples in the training set (common distance metrics include Euclidean distance, Manhattan distance, etc.). Then, based on these distances, the algorithm finds the K training samples closest to the test sample. Finally, for classification problems, the categories of these K nearest neighbors will determine the category of the test sample by majority voting (mode); for regression problems, the average or weighted average of the target values of these K neighbors is usually calculated as the prediction result.
[0004] K-nearest neighbor algorithm is one of the algorithms that can be used for classification and regression in machine learning. The principle of KNN is to compare the training sample set with a set of specified test sets, and divide the comparison results into the final model. This algorithm has a fast training time and ideal prediction effect. However, if there is a large-scale data set of training samples, it will increase the prediction deviation value and reduce the accuracy. Summary of the invention
[0005] The purpose of the present invention is to provide an optimization algorithm method and system based on K nearest neighbors to solve the problems raised in the above background technology.
[0006] To achieve the above object, the present invention provides the following technical solution: an optimization algorithm method based on K nearest neighbors, the method comprising the following steps:
[0007] Preprocessing of training data sets;
[0008] Post-processing of the test set;
[0009] Adjustment of K value.
[0010] Preferably, the specific operations of preprocessing the training data set include:
[0011] Before training, the size of the data set is divided, and the data set that exceeds the specified size is divided into multiple data sets that meet the size standard using the Kmeans method;
[0012] When segmenting, the number of segments is determined according to the size of the training set and the specified size, ensuring that the size of each data set after segmentation is lower than the specified data set size; micro-processing is performed on irrelevant data sets, and Avoid situations where irrelevant datasets are overlooked.
[0013] Preferably, the large-scale data set is divided according to the Kmeans criterion, the centroid of each cluster is specified according to the quantity, the distance between each centroid and the data point is calculated, and the nearest K data are taken as the training set. The complexity of the KNN model is understood by its tolerance to noise;
[0014] If the model is sensitive to noise, the complexity of the model is very high; on the contrary, the complexity of the model is low; due to the different degrees of influence of data attributes on clustering results, weights are introduced on the basis of traditional Euclidean distance to increase the degree of distinction between different data attributes and eliminate the influence of noise points in the data set; effectively overcome the difficulties in selecting the initial cluster center in the traditional K-Means algorithm and the influence of noise points on clustering results, and improve clustering performance; the specific distance calculation method is as follows:
[0015]
[0016] Preferably, the specific operations of post-processing the test set include:
[0017] If the total amount of training data set is small, a pseudo test set is generated after each training of a training set, and the pseudo test set is regarded as a test set to train the remaining training sets; if the total amount of training data set is large, a pseudo test set is generated after each training of several training sets; the specific method of generating the pseudo test set is as follows:
[0018] Test the accuracy of the trained model. During the model training process, all parameters are modified and fitted based on the data in the existing training set, so there is a possibility of overfitting. Rounding is used to calculate the approximately equal sample size of the k-th source domain.
[0019] Preferably, the specific operation of adjusting the K value includes:
[0020] Based on the pseudo-test set generation method, each condition is averaged and then the value is taken; when the value of K is small, such as K = 1, only using training samples from a smaller domain for prediction will lead to stronger model fitting ability;
[0021] When the value of K is large, cross validation is used to evaluate each hyperparameter and select the best combination of parameters to build the model. The distance between the test set and the training set is calculated as the preferred distance. The two-dimensional partition is calculated using the Chebyshev distance to speed up the calculation.
[0022] dAB=max(|x1-x2|,|y1-y2|); Manhattan distance is used as a preferred method for multi-dimensional distance calculation.
[0023] A K-nearest neighbor optimization algorithm system, the system comprising:
[0024] Preprocessing module, used for preprocessing the training data set;
[0025] Post-processing module, used for post-processing of the test set;
[0026] The adjustment module is used to adjust the K value.
[0027] Preferably, the preprocessing module divides the size of the data set before training, and uses the Kmeans method to divide the data set that exceeds the specified size into multiple data sets that meet the size standard; when dividing, the number of divisions is determined according to the training set size and the specified size, ensuring that the size of each data set after division is lower than the specified data set size; micro-processing is performed on irrelevant data sets, and Avoid situations where irrelevant datasets are overlooked.
[0028] Preferably, the large-scale data set is divided according to the Kmeans criterion, the centroid of each cluster is specified according to the quantity, the distance between each centroid and the data point is calculated, and the nearest K data are taken as the training set. The complexity of the KNN model is understood by its tolerance to noise; if the model is sensitive to noise, the complexity of the model is very high; on the contrary, the complexity of the model is low; due to the different degrees of influence of data attributes on clustering results, weights are introduced on the basis of traditional Euclidean distance to increase the degree of distinction between different attributes of the data and eliminate the influence of noise points in the data set; effectively overcome the difficulties in selecting the initial cluster center in the traditional K-Means algorithm and the influence of noise points on the clustering results, and improve the clustering performance; the specific distance calculation method is as follows:
[0029]
[0030] Preferably, if the total amount of the training data set is low, a pseudo test set is generated after each training of a training set, and the pseudo test set is regarded as a test set to train the remaining training sets; if the total amount of the training data set is high, a pseudo test set is generated after each training of several training sets; the specific method for generating the pseudo test set is as follows: to test the accuracy of the training model, during the model training process, all parameters are modified and fitted based on the data in the existing training set, so there is a possibility of overfitting, and rounding is used to calculate the approximately equal sample size of the kth source domain.
[0031] Preferably, the adjustment module takes the value after averaging each condition based on the method of generating a pseudo test set; when the value of K is small, for example, K=1, using only training samples from a smaller domain for prediction will result in a stronger model fitting ability; when the value of K is large, cross-validation is used to evaluate each hyperparameter, and the best combination of parameters is selected to establish a model. As the preferred calculation of the distance between the test set and the training set, the two-dimensional partition is calculated using the Chebyshev distance to speed up the calculation, dAB=max(|x1-x2|,|y1-y2|); as the preferred calculation of multi-dimensional distance, Manhattan distance is used.
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] The K-nearest neighbor-based optimization algorithm method and system proposed in the present invention improve the availability of the original K-nearest neighbor algorithm. For a large-scale input data set, it is decomposed into several data sets that meet the requirements and trained according to the Kmeans algorithm, avoiding the disadvantage that other small-scale training sets are affected by the large data scale.
[0034] A pseudo-test set is added to the test set, and the probability of irrelevant data appearing in the subsequent training set is reduced by targeted broadening of conditions. Micro-processing is used on irrelevant data so that irrelevant data will be identified but will not affect the experimental results. The K value is adjusted so that the qualified results can be calculated more accurately for small-scale training sets. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0036] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0037] For example, see Figure 1 The present invention provides a technical solution: an optimization algorithm method based on K nearest neighbors, the method comprising the following steps:
[0038] (I) Preprocessing of training data set
[0039] Before training, the data set will be divided into different sizes. Data sets that exceed the specified size will be divided into multiple data sets that meet the size standard using the Kmeans method. When dividing, the number of divisions needs to be determined based on the training set size and the specified size to ensure that the size of each data set after division is lower than the specified data set size. Irrelevant data sets are micro-processed and added. Avoid situations where irrelevant datasets are overlooked.
[0040] Preferably, the large-scale data set is divided according to the Kmeans criterion, the centroid of each cluster is specified according to the quantity, the distance between each centroid and the data point is calculated, and the nearest K data are taken as the training set. The complexity of the KNN model can be understood by its tolerance to noise. If the model is sensitive to noise, the complexity of the model is very high; on the contrary, the complexity of the model is low. Since the data attributes have different degrees of influence on the clustering results, weights are introduced on the basis of the traditional Euclidean distance, which increases the degree of distinction between different attributes of the data and eliminates the influence of noise points in the data set. This algorithm effectively overcomes the difficulties in selecting the initial cluster center in the traditional K-Means algorithm and the influence of noise points on the clustering results, and improves the clustering performance. The specific distance calculation method is as follows:
[0041]
[0042] (II) Post-processing of the test set
[0043] If the total amount of training data set is small, a pseudo test set can be generated after each training of a training set, and the pseudo test set can be regarded as a test set to train the remaining training sets; if the total amount of training data set is large, a pseudo test set can be generated after each training of several training sets. The specific method of generating a pseudo test set is as follows:
[0044] Test the accuracy of the trained model. Because all parameters are modified and fitted based on the data in the existing training set during model training, there is a possibility of overfitting, which means that the parameter only accurately fits the information in the training set. If the model results need to be used to predict new data, the accuracy may be very poor. According to the principle of discarding the remainder and repeating the sample to make up for the shortfall, ensure the size of the first few subsets, and the last subset includes all residuals, using rounding to calculate the approximately equal sample size of the k-th source domain. The amount of training set data usually accounts for 2 / 3 to 4 / 5. In practical applications, based on the size of the entire data set, the division ratio between the training set data and the test set data can be 6:4 to 8:2. In the case of a large amount of data, 9:1 to 99:1 can be used.
[0045] (III) Adjustment of K value
[0046] Based on the above method of generating a pseudo test set, each condition can be averaged and then valued. When the value of K is small (for example, K = 1), using only training samples from a smaller domain for prediction will result in stronger model fitting ability. As long as the decision closely follows the results of the nearest training samples (neighbors), the decision can be made. However, when the training set contains noise samples, the model is also easily affected by these noise samples, which increases the variance of learning and is prone to overfitting. In this case, listening to the opinions of other adjacent training samples can minimize the impact of these noises.
[0047] When the value of K is too large, the opposite is true, and insufficient filling is likely to occur. The value of K is adjusted after each pseudo test set is generated for the next training. K needs to be manually specified as a hyperparameter. However, the manual process is quite cumbersome, so this method requires presetting several hyperparameter combinations for the model. Use cross-validation to evaluate each hyperparameter and select the best combination of parameters to build the model.
[0048] As the preferred method for calculating the distance between the test set and the training set, the two-dimensional partition is calculated using the Chebyshev distance, which can speed up the calculation.
[0049] dAB=max(|x1-x2|,|y1-y2|) is preferably used as a multi-dimensional distance calculation, using Manhattan distance.
[0050] Embodiment 2, based on embodiment 1, proposes a K-nearest neighbor-based optimization algorithm system according to the K-nearest neighbor-based optimization algorithm method according to any one of claims 1 to 5, the system comprising:
[0051] The preprocessing module is used to preprocess the training data set. Before training, the data set is divided into different sizes. The data set that exceeds the specified size is divided into multiple data sets that meet the size standard using the Kmeans method. When dividing, the number of divisions is determined according to the training set size and the specified size to ensure that the size of each data set after division is lower than the specified data set size. The irrelevant data sets are micro-processed and added. Avoid situations where irrelevant datasets are overlooked.
[0052] The large-scale data set is divided according to the Kmeans criterion. The centroid of each cluster is specified according to the quantity, the distance between each centroid and the data point is calculated, and the nearest K data are taken as the training set. The complexity of the KNN model is understood by its tolerance to noise; if the model is sensitive to noise, the complexity of the model is very high; on the contrary, the complexity of the model is low; due to the different degrees of influence of data attributes on clustering results, weights are introduced on the basis of traditional Euclidean distance to increase the degree of distinction between different attributes of the data and eliminate the influence of noise points in the data set; effectively overcome the difficulties in selecting the initial cluster center in the traditional K-Means algorithm and the influence of noise points on the clustering results, and improve clustering performance; the specific distance calculation method is as follows:
[0053]
[0054] The post-processing module is used for post-processing the test set. If the total amount of the training data set is low, a pseudo test set is generated after each training of a training set, and the pseudo test set is regarded as a test set to train the remaining training sets. If the total amount of the training data set is high, a pseudo test set is generated after each training of several training sets. The specific method of generating the pseudo test set is as follows: to test the accuracy of the training model, during the model training process, all parameters are modified and fitted based on the data in the existing training set, so there is a possibility of overfitting, and rounding is used to calculate the approximately equal sample size of the kth source domain.
[0055] The adjustment module is used to adjust the K value. Based on the method of generating a pseudo test set, each condition is averaged and then the value is taken. When the value of K is small, for example, K=1, only using training samples from a smaller domain for prediction will lead to stronger model fitting ability. When the value of K is large, cross-validation is used to evaluate each hyperparameter, and the best combination of parameters is selected to establish the model. As the preferred calculation of the distance between the test set and the training set, the two-dimensional partition uses the Chebyshev distance calculation to speed up the calculation speed, dAB=max(|x1-x2|,|y1-y2|); as the preferred calculation of multi-dimensional distance, Manhattan distance is used.
[0056] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. An optimization algorithm method based on K nearest neighbors, characterized in that: The method comprises the following steps: Preprocessing of training data sets; Post-processing of the test set; Adjustment of K value.
2. The K-nearest neighbor optimization algorithm according to claim 1, characterized in that: The specific operations of preprocessing the training data set include: Before training, the size of the data set is divided, and the data set that exceeds the specified size is divided into multiple data sets that meet the size standard using the Kmeans method; When segmenting, the number of segments is determined according to the size of the training set and the specified size, ensuring that the size of each data set after segmentation is lower than the specified data set size; micro-processing is performed on irrelevant data sets, and Avoid situations where irrelevant datasets are overlooked.
3. The K-nearest neighbor optimization algorithm according to claim 2, characterized in that: The large-scale data set is divided according to the Kmeans criterion. The centroid of each cluster is specified according to the quantity, the distance between each centroid and the data point is calculated, and the nearest K data are taken as the training set. The complexity of the KNN model is understood by its tolerance to noise; If the model is sensitive to noise, the complexity of the model is high; On the contrary, the complexity of this model is relatively low. Due to the different degrees of influence of data attributes on clustering results, weights are introduced on the basis of traditional Euclidean distance to increase the degree of distinction between different data attributes and eliminate the influence of noise points in the data set. It effectively overcomes the difficulties in selecting the initial cluster center in the traditional K-Means algorithm and the influence of noise points on clustering results, and improves clustering performance. The specific distance calculation method is as follows:
4. The K-nearest neighbor optimization algorithm according to claim 1, characterized in that: The specific operations of post-processing the test set include: If the total amount of training data set is small, a pseudo test set is generated after each training of a training set, and the pseudo test set is regarded as a test set to train the remaining training sets; if the total amount of training data set is large, a pseudo test set is generated after each training of several training sets; the specific method of generating the pseudo test set is as follows: Test the accuracy of the trained model. During the model training process, all parameters are modified and fitted based on the data in the existing training set, so there is a possibility of overfitting. Rounding is used to calculate the approximately equal sample size of the k-th source domain.
5. The K-nearest neighbor optimization algorithm according to claim 1, characterized in that: The specific operations for adjusting the K value include: Based on the pseudo-test set generation method, each condition is averaged and then the value is taken; when the value of K is small, such as K = 1, only using training samples from a smaller domain for prediction will lead to stronger model fitting ability; When the value of K is large, cross validation is used to evaluate each hyperparameter and select the best combination of parameters to build the model. The distance between the test set and the training set is calculated as the preferred distance. The two-dimensional partition is calculated using the Chebyshev distance to speed up the calculation. dAB=max(|x1-x2|,|y1-y2|); Manhattan distance is used as a preferred method for multi-dimensional distance calculation.
6. A K-nearest neighbor based optimization algorithm system according to the K-nearest neighbor based optimization algorithm method according to any one of claims 1 to 5, characterized in that: The system comprises: Preprocessing module, used for preprocessing the training data set; Post-processing module, used for post-processing of the test set; The adjustment module is used to adjust the K value.
7. The K-nearest neighbor optimization algorithm system according to claim 6, characterized in that: The preprocessing module divides the size of the data set before training. The data set that exceeds the specified size is divided into multiple data sets that meet the size standard using the Kmeans method; when dividing, the number of divisions is determined according to the training set size and the specified size to ensure that the size of each data set after division is lower than the specified data set size; micro-processes irrelevant data sets and adds Avoid situations where irrelevant datasets are overlooked.
8. The K-nearest neighbor optimization algorithm system according to claim 7, characterized in that: The large-scale data set is divided according to the Kmeans criterion. The centroid of each cluster is specified according to the quantity, the distance between each centroid and the data point is calculated, and the nearest K data are taken as the training set. The complexity of the KNN model is understood by its tolerance to noise; if the model is sensitive to noise, the complexity of the model is high; On the contrary, the complexity of this model is relatively low. Due to the different degrees of influence of data attributes on clustering results, weights are introduced on the basis of traditional Euclidean distance to increase the degree of distinction between different data attributes and eliminate the influence of noise points in the data set. It effectively overcomes the difficulties in selecting the initial cluster center in the traditional K-Means algorithm and the influence of noise points on clustering results, and improves clustering performance. The specific distance calculation method is as follows:
9. The K-nearest neighbor optimization algorithm system according to claim 6, characterized in that: The post-processing module, if the total amount of the training data set is low, generates a pseudo test set after each training of a training set, and regards the pseudo test set as a test set to train the remaining training sets; if the total amount of the training data set is high, generates a pseudo test set after each training of several training sets; the specific generation method of the pseudo test set is as follows: to test the accuracy of the training model, during the model training process, all parameters are modified and fitted based on the data in the existing training set, so there is a possibility of overfitting, and rounding is used to calculate the approximately equal sample size of the kth source domain.
10. The K-nearest neighbor optimization algorithm system according to claim 6, characterized in that: The adjustment module takes the average value of each condition based on the pseudo test set generation method; When the value of K is small, such as K=1, using only training samples from a smaller domain for prediction will result in stronger model fitting ability; when the value of K is large, cross-validation is used to evaluate each hyperparameter and the best combination of parameters is selected to build the model. As the preferred calculation of the distance between the test set and the training set, the two-dimensional partition is calculated using the Chebyshev distance to speed up the calculation, dAB=max(|x1-x2|,|y1-y2|); as the preferred calculation of multi-dimensional distance, Manhattan distance is used.