Large-scale data similar feature detection method based on inverted indexes

Abstract

The invention discloses a large-scale data similarity feature detection method based on inverted indexes. According to the method, feature data columns of the corresponding types are sampled, the corresponding inverted indexes are extracted, then the hash table is established for the inverted indexes and the features in the form of key value pairs, candidate feature subsets are generated, and therefore the purpose of feature set dimension reduction is achieved; and the features in the feature subset are combined after dimension reduction in pairs, a Pearson correlation coefficient algorithm and a non-repetitive counting method aiming at the numeric features and the category features are respectively applied to obtain correlation coefficients of feature pairs, a threshold value is set, anda result is outputted. According to the method, the defect that original feature sets need to be combined pairwise in the past is overcome, the calculation time can be reduced by one order of magnitude, and a large number of resources are saved; and meanwhile, accuracy and the recall rate can be kept at an extremely high level;.

Description

technical field [0001] The invention belongs to the field of machine learning and data mining, and relates to a feature similarity detection method in big data feature engineering, in particular to a large-scale data similarity feature detection method based on an inverted index. Background technique [0002] Feature Similarity Detection (Feature Similarity Detection) is a crucial link in the data mining process, and it is also a necessary process for machine learning model training. There are often a large number of similar features in the original data set, which will disperse the importance of features during the model training process, affect the selection of features, and affect the performance of the model; and increase unnecessary computing overhead and waste a lot of resources. [0003] At present, most of the main feature similarity detection methods need to traverse and combine all the features for analysis. When the size of the original feature set is large, the s...

AI Technical Summary

Problems solved by technology

There are often a large number of similar features in the original data set, which will disperse the importance of features during model training, affect feature selection, and affect model performance; and increase unnecessary computing overhead and waste a lot of resources

[0003] At present, most of the main feature similarity detection methods need to traverse and combine all the features for analysis. When the size of the original feature set is large, the size of the feature pair set obtained after combination will also be very large, which makes this method in The performance is not good on large-scale data sets; or the locality sensitive hash (Locality sensitive hash) method is used to reduce the dimension first and then analyze the similarity. The disadvantage of this method is that although the dimension of the data set is reduced, the method is currently Can only be applied to categorical features (including numerical one-hot encoded features), but cannot be applied to numerical features

Images