Data set partitioning method and device

Abstract

The invention provides a data set division method and device, and the method comprises the steps: determining a target clustering number of a to-be-divided data set, carrying out the clustering of the to-be-divided data set, and obtaining a clustering result, and the target clustering number is the number of the clustering results; and according to a preset division standard, dividing the clustering result into an effective clustering result and an invalid clustering result, and merging data in the invalid clustering result into the effective clustering result. According to the data set division method and device provided by the invention, the invalid clustering results are merged into the effective clustering results, so that the number of target clusters is reduced, and only the effective clustering results are reserved, thereby improving the efficiency of model training.

Description

technical field [0001] The invention relates to the field of data clustering, in particular to a data set division method and device. Background technique [0002] Using machine learning algorithms to identify malware has become one of the current research and development trends in the security field. When applying machine learning algorithms, it is necessary to continuously provide high-quality labeled samples for model training and updating. [0003] Sample collection platforms such as VT use simple construction rules to obtain rough data sets, and the data labeling results in this data set are relatively low in purity. In order to improve the efficiency of training models, the existing technology divides the training data set into several types of groups. To train the model separately, but if the distribution of various data in the original data set is unbalanced, the multiple clustering results obtained after grouping the data set will not have a positive effect on the s...

AI Technical Summary

Problems solved by technology

[0005] The invention provides a data set division method and device, which is used to solve the problem that multiple clustering results obtained after dividing the data set in the prior art cannot have a positive effect on the subsequent training model, thereby increasing the overhead of the model during training defects, so that it can adaptively complete the process of data division and improve the training efficiency of the model

Images