Massive data classification method and system for big data platform based on LightGBM model

Abstract

The present invention discloses a method and system for classifying massive data on a big data platform based on LightGBM model integration, including: S100: batching asset data; S200: training a LightGBM weak classifier with current batch data; S300: classifying and predicting a test sample set with the LightGBM weak classifier, and selecting the optimal LightGBM weak classifier corresponding to each classification by a simple voting method; S400: using the optimal LightGBM weak classifier corresponding to each category as the basic model for the next batch of data, and repeating steps S200-S400 to obtain a global optimal classifier; S500: classifying and predicting asset data to be classified using the global optimal classifier. The present invention can greatly improve the training speed, efficiency and stability of large batches of data; can greatly improve the accuracy and generalization ability of the classifier, and can meet the processing requirements of massive data.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent classification of massive data assets, and in particular to a method and system for classifying massive data on a big data platform based on LightGBM model integration. Background Art

[0002] When sorting out assets and classifying data, the data we face is often disorganized and the data types are complex and diverse. At present, the common data types are numerical and categorical. For example, financial data such as income, receivables, turnover, wages, and taxes are mostly numerical, while data such as credit rating, company type, investment type, and investment scale are mostly categorical, with complex data structures and many data features. At the same time, big data platforms often face massive amounts of data, so it is unrealistic to use manual data sorting and classification marking. The investment cost is too high, there is no standard, and the maintenance cost is also high. It is an inevitable trend to use machine learning methods to sort and classify massive data.

[0003] At present, the data classification processing of big data platforms basically adopts general classification methods to perform multi-classification. The latest classification method is the XGBoost ensemble method based on K-means clustering. The principle of this latest method is: first train multiple XGBoost weak classifiers, then use K-means clustering method to select the best classifier from multiple XGBoost classifiers, and use the best classifier to predict the type of multi-classification data.

[0004] The above classification method has the following disadvantages: (1) the training and reasoning speeds are slow, and when encountering large amounts of data, there will be training freezes; (2) the number of trained classifiers is small, and the accuracy and generalization ability of the model are not high. Therefore, when faced with massive data in the big data platform, the processing performance and accuracy of this classification method cannot achieve satisfactory results. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for classifying massive data on a big data platform based on the LightGBM model integration. The method and system are more efficient and accurate and have higher generalization capabilities, and can meet the processing requirements of massive data in the big data platform.

[0006] The idea of ​​the present invention is as follows:

[0007] Consider using multiple LightGBM model integration, and use the K-means method to select and obtain the optimal LightGBM classifier; in addition, considering that it is unrealistic to conduct global training on all data when facing massive data, the cost of building a special big data processing platform is also high and not cost-effective. Therefore, the present invention adopts mini_batch for data division, uses the LightGBM model for batch training, and performs incremental training; trains multiple LightGBM classifiers in each batch, and uses the K-means method for clustering to select the optimal LightGBM classifier. The obtained LightGBM classifier will have better generalization ability and can improve the recognition accuracy to a certain extent.

[0008] The method for classifying massive data on a big data platform based on LightGBM model integration provided by the present invention comprises the following steps:

[0009] S100: Divide the asset data with classification labels into batches to obtain N batches of data, where N is a multiple of the total number of asset data categories;

[0010] Each batch of data is used as the current batch of data in turn, and steps S200-S500 are executed:

[0011] S200: Randomly sample N sample sets from the current batch data, and use the N sample sets to train N LightGBM weak classifiers respectively;

[0012] S300: Use the trained N LightGBM weak classifiers to classify and cluster the test sample sets, and use a simple voting method to select the optimal LightGBM weak classifier corresponding to each classification;

[0013] S400: Use the optimal LightGBM weak classifier corresponding to each category as the basic model for the next batch of data to continue training, and use the next batch of data as the current batch of data to repeat steps S200-S400; the final optimal LightGBM weak classifier corresponding to each category is the global optimal classifier;

[0014] S500: Input the asset data to be classified into each global optimal classifier respectively, and use a simple voting method to vote on the prediction results to obtain the final classification prediction results.

[0015] Furthermore, when training the LightGBM weak classifier, the sample set and the corresponding classification labels are used as the input of the LightGBM weak classifier, and the model parameters of the LightGBM weak classifier are used as the output for training.

[0016] Furthermore, in step S300, the prediction results are clustered using the Kmeans clustering method.

[0017] Furthermore, the simple voting method described in steps S300 and S600 is specifically:

[0018] The LightGBM weak classifier whose clustering result is the smallest difference with the known category is the optimal LightGBM weak classifier corresponding to the known category.

[0019] The present invention provides a massive data classification system for a big data platform based on LightGBM model integration, which corresponds to the massive data classification method for the big data platform, and includes:

[0020] The first module is used to divide the asset data that has been tagged into categories into batches to obtain N batches of data, where N is a multiple of the total number of asset data categories;

[0021] The second module is used to randomly sample N sample sets from the current batch data, and use the N sample sets to train N LightGBM weak classifiers respectively; each batch of data is used as the current batch data in turn;

[0022] The third module is used to use the trained N LightGBM weak classifiers to classify and cluster the test sample sets, and use a simple voting method to select the optimal LightGBM weak classifier corresponding to each classification;

[0023] The fourth module is used to continue training the optimal LightGBM weak classifier corresponding to each category as the basic model of the next batch of data, and repeat steps S200-S400 with the next batch of data as the current batch of data; the final optimal LightGBM weak classifier corresponding to each category is the global optimal classifier;

[0024] The fifth module is used to input the asset data to be classified into each global optimal classifier respectively, and use a simple voting method to vote on the prediction results to obtain the final classification prediction results.

[0025] Furthermore, in the second module, when training the LightGBM weak classifier, the sample set and the corresponding classification labels are used as the input of the LightGBM weak classifier, and the model parameters of the LightGBM weak classifier are used as the output for training.

[0026] Furthermore, in the second module, the Kmeans clustering method is used to cluster the prediction results.

[0027] Furthermore, the simple voting method described in the second and sixth modules is as follows:

[0028] The LightGBM weak classifier whose clustering result is the smallest difference with the known category is the optimal LightGBM weak classifier corresponding to the known category.

[0029] Compared with the prior art, the present invention has the following characteristics and beneficial effects:

[0030] By using the distributed idea and integrating the mini_batch, K-means and LightGBM models, the training speed and efficiency can be greatly improved in the case of large amounts of data, and the training stability can be ensured. The accuracy and generalization ability of the classifier model can be greatly improved, and the accuracy can be kept relatively stable for any scenario. It can meet the processing requirements of massive data in the big data platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a training flow chart of the global optimal classifier model of the present invention;

[0032] Figure 2 It is a business process diagram in a specific implementation method. DETAILED DESCRIPTION

[0033] The specific implementation of the present invention will be described in detail below in conjunction with the accompanying drawings. Obviously, the specific implementation described is only a part of the embodiments of the present invention, not all of the embodiments. Based on the specific implementation described, all other specific implementations obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0034] In this specific embodiment, the present invention is aimed at the business data of enterprises. The data characteristics presented by the business data are: mostly numerical data, with a small amount of categorical data, and the data is disorganized, with a total feature quantity of hundreds or even thousands. Taking corporate financial data as an example, most of the data is numerical data, including income, receivables, turnover, wages, taxes, etc.; it also contains a small amount of categorical data, including credit rating, company type, investment type, etc. Considering the complexity of data features and the amount of data, using traditional methods for data combing and feature engineering will take a lot of time, and it is easy to get stuck or fail to train during full training. Even if the training is successful, the accuracy is difficult to guarantee.

[0035] See also Figure 1 , which shows the training process of the global optimal classifier model. Figure 2 The following is the business process for dealing with the above scenario. Figure 1-2 , the specific implementation process of the present invention is described in detail, and the specific steps are as follows:

[0036] S100: Divide the classified and labeled asset data into batches to obtain several batches of data.

[0037] Take the local full asset data as the original data set, set the batch size batch_size, and record the set batch size as M; divide the original data set into batches according to the batch size: make each batch have M samples, perform N divisions in total, and obtain N batches of data containing M samples, which are recorded as batch1, batch2, ... batchN respectively.

[0038] In specific implementation, the number of samples in each batch of data (i.e., batch size) cannot be too small. There should be tens of thousands of data, and the number of samples corresponding to each final classification category in a batch should not be less than 500. The amount of data corresponding to each category in each batch of data should be balanced to achieve the best classification effect; the N value should be a multiple of the total number of asset data categories, and the N value should be adjusted according to the machine configuration. In specific implementation, the M and N values ​​will be continuously adjusted, and the optimal value will be determined through comparative experiments.

[0039] In specific implementation, a shuffle operation is performed before the asset data is divided into batches.

[0040] Each batch of data is used as the current batch of data in turn, and steps S200-S500 are executed:

[0041] S200: Randomly sample N sample sets from the current batch data, and use the N sample sets to train N LightGBM weak classifiers respectively. The current batch data initially selects the first batch of data batch1.

[0042] In specific implementation, before training the weak classifier, the sample set needs to be cleaned and data feature engineered in sequence; that is, use the same logic and rules, and discard data rules: count the global proportion of outliers in each field. When the number of discarded data in this batch is large, it is necessary to consider discarding the training of this batch and skipping the training of this batch of data.

[0043] In the present invention, the training process is: taking the sample set and the corresponding classification label as the input of the LightGBM weak classifier, and taking the model parameters of the LightGBM weak classifier as the output for training.

[0044] S300: Use the N LightGBM weak classifiers trained in step S200 to perform classification prediction on the pre-prepared test sample set, cluster the prediction results, and use a simple voting method to select the optimal LightGBM weak classifier corresponding to each classification cluster from the N LightGBM weak classifiers.

[0045] In the specific implementation, the test sample set is randomly collected from the current batch data; the prediction results are clustered using the K-means clustering method. The selection criteria for the optimal LightGBM weak classifier using the simple voting method are: the LightGBM weak classifier with the smallest difference between the clustering results and the known category is the optimal LightGBM weak classifier corresponding to the known category, and the difference is represented by the minimum variance value of the two.

[0046] S400: The optimal LightGBM weak classifier of each category cluster is used as the basic model for training the next batch of data, and the next batch of data is used as the current batch of data to repeat steps S200-S400. Finally, the optimal LightGBM weak classifier corresponding to each category is the global optimal classifier.

[0047] S500: Input the asset data to be classified into each global optimal classifier respectively, and use a simple voting method to vote on the prediction results to obtain the final prediction results.

[0048] When implementing it, the following points should be noted:

[0049] (1) A portion of each batch of data will be randomly sampled as a test sample set. Based on the test sample set, the relevant indicators of each classifier model, such as AUC, ACC, and ROC, are obtained so as to compare with the classifier models trained by the subsequent batches of data to determine whether the model performance has been improved;

[0050] (2) The classifier model trained with the current batch of data is not as good as the classifier model trained with the previous batch of data. Further strategies are used for comparison to determine whether to discard the classifier model trained with the previous batch of data. The strategies include judging from the sample distribution, model parameters, whether it is overfitting, underfitting, and the criteria for evaluating the quality of a single model;

[0051] (3) It is possible to roll back the final global optimal classifiers to test each batch of data, and to collect statistics on the test indicators of all batches of data, such as auc, acc, roc and other model evaluation indicators, and draw a graph for statistical description;

[0052] (4) When entering new asset data to be classified for prediction, batch prediction is also used to label the data in batches and store them in the database in batches.

[0053] The method of the present invention can be applied to classify the massive asset data stored in the enterprise big data platform. Before classifying the massive asset data, the classification categories are predefined, such as personnel data, financial data, operation data and other classification categories. The method of the present invention can classify the chaotic massive asset data according to the predefined categories.

[0054] Technical effects of the present invention:

[0055] (1) Compared with the training speed of the XGBoost weak classifier, the present invention adopts the LightGBM weak classifier, and the training speed can generally be increased by about 10 times, and can be increased by up to about 15 times. In terms of improving the training speed, the present invention has more advantages.

[0056] (2) The method of the present invention is compared with the traditional XGBoost ensemble method based on Kmeans clustering (denoted as comparison method 1) and the LightGBM model method (denoted as comparison method 2) to classify and predict the data in the existing standard databases Higgs, Yahoo LTR, MSLTR, and EXPO, and the obtained accuracy is shown in Table 1. As can be seen from Table 1, the prediction accuracy of the method of the present invention is improved for the four standard databases. Combining points (1) and (2), the classification and grading method of the present invention is faster and more accurate than the existing commonly used methods.

[0057] Table 1 Classification prediction accuracy

[0058] database Evaluation Criteria Comparison method 1 Comparison Method 2 Method of the present invention Higgs AUC 0.839528 0.845123 0.847819 Yahoo LTR NDCG@5 0.740316 0.756369 0.756217 MSLTR NDCG@5 0.476445 0.510153 0.512286 EXPO AUC 0.75548 0.781061 0.786223

[0059] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: (steps of the method), the storage medium, such as: ROM / RAM, disk, CD, etc.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for classifying massive data on a big data platform based on LightGBM model integration, characterized by: Includes steps: S100: Divide the asset data with classification labels into batches to obtain N batches of data, where N is a multiple of the total number of asset data categories; Each batch of data is used as the current batch of data in turn, and steps S200-S500 are executed: S200: Randomly sample N sample sets from the current batch data, and use the N sample sets to train N LightGBM weak classifiers respectively; S300: Use the trained N LightGBM weak classifiers to classify and cluster the test sample sets, and use a simple voting method to select the optimal LightGBM weak classifier corresponding to each classification; S400: Use the optimal LightGBM weak classifier corresponding to each category as the basic model for the next batch of data to continue training, and use the next batch of data as the current batch of data to repeat steps S200-S400; the final optimal LightGBM weak classifier corresponding to each category is the global optimal classifier; S500: Input the asset data to be classified into each global optimal classifier respectively, and use a simple voting method to vote on the prediction results to obtain the final classification prediction results.

2. The method for classifying massive data on a big data platform based on LightGBM model integration as claimed in claim 1, characterized in that: When training the LightGBM weak classifier, the sample set and the corresponding classification label are used as the input of the LightGBM weak classifier, and the model parameters of the LightGBM weak classifier are used as the output for training.

3. The method for classifying massive data on a big data platform based on LightGBM model integration as claimed in claim 1, characterized in that: In step S300, the prediction results are clustered using the Kmeans clustering method.

4. The method for classifying massive data on a big data platform based on LightGBM model integration as claimed in claim 1, characterized in that: The simple voting method described in steps S300 and S600 is specifically: The LightGBM weak classifier whose clustering result has the smallest gap with the known category is the optimal LightGBM weak classifier corresponding to the known category.

5. A massive data classification system based on the LightGBM model integrated on a big data platform, characterized by: include: The first module is used to divide the asset data that has been tagged into categories into batches to obtain N batches of data, where N is a multiple of the total number of asset data categories; The second module is used to randomly sample N sample sets from the current batch data, and use the N sample sets to train N LightGBM weak classifiers respectively; Take each batch of data as the current batch of data in turn; The third module is used to use the trained N LightGBM weak classifiers to classify and cluster the test sample sets, and use a simple voting method to select the optimal LightGBM weak classifier corresponding to each classification; The fourth module is used to continue training the optimal LightGBM weak classifier corresponding to each category as the basic model of the next batch of data, and repeat steps S200-S400 with the next batch of data as the current batch of data; the final optimal LightGBM weak classifier corresponding to each category is the global optimal classifier; The fifth module is used to input the asset data to be classified into each global optimal classifier respectively, and use a simple voting method to vote on the prediction results to obtain the final classification prediction results.

6. The massive data classification system for a big data platform based on LightGBM model integration as claimed in claim 5, characterized in that: In the second module, when training the LightGBM weak classifier, the sample set and the corresponding classification label are used as the input of the LightGBM weak classifier, and the model parameters of the LightGBM weak classifier are used as the output for training.

7. The massive data classification system for a big data platform based on LightGBM model integration as claimed in claim 5, characterized in that: In the second module, Kmeans clustering method is used to cluster the prediction results.

8. The massive data classification system for a big data platform based on LightGBM model integration as claimed in claim 5, characterized in that: The simple voting method described in Modules 2 and 6 is as follows: The LightGBM weak classifier whose clustering result has the smallest gap with the known category is the optimal LightGBM weak classifier corresponding to the known category.

Images