Big data out-of-kernel learning framework based on Bayesian network classification

Abstract

The invention discloses a big data out-of-kernel learning framework based on Bayesian network classification, which comprises the following steps of: firstly, carrying out attribute selection by taking memory constraint as a benchmark, namely sorting attributes, determining the number of selected attributes and learning model parameters on the selected attributes, and then carrying out model selection based on DME (Discrete Model Evaluation), namely, according to a model evaluation obtained by leaving one-out cross validation, carrying out one-out-of-kernel learning on the selected attributes and carrying out one-out-of-kernel learning on the selected attributes; selecting an optimal one from the approximation models formed by the selected attributes in the previous stage for use in classification; then, performing primary scanning on the sample data by adopting an on-line one-leaving cross validation model selection method, and determining Bayesian network classification based on model weighting and model selection of the model selection method;

Description

Technical Field

[0001] The present invention relates to the field of big data mining technology, and in particular to a big data out-of-core learning framework based on Bayesian network classification. Background Art

[0002] With the rapid development and popularization of computer and information technology, the data generated in social networks, biomedicine, e-commerce and other fields has exploded. Sina Weibo posts 80 million posts every day; the amount of human genome and transcriptome (multiple tissues and multiple time points) sequencing data of a single sample will exceed 100GB and 30GB respectively (based on a 3GB human genome and 10-30 times sequencing depth); Taobao's nearly 400 million members generate about 20TB of commodity transaction data every day. This massive, high-growth and diverse information asset is called big data. Big data not only contains more samples, but also the number of data attributes, that is, the dimension of the data will be higher. How to discover new knowledge from these large-scale data is a research hotspot in the field of data mining. Big data has brought huge challenges to existing data mining. A practical problem is that the computer's storage space is insufficient and it is difficult to load all the data into the computer's memory. Therefore, traditional data mining algorithms that require random access to data are no longer applicable. A feasible solution is to store the data in external memory, and the data mining algorithm accesses the sample data in the external memory one by one. In this case, the huge number of samples also increases the time cost of accessing samples, so only those data mining algorithms with lower time complexity are feasible on big data.

[0003] Weka is a general data mining work platform that brings together a large number of machine learning algorithms that can undertake data mining tasks, including data preprocessing, classification, regression, clustering, association rules, and visualization on a new interactive interface. However, the platform does not support out-of-core learning, so memory overflow and inability to process large data sets will occur. Vowpal Wabbit is a machine learning system that pushes the frontier of machine learning through online, hashing, dimensionality reduction, learning search, active and interactive learning techniques. It supports online learning and out-of-core learning, but focuses on linear methods such as logistic regression. Summary of the invention

[0004] The present invention provides a big data out-of-core learning framework, aiming to solve the problems of insufficient memory and insufficient efficiency of big data.

[0005] The present invention is a big data out-of-core learning framework based on Bayesian network classification, including two parts: incremental model training and model testing. The incremental model training performs model parameter learning on input data. When reading sample data, it is always read from a disk file. The reading process of the sample data needs to be controlled. If it is continuous data, the continuous value needs to be discretized. The number of sample readings is controlled in the model, and the network structure is learned and the model is selected after each training.

[0006] Furthermore, online leave-one-out cross-validation is a model method for online leave-one-out cross-validation, that is, in the training method of AnDE, each current nDE is first used to predict the current sample, then the current sample is used for training, and finally the classification accuracy of each nDE is calculated; when predicting the test data, the classification accuracy of each nDE is used as a weight to average the posterior probability obtained by each nDE. For incremental leave-one-out cross-validation, an additional data scan is added, and for each current sample, its influence is removed from the model parameters, and each nDE is used to predict the current sample, and then the current sample is attached to the model parameters.

[0007] The present invention constructs multiple nested approximate models in the Bayesian classification model, realizes model selection based on cross-validation empirical risk, and evaluates the empirical risk of the approximate model in an additional scan of the training sample, thereby realizing model selection by minimizing the empirical risk. 1. Observe and solve classification problems from a clustering perspective. In this way, the advantages of the two can be combined to achieve a unity. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0009] Figure 1 Technology Roadmap DETAILED DESCRIPTION

[0010] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0011] The present invention is a big data out-of-core learning method based on Bayesian network classification, including two parts: incremental model training and model testing. The incremental model training performs model parameter learning on input data. When reading sample data, it is always read from a disk file. The reading process of the sample data needs to be controlled. If it is continuous data, the continuous value needs to be discretized. The number of sample readings is controlled in the model, and the network structure is learned and the model is selected after each training.

[0012] The present invention is further described below in conjunction with a specific embodiment. Online leave-one-out cross-validation is a model method for online leave-one-out cross-validation, that is, in the training method of AnDE, each current nDE is first used to predict the current sample, then the current sample is used for training, and finally the classification accuracy of each nDE is calculated; when predicting the test data, the classification accuracy of each nDE is used as a weight to average the posterior probability obtained by each nDE. For incremental leave-one-out cross-validation, an additional data scan is added, and for each current sample, its influence is removed from the model parameters, and each obtained nDE is used to predict the current sample, and then the current sample is added to the model parameters.

Claims

1. A big data out-of-core learning framework based on Bayesian network classification, characterized by: include: The incremental model training consists of two parts: incremental model training and model testing. The incremental model training performs model parameter learning based on the input data. When reading sample data, it is always read from the disk file. The sample data reading process needs to be controlled. If it is continuous data, the continuous value needs to be discretized. The number of sample readings is controlled in the model. After each training, the network structure is learned and the model is selected.

2. The big data out-of-core learning method according to claim 1, characterized in that: The online leave-one-out cross-validation is a model method of online leave-one-out cross-validation, that is, in the training method of AnDE, each current nDE is first used to predict the current sample, then the current sample is used for training, and finally the classification accuracy of each nDE is calculated; when predicting the test data, the classification accuracy of each nDE is used as a weight to average the posterior probability obtained by each nDE. For incremental leave-one-out cross-validation, an additional data scan is added, and for each current sample, its influence is removed from the model parameters, and each obtained nDE is used to predict the current sample, and then the current sample is added to the model parameters.

Images