Systems and methods for sequential modeling in less than one sequential scan

Abstract

Most recent research of scalable inductive learning on very large streaming dataset focuses on eliminating memory constraints and reducing the number of sequential data scans. However, state-of-the-art algorithms still require multiple scans over the data set and use sophisticated control mechanisms and data structures. There is discussed herein a general inductive learning framework that scans the dataset exactly once. Then, there is proposed an extension based on Hoeffding's inequality that scans the dataset less than once. The proposed frameworks are applicable to a wide range of inductive learners.

Description

FIELD OF THE INVENTION [0001] The present invention relates generally to sequential modeling in the interpretation of, and other extraction of information from, large quantities of data. BACKGROUND OF THE INVENTION [0002] Herebelow, numeral in square brackets—[ ]—are keyed to the numbered list of references found towards the end of the disclosure. [0003] During the last two decades, our ability to collect and store data has significantly out-paced our ability to analyze, summarize and extract “knowledge” from the continuous stream of input. Traditional data mining methods that require all data to be held in memory are becoming inadequate. Securing an effective interface between data mining and very large database essentially requires scalability. The scalability and accuracy of data mining methods are constantly being challenged by real-time production systems that generate tremendous amount of data continuously at unprecedented rate. Examples of such data streams include security b...

AI Technical Summary

Benefits of technology

[0009] Significant differences exist in comparison with [8] in terms of determining whether to change the shape of a decision tree. Unlike previous research [8, 6], the presently proposed methods and arrangements are not limited to decision trees, but are applicable to a wide range of inductive learners. When applied to decision tree learning, accuracy will be greater than in the case of a single decision tree. Another advantage is that the ensemble reduces the asymptotic complexity of the algorithm, in addition to simply scanning less data.

Problems solved by technology

Traditional data mining methods that require all data to be held in memory are becoming inadequate.

The scalability and accuracy of data mining methods are constantly being challenged by real-time production systems that generate tremendous amount of data continuously at unprecedented rate.

Outside of decision trees, there hasn't been much research on reducing the number of data scans for other inductive learners.

Images