Couf, a data analysis method

Abstract

A calculate once use forever (COUF) method to efficiently determine statistics on data records of a data set. Data records are read from an original data set and grouped together into units and simple statistic on each unit are calculated and stored in a database. When a query asking for an additional statistic comes in, the additional statistic can be determined quickly and efficiently from the already calculated unit simple statistics so that read redundancies and computational redundancies can be eliminated.

Description

CROSS REFERENCE TO RELATED APPLICATIONS[0001]This application is a continuation in part of U.S. application Ser. No. 15 / 404,141, filed on Jan. 11, 2017 which is a continuation of U.S. application Ser. No. 15 / 361,441, filed on Nov. 27, 2016. The benefit under 35 USC § 120 for both earlier non-provisional applications is hereby claimed, and the aforementioned applications are hereby both herein incorporated by reference in their entireties.FIELD OF THE INVENTION[0002]This invention relates generally to the field of data analysis, data mining and more specifically to big-data analysis.BACKGROUND OF THE INVENTION[0003]Nowadays more and more businesses, governmental agencies and other organizations need to deal with huge amount of data every day. With the spread of Internet and widely adopted e-commerce, the data sizes grow exponentially and they easily exceed terabytes, petabytes or even beyond. In order to find consumer trends, uncover fraud, or detect terrorists, or for other purposes...

AI Technical Summary

Benefits of technology

This patent aims to provide an improved method and apparatus to efficiently calculate statistics, which reduces processing time and resources. It also introduces a new and simple big-data analysis method and apparatus, as well as a new and easy data analysis method for data mining and a new and simple process for risk analysis. The invention is a computer system with a database that stores simple statistics on each unit of data, which can be easily used to calculate subsequent query statistics on the records in any arbitrary combination of units. Overall, this patent presents a solution for effective and efficient data analysis, especially for big-data.

Problems solved by technology

However, the huge amount of data makes the analyses difficult, if not impossible.

However, there are two obvious disadvantages associated with Hadoop.

This is redundant and a waste of time and it makes the whole process slow.

But there is a setback.

In order for Spark to work well, entire data set needs to be fitted into the memory, Spark needs a lot of memory and memory is more expensive.

When the data set cannot be fitted into the memory, Spark does not work well.

Furthermore, when the same query is executed continually, it runs very fast.

Images