Fast data and big data combined data processing method and system

Abstract

The invention discloses a fast data and big data combined data processing method which includes steps: (1) data input of different data sources is received and is classified and transmitted according to fast data and big data; (2) fast data enter a real-time trading module which performs real-time calculation and inquiring on fast data by aid of a distributed memory; (3) a full-text retrieval module performs full-text retrieval according to the fast data result; (4) big data enter a volume historical data analysis module, are stored and are subjected to complete inquiring and statistic analysis; and (5) an application module receives data processed in the step (2), the step (3) and the step (4), and terminal display is carried out as required. The invention further provides a fast data and big data combined data processing system. The fast data and big data combined data processing method and system are low in cost and convenient to maintain, resources are distributed according to needs, and the performance is linearly expanded.

Description

technical field [0001] The present invention relates to a data processing method and system thereof, more specifically, to a data processing method and system combining fast data and big data, and belongs to the technical field of cloud computing data processing. Background technique [0002] Today's world has entered the era of "massive data", and the processing of massive data is generally divided into two categories. One is the processing of "active data", also known as "fast data", which needs to be able to be analyzed and processed in real time and the results can be fed back to users; the other is the processing of "historical data", also known as "big data". "A large amount of historical data needs to be saved using low-cost servers and storage disks, and statistical analysis and data mining can be easily performed through the interface. [0003] In the current market, the existing technologies are mainly divided into the following categories. [0004] 1. Databases ...

AI Technical Summary

Problems solved by technology

[0010] To sum up, in the era of "massive data", existing technologies cannot effectively deal with the following problems derived from "massive data":

[0011] (1) The problem of data storage. The storage capacity of a single machine is not enough to accommodate PB-level data. Using a larger machine will cause a substantial increase in hardware costs

[0012] (2) The problem of data calculation. The structure of a single machine makes the calculation of data performed serially. In the era of "massive data" (hundreds of millions to billions of data), the calculation time is greatly increased. Unable to get results within the time requested by the client

[0013] (3) The existing technologies are all based on disks, and the problem of slow disk read and write throughput is more prominent under "massive data", making the read and write efficiency extremely low

[0014] (4) The existing technology cannot dynamically expand server clusters and storage problems horizontally, so that hardware resources must be allocated according to the highest demand, and "elastic expansion on demand" cannot be achieved, resulting in waste of resources

[0015] To sum up, the existing technology cannot meet the requirements of real-time processing of "fast data" and batch processing of "big data" at the same time at a low cost, nor can it achieve resource allocation on demand and linear performance Expansion and easy maintenance

Images