Fast data and big data combined data processing method and system

A technology for data processing and big data, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of increased hardware cost, increased computing time, low read and write efficiency, etc., to achieve linear performance expansion, reduce reading The effect of slow writing and cheap hardware

Inactive Publication Date: 2013-08-28
刘峰
View PDF2 Cites 55 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] To sum up, in the era of "massive data", existing technologies cannot effectively deal with the following problems derived from "massive data":
[0011] (1) The problem of data storage. The storage capacity of a single machine is not enough to accommodate PB-level data. Using a larger machine will cause a substantial increase in hardware costs
[0012] (2) The problem of data calculation. The structure of a single machine makes the calculation of data performed serially. In the era of "massive data" (hundreds of millions to billions of data), the calculation time is greatly increased. Unable to get results within the time requested by the client
[0013] (3) The existing technologies are all based on

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Fast data and big data combined data processing method and system
  • Fast data and big data combined data processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0036] A data processing system combining fast data and big data provided in this embodiment, its structural diagram is as follows figure 1 As shown, it includes data input and classification module, real-time transaction (OLTP) module, full-text search (Lucene) module, massive historical data analysis (OLAP) module and application module (Apps).

[0037]The data input and classification module is used to receive data input from different data sources, organize them into a unified format and classify them according to "fast data" and "big data", and transmit "fast data" to the real-time transaction (OLTP) module , to transfer "big data" to the massive historical data analysis (OLAP) module.

[0038] The real-time transaction (OLTP) module is used to use distributed memory (In Memory Computing) to perform real-time calculation and query on "fast data", and output the calculation or query results to the application module (Apps) and the full-text retrieval module (Lucene) .

...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a fast data and big data combined data processing method which includes steps: (1) data input of different data sources is received and is classified and transmitted according to fast data and big data; (2) fast data enter a real-time trading module which performs real-time calculation and inquiring on fast data by aid of a distributed memory; (3) a full-text retrieval module performs full-text retrieval according to the fast data result; (4) big data enter a volume historical data analysis module, are stored and are subjected to complete inquiring and statistic analysis; and (5) an application module receives data processed in the step (2), the step (3) and the step (4), and terminal display is carried out as required. The invention further provides a fast data and big data combined data processing system. The fast data and big data combined data processing method and system are low in cost and convenient to maintain, resources are distributed according to needs, and the performance is linearly expanded.

Description

technical field [0001] The present invention relates to a data processing method and system thereof, more specifically, to a data processing method and system combining fast data and big data, and belongs to the technical field of cloud computing data processing. Background technique [0002] Today's world has entered the era of "massive data", and the processing of massive data is generally divided into two categories. One is the processing of "active data", also known as "fast data", which needs to be able to be analyzed and processed in real time and the results can be fed back to users; the other is the processing of "historical data", also known as "big data". "A large amount of historical data needs to be saved using low-cost servers and storage disks, and statistical analysis and data mining can be easily performed through the interface. [0003] In the current market, the existing technologies are mainly divided into the following categories. [0004] 1. Databases ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 刘峰陈一飞杨冬郭鑫刘云程
Owner 刘峰
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap