Unlock instant, AI-driven research and patent intelligence for your innovation.

Method achieving real-time processing of big data

A real-time processing, big data technology, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of waste of system resources, long data processing time, etc., to achieve the effect of improving efficiency

Inactive Publication Date: 2012-12-26
NANJING YUNCHUANG LARGE DATA TECH CO LTD
View PDF4 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to overcome the disadvantages of waste of system resources and long data processing time caused by commonly used data processing methods in existing cloud computing solutions, and provide an effective real-time processing method for massive data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method achieving real-time processing of big data
  • Method achieving real-time processing of big data
  • Method achieving real-time processing of big data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0016] The technical scheme of the present invention is described in detail below in conjunction with accompanying drawing:

[0017] All the processing of the present invention is performed concurrently, which utilizes the hardware equipment of the computer to the greatest extent and greatly improves the processing efficiency. Enable users to obtain query results when they perform query operations.

[0018] The invention includes real-time storage of data, real-time query, real-time result transmission, data storage, query and transmission are all concurrent and real-time.

[0019] 1. Real-time storage

[0020] Based on the existing HDFS, start multi-threaded index creation on each datanode, and create index files in parallel. Indexes are created as figure 1 Shown:

[0021] Build indexes for some important fields and generate them in the structure of B+ tree. Each new record only needs to be inserted into the B+ tree. The insertion of the B+ tree is only performed on the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a big data real-time processing method and relates to the field of computer application systems. According to the method, storage of data in a database, query and transmission are concurrent and real-time. At the time of distributing tasks, filtering and indexing is conducted simultaneously. At the time of filtering and indexing, filtered index files are distributed onto a data node which simultaneously finishes queries of local files and returns data to a client. After finishing any query, the data node immediately returns query results to a user. Due to the fact that the processing process of the method is executed concurrently, hardware equipment of a computer is utilized to the uttermost degree. Concurrent execution of an efficient B+ structure and queries enables the queries to be finished in real time and greatly improves efficiency of the queries. Therefore, the user can obtain the query results when executing query operation.

Description

technical field [0001] The invention relates to the field of computer application systems, in particular to a method involving real-time processing of massive data. Background technique [0002] With the development of informatization, the data to be processed by enterprises has grown explosively, and the amount of data has reached terabytes and petabytes, which has brought a series of problems. As the amount of data increases, the load on the system increases, and the performance of data storage and query decreases accordingly. How to give full play to the maximum performance of the system and make the storage and query the fastest without increasing the hardware cost is a difficult problem faced by many enterprises. [0003] The emergence of cloud computing provides an effective solution for massive data processing. In common cloud computing solutions, HDFS (a distributed file system) of Hadoop (a distributed system infrastructure) can easily realize massive data processi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 张真王磊陈伟王胤然杨震宇周亮亮
Owner NANJING YUNCHUANG LARGE DATA TECH CO LTD