IndexR real-time data analyzing library

A real-time data and data technology, applied in the Internet field, can solve the problems of good scanning, compression performance, old data cannot be deleted, lack of index flexibility, etc., to achieve the effect of simple and reliable structure, high availability, and efficient hardware utilization

CN107133342AInactive Publication Date: 2017-09-05广州舜飞信息科技有限公司
0 Cites 33 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2017-09-05
Estimated Expiration
Not applicable · inactive patent

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
Patent Text Reader

Abstract

The invention discloses an IndexR real-time data analyzing library. The IndexR real-time data analyzing library realizes a structure data format which can be arranged in a distribution environment, can be subjected to parallel processing, is provided with an index and is of a rank type. On the basis of the data format, a data warehouse system is built through the IndexR; mass data sets can be subjected to rapid statistical analysis (OLAP) on the basis of Hadoop ecology; the data can be imported in real time, moreover, the zero delay of querying is realized. The IndexR is deigned to solve the problems such as slow analyzing under a big data scene, data delay, and complex system. According to the IndexR real-time data analyzing library, the data is stored in HDFS; the Zookeeper is used for communicating and negotiating in the cluster; the Hive is used for conveniently managing partitioned data; the data can be rapidly imported in real time through Kafka; an outstanding distributive querying engine Apache Drill is used in a querying layer.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the technical field of the Internet, in particular to an IndexR real-time data analysis library. Background technique

[0002] The programmatic advertising business needs to connect with major media on the entire network, and generate millions of analytical data per second. These data track and describe the process of advertising activities in detail, such as the number of creative impressions, clicks, registrations generated by activities, and return visits. We need to analyze and process these data in real time, including customer reporting, delivery optimization, fraud analysis, fee settlement, etc. The query pattern of data users is non-fixed and unpredictable, and with the surge of business volume, the amount of data also increases sharply. We need a new technology to solve these requirements: 1. Super large data set, low query latency: the query mode cannot be predicted and cannot be pre-calculated; the amount of table ...

Examples

specific Embodiment

[0031]An IndexR real-time data analysis library, including: system architecture, deployment architecture, storage structure and real-time modules;

[0032] The system architecture is responsible for file storage format, including index and data, real-time data import, table definition operation, query optimization, and data cache. The distributed computing framework (Drill / Spark) is responsible for specific query operations on IndexR data, as well as other computing tasks, Hadoop and peripheral tools - providing distributed file storage, offline batch computing, offline data management, and various offline ETL tasks, The perfect combination of IndexR and Hadoop can be used as a highly compressed file format with its own index, compatible with all operations of Hive, Kafka-message queue, data flows into IndexR through Kafka, Zookeeper-cluster status management;

[0033] In the Hadoop system environment, deploying IndexR on an existing cluster can usually be completed within hal...