Implementation scheme for improving Impala query capacity

A scheme and capacity technology, applied in the field of big data, which can solve the problems that table metadata cannot be updated and query operations cannot be performed.

Pending Publication Date: 2021-07-20
南京中新赛克科技有限责任公司
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Impala uses the thrift framework to implement underlying communication and serialization. Each table metadata update supports a maximum of 4G. When the data involved exceeds 4G, the table metadata cannot be updated, resulting in the inability to perform query operations.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Implementation scheme for improving Impala query capacity
  • Implementation scheme for improving Impala query capacity
  • Implementation scheme for improving Impala query capacity

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] The invention discloses an implementation scheme for improving the query capacity of Impala, which comprises the following steps:

[0023] Step 1: Save the collected, filtered and preprocessed data from the data source in the Hadoop cluster;

[0024] In this step, the data collected, filtered and preprocessed from the data source includes structured data and unstructured data; the data source includes a variety of different types of data sources, and the collected data is stored in the Hadoop cluster by partition in HDFS.

[0025] Step 2: If image 3 As shown, after Impala associates with the Hadoop cluster, it serializes all file metadata in the table and stores them in the memory database Redis;

[0026] In this step, serializing all file metadata in the table and storing it in the memory database Redis includes the following steps:

[0027] Step 2.1: Impala opens the Redis connection switch;

[0028] Step 2.2: Configure the thread pool;

[0029] Step 2.3: When m...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an implementation scheme for improving Impala query capacity, which comprises the following steps: storing data collected and processed from a data source into a Hadoop cluster, associating Impala with the Hadoop cluster, caching metadata into Redis, acquiring the metadata from the Redis when a user submits a query request for the first time and an SQL engine creates an execution plan, deserializing the metadata, storing the deserialized metadata into Lrucache, and enabling the subsequent request creation execution plan to directly obtain the metadata from the Lrucache. The scheme can support updating of the table with the metadata larger than 4G, so that query of the large table with the metadata larger than 4G is supported.

Description

technical field [0001] The invention relates to the field of big data, in particular to an implementation scheme for improving the query capacity of Impala. Background technique [0002] Impala is a distributed large-scale parallel processing database query engine that supports queries from data sources such as hdfs and hbase. Impala consists of three services: impalad, catalogd and statestored, the specific architecture is attached figure 1 As shown, Impalad is responsible for accepting user query requests, generating execution plans, and distributing them to other Impalad processes for execution, and finally collecting results; Catalogd provides metadata services, which can pull metadata from hive metastore; Statestored mainly provides message subscriptions Serve. [0003] Impala needs to load metadata for the first query or execute dml operations. The metadata update process is as follows: figure 2 As shown, when an Impalad executes a dml operation and triggers catalo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/215G06F16/22G06F16/2455G06F16/27
CPCG06F16/215G06F16/2228G06F16/24552G06F16/27
Inventor 王东张晨飞
Owner 南京中新赛克科技有限责任公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap