Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Impala component-based batch processing method and system

A batch processing and component technology, applied in the field of data processing, can solve error-prone problems, avoid program development and improve execution efficiency

Inactive Publication Date: 2018-09-28
BANK OF NINGBO
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

At present, many organizations are trying to use big data platforms and Impala components for data batch processing. However, due to the system differences and business differences of different organizations, the existing big data platforms and Impala components cannot meet the batch processing needs of different organizations. When an organization needs to use the big data platform and Impala components for data batch processing, it often needs to write a large number of complex programs to call Impala components, which brings a huge workload to developers and is also prone to errors

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Impala component-based batch processing method and system
  • Impala component-based batch processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0040] see figure 1 , figure 1 A flow chart of a batch processing method based on the Impala component disclosed in this embodiment specifically includes the following steps:

[0041] S101: Obtain the SQL statement corresponding to the target batch processing task input by the user in the preset template and conforming to the format of the Impala component;

[0042] The preset template is pre-developed, preferably, the preset template is developed by Python ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides an Impala component-based batch processing method and system. A user only needs to input an SQL statement meeting an Impala component format and corresponding to a target batchprocessing task in a preset template; a preset format file embedded with the SQL statement is generated; the preset format file runs; and by calling at least one packaged function pack corresponding to the target batch processing task in a running environment of the preset format file and calling an application programming interface in an Impala component, the SQL statement is sent to the Impala component, thereby enabling the Impala component to execute the target batch processing task. A large amount of program development performed for calling the Impala component by a developer and programwriting errors during each batch processing are avoided, so that the executive efficiency of batch processing is improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, and more specifically, to a batch processing method and system based on Impala components. Background technique [0002] In recent years, big data processing and analysis has become a global problem. With the continuous improvement of my country's economic and social informatization and automation, many fields such as government, public services, scientific research and commercial applications are also facing big data problems. The big data platform proposes a service-oriented big data analysis platform solution based on the Internet and data centers to meet the growing demand for big data processing. [0003] The Impala component is a real-time interactive SQL query engine running on the existing big data platform, which enables the data in the HDFS file system and HBase database to support real-time and fast query. At present, many organizations are trying to use big data platf...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F8/20G06F8/35G06F8/41
CPCG06F8/24G06F8/35G06F8/41
Inventor 翁宇哲林强陈翔
Owner BANK OF NINGBO
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com