Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

SQL-based data processing method, device and equipment

A data processing device and data processing technology, applied in the field of data processing, can solve problems such as low efficiency, and achieve the effects of improving efficiency, avoiding pressure, and saving labor costs

Pending Publication Date: 2021-04-09
SHANGHAI ZHONGTONGJI NETWORK TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the object of the present invention is to provide a kind of data processing method, device and equipment based on SQL, to overcome the problem that the data in the current HBase database is imported into the efficiency of Hive database is not high

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • SQL-based data processing method, device and equipment
  • SQL-based data processing method, device and equipment
  • SQL-based data processing method, device and equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0047]In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other implementations obtained by persons of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

[0048] figure 1 It is a flowchart provided by an embodiment of the SQL-based data processing method of the present invention.

[0049] like figure 1 As shown, the SQL-based data processing method of this embodiment may include the following steps:

[0050] S101. Obtain the sampling SQL written by the user based on actual needs, and call the pre-created Hive table and HBase virtual table from the metadata database.

[0051] Hive is a data warehouse framework b...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to an SQL-based data processing method, device and equipment, and the method comprises the steps of obtaining an extraction SQL written by a user, and calling a Hive table and an HBase virtual table which are created in advance from a meta-database, wherein the HBase virtual table is used for mapping an HBase entity table; in the Calcite, in combination with the extraction SQL, the HBase virtual table and the Hive table, generating a physical execution plan, and sending the physical execution plan to the Hadoop Yarn; and scheduling the physical execution plan by utilizing the Yarn, reading the data in the HBase entity table, and writing the data into the Hive table. According to the invention, data extraction can be performed without using an HBase Region Server, pressure on HBase service is avoided, a user only needs to compile a simple extraction SQL on a scheduling platform, the labor cost is saved, and the efficiency of importing the data in the HBase database into the Hive database is effectively improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to an SQL-based data processing method, device and equipment. Background technique [0002] During data processing, it is necessary to frequently import data from the HBase database into the Hive database. In the prior art, it is generally implemented by manually creating a Hive external table corresponding to the HBase table, or by using HBaseSnapshot. [0003] However, manually establishing the Hive external table corresponding to the HBase table is not only redundant and inefficient, but also requires a full table scan of HBase, which generates a large number of requests to the HBase Region Server, resulting in excessive server load during task execution; use The manual configuration of HBase Snapshot is cumbersome. For example, it needs to configure fields, filter conditions, etc., and it cannot realize advanced data extraction requirements such as aggregation, ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/28
CPCG06F16/25G06F16/284
Inventor 秦瑞
Owner SHANGHAI ZHONGTONGJI NETWORK TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com