Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data storage and data processing method and device based on hive table

A data processing and data storage technology, applied in the information field, can solve problems such as resource waste and high system operating pressure, and achieve the effect of reducing operating pressure, reducing quantity, and avoiding repeated loading

Active Publication Date: 2019-08-02
BEIJING SANKUAI ONLINE TECH CO LTD
View PDF5 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The embodiment of this specification provides a method and device for data storage and data processing based on hive table, which is used to solve the problem that when using hive partition table for data processing in the prior art, it is necessary to traverse multiple hive partition tables to obtain data, which is easy Repeatedly calling data will lead to high pressure on system operation and waste of resources

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data storage and data processing method and device based on hive table
  • Data storage and data processing method and device based on hive table
  • Data storage and data processing method and device based on hive table

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0052]In order to make the purpose, technical solution and advantages of this specification clearer, the technical solution of this application will be clearly and completely described below in conjunction with specific embodiments of this specification and corresponding drawings. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present application.

[0053] The technical solutions provided by various embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0054] figure 1 A hive table-based data storage process provided for the embodiment of this specification may specifically include the following steps:

[0055] S102: Obtain data to be store...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data storage and data processing method and device. The method comprises the steps of after obtaining to-be-stored data, determining each field name corresponding to to-be-stored data can be determined firstly; then, aiming at each field name, determining a field value corresponding to the field name in the at least one hive partition table, and finally storing the fieldvalue corresponding to the field name in the to-be-stored data and the determined field value corresponding to the field name in the at least one hive partition table into the field value corresponding to the field name of the specified hive partition table.

Description

technical field [0001] The present application relates to the field of information technology, in particular to a method and device for data storage and data processing based on a hive table. Background technique [0002] Hive is a data warehouse tool applied in Hadoop Distributed File System (HDFS), which is used to map structured data files into a database table (usually called hive table), and Provides a simple structured query language (Structured Query Language, SQL) query function, and converts SQL statements into MapReduce tasks for execution. [0003] Since the address of the field is not recorded in the database table produced by hive, when using the data in the hive table for processing, it is necessary to traverse all the fields of the entire hive table to obtain the data required for processing. [0004] In order to reduce the amount of data to be scanned for data acquisition, in the prior art, hive partition tables are usually created according to dates, and da...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/28
CPCG06F16/2282G06F16/283
Inventor 杨晓明冯杰徐进
Owner BEIJING SANKUAI ONLINE TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com