Unlock instant, AI-driven research and patent intelligence for your innovation.

Zipper type model data aggregation method and system based on HIVE

A technology for model data and data aggregation, applied in database management system, database update, electronic digital data processing, etc., can solve the problems of not recommending update, delete, and unfriendly support, avoid update and delete operations, reduce data Redundancy, the effect of reducing development workload and operation and maintenance difficulty

Pending Publication Date: 2022-03-25
北京思特奇信息技术股份有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, Hive is based on HDFS (Hadoop Distributed File System, distributed file system) file storage method, which is not very friendly to update and delete support, and it is not recommended to perform update and delete operations in hive

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Zipper type model data aggregation method and system based on HIVE
  • Zipper type model data aggregation method and system based on HIVE
  • Zipper type model data aggregation method and system based on HIVE

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0040] Below in conjunction with accompanying drawing, the present invention is described in further detail:

[0041] Such as figure 1 As shown, a HIVE-based zipper type model data aggregation method provided according to the present invention includes:

[0042] Connect with the business system and extract the original data from the busines...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a zipper type model data aggregation method and system based on HIVE, which are applied to the technical field of data aggregation, and the method comprises the following steps: docking a business system, and extracting original data; cleaning and standardizing the original data to determine a standard interface model; obtaining current valid data from the target result model, performing internal association with the standard interface model, and determining a to-be-updated data model; associating the left side of the standard interface model with a to-be-updated data model, eliminating data with repeated primary keys to generate incremental data, and inserting the incremental data into a target result model; and associating a to-be-updated data model in the standard interface model, deleting the data matched with the full field, and updating the to-be-updated data model. By means of the technical scheme, file-level operation is avoided, data zipper type storage is achieved, all data change tracks can be reserved, meanwhile, data redundancy can be reduced, and the development workload and operation and maintenance difficulty are reduced through a templated development mode.

Description

technical field [0001] The invention relates to the technical field of data aggregation, in particular to a HIVE-based zipper model data aggregation method and a HIVE-based zipper model data aggregation system. Background technique [0002] In the carrier industry, technical scenarios such as de-O and MPP have been fully promoted, and big data technology has been introduced for data storage, calculation and application. According to the timeliness of data, according to business scenarios, it is mainly divided into three scenarios: real-time business, quasi-real-time business and offline business. No matter what kind of business scenario it is, the data must be operated according to a certain frequency cycle. In the offline business scenario, under the background of big data technology, Hive is generally used to replace Oracle as the main technical solution for data storage and calculation. However, Hive is based on the HDFS (Hadoop Distributed File System) file storage meth...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/215G06F16/23
CPCG06F16/254G06F16/215G06F16/2358
Inventor 时敏
Owner 北京思特奇信息技术股份有限公司