Supercharge Your Innovation With Domain-Expert AI Agents!

Data increment updating method

A technology of incremental update and data update, which is applied in the field of database and data warehouse, can solve the problems affecting the performance of data platform, platform performance, cost operation stability and other problems, and achieve the effect of saving resources

Active Publication Date: 2021-04-27
食亨(上海)科技服务有限公司
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This affects the overall performance of the data platform to a certain extent, and there are contradictions in platform performance, implementation cost and operational stability

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data increment updating method
  • Data increment updating method
  • Data increment updating method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] figure 2 A flow chart of a data incremental update method according to an embodiment of the present invention is disclosed. refer to figure 2 As shown, the data incremental update method is used to incrementally update data in a storage architecture composed of a database, an operational data store, and a data warehouse, and the data incremental update method includes:

[0029] S1. Data configuration step. In the data configuration step, configure data from message queues, such as Kafka queues. In one embodiment, the data is configured in two ways, namely, the first Flume configured with the date of the data itself as the partition and the second Flume configured with the data collection date as the partition. The data written to the DW via the first Flume is the first ODS data, and the data written to the DW via the second Flume is the second ODS data. The partition granularity of the first Flume or the second Flume is related to the computing granularity. The c...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data increment updating method which comprises the following steps: a data configuration step: configuring data from a message queue, including a first Flume configured by taking the date of the data as a partition and a second Flume configured by taking a data acquisition date as a partition, writing the data into first ODS data through the first Flume, and writing the data into second ODS data through the second Flume; a data initialization step: initializing the DW data, and selecting a partition meeting a screening condition from the first ODS data as a latest partition of the DW; in the data merging step, taking the second ODS data cut off to the current as latest write-in data, and merging and marking the latest partition of the DW and the latest write-in data; a data replacement step: writing the combined data back to the latest partition of the DW to cover the original data; an update determination step of performing update determination in the DW including the merged data, and performing marking to participate in an update determination operation; and a data updating step of synchronizing the data with the marks to the database for incremental updating if it is judged that updating exists.

Description

technical field [0001] The present invention relates to the field of software technology, more specifically, to database and data warehouse technology. Background technique [0002] Data is becoming an important resource, and more and more applications will call or store a large amount of data for application services, or analyze and calculate the stored data to improve their own functions. The storage and management of massive data is becoming an important issue. [0003] Most of the current data platforms adopt the architecture of database (DB) + operational data storage (ODS) + data warehouse (DW). figure 1 A schematic diagram of the architecture of the data platform is revealed, including a database DB 101 , an operational data store ODS 102 and a data warehouse DW 103 . The database (Database) layer is mostly a relational database, which is used to save the underlying data and the relationship between the data. Operational Data Store (Operational Data Store) is betwe...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/23G06F16/27G06F16/28G06F16/21G06F9/445
CPCG06F16/23G06F16/27G06F16/283G06F16/21G06F9/4451
Inventor 王泰舟
Owner 食亨(上海)科技服务有限公司
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com