Storage architecture method compatible with real-time and offline data processing

A technology of data processing and storage architecture, applied in the field of data processing, can solve problems such as low data quality and low data processing timeliness, and achieve the effect of realizing storage and improving capabilities

CN113051275APending Publication Date: 2021-06-29银盛支付服务股份有限公司
0 Cites 2 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2021-06-29

Smart Images

  • Figure 1
    Figure 1
Patent Text Reader

Abstract

The invention provides a storage architecture method compatible with real-time and offline data processing. The storage architecture method comprises the following steps: S1, synchronizing each single table of a source end to a corresponding Mongodb cluster; S2, updating the updated data in the Mongodb cluster to a ClickHouse cluster every t minutes according to the last update time, the value of t being set according to service requirements, the Last Update Time being a field existing in all the data, and when the data are updated, the field being updated to the current system time; and S3, synchronizing the data in the Mongodb cluster to a CDH cluster according to T+1 by adopting a synchronization tool according to the Last Update Time, T+1 meaning delaying by one day. The aggregation calculation performance requirement of real-time data is met, and storage of real-time incremental data and off-line data is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present invention relates to the technical field of data processing, and in particular, the present invention relates to a storage architecture method compatible with real-time and offline data processing. Background technique

[0002] In big data applications, the underlying data storage system needs to efficiently perform multi-functional applications such as real-time data processing, offline computing processing, and data analysis. At the same time, it also needs to provide strong support for data access timeliness, data access volume, and data concurrency. , with the accumulation of a large amount of data, there will be problems such as low data quality and low data processing timeliness. How to efficiently solve the real-time incremental update of data, meet the aggregate computing performance requirements of real-time data and offline storage, has become an urgent problem to be solved . Contents of the invention

[0003] In order to overc...

Examples

Embodiment Construction

[0021] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0022] The idea, specific structure and technical effects of the present invention will be clearly and completely described below in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, features and effects of the present invention. Apparently, the described embodiments are only some of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative efforts belong to The protection scope of the present invention. In addition, all the connection / connection relationships involved in the patent do not simply refer to the direct connection of components, but mean that a better connection structure can be formed by adding or reducing connection accessories according to specific implementat...