Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Storage architecture method compatible with real-time and offline data processing

A technology of data processing and storage architecture, applied in the field of data processing, can solve problems such as low data quality and low data processing timeliness, and achieve the effect of realizing storage and improving capabilities

Pending Publication Date: 2021-06-29
银盛支付服务股份有限公司
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] In big data applications, the underlying data storage system needs to efficiently perform multi-functional applications such as real-time data processing, offline computing processing, and data analysis. At the same time, it also needs to provide strong support for data access timeliness, data access volume, and data concurrency. , with the accumulation of a large amount of data, there will be problems such as low data quality and low data processing timeliness. How to efficiently solve the real-time incremental update of data, meet the aggregate computing performance requirements of real-time data and offline storage, has become an urgent problem to be solved

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Storage architecture method compatible with real-time and offline data processing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0021] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0022] The idea, specific structure and technical effects of the present invention will be clearly and completely described below in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, features and effects of the present invention. Apparently, the described embodiments are only some of the embodiments of the present invention, rather than all of them. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative efforts belong to The protection scope of the present invention. In addition, all the connection / connection relationships involved in the patent do not simply refer to the direct connection of components, but mean that a better connection structure can be formed by adding or reducing connection accessories according to specific implementat...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a storage architecture method compatible with real-time and offline data processing. The storage architecture method comprises the following steps: S1, synchronizing each single table of a source end to a corresponding Mongodb cluster; S2, updating the updated data in the Mongodb cluster to a ClickHouse cluster every t minutes according to the last update time, the value of t being set according to service requirements, the Last Update Time being a field existing in all the data, and when the data are updated, the field being updated to the current system time; and S3, synchronizing the data in the Mongodb cluster to a CDH cluster according to T+1 by adopting a synchronization tool according to the Last Update Time, T+1 meaning delaying by one day. The aggregation calculation performance requirement of real-time data is met, and storage of real-time incremental data and off-line data is achieved.

Description

technical field [0001] The present invention relates to the technical field of data processing, and in particular, the present invention relates to a storage architecture method compatible with real-time and offline data processing. Background technique [0002] In big data applications, the underlying data storage system needs to efficiently perform multi-functional applications such as real-time data processing, offline computing processing, and data analysis. At the same time, it also needs to provide strong support for data access timeliness, data access volume, and data concurrency. , with the accumulation of a large amount of data, there will be problems such as low data quality and low data processing timeliness. How to efficiently solve the real-time incremental update of data, meet the aggregate computing performance requirements of real-time data and offline storage, has become an urgent problem to be solved . Contents of the invention [0003] In order to overc...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/23G06F16/25G06F16/27
CPCG06F16/2282G06F16/23G06F16/254G06F16/27
Inventor 刘跃红管正爽周小润
Owner 银盛支付服务股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com