Supercharge Your Innovation With Domain-Expert AI Agents!

Data synchronization method from OLTP to OLAP

A data synchronization and data technology, applied in the field of data processing, can solve problems such as repeated memory occupation and affect the overall storage performance, achieve high synchronization performance, reduce data flow levels, and improve performance

Pending Publication Date: 2022-01-11
SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Data synchronization products usually use Record as an intermediate object. After the destination end receives the data from the source end, it reconstructs a record object that matches the destination end. It caches multiple records in memory and then processes them in batches. This will repeatedly occupy memory and affect Overall storage performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data synchronization method from OLTP to OLAP
  • Data synchronization method from OLTP to OLAP

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0030] combined with figure 1 , the present embodiment proposes a data synchronization method from OLTP to OLAP, and its implementation includes:

[0031] Deploy the acquisition unit, encoding unit and transmission unit 1 in OLTP, the acquisition unit collects OLTP data, the encoding unit encodes the collected data into messages, and transmits them to OLAP through the transmission unit 1;

[0032] Deploy the transmission unit 2, decoding unit and writing unit in OLAP. The transmission unit 2 receives the message output by the transmission unit 1, and decodes the message into data through the decoding unit. The decoding unit further constructs the decoded data into tasks. Then synchronize to OLAP in batches by writing the unit;

[0033] In the above process, the acquisition unit includes a conversion unit that does not exist independently. The conversion unit is an extension plug-in of the acquisition unit. The conversion unit converts the collected data into CSV, ORC, and Par...

Embodiment 2

[0043] combined with figure 2 , the present embodiment proposes a data synchronization method from OLTP to OLAP, and its implementation includes:

[0044] Deploy the acquisition unit, encoding unit and transmission unit 1 in OLTP, the acquisition unit collects OLTP data, the encoding unit encodes the collected data into messages, and transmits them to OLAP through the transmission unit 1;

[0045] Deploy the transmission unit 2, decoding unit and writing unit in OLAP. The transmission unit 2 receives the message output by the transmission unit 1, and decodes the message into data through the decoding unit. The decoding unit further constructs the decoded data into tasks. Then synchronize to OLAP in batches by writing the unit;

[0046] In the above process, the writing unit includes a conversion unit that does not exist independently. The conversion unit is an extension plug-in of the writing unit. The conversion unit converts the data contained in the output task of the dec...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data synchronization method from OLTP to OLAP, and relates to the technical field of data processing, and the method comprises the steps: deploying an acquisition unit, a coding unit and a first transmission unit in the OLTP, wherein the acquisition unit acquires the data of the OLTP, and the coding unit codes the acquired data into a message and transmits the message to the OLAP through the first transmission unit; and deploying a second transmission unit, a decoding unit and a writing unit in the OLAP, wherein the second transmission unit receives a message output by the first transmission unit and the decoding unit decodes the message into data and further constructs the decoded data into tasks, and then the writing unit synchronizes the tasks to the OLAP in batches. In the above process, the acquisition unit converts the acquired data into a format supported by the OLAP and then the coding unit codes the acquired data into the message, or the writing unit converts the data contained in the output task of the decoding unit into the format supported by the OLAP and then synchronizes the data to the OLAP in batches. According to the invention, the data synchronization performance can be improved, and the constant memory consumption is kept.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a data synchronization method from OLTP to OLAP. Background technique [0002] The online transaction processing (OLTP) system usually synchronizes data to the online analytical processing (OLAP) system on a regular or continuous basis. Processing tens of billions of records and PB-level data volume has become a common scale. This poses a challenge to data synchronization performance and stability. high demands. On the other hand, the number of tables may reach tens of thousands. In order to synchronize multiple tables at the same time, data synchronization can support the concurrency of multiple tables. [0003] The data query engine in the field of big data can generally directly use the data files stored in HDFS by defining external tables. The optional file formats include CSV, Parquet, ORC, etc. Different file formats are different in terms of writing speed, query s...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F9/54
CPCG06F16/27G06F9/546G06F2209/548G06F2209/547
Inventor 张明邓光超高传集江燕
Owner SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com