Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Incremental data synchronization method and device, computer equipment and storage medium

An incremental data and incremental technology, applied in the field of devices, computer equipment and storage media, and incremental data synchronization methods, can solve problems such as time-consuming, long reading and querying time, and achieve the effect of improving efficiency

Pending Publication Date: 2021-07-16
BEIJING QIANXIN TECH +1
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

A synchronization method for incremental data is proposed in the prior art. In this method, the incremental data in the data source is read and queried through the data reading plug-in, and the queried incremental data is synchronized to the downstream system. However, , when the amount of data in the data source is large, the reading and query process takes a long time, and there is still a time-consuming problem

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Incremental data synchronization method and device, computer equipment and storage medium
  • Incremental data synchronization method and device, computer equipment and storage medium
  • Incremental data synchronization method and device, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0027] The embodiment of the present invention provides a synchronization method of incremental data, which can be applied to a large data scenario, synchronized data source data to the downstream service system in an incremental synchronization, and can increase incremental data synchronization. Efficiency, specifically, figure 1 A flow diagram of a synchronization method for incremental data provided by the embodiment of the present invention, such as figure 1 As shown, the synchronization method of the incremental data provided by this embodiment includes steps S101 to S104 as follows.

[0028] Step S101: Get the state diagram of the data source in the first time to obtain the first state diagram.

[0029] The state diagram is a tree data structure, and the state diagram includes several node layers, each node layer includes several nodes, the node corresponding to a data object, the node information includes the properties of the data object, the node layer is used to characte...

Embodiment 2

[0058] The second embodiment of the present invention provides a synchronous method of incremental data, the related art characteristics, and corresponding technical effects can be referred to the above-described embodiments, in particular, the embodiment relying on the use scenario of large data, monitoring data sources New and delete changes to achieve compatibility of heterogeneous data sources. The synchronization method of the incremental data provided by this embodiment operates in a data synchronization system, which includes a data source monitoring module, a built-in metadata management module, a data increment identification module, a data ETL conversion module, a data storage module, and a data storage module. Incremental data information broadcast module to achieve a full-flow implementation of different data sources from monitoring, to final analysis, storage.

[0059] Among them, the data source monitoring module supports multiple data sources such as ES, HIVE, MYSQL...

Embodiment 3

[0071] According to the above-described embodiments, the third embodiment of the present invention provides a synchronization device of incremental data, the related art characteristics, and the corresponding technical effects can be refer to the above-described embodiments, and the second embodiment, specifically, figure 2 A block diagram of the synchronization device of the incremental data provided in the third embodiment of the present invention, such as figure 2 As shown, the apparatus includes: acquisition module 201, monitoring module 202, alignment module 203, and synchronization module 204.

[0072] Wherein, the acquisition module 201 is used to obtain a state diagram of the data source in the first time, where the state drawing is a tree data structure, and the state diagram includes several node layers, each node layer comprising several nodes. The node corresponds to a data object, the information of the node includes the properties of the data object, and the node lay...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides an incremental data synchronization method and device, computer equipment and a storage medium. The incremental data synchronization method comprises the steps that a state diagram of a data source at the first time is obtained, a first state diagram is obtained, wherein the state diagram is of a tree data structure and comprises a plurality of node layers, each node layer comprises a plurality of nodes, the nodes correspond to one data object, information of the nodes comprises attributes of the data objects, and the the node layer is used for representing the hierarchy of the data object in the data source; state information of the data source is monitored at second time, second state information is obtained, wherein the state information comprises attributes of data objects in the data source; the method also includes comparing the second state information with the first state diagram to determine an incremental data object in the data source; and acquiring the incremental data object from the data source and synchronizing the incremental data object to the target storage medium. According to the invention, the incremental data synchronization efficiency is improved.

Description

Technical field [0001] The present invention relates to the field of large data technology, and more particularly to a synchronization method, apparatus, computer device, and storage medium of incremental data. Background technique [0002] In some data, the upstream system using these data needs to be refreshed with these changes every day because the upstream data is changed every day. When the amount of data is small, the usual refresh method is usually updated each time the upstream data is performed. However, with the growth of business, especially in large data application scenarios, the amount of data will grow in geometric mode, to achieve a billion-level or even more, at this time, if each quadrant update is performed, it will be very expensive. And the downstream system service cannot endure, at which point the change amount synchronous update is required to become increment data synchronization. [0003] Incremental data synchronization, no longer need to copy all upst...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/25G06F16/27G06F16/23
CPCG06F16/254G06F16/27G06F16/23
Inventor 李宇彭博文
Owner BEIJING QIANXIN TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com