Data synchronization method, system and equipment for multiple data sources and medium

A data synchronization and multi-data source technology, applied in the direction of structured data retrieval, database distribution/replication, etc., can solve the problems of incompatibility with full-scale big data and incremental small data business needs, and achieve efficient data synchronization Effect

Pending Publication Date: 2022-04-01
CTRIP COMP TECH SHANGHAI
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to overcome the defect that the data synchronization method in the prior art cannot be compatible with the business requirements of full-scale large data and incremental small data, and provide a data synchronization method, system, and Equipment and media

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data synchronization method, system and equipment for multiple data sources and medium
  • Data synchronization method, system and equipment for multiple data sources and medium
  • Data synchronization method, system and equipment for multiple data sources and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0048] In this embodiment, a data synchronization method for multiple data sources is provided, such as figure 1 As shown, the data synchronization method includes:

[0049] S11. Receive raw data within a preset period; the raw data includes at least one initial data source of the database, and the initial data source includes incremental data identifiers and full data identifiers, incremental data identifiers are used to identify dynamically added data, and full data identifiers are used for identifying static historical data.

[0050] S12. Acquire at least one target data source from the initial data source based on the incremental data identifier and / or the full data identifier.

[0051] S13. Data synchronization is performed on all target data sources in sequence and stored in the synchronization database.

[0052] S14. Collect configuration information of multiple storage databases; the configuration information includes a data format.

[0053] S15. Convert the synchro...

Embodiment 2

[0067] On the basis of Example 1, such as image 3 As shown, this embodiment provides a data synchronization method for multiple data sources, and step S12 specifically includes:

[0068] S123. Based on the incremental data identifier, intercept the data in the second access period from each initial data source to generate an intermediate data source.

[0069] S124. Intercept several second sample data from the intermediate data source in sequence according to the length of the second field, and generate a target data source according to the second sample data.

[0070] Specifically, set the second data fetching period as 15 minutes, that is, the data fetching is performed every 15 minutes. Exemplarily, based on the incremental data identifier, the first time range corresponding to the second access period in the initial data source (for example, 2021-11-18 18:00:00 to 2021-11-18 18:15:00 ) dynamic incremental data, due to the large data volume of the dynamic incremental dat...

Embodiment 3

[0074] This embodiment provides a data synchronization system with multiple data sources, such as Figure 4 As shown, it specifically includes: a receiving module 210 , an acquiring module 220 , a synchronizing module 230 , a collecting module 240 and a converting module 250 .

[0075] Among them, the receiving module 210 is used to receive the original data within a preset period; the original data includes at least one initial data source of the database, and the initial data source includes an incremental data identifier and a full amount of data identifier, and the incremental data identifier is used to identify dynamically newly added Data, full data identification is used to identify static historical data;

[0076] An acquisition module 220, configured to acquire at least one target data source from the initial data source based on the incremental data identifier and / or the full data identifier;

[0077] The synchronization module 230 is configured to sequentially perf...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data synchronization method, system and device for multiple data sources and a medium. The data synchronization method comprises the steps of receiving original data in a preset period; acquiring at least one target data source from the initial data source based on the incremental data identifier and/or the total data identifier; and performing data synchronization on all the target data sources in sequence and storing the data in a synchronous database. According to the method, time-consuming, stable and efficient second-level data synchronization is realized, and the service requirements of obtaining and data synchronization compatible with full-amount big data and incremental small data at the same time are met.

Description

technical field [0001] The invention relates to the technical field of database synchronization, in particular to a data synchronization method, system, device and medium of multiple data sources. Background technique [0002] With the development of science and technology, there are more and more demands for real-time data synchronization and mutual transmission between multiple devices, multiple nodes, and multiple systems, and data synchronization between different data sources in different computer network domains is required. The larger the data volume of the synchronous data, the higher the quality requirement of the synchronous data. [0003] In the database field, for Sqlserver database and Mysql database, the use of datax components can only realize the synchronization of a large amount of data in the database, and the amount of data transferred is large, but it cannot support the synchronization of a small amount of variable data; However, using the canal componen...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27
Inventor 张帆陈康
Owner CTRIP COMP TECH SHANGHAI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap