Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data synchronization method and system

A data synchronization and data technology, applied in the fields of electronic devices and computer-readable storage media, systems, and data synchronization methods, to achieve the effect of ensuring data transmission quality, order and uniqueness

Active Publication Date: 2021-10-22
SHANGHAI BILIBILI TECH CO LTD
View PDF13 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The main purpose of this application is to propose a data synchronization method, system, electronic device and computer-readable storage medium, aiming to solve the problems of how to get through the synchronization of full data and incremental data, and improve the timeliness and transparency of processing

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data synchronization method and system
  • Data synchronization method and system
  • Data synchronization method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0060] Such as figure 2 As shown, it is a flow chart of a data synchronization method proposed in the first embodiment of the present application. It can be understood that the flowchart in this method embodiment is not used to limit the sequence of execution steps. Some steps in the flow chart can also be added or deleted as needed. The method will be described below with the data synchronization service platform as the execution subject.

[0061] The method includes the following steps:

[0062] S200, collecting full data and incremental data from data sources.

[0063] The data source comes from the data source platform, and in this embodiment, it may be a database such as MySQL or TiDB.

[0064] In view of the defect that the existing technology can only synchronize full data and incremental data separately, and the processing is not timely enough, this embodiment can integrate and process full data and incremental data, and realize minute-level synchronization. In t...

Embodiment 2

[0085] Such as Figure 4 As shown, it is a flow chart of a data synchronization method proposed in the second embodiment of the present application. In the second embodiment, the data synchronization method further includes steps S308-S310 on the basis of the above-mentioned first embodiment. It can be understood that the flowchart in this method embodiment is not used to limit the sequence of execution steps. Some steps in the flow chart can also be added or deleted as needed.

[0086] The method includes the following steps:

[0087] S300, collecting full data and incremental data from data sources.

[0088] The data source comes from the data source platform, and in this embodiment, it may be a database such as MySQL or TiDB.

[0089] In view of the defect that the existing technology can only synchronize full data and incremental data separately, and the processing is not timely enough, this embodiment can integrate and process full data and incremental data, and reali...

Embodiment 3

[0117] Such as Figure 6 As shown, it is a schematic diagram of a hardware architecture of an electronic device 20 proposed in the third embodiment of the present application. In this embodiment, the electronic device 20 may include, but is not limited to, a memory 21 , a processor 22 , and a network interface 23 that may be communicatively connected to each other through a system bus. It should be pointed out that, Figure 6 Only electronic device 20 is shown with components 21-23, but it should be understood that implementing all of the illustrated components is not a requirement and that more or fewer components may instead be implemented. In this embodiment, the electronic device 20 may be the data synchronization service platform 4 .

[0118] The memory 21 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), random access me...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data synchronization method. The method comprises the following steps: collecting total data and incremental data from a data source; allocating an ordered and unique number to each piece of collected data; performing global ordered arrangement and duplicate removal processing on the data according to the numbers; and writing the processed data into a data lake table in sequence. The invention further discloses a data synchronization system, an electronic device and a computer readable storage medium. Therefore, the full-amount data and the incremental data can be integrally communicated, minute-level data synchronization and one-key incremental warehousing are realized, the orderliness and uniqueness of the full-amount data and the incremental data are ensured, and the data transmission quality is ensured.

Description

technical field [0001] The present application relates to the technical field of data transmission and processing, and in particular to a data synchronization method, system, electronic device, and computer-readable storage medium. Background technique [0002] At present, for the data synchronization process from the data source to the data lake table, generally only the synchronization of full data and the synchronization of incremental data can be performed separately. For example, the full data is pulled on a daily basis, the incremental data is pulled every hour, and the full data and the incremental data are merged online. This solution can only achieve hour-level data synchronization, with low processing timeliness and opaque user experience. [0003] It should be noted that the above content is not intended to limit the protection scope of the application. Contents of the invention [0004] The main purpose of this application is to propose a data synchronization...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/27G06F16/25G06F16/22G06F9/54
CPCG06F16/27G06F16/254G06F16/2282G06F9/546G06F2209/548
Inventor 肖宇喻兆靖白鑫王翔宇
Owner SHANGHAI BILIBILI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com