Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data synchronization method and system

A data synchronization and corresponding data technology, applied in the field of data processing, can solve problems such as wasting resources, synchronization results cannot be retained, and operation time is affected

Inactive Publication Date: 2017-04-05
ALIBABA GRP HLDG LTD
View PDF4 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The offline synchronization cycle is long, and the synchronization process is heavily dependent on the stability of the source, execution path (gateway), and destination.
During the synchronization process, a task can be divided into multiple task shards for processing. However, if the synchronization of a shard fails, the entire task will fail, and the synchronization results of other shards cannot be retained.
[0004] If the above-mentioned shard synchronization failure occurs, it is often necessary to reprocess the entire task, wasting resources and affecting operation time
[0005] Therefore, a technical problem that needs to be urgently solved by those skilled in the art is: to propose a data synchronization method and system to solve the synchronization failure problem of the above-mentioned data synchronization

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data synchronization method and system
  • Data synchronization method and system
  • Data synchronization method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0031] Reference figure 1 , Shows a step flowchart of an embodiment of a data synchronization method of the present application, which may specifically include the following steps:

[0032] Step 102: Assign a task to each data segment of the data set to be processed.

[0033] Step 104: Start the task thread of the task, and perform offline data synchronization of corresponding data fragments between the source and the destination.

[0034] When performing offline data synchronization between different databases / file systems, set the data set to be synchronized as the data set to be processed. You can set the source and destination of offline data synchronization, and use the database / file system where the data set to be processed is located. At the source, the database / file system to which the data set to be processed is to be synchronized is used as the destination. Among them, the data set to be processed can be regarded as a collection of business data, which includes a large amo...

Embodiment 2

[0042] On the basis of the foregoing embodiment, this embodiment discusses in detail the offline data synchronization operation based on failover.

[0043] The offline data synchronization in the embodiments of this application can be applied to the offline synchronization of datax. Datax is a tool for high-speed data exchange between heterogeneous databases / file systems. It can be used in any data processing system (such as RDBMS / Hdfs / Local filesystem) data exchange.

[0044] DataX has the following characteristics: high-speed exchange of data between heterogeneous databases / file systems; using Framework+plugin architecture, Framework handles most of the technical issues of high-speed data exchange such as buffering, flow control, concurrency, context loading, etc. Simple interface and plug-in interaction, the plug-in only needs to realize the access to the data processing system; the operating mode is stand-alone; the data transmission process is completed in a single process, f...

Embodiment 3

[0091] On the basis of the foregoing embodiment, this embodiment also discloses a data synchronization system.

[0092] Reference image 3 , Shows a structural block diagram of an embodiment of a data synchronization system of the present application, which may specifically include the following modules:

[0093] The task allocation module 302 is configured to allocate a task to each data segment of the data set to be processed; and to re-allocate a task to the data segment corresponding to the failed task.

[0094] The data synchronization module 304 is used to start the task thread of the task, perform offline data synchronization of the corresponding data segment between the source and the destination; and start the task thread of the reallocated task to execute the data segment at the source Offline data synchronization between the end and the destination end.

[0095] The failover module 306 is configured to clean up the processing resources of the data segment corresponding to t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

Embodiments of the invention provide a data synchronization method and system, and aim to solve the problem of data synchronization failure. The method comprises the steps of allocating a task to each data segment of a to-be-processed data set; starting task threads of the tasks, and executing offline data synchronization of the corresponding data segments between a source end and a destination end; after it is judged that the synchronization of a task corresponding to any data segment fails, if it is determined that the failed task supports failover operation, clearing processing resources of the data segment corresponding to the failed task; and reallocating a task to the data segment corresponding to the failed task, and starting a task thread of the reallocated task to execute offline data synchronization of the data segment between the source end and the destination end. The data segment of the failed task is directly re-synchronized, and the whole to-be-processed data set does not need to be reprocessed, so that the resources are saved and the synchronization time is shortened.

Description

Technical field [0001] This application relates to the field of data processing technology, and in particular to a data synchronization method and a data synchronization system. Background technique [0002] With the development of network technology, there are more and more interactions between various databases or file systems, but there are many types of databases and file systems, so there are often data reading and writing between different types of databases / file systems. [0003] When performing data reading and writing between many different types of databases / file systems, that is, when data is imported and exported, it is sometimes necessary to perform offline synchronization. The offline synchronization cycle is long, and the synchronization process is heavily dependent on the stability of the source, the execution path (gateway), and the destination. During the synchronization process, a task can be divided into multiple task fragments for processing. However, if the s...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/50
CPCG06F9/50H04L67/1095G06F16/27G06F9/4856G06F9/5027G06F21/1085G06F21/1087G06F9/4881G06F11/1662G06F2201/80
Inventor 刘益
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com