Data synchronization method, system and device supporting breakpoint resume and medium

Abstract

The invention discloses a data synchronization method, system and device supporting breakpoint resume, and a medium. The method comprises the following steps: reading data to be synchronized; dividing the data to be synchronized into a plurality of data blocks; the method comprises the following steps: creating a Checkpoint, wherein the Checkpoint comprises a transmission progress table; the data blocks are sent to the target database one by one, when any data block is successfully transmitted, the transmission progress table in the Checkpoint is updated, and when any data block is interrupted in transmission, the transmission progress table in the Checkpoint is read, the data blocks which are not successfully transmitted are inquired, and then the data blocks are retransmitted until all the data blocks are successfully transmitted. According to the method, the system, the equipment and the medium, the reliability of data transmission and the consistency of data can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data synchronization, and relates to a data synchronization method, system, device and medium supporting breakpoint resume transmission. Background Art

[0002] Enterprises usually have multiple different systems to support their respective business functions. Data synchronization is mainly to solve the data island problem faced by enterprises in a multi-system environment and improve operational efficiency. If the traditional data synchronization method encounters a network interruption or other failure during the transmission process, it is usually necessary to restart the transmission, resulting in resource waste and low efficiency, and the consistency of the data cannot be guaranteed. Therefore, there is an urgent need for a data synchronization method that can support breakpoint resumption to improve the reliability of data transmission and data consistency. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a data synchronization method, system, device and medium that supports breakpoint resumption. The method, system, device and medium can improve the reliability of data transmission and the consistency of data.

[0004] To achieve the above object, the present invention discloses a data synchronization method supporting breakpoint resume transmission, comprising:

[0005] Read the data to be synchronized;

[0006] Splitting the data to be synchronized into a plurality of data blocks;

[0007] Create a Checkpoint, wherein the Checkpoint includes a transmission schedule;

[0008] Each data block is sent to the target database one by one. When any data block is successfully transmitted, the transmission schedule in the Checkpoint is updated. When the transmission of any data block is interrupted, the transmission schedule in the Checkpoint is read to query the data block that has not been successfully transmitted, and then retransmit the data block until all data blocks are successfully transmitted.

[0009] The further improvement of the data synchronization method supporting breakpoint resume transmission of the present invention is:

[0010] Furthermore, the transmission schedule records the task to which each data block belongs, the number of data items, the database storage location and the transmission status, and the transmission status includes to be transmitted, transmitted and transmission failed.

[0011] Furthermore, the process of reading the data to be synchronized is:

[0012] Establish the connection between the data source management module and the source database and the target database;

[0013] The data source management module is used to obtain the data to be synchronized from the source database.

[0014] Furthermore, the data source management module is connected to the source database by using the JDBC protocol.

[0015] Furthermore, the data source management module is connected to the target database by using the JDBC protocol.

[0016] The present invention discloses a data synchronization system supporting breakpoint resume transmission, comprising:

[0017] A reading module, used for reading data to be synchronized;

[0018] A segmentation module, used for segmenting the data to be synchronized into a plurality of data blocks;

[0019] A creation module, used for creating a Checkpoint, wherein the Checkpoint includes a transmission schedule;

[0020] The transmission module is used to send each data block to the target database one by one. When any data block is successfully transmitted, the transmission schedule in the Checkpoint is updated. When the transmission of any data block is interrupted, the transmission schedule in the Checkpoint is read to query the data block that has not been successfully transmitted, and then retransmit the data block until all data blocks are successfully transmitted.

[0021] The further improvement of the data synchronization system supporting breakpoint resume transmission of the present invention is:

[0022] Furthermore, the transmission schedule records the task to which each data block belongs, the number of data items, the database storage location and the transmission status, and the transmission status includes to be transmitted, transmitted and transmission failed.

[0023] Furthermore, the reading module includes:

[0024] Establishing module, used to establish the connection between data source management module and source database and target database;

[0025] The acquisition module is used to acquire the data to be synchronized from the source database using the data source management module.

[0026] The present invention discloses a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data synchronization method supporting breakpoint resume are implemented.

[0027] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data synchronization method supporting breakpoint resume transmission are implemented.

[0028] The present invention has the following beneficial effects:

[0029] The data synchronization method, system, device and medium supporting breakpoint resumption described in the present invention create a Checkpoint during specific operation. The Checkpoint includes a transmission schedule that records the status of each data block. When the transmission of any data block is interrupted, the transmission schedule in the Checkpoint is read to query the data block that has not been successfully transmitted, and then retransmit the data block until all data blocks are successfully transmitted, thereby improving the reliability of data transmission and the consistency of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0031] Figure 1 is a flow chart of the method of the present invention;

[0032] Figure 2 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0036] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0037] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0038] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0041] Embodiment 1

[0042] The principle of the present invention is as follows: read all data to be synchronized in this task from the source database; divide the read data into blocks, generate a unique identifier for each data block, record the data block information in the Checkpoint, and store the Checkpoint on the disk of the server to which the synchronization device belongs; send the data blocks to the target database respectively, and update the data block transmission progress in the Checkpoint; if all data blocks are synchronized successfully, the task is marked as successful, and the Checkpoint information and data blocks are deleted, otherwise, the task is marked as failed; if the synchronization task fails due to network interruption or other abnormalities, after the network is restored, read the unsynchronized data blocks recorded in the Checkpoint and retransmit them.

[0043] refer to Figure 1 The data synchronization method supporting breakpoint resume of the present invention comprises the following steps:

[0044] 1) Initialization phase;

[0045] 11) Establish a connection;

[0046] The data source management module uses the JDBC protocol to establish a connection with the source database.

[0047] 12) Prepare data;

[0048] Read the data to be synchronized from the source database.

[0049] 13) Data segmentation;

[0050] The data to be synchronized is divided into several data blocks, and the size of each data block can be fixed (eg, 1 MB) or dynamically adjusted according to the network bandwidth.

[0051] Generate a unique identifier for each data block. Specifically, generate a unique identifier for each data block through a hash function or uuid.

[0052] The generated data blocks are stored in the disk of the server where the synchronization device is located.

[0053] 14) Create Checkpoint;

[0054] A directory for storing Checkpoints is created in the server where the synchronization device is located. Checkpoints contain a transmission schedule table for recording the tasks to which each data block belongs, the number of data items, the database storage location, and the transmission status. The transmission status includes pending transmission, transmitted, and transmission failure. The transmission schedule table structure is shown in Table 1:

[0055] Table 1

[0056]

[0057] 2) Data transmission stage;

[0058] 21) Data block sending;

[0059] The data source management module establishes a connection with the target database through the JDBC protocol and sends each data block to the data target database one by one.

[0060] 22) Checkpoint records;

[0061] During the data transmission process, each time a data block is successfully transmitted, the transmission schedule in the Checkpoint is updated and the transmission status of the data block is changed to transmitted.

[0062] 3) Breakpoint resume stage;

[0063] 31) Restore the connection;

[0064] When the network is restored or the device is restarted, the data source management module re-establishes the communication connection with the target database.

[0065] 32) Read Checkpoint;

[0066] Read the transfer progress table in the local checkpoint to obtain the status when the last transfer was interrupted.

[0067] 33) Continue transmission;

[0068] Find the data block that has not been successfully transmitted from the transmission progress table in the Checkpoint, and continue transmitting the data block from the last interrupted position until all data blocks are successfully transmitted.

[0069] 4) Clean up the checkpoint;

[0070] When all data blocks are successfully synchronized to the target database, the synchronization status is marked as completed, and the Checkpoint file and all data blocks are deleted.

[0071] It should be noted that the present invention effectively solves the problem that all data need to be re-transmitted due to network interruption or equipment failure during the transmission process of traditional data synchronization methods, reduces resource usage, improves data transmission efficiency, and ensures data consistency.

[0072] Embodiment 2

[0073] refer to Figure 2 The data synchronization system supporting breakpoint-resume transmission of the present invention comprises:

[0074] A reading module, used for reading data to be synchronized;

[0075] A segmentation module, used for segmenting the data to be synchronized into a plurality of data blocks;

[0076] A creation module, used for creating a Checkpoint, wherein the Checkpoint includes a transmission schedule;

[0077] The transmission module is used to send each data block to the target database one by one. When any data block is successfully transmitted, the transmission schedule in the Checkpoint is updated. When the transmission of any data block is interrupted, the transmission schedule in the Checkpoint is read to query the data block that has not been successfully transmitted, and then retransmit the data block until all data blocks are successfully transmitted.

[0078] As an implementation mode of the present invention, the transmission schedule records the task to which each data block belongs, the number of data items, the database storage location and the transmission status, and the transmission status includes to be transmitted, transmitted and transmission failed.

[0079] As an implementation manner of the present invention, the reading module includes:

[0080] Establishing module, used to establish the connection between data source management module and source database and target database;

[0081] The acquisition module is used to acquire the data to be synchronized from the source database using the data source management module.

[0082] As an implementation mode of the present invention, the data source management module is connected to the source database by using the JDBC protocol.

[0083] As an implementation of the present invention, the data source management module is connected to the target database by using the JDBC protocol.

[0084] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0085] Embodiment 3

[0086] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data synchronization method supporting breakpoint resumption when executing the computer program, for example, comprising: establishing a connection between a data source management module and a source database and a target database; obtaining data to be synchronized from the source database using the data source management module; dividing the data to be synchronized into a plurality of data blocks; creating a Checkpoint, wherein the Checkpoint comprises a transmission schedule; and sending each data block to the target database one by one, wherein when any data block is successfully transmitted, the transmission schedule in the Checkpoint is updated, and when any data block transmission is interrupted, the transmission schedule in the Checkpoint is read, the data block that has not been successfully transmitted is queried, and the data block is retransmitted until all data blocks are successfully transmitted, wherein the transmission schedule records the task to which each data block belongs, the number of data items, the database storage location and the transmission status, wherein the transmission status comprises to be transmitted, transmitted and transmission failed. The memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc. The processor, network interface, and memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0087] Embodiment 4

[0088] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data synchronization method supporting breakpoint resumption are implemented, for example, including: establishing a connection between a data source management module and a source database and a target database; using the data source management module to obtain data to be synchronized from the source database; dividing the data to be synchronized into a plurality of data blocks; creating a Checkpoint, wherein the Checkpoint includes a transmission schedule; sending each data block to the target database one by one, wherein when any data block is successfully transmitted, the transmission schedule in the Checkpoint is updated, and when any data block transmission is interrupted, the transmission schedule in the Checkpoint is read, the data block that has not been successfully transmitted is queried, and the data block is retransmitted until all data blocks are successfully transmitted, and the transmission schedule records the task to which each data block belongs, the number of data items, the database storage location and the transmission status, and the transmission status includes to be transmitted, transmitted and failed transmission. Specifically, the computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory, etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, and the like.

[0089] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0093] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0094] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0095] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A data synchronization method supporting breakpoint resuming, characterized in that: include: Read the data to be synchronized; Splitting the data to be synchronized into a plurality of data blocks; Create a Checkpoint, wherein the Checkpoint includes a transmission schedule; Each data block is sent to the target database one by one. When any data block is successfully transmitted, the transmission schedule in the Checkpoint is updated. When the transmission of any data block is interrupted, the transmission schedule in the Checkpoint is read to query the data block that has not been successfully transmitted, and then retransmit the data block until all data blocks are successfully transmitted.

2. The data synchronization method supporting breakpoint resuming according to claim 1, characterized in that: The transmission schedule records the task to which each data block belongs, the number of data items, the database storage location, and the transmission status, wherein the transmission status includes to be transmitted, transmitted, and transmission failed.

3. The data synchronization method supporting breakpoint resuming according to claim 1, characterized in that: The process of reading the data to be synchronized is as follows: Establish the connection between the data source management module and the source database and the target database; The data source management module is used to obtain the data to be synchronized from the source database.

4. The data synchronization method supporting breakpoint resume according to claim 1, characterized in that: The data source management module is connected to the source database using the JDBC protocol.

5. The data synchronization method supporting breakpoint resume according to claim 1, characterized in that: The data source management module is connected to the target database using the JDBC protocol.

6. A data synchronization system supporting breakpoint resuming, characterized in that: include: A reading module, used for reading data to be synchronized; A segmentation module, used for segmenting the data to be synchronized into a plurality of data blocks; A creation module, used for creating a Checkpoint, wherein the Checkpoint includes a transmission schedule; The transmission module is used to send each data block to the target database one by one. When any data block is successfully transmitted, the transmission schedule in the Checkpoint is updated. When the transmission of any data block is interrupted, the transmission schedule in the Checkpoint is read to query the data block that has not been successfully transmitted, and then retransmit the data block until all data blocks are successfully transmitted.

7. The data synchronization system supporting breakpoint resume according to claim 1, characterized in that: The transmission schedule records the task to which each data block belongs, the number of data items, the database storage location, and the transmission status, wherein the transmission status includes to be transmitted, transmitted, and transmission failed.

8. The data synchronization system supporting breakpoint resume according to claim 1, characterized in that: The reading module comprises: Establishing module, used to establish the connection between data source management module and source database and target database; The acquisition module is used to acquire the data to be synchronized from the source database using the data source management module.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the data synchronization method supporting breakpoint resumption as described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data synchronization method supporting breakpoint resumption as claimed in any one of claims 1 to 5 are implemented.

Images