Incremental data synchronization method, device and electronic device

Abstract

The embodiments of the present invention provide an incremental data synchronization method, device and electronic device. It relates to the field of cloud computing technology and is applied to the synchronization process of incremental data between a source database and a destination database, including: when executing the first task of incremental data synchronization, writing the task information corresponding to the first task into a preset metadata table; when an exception occurs in the first task, determining the starting point of the second task of incremental data synchronization based on the task information corresponding to the first task in the preset metadata table, the second task being an overloaded task of the first task; based on the starting point of the second task, synchronizing the incremental data in the source database to the destination database. Therefore, the starting point of the incremental data synchronization overload task can be quickly determined by querying the metadata table, and the breakpoint-resume transmission of the incremental data can be realized based on the starting point. At the same time, the metadata table is stored in the metadata, and the source database or the destination database will not be modified, thereby achieving the purpose of avoiding data intrusion.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and more specifically, to an incremental data synchronization method, an incremental data synchronization device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Incremental data synchronization is an important part of Data Transmission Service (DTS), which is used to achieve real-time synchronization and update of data between the target database and the source database. When performing incremental data synchronization, you may encounter situations such as database connection failure or task abnormal interruption, in which case you need to resume the transfer from breakpoint.

[0003] In the related art, the commonly used method of breakpoint resuming is to record point information, such as recording the interruption time point or checkpoint in the destination database. However, this method may cause data inconsistency between the destination database and the source database, and recording a point information is not suitable for multi-threaded transmission. Therefore, it is necessary to improve the above defects. Summary of the invention

[0004] An object of an embodiment of the present invention is to provide a new technical solution for incremental data synchronization.

[0005] According to a first aspect of the present invention, a method for synchronizing incremental data is provided. The method is applied to a process of synchronizing incremental data between a source database and a target database. The method comprises:

[0006] When executing a first task of incremental data synchronization, writing task information corresponding to the first task into a preset metadata table;

[0007] When an exception occurs in the first task, a starting point of a second task for synchronizing the incremental data is determined based on the task information corresponding to the first task in the preset metadata table, where the second task is an overload task of the first task;

[0008] Based on the starting point of the second task, the incremental data in the source database is synchronized to the destination database.

[0009] Optionally, the method further comprises:

[0010] After synchronizing the incremental data in the source database to the destination database based on the starting point of the second task, the task information corresponding to the second task is updated to the preset metadata database.

[0011] Optionally, the task information includes task identification information of the incremental data synchronization, thread identification information of all threads corresponding to the task of the incremental data synchronization, and point information corresponding to each thread identification information.

[0012] When an exception occurs in the first task, determining the starting point of the second task of the incremental data synchronization based on the task information corresponding to the first task in the preset metadata table includes:

[0013] Determine thread identification information of all threads corresponding to the task identification information of the incremental data synchronization;

[0014] Based on the thread identification information, determining the point information corresponding to each thread;

[0015] The starting point of the second task of the incremental data synchronization is determined from the point information corresponding to all threads.

[0016] Optionally, determining the starting point of the second task of the incremental data synchronization from the point information corresponding to all threads includes:

[0017] Compare the sizes of the points corresponding to the point information of all threads;

[0018] The minimum point is determined as the starting point of the second task of the incremental data synchronization.

[0019] Optionally, when an exception occurs in the first task, before determining the starting point of the second task of the incremental data synchronization based on the task information corresponding to the first task in the preset metadata table, the method further includes:

[0020] Obtaining a task reload instruction; wherein the task reload instruction at least includes task identification information of incremental data synchronization to be performed on the task reload;

[0021] The determining the starting point of the second task of the incremental data synchronization based on the task information corresponding to the first task in the preset metadata table includes:

[0022] Finding the task identification information contained in the task reload instruction from the task information corresponding to the first task in the preset metadata table;

[0023] The starting point of the second task corresponding to the task identification information and to be synchronized with the incremental data to be task-overloaded is determined by using the task information of the first task corresponding to the found task identification information.

[0024] Optionally, when executing the first task of incremental data synchronization, writing the task information corresponding to the first task into a preset metadata table includes:

[0025] Obtaining a task start instruction for the first task of the incremental data synchronization;

[0026] According to the task start instruction, a first task of incremental data synchronization is executed to synchronize the incremental data from the source database to the destination database, and task information corresponding to the first task is written into a preset metadata table.

[0027] According to a second aspect of the present invention, there is provided an incremental data synchronization device, the device comprising:

[0028] A writing module, used to write task information corresponding to the first task into a preset metadata table when executing the first task of incremental data synchronization;

[0029] A determination module, used for determining a starting point of a second task of the incremental data synchronization based on the task information corresponding to the first task in the preset metadata table when an exception occurs in the first task, the second task being an overload task of the first task;

[0030] A synchronization module is used to synchronize the incremental data in the source database to the destination database based on the starting point of the second task.

[0031] Optionally, the writing module is further used to: after synchronizing the incremental data in the source database to the destination database based on the starting point of the second task, update the task information corresponding to the second task into the preset metadata database.

[0032] According to a third aspect of the present invention, there is provided an electronic device, comprising the incremental data synchronization device according to the second aspect of the present invention, or the electronic device comprises:

[0033] A memory for storing executable commands;

[0034] A processor is used to execute the incremental data synchronization method as described in any one of the first aspects of the present invention under the control of the executable command.

[0035] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing executable instructions, which, when executed by a processor, executes the incremental data synchronization method as described in any one of the first aspect of the present invention.

[0036] According to an embodiment of the present invention, in the process of synchronizing incremental data between a source database and a destination database, when executing the first task of incremental data synchronization, the task information corresponding to the first task is written into a preset metadata table; when an exception occurs in the first task, the starting point of the second task of incremental data synchronization is determined based on the task information corresponding to the first task in the preset metadata table, and the second task is an overloaded task of the first task; based on the starting point of the second task, the incremental data in the source database is synchronized to the destination database. In this way, when an exception occurs in the incremental data synchronization task, the starting point of the incremental data synchronization overload task can be quickly determined by querying the metadata table, and the breakpoint-resume transmission of the incremental data can be realized based on the starting point; at the same time, the metadata table used to record the task information is stored in the metadata, and the source database or the destination database will not be modified, thereby achieving the purpose of avoiding data intrusion.

[0037] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0039] Figure 1 A schematic diagram of the composition structure of a server to which the incremental data synchronization method according to an embodiment of the present invention can be applied;

[0040] Figure 2 is a schematic flow chart of an incremental data synchronization method according to an embodiment of the present invention;

[0041] Figure 3 is a schematic flow chart of an example of an incremental data synchronization method according to an embodiment of the present invention;

[0042] Figure 4 is a principle block diagram of an incremental data synchronization device according to an embodiment of the present invention;

[0043] Figure 5 is a principle block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0045] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0046] Technologies, methods, and apparatuses known to persons of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods, and apparatuses should be considered part of the specification.

[0047] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0048] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0049] <Hardware Configuration>

[0050] Figure 1 The figure is a schematic diagram of the composition structure of a server to which the incremental data synchronization method according to an embodiment of the present invention can be applied.

[0051] The server 1000 may be, for example, a blade server, etc. In one example, the server 1000 may be a computer.

[0052] In another example, the server 1000 may be Figure 1 As shown, the server includes a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, and an input device 1600. Although the server may also include a speaker, a microphone, etc., these components are irrelevant to the present invention and are therefore omitted here.

[0053] The processor 1100 may be, for example, a central processing unit (CPU), a microprocessor (MCU), etc. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 1300 may include, for example, a USB interface, a serial interface, etc. The communication device 1400 may be capable of wired or wireless communication, for example. The display device 1500 may be, for example, a liquid crystal display screen. The input device 1600 may include, for example, a touch screen, a keyboard, etc.

[0054] Figure 1 The server shown is merely illustrative and does not in any way imply any limitation on the present invention, its application or use. In the embodiments of the present invention, the memory 1200 of the server 1000 is used to store instructions, which are used to control the processor 1100 to operate to perform any incremental data synchronization method provided in the embodiments of the present invention.

[0055] It should be understood by those skilled in the art that although Figure 1In the figure, multiple devices of the server 1000 are shown, but the present invention may only involve some of the devices, for example, the server 1000 only involves the processor 1100 and the storage device 1200. A technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0056] <Method Example>

[0057] This embodiment provides an incremental data synchronization method, which is applied to the synchronization process of incremental data between a source database and a target database. The method can be executed by a server, for example, which can be Figure 1 Server 1000 is shown.

[0058] like Figure 2 As shown, the method includes the following steps 2100 to 2300:

[0059] In step 2100, when executing a first task of incremental data synchronization, task information corresponding to the first task is written into a preset metadata table.

[0060] The first task is used to synchronize the incremental data from the source database to the destination database. Specifically, the server 1000 may first obtain a task start instruction for the first task of the incremental data synchronization; then, according to the task start instruction, execute the first task of the incremental data synchronization to synchronize the incremental data from the source database to the destination database, and write the task information corresponding to the first task into the preset metadata table.

[0061] In one example, the metadata table may be established by the server 1000 when executing the first task of incremental data synchronization. In other embodiments, the metadata table may be established before executing the first task and used to store the task information during incremental data synchronization. The metadata table may be established in the source database or in another database or server that is independent of the source database and the destination database.

[0062] The task information includes the task identification information of the incremental data synchronization, the thread identification information of all threads corresponding to the task of the incremental data synchronization, and the point information corresponding to each of the thread identification information.

[0063] In actual applications, the incremental data synchronization task can be executed by multiple threads, that is, the server 1000 can distribute the incremental data to multiple threads for synchronization. While the multiple threads are transmitting the incremental data from the source database to the destination database, the server 1000 can update the thread identification information of each thread and the point information corresponding to each thread identification information to the metadata table. For example, the server 1000 can update the thread identification information of each thread and the point information corresponding to each thread identification information to the metadata table at a preset time interval, or can update the thread identification information of each thread and the point information corresponding to each thread identification information to the metadata table in real time, which is not limited in this embodiment.

[0064] Step 2200, when an exception occurs in the first task, a starting point of a second task for synchronizing the incremental data is determined based on the task information corresponding to the first task in the preset metadata table, where the second task is an overloaded task of the first task.

[0065] In one example, before this step 2200, when the server 1000 is synchronizing incremental data from a source database to a destination database, a task exception may occur, such as a database connection failure or an abnormal task interruption. In this case, when the server 1000 detects that an exception has occurred in the first task of the incremental data synchronization, it will obtain a task reload instruction, which includes at least the task identification information of the incremental data synchronization to be reloaded.

[0066] When determining the starting point of the second task of the incremental data synchronization, the server 1000 can specifically find the task identification information contained in the task overload instruction from the task information corresponding to the first task in the preset metadata table, and then determine the starting point of the second task of the incremental data synchronization to be overloaded corresponding to the task identification information through the task information of the first task corresponding to the found task identification information.

[0067] Since the task of incremental data synchronization is executed by multiple threads separately, the incremental data points synchronized by each thread are different. Therefore, when a task exception occurs, the server 1000 needs to determine the minimum point from the metadata table as the starting point for task reloading, otherwise data leakage may occur.

[0068] Specifically, the server 1000 first determines the thread identification information of all threads corresponding to the task identification information of the incremental data synchronization; then, based on the thread identification information, determines the point information corresponding to each thread; and then determines the starting point of the second task of the incremental data synchronization from the point information corresponding to all threads.

[0069] When determining the starting point of the second task of incremental data synchronization, the server 1000 may specifically compare the sizes of the points corresponding to the point information corresponding to all threads, and determine the smallest point as the starting point of the second task of incremental data synchronization.

[0070] In summary, the embodiment of the present invention records the task identification information while also synchronously recording the thread identification information. During the incremental synchronization process, when breakpoint transmission is performed after an exception occurs, the incremental synchronization task can be reloaded by searching the thread identification information, and breakpoint transmission can be implemented for multiple threads of any incremental synchronization task (for example, breakpoint transmission is performed when the second task is started). In addition, in the embodiment of the present invention, the minimum point position (for example, the point position of the slowest progressing thread in the first task) is determined as the starting point of the breakpoint transmission, which can minimize the omission of incremental data transmission.

[0071] In order to enable those skilled in the art to clearly and accurately understand the above embodiment, the above embodiment will be described below in conjunction with a specific type of database:

[0072] For example, in a MySQL database, if the global transaction identifier (GTID) is enabled, the point information corresponding to each thread includes transaction_id. When comparing the sizes of the points corresponding to the point information, the server 1000 can compare the transaction_id and determine the point with the smallest transaction_id as the starting point of the second task of the incremental data synchronization.

[0073] If GTID is not enabled, the point information corresponding to each thread includes binlog file and position. When comparing the sizes of the points corresponding to the point information, the server 1000 may first compare the binlog files. If the binlog files are the same, the position is compared, and the point with the smallest position is determined as the starting point of the second task of the incremental data synchronization; if the binlog files are different, the point with the smallest binlog file is determined as the starting point of the second task of the incremental data synchronization.

[0074] After determining the starting point of the second task of the incremental data synchronization, the server 1000 executes step 2300 .

[0075] In step 2300, based on the starting point of the second task, the incremental data in the source database is synchronized to the destination database.

[0076] Furthermore, after this step, the server 1000 may also update the task information corresponding to the second task into the preset metadata database.

[0077] <Example>

[0078] This embodiment takes the data migration task of MySQL database as an example to illustrate the above method. Figure 1 The server 1000 shown in FIG.

[0079] like Figure 3 As shown, in this example, the incremental data synchronization method may specifically include steps 3100 to 3900:

[0080] Step 3100, obtaining the start instruction of the first task of the incremental data synchronization.

[0081] The start instruction of the first task of the incremental data synchronization may be, for example, a start instruction of a data migration task.

[0082] Step 3200: According to the startup instruction, synchronize the incremental data from the source database to the target database, and establish the metadata table.

[0083] The metadata table includes multiple task information, each of which may include the task identification information of the incremental data synchronization, the thread identification information of all threads corresponding to the task of the incremental data synchronization, and the point information corresponding to each of the thread identification information. While synchronizing the incremental data from the source database to the destination database, the thread identification information of each thread and the point information corresponding to each thread identification information are updated to the preset metadata table.

[0084] Step 3300: when it is detected that an exception occurs in the first task of the incremental data synchronization, a task reload instruction is obtained; the task reload instruction includes at least task identification information of the incremental data synchronization to be reloaded.

[0085] Step 3400, based on the task information corresponding to the first task of incremental data synchronization obtained, search for the task identification information contained in the task reload instruction, and determine the starting point of the second task of incremental data synchronization to be reloaded corresponding to the task identification information through the task information of the first task corresponding to the found task identification information.

[0086] Step 3500: determine the thread identification information of all threads corresponding to the task identification information of the incremental data synchronization; and then determine the point information corresponding to each thread based on the thread identification information.

[0087] The point information includes, for example, transaction_id.

[0088] Step 3600, compare the size of transaction_id.

[0089] Step 3700: determine the smallest transaction_id as the starting point of the second task of the incremental data synchronization.

[0090] Step 3800: Based on the starting point of the second task, synchronize the incremental data in the source database to the destination database.

[0091] Step 3900: Update the task information corresponding to the second task into the preset metadata database.

[0092] The incremental data synchronization method of the present embodiment has been described above with reference to the accompanying drawings and specific examples. When executing the first task of incremental data synchronization, the task information corresponding to the first task is written into the preset metadata table; when an exception occurs in the first task, the starting point of the second task of incremental data synchronization is determined based on the task information corresponding to the first task in the preset metadata table, and the second task is an overloaded task of the first task; based on the starting point of the second task, the incremental data in the source database is synchronized to the destination database. In this way, when an exception occurs in the incremental data synchronization task, the starting point of the incremental data synchronization overload task can be quickly determined by querying the metadata table, and the breakpoint resumption of incremental data can be achieved based on the starting point; at the same time, the metadata table used to record task information is stored in the metadata, and the source database or the destination database will not be modified, thereby achieving the purpose of avoiding data intrusion.

[0093] <Device Example>

[0094] This embodiment provides an incremental data synchronization device, which is, for example, Figure 4 The incremental data synchronization device 4000 shown in FIG. 4 can be applied to, for example, Figure 1 In the server 1000 shown.

[0095] Specifically, the incremental data synchronization device 4000 may include: a writing module 4100 , a determining module 4200 and a synchronization module 4300 .

[0096] The writing module 4100 is used to write task information corresponding to the first task into a preset metadata table when executing the first task of incremental data synchronization.

[0097] The determination module 4200 is used to determine the starting point of the second task of the incremental data synchronization based on the task information corresponding to the first task in the preset metadata table when an exception occurs in the first task, and the second task is an overload task of the first task.

[0098] The synchronization module 4300 is used to synchronize the incremental data in the source database to the destination database based on the starting point of the second task.

[0099] Optionally, the writing module 4100 may also be used to: after synchronizing the incremental data in the source database to the destination database based on the starting point of the second task, update the task information corresponding to the second task into the preset metadata database.

[0100] In one example, the task information includes task identification information of the incremental data synchronization, thread identification information of all threads corresponding to the incremental data synchronization task, and point information corresponding to each thread identification information.

[0101] Correspondingly, the determination module 4200 can be used to determine the thread identification information of all threads corresponding to the task identification information of the incremental data synchronization; based on the thread identification information, determine the point information corresponding to each thread; and determine the starting point of the second task of the incremental data synchronization from the point information corresponding to all threads.

[0102] Among them, when the determination module 4200 determines the starting point of the second task of the incremental data synchronization from the point information corresponding to all threads, it can specifically compare the sizes of the points corresponding to the point information corresponding to all threads; and determine the smallest point as the starting point of the second task of the incremental data synchronization.

[0103] In one example, the incremental data synchronization device 4000 may also include: an acquisition module for acquiring a task reload instruction; the task reload instruction at least includes task identification information of the incremental data synchronization to be reloaded. Accordingly, the determination module 4200 may be used to find the task identification information contained in the task reload instruction from the task information corresponding to the first task in the preset metadata table; and determine the starting point of the second task of the incremental data synchronization to be reloaded corresponding to the task identification information by using the task information of the first task corresponding to the found task identification information.

[0104] In one example, the writing module 4100 is specifically used to obtain the task start instruction of the first task of the incremental data synchronization; according to the task start instruction, the first task of the incremental data synchronization is executed to synchronize the incremental data from the source database to the destination database, and the task information corresponding to the first task is written into the preset metadata table.

[0105] The incremental data synchronization device of this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0106] <Equipment Embodiment>

[0107] In this embodiment, an electronic device is also provided, the electronic device includes the incremental data synchronization device 4000 described in the device embodiment of the present invention; or the electronic device is Figure 5 The electronic device 5000 shown includes:

[0108] The memory 5100 is used to store executable commands.

[0109] The processor 5200 is used to execute the method described in any method embodiment of the present invention under the control of the executable instructions stored in the memory 5100.

[0110] The implementation subject of the method embodiment executed in the electronic device may be a server or a terminal device.

[0111] <Computer Readable Storage Medium Embodiment>

[0112] This embodiment provides a computer-readable storage medium, in which executable commands are stored. When the executable commands are executed by a processor, the method described in any method embodiment of the present invention is executed.

[0113] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0114] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0115] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0116] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0117] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0118] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0119] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0120] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and a part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0121] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for synchronizing incremental data, characterized in that: The method is applied to the synchronization process of incremental data between a source database and a target database, and the method includes: When executing the first task of incremental data synchronization, the task information corresponding to the first task is written into the preset metadata table, wherein the task information includes the task identification information of the incremental data synchronization, the thread identification information of all threads corresponding to the task of the incremental data synchronization, and the point information corresponding to each of the thread identification information; When an exception occurs in the first task, the starting point of the second task of the incremental data synchronization is determined based on the task information corresponding to the first task in the preset metadata table, and the second task is an overloaded task of the first task, specifically including: determining the thread identification information of all threads corresponding to the task identification information of the incremental data synchronization; determining the point information corresponding to each thread based on the thread identification information; determining the starting point of the second task of the incremental data synchronization from the point information corresponding to all threads, specifically by comparing the sizes of the points corresponding to the point information corresponding to all threads; and determining the smallest point as the starting point of the second task of the incremental data synchronization; Based on the starting point of the second task, the incremental data in the source database is synchronized to the destination database.

2. The method according to claim 1, characterized in that After synchronizing the incremental data in the source database to the destination database based on the starting point of the second task, the method further includes: The task information corresponding to the second task is updated into the preset metadata table.

3. The method according to claim 1, characterized in that When an exception occurs in the first task, before determining the starting point of the second task of the incremental data synchronization based on the task information corresponding to the first task in the preset metadata table, the method further includes: Obtaining a task reload instruction; wherein the task reload instruction at least includes task identification information of incremental data synchronization to be performed on the task reload; The determining the starting point of the second task of the incremental data synchronization based on the task information corresponding to the first task in the preset metadata table includes: Finding the task identification information contained in the task reload instruction from the task information corresponding to the first task in the preset metadata table; The starting point of the second task corresponding to the task identification information and to be synchronized with the incremental data to be task-overloaded is determined by using the task information of the first task corresponding to the found task identification information.

4. The method according to claim 1, characterized in that: When executing the first task of incremental data synchronization, writing the task information corresponding to the first task into the preset metadata table includes: Obtaining a task start instruction for the first task of the incremental data synchronization; According to the task start instruction, a first task of incremental data synchronization is executed to synchronize the incremental data from the source database to the destination database, and task information corresponding to the first task is written into the preset metadata table.

5. An incremental data synchronization device, characterized in that: The device comprises: A writing module, used for writing task information corresponding to the first task of incremental data synchronization into a preset metadata table when executing the first task of the incremental data synchronization, wherein the task information includes the task identification information of the incremental data synchronization, the thread identification information of all threads corresponding to the task of the incremental data synchronization, and the point information corresponding to each of the thread identification information; A determination module, used for determining a starting point of a second task of the incremental data synchronization based on the task information corresponding to the first task in the preset metadata table when an exception occurs in the first task, the second task being an overload task of the first task; A synchronization module, used to synchronize the incremental data in the source database to the destination database based on the starting point of the second task; wherein, The determination module is specifically used to determine the thread identification information of all threads corresponding to the task identification information of the incremental data synchronization; based on the thread identification information, determine the point information corresponding to each thread; and determine the starting point of the second task of the incremental data synchronization from the point information corresponding to all threads; The determination module is also specifically used to compare the sizes of the points corresponding to the point information corresponding to all threads; and determine the smallest point as the starting point of the second task of the incremental data synchronization.

6. The device according to claim 5, characterized in that The writing module is further used to: after synchronizing the incremental data in the source database to the destination database based on the starting point of the second task, update the task information corresponding to the second task into the preset metadata table.

7. An electronic device, characterized in that: The device comprises the incremental data synchronization device as claimed in claim 5 or 6, or the electronic device comprises: A memory for storing executable commands; A processor is used to execute the incremental data synchronization method as described in any one of claims 1-4 under the control of the executable command.

8. A computer-readable storage medium, characterized in that: Executable instructions are stored, and when the executable instructions are executed by a processor, the incremental data synchronization method according to any one of claims 1 to 4 is executed.

Images