A method and device for data synchronization and full data breakpoint resumption

Abstract

The present invention belongs to the field of database data synchronization and relates to a method and device for data synchronization and full data breakpoint resumption. The method comprises grouping a table to be loaded to obtain group information, saving the group information to a source end and a target end respectively, establishing a connection between the source end and the target end, loading group data according to the group information, updating the group information according to the result of the group data loading, updating the initial loading state of the table to be loaded according to the updated group information, and when a failure occurs during the group data loading process, determining whether to reload according to the updated loading state of the table to be loaded, and if so, determining the group to be reloaded according to the loading state of each group in the table to be reloaded. The grouping is used to reduce the amount of data loaded at a single time. After a failure, only the group with loading error needs to be reloaded, and the amount of group data is greatly reduced, thereby reducing the cost of fault recovery.

Description

Technical Field

[0001] The present invention belongs to the field of database data synchronization, and more specifically, relates to a method and device for data synchronization and full data breakpoint resumption. Background Art

[0002] Currently, heterogeneous database replication technology based on database log analysis is widely used. This technology captures incremental database data at the source end and then sends it to the destination end. At the destination end, the incremental data is applied to the target database through a universal database access interface to achieve data replication. Because this technology uses a universal database interface, it supports the replication of heterogeneous database systems and heterogeneous operating system environments. The destination standby database system is readable and writable, making it a "dual-active" system.

[0003] When building a data synchronization system, the destination database is often empty. The full data in the source database must be synchronized to the destination database before incremental data synchronization can begin. This full data synchronization process is called initialization loading. If a table in the source database has a large amount of data, full data synchronization of that table will take a considerable amount of time. If an unexpected situation occurs during synchronization that causes a downtime in the synchronization service, or if synchronization errors occur for other reasons, the target database table with synchronization errors must be cleared and the full data for that table must be resynchronized. This places very high demands on the stability and reliability of the entire synchronization system. If the current system itself is a production system, initialization loading errors are often caused by uncontrollable external environmental factors.

[0004] Therefore, how to reduce the demand for synchronization environment stability during initial loading, handle errors that occur during initial loading at a lower cost, and synchronize the full data in the source database to the destination database has become an important technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: how to reduce the requirements of initialization loading on the stability of the synchronization environment, handle errors that occur during initialization loading at a lower cost, and synchronize the full amount of data in the source database to the destination database.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] In a first aspect, a method for resuming data synchronization and full data transmission is provided, comprising:

[0008] When initializing the load, set the initial load state for the table to be loaded;

[0009] Grouping the tables to be loaded to obtain grouping information, and saving the grouping information to the source end and the target end respectively;

[0010] Establishing a connection between the source end and the target end, and loading packet data according to the packet information;

[0011] updating the grouping information according to the result of the grouping data loading, and updating the initial loading state of the to-be-loaded table according to the updated grouping information;

[0012] When a failure occurs during the packet data loading process, whether the loading needs to be continued is determined based on the loading status of the updated table to be loaded. If so, the group to be continued is determined based on the loading status of each group in the table to be continued.

[0013] Preferably, setting an initial loading state for the table to be loaded during initial loading includes:

[0014] STATE1 is set to indicate the loading status of the table to be loaded. When the STATE1 value of the table to be loaded is 1, it indicates that the loading is successful. When the STATE1 value of the table to be loaded is 0, it indicates that the loading fails. When initializing the loading, the STATE1 value of the table to be loaded is saved as 0.

[0015] Preferably, grouping the to-be-loaded tables to obtain group information, and saving the group information to the source end and the target end respectively, comprises:

[0016] The group information includes STATE2, which indicates the loading state of each group. When STATE2 is 1, it indicates that the loading is successful. When STATE2 is 0, it indicates that the loading fails. Initially, the value of STATE2 of each group is saved as 0.

[0017] A source-side auxiliary table and a target-side auxiliary table are created, and the grouping information is saved in the source-side auxiliary table and the target-side auxiliary table respectively.

[0018] Preferably, establishing a connection between the source end and the target end and loading packet data according to the packet information includes:

[0019] The source end creates a loading thread for each packet according to the source end auxiliary table, and the loading threads respectively establish connections with the execution threads of the target end;

[0020] The source end delivers the entire packet as a single transaction to the execution thread of the target end according to the source end auxiliary table.

[0021] Preferably, the source end delivers the entire packet as a single transaction to the execution thread of the target end according to the source end auxiliary table, including:

[0022] The target end executes storage of the single transaction through the execution thread;

[0023] When the loading program is abnormal, the target end will automatically roll back the uncommitted single transaction;

[0024] Before submitting the successful storage operation of the single transaction, the target end will update the value of STATE2 of the grouping information in the target end auxiliary table to 1, and then submit it.

[0025] Preferably, updating the grouping information according to the result of loading the grouped data, and updating the initial loading state of the table to be loaded according to the updated grouping information, includes:

[0026] When the source end completes delivering the entire packet to the execution thread of the target end, the target end determines whether the packet has been stored by checking the value of STATE2 of the packet in the target end auxiliary table;

[0027] The source end receives the result returned by the target end. If the storage is completed, STATE2 in the source end auxiliary table is updated to 1. If the storage is not completed, the group with STATE2 1 is skipped and the group that is not completed is checked again.

[0028] When all the groups of a table are successfully stored, STATE1 in the initial loading state set for the table to be loaded during initial loading is updated to 1, indicating that the group data loading of the table is completed.

[0029] Preferably, when a failure occurs during the loading of the packet data, determining whether reloading is required based on the loading status of the updated table to be loaded, and if so, determining the group to be reloaded based on the loading status of each group in the table to be reloaded, including:

[0030] Based on the updated initial loading status of each table, reload the tables whose STATE1 value is 0, i.e., tables that have not been loaded into the warehouse;

[0031] In the table where the STATE1 value is 0, i.e., the storage is not completed, the group whose STATE2 value is 0 in its group information is continued.

[0032] Preferably, in the table where the STATE1 value is 0, i.e., the storage is not completed, the group with the STATE2 value of 0 in the group information is continued to be loaded, including:

[0033] Determine whether the grouping information of the table that has not been completed into the warehouse is consistent on the source end and the target end. If consistent, perform a classification operation based on the value of STATE2 in the grouping information of the table;

[0034] If they are inconsistent, it is necessary to delete the grouping information of the table at the source end and the target end, and regroup the table.

[0035] Preferably, the step of determining whether the grouping information of the table not yet completed in storage is consistent between the source end and the target end, and if consistent, performing a classification operation based on the value of STATE2 in the grouping information of the table, wherein the classification operation specifically includes:

[0036] When the value of STATE2 is 1, skip directly and take the next group information;

[0037] When the value of STATE2 is 0, it indicates that the packet loading has failed and the packet needs to be rechecked;

[0038] The source end sends a recheck message to the target end, and the target end checks the storage status of the group again: if it has been successfully stored, it returns the completed storage to the source end, and the source end updates the value of STATE2 of the group to 1 and skips, taking the next group information, otherwise it returns the unfinished storage to the source end, indicating that the group is confirmed to have failed to load and needs to be reinstalled.

[0039] On the second aspect, a device for data synchronization and full data breakpoint resumption is provided, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, the method for completing data synchronization and full data breakpoint resumption is completed.

[0040] The beneficial effects of the present invention are as follows: the present invention splits the data of a table into N groups, each group is sent after an independent connection query, which is called group loading. Grouping is to divide the data of a table according to the number of rows or the physical storage unit of the data, so as to reduce the amount of data loaded at a single time. After a failure, only the groups with loading errors need to be reloaded, and the groups that are loaded successfully will not be affected. Compared with the previous reloading method based on tables, the amount of data in the groups is greatly reduced, thereby reducing the cost of failure recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The drawings are used to provide a further understanding of the present disclosure and constitute part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure, but do not constitute a limitation of the present disclosure. In the drawings:

[0042] Figure 1This is a flow chart of a method for resuming full data synchronization breakpoint transmission provided by an embodiment of the present invention;

[0043] Figure 2 1 is a schematic diagram of grouping information of a method for resuming full data synchronization breakpoint transmission provided by an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of establishing a connection between a source end and a target end in a method for resuming full data synchronization provided by an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of group loading of a method for resuming breakpoint transmission of full data synchronization provided by an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of a check group loading result of a method for data synchronization full data breakpoint resume transmission provided by an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of a group re-installation process of a method for resuming breakpoint transmission of full data synchronization provided by an embodiment of the present invention;

[0048] Figure 7 This is a schematic diagram of the architecture of a device for data synchronization and full data breakpoint resumable transmission provided by an embodiment of the present invention;

[0049] Figure 8 This is a first example schematic diagram of a method and apparatus for data synchronization and full data breakpoint resume transmission provided by an embodiment of the present invention;

[0050] Figure 9 This is a second example schematic diagram of a method and apparatus for data synchronization and full data breakpoint resume transmission provided by an embodiment of the present invention;

[0051] Figure 10 This is a schematic diagram of a third example of a method and apparatus for data synchronization and full data breakpoint resumable transmission provided by an embodiment of the present invention;

[0052] Figure 11 This is a fourth example schematic diagram of a method and apparatus for data synchronization and full data breakpoint resume transmission provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0055] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0056] Example 1

[0057] To achieve real-time data synchronization between two or more databases, full data synchronization is paramount. Data loading is the synchronization method for initializing full data. To speed up the transmission efficiency of loaded data, the data of a table is split into N groups and then sent after query, forming group loading. Grouping is to divide the data of a table by the number of rows or physical storage units of data to reduce the amount of data loaded at a single time.

[0058] In existing solutions, to speed up data loading, when the source sends grouped data, there is a commit threshold. That is, for every certain number of rows of data in a single group, the data is packaged into a single transaction and sent to the target. The target executes the data in parallel according to the received transactions and stores it in the database. For a single group, the multiple transactions are independent. Although multiple connections can be used to execute the data in parallel between transactions to improve the efficiency of full data initialization, the integrity of a single group of data cannot be guaranteed after the synchronization service fails. Some transactions may have been successfully stored before the failure, while others may fail due to the failure. The end result is that only part of the data in the group is successfully stored.

[0059] In the above failure situation, if you want to reload the table by group, you need to delete the corresponding data on the target side for the group that has only partially loaded data to prevent data duplication after reloading. Currently, there is no effective solution to accurately clean up the data in the group loading on the target side.

[0060] It can be seen from this that although the above group sending and execution strategies can improve the loading rate to a certain extent, once a failure occurs, the entire table needs to be cleared and restarted, which will greatly reduce the loading efficiency. It is necessary to invent new group sending and execution strategies to efficiently achieve re-loading after a failure.

[0061] Example 2

[0062] In response to the problems existing in the existing technology, a method for data synchronization and full data resuming after breakpoint is provided in this embodiment. First, synchronization-related services are deployed and started on the databases on both the source and target ends that need to be synchronized. The service includes the environment required for loading. The source end of loading includes a loading thread and a loading management thread. The loading thread is responsible for taking grouped data from the source database and packaging it into a single transaction and sending it to the target end. The loading management thread is responsible for polling whether each group has completed the storage at the target end. The target end includes an execution thread, which is responsible for submitting the received grouped data through a single transaction and inserting it into the target database.

[0063] like Figure 1 As shown, the method for resuming data synchronization and full data transmission specifically includes:

[0064] Step 101: When initializing loading, set the initial loading state for the table to be loaded.

[0065] STATE1 is set to indicate the loading status of the table to be loaded. When the STATE1 value of the table to be loaded is 1, it indicates that the loading is successful. When the STATE1 value of the table to be loaded is 0, it indicates that the loading fails. When initializing the loading, the STATE1 value of the table to be loaded is saved as 0.

[0066] Before each table starts loading, the status information of each table is recorded, including TABLEID and STATE1. TABLEID is a unique identifier for a table, and each table corresponds to its TABLEID. STATE1 indicates the loading status of the table. From the results, there are two states: 1 for successful loading and 0 for failed loading. When initializing the load, the value of STATE1 for each table is saved as 0, that is, the unloaded state.

[0067] Step 102: Grouping the tables to be loaded to obtain grouping information, and saving the grouping information to the source end and the target end respectively.

[0068] The group information includes STATE2, which represents the loading status of each group. When STATE2 is 1, it means that the loading is successful. When STATE2 is 0, it means that the loading fails. Initially, the value of STATE2 of each group is saved as 0. A source-side auxiliary table and a target-side auxiliary table are created, and the group information is saved to the source-side auxiliary table and the target-side auxiliary table respectively.

[0069] Before initializing the loading of each table to be loaded, each table to be loaded needs to be divided according to the number of rows or the physical storage unit of data to form N groups and save the group information. After the table is grouped, the group information will no longer change. After a failure, the data can be resumed based on the saved group information. The information of each group mainly includes: TABLEID, GROUPID, LRID, RRID, STATE2, etc. Figure 2 As shown, GROUPID is the serial number of this group, starting from 1, LRID is the left boundary value of the ROWID range of this grouped data, ROWID is the address of the data row, which is equivalent to the identifier of the data row. Each row of data corresponds to its ROWID, RRID is the right boundary value of the ROWID range of this grouped data, and STATE2 is the loading status of this group. From the results, it can be seen that there are two states: loading success 1 and loading failure 0. Initially, STATE2 of each group is uniformly saved as 0.

[0070] After grouping, save the grouping information on the target and source ends respectively, and update the GROUPS value in the table information, including:

[0071] (1) After the source end obtains the group information, it first sends the group information to the target end for storage. The target end will first create an auxiliary table containing each column of all group information, and then insert all group information records into the table.

[0072] (2) The source end saves the grouping information to the source end auxiliary table.

[0073] Step 103: Establish a connection between the source end and the target end, and load packet data according to the packet information.

[0074] The source end creates a loading thread for each packet according to the source end auxiliary table, and the loading threads respectively establish connections with the execution threads of the target end.

[0075] like Figure 3As shown, specifically, the source creates a loading thread for each group in turn and attempts to establish a connection with the execution thread of the target. After receiving the start command for group loading, the target attempts to create a corresponding loading transaction execution thread. To prevent too many threads from establishing connections to the database at the same time, which would affect other third-party applications of the database, the target sets thresholds for source loading threads and target execution threads. If the source loading thread threshold is set to X, then at most X groups can load data simultaneously through X threads. If the target execution thread threshold is set to N, then at most N groups can execute data storage through N threads simultaneously. When the target reaches the upper limit of the number of loading execution threads, the source will not be able to successfully establish a connection even if it attempts to establish a connection. It must wait until other loading thread transactions on the target are completed and resources are released before it can successfully obtain relevant resources and establish a connection.

[0076] In a preferred embodiment, the source end delivers the entire packet as a single transaction to the execution thread of the target end according to the source end auxiliary table.

[0077] Among them, in a preferred embodiment, the target end executes the storage of the single transaction through the execution thread. When the loading program is abnormal, the target end will automatically roll back the uncommitted single transaction. Before committing the successful storage operation of the single transaction, the target end will update the value of STATE2 of the grouping information in the target end auxiliary table to 1, and then commit it.

[0078] like Figure 4 As shown, the above process is specifically as follows: the source-side loading thread takes a group to start loading, first establishes a connection with the source-side database, and obtains data according to the group's ROWID range information [LRID, RRID]. The data of the entire group is delivered to the target-side execution thread as a single transaction. After receiving the data, the target side establishes a connection with the target-side database through the execution thread, and then the group data is stored in the database through the execution thread single transaction. In this way, after the loading program of the group is abnormal, the target-side database will automatically roll back the uncommitted group transaction. When the subsequent installation is completed after the fault is recovered, there is no need to clean up the data of the group that was not successfully loaded last time in the target-side database. Before the operation on the entire group is committed, the STATE2 value of the group information in the target-side auxiliary table will be updated to 1, and then it will be committed. This ensures the integrity of the entire group transaction and realizes the atomicity of the synchronous update of the group loading status. The data insertion of the entire group will succeed or fail together with the update of the group's loading completion flag STATE2.

[0079] Step 104: updating the grouping information according to the result of the grouping data loading, and updating the initial loading state of the to-be-loaded table according to the updated grouping information.

[0080] Among them, in a preferred embodiment, after the source end completes delivering the entire group to the execution thread of the target end, the target end determines whether the group has completed warehousing by checking the value of STATE2 of the group in the auxiliary table of the target end. The source end receives the result returned by the target end. If the warehousing is completed, STATE2 in the auxiliary table of the source end is updated to 1. If the warehousing is not completed, the group with STATE2 of 1 is skipped, and the group that has not completed warehousing is checked again. When all the groups of a table are successfully warehousing, STATE1 in the initial loading state set for the table to be loaded during initial loading is updated to 1, indicating that the table has completed the loading of group data.

[0081] like Figure 5 As shown, specifically, after the source end group loading thread completes sending data, the loading management thread will poll and send a group loading completion check message to inquire the target end whether the group has completed warehousing. The target end judges by checking the value of STATE2 saved in the target end auxiliary table. STATE2 is 1, indicating that the warehousing has been completed and a success is returned. STATE2 is 0, indicating that it has not been completed and a failure is returned. The source end updates STATE2 of the group information saved in the source end auxiliary table to 1 based on the result returned by the target end. If it is successful, it will wait and check again later. When checking again, it will skip the group with STATE2 1, which means it has succeeded, and will only check the group that has not yet succeeded. When all the groups of a table are loaded successfully, STATE1 in the status information of the table saved at the beginning will be updated to 1, indicating that the group loading of the table has been completed and its full data has been copied successfully.

[0082] According to the group loading strategy at both ends described in steps 101 to 104, when a failure occurs during the normal loading process, causing some tables to fail to load, it is necessary to re-install the tables that failed to load in groups. The re-installation does not require further grouping, and the saved historical grouping information is directly read as the basis for grouping. Since each group is committed by a single transaction, the data of the entire group will only be successfully or unsuccessfully stored together. Therefore, the verification of the group information and the checking of the group status are key. Even if the time of the failure is different, the operation of step 105 below must be performed when re-installing.

[0083] Step 105: When a failure occurs during the packet data loading process, determine whether reloading is required based on the loading status of the updated table to be loaded. If so, determine the group to be reloaded based on the loading status of each group in the table to be reloaded.

[0084] Among them, in a preferred embodiment, according to the initial loading status after each table is updated, the table whose STATE1 value is 0, that is, the table that has not been completed into the warehouse is reloaded, and among the tables whose STATE1 value is 0, that is, the table that has not been completed into the warehouse, the group whose STATE2 value is 0 in its grouping information is reloaded, and it is judged whether the grouping information of the table that has not been completed into the warehouse at the source end and the target end is consistent. If consistent, the classification operation is performed according to the STATE2 value in the grouping information of the table. If inconsistent, it is necessary to delete the grouping information of the table at the source end and the target end, and regroup the table.

[0085] The reinstallation process is as follows Figure 6 As shown, specifically including:

[0086] 1. Query the loading status of each table to be loaded, filter out the tables that failed to load from the table that records the loading status of each table, and select a table to start reloading.

[0087] 2. Determine whether the grouping information of the table on the source and target ends is consistent by checking the number of GROUPS and grouping information items saved on the source and target ends, and take different steps depending on the results.

[0088] (21) When the grouping information items on both the source and target sides are complete and the number of groups is equal to the number of GROUPS, it means that the table meets the prerequisites for reinstallation and can continue to execute reinstallation process 3.

[0089] (22) When the condition (21) is not met, it means that the table cannot be reloaded. The only way is to first delete the grouping information about the table at both the source and target ends, and then regroup and load it starting from step 102 of the normal loading above.

[0090] 3. After confirming that the reload conditions are met (i.e., the above 21 is met), the source side takes different actions based on the value of STATE2 in the source side auxiliary table. Specifically:

[0091] (31) STATE2 is 1, which means the group is loaded successfully. Skip directly and take the next group information.

[0092] (32) STATE2 is 0, which means that the group loading has failed and needs to be rechecked. The source sends a message to check that the group loading has ended. The target checks the storage status of the group again. If it has been successfully stored, it returns a success message to the source. The source updates STATE2 in the group information to 1 and skips to the next group information. Otherwise, it returns a failure message to the source, indicating that the group is confirmed to have failed loading and needs to be reloaded.

[0093] 4. Confirm that the failed packets are reloaded. The source-side loading thread takes the failed packets and restarts loading. The loading process is consistent with step 103 during normal loading. The saved ROWID range is used as the basis for querying packet data.

[0094] 5. Check the loading results of the reloaded groups. After the reloaded group data is sent, the loading management thread will also check the completion status of the group at the target end, consistent with step 104 during normal loading. When all the reloaded groups of the table have been checked and loaded, it means that the table has completed the reload. When all the tables that failed to load have completed the reload, it means that the reload of this failure has been completed.

[0095] The basic steps of this embodiment can be explained as follows:

[0096] First, a status is set for each table during initial loading, and the table is divided into N groups as needed. Then, the group information is saved in auxiliary tables on the source and target ends respectively. When a failure occurs, the status of the table can be used to determine whether to re-install. If necessary, the status of each group in the table can be used to determine which groups need to be re-installed.

[0097] Secondly, when loading in groups, each group on the source side has an independent connection on both the source and target sides, and the data on the target side is stored in a single transaction. When the data is submitted, the status of the group is registered in the auxiliary table and then submitted as a whole. When a failure occurs during group loading, the target database will automatically roll back the data of the entire group that has been stored. In this way, when reloading after the failure is recovered, the source side does not need to clean up the previously loaded data of the group in the target database and can directly reload it.

[0098] This solution has two major technical advantages over existing solutions:

[0099] First, divide the table into smaller groups for initial loading. When reloading after a failure, use the group as the unit for reloading. Compared with the technical solution that uses the table as the unit, it can effectively reduce the cost of reloading.

[0100] Secondly, during group loading, the source and target ends use a one-to-one connection, and the grouped data is stored in the database as a single transaction. By utilizing the database's automatic rollback mechanism, the data stored before the failure can be automatically cleaned up when a loading failure occurs. After the failure is recovered, no additional cleanup work is required compared to existing solutions.

[0101] Example 3

[0102] In this embodiment, a device for data synchronization and full data breakpoint resumption is proposed, including at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the method for data synchronization and full data breakpoint resumption as described in Example 2.

[0103] like Figure 7 FIG. 1 is a schematic diagram of the architecture of a device for data synchronization and full data breakpoint resuming according to an embodiment of the present invention. The device for data synchronization and full data breakpoint resuming according to this embodiment includes one or more processors 21 and a memory 22. Figure 7 A processor 21 is taken as an example.

[0104] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0105] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for data synchronization and full data breakpoint resumption described in Example 2. The processor 21 executes the method for data synchronization and full data breakpoint resumption by running the non-volatile software programs and instructions stored in the memory 22.

[0106] The memory 22 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 may optionally include a memory remotely located relative to the processor 21. These remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0107] The program instructions / modules are stored in the memory 22 and, when executed by the one or more processors 21 , perform the method for data synchronization and full data breakpoint resumption in the above-mentioned embodiment 2.

[0108] Example 4

[0109] The method for data synchronization and full data breakpoint resuming proposed in Example 2 and the device for data synchronization and full data breakpoint resuming proposed in Example 3 will be given as examples in this embodiment to further illustrate the present invention.

[0110] like Figure 8As shown in the figure, there is a table T with TABLEID 66880, which contains 300 rows of data. The data is divided into three groups of 100 rows each and loaded from the source database to the target database.

[0111] like Figure 9 The figure shows the process of grouping and loading data into table T. Figure 10 The following shows the packet loading check process: Figure 11 The following figure shows the subsequent installation process after the failure.

[0112] The specific group loading and reloading process is shown in Example 2 and will not be described in detail in this example.

[0113] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for data synchronization and full data resuming, characterized in that: include: When initializing the load, set the initial load state for the table to be loaded; Grouping the tables to be loaded to obtain group information, and saving the group information to the source end and the target end respectively, including: The group information includes STATE2, which indicates the loading state of each group. When STATE2 is 1, it indicates that the loading is successful. When STATE2 is 0, it indicates that the loading fails. Initially, the value of STATE2 of each group is saved as 0. Creating a source-side auxiliary table and a target-side auxiliary table, and saving the grouping information to the source-side auxiliary table and the target-side auxiliary table respectively; Establishing a connection between the source end and the target end, and loading packet data according to the packet information, including: The source end creates a loading thread for each packet according to the source end auxiliary table, and the loading threads respectively establish connections with the execution threads of the target end; The source end delivers the entire packet as a single transaction to the execution thread of the target end according to the source end auxiliary table; updating the grouping information according to the result of the grouping data loading, and updating the initial loading state of the to-be-loaded table according to the updated grouping information; When a failure occurs during the packet data loading process, whether the loading needs to be continued is determined based on the loading status of the updated table to be loaded. If so, the group to be continued is determined based on the loading status of each group in the table to be continued.

2. The method for data synchronization and full data resuming according to claim 1, characterized in that: The step of setting an initial loading state for the table to be loaded during initial loading includes: STATE1 is set to indicate the loading status of the table to be loaded. When the STATE1 value of the table to be loaded is 1, it indicates that the loading is successful. When the STATE1 value of the table to be loaded is 0, it indicates that the loading fails. When initializing the loading, the STATE1 value of the table to be loaded is saved as 0.

3. The method for data synchronization and full data resuming according to claim 1, characterized in that: The source end delivers the entire packet as a single transaction to the execution thread of the target end according to the source end auxiliary table, including: The target end executes storage of the single transaction through the execution thread; When the loading program is abnormal, the target end will automatically roll back the uncommitted single transaction; Before submitting the successful storage operation of the single transaction, the target end will update the value of STATE2 of the grouping information in the target end auxiliary table to 1, and then submit it.

4. The method for data synchronization and full data resuming according to claim 3, characterized in that: The updating of the grouping information according to the result of the grouping data loading, and updating the initial loading state of the to-be-loaded table according to the updated grouping information, comprises: When the source end completes delivering the entire packet to the execution thread of the target end, the target end determines whether the packet has been stored by checking the value of STATE2 of the packet in the target end auxiliary table; The source end receives the result returned by the target end. If the storage is completed, STATE2 in the source end auxiliary table is updated to 1. If the storage is not completed, the group with STATE2 1 is skipped and the group that is not completed is checked again. When all the groups of a table are successfully stored, STATE1 in the initial loading state set for the table to be loaded during initial loading is updated to 1, indicating that the group data loading of the table is completed.

5. The method for data synchronization and full data resuming according to claim 4, characterized in that: When a failure occurs during the packet data loading process, determining whether reloading is required based on the loading status of the updated table to be loaded, and if so, determining the group to be reloaded based on the loading status of each group in the table to be reloaded, including: Based on the updated initial loading status of each table, reload the tables whose STATE1 value is 0, i.e., tables that have not been loaded into the warehouse; In the table where the STATE1 value is 0, i.e., the storage is not completed, the group whose STATE2 value is 0 in its group information is continued.

6. The method for data synchronization and full data resuming according to claim 5, characterized in that: The method of reloading the group whose STATE2 value is 0 in the group information in the table where the STATE1 value is 0, i.e., the storage is not completed, includes: Determine whether the grouping information of the table that has not been completed into the warehouse is consistent on the source end and the target end. If consistent, perform a classification operation based on the value of STATE2 in the grouping information of the table; If they are inconsistent, it is necessary to delete the grouping information of the table at the source end and the target end, and regroup the table.

7. The method for data synchronization and full data resuming according to claim 6, characterized in that: The step of determining whether the grouping information of the table not yet completed in storage is consistent at the source end and the target end, and if consistent, performing a classification operation based on the value of STATE2 in the grouping information of the table, wherein the classification operation specifically includes: When the value of STATE2 is 1, skip directly and take the next group information; When the value of STATE2 is 0, it indicates that the packet loading has failed and the packet needs to be rechecked; The source end sends a recheck message to the target end, and the target end checks the storage status of the group again: if it has been successfully stored, it returns the completed storage to the source end, and the source end updates the value of STATE2 of the group to 1 and skips, taking the next group information, otherwise it returns the unfinished storage to the source end, indicating that the group is confirmed to have failed to load and needs to be reinstalled.

8. A device for data synchronization and full data breakpoint resuming, characterized in that: It includes at least one processor and a memory, and the at least one processor and the memory are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the method for data synchronization and full data breakpoint resumption as described in any one of claims 1 to 7.

Images