Replicating data changes using multiple storage devices and tracking records of pending data changes stored on the storage devices

Abstract

Methods, computer program products, and / or systems that perform the following are provided: generating log files while performing data changes to a source database system table; storing the log files on a first memory device having a first access time; archiving a portion of the log files from the first memory device on a second memory device having a second access time, the second access time being higher than the first access time; tracking a second number of pending data changes, the pending data changes being data changes that are logged in the data changes that are archived on the second memory device that have not yet been replicated into a target database system table; selecting a type of data replication from a first type of data replication and a second type of data replication based on the second number; and replicating the data changes into the target database system table.

Description

BACKGROUND

[0001] The present invention relates to the field of database technology and more specifically to replicating data changes of a table of a source database system into a table of a target database system.

[0002] The replication of data changes made in a table of a source database system in a table of a target database system can be performed to create a backup of the table of the source database. Another application of such replication of data changes can be to synchronize a table of a target database system with a table of a source database system, for example, in case the table of the target database system comprises a different ordering scheme than the table of the source database system. Generally, the replication of data changes should be performed as fast as possible to maintain the data of the table of the target database system and the data of the table of the source database system as synchronously as possible. The replication of data changes can be performed according to different types of replication strategies. A first type can comprise replicating individual rows of a table of a source database system individually. Another type of replication strategy can comprise replicating several rows, even a majority of rows, of a table of a source database at once. SUMMARY

[0003] Various embodiments provide a method, a computer system and a computer program product for replicating data changes as recited in the independent claims. Advantageous embodiments are described in the dependent claims. Embodiments of the invention can be freely combined with each other if they are not mutually exclusive.

[0004] According to aspects of the invention, a method for replicating data changes of a table of a source database system into a table of a target database system is provided. The method comprises repeatedly generating log files while performing data changes of a table of a source database system, each log file comprising at least one record of a data change of the table of the source database system. The method further comprises storing the log files on a first memory device, the first memory device having a first access time. The method further comprises archiving a portion of the log files stored on the first memory device on a second memory device, the second memory device having a second access time, wherein the second access time is higher than the first access time. The method further comprises tracking a second number of second pending data changes, the second pending data changes being data changes in records of data changes of the log files archived on the second memory device that have not yet been replicated into the table of the target database system. The method further comprises selecting a type of data replication from a first type of data replication and a second type of data replication based on the second number. The method further comprises replicating data changes of the table of the source database system into the table of the target database system according to the selected type of data replication using the records of data changes of the log files.

[0005] In some embodiments, the first type of data replication includes separately loading each of the records of the data changes, and replicating respective data changes of individual rows of the table of the source database system into the table of the target database system according to the corresponding separately loaded records of the data changes. In some embodiments, the second type of data replication includes loading information for updating several rows of the table of the target database system together and updating the several rows of the table of the target database system based on the information.

[0006] In some embodiments, the method can optionally further include tracking a first number of first pending data changes, the first pending data changes being data changes stored in the records of the data changes of the log files on the first memory device that have not yet been replicated into the table of the target database system, wherein the selection of the type of data replication is based on the first number and the second number.

[0007] Optionally, in some embodiments, the selection of the type of data replication can include a weighting of the first number and a weighting of the second number, wherein the weighting of the first number is based on the first access time and the weighting of the second number is based on the second access time.

[0008] In some embodiments, the method can optionally further include selecting the first type of data replication if a ratio between the first number and the second number is greater than or equal to a third threshold, and selecting the second type of data replication if the ratio between the first number and the second number is less than the third threshold. The ratio can be a weighted ratio, wherein the first number is weighted depending on the first access time and the second number is weighted depending on the second access time.

[0009] In some embodiments, the method can optionally further include calculating a weighted sum of the first number and the second number, wherein the first number is weighted based on the first access time and the second number is weighted based on the second access time. In such embodiments, the first type of data replication is selected if the weighted sum is less than a fourth threshold, and the second type of data replication is selected if the weighted sum is greater than or equal to the fourth threshold.

[0010] In some embodiments, the method can optionally further include obtaining respective log numbers, wherein each respective log number is assigned to a respective one of the log files, and each respective log number indicates a total number of records of data changes of the table of the source database system that have been written up to a respective time of a beginning of generation of the respective log file. In such embodiments, the method can further include using the respective log numbers assigned to the log files to track the first number and the second number.

[0011] Optionally, in some embodiments, the first number of tracks includes calculating a difference between a respective log number assigned to a log file stored on the first memory device for a shortest time and a respective log number assigned to a log file stored on the first memory device for a longest time.

[0012] Optionally, in some embodiments, the first number of tracks includes calculating a difference between an actual total number of data changes of a table of the source database system at an actual time and a respective log number assigned to a log file stored on the first memory device for a longest time.

[0013] In some embodiments, the method can optionally further include tracking a record number of a last record of data changes read out from the first memory device or the second memory device and used to replicate data changes to a table of the target database system according to the last record of data changes; and using the record number of the last record of data changes to track the first number or the second number.

[0014] In some embodiments, the method can optionally further include calculating the second number as a difference between a respective log number assigned to a log file stored on the first memory device for a longest time and a record number of the last record of data changes, wherein the last record of data changes is included in one of the log files archived on the second memory device.

[0015] In some embodiments, the method can optionally further include calculating the first number as a difference between a respective log number assigned to a log file stored on the first memory device for a shortest time and a record number of the last record of data changes, wherein the last record of data changes is included in one of the log files stored on the first memory device.

[0016] In some embodiments, the method can optionally further comprise counting, using a counter, a total number of records of data changes of the table of the source database system. In such embodiments, the method can further comprise determining whether the counter is in a state of nearing an overflow of the counter. In such embodiments, the method can further comprise selecting the second type of data replication if the counter is in the state of nearing the overflow of the counter. In such embodiments, the method can further comprise resetting the counter after replicating the data changes of the table of the source database system into the table of the target database system according to the second type of data replication using the records of data changes of the log file. In some embodiments, the second type of data replication comprises loading all records of data changes stored on the first memory device and the second memory device together, and replicating more than one row of data changes of the table of the source database system into the table of the target database system according to all loaded records of data changes.

[0017] According to another aspect of the present application, there is provided a computer program product comprising a computer readable storage medium having computer readable program instructions embodied therewith, the computer readable program instructions capable of being executed by one or more processors to perform the method according to any of the preceding embodiments.

[0018] According to another aspect of the present application, there is provided a computer system for replicating data changes of a table of a source database system into a table of a target database system. The computer system can be configured to perform the method according to any of the preceding embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0019] In the following, embodiments of the present application are explained in more detail, by way of example only, with reference to the accompanying drawings, wherein:

[0020] Figure 1 a computer system according to embodiments of the present application is shown;

[0021] Figure 2 a second computer system / server of the computer system shown in Figure 1

[0022] Figure 3 depicting a network of computer systems connected to Figure 1

[0023] Figure 4 depicting a source database system, a target database system, a first memory device and a second memory device for performing data replication of a source database system in a target database system according to embodiments of the present application;​​

[0024] Figure 5 depicts a table of a source database system according to an embodiment of the invention;

[0025] Figure 6 depicts a table of a target database system according to an embodiment of the invention;

[0026] Figure 7 depicts a log file generated when performing a data change to a table of a source database system according to an embodiment of the invention;

[0027] Figure 8 depicts a record of a data change generated when performing a data change to a table of a source database system according to an embodiment of the invention;

[0028] Figure 9 depicts a first record of a data change and a second record of a data change according to an embodiment of the invention;

[0029] Figure 10 depicts a source database system, a target database system, a first memory device and a second memory device storing a log file as shown in Figure 4 depicts a source database system, a target database system, a first memory device and a second memory device storing a log file as shown in Figure 7 depicts a source database system, a target database system, a first memory device and a second memory device storing a log file as shown in

[0030] Figure 11 depicts a source database system, a target database system, a first memory device and a second memory device storing a log file as shown in Figure 4 depicts a source database system, a target database system, a first memory device and a second memory device storing a log file as shown in Figure 7 depicts a source database system, a target database system, a first memory device and a second memory device storing a log file as shown in

[0031] Figure 12 depicts a computer-implemented method for replicating a data change to a table of a source database system as shown in Figure 5 into a table of a target database system as shown in Figure 6 DETAILED DESCRIPTION

[0032] The description of the various embodiments of the invention will be presented for the purpose of illustration and description, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The terminology used is chosen to best explain the principles of the embodiments, the practical application, or technical improvement over technology found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0033] ​Hereinafter, a table of a source database system is referred to as a source table, and a table of a target database system is referred to as a target table. A record of each log file generated at the time of execution of a data change of a source table can include all necessary information to replicate a single data change of the source table, respectively. A single data change of a source table can include a change of a value of a single data field of the source table or a change of values of several data fields of a single row of the source table. A single data field can be specified by a number of a row and a number of a column of the source table. A value of a single data field can be a number and / or a string. Necessary information to replicate a single data change can include, for example, a new data value of a single data field of a source table after execution of a data change and a number of a row and a number of a column of the source table that specify the single data field. In a case where a single data change can include a change of values of several data fields of a single row of a source table, necessary information to replicate a single data change can include a respective new data value of each data field modified at the time of execution of a single data change, a number of a row including these modified data fields, and a respective column indicating each modified data field.

[0034] In a case where each new data change of a source table is executed, a new record of a data change can be generated, which includes necessary information to replicate the data change into a target table. The new record of a data change can be written into an actual log file. The new record can be written into the actual log file at a position after a last written record of a data change in the actual log file, which indicates a last executed data change of a source table. Thus, an order of data changes of a source table can be represented by an order of records of data changes in a log file.

[0035] If a size of an actual log file reaches a given threshold, a new log file can be generated. In this way, generation of log files can be repeated. Since a source table can be maintained for a long period of time, for example, for years or decades, a total number of data changes of a source table can be greater than, for example, one thousand, one million, one billion, etc. The total number of data changes can be higher than a number of data fields or rows of a source table, since data fields can change several times during a lifetime of a source table.

[0036] The term "tracking a second number" as used herein refers to any type of determination of the second number, for example, counting, calculating, and / or updating a second counter representing the second number. The tracking of the second number can involve updating a second counter whose value can represent the second number. The second counter can be updated if one of the second pending data changes has been recently replicated into the target table. In this case, the second counter can be decreased by one. Further, the tracking of the second number can involve updating the second counter if a new log file can be stored on the second memory device. In this case, the second counter can be increased by a number of records included in the new log file stored on the second memory device.

[0037] The term "archiving a portion of the log files on the second memory device" as used herein refers to storing a portion of the log files on the second memory device. The term "archiving" is used to indicate that the second access time is higher than the first access time. The first access time and the second access time can be defined by a first time and a second time taken to execute a write or read command after the write or read command has been received by the first memory device or the second memory device, respectively. The second memory device can be a hard disk drive, a magnetic tape, etc. The second memory device can comprise moving elements, such as rotating disks or a magnetic tape, for accessing data stored on the second memory device. The second access time can comprise, for example, a time interval for positioning a read-write head of the second memory according to a read command, also referred to as seek time. The first memory device can be a solid state drive (SSD), a flash memory, etc. The first memory device can not comprise moving elements for reading out data stored on the first memory device.

[0038] Generally, in order to achieve high performance for replicating data changes, it is advantageous to store all log files on the first memory device (e.g. lower access time, etc.). However, the first memory device can be more expensive than the second memory device. For this reason, if the available space on the first memory is close to zero, the size of the first memory can be limited and a portion of the log files stored on the first memory device can be archived on the second memory device.

[0039] When the second access time (e.g. the second memory device, etc.) is higher than the first access time, if the second number of second pending data changes exceeds a certain first level and the frequency of data changes performed in the source table exceeds a certain second level, the replication of data changes can slow down over a longer period of time. In such a case, performing a first type of data replication (e.g. incremental data replication, etc.) can lead to a deadlock situation, in which more log files are archived on the second memory device than can be replicated within a given time interval. In such a case, performing a second type of data replication (e.g. batch data replication, etc.) can resolve or avoid the deadlock situation. According to aspects of the present invention, by tracking the second number and selecting the type of data replication based at least in part on the second number, a deadlock situation or a situation close to a deadlock situation can be detected and prevented. In some embodiments, the tracking of the second number can be performed automatically by checking the actual value of the second number and / or by checking the temporal change of the actual value of the second number.

[0040] The first level and the second level can depend on the design of the computer system used to perform the replication of data changes, in particular on the second access time. The frequency of data changes performed in the source table can depend on the usage behavior of the source database system.

[0041] In some embodiments, the second type of data replication can be selected if the second number is greater than a second threshold. The second threshold can depend on the second access time and / or the bandwidth of the replication of data changes. Such embodiments can provide a fast way to perform the selection of the type of data replication, as only one value needs to be compared. Moreover, the selection of the type of data replication based on only one value can alleviate the need to calibrate the computer system used to perform the data replication, as only one threshold (here the second threshold) needs to be adapted to the usage of the source database system.

[0042] In some embodiments, the determining of the selection of the type of data replication can comprise tracking the first number of first pending data changes, the first pending data changes being data changes of records of data changes of log files stored on the first memory device in the table of the target database system which have not yet been replicated. Thus, in some embodiments, the selection can depend on the first number and the second number. In this way, the first pending data changes can also be taken into account and the deadlock situation can be anticipated more accurately.

[0043] The term "tracking the first number" as used herein refers to any type of determination of the first number, e.g. a counting, a calculation or an updating of a first counter representing the first number. The tracking of the first number can involve updating a first counter whose value can represent the first number. The first counter can be updated if one of the first pending data changes has recently been replicated into the target table. In this case, the first counter can be decreased by one. Moreover, the tracking of the first number can involve updating the first counter if a new log file is stored on the first memory device. In this case, the first counter can be increased by the number of records comprised in the new log file.

[0044] In some embodiments, the loading of the information for updating the number of rows of the table of the target database system can comprise loading the number of records of the records of data changes together. In some embodiments, the updating of the number of rows of the table of the target database system can comprise replicating the data changes of the number of rows of the table of the source database system into the table of the target database system according to the loaded records of data changes. Each of the number of rows of the table of the source database system can correspond to one of the number of rows of the table of the target database system. The number of records of the records of data changes can be loaded from the first memory device or the second memory device. By loading the number of records of the records of data changes together, the data replication can be accelerated. Thus, the deadlock situation can be prevented.

[0045] In some embodiments, loading information for updating the number of rows of the target database system's table can include loading data values of the number of rows of the source database system's table, which can also be referred to herein as a second case of the second type of data replication. Thus, in some embodiments, the record of data changes can not be needed to perform the second type of data replication. The number of rows of the target table and the number of rows of the source table can be partitions of the target table and the source table, respectively. As an example, the number of rows of the target table and the number of rows of the source table can constitute the entire target table and the entire source table, respectively. The data values of the number of rows of the source table can each provide actual data values of a single data field of the number of rows of the source table. Once the data values of the number of rows of the source table are loaded, these values can be replicated to corresponding data fields of the target table to update the target table. The data values can be loaded directly from the source database system. By loading the data values of the number of rows of the source table, replication of a number of data changes of one of the single data fields of the source table to update the corresponding data field in the target table can be skipped. In this way, data replication can be accelerated and deadlock situations can be prevented.

[0046] In some embodiments, the selection of the type of data replication can include comparing a first computational cost needed to perform all pending data changes (e.g., the first pending data change and the second pending data change) using the first type of data replication to a second computational cost needed to perform the update of all rows of the target table according to the second case of the second type of data replication. If the first computational cost is greater than the second computational cost, the second type of data replication can be selected, otherwise the first type of data replication can be selected.

[0047] In some embodiments, determining the selection of the type of data replication can include selecting the second type of data replication if the first number is greater than a first threshold and the second number is greater than a second threshold. Such embodiments can present a simple implementation to consider the first pending data change and the second pending data change. If the first number is greater than the first threshold and the second number is greater than the second threshold, it can be assumed that a deadlock situation can be reached soon. Similar to the second threshold, the first threshold can be adapted to the usage of the source database system and the achievable bandwidth of replicating data changes.

[0048] In some embodiments, the selection of the type of data replication can comprise a weighting of the first number and the second number. As an example, the weighting of the first number can depend on the first access time and the weighting of the second number can depend on the second access time. By performing the weighting of the first number and the second number with respect to the respective access times, a difference between the first access time and the second access time can be taken into account. This can mitigate a need to adapt the method of replicating data changes to a design of the computer system. For example, the higher the second access time, the more the second number can be weighted in dependence on the second access time to yield a key value based on which the selection of the type of data replication can be performed. The weighting can be used to determine the first computation cost mentioned above.

[0049] In some embodiments, the selection of the type of data replication can comprise selecting the first type of data replication if a ratio between the first number and the second number is greater than or equal to a third threshold value. Thus, in some embodiments, the selection can comprise selecting the second type of data replication if the ratio between the first number and the second number is less than the third threshold value. The ratio can be in the form of a quotient, wherein the first number is the numerator of the quotient and the second number is the denominator of the quotient. The third threshold value can be adapted to a further ratio between the first access time and the second access time. By comparing the ratio between the first number and the second number to the third threshold value, the difference between the first access time and the second access time can be taken into account in a simple manner. The ratio can be used as a key value, e.g. as a single key value, based on which the selection of the type of data replication can be performed. The computer system can adapt to a usage of the source database system by adjusting the third threshold value.

[0050] In some embodiments, the ratio can be a weighted ratio comprising the first number weighted in dependence on the first access time and the second number weighted in dependence on the second access time. The ratio as a weighted ratio can be used as a key value, based on which the selection of the type of data replication can be performed, more precisely, as it can take into account the first access time and the second access time.

[0051] In some embodiments, the selection of the type of data replication can include computing a weighted sum of a first quantity and a second quantity, the first quantity being weighted based on the first access time and the second quantity being weighted based on the second access time. Thus, in some embodiments, the selection can include selecting the first type of data replication if the weighted sum is less than a fourth threshold, and selecting the second type of data replication if the weighted sum is greater than or equal to the fourth threshold. The weighted sum can represent an amount of total access time for accessing all records of the first memory device and the second memory device to perform all of the first pending data changes and the second pending data changes. To this end, the weighted sum can serve as a key value for estimating whether a deadlock situation is likely to occur soon. Thus, the weighted sum can serve as a key value based on which the selection of the type of data replication can be performed. The weighted sum can be used to determine the first computation cost described above.

[0052] In some embodiments, the second type of data replication can include loading information for updating all rows of a table of the target database system together and updating all rows of the table of the target database system based on the information for updating all rows of the table of the target database system. The information for updating all rows of the target table can be in the form of all records of the first pending data changes and the second pending data changes in the first case, or in the form of all data values of the source table in the second case. Thus, in the first case, the updating of all rows of the target table can include performing a replication of all data changes according to the first pending data changes and the second pending data changes. In the second case, the updating of all rows of the target table can include setting each data value of the target table to equal a corresponding one of the data values of the source table.

[0053] In some embodiments, after all rows have been updated, all log files can be deleted from the first memory device and the second memory device. It can be assumed, for example, that after such a cleanup of the first memory device and the second memory device, it can take some time until further records of data changes stored on the first memory device can be archived on the second memory device. In this way, some embodiments can provide transitioning the computer system from a state in which records of data changes are continuously loaded from the second memory device to perform a replication of the data changes to a state in which records of data changes are continuously loaded from the first memory device to perform the replication. As a result, the performance of the replication can be improved.

[0054] In some embodiments, the selection of the type of data replication can comprise using respective log numbers. For example, each respective log number can be assigned to a respective one of the log files. The respective log number can indicate a total number of records of data changes of the table of the source database system that have been written at a respective time instant at which generation of the respective log file started. In some embodiments, the log numbers assigned to the log files can be used when performing the tracking of the first number and the second number. This can reduce the computational cost for determining the first number and / or the second number. For example, it can not be necessary to read all records of a log file stored on the second memory device to count these records. The same applies for determining the second number.

[0055] In some embodiments, the log files each comprise a respective log number assigned to them. For example, the log number assigned to each respective log file can be embedded in the respective log file. This can provide the advantage that the log numbers and their assignment to the respective log files do not need to be stored on a further memory device than the first memory device and the second memory device.

[0056] In some embodiments, the tracking of the first number can comprise calculating a difference between the log number assigned to the log file stored on the first memory device for the shortest time and the log number assigned to the log file stored on the first memory device for the longest time. The first number can be set to the difference between the log numbers. In some embodiments, this calculation can provide a simple and fast way to determine the first number. The log file stored on the first memory device for the longest time can also be referred to herein as the oldest log file. The log file stored on the first memory device for the shortest time can also be referred to herein as the youngest log file.

[0057] In some embodiments, the tracking of the first number can comprise calculating a difference between an actual total number of data changes of the table of the source database system at an actual time instant and the log number assigned to the log file stored on the first memory device for the longest time. The actual total number of data changes can be retrieved from the source database. The source database system can comprise a record counter for counting the actual total number of data changes. Using the actual total number of data changes instead of the log number of the youngest log file can allow a more precise determination of the first number. Typically, the last record generated by the source database system, for example referred to herein as the last generated record, can be written directly into the youngest log file.

[0058] In some embodiments, the selection of the type of data replication can include tracking a record number of a last record of data changes (e.g., also referred to as a last processed record or a last record of data changes) read out from the first memory device or the second memory device and used to replicate data changes to a table in the target database system in accordance with the last record of data changes. In some embodiments, the selection of the type of data replication can include tracking the first quantity or the second quantity using the record number of the last record of data changes. The record number of the last processed record can represent a total number of records of data changes of the table of the source database system written at a time of writing the last processed record, including the last processed record. Each record of data changes can include a record number equal to a total number of records of data changes of the table of the source database system written at a time of writing the record, including the record. Tracking the record number of the last processed record can facilitate tracking the first quantity and / or the second quantity. For example, the first counter and the second counter described above can not need to be used and updated in such cases. The tracking of the record number of the last processed record can include reading the record number of the last processed record.

[0059] In some embodiments, the selection of the type of data replication can include calculating the second quantity as a difference between a log number assigned to a log file stored on the first memory device for the longest time and the record number assigned to the last record of data changes. In some embodiments, the last record of data changes can be included or provided by one of the log files archived on the second memory device. Such embodiments can provide a simple way of determining the second quantity without using the first counter or the second counter, but using only the record number of the last processed record and the log number assigned to the oldest log file.

[0060] In some embodiments, the selection of the type of data replication can include calculating the first quantity as a difference between a log number assigned to a log file stored on the first memory device for the shortest time and the record number assigned to the last record of data changes. In some embodiments, the last record of data changes can be included in or provided by one of the log files stored on the first memory device. Such embodiments can provide a simple way of determining the first quantity without using the first counter or the second counter, but using only the record number of the last processed record and the log number assigned to the youngest log file.

[0061] In some embodiments, the selection of the type of data replication can include restoring the log files. The restoring can include storing the log files archived on the second memory device on the first memory device. The restored log files can be read and processed from the first memory device instead of from the second memory device to perform the replication of the data changes. Thus, the replication of the data changes can be accelerated. The restoring can be performed based on an estimation of a computational cost of the restoring of the log files.

[0062] In some embodiments, the selection of the type of data replication can include performing the restoring of the log files based on respective sizes of the records of the data changes of the log files archived on the second memory device. The estimation of the computational cost of the restoring of the log files can be performed based on the respective sizes of the records of the data changes of the log files archived on the second memory device. In this way, the decision whether the restoring of the log files should be performed can be made based on more detailed information about the computational cost of the restoring of the log files.

[0063] In some embodiments, the selection of the type of data replication can include counting, using a counter, a total number of records of the data changes of the table of the source database system. The counter can be the record counter described above. In some embodiments, the selection can include determining whether the counter is in a state close to an overflow of the counter. In some embodiments, if the counter is in the state close to the overflow of the counter, the second type of data replication can be selected. Using the records of the data changes of the log files, according to the second type of data replication, the counter can be reset after the data changes of the table of the source database system are replicated into the table of the target database system. In some cases, determining when the counter can be in the state close to the overflow of the counter can allow preventing the overflow of the counter.

[0064] In some embodiments, the second type of data replication can include loading all the records of the data changes stored on the first memory device and the second memory device together and replicating more than one row of the data changes of the table of the source database system into the table of the target database system according to all the loaded records of the data changes. Loading all the records of the data changes stored on the first memory device and the second memory device together can prevent having to perform the recalculation of the log numbers of the log files stored on the first memory device and the second memory device. After loading all the records of the data changes stored on the first memory device and the second memory device, these records can be deleted from the first memory device and the second memory device.

[0065] Embodiments of the present invention can be implemented using a computing device, which can also be referred to as a computer system, a client, a server, etc. Referring now to the figures in general, and to Figure 1 in particular, that that shows one Figure 1The diagram illustrates an example of computer system 10. Computer system 10 is merely one example of a suitable computer system and is not intended to impose any limitation on the scope of use and / or functionality of the embodiments of the invention described herein. In any case, computer system 10 is capable of implementing and / or performing any of the functions set forth herein.

[0066] Computer system 10 may include a first computer system / server 12, which may operate with many other general-purpose or special-purpose computing system environments and / or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the first computer system / server 12 include, but are not limited to, the first memory 411 of computer system 10 (e.g., Figure 4 (as shown), personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed computing environments including any of the above systems or devices.

[0067] The first computer system / server 12 can be described within the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. As an example, the first computer system / server 12 can be implemented in a distributed computing environment where tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside in local and remote computer system storage media, including memory storage devices.

[0068] like Figure 1 As shown, a first computer system / server 12 in computer system 10 is illustrated as a general-purpose computing device. Components of the first computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that couples the various system components, including system memory 28, to the processor 16. Bus 18 represents one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of a variety of bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0069] The first computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by the first computer system / server 12 and includes both volatile and non-volatile media, removable and non-removable media.

[0070] The system memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The first computer system / server 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not shown, a magnetic disk drive can also be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be provided for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media). In such instances, each can be connected to the bus 18 by one or more data media interfaces. As will be further depicted and described below, the memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0071] Program / utility 40 having a set (at least one) of program modules 50, can be stored in memory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, can include an implementation of a network environment. Program modules 50 generally can be configured to carry out the functions and / or methodologies of embodiments of the application as described herein.

[0072] The term "program" or "program module" as used herein, refers to a set of instructions that can include commands that, when read by the processor 16, cause the processor 16 to perform actions. The set of instructions can be in the form of a computer readable program, routine, subroutine, or library of subroutines of a part of a program, which can be executed by the processor 16 and / or can be called by the processor 16 for execution. Preferably, the program modules 50 can be executable programs compiled according to the type of hardware platform of the first computer system / server 12.

[0073] The first computer system / server 12 may also communicate with one or more external devices 14 (such as a keyboard, pointing device, display 24, etc.); one or more devices that enable a user to interact with the first computer system / server 12; and / or any device that enables the first computer system / server 12 to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Such communication may occur via input / output (I / O) interface 22. Additionally, the first computer system / server 12 may communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 20. As depicted, network adapter 20 may communicate with other components of the first computer system / server 12 via bus 18. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the first computer system / server 12. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0074] As an example, the first computer system / server 12 can be designed in the form of a source database system, such as... Figure 4 The source database system 401 is shown. Source database system 401 may include source table 410, such as... Figure 4 and Figure 5 As shown in the diagram. Source table 410 may include data fields, such as... Figure 5 The boxes in the table are shown. Data fields can each include entries, such as numbers and / or strings. Each data field in source table 410 can be assigned a specific number of rows and columns from source table 410. The specific number of rows in source table 410 can be... Figure 5 The number of columns in the source table 410 increases in the direction of the first arrow 501 shown. Figure 5 The direction of the second arrow 502 shown increases. The number of rows and columns for each data field can be used to perform data changes on the corresponding data fields. Furthermore, the selected number of rows and several different selected columns of the source table 410 can be used to perform several data changes on different data fields of a single row specified by the selected row number. As an example, the source table 410 can be stored in the storage system 34 of the first computer system / server 12.

[0075] Computer system 10 may include a second computer system / server, for example, such as Figure 2The computer system / server 212 is operational with numerous other general purpose or special purpose computing system environments. Examples of well- known computing systems, environments, and / or configurations that can be suitable for use with second computer system / server 212 include, but are not limited to, first memory 411, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed computing environments that include any of the above systems or devices, and the like.

[0076] The second computer system / server 212 can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. The second computer system / server 212 can be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

[0077] As Figure 2 The second computer system / server 212 in the computer system 10 is shown in the form of a general-purpose computing device. The components of second computer system / server 212 can include, but are not limited to, one or more processors or processing units 216, a system memory 228, and a bus 218 that couples various system components including system memory 228 to processor 216. Bus 218 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0078] The second computer system / server 212 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by second computer system / server 212 and includes both volatile and non-volatile media, removable and non-removable media.

[0079] The system memory 228 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 230 and / or cache memory 232. The second computer system / server 212 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 234 can be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not specifically shown, a magnetic disk drive can also be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be provided for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media). In such instances, each can be connected to the bus 218 by one or more data media interfaces. As will be further depicted and described below, the memory 228 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0080] The program / utility 240, having a set (at least one) of program modules 250, can be stored in memory 228 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, program data or some combination thereof, can include an implementation of a networking environment. Program modules 250 generally can be configured to carry out the functions and / or methodologies of embodiments of the application as described herein.

[0081] The term "program" or "program module" as used herein, refers to a set of instructions that can include commands that, when read by the processor 216, cause actions to be performed by the processor 216. The set of instructions can be in the form of a computer readable program, routine, subroutine, or library of subroutines of a part of a program, which can be executed by the processor 216 and / or can be called by the processor 216 for execution. Preferably, the program modules 250 can be executable programs compiled according to the type of hardware platform of the second computer system / server 212.

[0082] The second computer system / server 212 can also be communicably coupled to one or more external devices 214 such as a keyboard, a pointing device, a display 224; one or more devices that enable a user to interact with the second computer system / server 212; and / or any devices (e.g., network card, modem, etc.) that enable the second computer system / server 212 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interfaces 222. Still yet, the second computer system / server 212 can be communicably coupled to a network 226, such as a Local Area Network (LAN), a General Wide Area Network (WAN), and / or the Internet through a network adapter 220. As depicted, the network adapter 220 can communicate with the other components of the second computer system / server 212 via a bus 218. It should be understood that, although not shown, other hardware and / or software components could be used in conjunction with the second computer system / server 212. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0083] By way of example, the second computer system / server 212 can be designed in the form of a target database system, such as the target database system 402 shown. Figure 4 The target database system 402 can include a target table 420 as shown in Figure 4 and Figure 6 The target table 420 can include data fields. Each data field of the target table 420 can include entries, such as numbers and / or strings. Each data field of the target table 420 can be assigned a specific number of rows of the target table 420 and a specific number of columns of the target table 420. As shown in Figure 6 the specific number of rows of the target table 420 can be increased in the direction of a first arrow 601. The specific number of columns of the target table 420 can be increased in the direction of a second arrow 602 as shown in Figure 6 The number of rows and the number of columns of each data field can be used to perform data changes of the respective data field. Further, a selected number of rows of the target table 420 and a number of different selected numbers of columns of the target table 420 can be used to perform a number of data changes of different data fields of a single row specified by the selected number of rows. By way of example, the target table 420 can be stored in the storage system 234 of the second computer system / server 212.

[0084] In some embodiments, a computer system, such as the computer system 10 shown in Figure 1 and / or Figure 2 may be used to perform the operations disclosed herein, at least such as the first operation, the second operation, the third operation, the fourth operation, the fifth operation, the sixth operation, the seventh operation, and the eighth operation.

[0085] In some embodiments, the computer system 10 can be configured for replicating data changes of the source table 410 into the target table 420.

[0086] In some embodiments, the first operation can comprise repeatedly generating log files while performing data changes of the source table 410. As an example, Figure 7 A set 430 of generated log files is shown in Fig. 4. The processor 16 can perform each data change of the source table 410 by initiating a write to the storage system 34 which can store the source table 410. Each log file 430 i may comprise at least one record of a data change of the source table 410. The first operation can be performed by means of the source database system 401, e.g. by means of the processor 16.

[0087] In some embodiments, the processor 16 can perform a new data change of the source table 410 and can generate a new record 440 n of a data change comprising necessary information, such as described above, to replicate the new data change of the source table 410 into the target table 420. Thus, for each new data change, a new record 440 n of a data change can be generated. Moreover, log files can be repeatedly generated while performing data changes. In some embodiments, the processor 16 can also perform further data changes of further source tables which are not shown in the figure. Moreover, the processor 16 can generate further records of data changes comprising necessary information to replicate the further data changes of the further source tables. The further records can be written into further log files which are not shown in the figure and / or into the log files 430. Thus, in some embodiments, the log files 430 can comprise one or more of the further records of data changes which are not shown in the figure. In some embodiments, one or more further log files can be stored between the log files 430.

[0088] In some embodiments, the new record 440 m may be generated by writing the necessary information to replicate the new data change into an actual log file 430 n . For each further new data change, a further new record of a data change can be generated in the same way as the new record 440 n . In this way, for several further new data changes, several further new records of data changes can be generated. Figure 8 A set 440 of generated records of data changes is shown in Fig. 4 which the source database system 401 can have generated. Each record 440 i of a data change can comprise necessary information, such as described above, to replicate the respective record 440 iThe corresponding data in source table 410 has changed. For example, source database system 401 may have already generated the first record 4401 with changed data, followed by the second record 4402 with changed data, and so on. The last record generated may be a new record 440. n .

[0089] In some embodiments, the second operation may include storing the log file 430 on the first memory device 411. In some embodiments, the storage of the log file 430 may be performed, for example, by a separate processor 450. The source database 401 may be connected to the first memory device 411 and / or the separate processor 450 via (I / O) interface 22 or via network adapter 20. The first memory device 411 may include or provide a first access time. For example, the first access time may be in the range of 35 to 100 microseconds. Figure 4 This shows 430 actual log files. m 430 per log file i Examples that can be generated directly on the first memory device 411. In some embodiments, this can be performed, for example, by sending a write command from the source database system 401 to another processor 450 of the computer system 10.

[0090] However, in some embodiments, log file 430 i It can be internally written to the source database system 401 and sent from the source database system 401 to the first storage device 411. New record 440 n Writing directly to the first memory device 411 instead of storing it inside the source database system 401 can enable the target database system 402 to read new records 440 as quickly as possible. n Advantages. When writing to the log file 430 i Afterwards and / or during this period, log file 430 can be accessed. i It is stored on the first storage device 411. In some cases, if the actual log file 430... m If the size exceeds a given log file size threshold, then another processor 450 can lock the actual log file 430. m And generate a new actual log file ( Figure 4 (Not shown in the image). If the actual log file is 430... m If locked, processor 16 may no longer be able to write one of the new records whose data has been changed to the actual log file 430. m Instead, processor 16 can write one of the additional new records to a new actual log file.

[0091] In some embodiments, the third operation may include archiving a portion of the log file 430 stored on the first storage device 411 to a second storage device 412. The second storage device 412 may include or provide a second access time. The second access time may be longer than the first access time, for example, the second access time may be about 100 to 200 times longer than the first access time. In some embodiments, an additional processor 450 may perform the third operation. The additional processor 450 may load a portion of the log file from the first storage device 411 and archive it to the second storage device 412. Furthermore, the additional processor 450 may delete the archived portion of the log file 430 on the first storage device 411. This creates space on the first storage device 411 to allow additional new records to be written to the actual log file 430. m Neutralize and / or generate new actual log files.

[0092] In some embodiments, the additional processor 450, the first memory device 411, and the second memory device 412 may be considered as a replication engine.

[0093] In some embodiments, in the fourth operation, the computer system 10 may provide the execution of the first type of data copying, for example, by performing the fourth operation via the processor 216. The first type of data copying may include individually and repeatedly loading records 440 with changed data. i Each record in the database, and records 440 that are modified based on the corresponding individually loaded data. i Copy the corresponding changes to a single row in the source table 410 to the target table 420 (e.g., incremental copying). For example, Figure 9 The first record 4401 showing data changes may include a first value ("2" in this example) and a first value ("1" in this example) for the number of rows and columns of the source table 410 of the modified data field, as specified by these values, along with the new value of the data field ("4812" in this example). The second record 4402 showing data changes may include a second value ("1" in this example) and a second value ("2" in this example) for the number of rows and columns of the source table 410 of the modified data field, as specified by these values, along with the new value of the data field ("9001" in this example).

[0094] 440 per record i This can include the record number field 441 i The record number field is 441. i Contains 440 corresponding records i The record number indicates when the corresponding record 440 was generated. ithe total number of records of data changes of the source table 410 written at the corresponding time.

[0095] According to the first type of data replication, the target database system 402 can load the first record 4401 from the first memory device 411 or the second memory device 412 and replicate the data change of the second row of the source table 410 into the target table 420 according to the first record 4401. As an example, the processor 216 can provide setting the value of the data field of the target table 420 specified by the row number "2" and the column number "1" to be equal to 4812. Further, according to the first type of data replication, after the replication of the data change in the target table 420 has been performed according to the first record 4401, the target database system 402 can load the second record 4402 from the first memory device 411 or the second memory device 412 and replicate the data change of the first row of the source table 410 into the target table 420 according to the second record 4402. For example, the processor 216 can provide setting the value of the data field of the target table 420 specified by the row number "1" and the column number "2" to be equal to 9001. Thus, in some embodiments, the additional processor 450 can access the first memory device 411 or the second memory device 412 each time one of the first record 4401 and the second record 4402 is loaded from the first memory device 411 or the second memory device 412. For example, in this case, to load the first record 4401 and the second record 4402, the first memory device 411 or the second memory device 412 can be accessed twice or the first memory device 411 and the second memory device 412 can each be accessed once.

[0096] In some embodiments, in a fifth operation, the computer system 10 can provide for performing a second type of data replication, e.g., by performing the fifth operation via the processor 216, the second type of data replication can be performed. The second type of data replication can include loading information for updating several rows of the target table 420 together (e.g., batch replication, etc.). In some embodiments, the information can be presented by several of the records of data changes 440 i , e.g., two of the records 440 i , one hundred of the records 440 i , all of the records 440 i , etc. In this way, several of the records of data changes 440 i can be loaded together from the first memory device 411 or the second memory device 412. Further, the loaded records of data changes 440 iMore than one row of data of the source table 410 is changed into the target table 420. As an example, the processor 216 can load the first record 4401 and the second record 4402 together from the first memory device 411 or the second memory device 412 and perform a first data change in the target table 420 according to the first record 4401 and a second data change in the target table 420 according to the second record 4402. Loading several records together can involve accessing the first memory device 411 or the second memory device 412 only once and reading the several records consecutively starting from a read position that can be achieved by a single access of the first memory device 411 or the second memory device 412.

[0097] In some embodiments, for the second type of data replication, the information used to update several rows of the target table 420 can be in the form of data values of the source table 410, e.g. a partition of data values in the source table or all data values in the source table. As an example, in case the source table 410 is ordered with respect to a date, a partition of data values of the source table 410 can represent a certain month. The data values can be loaded from the source database system 401, e.g. from the storage system 34 or the like, directly. The data values of the source table 410 can be loaded together from the source database system 401 to the target database system 402, e.g. by means of the further processor 450 or the like, in the form of a matrix which can have multiple dimensions. In some embodiments, the records 440 of data changes can not be used for the second type of data replication.

[0098] Figure 10 An example embodiment is shown in which the first record 4401 and the second record 4402 are stored on the second memory device 412. The total number of log files 430 stored on the first memory device 411 and the second memory device 412 can be equal to m. The number of log files 430 stored on the first memory device 411 can be equal to l.

[0099] The m-th log file 430 of the log files 430 m may be the youngest log file of the first memory device 411, the (m-l+1)-th log file 430 of the log files 430 m-l+1 may be the oldest log file of the first memory device 411. The number of log files 430 stored on the second memory device 412 can be equal to m-l. The (m-l)-th log file 430 of the log files 430 m-l may be the youngest log file of the second memory device 412. The first log file 430 l may be the oldest log file of the second memory device 412. The records of the log files 430 are shown as boxes in Figure 10

[0100] ​In some embodiments, the sixth operation can include tracking a second number of second pending data changes. The sixth operation can be performed by way of the further processor 450, etc. The second pending data changes can be data changes of records 440 of the journal files 430 archived on the second memory device 412 that have not yet been copied into the target table 420 i The second number can be tracked by counting the number of second pending records. In some embodiments, a record number of a last record of data changes (e.g., referred to as a last record of data changes) that is read out from the first memory device 411 or the second memory device 412 and used for copying of data changes in the target table 420 can be tracked. For example, if the last record of data changes is the second record 4402, the record number of the last record of data changes can equal to "2". The record number of the last record of data changes can be tracked by reading the value of the second record number field 4412 of the second record 4402. In some embodiments, the second number can be calculated by subtracting the record number of the last record of data changes from the record number of the youngest record of the youngest journal file 430 of the second memory device 412 m-l In some embodiments, the second number can be determined by counting the records stored on the journal files of the second memory device 412 from the last record of data changes.

[0101] In some embodiments, the seventh operation can include selecting a type of data copying from the first type of data copying and the second type of data copying based on the second number. For example, if the second number is greater than a second threshold, the second type of data copying can be selected, e.g., by the processor 216, etc. Otherwise, the first type of data copying can be selected (e.g., the second number is not greater than the second threshold). The seventh operation can be performed by the further processor 450, the processor 216, etc.

[0102] In some embodiments, a first number of first pending data changes can be tracked. The first pending data changes can be data changes of records 440 of the journal files 430 stored on the first memory device 411 that have not yet been copied into the target table 420 i In some embodiments, if the first number and the second number are tracked, the selection can be based on the first number and the second number. In some embodiments, the first number can be determined by counting the records stored on the first memory device 411 from the record number of the last record of data changes forward to the youngest record of the newest journal file 430 of the first memory device 411 m This can be the case when the last record of data changes is read out from the first memory device 411.

[0103] In some embodiments, the selection of the type of data replication can involve performing a weighting of the first number and the second number. For example, the weighting of the first number can be based on the first access time (e.g., access time 1), and the weighting of the second number can be based on the second access time (e.g., access time 2).

[0104] As an example, the weighting of the first number n1 and the second number n2 and S can be computed according to the following equation:

[0105] S = access time 1 * n1 + access time 2 * n2.

[0106] In some embodiments, the selection of the type of data replication can be performed such that if the weighted sum S is less than a fourth threshold, then the first type of data replication is selected, and if the weighted sum S is greater than or equal to the fourth threshold, then the second type of data replication is selected. In some embodiments, as an example, the fourth threshold can represent an approximate time for performing the updates of all rows of the target table 420 according to the second type of data replication. In another example, in some embodiments, the fourth threshold can be determined based on past experiences, such as when a deadlock situation is reached or a situation is reached that is close to a deadlock situation. For example, the fourth threshold can be determined by computing another weighted sum (such as according to the formula given above), but using another number of the first pending data changes and the second pending data changes at the time when the deadlock situation or the situation close to the deadlock situation is reached, instead of using the first number n1 and the second number n2. In some embodiments, the fourth threshold can then be determined by multiplying the other weighted sum by a value between 0.1 and 0.9 to prevent the deadlock situation.

[0107] In some embodiments, the eighth operation can include using the records 440 of data changes of the log file 430 to replicate the data changes of the source table 410 into the target table 420 according to the selected type of data replication. The eighth operation can be performed by the processor 216, etc. i

[0108] In some embodiments, the computer system 10 can be a standalone computer that does not have network connectivity, which can receive data to be processed through a local interface. However, such operations can equally be performed using a computer system that is connected to a network, such as a communication network and / or a computing network.

[0109] Figure 3 ​An example computing environment is shown in which a computer system, such as computer system 10, is connected to a network 200, e.g., using network adapter 20. Without limitation, network 200 can be a communications network, such as the Internet or other public network, a wide area network (WAN), a local area network (LAN), a wireless network, such as a mobile communications network, etc. Network 200 can include a computing network, such as a cloud computing network. Computer system 10 can receive data to be processed from network 200 and / or can provide computational results to another computing device connected to computer system 10 via network 200.

[0110] In some embodiments, computer system 10 can perform, in whole or in part, the operations described herein, such as the first operation, the second operation, the third operation, the fourth operation, the fifth operation, the sixth operation, the seventh operation, and the eighth operation, in response to a request received via network 200. In particular, computer system 10 can perform such operations in distributed computing with one or more additional computer systems, which can be connected to computer system 10 via network 200. To this end, computing system 10 and / or any additionally involved computer systems can access additional computing resources, such as dedicated memory or shared memory, etc., using network 200.

[0111] Figure 11 An example of computer system 10 is shown in which each log file 430 i may include a respective log number 431 i , as shown in the form of a circle in Figure 11 some embodiments. In some embodiments, respective log number 431 i may indicate the total number of records of data changes of source table 410 that have been generated until the respective instant at which the generation of respective log file 430 i begins. Respective log number 431 i may be included in respective log file 430 i in the form of a respective header row. In the following example, it can be assumed that there are two log files stored on first memory device 411, namely first log file 4301 and second log file 4302, and there are two log files stored on second memory device 412, namely third log file 4303 and fourth log file 430 m . Thus, in accordance with this example, m is equal to “4” and l is equal to “2”. Log number 4311 of first log file 4301 can be equal to “10”, log number 4312 of second log file 4302 can be equal to “32”, log number 4313 of third log file 4303 (e.g., 431 m-l+1) can equal "42", the log number 4314 (e.g., 431 m ) of the fourth log file 4304. In this example, the fourth log file 4304 can be the actual log file 430 m .

[0112] The first number can be determined by calculating the difference between the log number (here, the log number 431 m ) of the youngest log file 430 m assigned to the first memory device 411 and the log number 431 m-1+1 of the oldest log file 430 m-1+1 assigned to the first memory device 411. The first number can be determined in this way if the last record of a data change is stored on the second memory device 412.

[0113] In some embodiments, the first number can be determined by calculating the difference between the actual total number of data changes of the source table 410 at the actual point in time and the log number 431 m-1+1 of the oldest log file 430 m-1+1 of the first memory device 411. In some embodiments, the source database system 401 can comprise a counter for counting the actual total number of data changes of the source table 410 and can send the actual total number of data changes to the further processor 450, the target database system 402, etc.

[0114] Furthermore, in some embodiments, using the log number 431 i , the second number can be determined as the difference between the log number 431 m-1+1 of the oldest log file 430 m-1+1 of the first memory device 411 and the record number of the last record of a data change. In this case, the last record of a data change can be stored in one of the log files 430 archived on the second memory device 412.

[0115] Furthermore, in some embodiments, using the log number 431 i , the first number can be calculated as the difference between the log number 431 m of the youngest log file 430 m of the first memory device 411 and the record number of the last record of a data change. In this case, the last record of a data change can be one of the records 440 i of a log file 430 stored on the first memory device 411.

[0116] Figure 12A computer-implemented method for replicating data changes of a source table 410 into a target table 420 according to an embodiment of the present application is shown.

[0117] In step 1201, log files 430 are repeatedly generated while performing data changes of the table 410 of the source database system 401 i each log file 430 i comprises at least one record of a data change of the table 410 of the source database system 401.

[0118] In step 1202, the log files 430 i are stored on a first memory device 411, the first memory device 411 having a first access time.

[0119] In step 1203, a part of the log files 430 i stored on the first memory device 411 are archived on a second memory device 412, the second memory device 412 having a second access time. The second access time can be higher than the first access time.

[0120] In step 1204, a first type of data replication can be provided.

[0121] In step 1205, a second type of data replication can be provided.

[0122] In step 1206, a second number indicating a number of second pending data changes can be tracked.

[0123] In step 1207, a type of data replication can be selected from the first type of data replication and the second type of data replication depending on the second number and / or the first number.

[0124] In step 1208, data changes 440 i of the log files 430 i are replicated into the target table 420 of the target database system 402 according to the selected type of data replication.

[0125] The present application can be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0126] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0127] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0128] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and a procedural programming language such as the "C" programming language or the like. The computer readable program instructions can execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0129] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0130] These computer readable program instructions can be provided to a processor of a computer, or other programmable data processing apparatus, to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include non- transitory computer-readable storage media, that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions for causing an apparatus to implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0131] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0133] The invention will be described again below by a list of terms that emphasize several possible non-exclusive combinations of the features disclosed herein.

[0134] Clause 1: A computer-implemented method for copying data changes of a table in a source database system to a table in a target database system, the method comprising: repeatedly generating log files while performing data changes on a table in the source database system, each log file including at least one record of a data change in the table of the source database system; storing the log files on a first storage device having a first access time; archiving a portion of the log files stored on the first storage device on a second storage device having a second access time, wherein the second access time is longer than the first access time; providing a first type of data replication, the first type of data replication comprising repeatedly and individually loading one of the records of the data changes, and copying the source database system data to a table based on the corresponding individually loaded record of the data changes. The process involves: copying corresponding data changes of a single row in a table of a database system to a table in the target database system; providing a second type of data copying, which includes loading information for updating several rows in a table of the target database system and updating those rows based on that information; tracking a second number of second pending data changes, which are data changes recorded in a log file archived on a second storage device that have not yet been copied to a table in the target database system; selecting a data copying type from the first and second types of data copying based on the second number; and using the records of data changes in the log file, copying the data changes from the table of the source database system to a table in the target database system according to the selected data copying type.

[0135] Clause 2: The method of clause 1, further comprising tracking a first number of first pending data changes, the first pending data changes being data changes in records of data changes of the log files stored on the first memory device that have not yet been copied into the tables of the target database system, the selection depending on the first number and the second number.

[0136] Clause 3: The method of clause 1 or 2, wherein loading information for updating the number of rows of the table of the target database system comprises loading a number of records of the records of data changes together, wherein updating the number of rows of the table of the target database system comprises copying data changes of the number of rows of the table of the source database system into the table of the target database system according to the loaded records of data changes, each of the number of rows of the table of the source database system corresponding to one of the number of rows of the table of the target database system.

[0137] Clause 4: The method of clause 1 or 2, wherein loading information for updating the number of rows of the table of the target database system comprises loading data values of the number of rows of the table of the source database system.

[0138] Clause 5: The method of one of the preceding clauses, wherein the selection of the type of data replication comprises a weighting of the first number and the second number, the weighting of the first number being based on the first access time and the weighting of the second number being based on the second access time.

[0139] Clause 6: The method of one of the preceding clauses, further comprising selecting the first type of data replication if a ratio between the first number and the second number is greater than or equal to a third threshold value, and selecting the second type of data replication if the ratio between the first number and the second number is less than the third threshold value, the ratio being a weighted ratio comprising the first number weighted based on the first access time and the second number weighted based on the second access time.

[0140] Clause 7: The method of one of the preceding clauses, wherein the selection of the type of data replication comprises computing a weighted sum of the first number and the second number, the first number being weighted based on the first access time and the second number being weighted based on the second access time; selecting the first type of data replication if the weighted sum is less than a fourth threshold value; and selecting the second type of data replication if the weighted sum is greater than or equal to the fourth threshold value.

[0141] Clause 8: The method of one of the preceding clauses, wherein the second type of data replication comprises loading information for updating all rows of the table of the target database system together, and updating all rows of the table of the target database system based on the information.

[0142] Clause 9: The method of one of the preceding clauses, further comprising tracking the first number and the second number using respective log numbers, each respective log number being assigned to a respective one of the log files, wherein each respective log number indicates a total number of records of data changes of the table of the source database system that have been written up to a respective time at which generation of the respective log file began, and wherein the first number and the second number are tracked based on the respective log numbers assigned to the log files.

[0143] Clause 10: The method of any one of the preceding clauses, wherein the log files each comprise a respective log number, each respective log number being assigned to a respective one of the log files, wherein each respective log number indicates a total number of records of data changes of the table of the source database system that have been written up to a respective time at which generation of the respective log file began, and wherein the first number and the second number are tracked based on the respective log numbers of the log files.

[0144] Clause 11 : The method of clause 9 or 10, wherein the tracking of the first number comprises calculating a difference between the respective log number assigned to the log file stored on the first memory device for the shortest time and the respective log number assigned to the log file stored on the first memory device for the longest time.

[0145] Clause 12: The method of clause 9 or 10, wherein the tracking of the first number comprises calculating a difference between an actual total number of data changes of the table of the source database system at an actual time and the respective log number assigned to the log file stored on the first memory device for the longest time.

[0146] Clause 13: The method of one of the preceding clauses, further comprising tracking a record number of a last record of data changes read out of the first memory device or the second memory device and used to replicate data changes into the table of the target database system according to the last record of data changes, and using the record number of the last record of data changes to track the first number or the second number.

[0147] Clause 14: The method of clauses 9 or 10 and 13, further comprising calculating the second number as a difference between the respective log number assigned to the log file stored on the first memory device for the longest time and the record number of the last record of data changes, wherein the last record of data changes is included in one of the log files archived on the second memory device.

[0148] Clause 15: The method of clause 9 or 10 and 13, further comprising calculating the first number as a difference between a respective log number assigned to a log file stored on the first memory device for a shortest time and a record number assigned to a last record of the data changes, wherein the last record of the data changes is included in one of the log files stored on the first memory device.

[0149] Clause 16: The method of one of the preceding clauses, further comprising restoring the log files, the restoring comprising storing the log files archived on the second memory device on the first memory device.

[0150] Clause 17: The method of clause 16, further comprising performing the restoring based on respective sizes of records of data changes of the log files archived on the second memory device.

[0151] Clause 18: The method of clause 9 or 10, further comprising counting a total number of records of data changes of a table of the source database system using a counter; determining whether the counter is in a state close to an overflow of the counter; selecting the second type of data replication if the counter is in the state close to the overflow of the counter; resetting the counter after replicating the data changes of the table of the source database system into the table of the target database system according to the second type of data replication using the records of data changes of the log files; wherein the second type of data replication comprises loading all records of data changes stored on the first memory device and the second memory device together, and replicating more than one row of data changes of the table of the source database system into the table of the target database system according to all loaded records of data changes.

[0152] Clause 19: A computer program product comprising a computer readable storage medium having computer readable program instructions embodied therewith, the computer readable program instructions configured to implement steps of the method according to any one of the preceding method clauses 1 to 18.

[0153] Clause 20: A computer system for replicating data changes of a table of a source database system into a table of a target database system, the computer system comprising the source database system and the target database system, the computer system configured for: repeatedly generating log files while executing data changes of a table of the source database system, each log file comprising at least one record of data changes of the table of the source database system; storing the log files on a first memory device, the first memory device having a first access time; archiving a portion of the log files stored on the first memory device on a second memory device, the second memory device having a second access time, wherein the second access time is higher than the first access time; tracking a second number of second pending data changes, the second pending data changes being data changes in records of data changes of the log files archived on the second memory device that have not been replicated into the table of the target database system; selecting a type of data replication from a first type of data replication and a second type of data replication according to the second number; and replicating the data changes of the table of the source database system into the table of the target database system according to the selected type of data replication using the records of data changes of the log files.

[0154] Clause 21 : The computer system of clause 20, wherein the first type of data replication comprises repeatedly individually loading one of the records of data changes and replicating respective data changes of a single row of the table of the source database system into the table of the target database system according to the corresponding individually loaded record of data changes.

[0155] Clause 22: The computer system of clause 20, wherein the second type of data replication comprises loading information for updating several rows of the table of the target database system together and updating the several rows of the table of the target database system based on the information.

[0156] Clause 23: A computer system for replicating data changes of a table of a source database system into a table of a target database system, the computer system comprising the source database system and the target database system, the computer system configured to implement the steps of the method according to any of the preceding method clauses 1 to 18.

Claims

1. A computer-implemented method, comprising: Log files are repeatedly generated while data changes are made to the tables in the source database system. Each log file includes: At least one record of the table in the source database system whose data has been changed; Log number, which indicates the total number of records in the source database system whose data has been changed up to the point at which the corresponding log file was generated; The log file is stored on a first storage device, the first storage device having a first access time; A portion of the log file stored on the first storage device is archived on a second storage device, the second storage device having a second access time, wherein the second access time is longer than the first access time; Track a first number of first pending data changes, wherein the first pending data changes are data changes in records of data changes in the log file stored on the first storage device that have not yet been copied to the target database system; Track a second number of second pending data changes, which are data changes recorded in the log file archived on the second storage device that have not yet been copied to the target database system; Based on the first quantity and the second quantity, select the data replication type from the first type of data replication and the second type of data replication; and Using the records of data changes in the log file, and depending on the selected data replication type, the data changes in the tables of the source database system are copied to the tables of the target database system.

2. The computer-implemented method according to claim 1, wherein, The first type of data replication includes loading each of the records in which the data has been changed individually, and copying the corresponding data changes of a single row in the source database system's table to the table in the target database system based on the corresponding individually loaded record of data change.

3. The computer-implemented method according to claim 1, wherein, The second type of data replication includes loading information for updating several rows of a table in the target database system and updating those rows based on the information.

4. The computer-implemented method according to claim 1, wherein, The selection of the data replication type includes a first weighted number and a second weighted number, wherein the first weighted number is based on the first access time and the second weighted number is based on the second access time.

5. The computer-implemented method according to claim 1, further comprising: In response to the ratio between the first quantity and the second quantity being greater than or equal to a third threshold, the first type of data copying is selected; The ratio is a weighted ratio, which includes a first quantity weighted according to the first access time and a second quantity weighted according to the second access time.

6. The computer-implemented method according to claim 1, further comprising: If the ratio between the first quantity and the second quantity is less than a third threshold, the second type of data replication is selected; The ratio is a weighted ratio, which includes a first quantity weighted according to the first access time and a second quantity weighted according to the second access time.

7. The computer-implemented method according to claim 1, wherein, The selection of the type of data replication includes: Calculate a weighted sum of the first quantity and the second quantity, wherein the first quantity is weighted based on the first access time and the second quantity is weighted based on the second access time; and If the weighted sum is less than the fourth threshold, the first type of data replication is selected; if the weighted sum is greater than or equal to the fourth threshold, the second type of data replication is selected.

8. The computer-implemented method according to claim 1, wherein, The first quantity tracking includes calculating the difference between the corresponding log number assigned to the log file stored on the first storage device for the shortest duration and the corresponding log number assigned to the log file stored on the first storage device for the longest duration.

9. The computer-implemented method according to claim 1, wherein, The tracking of the first quantity includes calculating the difference between the actual total number of data changes to the tables of the source database system at any given time and the corresponding log number assigned to the log file that has been stored for the longest time on the first storage device.

10. The computer-implemented method according to claim 1, further comprising: The record number of the last record of data change read from the first memory device or the second memory device and used to copy the data change to the table of the target database system based on the last record of data change; as well as Use the record number of the last record where the data changed to track the first quantity or the second quantity.

11. The computer-implemented method of claim 10, further comprising calculating the second quantity as the difference between a corresponding log number assigned to the longest-running log file stored on the first storage device and a record number assigned to the last record of the data change, wherein, The last record of the data change is included in one of the log files archived on the second storage device.

12. The computer-implemented method of claim 10, further comprising calculating the first quantity as the difference between a corresponding log number assigned to the log file stored for the shortest time on the first storage device and a record number assigned to the last record of the data change, wherein, The last record of the data change is included in a log file stored on the first storage device.

13. The computer-implemented method of claim 1, further comprising restoring the log file, the restoration comprising storing the log file archived on the second storage device on the first storage device.

14. The computer-implemented method of claim 13, further comprising performing the recovery based on the corresponding size of a record of data changes in the log file archived on the second storage device.

15. The computer-implemented method according to claim 1, further comprising: Use a counter to count the total number of records in the tables of the source database system whose data has been changed; Determine that the counter is in a state close to overflowing; In response to the counter being close to overflowing, the second type of data copying is selected; as well as Using the records of data changes in the log file, and based on the second type of data copying, after copying the data changes from the tables in the source database system to the tables in the target database system, the counter is reset; The second type of data replication includes loading all records of data changes stored on the first and second storage devices together, and copying more than one row of data changes from a table in the source database system to a table in the target database system based on all loaded records of data changes.

16. A computer program product comprising a computer-readable storage medium having computer-readable program instructions embodied therein, the computer-readable program instructions being executable by one or more processors to perform a method comprising: Log files are repeatedly generated while data changes are made to the tables in the source database system. Each log file includes: At least one record of the table in the source database system whose data has been changed; Log number, which indicates the total number of records in the source database system whose data has been changed up to the point at which the corresponding log file was generated; The log file is stored on a first storage device, the first storage device having a first access time; A portion of the log file stored on the first storage device is archived on a second storage device, the second storage device having a second access time, wherein the second access time is longer than the first access time; Track a first number of first pending data changes, wherein the first pending data changes are data changes in records of data changes in the log file stored on the first storage device that have not yet been copied to the target database system; Track a second number of second pending data changes, which are data changes recorded in the log file archived on the second storage device that have not yet been copied to the target database system; Based on the first quantity and the second quantity, select the data replication type from the first type of data replication and the second type of data replication; and Using the records of data changes in the log file, and depending on the selected data replication type, the data changes in the tables of the source database system are copied to the tables of the target database system.

17. A computer system, comprising: At least one processor; At least one computer-readable storage medium; Source database system; as well as Target database system; in: The at least one processor is structured, positioned, linked, and programmed to execute program instructions stored on the at least one computer-readable storage medium; and The stored program instructions include: Program instructions programmed to repeatedly generate log files while executing data changes to tables in the source database system, each log file including: At least one record of the table in the source database system whose data has been changed; Log number, which indicates the total number of records in the source database system whose data has been changed up to the point at which the corresponding log file was generated; Program instructions programmed to store the log file on a first memory device having a first access time; Program instructions programmed to archive a portion of the log file stored on the first memory device on a second memory device, the second memory device having a second access time, wherein the second access time is longer than the first access time; Program instructions are programmed to track a first number of first pending data changes, the first pending data changes being data changes in records of data changes in log files stored on the first storage device that have not yet been copied to tables in the target database system; Program instructions are programmed to track a second number of second pending data changes, the second pending data changes being data changes recorded in the log file archived on the second storage device and not yet copied to the target database system; Program instructions programmed to select the type of data replication from a first type of data replication and a second type of data replication based on the first quantity and the second quantity; and Program instructions programmed to use records of data changes in the log file to copy data changes from tables in the source database system to tables in the target database system, based on the selected type of data replication.

Images