Replicating data changes using multiple storage devices and tracking records of pending data changes stored on the multiple storage devices

Abstract

A method, computer program product, or system, or combination thereof, is provided that performs the operations of generating a plurality of log files while performing data changes of a table of a source database system; storing the plurality of log files on a first memory device having a first access time; archiving a portion of the plurality of log files from the first memory device on a second memory device having a second access time, where the second access time is greater than the first access time; tracking a second number of pending data changes, where the pending data changes are the data changes of the records of the data changes of the plurality of log files archived on the second memory device that have not yet been replicated into a table of a target database system; selecting a type of data replication from a first type of data replication and a second type of data replication based on the second number; and replicating the data changes into the table of the target database system.

Description

[Technical Field]

[0001] The present invention relates to the field of database technology, and more particularly to replicating data changes in tables of a source database system into tables of a target database system. [Background technology]

[0002] Replicating data changes made in a table of a source database system to a table of a target database system may be performed to create a backup of the table of the source database system. Another use of such replication of data changes may be to synchronize the table of the target database system with the table of the source database system, for example, when the table of the target database system includes a different ordering scheme than the table of the source database system. Generally, the replication of data changes should be performed as quickly as possible to keep the data of the table of the target database system as synchronized as possible with the data of the table of the source database system. The data changes may be replicated according to different types of replication strategies. One type may involve individually replicating a single row of the table of the source database system. Another type of replication strategy may involve replicating multiple rows of the table of the source database system at once, even a large portion of the rows. Summary of the Invention [Means for solving the problem]

[0003] Various embodiments provide a method, a computer system and a computer program product for replicating data changes as set forth in the independent claims. Advantageous embodiments are set forth in the dependent claims. The embodiments of the invention can be freely combined with one another if they are not mutually exclusive.

[0004] According to an aspect of the present invention, there is provided a method for replicating data changes in tables of a source database system into tables of a target database system. The method includes repeatedly generating log files while executing data changes in tables of the source database system, where each log file contains at least one record of the data changes in the tables of the source database system. The method further includes storing the log files on a first memory device, where the first memory device has a first access time. The method further includes archiving a portion of the log files stored on the first memory device onto a second memory device, where the second memory device has a second access time, where the second access time is longer than the first access time. The method further includes tracking a second number of second pending data changes, where the second pending data changes are data changes of the records of the data changes in the plurality of log files archived on the second memory device that have not yet been replicated into tables of the target database system. The method further includes selecting a type of data replication from a first type of data replication and a second type of data replication based on the second number, and replicating the data changes of the table in the source database system into the table in the target database system according to the selected type of data replication using the records of data changes in the plurality of log files.

[0005] In some embodiments, the first type of data replication includes individually loading each of the records of data changes and replicating each data change of a single row of the table of the source database system into the table of the target database system according to the individually loaded corresponding record of data changes. In some embodiments, the second type of data replication includes loading information for jointly updating multiple rows of the table of the target database system and updating the multiple rows of the table of the target database system based on the information.

[0006] In some embodiments, the method may optionally further include tracking a first number of first pending data changes, where the first pending data changes are the data changes of the records of the data changes of the plurality of log files stored on the first memory device that have not yet been replicated to the table of the target database system; and selecting the type of data replication based on the first number and the second number.

[0007] In some embodiments, optionally, selecting the type of data replication may include weighting the first number and weighting the second number, where the weighting of the first number is based on the first access time and the weighting of the second number is based on the second access time.

[0008] In some embodiments, the method may optionally further include selecting the first type of data replication if the ratio of the first number to the second number is greater than or equal to a third threshold, and selecting the second type of data replication if the ratio of the first number to the second number is less than a third threshold. The ratio may be a weighted ratio including the first number weighted depending on the first access time and the second number weighted depending on the second access time.

[0009] In some embodiments, the method may optionally further include calculating a weighted sum of the first number and the second number, where the first number is weighted based on the first access time and the second number is weighted based on the second access time. In such embodiments, if the weighted sum is less than a fourth threshold, the first type of data replication is selected, or if the weighted sum is greater than or equal to a fourth threshold, the second type of data replication is selected.

[0010] In some embodiments, the method may optionally further include obtaining a respective log number assigned to a respective one of the plurality of log files, where each log number indicates a total number of records of the data changes in the table of the source database system that have been written up to a respective moment when generation of the respective log file begins. In such embodiments, the method may further include tracking the first number and the second number using the respective log numbers assigned to the plurality of log files.

[0011] Optionally, in some embodiments, tracking the first number may include calculating the difference between the respective log number assigned to the log file that has been stored on the first memory device for the shortest time and the respective log number assigned to the log file that has been stored on the first memory device for the longest time.

[0012] Optionally, in some embodiments, tracking the first number may include calculating a difference between the actual total number of data changes of the table in the source database system at the actual moment and the respective log number assigned to the log file that has been stored for the longest time on the first memory device.

[0013] In some embodiments, the method may optionally further include tracking a record number of the last record of data changes that is read from the first memory device or the second memory device and used for replicating data changes in the table of the target database system according to the last record of data changes; and using the record number of the last record of data changes to track the first number or the second number.

[0014] In some embodiments, the method may optionally further include calculating the second number as the difference between the respective log number assigned to the log file that has been stored for the longest time on the first memory device and the record number assigned to the last record of a data change, where the last record of a data change is included in one of the plurality of log files archived on the second memory device.

[0015] In some embodiments, the method may optionally further include calculating the first number as the difference between the respective log number assigned to the log file stored on the first memory device for the shortest time and the record number assigned to the last record of a data change, where the last record of a data change is included in one of the plurality of log files stored on the first memory device.

[0016] In some embodiments, the method may optionally further include counting the total number of records of data changes for the table in the source database system using a counter. In such embodiments, the method may further include determining whether the counter is close to overflowing. In such embodiments, the method may further include selecting the second type of data replication if the counter is close to overflowing. In such embodiments, the method may further include resetting the counter after replicating the data changes for the table in the source database system into the table in the target database system according to the second type of data replication using the records of data changes in the multiple log files. In some embodiments, the second type of data replication may include loading all records of data changes stored on the first memory device and the second memory device together, and replicating the data changes for more than one row of the table in the source database system into the table in the target database system according to the loaded records of data changes.

[0017] According to another aspect of the present invention, there is provided a computer program product comprising a computer-readable storage medium having a plurality of computer-readable program instructions thereon, wherein the plurality of computer-readable program instructions are executable by one or more processors to perform a method according to any of the above-described embodiments.

[0018] According to another aspect of the present invention, there is provided a computer system for replicating data changes in a table of a source database system into a table of a target database system, the computer system being configured to perform a method according to any of the above-described embodiments.

[0019] In the following, embodiments of the invention will be described in more detail, by way of example only, and with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 illustrates a computer system according to an embodiment of the present invention. [Figure 2] FIG. 2 illustrates a second computer system / server of the computer system shown in FIG. 1 according to an embodiment of the present invention. [Figure 3] FIG. 3 illustrates a network connected to the computer system of FIG. 1 in accordance with an embodiment of the present invention. [Figure 4] FIG. 4 illustrates a source database system, a target database system, a first memory device, and a second memory device for performing data replication of the source database system in the target database system according to an embodiment of the present invention. [Figure 5] FIG. 5 illustrates tables in a source database system according to an embodiment of the present invention. [Figure 6] FIG. 6 illustrates a table of a target database system according to an embodiment of the present invention. [Figure 7] FIG. 7 illustrates multiple log files generated while performing data changes on tables in a source database system according to an embodiment of the present invention. [Figure 8] FIG. 8 illustrates a record of data changes generated while performing data changes on a table in a source database system according to an embodiment of the present invention. [Figure 9] FIG. 9 illustrates a first record of data changes and a second record of data changes according to an embodiment of the present invention. [Figure 10]FIG. 10 illustrates the source database system, the target database system, the first memory device, and the second memory device shown in FIG. 4 according to an embodiment of the present invention, where the first memory device and the second memory device store the plurality of log files shown in FIG. [Figure 11] FIG. 11 illustrates the source database system, the target database system, the first memory device, and the second memory device shown in FIG. 4 according to an embodiment of the present invention, wherein the first memory device and the second memory device store the plurality of log files shown in FIG. 7, and wherein each log file includes a log number. [Figure 12] FIG. 12 illustrates a flowchart of a computer-implemented method for replicating data changes in the table of the source database system shown in FIG. 5 into the table of the target database system shown in FIG. 6 according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] The description of various embodiments of the present disclosure has been presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0022] Hereinafter, a table in the source database system is referred to as a source table, and a table in the target database system is referred to as a target table. Each log file record generated while executing a data change in the source table may contain all the information necessary to replicate a single data change in the source table. The single data change in the source table may include a change in the value of a single data field in the source table, or changes in the values ​​of multiple data fields in a single row in the source table. The single data field may be identified by a row number and a column number in the source table. The value of the single data field may be a number, a string, or a combination thereof. The information necessary to replicate the single data change may include, for example, the new data value of the single data field in the source table after the data change is executed, and the row number and column number in the source table that identify the single data field. If the single data change may include changing the values ​​of multiple data fields in a single row of the source table, the information required to replicate the single data change may include the respective new data values ​​for each data field changed in performing the single data change, the row numbers containing these changed data fields, and respective columns indicating each changed data field.

[0023] Each time a new data change is performed for the source table, a new record of the data change may be generated that contains the necessary information to replicate the data change in the target table. The new record of the data change may be written in the actual log file. The new record is written in the actual log file at a position following the last-written record of the data change, where the last-written record indicates the last-performed data change for the source table. Thus, the order of the data changes for the source table may be represented by the order of the records of the data changes in the multiple log files.

[0024] A new log file may be generated when the actual log file size reaches a given threshold. In this manner, the log file may be generated repeatedly. Because the source table may be maintained for an extended period of time, e.g., years or decades, the total number of data changes in the source table may be greater than, e.g., thousands, hundreds, billions, etc. Because data fields may change several times during the life of the source table, the total number of data changes may be greater than the number of data fields or rows in the source table.

[0025] As used herein, the term "tracking a second number" refers to any type of determining the second number, such as counting, calculating, or updating a second counter representing the second number, or a combination thereof. Tracking the second number may include updating the second counter, whose value may represent the second number. The second counter may be updated if one of the second pending data changes has recently been replicated in the target table. In this case, the second counter may be decremented by one. Additionally, tracking the second number may include updating the second counter when a new log file may be stored on the second memory device. In this case, the second counter may be incremented by the number of records contained in the new log file stored on the second memory device.

[0026] As used herein, the term "archiving portions of a plurality of log files to a second memory device" refers to storing portions of a plurality of log files on a second storage device. The term "archiving" is used to express that the second access time is longer than the first access time. The first access time and the second access time may be defined by the first time and the second time it takes the first memory device or the second memory device to execute a write command or a read command after receiving the command, respectively. The second memory device may be a hard disk drive, a tape, or the like. The second memory device may include a moving element, such as a rotating disk or a tape, for accessing data stored on the second memory device. The second access time may include, for example, the time interval (known as seek time) used to position the read / write head of the second memory according to a read command. The first memory device may be a solid-state drive (SSD), a flash memory, or the like. The first memory device may not include a moving element for reading data stored on the first memory device.

[0027] Generally, to achieve high performance in replicating the data changes, it is advantageous to store all log files on the first memory device (e.g., shorter access time). However, the first memory device may be more expensive than the second memory device. Therefore, if the size of the first memory is limited and the available capacity of the first memory is close to zero, some of the log files stored on the first memory device may be archived on the second memory device.

[0028] Because the second access time (e.g., a second memory device, etc.) is longer than the first access time, replication of the data changes may be slowed for a longer period of time when a second number of second pending data changes exceeds a first level and the frequency of data changes performed in the source table exceeds a second level. In such a case, performing the first type of data replication (e.g., incremental data replication, etc.) may result in a deadlock situation in which more log files are archived on the second memory device within a given time interval than can be replicated. In such a situation, performing the second type of data replication (e.g., bulk data replication, etc.) may resolve or avoid the deadlock situation. According to an aspect of the present invention, by tracking the second number and selecting a type of data replication based at least in part on the second number, a deadlock situation or near-deadlock situation may be detected and prevented. In some embodiments, tracking the second number may be performed automatically by checking the actual value of the second number, or by checking the change in the actual value of the second number over time, or by checking a combination thereof.

[0029] The first and second levels may depend on the design of a computer system for performing the replication of the data changes, and in particular, on the second access time. The frequency of data changes performed in the source table may depend on the usage behavior of the source database system.

[0030] In some embodiments, the second type of data replication may be selected if the second number is greater than a second threshold. The second threshold may depend on the second access time or the bandwidth for replicating the data changes, or a combination thereof. Such embodiments may provide a fast method for performing data replication type selection because only one value needs to be compared. Moreover, because only one threshold, here the second threshold, needs to be adapted to the usage of the source database system, selecting a data replication type based on only one value may alleviate the need to calibrate a computer system for performing the data replication.

[0031] In some embodiments, determining the selection of the type of data replication may include tracking a first number of first pending data changes, where the first pending data changes are data changes in records of data changes in a plurality of log files stored on the first memory device that have not yet been replicated into the table of the target database system. Thus, in some embodiments, the selection may depend on the first number and the second number. In this way, the first pending data changes are also taken into account, and deadlock situations may be more accurately predicted.

[0032] As used herein, the term "tracking a first number" refers to any type of determining a first number, such as counting, calculating, or updating a first counter representing the first number, or a combination thereof. Tracking the first number may include updating the first counter, whose value may represent the first number. The first counter may be updated if one of the first pending data changes has recently been replicated in the target table. In this case, the first counter may be decremented by one. Additionally, tracking the first number may include updating the first counter when a new log file may be stored on the first memory device. In this case, the first counter may be incremented by the number of records contained in the new log file.

[0033] In some embodiments, loading the information for updating several rows of the table in the target database system may include loading several rows of the record of data changes together. In some embodiments, updating the rows of the table in the target database system may include replicating data changes of the rows of the table in the source database system in a table of the target database system according to the loaded record of data changes. Each of the rows of the table in the source database system may correspond to one of the rows of the table in the target database system. Some records of the records of data changes may be loaded from the first memory device or the second memory device. By loading some records of the records of data changes together, the data replication may be accelerated. Therefore, the occurrence of deadlocks may be prevented.

[0034] In some embodiments, loading the information for updating the rows of the table in the target database system may include loading data values ​​of the rows of the table in the source database system, which may also be referred to herein as a second case of the second type of data replication. Thus, in some embodiments, the record of data changes may not be necessary to perform the second type of data replication. The rows of the target table and the rows of the source table may be partitions of the target table and the source table, respectively. As an example, the rows of the target table and the source table may constitute the entire target table and the entire source table, respectively. The data values ​​of the rows of the source table may each provide the actual data value of a single data field of the rows of the source table. Once the data values ​​of the rows of the source table are loaded, these values ​​may be copied to the corresponding data field of the target table to update the target table. The data values ​​may be loaded directly from the source database system. By loading data values ​​of the multiple rows of the source table to update the corresponding data field in the target table, replication of multiple data changes to one of the single data fields of the source table can be skipped. In this way, the data replication can be accelerated and deadlock situations can be prevented.

[0035] In some embodiments, selecting the type of data replication may include comparing a first computational cost required to perform all pending data changes (e.g., the first pending data change and the second pending data change) using the first type of data replication with a second computational cost required to perform updates of all rows of the target table according to a second case of the second type of data replication. If the first computational cost is greater than the second computational cost, the second type of data replication may be selected; otherwise, the first type of data replication may be selected.

[0036] In some embodiments, determining the type of data replication to select may include selecting the second type of data replication if the first number is greater than a first threshold and the second number is greater than a second threshold. Such an embodiment may offer a simple implementation for considering the first pending data changes and the second pending data changes. If the first number is greater than a first threshold and the second number is greater than a second threshold, it may be estimated that a deadlock situation may soon be reached. Similar to the second threshold, the first threshold may be adapted to the usage of the source database system and the achievable bandwidth for replicating the data changes.

[0037] In some embodiments, selecting the type of data replication may include weighting the first number and the second number. As an example, the weighting of the first number may depend on the first access time, and the weighting of the second number may depend on the second access time. By weighting the first number and the second number with respect to the respective access times, the difference between the first access time and the second access time may be taken into account. This may alleviate the need to adapt the method of replicating data changes to the design of the computer system. For example, the longer the second access time, the more the second number may be weighted depending on the second access time, which may generate a key figure by which the selection of the type of data replication may be performed. The weighting may be used to determine the first computational cost described above.

[0038] In some embodiments, selecting a type of data replication may include selecting the first type of data replication if the ratio of the first number to the second number is greater than or equal to a third threshold. Accordingly, in some embodiments, selecting may include selecting the second type of data replication if the ratio of the first number to the second number is less than a third threshold. The ratio may be in the form of a quotient, with the first number being the numerator and the second number being the divisor. The third threshold may be adapted to a further ratio between the first access time and the second access time. By comparing the ratio of the first number to the second number with the third threshold, the difference between the first access time and the second access time may be taken into account in a simple manner. The ratio may be used, for example, as a single primary indicator, based on which selecting the type of data replication may be performed. The computer system may be adapted to the usage status of the source database system by adjusting the third threshold.

[0039] In some embodiments, the ratio may be a weighted ratio, with the first number being weighted depending on the first access time and the second number being weighted depending on the second access time. Because the weighted ratio takes into account the first access time and the second access time, it may more accurately serve as a primary indicator based on which the selection of the type of data replication may be performed.

[0040] In some embodiments, selecting the type of data replication may include calculating a weighted sum of the first number and the second number, where the first number is weighted based on the first access time and the second number is weighted based on the second access time. Thus, in some embodiments, selecting may include selecting the first type of data replication if the weighted sum is less than a fourth threshold, or selecting the second type of data replication if the weighted sum is greater than or equal to a fourth threshold. The weighted sum may represent the amount of total access time required to access all records on the first storage device and the second storage device to perform all first pending and second pending data modifications. Therefore, the weighted sum may be sufficient as a key indicator for estimating whether a deadlock situation may soon occur. Therefore, the weighted sum may be sufficient as a key indicator based on which the selection of the type of data replication may be performed. The weighted sum may be used to determine the first computational cost described above.

[0041] In some embodiments, the second type of data replication may include loading information for updating all rows of the table in the target database system together, and updating all rows of the table in the target database system based on the information for updating all rows of the target table. The information for updating all rows of the target table may be in the form of all records of first and second pending data changes in the first case, and in the form of all data values ​​of the source table in the second case. Thus, in the first case, updating all rows of the target table may include performing replication of all data changes according to the first and second pending data changes. In the second case, updating all rows of the target table may include setting each data value in the target table equal to a corresponding one of the data values ​​in the source table.

[0042] In some embodiments, after all rows have been updated, all log files may be deleted from the first memory device and the second memory device. For example, after such a type of cleanup of the first memory device and the second memory device, it may be estimated that some time will pass before further records of data changes stored on the first memory device are archived on the second memory device. Thus, some embodiments may provide for shifting a computer system from a state in which the records of the data changes are continuously loaded from the second memory device to perform replication of the data changes to a state in which the records of the data changes are continuously loaded from the first memory device to perform the replication. As a result, the performance of the replication may be improved.

[0043] In some embodiments, selecting the type of data replication may include using a respective log number. For example, a respective log number may be assigned to each of the plurality of log files. The respective log number may indicate the total number of records of data changes in the table of the source database system written at each moment when generation of the respective log file began. In some embodiments, the log numbers assigned to the plurality of log files may be used when tracking the first number and the second number. This may reduce the computational cost of determining the first number, the second number, or a combination thereof. For example, it may not be necessary to read all records of the plurality of log files stored on the second memory device to count these records. The same applies to determining the second number.

[0044] In some embodiments, the plurality of log files each include a respective log number assigned thereto. For example, the log number assigned to each log file may be embedded within the respective log file. This may provide the advantage that the log numbers and their assignment to each log file do not need to be stored on an additional memory device different from the first memory device and the second memory device.

[0045] In some embodiments, tracking the first number may include calculating the difference between the log number assigned to the log file stored on the first memory device for the shortest time and the log number assigned to the log file stored on the first memory device for the longest time. The first number may be set to the difference between multiple log numbers. This calculation may, in some embodiments, provide a simple and fast way to determine the first number. The log file stored on the first memory device for the longest time may also be referred to herein as the oldest log file. The log file stored on the first memory device for the shortest time may also be referred to herein as the youngest log file.

[0046] In some embodiments, tracking the first number may include calculating the difference between the actual total number of data changes for the table in the source database system at the actual moment and the log number assigned to the log file that has been stored on the first memory device for the longest time. The actual total number of data changes may be obtained from the source database system. The source database system may include a record counter for counting the actual total number of data changes. Using the actual total number of data changes instead of the log number of the youngest log file may enable more accurate determination of the first number. Generally, the last record being generated by the source database system (e.g., referred to herein as the last generated record) may be written directly to the youngest log file.

[0047] In some embodiments, selecting a type of data replication may include tracking a record number of the last record of data change read from the first memory device or the second memory device and used to replicate data changes for the table in the target database system according to the last record of data change (e.g., also referred to as the last processed record or the last record of data change). In some embodiments, selecting a type of data replication may include using the record number of the last record of data change to track a first number or a second number. The record number of the last processed record may represent the total number of records of data changes (including the last processed record) for the table in the source database system that have been written at the moment the last processed record was written. Each record of data change may include a record number equal to the total number of records of data changes (including that record) for the table in the source database system that have been written at the moment the record was written. Tracking the record number of the last processed record may facilitate tracking of the first number, the second number, or a combination thereof. For example, in such a case, the first and second counters described above are not needed and may be updated. Tracking the record number of the last processed record may include reading the record number of the last processed record.

[0048] In some embodiments, selecting a type of data replication may include calculating the second number as the difference between a log number assigned to a log file that has been stored the longest on the first memory device and a record number assigned to the last record of a data change. In some embodiments, the last record of a data change may be included in or provided by one of the log files archived on the second memory device. Such an embodiment may provide a simple way to determine the second number without using the first counter or the second counter, but instead by simply using the record number of the last processed record and the log number assigned to the oldest log file.

[0049] In some embodiments, selecting a type of data replication may include calculating the first number as the difference between a log number assigned to a log file that has been stored on the first memory device for the shortest time and a record number assigned to the last record of a data change. In some embodiments, the last record of a data change may be included in or provided by one of the multiple log files stored on the first memory device. Such an embodiment may provide a simple way to determine the first number without using the first counter or the second counter, but instead by simply using the record number of the last processed record and the log number assigned to the youngest log file.

[0050] In some embodiments, selecting a type of data replication may include restoring the plurality of log files. The restoring may include storing the plurality of log files archived in the second memory device on the first memory device. The restored log files may be read and processed from the first memory device, rather than from the second memory device, to perform the replication of the data changes. Thus, the replication of the data changes may be accelerated. The restoring may be performed based on an estimate of the computational cost of restoring the plurality of log files.

[0051] In some embodiments, selecting a type of data replication may include performing a restoration of the plurality of log files based on a size of each of the plurality of records of data changes in the plurality of log files archived on the second memory device. Estimating the computational cost of restoring the plurality of log files may be performed based on a size of each of the plurality of records of data changes in the plurality of log files archived on the second memory device. In this manner, a determination of whether to perform a restoration of the plurality of log files may be made based on more detailed information regarding the computational cost of restoring the plurality of log files.

[0052] In some embodiments, selecting a type of data replication may include using a counter to count the total number of records of data changes for the table in the source database system. The counter may be the record counter described above. In some embodiments, the selecting may include determining whether the counter is close to overflowing. In some embodiments, the second type of data replication may be selected if the counter is close to overflowing. The counter may be reset after replicating data changes for the table in the source database system into the table in the target database system according to the second type of data replication using the multiple records of data changes in the multiple log files. In some cases, determining that the counter may be close to overflowing may prevent the counter from overflowing.

[0053] In some embodiments, the second type of data replication may include loading all records of data changes stored on the first memory device and the second memory device together, and replicating multiple data changes of multiple rows of the table of the source database system into the table of the target database system according to the loaded all records of data changes. By loading all records of data changes stored on the first memory device and the second memory device together, the need to recalculate the log numbers of the multiple log files stored on the first memory device and the second memory device may be avoided. After loading all records of data changes stored on the first memory device and the second memory device, these records may be deleted from the first memory device and the second memory device.

[0054] Embodiments of the present invention may be practiced using computing devices, which may also be referred to as computer systems, clients, servers, etc. Referring now to FIG. 1, a schematic diagram of an example computer system 10 is shown. Computer system 10 provides only one example of a suitable computer system and is not intended to suggest any limitations on the scope of use or functionality of the embodiments of the present invention described herein, or on their combinations. Nevertheless, computer system 10 is capable of implementing or performing, or both, any of the functions described herein.

[0055] Computer system 10 may comprise a first computer system / server 12 that may be operable with numerous other general-purpose or special-purpose computing system environments or configurations, or combinations thereof. Examples of well-known computing systems, environments, or configurations, or combinations thereof, suitable for use with first computer system / server 12 include, but are not limited to, first memory 411 of computer system 10 (e.g., as shown in FIG. 4), personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, distributed computing environments comprising any of the above-listed systems or devices, and the like.

[0056] The first computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. By way of example, the first computer system / server 12 may be practiced in a distributed computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer system storage media, including, for example, memory storage devices.

[0057] As shown in FIG. 1 , a first computer system / server 12 in computer system 10 is shown in the form of a general-purpose computing device. Components of first computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 connecting various system components, including the system memory 28, to processor 16. Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0058] The first computer system / server 12 typically includes a variety of computer system-readable media, which may be any available media that can be accessed by the first computer system / server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

[0059] The system memory 28 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 or cache memory 32, or a combination thereof. The first computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a floppy disk) and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 18 by one or more data media interfaces. As further depicted and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0060] The program / utility 40 having a set (at least one) of program modules 50 may be stored in the memory 28, by way of example and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may comprise an implementation of a network environment. The program modules 50 may generally be configured to perform the functions or methodologies, or combinations thereof, of embodiments of the present invention described herein.

[0061] As used herein, the term "program" or "program module" refers to a set of instructions that may include commands for causing actions to be executed by processor 16 when processor 16 can read the commands. The set of instructions may be in the form of a computer-readable program, routine, subroutine, or part of a library, and may be executed by processor 16, or called by further programs being executed by processor 16, or executed by processor 16 and called by further programs being executed by processor 16. Preferably, program module 50 may be an executable program that is compiled according to the type of hardware platform of first computer system / server 12.

[0062] The first computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, one or more devices that allow a user to interact with the first computer system / server 12, or any device (e.g., a network card, modem, etc.) that allows the first computer system / server 12 to communicate with one or more other computer devices, or a combination thereof. Such communication may occur via one or more input / output (I / O) interfaces 22. Additionally, the first computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 20. As shown, the network adapter 20 may communicate with other components of the first computer system / server 12 via a bus 18. Although not shown, it should be understood that other hardware or software components, or combinations thereof, may be used with the first computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0063] As an example, the first computer system / server 12 may be designed in the form of a source database system, e.g., source database system 401, shown in FIG. 4. The source database system 401 may include a source table 410, as shown in FIGS. 4 and 5. The source table 410 may include, for example, data fields, shown as boxes in FIG. 5. Each of the data fields may include an entry, e.g., a numeric value or a string, or a combination thereof. Each data field in the source table 410 may be assigned a specific row number in the source table 410 and a specific column number in the source table 410. The specific row numbers in the source table 410 may increase in the direction of a first arrow 501 shown in FIG. 5. The specific column numbers in the source table 410 may increase in the direction of a second arrow 502 shown in FIG. 5. The row number and column number of each data field may be used to perform data modifications of the respective data field. Additionally, a selected row number of source table 410 and multiple different selected column numbers of source table 410 may be used to perform multiple data modifications of different data fields of a single row identified by the selected row number. As an example, source table 410 may be stored in storage system 34 of first computer system / server 12.

[0064] 2, which may be operable with numerous other general-purpose or special-purpose computing system environments or configurations, or combinations thereof. Examples of well-known computing systems, environments, or configurations, or combinations thereof, suitable for use with second computer system / server 212 include, but are not limited to, first memory 411, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, distributed computing environments comprising any of the above-listed systems or devices, and the like.

[0065] The second computer system / server 212 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The second computer system / server 212 may be practiced in a distributed computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer system storage media, including, for example, memory storage devices.

[0066] 2, second computer system / server 212 in computer system 10 is shown in the form of a general-purpose computing device. Components of second computer system / server 212 may include, but are not limited to, one or more processors or processing units 216, a system memory 228, and a bus 218 connecting various system components, including the system memory 228, to processor 216. Bus 218 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0067] The second computer system / server 212 typically includes a variety of computer system-readable media. Such media can be any available media that can be accessed by the second computer system / server 212 and includes both volatile and nonvolatile media, removable and non-removable media.

[0068] The system memory 228 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 230 or cache memory 232, or a combination thereof. The second computer system / server 212 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 234 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a floppy disk) and an optical disk drive for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media, may be provided. In such cases, each may be connected to the bus 218 by one or more data medium interfaces. As further depicted and described below, the memory 228 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0069] Programs / utilities 240 having a set (at least one) of program modules 250 may be stored in memory 228, by way of example and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may comprise an implementation of a networking environment. Program modules 250 may generally be configured to perform the functions or methodologies, or combinations thereof, of embodiments of the present invention described herein.

[0070] As used herein, the term "program" or "program module" refers to a set of instructions that may include commands for causing actions to be executed by processor 216 when processor 216 can read the commands. The set of instructions may be in the form of a computer-readable program, routine, subroutine, or part of a library, and may be executed by processor 216, or called by further programs being executed by processor 216, or executed by processor 216 and called by further programs being executed by processor 216. Preferably, program module 250 may be an executable program that is compiled according to the type of hardware platform of second computer system / server 212.

[0071] The second computer system / server 212 may also communicate with one or more external devices 214, such as a keyboard, pointing device, display 224, one or more devices that allow a user to interact with the second computer system / server 212, or any device (e.g., a network card, modem, etc.) that allows the second computer system / server 212 to communicate with one or more other computer devices, or a combination thereof. Such communication may occur via one or more input / output (I / O) interfaces 222. Additionally, the second computer system / server 212 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 220. As shown, the network adapter 220 may communicate with other components of the second computer system / server 212 via a bus 218. It should be understood that, although not shown, other hardware or software components, or combinations thereof, may be used with the second computer system / server 212. Examples include, but are not limited to, microcode, device drivers, redundant processing devices, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

[0072] As an example, the second computer system / server 212 may be designed in the form of a target database system, such as the target database system 402, shown in FIG. 4. The target database system 402 may include a target table 420, as shown in FIGS. 4 and 6. The target table 420 may include data fields. Each data field in the target table 420 may include an entry, such as a numeric value or a string, or a combination thereof. Each data field in the target table 420 may be assigned a specific row number in the target table 420 and a specific column number in the target table 420. The specific row numbers in the target table 420 may increase in the direction of a first arrow 601 shown in FIG. 6. The specific column numbers in the target table 420 may increase in the direction of a second arrow 602 shown in FIG. 6. The row number and column number of each data field may be used to perform data modifications of the respective data field. Additionally, a selected row number of target table 420 and multiple different selected column numbers of target table 420 may be used to perform multiple data modifications of different data fields of a single row identified by the selected row number. As an example, target table 420 may be stored in storage system 234 of second computer system / server 212.

[0073] In some embodiments, a computer system, such as the computer system 10 shown in FIG. 1 or FIG. 2 or FIG. 1 and FIG. 2, may be used to perform the operations disclosed herein, such as at least the first, second, third, fourth, fifth, sixth, seventh, and eighth operations described below.

[0074] In some embodiments, computer system 10 may be configured to replicate data changes in source table 410 into target table 420 .

[0075] In some embodiments, the first operation may include repeatedly generating multiple log files while executing data changes to the source table 410. As an example, a set of generated log files 430 is shown in FIG. 7. The processor 16 may execute each data change to the source table 410 by initiating a write on the storage system 34 where the source table 410 may be stored. Each log file 430i may include at least one record of the data change to the source table 410. The first operation may be executed by means of the source database system 401, such as by means of the processor 16.

[0076] In some embodiments, processor 16 may perform a new data change on source table 410 and create a new record 440 for the data change that contains the necessary information, such as the necessary information described above, to replicate the new data change on source table 410 into target table 420. n Therefore, for each new data change, a new record 440 of the data change may be generated. n may be generated. Additionally, multiple log files may be generated iteratively while performing data changes. In some embodiments, processor 16 may also perform additional data changes for additional source tables not shown in the figures. Moreover, processor 16 may generate additional records of data changes that contain information necessary to replicate the additional data changes for the additional source tables. The additional records may be written to additional log files not shown, or multiple log files 430, or a combination thereof. Thus, in some embodiments, multiple log files 430 may contain one or more additional records of data changes not shown. In some embodiments, one or more of the additional log files may be stored among multiple log files 430.

[0077] In some embodiments, the new record 440 nThe information needed to replicate new data changes is stored in the actual log file 430. m For each further new data change, a further new record of the data change can be created by writing into new record 440. n 4. In this way, for several new data changes, several new records of the data change may be generated. Figure 8 shows a set of generated records 440 of the data change that the source database system 401 may have generated. Each record 440 of the data change i Each record of data change i 4401, the source database system 401 may generate a first record 4401 of the data change, followed by a second record 4402 of the data change. The last record generated may be a new record 4403. n It is possible.

[0078] In some embodiments, the second operation may include storing a plurality of log files 430 on the first memory device 411. In some embodiments, storing the log files 430 may be performed by, for example, the additional processor 450. The source database 401 may be connected to the first memory device or the additional processor 450, or a combination thereof, via an (I / O) interface 22 or via a network adapter 20. The first memory device 411 may include or provide a first access time. As an example, the first access time may be in the range of 35 to 100 microseconds. FIG. 4 illustrates an actual log file 430. m Multiple log files containing 430 i may be generated directly on the first memory device 411. In some embodiments, this may be performed, for example, by sending a write command from the source database system 401 to the further processor 450 of the computer system 10.

[0079] However, in some embodiments, the log file 430i may be written internally within the source database system 401 and then transmitted from the source database system 401 to the first memory device 411. Writing new records 440n directly onto the first memory device 411, without storing the new records 440n internally within the source database system 401, may provide the advantage that the target database system 402 may read the new records 440n as soon as possible. i The plurality of log files 430 i The actual log file 430 may be stored on the first memory device 411 after or during the writing of the log file 430, or a combination thereof. m If the size of the actual log file 430 exceeds a size threshold for a given log file, a further processor 450 m , and may generate a new actual log file (not shown in FIG. 4). m If the actual log file 430 is locked, the processor 16 m If the log file is updated, it may no longer be possible to write one of the further new records of data changes in the new log file. Instead, processor 16 may write one of the further new records in the new actual log file.

[0080] In some embodiments, the third operation may include archiving a portion of the plurality of log files 430 stored on the first memory device 411 onto the second memory device 412. The second memory device 412 may include or provide a second access time. The second access time may be longer than the first access time, for example, the second access time may be approximately 100 to 200 times longer than the first access time. In some embodiments, the further processor 450 may perform the third operation. The further processor 450 may load a portion of the plurality of log files from the first memory device 411 and archive it onto the second memory device 412. Additionally, the further processor 450 may delete the archived portion of the plurality of log files 430 on the first memory device 411. In this manner, space on the first memory device 411 is freed up for the actual log files 430. m A new log file may be created to allow more new records to be written in the log file, or to create a new actual log file, or to do a combination of both.

[0081] In some embodiments, the additional processor 450, the first memory device 411, and the second memory device 412 may be considered a replication engine.

[0082] In some embodiments, in a fourth operation, computer system 10 may provide for performing the first type of data replication, e.g., the fourth operation may be performed via processor 216, thereby performing the first type of data replication. The first type of data replication may include: i and the individually loaded corresponding records 440 of the data changes. i9 may include replicating (e.g., incrementally replicating, etc.) each data change of a single row of source table 410 into target table 420 according to the following: For example, a first record 4401 of data changes shown in FIG. 9 may include a first value of a row number (in this example, "2") and a first value of a column number (in this example, "1") in source table 410 for a changed data field identified by these values, and a new value for this data field (in this example, "4812"). A second record 4402 of data changes may include a second value of a row number (in this example, "1") and a second value of a column number (in this example, "2") in source table 410 for a changed data field identified by these values, and a new value for this data field (in this example, "9001").

[0083] Each record 440 i Each record 440 i Record number field 441 contains the record number of i , which includes each record 440 i The table may indicate the total number of records of data changes in the source table 410 that are being written at each moment of generation.

[0084] According to the first type of data replication, the target database system 402 may load the first record 4401 from the first memory device 411 or the second memory device 412, and then replicate the data change of the second row of the source table 410 in the target table 420 according to the first record 4401. As an example, the processor 216 may provide setting the value of the data field in the target table 420 identified by the row number “2” and the column number “1” equal to 4812. Moreover, according to the first type of data replication, after performing the replication of the data change in the target table 420 according to the first record 4401, the target database system 402 may load the second record 4402 from the first memory device 411 or the second memory device 412, and then replicate the data change of the first row of the source table 410 in the target table 420 according to the second record 4402. For example, processor 216 may be provided to set the value of a data field in target table 420 identified by row number "1" and column number "2" equal to 9001. Thus, in some embodiments, each time one of first record 4401 and second record 4402 is loaded from first memory device 411 or second memory device 412, further processor 450 may access first memory device 411 or second memory device 412. For example, in this case, first memory device 411 or second memory device 412 may be accessed twice to load first record 4401 and second record 4402, or first memory device 411 and second memory device 412 may each be accessed once.

[0085] In some embodiments, in a fifth operation, computer system 10 may provide for performing the second type of data replication, e.g., the fifth operation may be performed via processor 216 to perform the second type of data replication. The second type of data replication may include loading information to update multiple rows of target table 420 together (e.g., bulk replication, etc.). In some embodiments, this information may include multiple records 440 of data changes. i Some records, e.g., multiple records 440 i Two of the multiple records 440 i 100 of the records 440, all of the records 440, etc. In this way, multiple records 440 of data changes may be represented. i Several records of the data change may be loaded together from the first memory device 411 or the second memory device 412. Furthermore, data changes in multiple rows of the source table 410 may be stored in the loaded multiple records 440 of the data change. i As an example, the processor 216 may load a first record 4401 and a second record 4402 together from the first memory device 411 or the second memory device 412, and then perform a first data modification in the target table 420 according to the first record 4401 and a second data modification in the target table 420 according to the second record 4402. Loading multiple records together may include accessing the first memory device 411 or the second memory device 412 only once and reading the multiple records consecutively starting from a read position that may be achieved by the single access of the first memory device 411 or the second memory device 412.

[0086] In some embodiments, in this second type of data replication, the information for updating multiple rows in target table 420 may be in the form of data values ​​in source table 410, such as a partition of data values ​​in the source table or all of the data values ​​in the source table. As an example, a partition of data values ​​in source table 410 may represent a month if source table 410 is ordered by date. The data values ​​may be loaded directly from source database system 401, such as from storage system 34. The data values ​​of source table 410 may be loaded together from source database system 401 to target database system 402, such as by further processor 450, in the form of a matrix, which may have multiple dimensions. In some embodiments, data change records 440 may not be used in this second type of data replication.

[0087] 10 is an exemplary implementation in which a first record 4401 and a second record 4402 are stored on a second memory device 412. The total number of log files 430 stored on the first memory device 411 and the second memory device 412 may be equal to m. The number of log files 430 stored on the first memory device 411 may be equal to l.

[0088] The mth log file 430 among multiple log files 430 m may be the youngest log file in the first memory device 411, and may be the (m-l+1)th log file 430 among the plurality of log files 430. m-l+1 may be the oldest log file in the first memory device 411. The number of the plurality of log files 430 stored on the second memory device 412 may be equal to ml. The (ml)-th log file 430 among the plurality of log files 430 may be m-lmay be the youngest log file in the second memory device 412. The first log file 4301 of the plurality of log files 430 may be the oldest log file in the second memory device 412. The records of the plurality of log files 430 are shown as boxes in FIG.

[0089] In some embodiments, the sixth operation may include tracking a second number of second pending data modifications. The sixth operation may be performed by, for example, the further processor 450. The second pending data modifications may include records 440 of data modifications in the plurality of log files 430 archived on the second memory device 412 that have not yet been replicated into the target table 420. i The second number may be performed by calculating the number of second pending records. In some embodiments, the record number of the last record of a data change (e.g., referred to as the last record of a data change) that is read from the first memory device 411 or the second memory device 412 and used for replicating the data change in the target table 420 may be tracked. For example, if the last record of a data change is the second record 4402, the record number of the last record of a data change may be equal to "2." The record number of the last record of a data change may be tracked by reading the value of the second record number field 4412 of the second record 4402. In some embodiments, the second number may be calculated ... m-l In some embodiments, the second number may be determined by counting the records stored on the plurality of log files in second memory device 412 from the last record of data changes.

[0090] In some embodiments, a seventh operation may include selecting a type of data replication from the first type of data replication and the second type of data replication based on the second number. For example, if the second number is greater than a second threshold, the second type of data replication may be selected, such as by processor 216. Otherwise, the first type of data replication may be selected (e.g., if the second number is not greater than a second threshold). The seventh operation may be performed by a further processor 450, processor 216, or the like.

[0091] In some embodiments, a first number of first pending data changes may be tracked, where the first pending data changes may be a number of records 440 of data changes in a number of log files 430 stored on the first memory device 411 that have not yet been replicated into the target table 420. i In some embodiments, if the first number and the second number are tracked, the selecting may be based on the first number and the second number. In some embodiments, the first number is calculated by subtracting the record number of the last record of a data change from the youngest log file 430 in the first memory device 411. m This may be the case when the last record of a data change is read from the first memory device 411.

[0092] In some embodiments, selecting the type of data replication may include performing a first number of weightings and a second number of weightings, for example, the first number of weightings may be based on a first access time (e.g., access_time1) and the second number of weightings may be based on a second access time (e.g., access_time2).

[0093] As an example, the weighted sum S of the first number n1 and the second number n2 may be calculated according to the following formula:

number

[0094] In some embodiments, selecting the type of data replication may be performed such that the first type of data replication is selected if the weighted sum S is less than a fourth threshold, or such that the second type of data replication is selected if the weighted sum S is greater than or equal to the fourth threshold. In some embodiments, as an example, the fourth threshold may represent an estimated time to perform updates of all rows in the target table 420 according to the second type of data replication. In another example, in some embodiments, the fourth threshold may be determined based on past experience when a deadlock situation or a near-deadlock situation is reached. For example, the fourth threshold may be determined by calculating a further weighted sum according to the above-described formula, but instead of using the first number n1 and the second number n2, the fourth threshold may be determined using the further numbers of the first pending and the second pending data changes at the moment when a deadlock situation or a near-deadlock situation is reached. In some embodiments, the fourth threshold may then be determined by multiplying the further weighted sum by a value between 0.1 and 0.9 to prevent a deadlock situation.

[0095] In some embodiments, an eighth operation is to retrieve a plurality of records 440 of data changes in a plurality of log files 430. i The eighth operation may include replicating data changes in source table 410 into target table 420 according to a selected type of data replication using the data replication function. The eighth operation may be performed by processor 216, etc.

[0096] In some embodiments, computer system 10 may be a stand-alone computer without network connectivity that may receive data to be processed via a local interface, although such operations may likewise be performed using a computer system connected to a network, such as a communications network or a computing network, or a combination thereof.

[0097] 3 illustrates an exemplary computing environment in which a computer system, e.g., computer system 10, is connected to a network 200, e.g., using a network adapter 20. Network 200 may be, but is not limited to, a communications network, e.g., the Internet or other public network, a wide area network (WAN), a local area network (LAN), a wireless network (e.g., a mobile communications network), etc. Network 200 may also include a computing network, e.g., a cloud computing network. Computer system 10 may receive data to be processed from network 200, or may provide computing results to another computing device connected to computer system 10 via network 200, or may perform a combination thereof.

[0098] In some embodiments, computer system 10 may perform, in whole or in part, the operations described herein, e.g., first operation, second operation, third operation, fourth operation, fifth operation, sixth operation, seventh operation, and eighth operation, in response to a request received over network 200. In particular, computer system 10 may perform such operations in a distributed computation manner with one or more additional computer systems that may be connected to computer system 10 via network 200. To do so, computer system 10 or any additional participating computer systems may use network 200 to access additional computational resources, e.g., dedicated memory or shared memory.

[0099] FIG. 11 shows an example of a computer system 10, where each log file 430 i As shown in the circle in Figure 11, each log number 431 i In some embodiments, each log number 431 i are the respective log files 430 i Each log number 431 may indicate the total number of records of data changes in the source table 410 that have been generated up to the moment the generation of each log number 431 is started. i are the respective log files 430 i In the following example, there are two log files stored on the first memory device 411, namely, a first log file 4301 and a second log file 4302, and two log files, namely, a third log file 4303 and a fourth log file 4304. m , may be assumed to be stored on the second memory device 412. Thus, following this example, m may be equal to "4" and l may be equal to "2". The log number 4311 of the first log file 4301 may be equal to "10", and the log number 4312 of the second log file 4302 may be equal to "32", and the log number 4313 of the third log file 4303 (e.g., 431 m-l+1) may be equal to "42", and the log number 4314 (e.g., 431 m ) may be equal to "112". The fourth log file 4304 is the actual log file 430 in this example. m may be.

[0100] The first number is the youngest log file 430 in the first memory device 411. m The log number assigned to the log file, i.e., log number 431 in this case. m and the oldest log file 430 in the first memory device 411. m-l+1 Log number 431 assigned to m-l+1 The first number may be determined in such a manner if the last record of the data change is stored on the second memory device 412.

[0101] In some embodiments, the first number is the sum of the actual total number of data changes in the source table 410 at the actual moment and the oldest log file 430 in the first memory device 411. m-l+1 Log number 431 m-l+1 In some embodiments, the source database system 401 may include a counter to count the actual total number of data changes in the source table 410, and may send the actual total number of data changes to a further processor 450, the target database system 402, etc.

[0102] Additionally, in some embodiments, log number 431 i The second number is calculated by using the oldest log file 430 in the first memory device 411. m-l+1 Log number 431 m-l+1 and the record number of the last record of the data change, where the last record of the data change may be stored in one of a plurality of log files 430 archived on the second memory device 412.

[0103] Additionally, in some embodiments, log number 431 i The youngest log file 430 in the first memory device 411 is then retrieved using m Log number 431 m The first number may be calculated as the difference between the record number of the last record of the data change and the record number of the last record of the data change, where the last record of the data change is one of the records 440 in the plurality of log files 430 stored on the first memory device 411. i It can be one of the following:

[0104] FIG. 12 illustrates a flowchart of a computer-implemented method for replicating data changes in a source table 410 into a target table 420 in accordance with an embodiment of the present invention.

[0105] In step 1201, while executing the data change in the table 410 of the source database system 401, a plurality of log files 430 are read. i are generated repeatedly, where each log file contains at least one record of the data change in table 410 of source database system 401.

[0106] In step 1202, a plurality of log files 430 i is stored on a first memory device 411, where the first memory device 411 has a first access time.

[0107] In step 1203, a plurality of log files 430 stored on the first memory device 411 are i is archived on a second memory device 412, where the second memory device 412 has a second access time, the second access time being longer than the first access time.

[0108] In step 1204, a first type of data replication may be provided.

[0109] In step 1205, a second type of data replication may be provided.

[0110] In step 1206, a second number indicating a second number of pending data changes may be tracked.

[0111] In step 1207, a type of data replication may be selected from the first type of data replication and the second type of data replication depending on the second number or the first number or a combination thereof.

[0112] In step 1208, data changes in the source table 410 of the source database system 401 are recorded in a plurality of log files 430. i Data changes for multiple records 440 i , data may be replicated into the target table 420 of the target database system 402 according to the selected type of data replication.

[0113] The present invention may be a system, method, or computer program product, or combination thereof, at any level of technical detail that may be integrated. The computer program product may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0114] The computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction-execution device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer diskette (登録商標), hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded device such as punch cards or raised structures in grooves on which instructions are recorded, or any suitable combination thereof. As used herein, the computer-readable storage medium should not be construed to be a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0115] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing device / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may be comprised of copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer-readable program instructions from the network and transmits the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing device / processing device.

[0116] The computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, such as object-oriented programming languages, e.g., Smalltalk, C++, etc., or conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any kind of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0117] Aspects of the present invention are described herein with reference to flowchart illustrations or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products or computer programs according to embodiments of the invention. It will be understood that each block of the flowchart illustrations or block diagrams, or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams, or combinations thereof, can be implemented by computer-readable program instructions.

[0118] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts identified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof, to produce a machine. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer-programmable data processing apparatus or other device, or a combination thereof, to function in a particular manner, such that a computer-readable storage medium having stored instructions includes an article of manufacture including instructions that implement aspects of the functions / acts identified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof.

[0119] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable data processing apparatus, or other device, implement the functions / acts identified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof, to cause the computer, other programmable apparatus, or other device to perform a series of operating steps to generate a computer-implemented process.

[0120] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be accomplished as a single step performed simultaneously, substantially simultaneously, partially, or fully in a time-overlapping manner, depending on the functionality involved, or the blocks may be performed in the reverse order. It should be noted that each block of the block diagrams or flowchart diagrams or combinations thereof, and combinations of multiple blocks in the block diagrams or flowchart diagrams or combinations thereof, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations, or may execute a combination of special-purpose hardware and computer instructions.

[0121] In the following, the invention will be restated by a list of clauses highlighting some possible non-exclusive combinations of the features disclosed herein.

[0122] Item 1: A computer-implemented method for replicating data changes of a table in a source database system into a table in a target database system, the method including: repeatedly generating a plurality of log files while performing data changes of the table in the source database system, where each log file includes at least one record of the data changes of the table in the source database system; storing the plurality of log files on a first memory device, where the first memory device has a first access time; archiving a portion of the plurality of log files stored on the first memory device on a second memory device, where the second memory device has a second access time, the second access time being longer than the first access time; providing a first type of data replication, where the first type of data replication individually and repeatedly loads one of the plurality of records of data changes, and replicating a single record of the table in the source database system according to the individually loaded corresponding record of data changes. The method includes replicating each data change of a row in the table of the target database system within the target database system; providing a second type of data replication, where the second type of data replication includes loading information for updating multiple rows of the table of the target database system together and updating the multiple rows of the table of the target database system based on the information; tracking a second number of second pending data changes, where the second pending data changes are the data changes of the records of data changes of the multiple log files archived on the second memory device that have not yet been replicated into the table of the target database system; selecting a type of data replication from the first type of data replication and the second type of data replication depending on the second number; and replicating the data changes of the table of the source database system within the table of the target database system according to the selected type of data replication using the records of data changes of the multiple log files.

[0123] Item 2: The method of Item 1, further comprising tracking a first number of first pending data changes, wherein the first pending data changes are the plurality of data changes of the records of the data changes of the plurality of log files stored on the first memory device that have not yet been replicated into the table of the target database system, and wherein the selecting is dependent on the first number and the second number.

[0124] Item 3: The method described in Item 1 or 2, wherein loading information for updating multiple rows of the table of the target database system together includes loading multiple rows of the record of data changes together, and wherein updating the multiple rows of the table of the target database system includes replicating data changes of the multiple rows of the table of the source database system into the table of the target database system according to the loaded record of data changes, and wherein each of the multiple rows of the table of the source database system corresponds to one of the multiple rows of the table of the target database system.

[0125] Item 4: The method of items 1 or 2, wherein loading the information for updating the plurality of rows of the table of the target database system includes loading data values ​​of the plurality of rows of the table of the source database system.

[0126] Clause 5: The method of any one of clauses 1 to 4, wherein selecting the type of data replication includes weighting the first number and weighting the second number, wherein the weighting of the first number is based on the first access time, and the weighting of the second number is based on the second access time.

[0127] Clause 6: The method of any one of clauses 1 to 5, further comprising: selecting the first type of data replication if the ratio of the first number to the second number is greater than or equal to a third threshold; and selecting the second type of data replication if the ratio of the first number to the second number is less than a third threshold, wherein the ratio is a weighted ratio including the first number being weighted based on the first access time and the second number being weighted based on the second access time.

[0128] Clause 7: The method of any one of clauses 1 to 6, wherein selecting the type of data replication includes calculating a weighted sum of the first number and the second number, where the first number is weighted based on the first access time and the second number is weighted based on the second access time; and selecting the first type of data replication if the weighted sum is less than a fourth threshold, or selecting the second type of data replication if the weighted sum is greater than or equal to a fourth threshold.

[0129] Clause 8: A method according to any one of clauses 1 to 7, wherein the second type of data replication includes loading information for updating all rows of the table in the target database system together, and updating all rows of the table in the target database system based on the information.

[0130] Clause 9: The method of any one of clauses 1 to 8, wherein the method further includes using a respective log number, where each log number is assigned to a respective one of the plurality of log files, where each log number indicates a total number of records of data changes in the table of the source database system that have been written up to a respective moment when generation of the respective log file is initiated, and where tracking the first number and the second number is based on the respective log numbers assigned to the plurality of log files.

[0131] Item 10: The method of any one of items 1 to 9, wherein each of the plurality of log files includes a respective log number, wherein each log number is assigned to a respective one of the plurality of log files, wherein each log number indicates a total number of records of data changes in a table of the source database system that have been written up to a respective moment when generation of the respective log file is initiated, and wherein tracking the first number and the second number is based on the respective log numbers of the plurality of log files.

[0132] Item 11: The method of item 9 or 10, wherein tracking the first number includes calculating the difference between the respective log number assigned to the log file that has been stored on the first memory device for the shortest time and the respective log number assigned to the log file that has been stored on the first memory device for the longest time.

[0133] Item 12: The method of item 9 or 10, wherein tracking the first number includes calculating the difference between the actual total number of data changes in the table of the source database system at the actual moment and the respective log number assigned to the log file that has been stored for the longest time on the first memory device.

[0134] Clause 13: The method according to any one of clauses 1 to 12, wherein the method further comprises tracking a record number of the last record of data changes that is read from the first memory device or the second memory device and used for replicating data changes in the table of the target database system according to the last record of data changes; and using the record number of the last record of data changes to track the first number or the second number.

[0135] Item 14: The method of items 9 or 10 and 13, wherein the method further includes calculating the second number as the difference between the respective log number assigned to the log file that has been stored for the longest time on the first memory device and the record number assigned to the last record of a data change, wherein the last record of a data change is included in one of the plurality of log files archived on the second memory device.

[0136] Item 15: The method of items 9 or 10 and 13, wherein the method further includes calculating the first number as the difference between the respective log number assigned to the log file stored on the first memory device for the shortest time and the record number assigned to the last record of data change, wherein the last record of data change is included in one of the plurality of log files stored on the first memory device.

[0137] Item 16: The method of any one of items 1 to 15, wherein the method further includes restoring the plurality of log files, wherein the restoring includes storing the plurality of log files archived on the second memory device on the first memory device.

[0138] Clause 17: The method of clause 16, wherein the method further includes performing the restoring based on the size of each of the records of data changes in the plurality of log files archived on the second memory device.

[0139] Item 18: The method of Item 8 or 9, wherein the method further includes counting the total number of records of data changes in the table of the source database system using a counter; determining whether the counter is in a state close to overflowing the counter; selecting the second type of data replication if the counter is in a state close to overflowing the counter; and resetting the counter after replicating the data changes in the table of the source database system according to the second type of data replication using the records of data changes in the multiple log files, wherein the second type of data replication includes loading all records of data changes stored on the first memory device and the second memory device together, and replicating the data changes of more than one row of the table of the source database system into the table of the target database system according to the loaded records of data changes.

[0140] Clause 19: A computer program product comprising a computer-readable storage medium having a plurality of computer-readable program instructions thereon, the plurality of computer-readable program instructions being configured to perform the steps of the method of any one of clauses 1 to 18.

[0141] Item 20: A computer system for replicating data changes in a table of a source database system into a table of a target database system, the computer system comprising the source database system and the target database system, the computer system repeatedly generating a plurality of log files while executing the data changes in the table of the source database system, where each log file contains at least one record of the data changes in the table of the source database system; storing the plurality of log files on a first memory device, where the first memory device has a first access time; archiving a portion of the plurality of log files stored on the first memory device on a second memory device, where the first The second memory device has a second access time, the second access time being longer than the first access time; tracking a second number of second pending data changes, where the second pending data changes are the data changes of the records of data changes of the plurality of log files archived on the second memory device that have not yet been replicated into a table of a target database system; selecting a type of data replication from a first type of data replication and a second type of data replication depending on the second number; and replicating the data changes of the table of the source database system into the table of the target database system using the records of data changes of the plurality of log files in accordance with the selected type of data replication.

[0142] Item 21: The computer system of Item 20, wherein the first type of data replication includes repeatedly loading each one of the records of data changes individually, and replicating each data change of a single row of the table of the source database system into the table of the target database system according to the individually loaded corresponding record of the data change.

[0143] Item 22: The computer system described in Item 20, wherein the second type of data replication includes loading information for updating multiple rows of the table in the target database system together, and updating the multiple rows of the table in the target database system based on the information.

[0144] Clause 23: A computing system for replicating data changes in a table of a source database system into a table of a target database system, the computing system comprising the source database system and the target database system, the computing system being configured to perform the steps of the method of any one of clauses 1 to 18.

Claims

1. 1. A computer-implemented method comprising: repeatedly generating a plurality of log files while performing data changes in a table of a source database system, wherein each log file contains at least one record of the data changes in the table of the source database system; storing the plurality of log files on a first memory device, wherein the first memory device has a first access time; archiving a portion of the plurality of log files stored on the first memory device onto a second memory device, wherein the second memory device has a second access time, the second access time being greater than the first access time; tracking a second number of second pending data changes, where the second pending data changes are the data changes of the records of the data changes of the plurality of log files archived on the second memory device that have not yet been replicated into a table of a target database system; selecting a type of data replication from a first type of data replication and a second type of data replication based on the second number; and replicating the data changes of the table of the source database system into the table of the target database system according to the selected type of data replication using the records of data changes of the plurality of log files; The method comprising:

2. 2. The computer-implemented method of claim 1, wherein the first type of data replication includes individually loading each of the records of data changes and replicating each data change of a single row of the table of the source database system into the table of the target database system according to the individually loaded corresponding record of data change.

3. 2. The computer-implemented method of claim 1, wherein the second type of data replication includes loading information to jointly update multiple rows of the table in the target database system, and updating the multiple rows of the table in the target database system based on the information.

4. tracking a first number of first pending data changes, where the first pending data changes are the data changes of the records of the data changes of the plurality of log files stored on the first memory device that have not yet been replicated into the table of the target database system; and selecting the type of data replication based on the first number and the second number; The computer-implemented method of claim 1 , further comprising:

5. 5. The computer-implemented method of claim 4, wherein selecting the type of data replication includes weighting the first number and weighting the second number, wherein the weighting of the first number is based on the first access time and the weighting of the second number is based on the second access time.

6. The method comprises: selecting the first type of data replication in response to a ratio of the first number to the second number being greater than or equal to a third threshold. Further comprising: wherein the ratio is a weighted ratio including the first number weighted depending on the first access time and the second number weighted depending on the second access time.

5. The computer-implemented method of claim 4.

7. The method comprises: selecting the second type of data replication in response to a ratio of the first number to the second number being less than a third threshold. Further comprising: wherein the ratio is a weighted ratio including the first number weighted depending on the first access time and the second number weighted depending on the second access time.

5. The computer-implemented method of claim 4.

8. selecting the type of data replication; calculating a weighted sum of the first number and the second number, where the first number is weighted based on the first access time and the second number is weighted based on the second access time; and selecting the first type of data replication if the weighted sum is less than a fourth threshold, or selecting the second type of data replication if the weighted sum is greater than or equal to a fourth threshold.

5. The computer-implemented method of claim 4, comprising:

9. obtaining respective log numbers, where each log number is assigned to a respective one of the plurality of log files, where each log number indicates a total number of records of the data changes in the table of the source database system that have been written up to a respective moment when generation of the respective log file is initiated; and tracking the first number and the second number using the respective log numbers assigned to the plurality of log files; The computer-implemented method of claim 4 further comprising:

10. 5. The computer-implemented method of claim 4, wherein each of the plurality of log files includes a respective log number, wherein a respective log number is assigned to a respective one of the plurality of log files, and wherein the respective log numbers indicate a total number of records of the data changes in the table of the source database system that have been written up to a respective moment when generation of the respective log file is initiated, and wherein tracking the first number and the second number uses the respective log numbers of the plurality of log files.

11. 10. The computer-implemented method of claim 9, wherein tracking the first number comprises calculating a difference between the respective log number assigned to the log file that has been stored on the first memory device for the shortest time and the respective log number assigned to the log file that has been stored on the first memory device for the longest time.

12. 10. The computer-implemented method of claim 9, wherein tracking the first number comprises calculating a difference between an actual total number of data changes for the table in the source database system at an actual moment and the respective log number assigned to the log file that has been stored for the longest time on the first memory device.

13. Tracking the record number of the last record of data changes that is read from the first memory device or the second memory device and used for replicating the data changes in the table of the target database system according to the last record of data changes; and using the record number of the last record of data changes to track the first number or the second number.

10. The computer-implemented method of claim 9, further comprising:

14. The method comprises: calculating the second number as the difference between the respective log number assigned to the log file that has been stored for the longest time on the first memory device and the record number assigned to the last record of a data change; Further comprising: wherein the last record of the data change is included in one of the plurality of log files archived on the second memory device.

14. The computer-implemented method of claim 13.

15. The method comprises: calculating the first number as the difference between the respective log number assigned to the log file stored on the first memory device for the shortest time and the record number assigned to the last record of a data change; Further comprising: wherein the last record of a data change is included in one of the plurality of log files stored on the first memory device.

14. The computer-implemented method of claim 13.

16. 2. The computer-implemented method of claim 1, further comprising determining whether to perform restoration of the plurality of log files based on information regarding a computational cost of restoring the plurality of log files, wherein the restoring comprises storing the plurality of log files archived on the second memory device on the first memory device.

17. 17. The computer-implemented method of claim 16, wherein the method further comprises performing the restoring based on a size of each of the records of data changes in the plurality of log files archived on the second memory device.

18. The method comprises: counting the total number of records of data changes in the table of the source database system using a counter; determining whether the counter is close to overflowing; selecting the second type of data replication when the counter is close to overflowing; and resetting the counter after replicating the data changes of the table of the source database system into the table of the target database system according to the second type of data replication using the records of data changes of the plurality of log files. Further comprising: wherein the second type of data replication includes loading together all records of data changes stored on the first memory device and the second memory device, and replicating the data changes of more than one row of the table of the source database system into the table of the target database system according to the loaded records of data changes.

17. The computer-implemented method of claim 16.

19. A computer program comprising: repeatedly generating a plurality of log files while performing data changes in a table of a source database system, wherein each log file contains at least one record of the data changes in the table of the source database system; storing the plurality of log files on a first memory device, wherein the first memory device has a first access time; archiving a portion of the plurality of log files stored on the first memory device onto a second memory device, wherein the second memory device has a second access time, the second access time being greater than the first access time; tracking a second number of second pending data changes, where the second pending data changes are the data changes of the records of the data changes of the plurality of log files archived on the second memory device that have not yet been replicated into a table of a target database system; selecting a type of data replication from a first type of data replication and a second type of data replication based on the second number; and replicating the data changes of the table of the source database system into the table of the target database system according to the selected type of data replication using the records of data changes of the plurality of log files; The computer program causes one or more processors to execute each step of the method comprising:

20. 1. A computer system comprising: at least one processor; at least one computer-readable storage medium; the source database system; and Target Database System It is equipped with wherein the at least one processor is structured, arranged, connected, and programmed to execute a plurality of program instructions stored on the at least one computer-readable storage medium; and the stored plurality of program instructions: program instructions programmed to repeatedly generate a plurality of log files while executing data changes in a table of a source database system, wherein each log file contains at least one record of the data changes in the table of the source database system; program instructions programmed to store the plurality of log files on a first memory device, wherein the first memory device has a first access time; program instructions programmed to archive a portion of the plurality of log files stored on the first memory device onto a second memory device, wherein the second memory device has a second access time, the second access time being greater than the first access time; program instructions programmed to track a second number of second pending data changes, wherein the second pending data changes are the data changes of the records of data changes of the plurality of log files archived on the second memory device that have not yet been replicated into a table of a target database system; program instructions programmed to select a type of data replication from a first type of data replication and a second type of data replication based on the second number; and program instructions programmed to replicate the data changes of the table of the source database system into the table of the target database system according to the selected type of data replication using the records of data changes in the plurality of log files; The computer system.

Images