Method, system, and computer program product for tracking change data capture log history

Abstract

The present disclosure relates to tracking change data capture log history. A method includes obtaining a first snapshot of a source system and deriving a set of key-value pairs reflecting the first snapshot. A mirroring operation of the source system is performed to obtain CDC change operations. The method obtains a first CDC log as a first sequence of key-value pairs. A second snapshot of the source system is obtained and a set of key-value pairs reflecting the second snapshot is derived. The first sequence of key-value pairs is compared with the set of key-value pairs to derive corrective CDC operations captured as the set of key-value pairs. The corrective CDC operations represent corrections to be performed on the first sequence of key-value pairs. A second CDC log is obtained as a second sequence of key-value pairs. The corrective CDC operations ensure that the second sequence of key-value pairs is coherent with the set of key-value pairs.

Description

Technical Field

[0001] The present invention generally relates to a computerized method and computer program product for tracking change data capture (CDC) log history. In particular, the present invention relates to a method that relies on corrective CDC operations to produce a consistent CDC log. Background Art

[0002] Most enterprise data is stored in relational data warehouses where it is typically updated, processed, and queried, for example, to generate actionable information for the enterprise's business. Data lakes allow data from many different sources to be combined so that additional value can be extracted from such data. For example, the combination of weather data and supply chain data can lead to predictions about potential risks to those supply chains. Therefore, it is of interest to replicate relational data from multiple different sources to many different storage and processing systems, ideally in real time. In a hybrid cloud model, such systems would run on both a company's private cloud and the public clouds of one or more cloud providers. For example, a company's sales data may be stored in a trading system residing on the company's premises and also replicated to a public cloud where analytical processes can generate sales recommendations.

[0003] In a change data capture (CDC) system, only the data that has actually changed in the source system is updated in the target system. A key task in these systems is to identify which parts of the dataset have changed. In relational database systems, this can be efficiently achieved by examining the transaction log.

[0004] Typically, within a CDC system, an initial refresh / snapshot of the table is first performed into a messaging system such as a Kafka topic (or MQ queue), and then all subsequent changes are read from the change log and propagated to the topic. All changes, including the initial refresh, are stored as separate messages. The CDC system can ensure coherence between these two independent operations by noting the operation that performed the first refresh and ensuring that all operations performed after that refresh are correctly captured. Reading the topic will then allow a replica of the source database to be created at the target system. The data in the target database is defined to be coherent with respect to the data in the source database when the state at the target system represents a valid state of the source system. In contrast to consistency, which is a requirement for all databases, coherence relates more specifically to replicated databases. Summary of the Invention

[0005] According to a first aspect, the present invention is implemented as a method for tracking change data capture log history or CDC log history. First, a first snapshot of a source system is obtained and a set S1 of key-value pairs reflecting the first snapshot is derived. Then, a mirroring operation of the source system is performed to obtain CDC change operations accordingly. The CDC change operations represent changes to be performed on the set S1 of key-value pairs. Such operations are captured as a set S1 of key-value pairs. M Next, get the first CDC log as the first sequence S of key-value pairs A , these key-value pairs include set S1 and set S M In addition, a second snapshot of the source system is obtained (after obtaining the first snapshot), and a set S2 of key-value pairs is derived that reflects the second snapshot. Then, the first sequence S of key-value pairs is converted to A The corrective CDC operation is compared with the set S2 of key-value pairs to derive the corrective CDC operation captured as the set S3 of key-value pairs. A Finally, obtain the second CDC log as a second sequence S of key-value pairs B , which includes the sequence S A The corrected CDC operation ensures that the second sequence of key-value pairs S B Coherent as a whole is the set S2 of key-value pairs.

[0006] Preferably, the method further comprises interpreting the second sequence of key-value pairs to modify the current state of the target system so that the target system becomes a target state that is coherent with the state of the source system when the second snapshot is taken.

[0007] According to another aspect, the present invention is implemented as a computer program product for tracking CDC log history. The computer program product includes a computer-readable storage medium having program instructions embodied therewith, wherein the program instructions are executable by a processing device to cause the processing device to perform the steps of the above method.

[0008] A computerized method and a computer program product embodying the present invention will now be described, by way of non-limiting examples, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings serve to further illustrate various embodiments and to explain various principles and advantages of the present disclosure, wherein like reference numerals designate identical or functionally similar elements throughout the several views, and are incorporated into and form a part of this specification together with the following detailed description, wherein:

[0010] Figure 1schematically illustrates a CDC system as interacting with both a source database system and a target database system in an embodiment;

[0011] Figure 2 is a diagram illustrating a CDC log as typically obtained using existing methods;

[0012] Figure 3 is a diagram illustrating how a coherent CDC log may be generated according to an embodiment;

[0013] Figure 4A and 4B is an additional diagram illustrating a method for generating a coherent CDC log that is compatible with sorted compaction of the CDC log, as in an embodiment;

[0014] Figure 5 is a flowchart illustrating high-level steps of a method of tracking CDC log history according to an embodiment; and

[0015] Figure 6 A general purpose computerized system is schematically shown that is suitable for implementing one or more method steps involved in embodiments of the present invention.

[0016] The accompanying drawings show simplified representations of devices or parts thereof involved in the embodiments. Similar or functionally similar elements in the drawings have been assigned the same reference numerals unless otherwise specified. DETAILED DESCRIPTION

[0017] The initial part of the message sequence in this topic corresponds to a full snapshot (corresponding to an operation sometimes called a "refresh" or "load"), while the subsequent replication of the operation is called a mirror operation. The CDC system guarantees that mirroring begins with correct operation after the snapshot operation completes; therefore, such operations can be treated as a single integrated operation, namely a snapshot-mirror operation.

[0018] If a snapshot-mirror operation is performed a second time on the table, the topic being written must be empty. While the mirroring operation is guaranteed to begin after the snapshot is complete, there is no guarantee of data coherence between flushes. For example, if a row existed in the table during the first flush but not the second, it will still exist in the Kafka topic and will never be deleted.

[0019] Any resulting target table will therefore be incoherent with respect to the source table. This is a consequence of the fact that data can be updated in a relational database in two ways: by operating on it or by performing a full snapshot. At the same time, topics such as Kafka topics will still represent such operations in the same way. Note that doing this is normal practice in real systems. For example, databases are periodically backed up and restored to different machines for upgrades, maintenance, etc. Therefore, the above issue is not just a theoretical issue, but something that enterprise systems may want to consider. Some database systems (e.g., Microsoft SQL) disable snapshot operations on tables that are in capture mode, as well as some other operations such as truncation operations.

[0020] The problem is Figure 1 This is illustrated by a simple example in . Assume that a snapshot is performed at time t1. As a result, rows corresponding to Key1 and Key2 are added (Row A, Row B). Mirroring occurs between time t1 and t2, thereby changing the row value for Key1 and adding a new row corresponding to Key3. Next, at time t3, a new snapshot occurs, which causes any data found in the source database to be placed into the subject. In this example, this consists of Key3 and Key4, i.e., Key1 and Key2 no longer exist in the source. However, if the target database is created based on the values found in the subject, the target database will not be coherent with respect to the source database. In fact, the rows corresponding to Key1 and Key2 have never been explicitly deleted from the system, so there is no delete operation in the log.

[0021] In effect, each snapshot of a table creates a new version of that table, and this must be reflected in the (Kafka) representation of that table and in any tables created in the target database that are exported from that representation.

[0022] One solution known in the art is to simply delete the old CDC log and start again. This has the advantage of simplicity, but requires any downstream system reading the log to:

[0023] - Identify that old CDC logs have been deleted;

[0024] Correctly update any downstream systems, such as databases and Elasticsearch indexes;

[0025] - Switching to the new CDC log; and

[0026] - Reread all data.

[0027] As the inventors have observed and inferred, this last point can be particularly problematic, for example, when the source and target systems are separated by a wide area network (WAN, i.e., a network with high latency and low bandwidth), or if the target system does not support a transactional approach that switches to a new CDC log after reading new data. For example, when the source system is on an on-premises system and the target system is in the public cloud, the entire table needs to be transferred over the WAN. Now, this can be prohibitively long when the table is very large, as is often the case in practice.

[0028] As mentioned above, a commonly used approach in the art is to simply delete the target's old CDC log and start over. While this approach is conceptually simple, it typically requires downstream systems to read the log to reread all the data, which can be prohibitively long in some cases. The present invention addresses this issue by creating a coherent CDC log from a new snapshot and the existing CDC log. For example, a new snapshot can be performed on an existing topic that is already in mirror mode and already contains data, while allowing the data at the target to remain coherent relative to the source, as described in detail below.

[0029] The following description is structured as follows. First, a general embodiment and high-level variants are described (Section 1). The next section addresses more specific embodiments and technical implementation details (Sections 2 and 3). Note that the present method and its variants are collectively referred to as "the present method". All reference numerals Sij refer to Figure 5 The method steps of the flowchart are referenced by S x refers to a set or sequence of key-value pairs, and numerical reference numerals refer to physical parts or components of system 1 .

[0030] 1. General Examples and High-Level Variants

[0031] refer to Figures 1 to 3 as well as Figure 5 , firstly an aspect of the invention is described which relates to a method of tracking the history of CDC logs. Such a method may generally be performed by a CDC system 20 or any system which is allowed to interact with a source system 10 and possibly also with a target system 30 in order to update the latter, as in the embodiments described later. Furthermore, this CDC system may form part of the source system 10 or the target system 30. The CDC system may be executed, for example, on any physical or virtual machine. Note that for the sake of completeness, several target systems may be involved; for the sake of simplicity, Figure 1 Only one such target system 30 is shown.

[0032] According to the method, a first snapshot of the source system 10 is obtained at step S10. A set of key-value pairs S1 is then derived, see Figure 3The set S1 reflects the first snapshot obtained. Then, in step S20, a mirroring operation of the source system 10 is performed. Accordingly, CDC change operations are obtained, wherein such operations represent changes performed with respect to the set S1 of key-value pairs in view of the mirroring operation. Like the set S1, the CDC change operations are captured as a set S1 of key-value pairs. M The first CDC log may accordingly be a first sequence S of key-value pairs A Obtain S30, where the first sequence S A Including set S1 and set S M Key-value pairs for both.

[0033] In step S40 , a second snapshot of the source system 10 is obtained, for example, as a load operation, and a set S2 of key-value pairs is derived accordingly, wherein the set S2 reflects the second snapshot.

[0034] Then, the first sequence of key-value pairs S A The comparison S50 is performed with the set S2 of key-value pairs in order to derive the corrective CDC operation. The latter is captured as a set S3 of key-value pairs. The corrective CDC operation represents the set of key-value pairs to be compared with the first sequence S2 of key-value pairs. A Correction performed.

[0035] Finally, the S60 second CDC log is obtained as a second sequence of key-value pairs S B , where the second sequence S B Including sequence S A The CDC operation is corrected to ensure that the second sequence of key-value pairs S B is derived in a manner coherent with the set S2 of key-value pairs as a whole.

[0036] Note that the values of the key-value pairs mentioned above may include any data or data sets; they preferably include structured data. Such values typically correspond to database rows, i.e. rows of the source system 10. The term "key-value pairs" should be understood in this document in a broad sense; it refers to any association of data (values) with corresponding identifiers (keys). If necessary, the method may further cause a unique key to be generated if it is missing in the source system 10. In this way, it is ensured that a suitable key will always be available for the sets S1, S2, S3, the sequences S A and / or sequence S B For each key-value pair in . Such a unique key can be generated, for example, by hashing the contents of the corresponding value.

[0037] For example, a first snapshot may be obtained at a first time t1, and a mirroring operation may be performed during a time period ranging from the first time t1 to a second time t2, the second time t2 being after the time t1. Figure 3 As assumed in . This second snapshot will typically be taken at a third time t3 after t2. Note that this method also works if the second snapshot is taken at time t3 after t1, not necessarily after t2. In all cases, the target system 30 can eventually reach a target state that is coherent with the state of the source system 10 at time t3.

[0038] After the mirroring operation S20, the CDC system is in a mirrored state, ready for a subsequent load operation. However, the subsequent load operation S40 may cause the CDC system 20 to load a state corresponding to a different state (i.e., one that is irrelevant to the state of the source system after the mirroring operation S20). For example, the source system 10 may have meanwhile been restored to a backup state, which may be inconsistent with the state of the system 10 immediately after the mirroring operation S20.

[0039] However, due to the correction operation of capturing S50 as key-value pairs S3, the second sequence of key-value pairs S B coherent with the second snapshot corresponding to the key-value pair S2. That is, the second sequence can be interpreted as bringing the target system 30 to a state reflecting the second snapshot, ie, corresponding to the state of the source system 10 at time t3.

[0040] Therefore, the present method allows to create a coherent CDC log S from the latest snapshot S2 and the existing CDC log S1 by generating a series of corrective CDC operations B Such a correction operation results in the target system 30 being in the same state after processing as it would have been if it had read the new snapshot S2. However, since only CDC messages corresponding to changed values (e.g., rows) are added, when the difference between the new snapshot S2 and the first CDC log S1 is small (which is the most frequent case in practice), the new snapshot S2 can be corrected by interpreting the coherent CDC log S2. B The amount of data to be processed is much smaller (perhaps orders of magnitude smaller) than if the target system 30 had to read the new snapshot S2 completely as in the prior art. In addition, the target system 30 does not need to know that it needs to take some specific action at the target or handover subject.

[0041] Another advantage of the proposed approach is that it allows different types of operations to be interleaved while still making it possible to generate a consistent CDC log. That is, a database table can be modified in two completely different ways: via row operations (e.g., insert, update, etc.) and via table operations (e.g., refresh, truncate, etc.), as known per se. A previous CDC log tracking operations of the first type cannot be semantically directly combined with changes made from operations of the second type, as data coherence is compromised. However, the present approach allows the two types of operations induced above to be interleaved while still generating a consistent CDC log. This is achieved by changing the old CDC log state to a new, coherent CDC log state (a mechanism also referred to as "morphing" in this document), as it is preferably performed incrementally.

[0042] As in Figure 5 As further seen in the process of FIG, the method can further cause (eg, the CDC system 20 or the target system 30) to interpret S70 the second sequence of key-value pairs so as to modify the current state of the target system 30. B , which in turn allows the target system 30 to reach a target state that is coherent with the state of the source system 10 at the moment the second snapshot is taken (ie, from that moment on).

[0043] Preferably, a second sequence S of key-value pairs is obtained S60 B As an ordered sequence, where the key-value pairs of set S1 are in set S M Before the key-value pairs of , which itself precedes the key-value pairs of set S3, such as Figure 3 Similarly, when forming the first CDC log S30, the first sequence of key-value pairs S A Available as an ordered sequence. An ordered sequence allows comparisons to be performed in linear time.

[0044] In an embodiment, S50 corrective CDC operations are obtained incrementally, i.e., each operation reflects one change at a time. Thus, the second CDC log may include one or more of each (or any number) of delete operations, insert operations, and update operations. Each of these operations is captured as a key-value pair. However, in practice, the corrective CDC operation will typically include multiple delete, insert, and update operations.

[0045] As described above, the values of all the key-value pairs mentioned above generally correspond to database rows of the source system 10. Therefore, in an embodiment, if a given database row indexed in the first CDC log is not reflected in the second snapshot, then at step S50 (i.e., in the first sequence S A, and set S2) can result in the export of a corrective CDC operation as a delete operation for the given database row. Similarly, if a given database row indexed in the first CDC log is reflected in the second snapshot, but a non-key field of the row is changed, then comparison S50 can result in the export of a corrective CDC operation as a corresponding update operation to update the non-key field. Likewise, if a given database row (as reflected in the second snapshot) is not indexed in the first CDC log, then comparison S50 can result in the export of one of the corrective CDC operations as an insert operation for the given row. However, if a given database row (as indexed in the first CDC log) is identically reflected in the second snapshot, then comparison S50 does not produce any corrective CDC operation for the row.

[0046] Any suitable algorithm can be envisioned to derive the S50 corrective CDC operation. However, the algorithm is preferably based on the first sequence S A Thus, in an embodiment, step S50 further includes evaluating S A and S2 in order to select the most appropriate algorithm to derive the corrective CDC operation.

[0047] Interestingly, this method is compatible with sort compaction of CDC logs, as discussed in detail in Section 2. Furthermore, this method can still be executed by a CDC system that is configured to partition data into different partitions. In that case, the database rows need to be mapped based on key-value pairs according to the different partitions of the CDC system.

[0048] Next, according to another aspect, the present invention can be implemented as a computer program product. The computer program product includes a computer-readable storage medium having program instructions embodied therewith. Such program instructions can be executed, for example, by a processing device 105 of a CDC system 20. In variants, they can be executed on one or more appropriately connected physical machines, or, if desired, by a virtual machine in a cloud environment, for example. In all cases, such instructions cause the processing device to perform the steps described above. Additional considerations regarding computer program products and computerized systems are provided in Section 3.

[0049] The above embodiments have been briefly described with reference to the accompanying drawings and can accommodate a number of variations. Several combinations of the above features can be envisioned. Examples are given in the next section. 2. Specific Examples

[0051] This section describes embodiments that allow creation of a coherent CDC log based on a new snapshot and an existing CDC log by generating a synthetic series of CDC changes that, after processing, results in the target system being in the same state as if it had fully read the new snapshot.

[0052] Such an embodiment essentially involves adding only the CDC messages corresponding to the rows that have changed. Thus, a small difference between the new snapshot and the CDC log requires only a small amount of data to be read by the downstream system, and allows the target system to not be aware of the fact that it must take some specific action at the target or handover topic.

[0053] Without loss of generality, we can assume that each row in a table (and therefore each message in a topic) can be identified by a unique key. As noted in the previous section, if such a key does not exist, it can be created on the fly by, for example, hashing the contents of the row.

[0054] As can be realized, the comparison S50 of the CDC log with the new snapshot will in practice lead to one of the following situations:

[0055] - The row exists in the old CDC log but not in the new snapshot;

[0056] - The row exists in both and is identical;

[0057] - the row exists in both, but non-key fields have been changed in the snapshot; and

[0058] - This row exists only in the snapshot.

[0059] In the first case, a synthetic delete message can be created in the CDC log for the row. In the second case, no action is required because the row already exists in the CDC log. In the third case, a synthetic update message can be created in the CDC log. In the fourth case, a synthetic insert message can be created in the CDC log.

[0060] In effect, this reduces the snapshot to a certain number of operations. All necessary operations are added to the CDC log. The latter produces the same final state in the target system as would be produced if the target system had only read the snapshot alone.

[0061] A CDC system can, for example, store rows in a CDC log in key order. Similarly, a snapshot can be created using the rows in key order. This allows the first portion of the CDC log to be compared to the snapshot in linear time. Mirrored portions of the CDC log can be processed independently.

[0062] In systems like Kafka, data is partitioned across multiple different partitions. In this case, operations on a specific row are always stored on the same partition. This is achieved by mapping rows to partitions based on key values. Advantageously, this solution can be parallelized on a per-partition basis to ensure scalability.

[0063] Preferably, the method identifies the similarity between the snapshot and the existing CDC log to select the most appropriate algorithm, such as between those defined in Algorithms 1 and 2. It can do this in particular by retaining the keys of the rows added during the mirror phase in a given structure (herein referred to as "mirrorSet"), and then using the ratio of this key to the length of the old CDC log to select the most appropriate algorithm. For example, when this ratio is small and attention is paid to the key order in which the initial snapshot was written, the computation of the composite operation can be calculated in linear time.

[0064] A particularly preferred embodiment is reflected in the following (pseudo-code) Algorithm 1.

[0065]

[0066]

[0067] Additional technical details may be noted: If the mirrorSet is large compared to the old CDC log, then the algorithm of the embodiment captured in Algorithm 1 above may become inefficient.

[0068] Because while the snapshot portion of the log is sorted, the mirror portion is not. In this regard, modern log systems are equipped with a method called "compaction" to compact the snapshot and mirror portions, thereby creating a new snapshot portion. A method that is compatible with compaction but allows for sorted snapshots (hereinafter referred to as the sorted compaction method) is described below in Algorithm 2.

[0069] During a compaction, consumers of the topic cannot read the compaction log or mirrorSets marked for compaction. New records can still be added to and read from the topic as long as the algorithm does not change the offsets used for the new data. Figure 4A A flowchart describing the sort compaction method is provided, and Figure 4B An example on a theme is shown. Figure 4AThe last compaction point in describes the offsets in the log up to the point where the log was sorted and compacted. Mirror sets can contain unsorted and duplicate key entries. The goal of sorted compaction is to compact the current sorted compaction log (snapshot) and mirror set (new updates) into a newly sorted compaction log. To do this, the method first sorts and deduplicates the mirror set. Duplication is handled by taking the record with the highest offset for multiple records with the same key. After sorting and duplication, the mirror set becomes essentially another sorted compaction log that can be merged with the original snapshot, where the same merge rules apply: for two identical keys, taking the key with the higher offset, which will always be the mirror set's value since it was added last. This is similar to the merge phase of the merge sort algorithm. After executing the algorithm shown in Algorithm 2, the snapshot and mirror definitions have changed; the new snapshot becomes the newly sorted compaction log, and the mirror set is the new change added after the algorithm began. The complexity of sorted compaction is O(m log m) for sorting the mirrorSet and O(m + n) for merging it with the compaction log.

[0070]

[0071]

[0072] 3. Technical implementation details

[0073] 3.1 Computerized Systems and Equipment

[0074] Computerized systems and devices can be appropriately designed to implement embodiments of the present invention as described herein. In this regard, it will be appreciated that the methods described herein are largely non-interactive and automated. In exemplary embodiments, the methods described herein can be implemented in interactive, partially interactive, or non-interactive systems. The methods described herein can be implemented with software, hardware, or a combination thereof. In exemplary embodiments, the methods proposed herein are implemented as executable programs in software that are executed by suitable digital processing devices. More generally, embodiments of the present invention can be implemented using virtual machines and / or general-purpose digital computers, such as personal computers, workstations, etc.

[0075] For example, Figure 6 Schematically represented is a computerized unit 101 (for example a general or special purpose computer), which may interact with other similar units in order to be able to perform the steps according to the method.

[0076] In an exemplary embodiment, in terms of hardware architecture, as Figure 6As shown, each unit 101 includes at least one processor 105 and a memory 110 coupled to a memory controller 115. Several processors (CPUs and / or GPUs) may be involved in each unit 101. To this end, each CPU / GPU may be assigned a corresponding memory controller, as is known per se.

[0077] One or more input and / or output (I / O) devices 145, 150, 155 (or peripheral devices) are communicatively coupled via a local input / output controller 135. As is known in the art, the input / output controller 135 may be coupled to or include one or more buses and the system bus 140. The input / output controller 135 may have additional elements, omitted for simplicity, such as controllers, buffers (cache memories), drivers, repeaters, and receivers to enable communication. Further, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0078] The processor 105 is a hardware device for executing software instructions. The processor 105 can be any custom or commercially available processor. In general, they can refer to any type of semiconductor-based microprocessor (in the form of a microchip or chipset), or generally refer to any device for executing software instructions.

[0079] Memory 110 typically includes volatile memory elements (e.g., random access memory) and may also include non-volatile memory elements. In addition, memory 110 may incorporate electrical, magnetic, optical, and / or other types of storage media. Additional storage may be provided via storage device 120.

[0080] The software in the memory 110 may include one or more separate programs, each of which includes executable instructions for implementing logical functions. Figure 6 In the example of , the instructions loaded into the memory 110 may include instructions derived from executing the computerized method described herein according to the exemplary embodiments. The memory 110 may further load a suitable operating system (OS) 111. The OS 111 substantially controls the execution of other computer programs or instructions and provides scheduling, input-output control, file and data management, memory management, and communication control and related services.

[0081] Possibly, a conventional keyboard and mouse may be coupled to the input / output controller 135. Other I / O devices 140-155 may be included. The computerized unit 101 may further include a display controller 125 coupled to the display 130. Any computerized unit 101 will typically include a network interface or transceiver 160 to couple to a network and thereby enable data communication to / from other external components (e.g., other units 101).

[0082] The network sends and receives data between a given unit 101 and other devices 101. The network may be implemented wirelessly, for example, using wireless protocols and technologies such as Wifi, WiMax, etc. The network may be, among other things, a fixed wireless network, a wireless local area network (LAN), a wireless wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), an intranet, or other suitable network system and include devices for receiving and sending signals. Preferably, the network should allow very fast message transmission between units.

[0083] The network may also be an IP-based network for communicating between any given unit 101 and any external unit via a broadband connection. In an exemplary embodiment, the network may be a managed IP network managed by a service provider. Additionally, the network may be a packet-switched network such as a LAN, WAN, Internet network, IoT network, or the like.

[0084] 3.2 Computer Program Products

[0085] The present invention may be a method and / or a computer program product. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0086] Computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punch card or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the above. As used herein, computer readable storage medium should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by a fiber optic cable), or an electrical signal transmitted by a wire.

[0087] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0088] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and procedural programming languages, such as "C" programming language or similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, an electronic circuit (including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA)) can execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions, so as to perform aspects of the present invention.

[0089] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0090] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that, when executed by the processor of the computer or other programmable data processing apparatus, these instructions create a means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing apparatus, and / or other device to function in a particular manner, such that a computer-readable storage medium having instructions stored therein comprises an article of manufacture including instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0091] Computer-readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other apparatus, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0092] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to different embodiments of the present invention. To this end, each box in the flow chart or block diagram can represent a module, a segment or a part of an instruction, which includes one or more executable instructions for realizing the logical function of the specification. In some alternative embodiments, the functions marked in the box may not occur in the order marked in the figure. For example, depending on the function involved, the two boxes shown in succession can actually be performed substantially simultaneously, or these boxes can sometimes be performed in the opposite order. It will also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a system based on special-purpose hardware, and the system based on special-purpose hardware performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

[0093] 3.3 Cloud

[0094] Computations performed according to the present method may be provided as a cloud service. However, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed. Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly configured and released with minimal management effort or interaction with a service provider.

[0095] Although the present invention has been described with reference to a limited number of embodiments, variants and drawings, it will be understood by those skilled in the art that, without departing from the scope of the present invention, various changes may be made and equivalents may be substituted. Specifically, without departing from the scope of the present invention, the features (similar devices or similar methods) described in a given embodiment, variant or shown in the drawings may be combined with or replace another feature in another embodiment, variant or drawings. Different combinations of features described with respect to any of the above-mentioned embodiments or variants are contemplated, and are still within the scope of the appended claims. In addition, many minor modifications may be made to adapt specific circumstances or materials to the teachings of the present invention without departing from its scope. Therefore, it is intended that the present invention is not limited to the specific embodiments disclosed, but that the present invention will include all embodiments falling within the scope of the appended claims. In addition, many other variations other than those explicitly touched upon above may be envisioned.

Claims

1. A method of tracking a change data capture log history or a CDC log history, the method comprising: Obtain a first snapshot of the source system and export a set S1 of key-value pairs reflecting the first snapshot; The mirror operation of the source system is performed to obtain a CDC change operation representing the change to be performed on the set of key-value pairs S1, the CDC change operation being captured as the set of key-value pairs S M ; Get the first CDC log as including the set S1 and the set S M The first sequence of key-value pairs S A ; Obtain a second snapshot of the source system and derive a set S2 of key-value pairs reflecting the second snapshot; Compare the first sequence S of the key-value pairs A The corrected CDC operation is performed on the set S2 of key-value pairs to derive the set S3 of key-value pairs, wherein the corrected CDC operation represents the first sequence S of the key-value pairs. A Corrections performed; as well as Get the second CDC log as including the sequence S A and a second sequence S of key-value pairs of the set S3 B , wherein the corrective CDC operation ensures that the second sequence S of the key-value pairs B Coherent with the set S2 of key-value pairs as a whole.

2. The method according to claim 1, wherein The method further includes interpreting the second sequence of key-value pairs to modify a current state of the target system to a target state that is coherent with the state of the source system at the time the second snapshot was taken.

3. The method according to claim 1, wherein The second sequence S of key-value pairs B is obtained as an ordered sequence, whereby the key-value pairs of set S1 precede those of set S M The set S of key-value pairs M The key-value pairs of s3 precede the key-value pairs of collection S3.

4. The method according to claim 1, wherein The corrective CDC operation includes one or more delete operations, each of which is captured as a key-value pair.

5. The method according to claim 1, wherein The corrective CDC operation includes one or more insert operations, each of which is captured as a key-value pair.

6. The method according to claim 1, wherein The corrective CDC operation includes one or more update operations, each update operation being captured as a key-value pair.

7. The method according to claim 1, wherein The corrective CDC operation includes at least one of each of the following operations: a delete operation, an insert operation, and an update operation, each operation being captured as a key-value pair.

8. The method according to claim 1, wherein The values of all the key-value pairs correspond to database rows of the source system.

9. The method according to claim 8, wherein A given database row indexed in the first CDC log is not reflected in the second snapshot, thereby reducing the first sequence S A The comparison with the set S2 results in deriving one of the corrective CDC operations as a delete operation for the given database row.

10. The method according to claim 8, wherein A given database row indexed in the first CDC log is reflected in the second snapshot, although non-key fields are changed, thereby changing the first sequence S A Comparison with the set S2 results in deriving one of the corrective CDC operations as the corresponding update operation for the non-key field.

11. The method according to claim 8, wherein The given database row reflected in the second snapshot is not indexed in the first CDC log, so the first sequence S A Comparison with said set S2 results in deriving one of said corrective CDC operations as the corresponding insert operation for said given row.

12. The method according to claim 8, wherein A given database row indexed in the first CDC log is identically reflected in the second snapshot, thereby converting the first sequence S A Comparison with the set S2 results in no corrective CDC operation being derived for the given row.

13. The method according to claim 1, wherein The method further includes obtaining a sort compaction of the first CDC log.

14. The method according to claim 1, wherein The method is performed by a CDC system configured to divide data into different partitions.

15. The method according to claim 14, wherein The method further includes mapping database rows of the source system according to the different partitions of the CDC system based on the key-value pairs.

16. The method according to claim 1, wherein The first sequence S A The comparison with the set S2 further comprises evaluating the first sequence S A The similarity between S2 and the set S2, whereby the corrective CDC operation is derived based on an algorithm selected according to the evaluated similarity.

17. The method according to claim 1, wherein The method further comprises: if the unique key is lost in the source system, M , the sequence S A , the set S3 and the sequence S B Each of one or more of the key-value pairs of one or more of produces the unique key.

18. A computer program product for tracking a change data capture log history or a CDC log history, the computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a processing device to cause the processing device to perform the operations of the method according to any one of claims 1 to 17.

19. A system for tracking a change data capture log history or a CDC log history, comprising: one or more processors; as well as One or more computer-readable storage media storing program instructions that, when executed by the one or more processors, are configured to cause the one or more processors to perform the operations of the method according to any one of claims 1 to 17.

Images