Predictive staging for accelerated data extraction

Abstract

System, comprehensive: a storage facility; and at least one processor that is coupled to the memory and configured for: Receiving a data change for a data extraction from a producer job on a source system, where the source system is communicatively coupled to a first persistent storage; Storing the data change in a staging area of ​​a second persistent storage along with a corresponding sequence identifier, where the staging area corresponds to the data extraction; Transmitting a response to the producer job that saving the data change was successful, in order to cause the producer job to commit the data change, with the source system being designed to release an associated lock on the first persistent storage in response to the producer job committing the data change; Receiving a request for the next package of data changes in the staging area from a consumer job on a target system; Generating the next package from the staging area; Transferring the next package to the consumer job; Receiving a commit notification for the next package from the consumer job in response to the transfer of the next package to the consumer job; and Removing the data changes in the next package from the staging area in response to receiving the commit notification for the next package.

Description

background

[0001] Data often needs to be transferred from a source system to a target system. For example, a producer job might extract data from the source system and send the extracted data to a consumer job in the target system. The producer job can perform a delta extraction from the source system and send the extracted data changes to the consumer job in the target system. The source system can often create a packet (also called a transaction) to transfer data to the target system. A packet can represent a set of data changes that are either fully applied or not applied at all in the target system. In the case of delta extraction, packets must be fully applied in the target system in the order they were created to ensure data consistency between the source and target systems.

[0002] To ensure that a package is applied completely or not at all in the target system, the producer job can wait until it receives notification that the consumer job has successfully committed the package before processing a new one. While the producer job waits for notification from the consumer job, the source and target systems can lock their respective persistent storage to ensure data consistency between them. If a problem occurs anywhere between the source and target systems, the source system can roll back the uncommitted package and attempt to restore it in the target system.

[0003] However, this handshake process between the producer job and the consumer job can lead to long wait times, which reduce the data throughput between the two jobs. Furthermore, this handshake process can limit the number of producer and consumer jobs that can run in parallel, as each job may need to be assigned a separate data area or otherwise wait for its assigned data area to be released by another job. Brief description of the drawings

[0004] The attached drawings are included herein and form part of the description. Fig. Figure 1 is a block diagram of a system that uses look-ahead staging (LAS) to accelerate data extraction from a source system to a target system while ensuring data consistency and reproducibility, according to some embodiments. Fig. Figure 2 is an example of a staging area for data extraction from a producer job in a source system according to some embodiments. Fig. 3 is an example of a control area for accelerating data extraction according to some embodiments. Fig. Figure 4 is a flowchart showing a process for using a LAS to accelerate data extraction from a source system to a target system while ensuring consistency and reproducibility, according to some embodiments. Fig. 5 is an example of a computer system that is useful for implementing various embodiments.

[0005] In the drawings, identical reference numerals generally denote identical or similar elements. Additionally, the leftmost digit(s) of a reference numeral generally identifies the drawing in which the reference numeral first appears. Detailed description

[0006] This document provides embodiments of systems, devices, methods and / or computer program products and / or combinations and sub-combinations thereof for the use of look-ahead staging (LAS) to accelerate data extraction from a source system to a target system while ensuring data consistency and reproducibility.

[0007] One embodiment is configured to receive a data change for data extraction from a producer job on a source system. The embodiment stores the data change in a staging area of ​​persistent storage along with a corresponding sequence identifier, the staging area corresponding to the data extraction. The embodiment receives a request for the next packet of data changes in the staging area from a consumer job on a target system. The embodiment generates the next packet of data changes from the staging area. The embodiment transmits the next packet to the consumer job. The embodiment receives a commit notification for the next packet from the consumer job in response to the transmission. The embodiment then removes the data changes in the next packet from the staging area in response to receiving the commit notification for the next packet.

[0008] This embodiment can solve at least four technological problems when performing data extraction from a source system to a target system. First, it can increase data throughput by eliminating the dependency between a producer job on the source system and a consumer job on the target system. For example, the producer job can store a data change for data extraction in a LAS (Local Access Storage) and immediately commit the data change to persistent storage on the source system. This allows the producer job to proceed with data extraction without waiting for a consumer job on the target system to actually commit the data change. Because both the producer and consumer jobs are decoupled, they no longer need to be synchronized and can scale independently.As a result, there may be an increase in data throughput between the producer job and the consumer job.

[0009] Second, this implementation can improve load balancing between multiple producer jobs and multiple consumer jobs. For example, the LAS can generate a separate staging area in its persistent storage for each data extraction performed by a producer job. This allows the producer job to store data changes from its respective data extraction in its respective staging area without blocking another producer job. As a result, multiple producer jobs can run in parallel, increasing data throughput. Furthermore, the LAS can allow switching to another producer job when the staging area for the current producer job is full. Similarly, the LAS can allow switching to another consumer job when the staging area for the current consumer job is empty. This can further increase data throughput between the source and destination systems.

[0010] Third, the embodiment can increase data throughput between the source and destination systems by reducing the time the source system's persistent storage is locked. For example, a producer job can store data changes in its respective staging area and then immediately commit those data changes to the source system's persistent storage without waiting for them to be committed in the destination system. Similarly, a consumer job can retrieve a package of data changes from the LAS without preventing a producer job from processing further data changes for the consumer job. Finally, a consumer job can retrieve a package of data changes from the LAS without preventing another consumer job from retrieving a package of data changes from the LAS.

[0011] Fourth, the embodiment can increase data throughput between the source and target systems by reducing the size of the data extraction. For example, a producer job can reduce the size of the data extraction by applying filters and / or projections to the data modifications before they are stored in the LAS. This can increase data throughput by reducing the time the LAS's persistent storage is locked. Furthermore, it can increase data throughput by reducing the size of the data transfer to the target system.

[0012] Fig. Figure 1 is a block diagram of a system 100 that uses LAS to accelerate data extraction from a source system to a target system while ensuring data consistency and reproducibility, according to some embodiments. The system 100 can comprise a source system 102, an LAS 104, and a target system 108. As is known to those skilled in the art, the system 100 can comprise multiple source systems 102 and multiple target systems 108.

[0013] The source system 102 can be a desktop computer, a server, a virtual machine, a container, a laptop, a tablet, a smartphone, or any other device, as should be obvious to a professional. The source system 102 can also be a software platform for cloud computing.

[0014] The source system 102 can be communicatively coupled to a persistent memory 110. The persistent memory 110 can represent any storage device that retains data after the power supply to the storage device is switched off. For example, the persistent memory 110 can be a hard disk drive, a semiconductor memory drive, a database, a file system, an object storage device, or various other types of storage devices, as should be understandable to a person skilled in the art.

[0015] The LAS 104 can be a software module that is communicatively coupled between the source system 102 and the target system 108. The LAS 104 can also be a hardware module, device, or system that is communicatively coupled between the source system 102 and the target system 108.

[0016] The LAS 104 can also be communicatively coupled to a persistent memory 112. The persistent memory 112 can represent any storage device that retains data after the power supply to the storage device has been switched off. For example, the persistent memory 112 can be a hard disk drive, a solid-state drive, a database, a file system, object storage, or various other types of storage devices, as should be understandable to a person skilled in the art.

[0017] The target system 108 can be a desktop computer, a server, a virtual machine, a container, a laptop, a tablet, a smartphone, or any other device, as should be obvious to a specialist. The target system 108 can also be a software platform for cloud computing.

[0018] The target system 108 can be communicatively coupled to a persistent memory 114. The persistent memory 114 can represent any storage device that retains data after the power supply to the storage device is switched off. For example, the persistent memory 114 can be a hard disk drive, a solid-state drive, a database, a file system, an object storage device, or various other types of storage devices, as should be understandable to a person skilled in the art.

[0019] Source system 102 can have one or more producer jobs. A producer job can be a software- or hardware-implemented process that collects data changes made by source system 102 to persistent storage 110. A producer job can provide the data changes affecting persistent storage 110 to one or more consumer jobs in target system 108. The producer job can provide data changes for a specific data extraction (e.g., data changes to a specific database table). A consumer job can be a software- or hardware-implemented process that receives the data changes from a producer job and attempts to commit the data changes to persistent storage 114 of target system 108.

[0020] To accelerate data extraction, a producer job can provide data changes for extraction to LAS 104. LAS 104 can store the data changes in persistent storage 112. The producer job can then immediately commit the data changes to persistent storage 110.

[0021] After storing the data changes in persistent storage 112, the LAS 104 can construct a package from the data changes. A package (also called a transaction) can represent a set of data changes that are either fully applied (also called committed) in the target system 108 or not applied at all. The LAS 104 can then transfer the package to a consumer job in the target system 108. In this way, the LAS 104 can eliminate the dependency between the source system 102 and the target system 108. Because the LAS 104 decouples a producer job from a consumer job, the producer job and the consumer job may no longer need to be synchronized and can scale independently, thus enabling maximum data throughput between the source system 102 and the target system 108 during data extraction.

[0022] To accelerate data extraction, the LAS 104 can generate a separate staging area in persistent storage 112 for each data extraction. A staging area can be a log of the data changes made in persistent storage 110 for a specific data extraction. A producer job can append a data change for a data extraction to the appropriate staging area. The LAS 104 can then generate a package from the data changes in the staging area. The LAS 104 can then transfer the package to a consumer job on the target system 108 to apply the data changes to persistent storage 114.

[0023] Fig. Figure 2 is an example of a staging area 200 for data extraction from a producer job on the source system 102 according to some embodiments. Fig. 2 is referred to Fig. 1 described.

[0024] The LAS 104 can generate a staging area 200 for data extraction from a producer job in the source system 102. For example, the LAS 104 can generate the staging area 200 from a schema for a database object (such as a database table). The LAS 104 can generate the staging area 200 by extending the schema for the database object with additional fields. For example, the staging area 200 can contain data change entries 210, 212, 214, 216, and 218. A data change entry can have an underlying data change 202 to the database object, a sequence identifier 204, an operation type 206, and a packet identifier 208.

[0025] The LAS 104 can set the size of staging area 200. The LAS 104 can calculate the size of staging area 200 based on throughput. For example, the LAS 104 can calculate the size of staging area 200 based on a maximum packet size, a number of active producers, and a number of active consumer jobs. The maximum packet size, the number of active producers, and the number of active consumer jobs can be stored in control area 300 as a maximum packet size of 324, a number of active producers of 326, and a number of active consumers of 328, as shown in Fig. 3 described. The LAS 104 can calculate the size of the staging area 200 based on various other parameters, as known to a professional.

[0026] A producer job can append a data change read from persistent memory 110 to staging area 200 via LAS 104. LAS 104 can receive the data change at an input adapter (also called LAS IN adapter). The input adapter can be an application programming interface (API) to LAS 104. The producer job can use the input adapter to append the data change to staging area 200, thus storing the data change in persistent memory 112.

[0027] In some embodiments, a push producer job on source system 102 can transmit the data change to LAS 104 in a synchronous call. For example, the push producer job can call the input adapter of LAS 104 to transmit the data change to LAS 104 and thus store the data change in persistent memory 112. The push producer job can wait until the call is successful or fails (e.g., receives a response from LAS 104). If the call is successful (e.g., LAS 104 has stored the data change in persistent memory 112), the push producer job can commit the data change read from persistent memory 110. In response to the commit, source system 102 can release any associated locks on persistent memory 110, thereby reducing the duration of the lock on persistent memory 110. If the call fails (e.g.,If LAS 104 could not save the data change to persistent memory 112, the "push" creator job can retry calling LAS 104 to save the data change to persistent memory 112. In the case of a "push," the write to LAS 104 can be performed from within the creator job. As should be obvious to a person skilled in the art, a "push" creator job can be software that runs in its own execution environment (e.g., outside of LAS 104).

[0028] In some other embodiments, a "pull" producer job on the source system 102 can be called from an external execution environment (e.g., in the same environment as the LAS 104). For example, the input adapter of the LAS 104 can poll the "pull" producer job on the source system 102 for new data. In response, the "pull" producer software can transmit a data change representing the new data to the LAS 104, thus storing the data change in persistent memory 112. If the call is successful (e.g., the LAS 104 has stored the data change in persistent memory 112), the "pull" producer job can commit the data change read from persistent memory 110. In response to the commit, the source system 102 can release any associated locks on persistent memory 110, thereby reducing the duration of the lock on persistent memory 110. If the call fails (e.g.,If the LAS 104 could not save the data change to persistent memory 112, the "pull" creator job can be called again by the input adapter of the LAS 104. In the case of a "pull", the write to the LAS 104 can be initiated from a process within the LAS 104. As should be obvious to a person skilled in the art, a "pull" creator job can be software that is called in an external execution environment.

[0029] In response to receiving the data change from the producer job, the LAS 104 can generate a sequence identifier 204, which specifies the order of the data changes for data extraction. The consumer job can use the sequence identifier 204 to apply the data changes to persistent storage 114 in the correct order.

[0030] The LAS 104 can generate the sequence identifier 204 using a monotonically increasing number. The LAS 104 can also generate the sequence identifier 204 using a timestamp of the data change. The LAS 104 can also generate the sequence identifier 204 using various other techniques, as should be obvious to a person skilled in the art.

[0031] The LAS 104 can generate an operation type 206 to specify the type of change performed by the data modification. For example, operation type 206 can indicate that the change is an insertion, update, or deletion. The LAS 104 can generate operation type 206 by analyzing the data modification.

[0032] The LAS 104 can assign a packet ID 208 to the data change. The packet ID 208 can indicate that the data change belongs to a specific packet. The LAS 104 can initially assign a packet ID 208, indicating that the data change has not yet been assigned to any packet. For example, the LAS 104 can assign a packet ID 208 of 0 to the data change to indicate that the data change is not assigned to any packet. As a specialist would understand, the LAS 104 can use a different value for the packet ID 208 to indicate that the data change is not assigned to any packet.

[0033] The LAS 104 can store the data change, along with its respective sequence identifier 204, operation type 206, and packet identifier 208, as a data change entry (e.g., data change entry 210) in the staging area 200. The LAS 104 can store data changes for data extraction in the order they are received from the producer job.

[0034] Upon receiving a data change, LAS 104 can store the data change in the staging area 200 of persistent storage 112. If LAS 104 successfully stores the data change in persistent storage 112, the source system 102 can treat the data change as committed without any risk of data loss. In other words, once LAS 104 stores the data change in persistent storage 112, the source system 102 can unlock the associated area of ​​persistent storage 110 and read the next data change for transmission to LAS 104. This allows LAS 104 to begin processing another data change before a consumer job has fully processed the original data change. Thus, LAS 104 can process data changes asynchronously in sequence by one or more consumer jobs.

[0035] To accelerate data extraction, the source system 102 can reduce the size of data changes stored in the staging area 200 and the number of operations that a consumer job might need to perform. The source system 102 can reduce the number of rows moved to a producer job and, if possible, to persistent storage 110 by applying one or more filters. A filter can remove the loading of one or more unnecessary rows. For example, if source data in persistent storage 110 has a field for "Year" and a query is only for the current year, the producer job can skip loading the other years. The source system 102 can apply one or more filters if a producer job itself does not support filtering.

[0036] In some embodiments, a producer job can "move down" a filter by applying it within the producer job. In some other embodiments, the producer job can "move down" a filter to a source database on persistent storage 110 by including it in a database query.

[0037] The source system 102 can also reduce the number of columns moved to a producer job and, if possible, to persistent storage 110 by applying one or more projections. A projection can remove the loading of one or more unnecessary columns (also called fields).

[0038] In some embodiments, a producer job can "move down" a projection by applying it within the producer job. In some other embodiments, the producer job can "move down" a projection to a source database on persistent storage 110 by including it in a database query.

[0039] To accelerate data extraction, the LAS 104 can generate one or more packets from the staging area 200 for transfer to one or more consumer jobs on the target system 108. A packet (also called a transaction) can represent a set of data changes that are either fully applied (also called "committed") on the target system 108 or not applied at all. The LAS 104 can transfer the packet to a consumer job on the target system 108. In this way, the LAS 104 can eliminate the dependency between the source system 102 and the target system 108. Because the LAS 104 decouples a producer job and a consumer job, the producer job and the consumer job may no longer need to be synchronized and can scale independently, potentially enabling maximum throughput during data extraction between the source system 102 and the target system 108.

[0040] To reduce the time persistent storage 112 is locked, LAS 104 can perform a tagging and sweeping process to generate a packet for transmission to a consumer job on target system 108. The tagging and sweeping process can involve LAS 104 tagging data change entries in a staging area with a next packet identifier. LAS 104 can then identify data change entries in the staging area that are assigned the next packet identifier. LAS 104 can then generate a packet for the next packet identifier that contains the identified data change entries.

[0041] The LAS 104 can perform the tagging and sweeping process using the staging area 200 and a control area. The control area can track data changes in the staging area 200, reduce the load on the staging area 200, and coordinate the generation and transmission of one or more packets between a producer job on the source system 102 and a consumer job on the target system 108.

[0042] Fig. Figure 3 is an example of a control area 300 for accelerating data extraction according to some embodiments. Fig. 3 is referred to Fig. 1 and Fig. 2 described.

[0043] Control area 300 can have various control flags. Control area 300 can contain the following flags: Staging area is full (320), Data available (322), Maximum packet size (324), Number of active producers (326), Number of active consumers (328), and Next packet ID (330). "Staging area is full" (320) can indicate whether staging area 200 is currently full. A producer job can check "Staging area is full" (320) to determine if it can append a data change to staging area 200. A producer job can often determine whether staging area 200 is full much faster and with less computational effort by using "Staging area is full" (300). This is because checking a flag (e.g., staging area is full 300) is often much faster and computationally more cost-effective than calculating whether the staging area 200 is full every time.

[0044] Data Available 322 can indicate whether one or more data changes are available in staging area 200 for packet transmission to a consumer job on target system 108. A consumer job can check Data Available 322 to determine if it can retrieve a packet containing data changes from LAS 104.

[0045] Maximum packet size 324 can specify a maximum packet size. Maximum packet size 324 can also specify a maximum packet size as the maximum number of data changes or as the maximum data size. Maximum packet size 324 can specify a maximum packet size in various other ways, as should be understandable to a professional.

[0046] The number of active producers (326) can specify the number of active producer jobs interacting with the LAS 104. The LAS 104 can use this number to perform load balancing.

[0047] The number of active consumers (328) can specify the number of active consumer jobs interacting with the LAS 104. The LAS 104 can use this number to perform load balancing.

[0048] The control area 300 can contain package entries 310, 312, 314, 316, and 318. Each package entry can represent a package that has been processed, is being processed, or will be processed by a consumer job. Each package can have a subscription identifier 302, a package identifier 304, a package status 306, and data changes 308. As should be obvious to a professional in this field, each package can contain various other types of fields.

[0049] The subscription identifier 302 can uniquely identify a package subscription. A package subscription can represent a group of logically related packages. For example, a package subscription can represent a set of packages associated with a specific user.

[0050] The packet identifier 304 can identify a set of data changes in staging area 200 that are either fully applied to target system 108 or not applied at all. The packet identifier 304 can also uniquely identify the logical position of a packet within a subscription that is to be processed by a consumer job and committed to target system 108.

[0051] Packet status 306 can represent a packet's processing status. A "committed" status indicates that the corresponding packet has been committed from target system 108 to persistent storage 114. An "inprogress" status indicates that the corresponding packet is currently being processed by a consumer job and has not yet been committed from target system 108 to persistent storage 114. A "rolled back" status indicates that the attempt to commit the corresponding packet from target system 108 to persistent storage 114 failed, and it therefore needs to be reconstructed. A "new" status indicates that the corresponding packet has not yet been processed by a consumer job. As should be obvious to a professional, a packet can have various other states.

[0052] Data changes 308 can represent the actual set of data changes that make up a package. For example, data changes 308 might contain a set of data changes in a staging area 200, marked with the corresponding package identifier 304. Data changes 308 can represent a set of SQL (Structured Query Language) statements. Data changes 308 can also contain the actual contents of the package. Data changes 308 can also contain references to the contents of the package. As a professional would understand, data changes 308 can contain various other types of data.

[0053] To speed up data extraction, a consumer job can use control area 300 to determine if data changes are available in staging area 200. The consumer job can use control area 300 to determine if data changes are available via an output adapter of the LAS 104 (also known as LAS OUT). The output adapter of the LAS 104 can be an Apl.

[0054] The consumer job can determine if data changes are available by checking if `Data Available 322` is set. `Data Available 322` can be a flag indicating that data changes are available in staging area 200 for packet delivery to the consumer job. As any expert in this field would understand, the consumer job can also use various other techniques to determine if data changes are available.

[0055] In some embodiments, if no data changes are available for the current consumer job, the LAS 104 can allow another consumer job to proceed with requesting the next package of data changes. The other consumer job can determine whether data changes are available for it and then request a package of those data changes.

[0056] In some other embodiments, if no data changes are available for the current consumer job, the LAS 104 can determine a working list of consumer jobs for which data is available. The LAS 104 can then allow each of the consumer jobs in this list to proceed with requesting the next package of data changes.

[0057] For example, in the case of multiple simultaneous extractions to target system 108 that generate data at a slow rate (e.g., during a delta extraction where a table is not modified very often), LAS 104 can periodically receive a worklist of extractions. LAS 104 can then determine from the worklist which extraction to execute. This prevents one or more extractions from being idle most of the time.

[0058] When data changes are available for the current consumer job, the consumer job can request the next package from LAS 104. The consumer job can request the next package via LAS OUT. In response, LAS 104 can retrieve the first package that was rolled back after a failed commit of that package to persistent storage on target system 108 (persistent storage 114). If no rolled-back package exists, LAS 104 can generate a new one.

[0059] To generate a new packet, the LAS 104 can claim the next packet identifier by atomically incrementing the next packet identifier 330 in control area 300. The next packet identifier 330 can then specify the next packet identifier for a newly generated packet.

[0060] The LAS 104 can then perform a marking process. The LAS 104 can atomically mark data change entries in staging area 200 with the next packet identifier. The LAS 104 can mark one or more data change entries in staging area 200 such that their respective packet identifiers 208 are set to the next packet identifier. The LAS 104 can mark one or more data change entries in staging area 200 up to a maximum packet size of 324. Maximum packet size 324 can define a maximum packet size as the maximum number of data changes or as the maximum data size. Maximum packet size 324 can define a maximum packet size in various other ways, as should be understandable to an expert in this field.

[0061] The LAS 104 can prevent other consumer jobs from modifying the staging area 200 during the marking process. This prevents other consumer jobs from marking the same data change entries. The LAS 104 can perform the marking process using a single database operator.

[0062] After marking the data change entries in staging area 200, the LAS 104 can perform a sweep operation. The LAS 104 can identify data change entries in staging area 200 that are assigned the next packet ID. The LAS 104 can then generate a packet for the next packet ID that contains the identified data change entries.

[0063] For example, in Fig. 3. Package entry 310 represents a package containing data change entries 210, 212, and 214 from staging area 200. Similarly, package entry 312 can represent a package containing data change entries 216 and 218 from staging area 200.

[0064] The LAS 104 can execute the sweeping process without blocking other consumer jobs. This is possible because the data change entries have already been marked (e.g., claimed) in staging area 200, alerting the other consumer jobs so they do not re-mark these data change entries.

[0065] In some embodiments, the LAS 104 can compact the identified data changes as part of the sweeping process. For example, the LAS 104 can compact data changes for the same keys. The LAS 104 can compact data changes by combining multiple data changes for the same data record into a single data change. This can increase the data throughput between the source system 102 and the target system 108 due to the reduced size of the generated packet.

[0066] After the sweep process is complete, the LAS 104 can store the generated packet as a packet entry in control area 300. The LAS 104 can set the packet status 306 of the generated packet to "New". A "New" status indicates that a consumer job has not yet started processing the corresponding packet.

[0067] After completing the sweep process, LAS 104 can make the generated package available to a consumer job. For example, LAS 104 can provide the data change entries that comprise the generated package, along with the corresponding package ID 304. As part of making the generated package available to the consumer job, LAS 104 can set the package status 306 for the generated package to "in progress." An "in progress" status indicates that the corresponding package is currently being processed by the consumer job but has not yet been committed by the target system 108 to persistent storage 114.

[0068] In response to receiving a commit notification 120 for a packet from the target system 108, the LAS 104 can mark the status 306 of the corresponding packet in control area 300 as "committed". The LAS 104 can then delete the corresponding packet from control area 300 of persistent storage 112. The LAS 104 can also delete the corresponding data change entries in staging area 200 of persistent storage 112.

[0069] In some embodiments where an attempt is made to commit a packet to multiple target systems 108, the LAS 104 can mark the status 306 of the corresponding packet in the control area 300 as "committed" in response to receiving a commit notification 120 for the packet from each target system 108. The LAS 104 can then delete the corresponding packet from the control area 300 of the persistent storage 112 after receiving a commit notification 120 from each target system 108. The LAS 104 can also delete the corresponding data change entries in the staging area 200 of the persistent storage 112.

[0070] In response to receiving a rollback notification 122 for a packet from the target system 108, the LAS 104 can automatically drop all packets following the rolled-back packet into control area 300. This prevents the creation of duplicate packets in persistent storage 114. Furthermore, it can prevent inconsistencies due to packet order dependencies.

[0071] In response to receiving the rollback notification 122 for the packet from the destination system 108, the LAS 104 can also roll back and reconstruct all subsequent packets that are "in progress". In some embodiments, the LAS 104 can identify the subsequent packets that are "in progress" as those packets that have a packet status 306 "in progress" and a packet identifier 304 with a receive time (e.g., a larger packet identifier) ​​that is later than that of the rolled-back packet. In some other embodiments, the LAS 104 can identify the subsequent packets that are "in progress" as those packets that have a packet status 306 "in progress", a packet identifier 304 with a receive time (e.g., a larger packet identifier) ​​that is later than that of the rolled-back packet.a larger package identifier), which is later than that of the reversed package, and have a subscription identifier 302 that is identical to the subscription identifier 302 of the reversed package.

[0072] The LAS 104 can then generate new packets corresponding to all subsequent packets currently "in progress". The LAS 104 can change the packet status 306 of each generated packet to "rolled back". The LAS 104 can then sequentially transfer each generated packet to a consumer job. The LAS 104 can change the packet status 306 of each generated packet to "in progress" after the respective generated packet has been transferred to the consumer job.

[0073] The LAS 104 can generate these new packets using a variation of the mark-and-sweep procedure. The LAS 104 can generate these new packets in response to a consumer job requesting the next packet. The LAS 104 can retrieve the first packet that was rolled back after a commit of that packet to persistent storage on target system 108 failed. In other words, the LAS 104 can retrieve the packet identifier 304 of the first rolled-back packet.

[0074] The LAS 104 can skip the marking process for the first rolled-back packet because the data change entries in staging area 200 were previously marked with the corresponding packet ID 304. The LAS 104 can then perform the sweep process. The LAS 104 can identify the data change entries in staging area 200 that are assigned the packet ID 304 of the rolled-back packet. The LAS 104 can then generate a packet for the rolled-back packet that contains the identified data change entries.

[0075] In addition to the tagging and sweeping process, the LAS 104 can facilitate load balancing by controlling how often a producer job appends data changes to the staging area 200. In some embodiments, the LAS 104 can append data changes for a data extraction from a producer job to the staging area 200 until a flood level for data in the staging area 200 is reached. This can prevent the ordered list of data changes from growing indefinitely and free up the producer job to perform other tasks.

[0076] In some other embodiments, the LAS 104 can append data changes for a data extraction from a producer job to staging area 200 until staging area 200 contains a threshold number of data changes. For example, the LAS 104 can append data changes to staging area 200 even when staging area 200 contains fewer than a threshold number of data changes.

[0077] In some other embodiments, the LAS 104 can append data changes for a data extraction from a producer job to the staging area 200 until the staging area 200 contains a threshold amount of data. In some other embodiments, the LAS 104 can append data changes for a data extraction from a producer job to the staging area 200 until various other thresholds are reached.

[0078] In some embodiments, the LAS 104 can append data changes for a data extraction from a producer job to the staging area 200 until a global data flood mark is reached. The global data flood mark can be a data threshold that is independent of whether the next received data change is part of the same logical set of packets (e.g., has the same subscription identifier 302).

[0079] In some other embodiments, LAS 104 can append data changes for a data extraction from a producer job to staging area 200 until a subscription flood mark for data is reached. The subscription flood mark for data can be a data threshold specific to subscription identifier 302.

[0080] In some implementations, a producer job can determine whether it can append a data change to staging area 200 by checking if staging area 220 is full. "Staging area is full 220" can be a flag representing whether staging area 200 is full. LAS 104 can ensure that "Staging area is full 220" reflects whether staging area 200 is full. This allows the producer job to avoid having to calculate whether staging area 200 is full.

[0081] Fig. Figure 4 is a flowchart for a Method 400 for using a LAS to accelerate data extraction from a source system to a target system while ensuring consistency and reproducibility, according to one embodiment. The Method 400 can be executed by processing logic that may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be noted that not all steps may be required to perform the disclosure provided herein. Furthermore, some of the steps may be performed concurrently or in a different order than shown. Fig. 4 will be shown, as will be clear to a professional in this field.

[0082] Procedure 400 is referred to in the Fig. 1, Fig. 2 and Fig. 3 described. However, method 400 is not limited to this exemplary embodiment.

[0083] In 402, LAS 104 receives a data change for a data extraction from a producer job in the source system 102 to append to the staging area 200.

[0084] In some embodiments, the LAS 104 can generate staging area 200 for data extraction. The LAS 104 can generate staging area 200 based on a schema of a database object (e.g., a database table) associated with data extraction. The LAS 104 can also generate staging area 200 based on a schema of a database object (e.g., a database table) associated with data extraction and with a maximum packet size of 324. The LAS 104 can also generate staging area 200 based on a schema of a database object (e.g., a database table) associated with data extraction, with a maximum packet size of 324 and a number of active consumers of 328. The LAS 104 can also generate staging area 200 based on a schema of a database object (e.g., a database table).a database table), which is associated with data extraction, with a maximum package size of 324, with a number of active producers of 326 and a number of active consumers of 328.

[0085] In some embodiments, the LAS 104 can determine whether the staging area 200 is partially empty. The LAS 104 can then receive the data change for data extraction from the producer job in response to the finding that the staging area 200 is partially empty.

[0086] In 404, the LAS 104 stores the data change together with a corresponding sequence identifier 204 in the staging area 200 of the persistent storage 112.

[0087] In some embodiments, the LAS 104 can generate the respective sequence identifier 204 in response to receiving the data change. The respective sequence identifier 204 can identify the order of the data changes in the data extraction.

[0088] The LAS 104 can generate the sequence identifier 204 using a monotonically increasing number. The LAS 104 can also generate the sequence identifier 204 using a timestamp of the data change. The LAS 104 can also generate the sequence identifier 204 using various other techniques, as should be obvious to a person skilled in the art.

[0089] In response 406, the LAS 104 receives a request for the next packet of data changes in staging area 200 from a consumer job on the target system 108. The LAS 104 can receive the request via an output adapter (also known as LAS OUT). The output adapter of the LAS 104 can be an application programming interface (API).

[0090] In some embodiments, the LAS 104 can determine whether the staging area contains 200 data changes for a next packet. The LAS can determine whether the staging area contains 20 data changes for a next packet by checking if data available 322 is set. The LAS 104 can receive the request for a next packet of data changes from the consumer job in response to the determination that the staging area contains 200 data changes for a next packet.

[0091] In some embodiments, if no data changes are available for the consumer job, the LAS 104 can allow another consumer job to proceed with requesting the next package of data changes. The other consumer job can determine whether data changes are available for it and then request a package of those data changes. In some other embodiments, if no data changes are available for the consumer job, the LAS 104 can determine a working list of consumer jobs for which data is available. The LAS 104 can then allow each of the consumer jobs in this list to proceed with requesting the respective next package of data changes.

[0092] In 408, LAS 104 generates the next package of data changes from staging area 200.

[0093] In some embodiments, if there is no "rolled back" packet in control area 300, the LAS 104 can generate a new packet. To generate the new packet, the LAS 104 can claim the next packet identifier by atomically incrementing the next packet identifier 330 in control area 300. The LAS 104 can then atomically mark data change entries in staging area 200 with the next packet identifier. The LAS 104 can mark one or more data change entries in staging area 200 such that their respective packet identifiers 208 are set to the next packet identifier.

[0094] After marking the data change entries in staging area 200, the LAS 104 can identify data change entries in staging area 200 to which the next packet identifier is assigned. The LAS 104 can then generate a new packet for the next packet identifier that contains the identified data change entries. The LAS 104 can store the generated packet as a packet entry in control area 300. The LAS 104 can set the packet status 306 of the generated packet to "New".

[0095] In some other embodiments, if a rolled-back packet is located in control area 300, the LAS 104 can retrieve the first packet that was rolled back to persistent memory 114 after a failed commit attempt of the respective packet at the target system 108. In other words, the LAS 104 can retrieve the packet identifier 304 of the first rolled-back packet.

[0096] LAS 104 can skip the marking process for the first rolled-back packet because the data change entries in staging area 200 were previously marked with the corresponding packet ID 304. LAS 104 can then perform the sweep process. LAS 104 can identify the data change entries in staging area 200 that are assigned the packet ID 304 of the rolled-back packet. LAS 104 can then generate a packet for the rolled-back packet that contains the identified data change entries.

[0097] In some embodiments, after generating the next data package, the LAS 104 can set data available 322 in control area 300.

[0098] In step 410, LAS 104 transmits the next packet of data changes to the consumer job. After transmitting the next packet to the consumer job, LAS 104 can set the corresponding packet status 306 of the next packet to "in progress".

[0099] In port 412, LAS 104 receives a commit notification 120 for the next transmitted packet from the destination system 108. LAS 104 can receive the commit notification 120 via an output adapter (also called LAS OUT). The output adapter of LAS 104 can be an API. LAS 104 can mark the transmitted packet as "committed" in control area 300 in response to receiving the commit notification 120.

[0100] In 414, LAS 104 removes the data changes in the next packet from staging area 200, in response to receiving commit notification 120. LAS 104 can also delete the corresponding packet from control area 300.

[0101] Various embodiments can be achieved, for example, using one or more known computer systems, such as the one described in Fig. The computer system 500 shown in Figure 5 can be implemented. One or more computer systems 500 can, for example, be used to implement any of the embodiments discussed herein, as well as combinations and subcombinations thereof. The computer system 500 can, for example, be used to implement the method 400 from Figure 5. Fig. 4 to implement.

[0102] The Computer System 500 can have one or more processors (also called central processing units or CPUs), for example a Processor 504. The Processor 504 can be connected to a communication infrastructure or a Bus 506.

[0103] The computer system 500 may also include user input / output device(s) 503, such as monitors, keyboards, pointing devices, etc., which can communicate with the communication infrastructure 506 via user input / output interface(s) 502.

[0104] One or more of the 504 processors can be a graphics processing unit (GPU). In one embodiment, a GPU can be a processor that is a specialized electronic circuit designed to process mathematically intensive applications. The GPU can have a parallel architecture that is efficient for the parallel processing of large blocks of data, such as mathematically intensive data commonly found in computer graphics applications, images, videos, etc.

[0105] The Computer System 500 can also have main or primary memory, such as random access memory (RAM). The main memory can have one or more cache levels. The main memory can store control logic (i.e., computer software) and / or data.

[0106] The Computer System 500 may also include one or more secondary storage devices or memory 510. The secondary storage 510 may, for example, include a hard disk drive 512 and / or a removable storage device or drive 514. The removable storage drive 514 may be a floppy disk drive, a magnetic tape drive, a CD drive, an optical storage device, a tape backup device, and / or any other storage device / drive.

[0107] The removable storage drive 514 can interact with a removable storage unit 518. The removable storage unit 518 can be a computer-readable or -processable storage device on which computer software (control logic) and / or data are stored. The removable storage unit 518 can be a floppy disk, magnetic tape, compact disc, DVD, optical storage disk, and / or other computer data storage device. The removable storage drive 514 can read from and / or write to the removable storage unit 518.

[0108] The secondary storage device 510 may include other means, devices, components, instruments, or other approaches to enable the computer system 500 to access computer programs and / or other instructions and / or data. Such means, devices, components, instruments, or other approaches may include, for example, a removable storage device 522 and an interface 520. Examples of the removable storage device 522 and the interface 520 may include a program cartridge and a cartridge interface (such as those found in video game devices), a removable memory chip (such as an EPROM or PROM) and an associated socket, a memory stick and a USB connector, a memory card and an associated memory card slot, and / or any other removable storage device and an associated interface.

[0109] The computer system 500 may further include a communication or network interface 524. The communication interface 524 enables the computer system 500 to communicate and interact with any combination of external devices, external networks, external units, etc. (individually and collectively designated by the reference number 528). For example, the communication interface 524 enables the computer system 500 to communicate with external or remote devices 528 via a communication path 526, which may be wired and / or wireless (or a combination thereof) and which may include any combination of LANs, WANs, the Internet, etc. Control logic and / or data may be transmitted to and from the computer system 500 via the communication path 526.

[0110] The Computer System 500 can also be a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet, a smartphone, a smartwatch or other portable device, a consumer facility, part of the Internet of Things and / or an embedded system, to name just a few non-limiting examples, or any combination thereof.

[0111] The Computer System 500 can be a client or server that accesses or hosts applications and / or data via any deployment model, including but not limited to remote or distributed cloud computing solutions; locally or on-premises installed software (“on-premises” cloud-based solutions); “as-a-service” models (e.g., Content as a Service (CaaS), Digital Content as a Service (DCaaS), Software as a Service (SaaS), Managed Software as a Service (MSaaS), Platform as a Service (PaaS), Desktop as a Service (DaaS), Framework as a Service (FaaS), Backend as a Service (BaaS), Mobile Backend as a Service (MBaaS), Infrastructure as a Service (laaS), etc.); and / or a hybrid model that includes any combination of the foregoing examples or other services or deployment models.

[0112] All applicable data structures, file formats, and schemas in Computer System 500 can be derived from standards, including but not limited to JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or other functionally similar representations, alone or in combination. Alternatively, proprietary data structures, formats, or schemas can be used either exclusively or in combination with known or open standards.

[0113] In some embodiments, a tangible, non-volatile device or tangible, non-volatile product article comprising a tangible, non-volatile, computer-readable medium on which control logic (software) is stored may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, a computer system 500, a main memory 508, a secondary memory 510, and removable storage units 518 and 522, as well as tangible products embodying any combination of the aforementioned elements. Such control logic, when executed by one or more data processing devices (such as the computer system 500), can cause such data processing devices to operate as described herein.

[0114] Based on the teachings contained in this disclosure, it will be obvious to those skilled in the field how embodiments of this disclosure can be implemented using data processing devices, computer systems and / or computer architectures other than those described in Fig. The embodiments shown in section 5 can be manufactured and used. In particular, embodiments can be operated with software, hardware and / or operating system implementations other than those described here.

[0115] It should be noted that the section "Detailed Description" and no other section is to be used for the interpretation of the claims. Other sections may describe one or more, but not all, embodiments considered by the inventors and are therefore not intended to limit this disclosure or the accompanying claims in any way.

[0116] Although this disclosure describes exemplary embodiments for exemplary domains and applications, it should be understood that the disclosure is not limited thereto. Other embodiments and modifications thereof are possible and are within the scope and spirit of this disclosure. For example, and without limiting the generality of this paragraph, embodiments are not limited to the software, hardware, firmware, and / or entities shown in the figures and / or described herein. Furthermore, embodiments (whether or not explicitly described herein) have significant utility in domains and applications beyond the examples described herein.

[0117] Implementations have been described herein using functional blocks that illustrate the implementation of specified functions and their relationships. The boundaries of these functional blocks have been arbitrarily defined here for the sake of simplicity. Alternative boundaries may be defined as long as the specified functions and relationships (or their equivalents) are appropriately executed. Furthermore, alternative implementations may execute functional blocks, steps, operations, procedures, etc., using sequences that differ from those described herein.

[0118] References in this document to “an embodiment”, “an embodiment”, “an exemplary embodiment”, or similar expressions mean that the described embodiment may have a particular feature, structure, or property, but not every embodiment necessarily has to have that particular feature, structure, or property. Furthermore, such expressions do not necessarily refer to the same embodiment. Moreover, when a particular feature, structure, or property is described in connection with an embodiment, it would be understood by those skilled in the art to incorporate that feature, structure, or property into other embodiments, whether or not they are expressly mentioned or described herein.Furthermore, some embodiments can be described using the terms "coupled" and "connected," along with their derivatives. These terms are not necessarily intended to be synonymous. For example, some embodiments may be described using the terms "connected" and / or "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other but nevertheless work together or interact with each other.

[0119] The scope and extent of this disclosure are not limited by the exemplary embodiments described above, but are defined solely by the following claims and their equivalents.

Images