Pre-staged for accelerated data extraction

Abstract

Disclosed herein are system, method, and computer program product embodiments for accelerating data extraction from a source system to a target system using LAS. Embodiments operate by receiving data changes for a data extraction from a producer job at a source system. Embodiments store the data changes in a staging area along with corresponding sequence identifiers on a persistent storage device. Embodiments receive a request for the next package of data changes in the staging area from a consumer job at a target system. Embodiments generate the next package from the staging area. Embodiments send the next package to the consumer job. Embodiments receive a commit notification for the next package from the consumer job. Embodiments then remove the data changes in the next package from the staging area in response to receiving the commit notification.

Description

Technical Field

[0001] The present invention relates to system, apparatus, device, method and / or computer program product embodiments and / or combinations and sub-combinations thereof for accelerating data extraction from a source system to a target system using level-ahead (LAS) while ensuring data consistency and reproducibility. Background Art

[0002] It is often necessary to transfer data from a source system to a target system. For example, a producer job may extract data from a source system and send the extracted data to a consumer job at a target system. A producer job may perform a delta extraction from a source system and send the extracted data changes to a consumer job at a target system. The source system often creates packages (also known as transactions) to transfer data to the target system. A package may represent a set of data changes that are either applied in their entirety to the target system or not applied at all. In the case of a delta extraction, all packages must be applied to the target system in the order they were created to ensure data consistency between the source and target systems.

[0003] To ensure that a package is applied in its entirety or not at all to the target system, the producer job can wait to receive notification that the consumer job successfully submitted the package before processing a new package. While the producer job waits for notification from the consumer job, the source and target systems can lock their respective persistent storage devices to ensure data consistency between the source and target systems. If a problem occurs anywhere between the source and target systems, the source system can roll back the uncommitted package and attempt to rebuild it in the target system.

[0004] However, this handshake process between producer and consumer jobs can introduce long wait times, which reduces the data throughput between producer and consumer jobs. In addition, the handshake process can limit the number of producer and consumer jobs that can be executed in parallel, because each job may need to be assigned a separate data region or wait for its assigned data region to be unlocked by another job. Summary of the Invention

[0005] According to an embodiment of the present invention, a computer-implemented method for accelerating data extraction from a source system to a target system is provided, comprising: receiving, by at least one processor, data changes for a data extraction from a producer job at a source system; storing, by at least one processor, the data changes in a staging area along with corresponding sequence identifiers in a permanent storage device, wherein the staging area corresponds to the data extraction; receiving, by at least one processor, a request for a next data change package in the staging area from a consumer job at a target system; generating, by at least one processor, the next package from the staging area; sending, by at least one processor, the next package to the consumer job; receiving, by at least one processor, a submission notification of the next package from the consumer job in response to the sending; and removing, by at least one processor, the data changes in the next package from the staging area in response to receiving the submission notification of the next package.

[0006] According to another embodiment of the present invention, a system is provided, comprising: a memory; and at least one processor coupled to the memory and configured to: receive data changes for a data extract from a producer job at a source system; store the data changes along with corresponding sequence identifiers in a staging area of a permanent storage device, wherein the staging area corresponds to the data extract; receive a request for a next package of data changes in the staging area from a consumer job at a target system; generate the next package from the staging area; send the next package to the consumer job; receive a submission notification of the next package from the consumer job in response to the sending; and remove the data changes in the next package from the staging area in response to receiving the submission notification of the next package.

[0007] According to another embodiment of the present invention, a non-transitory computer-readable medium is provided, having instructions stored thereon, which, when executed by at least one computing device, cause the at least one computing device to perform operations, the operations including: receiving data changes for a data extract from a producer job at a source system; storing the data changes along with corresponding sequence identifiers in a staging area of a permanent storage device, wherein the staging area corresponds to the data extract; receiving a request for a next package of data changes in the staging area from a consumer job at a target system; generating the next package from the staging area; sending the next package to the consumer job; receiving a submission notification of the next package from the consumer job in response to the sending; and removing the data changes in the next package from the staging area in response to receiving the submission notification of the next package. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings are incorporated herein and form a part of this specification.

[0009] Figure 1 is a block diagram of a system for accelerating data extraction from a source system to a target system using LAS while ensuring data consistency and reproducibility, according to some embodiments.

[0010] Figure 2 is an example of a staging area for data extraction from a producer job at a source system according to some embodiments.

[0011] Figure 3 is an example control region for accelerating data extraction according to some embodiments.

[0012] Figure 4 is a flow chart illustrating a process for accelerating data extraction from a source system to a target system using LAS while ensuring consistency and reproducibility, according to some embodiments.

[0013] Figure 5 is an example computer system that can be used to implement various embodiments.

[0014] In the accompanying drawings, like reference numbers generally indicate like or similar elements. Additionally, generally, the leftmost digit(s) of a reference number shows the drawing in which the reference number first appears. DETAILED DESCRIPTION

[0015] Provided herein are system, apparatus, device, method, and / or computer program product embodiments and / or combinations and sub-combinations thereof for accelerating data extraction from a source system to a target system using level-ahead (LAS) while ensuring data consistency and reproducibility.

[0016] An embodiment operates by receiving data changes for a data extract from a producer job at a source system. The embodiment stores the data changes along with corresponding sequence identifiers in a staging area of a persistent storage device, where the staging area corresponds to the data extract. The embodiment receives a request for a next data change package in the staging area from a consumer job at a target system. The embodiment generates the next data change package from the staging area. The embodiment sends the next package to the consumer job. In response to the sending, the embodiment receives a submission notification for the next package from the consumer job. The embodiment then removes the data changes in the next package from the staging area in response to receiving the submission notification for the next package.

[0017] When performing data extraction from a source system to a target system, embodiments can solve at least four technical problems. First, embodiments can improve data throughput by breaking the dependency between the producer job at the source system and the consumer job at the target system. For example, the producer job can store the data changes of the data extraction in the LAS and immediately submit the data changes to the permanent storage device of the source system. This can allow the producer job to continue data extraction without waiting for the consumer job at the target system to actually submit the data changes. Since the producer job and the consumer job are decoupled, they no longer need to be synchronized with each other and can be scaled independently. Therefore, the data throughput between the producer job and the consumer job can be increased.

[0018] Second, the embodiment can improve the load balancing between multiple producer jobs and multiple consumer jobs. For example, for each data extraction performed by a producer job, LAS can generate a separate staging area in its permanent storage device. This can allow the producer job to store the data changes from its corresponding data extraction in its corresponding staging area without interfering with another producer job. Therefore, multiple producer jobs can be executed in parallel, thereby improving data throughput. In addition, when the staging area of the current producer job is full, LAS can initiate switching to a different producer job. Similarly, when the staging area of the current consumer job is empty, LAS can initiate switching to a different consumer job. This can further increase the data throughput between the source system and the target system.

[0019] Third, embodiments can increase data throughput between a source system and a target system by reducing the amount of time the source system's permanent storage is locked. For example, a producer job can store data changes in its corresponding staging area and then immediately commit those data changes to the source system's permanent storage without waiting for those data changes to be committed at the target system. Similarly, a consumer job can retrieve data change packages from the LAS without blocking the producer job from processing more data changes for the consumer job. Finally, a consumer job can retrieve data change packages from the LAS without blocking another consumer job from retrieving data change packages from the LAS.

[0020] Fourth, embodiments can increase data throughput between source and target systems by reducing the size of data extracts. For example, a producer job can reduce the size of a data extract by applying filters and / or projections to the extracted data changes before storing them in the LAS. This can increase data throughput by reducing the amount of time the LAS's permanent storage is locked. Furthermore, this can increase data throughput by reducing the size of data transfers to the target system.

[0021] Figure 1 1 is a block diagram of a system 100 that uses a LAS to accelerate data extraction from a source system to a target system while ensuring data consistency and reproducibility, according to some embodiments. The system 100 may include a source system 102, a LAS 104, and a target system 108. As one skilled in the art will appreciate, the system 100 may include multiple source systems 102 and multiple target systems 108.

[0022] The source system 102 may be a desktop computer, a server, a virtual machine, a container, a laptop computer, a tablet, a smartphone, or other devices as understood by one of ordinary skill in the art. The source system 102 may also be a software platform for cloud computing.

[0023] Source system 102 can be communicatively coupled to persistent storage 110. Persistent storage 110 can represent any storage device that retains data after power to the storage device is removed. For example, persistent storage 110 can be a hard drive, a solid-state drive, a database, a file system, an object store, or various other types of storage devices as understood by those of ordinary skill in the art.

[0024] The LAS 104 may be a software module that is communicatively coupled to the source system 102 and the target system 108. The LAS 104 may be a hardware module, device, or system that is communicatively coupled to the source system 102 and the target system 108.

[0025] The LAS 104 may also be communicatively coupled to a persistent storage device 112. The persistent storage device 112 may represent any storage device that retains data after power to the storage device is removed. For example, the persistent storage device 112 may be a hard drive, a solid-state drive, a database, a file system, an object store, or various other types of storage devices as understood by those skilled in the art.

[0026] The target system 108 may be a desktop computer, a server, a virtual machine, a container, a laptop computer, a tablet, a smartphone, or other devices as understood by one of ordinary skill in the art. The target system 108 may also be a software platform for cloud computing.

[0027] The target system 108 can be communicatively coupled to a persistent storage device 114. The persistent storage device 114 can represent any storage device that retains data after power to the storage device is removed. For example, the persistent storage device 114 can be a hard drive, a solid-state drive, a database, a file system, an object store, or various other types of storage devices as understood by those skilled in the art.

[0028] The source system 102 may include one or more producer jobs. A producer job may be a software or hardware-implemented process that collects data changes made to persistent storage 110 by the source system 102. The producer job may provide the data changes that affect persistent storage 110 to one or more consumer jobs in the target system 108. A producer job may provide data changes for a specific data extraction (e.g., data changes to a specific database table). A consumer job may be a software or hardware-implemented process that receives data changes from the producer job and attempts to commit the data changes to persistent storage 114 of the target system 108.

[0029] To speed up data extraction, the producer job may provide the data changes for the data extraction to the LAS 104. The LAS 104 may store the data changes to the persistent storage 112. The producer job may then immediately commit the data changes to the persistent storage 110.

[0030] After storing the data changes in the persistent storage 112, the LAS 104 can construct a package based on the data changes. A package (also called a transaction) can represent a set of data changes that have been applied in their entirety to the target system 108 (also called committed) or not applied at all. The LAS 104 can send the package to a consumer job in the target system 108. Thus, the LAS 104 can break the dependency between the source system 102 and the target system 108. Because the LAS 104 decouples the producer job from the consumer job, the producer job and the consumer job no longer need to be synchronized and can be scaled independently, thereby potentially allowing for maximum data throughput between the source system 102 and the target system 108 during data extraction.

[0031] To speed up data extraction, the LAS 104 can create a separate staging area in persistent storage 112 for each data extraction. The staging area can be a log of data changes made to persistent storage 110 for a given data extraction. A producer job can append the data changes for a data extraction to its corresponding staging area. The LAS 104 can generate a package based on the data changes in the staging area. The LAS 104 can then send the package to a consumer job at the target system 108 to apply the data changes to persistent storage 114.

[0032] Figure 2 is an example staging area 200 for data extraction from producer jobs at source systems 102 according to some embodiments. Figure 1 To describe Figure 2 .

[0033] The LAS 104 can generate a staging area 200 for a data extract based on a producer job at the source system 102. For example, the LAS 104 can generate the staging area 200 based on the schema of a database object (e.g., a database table). The LAS 104 can generate the staging area 200 by extending the schema of the database object with additional fields. For example, the staging area 200 can include data change entries 210, 212, 214, 216, and 218. The data change entries can include the underlying data change 202 of the database object, a sequence identifier 204, an operation type 206, and a package identifier 208.

[0034] The LAS 104 may set the size of the staging area 200. The LAS 104 may calculate the size of the staging area 200 based on throughput. For example, the LAS 104 may calculate the size of the staging area 200 based on the maximum packet size, the number of active producers, and the number of active consumer jobs. Figure 3 As described above, the maximum packet size, the number of active producers, and the number of active consumer jobs may be stored in the control area 300 as maximum packet size 324, the number of active producers 326, and the number of active consumers 328, respectively. The LAS 104 may calculate the size of the staging area 200 based on various other parameters as will be appreciated by one of ordinary skill in the art.

[0035] The producer job can append data changes read from the persistent storage 110 to the staging area 200 via the LAS 104. The LAS 104 can receive the data changes at an input adapter (also referred to as a LAS IN adapter). The input adapter can be an application programming interface (API) of the LAS 104. The producer job can use the input adapter to append the data changes to the staging area 200 and, thus, store the data changes in the persistent storage 112.

[0036] In some embodiments, a "push" producer job on the source system 102 can send data changes to the LAS 104 in a synchronous call. For example, the "push" producer job can call an input adapter of the LAS 104 to send the data changes to the LAS 104 and, as a result, store the data changes in the persistent storage 112. The "push" producer job can wait until the call succeeds or fails (e.g., receives a response from the LAS 104). If the call succeeds (e.g., the LAS 104 stores the data changes in the persistent storage 112), the "push" producer job can commit the data changes read from the persistent storage 110. In response to the commit, the source system 102 can release any associated locks on the persistent storage 110, thereby shortening the duration of the lock on the persistent storage 110. If the call fails (e.g., the LAS 104 cannot store the data changes in the persistent storage 112), the "push" producer job can retry calling the LAS 104 to store the data changes in the persistent storage 112. In the case of "push", writing to the LAS 104 can occur from within the producer job. As understood by those skilled in the art, a "push" producer job can be software running in its own execution environment (e.g., outside the LAS 104).

[0037] In some other embodiments, the Pull producer job on source system 102 can be invoked from an external execution environment (e.g., within the same environment as LAS 104). For example, an input adapter of LAS 104 can poll the Pull producer job on source system 102 for new data. In response, the Pull producer software can send data changes representing the new data to LAS 104 and, as a result, store the data changes to persistent storage 112. If the invocation is successful (e.g., LAS 104 stores the data changes to persistent storage 112), the Pull producer job can commit the data changes read from persistent storage 110. In response to the commit, source system 102 can release any associated locks on persistent storage 110, thereby shortening the duration of the lock on persistent storage 110. If the invocation fails (e.g., LAS 104 is unable to store the data changes to persistent storage 112), the Pull producer job can be reinvoked by the input adapter of LAS 104. In the "pull" case, writing to the LAS 104 may occur from a process internal to the LAS 104. As will be appreciated by one of ordinary skill in the art, a "pull" producer job may be software where its invocation occurs in an external execution environment.

[0038] In response to receiving data changes from a producer job, LAS 104 can generate a sequence identifier 204 that indicates the order of the data changes between the data changes extracted. Consumer jobs can use sequence identifier 204 to apply the data changes to persistent storage 114 in the correct order.

[0039] The LAS 104 may use a monotonically increasing number to generate the sequence identifier 204. The LAS 104 may also use a timestamp of a data change to generate the sequence identifier 204. The LAS 104 may also generate the sequence identifier 204 using various other techniques as understood by one of ordinary skill in the art.

[0040] The LAS 104 may generate an operation type 206 to indicate the type of change performed by the data change. For example, the operation type 206 may indicate whether the type of change is an insert, update, or delete. The LAS 104 may generate the operation type 206 by analyzing the data change.

[0041] The LAS 104 may assign a packet identifier 208 to the data change. The packet identifier 208 may indicate that the data change belongs to a particular packet. The LAS 104 may initially assign a packet identifier 208 that indicates that the data change has not yet been assigned to any packet. For example, the LAS 104 may assign a packet identifier 208 of 0 to the data change to indicate that the data change has not been assigned to any packet. As will be appreciated by one of ordinary skill in the art, the LAS 104 may use a different value for the packet identifier 208 to indicate that the data change has not been assigned to any packet.

[0042] The LAS 104 may store the data changes as data change entries (eg, data change entry 210) in the staging area 200 along with their corresponding sequence identifier 204, operation type 206, and package identifier 208. The LAS 104 may store the data changes for the data extraction in the order received from the producer job.

[0043] After receiving the data changes at the LAS 104, the LAS 104 may store the data changes in the staging area 200 of the permanent storage 112. If the LAS 104 successfully stores the data changes to the permanent storage 112, the source system 102 may consider the data changes committed without risk of data loss. In other words, once the LAS 104 stores the data changes to the permanent storage 112, the source system 102 may unlock the associated area of the permanent storage 110 and read the next data change to send to the LAS 104. This may allow the LAS 104 to begin processing another data change before the consumer job has completely processed the original data change. Thus, the LAS 104 may process data changes asynchronously in sequence through one or more consumer jobs.

[0044] To speed up data extraction, the source system 102 can reduce the size of the data changes stored in the staging area 200 and the number of operations that may need to be performed by the consumer job. The source system 102 can reduce the number of rows pushed into the producer job (and, if applicable, into the persistent storage 110) by applying one or more filters. The filter can remove the loading of one or more unnecessary rows. For example, if the source data in the persistent storage 110 has a field for "year" and the query applies only to the current year, the producer job can skip loading other years. In cases where the producer job itself does not support filtering, the source system 102 can apply one or more filters.

[0045] In some embodiments, the producer job can "push down" the filter by applying the filter to the producer job. In some other embodiments, the producer job can "push down" the filter to the source database on the permanent storage device 110 by incorporating the filter into the database query.

[0046] The source system 102 can also reduce the number of columns pushed into the producer job (and, if applicable, into persistent storage 110) by applying one or more projections. A projection can remove the loading of one or more unnecessary columns (also referred to as fields).

[0047] In some embodiments, the producer job can "push down" the projection by applying the projection to the producer job. In some other embodiments, the producer job can "push down" the projection to the source database on the persistent storage device 110 by incorporating the projection into the database query.

[0048] To speed up data extraction, the LAS 104 can generate one or more packages from the staging area 200 to send to one or more consumer jobs at the target system 108. A package (also called a transaction) can represent a set of data changes that have been applied in their entirety (also called committed) to the target system 108 or not applied at all. The LAS 104 can send the packages to the consumer jobs in the target system 108. Thus, the LAS 104 can break the dependency between the source system 102 and the target system 108. Because the LAS 104 decouples the producer job from the consumer job, the producer job and the consumer job no longer need to be synchronized and can be scaled independently, thereby potentially allowing for maximum throughput between the source system 102 and the target system 108 during data extraction.

[0049] To reduce the amount of time that persistent storage 112 is locked, LAS 104 can perform a mark and scan process to generate packages to send to consumer jobs on target systems 108. The mark and scan process can include LAS 104 marking data change entries in the staging area with a next package identifier. LAS 104 can then identify data change entries in the staging area that are assigned the next package identifier. LAS 104 can then generate a package for the next package identifier that includes the identified data change entries.

[0050] The LAS 104 can use the staging area 200 and the control area to perform the mark and scan process. The control area can track data changes in the staging area 200, reduce the pressure on the staging area 200, and coordinate the generation and sending of one or more packets between the producer job on the source system 102 and the consumer job on the target system 108.

[0051] Figure 3 is an example control area 300 for accelerating data extraction according to some embodiments. Figure 1 and Figure 2 describe Figure 3 .

[0052] The control area 300 may include various control flags. The control area 300 may include staging area is full 320, data available 322, maximum packet size 324, number of active producers 326, number of active consumers 328, and next packet identifier 330. Staging area is full 320 may indicate whether the staging area 200 is currently full. A producer job may check staging area is full 320 to determine whether it can append data changes to the staging area 200. A producer job may often use staging area is full 300 to determine whether the staging area 200 is full more quickly and computationally less expensively. This is because checking a flag (e.g., staging area is full 300) is often faster and computationally less expensive than calculating whether the staging area 200 is full each time.

[0053] Data available 322 may indicate whether one or more data changes are available in staging area 200 for package sending to a consumer job at target system 108. A consumer job may check data available 322 to determine whether it can retrieve a data change package from LAS 104.

[0054] Maximum packet size 324 may indicate a maximum packet size for a packet. Maximum packet size 324 may specify the maximum packet size as a maximum number of data changes or a maximum data size. Maximum packet size 324 may specify the maximum packet size in various other ways as would be understood by one of ordinary skill in the art.

[0055] The number of active producers 326 may indicate the number of active producer jobs interacting with the LAS 104. The LAS 104 may use the number of active producers 326 to perform load balancing.

[0056] The number of active consumers 328 may indicate the number of active consumer jobs interacting with the LAS 104. The LAS 104 may use the number of active consumers 328 to perform load balancing.

[0057] The control area 300 may include package entries 310, 312, 314, 316, and 318. Each package entry may represent a package that has been processed, is being processed, or will be processed by a consumer job. Each package may include a subscription identifier 302, a package identifier 304, a package status 306, and data changes 308. As will be appreciated by one of ordinary skill in the art, each package may include various other types of fields.

[0058] Subscription identifier 302 can uniquely identify a package subscription. A package subscription can represent a group of logically related packages. For example, a package subscription can represent a set of packages associated with a specific user.

[0059] Package identifier 304 may identify a set of data changes in staging area 200 that are all applied in target system 108 or not applied at all. Package identifier 304 may also uniquely identify the logical location of a package within a subscription to be processed and submitted to target system 108 by a consumer job.

[0060] The package status 306 may indicate the processing status of the package. A status of "Submitted" may indicate that the corresponding package has been submitted to permanent storage 114 by the target system 108. A status of "In Progress" may indicate that the corresponding package is currently being processed by a consumer job and has not yet been submitted to permanent storage 114 by the target system 108. A status of "Rolled Back" may indicate that the corresponding package could not be submitted to permanent storage 114 by the target system 108 and therefore needs to be rebuilt. A status of "New" may indicate that the corresponding package has not yet been processed by a consumer job. As will be appreciated by one of ordinary skill in the art, a package may have various other states.

[0061] Data changes 308 may represent the actual set of data changes that make up a package. For example, data changes 308 may include a set of data changes in staging area 200 that have been labeled with corresponding package identifiers 304. Data changes 308 may represent a series of Structured Query Language (SQL) statements. Data changes 308 may also include the actual contents of the package. Data changes 308 may also include references to the contents of the package. As will be appreciated by those skilled in the art, data changes 308 may include various other types of data.

[0062] To speed up data extraction, consumer jobs can use control area 300 to determine whether data changes are available in staging area 200. Consumer jobs can use control area 300 to determine whether data changes are available via an output adapter (also referred to as LAS OUT) of LAS 104. The output adapter of LAS 104 can be an API.

[0063] The consumer job can determine whether the data change is available by checking the set data available 322. Data available 322 can be a flag indicating that the data change is available in the staging area 200 for packet transmission to the consumer job. As will be appreciated by one of ordinary skill in the art, the consumer job can use various other techniques to determine whether the data change is available.

[0064] In some embodiments, if data changes are not available for the current consumer job, LAS 104 may allow another consumer job to proceed to request the next data change package. The other consumer job may determine whether data changes are available for it and then request a package of those data changes.

[0065] In some other embodiments, if data changes are not available for the current consumer job, LAS 104 may determine a work list of consumer jobs that have data available for them. LAS 104 may allow each of the consumer jobs in the list to proceed with requesting the corresponding next data change package.

[0066] For example, in the case of multiple concurrent extracts to the target system 108 that are generating data at a slow rate (e.g., during a delta extract where a table does not change frequently), the LAS 104 can periodically obtain a worklist of extracts. The LAS 104 can then determine which extract to execute from the worklist. This can avoid a situation where one or more extracts are idle most of the time.

[0067] If data changes are available for the current consumer job, the consumer job may request the next package from LAS 104. The consumer job may request the next package via LAS OUT. In response, LAS 104 may retrieve the first package that was "rolled back" after failing to commit the corresponding package at target system 108 to persistent storage 114. If there is no "rolled back" package, LAS 104 may generate a new package.

[0068] To generate a new packet, LAS 104 may obtain a next packet identifier by atomically increasing next packet identifier 330 in control region 300. Next packet identifier 330 may indicate the next packet identifier of the newly generated packet.

[0069] The LAS 104 may then perform a marking process. The LAS 104 may atomically mark the data change entries in the staging area 200 using the next packet identifier. The LAS 104 may mark one or more data change entries in the staging area 200 such that their corresponding packet identifiers 208 are set to the next packet identifier. The LAS 104 may mark one or more data change entries in the staging area 200 up to a maximum packet size 324. The maximum packet size 324 specifies the maximum packet size as a maximum number of data changes or a maximum data size. The maximum packet size 324 may specify the maximum packet size in various other ways, as would be understood by one of ordinary skill in the art.

[0070] The LAS 104 may block other consumer jobs from modifying the staging area 200 during the marking process. This may prevent other consumer jobs from marking the same data change entries. The LAS 104 may use a single database operator to perform the marking process.

[0071] After marking the data change entries in the staging area 200, the LAS 104 may perform a scan operation. The LAS 104 may identify the data change entries in the staging area 200 that are assigned a next packet identifier. The LAS 104 may then generate a packet for the next packet identifier that includes the identified data change entries.

[0072] For example, in Figure 3 , package entry 310 may represent a package that includes data change entries 210, 212, and 214 from staging area 200. Similarly, package entry 312 may represent a package that includes data change entries 216 and 218 from staging area 200.

[0073] LAS 104 can perform the scanning process without interfering with other consumer jobs. This is possible because the data change entries have already been marked (e.g., obtained) in staging area 200, and thus this will alert other consumer jobs so that they avoid re-marking these data change entries.

[0074] In some embodiments, the LAS 104 may compress identified data changes as part of the scanning process. For example, the LAS 104 may compress data changes for the same key. The LAS 104 may compress data changes by combining multiple data changes for the same record into a single data change. This may increase data throughput between the source system 102 and the target system 108 due to the reduced size of the generated packets.

[0075] After performing the scanning process, the LAS 104 may store the generated package as a package entry in the control area 300. The LAS 104 may set the package status 306 of the generated package to “new.” A status of “new” may indicate that a consumer job has not yet started processing the corresponding package.

[0076] After performing the scan process, the LAS 104 may provide the generated package to the consumer job. For example, the LAS 104 may provide the data change entries that constitute the generated package along with the corresponding package identifier 304. As part of providing the generated package to the consumer job, the LAS 104 may set the package status 306 of the generated package to "in progress." The "in progress" status may indicate that the corresponding package is currently being processed by the consumer job but has not yet been submitted to the persistent storage device 114 by the target system 108.

[0077] In response to receiving the commit notification 120 for the package from the target system 108, the LAS 104 may mark the status 306 of the corresponding package as "committed" in the control area 300. The LAS 104 may then delete the corresponding package from the control area 300 of the persistent storage 112. The LAS 104 may also delete the corresponding data change entry in the staging area 200 of the persistent storage 112.

[0078] In some embodiments, in the event of attempting to commit a package to multiple target systems 108, the LAS 104 may mark the status 306 of the corresponding package as "committed" in the control area 300 in response to receiving a commit notification 120 for the package from each target system 108. After receiving the commit notification 120 from each target system 108, the LAS 104 may then delete the corresponding package from the control area 300 of the persistent storage 112. The LAS 104 may also delete the corresponding data change entry in the staging area 200 of the persistent storage 112.

[0079] In response to receiving a rollback notification 122 of a package from the target system 108, the LAS 104 may automatically drop all packages in the control area 300 that follow the rolled-back package. This may prevent duplicate packages from being created in the persistent storage 114. This may further prevent inconsistencies due to order dependencies in the packages.

[0080] In response to receiving the rollback notification 122 for the package from the target system 108, the LAS 104 may also roll back and rebuild all subsequent "in progress" packages. In some embodiments, the LAS 104 may identify subsequent "in progress" packages as those packages that have a package status 306 of "in progress" and a package identifier 304 that was received at a later time (e.g., a larger package identifier) than the package being rolled back. In some other embodiments, the LAS 104 may identify subsequent "in progress" packages as those packages that have a package status 306 of "in progress," a package identifier 304 that was received at a later time (e.g., a larger package identifier) than the package being rolled back, and a subscription identifier 302 that is the same as the subscription identifier 302 of the package being rolled back.

[0081] The LAS 104 may then generate new packages corresponding to all subsequent "in progress" packages. The LAS 104 may change the package status 306 of each of the generated packages to "rollback". The LAS 104 may then send each of the generated packages to the consumer job in sequence. After sending the corresponding generated package to the consumer job, the LAS 104 may change the package status 306 of each of the generated packages to "in progress".

[0082] The LAS 104 may generate these new packages using a variation of the mark and scan process. The LAS 104 may generate these new packages in response to a consumer job requesting the next package. The LAS 104 may retrieve the first package that was "rolled back" after failing to submit the corresponding package at the target system 108 to the persistent storage 114. In other words, the LAS 104 may retrieve the package identifier 304 of the first "rolled back" package.

[0083] Since the data change entries in the staging area 200 were previously marked with the corresponding package identifier 304, the LAS 104 can skip the marking process of the first "rollback" package. The LAS 104 can then perform a scanning process. The LAS 104 can identify the data change entries in the staging area 200 that are assigned the package identifier 304 of the "rollback" package. The LAS 104 can then generate a package of the "rollback" package that includes the identified data change entries.

[0084] In addition to the mark and scan process, the LAS 104 can facilitate load balancing by controlling how often producer jobs append data changes to the staging area 200. In some embodiments, the LAS 104 can append data changes from the data extract of the producer job to the staging area 200 until a high water mark for the data is reached in the staging area 200. This can prevent the ordered list of data changes from growing indefinitely and can provide the producer job with an opportunity to do other work.

[0085] In some other embodiments, the LAS 104 may append data changes from the data extraction of the producer job to the staging area 200 until the staging area 200 contains a threshold number of data changes. For example, the LAS 104 may append the data changes to the staging area 200 when the staging area 200 contains less than a threshold number of data changes.

[0086] In some other embodiments, the LAS 104 may append data changes from the data extracts of the producer jobs to the staging area 200 until the staging area 200 contains a threshold level of data. In some other embodiments, the LAS 104 may append data changes from the data extracts of the producer jobs to the staging area 200 until various other thresholds are met.

[0087] In some embodiments, the LAS 104 can append data changes from the data ingested by the producer job to the staging area 200 until a global high water mark for the data is reached. The global high water mark for the data can be a threshold level of data that is independent of whether the next received data change is part of the same logical package set (e.g., sharing the same subscription identifier 302).

[0088] In some other embodiments, the LAS 104 may append data changes from the data extract of the producer job to the staging area 200 until a subscription high water mark for the data is reached. The subscription high water mark for the data may be a threshold level for the data specified by the subscription identifier 302 .

[0089] In some embodiments, a producer job can determine whether it can append data changes to the staging area 200 by checking if the staging area is full 220. The staging area is full 220 can be a flag indicating whether the staging area 200 is full. The LAS 104 can ensure that the staging area is full 220 reflects whether the staging area 200 is full. This can ensure that the producer job avoids having to calculate whether the staging area 200 is full.

[0090] Figure 4 4 is a flow chart of a method 400 for accelerating data extraction from a source system to a target system using LAS while ensuring consistency and reproducibility according to an embodiment. The method 400 may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps may be required to perform the present disclosure provided herein. Further, as will be understood by one of ordinary skill in the art, some of the steps may be performed simultaneously or in different steps. Figure 4 Execute in the order shown.

[0091] Should refer to Figure 1 、 Figure 2 as well as Figure 3 4 to describe the method 400. However, the method 400 is not limited to this example embodiment.

[0092] In 402 , the LAS 104 receives data changes for a data extraction from a producer job at the source system 102 to append to the staging area 200 .

[0093] In some embodiments, the LAS 104 can generate a staging area 200 for a data extract. The LAS 104 can generate the staging area 200 based on the schema of database objects (e.g., database tables) associated with the data extract. The LAS 104 can also generate the staging area 200 based on the schema of database objects (e.g., database tables) associated with the data extract and the maximum packet size 324. The LAS 104 can also generate the staging area 200 based on the schema of database objects (e.g., database tables) associated with the data extract, the maximum packet size 324, and the number of active consumers 328. The LAS 104 can also generate the staging area 200 based on the schema of database objects (e.g., database tables) associated with the data extract, the maximum packet size 324, the number of active producers 326, and the number of active consumers 328.

[0094] In some embodiments, the LAS 104 may determine whether the staging area 200 is partially empty. The LAS 104 may receive a data change for a data extraction from a producer job in response to determining that the staging area 200 is partially empty.

[0095] At 404 , the LAS 104 stores the data changes in the staging area 200 of the persistent storage 112 along with the corresponding sequence identifier 204 .

[0096] In some embodiments, the LAS 104 may, in response to receiving the data changes, generate corresponding sequence identifiers 204. The corresponding sequence identifiers 204 may identify the order of the data changes in the data extraction.

[0097] The LAS 104 may use a monotonically increasing number to generate the sequence identifier 204. The LAS 104 may also use a timestamp of a data change to generate the sequence identifier 204. The LAS 104 may also generate the sequence identifier 204 using various other techniques as understood by one of ordinary skill in the art.

[0098] In 406, LAS 104 receives a request for the next data change package in staging area 200 from a consumer job at target system 108. LAS 104 may receive the request via an output adapter (also referred to as LAS OUT) of LAS 104. The output adapter of LAS 104 may be an API.

[0099] In some embodiments, the LAS 104 may determine whether the staging area 200 contains the next package of data changes. The LAS may determine whether the staging area 200 contains the next package of data changes by checking whether data is set available 322. The LAS 104 may receive a request for the next package of data changes from the consumer job in response to determining that the staging area 200 contains the next package of data changes.

[0100] In some embodiments, if data changes are not available for a consumer job, the LAS 104 may allow another consumer job to proceed to request the next data change package. The other consumer job may determine whether data changes are available for it and then request a package of those data changes. In some other embodiments, if data changes are not available for a consumer job, the LAS 104 may determine a worklist of consumer jobs that have data available for them. The LAS 104 may allow each of the consumer jobs in the worklist to proceed to request the corresponding next data change package.

[0101] At 408 , the LAS 104 generates the next data change package from the staging area 200 .

[0102] In some embodiments, if there is no "rollback" package in the control area 300, the LAS 104 can generate a new package. To generate the new package, the LAS 104 can obtain a next package identifier by atomically increasing the next package identifier 330 in the control area 300. The LAS 104 can then use the next package identifier to atomically mark the data change entries in the staging area 200. The LAS 104 can mark one or more data change entries in the staging area 200 so that their corresponding package identifiers 208 are set to the next package identifier.

[0103] After marking the data change entry in the staging area 200, the LAS 104 may identify the data change entry in the staging area 200 assigned a next packet identifier. The LAS 104 may then generate a new packet for the next packet identifier that includes the identified data change entry. The LAS 104 may store the generated packet as a packet entry in the control area 300. The LAS 104 may set the packet state 306 of the generated packet to "new."

[0104] In some other embodiments, if “rollback” packages exist in control area 300, LAS 104 may retrieve the first package that was “rolled back” after failing to commit the corresponding package at target system 108 to persistent storage 114. In other words, LAS 104 may retrieve the package identifier 304 of the first “rollback” package.

[0105] Since the data change entries in the staging area 200 were previously marked with the corresponding package identifier 304, the LAS 104 can skip the marking process of the first "rollback" package. The LAS 104 can then perform a scanning process. The LAS 104 can identify the data change entries in the staging area 200 that are assigned the package identifier 304 of the "rollback" package. The LAS 104 can then generate a package of the "rollback" package that includes the identified data change entries.

[0106] In some embodiments, LAS 104 may set data available 322 in control region 300 after generating the next packet.

[0107] The LAS 104 sends the next data change packet to the consumer job in 410. The LAS 104 may set the corresponding packet status 306 of the next packet to "in progress" after sending the next packet to the consumer job.

[0108] At 412, the LAS 104 receives a commit notification 120 for the next packet to be sent from the target system 108. The LAS 104 may receive the commit notification 120 via an output adapter (also referred to as LAS OUT) of the LAS 104. The output adapter of the LAS 104 may be an API. In response to receiving the commit notification 120, the LAS 104 may mark the packet to be sent as "committed" in the control area 300.

[0109] In 414, the LAS 104 removes the data changes in the next package from the staging area 200 in response to receiving the commit notification 120. The LAS 104 may also delete the corresponding package from the control area 300.

[0110] For example, one could use Figure 5 The computer system 500 shown may be one or more well-known computer systems to implement various embodiments. For example, one or more computer systems 500 may be used to implement any of the embodiments discussed herein, as well as combinations and subcombinations thereof. For example, the computer system 500 may be used to implement Figure 4 Method 400.

[0111] Computer system 500 may include one or more processors (also referred to as central processing units or CPUs), such as processor 504. Processor 504 may be connected to a communication infrastructure or bus 506.

[0112] The computer system 500 may also include user input / output device(s) 503 , such as a display, keyboard, pointing device, etc., which may communicate with the communication infrastructure 506 via the user input / output interface(s) 502 .

[0113] One or more of the processors 504 may be a graphics processing unit (GPU). In an embodiment, a GPU may be a processor that is a specialized electronic circuit designed to process mathematically intensive applications. A GPU may have a parallel architecture that is efficient for parallel processing of large blocks of data, such as mathematically intensive data commonly used for computer graphics applications, images, videos, and the like.

[0114] The computer system 500 may also include a main or primary memory 508, such as random access memory (RAM). The main memory 508 may include one or more cache levels. The main memory 508 may store control logic (ie, computer software) and / or data.

[0115] The computer system 500 may also include one or more secondary storage devices or memories 510. The secondary storage 510 may include, for example, a hard drive 512 and / or a removable storage device or drive 514. The removable storage drive 514 may be a floppy disk drive, a tape drive, an optical drive, an optical storage device, a tape backup device, and / or any other storage device / drive.

[0116] Removable storage drive 514 can interact with removable storage unit 518. Removable storage unit 518 may include a computer-usable or readable storage device having computer software (control logic) and / or data stored thereon. Removable storage unit 518 may be a floppy disk, magnetic tape, optical disk, DVD, optical storage disk, and / or any other computer data storage device. Removable storage drive 514 can read from and / or write to removable storage unit 518.

[0117] Secondary memory 510 may include other tools, devices, elements, means, or other methods for allowing computer programs and / or other instructions and / or data to be accessed by computer system 500. These tools, devices, elements, means, or other methods may include, for example, a removable storage unit 522 and an interface 520. Examples of removable storage unit 522 and interface 520 may include a program cartridge and cartridge interface (such as those found in video game devices), a removable memory chip (such as an EPROM or PROM) and an associated socket, a memory stick and a USB interface, a memory card and an associated memory card slot, and / or any other removable storage unit and associated interface.

[0118] The computer system 500 may further include a communication or network interface 524. The communication interface 524 may enable the computer system 500 to communicate and interact with any combination of external devices, external networks, external entities, and the like (individually and collectively referenced by reference numeral 528). For example, the communication interface 524 may allow the computer system 500 to communicate with an external or remote device 528 via a communication path 526, which may be wired and / or wireless (or a combination thereof) and may include any combination of a LAN, a WAN, the Internet, and the like. Control logic and / or data may be transmitted to and from the computer system 500 via the communication path 526.

[0119] To name a few non-limiting examples, the computer system 500 may also be any of a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet, a smart phone, a smart watch or other wearable device, a home appliance, part of the Internet of Things, and / or an embedded system, or any combination thereof.

[0120] The computer system 500 can be a client or server that accesses or stores any applications and / or data through any delivery model, including but not limited to remote or distributed cloud computing solutions; local or on-site software ("on-site" cloud-based solutions); "as a service" models (e.g., Content as a Service (CaaS), Digital Content as a Service (DCaaS), Software as a Service (SaaS), Management Software as a Service (MSaaS), Platform as a Service (PaaS), Desktop as a Service (DaaS), Framework as a Service (FaaS), Backend as a Service (BaaS), Mobile Backend as a Service (MBaaS), Infrastructure as a Service (IaaS), etc.); and / or a hybrid model that includes any combination of the above examples or other services or delivery models.

[0121] Any applicable data structures, file formats, and architectures in the computer system 500 may be derived from standards including, but not limited to, JavaScript Object Notation (JSON), Extensible Markup Language (XML), another markup language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representations (alone or in combination). Alternatively, proprietary data structures, formats, or architectures may be used exclusively or in combination with known or public standards.

[0122] In some embodiments, a tangible, non-transitory device or article of manufacture comprising a tangible, non-transitory computer-usable or readable medium having control logic (software) stored thereon is also referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 500, main memory 508, secondary memory 510, and removable storage units 518 and 522, as well as tangible articles of manufacture comprising any combination thereof. When executed by one or more data processing devices (such as computer system 500), such control logic may cause these data processing devices to operate as described herein.

[0123] Based on the teachings contained in this disclosure, how to use different Figure 5The data processing devices, computer systems and / or computer architectures shown will be readily apparent to those skilled in the relevant art(s) to make and use the embodiments of the present disclosure. Specifically, the embodiments may operate using software, hardware and / or operating system implementations other than those described herein.

[0124] It should be understood that the detailed description section, and not any other section, is intended to be used to interpret the claims. The other sections may set forth one or more, but not all, exemplary embodiments contemplated by the inventor(s), and therefore are not intended to limit the present disclosure or the appended claims in any way.

[0125] Although the present disclosure describes exemplary embodiments of exemplary fields and applications, it should be understood that the present disclosure is not limited thereto. Other embodiments and modifications thereof are possible and are within the scope and spirit of the present disclosure. For example, and not limiting the intent of this paragraph, the embodiments are not limited to the software, hardware, firmware, and / or entities shown in the drawings and / or described herein. Furthermore, the embodiments (whether or not explicitly described herein) have significant utility in fields and applications beyond the examples described herein.

[0126] Embodiments have been described herein with reference to functional building blocks illustrating implementations of specified functions and relationships thereof. For ease of illustration, the boundaries of these functional building blocks have been arbitrarily defined herein. Alternative boundaries may be defined so long as the specified functions and relationships (or their equivalents) are appropriately implemented. Furthermore, alternative embodiments may implement functional blocks, steps, operations, methods, and the like using different orderings than those described herein.

[0127] References to "one embodiment," "an embodiment," "an example embodiment," or similar phrases herein indicate that the embodiment may include specific features, structures, or characteristics, but each embodiment may not necessarily include specific features, structures, or characteristics. In addition, these phrases do not necessarily refer to the same embodiment. Further, when describing specific features, structures, or characteristics with respect to an embodiment, it is within the knowledge of those skilled in the art to incorporate such features, structures, or characteristics into other embodiments (whether or not explicitly mentioned or described in this article). In addition, some embodiments may be described using the terms "coupled" and "connected," along with their derivatives. These terms are not necessarily intended to be synonyms for each other. For example, some embodiments may be described using the terms "connected" and / or "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also refer to two or more elements that are not in direct contact with each other, but still cooperate or interact with each other.

[0128] The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A computer-implemented method for accelerating data extraction from a source system to a target system, comprising: receiving, by at least one processor, data changes for the data extraction from a producer job at the source system, the source system being communicatively coupled to a first persistent storage device; storing, by the at least one processor, the data changes along with corresponding sequence identifiers in a staging area of a second persistent storage device, wherein the staging area corresponds to the data extraction; sending, by the at least one processor, a response to the producer job indicating success in storing the data change, so that the producer job commits the data change, wherein in response to the producer job committing the data change, the source system releases an associated lock on the first permanent storage device; receiving, by the at least one processor, a request from a consumer job at the target system for a next data change package in the staging area; generating, by the at least one processor, a next packet from the staging area; sending, by the at least one processor, the next packet to the consumer job; receiving, by the at least one processor, a submission notification of the next packet from the consumer job in response to sending the next packet; and The data changes in the next packet are removed from the staging area by the at least one processor in response to receiving the commit notification of the next packet.

2. The method according to claim 1, further comprising: determining, by the at least one processor, that the staging area is partially empty; as well as Wherein receiving the data change further comprises receiving, by the at least one processor, the data change from the producer job in response to determining that the staging area is partially empty.

3. The method according to claim 1, further comprising: The respective sequence identifiers are generated by the at least one processor in response to receiving the data changes, wherein the respective sequence identifiers identify an order of the data changes in the data extraction.

4. The method according to claim 1, wherein generating the next packet further comprises: generating, by at least one processor, a next packet identifier based on a next packet identifier in a control area of the second persistent storage, wherein generating the next packet identifier is performed atomically; marking, by the at least one processor, a data change set in the staging region using the next packet identifier based on a maximum packet size in the control region, wherein marking the data change set is performed atomically; as well as The next packet is generated, by the at least one processor, based on the set of data changes marked with the next packet identifier, wherein generating the next packet is performed non-atomically.

5. The method according to claim 4, further comprising: setting, by the at least one processor, a packet status of the next packet in the control region to "in progress"; as well as A data available flag in the control area is set by the at least one processor, wherein the data available flag indicates that data is available for the consumer job.

6. The method according to claim 1, wherein generating the next packet further comprises: retrieving, by the at least one processor, a next packet identifier corresponding to the packet that has been rolled back; as well as The next packet is generated by the at least one processor based on the set of data changes in the staging area marked with the next packet identifier.

7. The method according to claim 1, further comprising: The staging area of the data extract is generated by the at least one processor based on a schema of a database table and a maximum packet size, wherein the staging area includes a set of data changes, wherein each data change includes a sequence identifier, a packet identifier, and an operation type.

8. A system comprising: Memory; as well as at least one processor coupled to the memory and configured to: receiving data changes for a data extraction from a producer job at a source system, the source system being communicatively coupled to a first persistent storage device; storing the data changes along with corresponding sequence identifiers in a staging area of a second persistent storage device, wherein the staging area corresponds to the data extraction; sending, by the at least one processor, a response to the producer job indicating success in storing the data change, so that the producer job commits the data change, wherein in response to the producer job committing the data change, the source system is configured to release an associated lock on the first permanent storage device; receiving a request for a next data change package in the staging area from a consumer job at a target system; generating a next packet from the staging area; sending the next packet to the consumer job; receiving a submission notification of the next packet from the consumer job in response to sending the next packet to the consumer job; and The data changes in the next packet are removed from the staging area in response to receiving the commit notification for the next packet.

9. The system of claim 8, wherein the at least one processor is further configured to: determining that the staging area is partially empty; and Wherein to receive the data change, the at least one processor is further configured to receive the data change from the producer job in response to determining that the staging area is partially empty.

10. The system of claim 8, wherein the at least one processor is further configured to: The respective sequence identifiers are generated in response to receiving the data changes, wherein the respective sequence identifiers identify an order of the data changes in the data extraction.

11. The system of claim 8, wherein to generate the next packet, the at least one processor is further configured to: generating a next packet identifier based on a next packet identifier in a control area of the second persistent storage, wherein generating the next packet identifier is performed atomically; marking a data change set in the staging area with the next packet identifier based on a maximum packet size in the control area, wherein marking the data change set is performed atomically; and The next packet is generated based on the set of data changes marked with the next packet identifier, wherein generating the next packet is performed non-atomically.

12. The system of claim 11, wherein the at least one processor is further configured to: Setting the packet status of the next packet in the control region to "in progress"; and A data available flag in the control area is set, wherein the data available flag indicates that data is available for the consumer job.

13. The system of claim 8, wherein to generate the next packet, the at least one processor is further configured to: retrieving a next packet identifier corresponding to the packet that has been rolled back; and The next packet is generated based on a set of data changes in the staging area marked with the next packet identifier.

14. The system of claim 8, wherein the at least one processor is further configured to: The staging area of the data extract is generated based on a schema of a database table and a maximum packet size, wherein the staging area includes a set of data changes, wherein each data change includes a sequence identifier, a packet identifier, and an operation type.

15. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: receiving data changes for a data extraction from a producer job at a source system, the source system being communicatively coupled to a first persistent storage device; storing the data changes along with corresponding sequence identifiers in a staging area of a second persistent storage device, wherein the staging area corresponds to the data extraction; sending, by the at least one processor, a response to the producer job indicating success in storing the data change, so that the producer job commits the data change, wherein in response to the producer job committing the data change, the source system releases an associated lock on the first permanent storage device; receiving a request for a next data change package in the staging area from a consumer job at a target system; generating a next packet from the staging area; sending the next packet to the consumer job; receiving a submission notification of the next packet from the consumer job in response to sending the next packet; and The data changes in the next packet are removed from the staging area in response to receiving the commit notification for the next packet.

16. The non-transitory computer-readable medium of claim 15, wherein the operations further comprise: determining that the staging area is partially empty; and Wherein receiving the data changes further comprises receiving the data changes from the producer job in response to determining that the staging area is partially empty.

17. The non-transitory computer-readable medium of claim 15, wherein the operations further comprise: The respective sequence identifiers are generated in response to receiving the data changes, wherein the respective sequence identifiers identify an order of the data changes in the data extraction.

18. The non-transitory computer-readable medium of claim 15, wherein said generating the next packet further comprises: generating a next packet identifier based on a next packet identifier in a control area of the second persistent storage, wherein generating the next packet identifier is performed atomically; marking a data change set in the staging area using the next packet identifier based on a maximum packet size in the control area, wherein marking the data change set is performed atomically; as well as The next packet is generated based on the set of data changes marked with the next packet identifier, wherein generating the next packet is performed non-atomically.

19. The non-transitory computer-readable medium of claim 18, wherein the operations further comprise: Setting the packet status of the next packet in the control area to "in progress"; as well as A data available flag in the control area is set, wherein the data available flag indicates that data is available for the consumer job.

20. The non-transitory computer-readable medium of claim 15, wherein generating the next packet further comprises: Retrieving a next packet identifier corresponding to the packet that has been rolled back; as well as The next packet is generated based on a set of data changes in the staging area marked with the next packet identifier.

Images