Advance data classification for time-marching reconstruction

Abstract

Disclosed herein are systems, methods, and computer program product embodiments for utilizing Lookahead Data Analysis (LAS) to ensure the ability to roll back and rebuild packages while minimizing lock duration and enabling multiple packages to be processed simultaneously in a data pipeline. One embodiment operates by receiving a package from a source system for processing through a data pipeline. The embodiment stores the package in a permanent storage device along with the corresponding package state. The embodiment sends the package to the data pipeline in response to the storage. The embodiment receives a commit notification for the package from a target system in response to the sending. Thereafter, the embodiment removes the package from the permanent storage device in response to receiving the commit notification for the package.

Description

Technical Field

[0001] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for utilizing Lookahead Data Staging (LAS) to ensure rollback and reconstruction of packages while minimizing lock duration and enabling simultaneous processing of multiple packages in a data pipeline. Background Art

[0002] Packages (also called transactions) are often used to send data from a source system to a target system. A package can represent a set of changes that are applied in their entirety or not at all in the target system. When a problem occurs anywhere between the source and target systems, the source system can roll back the uncommitted package and attempt to rebuild the package in the target system.

[0003] However, when the source system attempts to submit a package asynchronously at the target system, rolling back and rebuilding the unsubmitted package is often not possible. This is because the source system may be continuously updated after the attempt to submit the package, and the data in the source system's persistent storage may have changed during this time. Some systems guarantee the ability to roll back and rebuild packages by ensuring that areas of the source system's persistent storage and areas of the target system's persistent storage are locked while the package is being processed. However, the duration of the lock can be long, which can prevent other users from using the source and target systems' persistent storage. This often occurs when processing packages using a data pipeline that has many intermediate processing stages or that crosses the boundaries of many different systems. In addition, such locks can prevent the data pipeline from processing more than one packet at a given point in time, which can reduce the throughput of the data pipeline. Summary of the invention

[0004] According to an embodiment of the present disclosure, a computer-implemented method for enabling reconstruction of a package processed through a data pipeline is provided, the method comprising: receiving a package from a source system for processing through a data pipeline by at least one processor, wherein the package includes a set of changes to be submitted to a target system; storing the package together with a corresponding package state in a permanent storage device by at least one processor; in response to the storing, sending the package to the data pipeline by at least one processor; in response to the sending, receiving a submission notification for the package from the target system by at least one processor; and in response to receiving the submission notification of the package, deleting the package from the permanent storage device by at least one processor.

[0005] According to another embodiment of the present disclosure, a system is provided, comprising: a memory; and at least one processor, the at least one processor being coupled to the memory and configured to: receive a package from a source system for processing through a data pipeline, wherein the package includes a set of changes to be submitted to a target system; store the package together with a corresponding package state in a permanent storage device; in response to the storing, send the package to the data pipeline; in response to the sending, receive a submission notification for the package from the target system; and in response to receiving the submission notification of the package, delete the package from the permanent storage device.

[0006] According to another embodiment of the present disclosure, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by at least one computing device, causes the at least one computing device to perform operations including: receiving a package from a source system for processing through a data pipeline, wherein the package includes a set of changes to be submitted to a target system; storing the package together with a corresponding package state in a permanent storage device; in response to the storing, sending the package to the data pipeline; in response to the sending, receiving a submission notification for the package from the target system; and in response to receiving the submission notification of the package, deleting the package from the permanent storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings are incorporated herein and form a part of the specification.

[0008] Figure 1 is a block diagram of a system that utilizes look-ahead data staging (LAS) to ensure the ability to roll back and rebuild packages while minimizing lock duration and enabling multiple packages to be processed simultaneously in a data pipeline, according to some embodiments.

[0009] Figure 2 is an example of an ordered list of packages maintained by the LAS module according to some embodiments.

[0010] Figure 3 is a flow chart illustrating a process of utilizing LAS to ensure packet reconstruction while minimizing lock duration and enabling multiple packets to be processed simultaneously in a data pipeline in accordance with some embodiments.

[0011] Figure 4 is an exemplary computer system for implementing various embodiments.

[0012] In the drawings, the same reference numbers generally refer to the same or similar elements. In addition, generally, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. DETAILED DESCRIPTION

[0013] Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for utilizing Lookahead Data Staging (LAS) to ensure rollback and reconstruction of packages while minimizing lock duration and enabling simultaneous processing of multiple packages in a data pipeline.

[0014] One embodiment operates by receiving a packet from a source system for processing through a data pipeline. The embodiment stores the packet in a permanent storage device along with a corresponding packet state. The embodiment sends the packet to the data pipeline in response to the storing. The embodiment receives a commit notification for the packet from a target system in response to the sending. Thereafter, the embodiment removes the packet from the permanent storage device in response to receiving the commit notification for the packet.

[0015] This embodiment can solve at least two technical problems when processing packets through a data pipeline between a source system and a target system. First, this embodiment can minimize the lock duration of the permanent storage devices of the source system and the target system, while maintaining the ability to roll back and rebuild uncommitted packets in response to problems occurring anywhere between the source system and the target system. Second, this embodiment enables each processing step of the data pipeline to act on different packets at the same time, thereby increasing the throughput of the data pipeline.

[0016] Figure 1 1 is a block diagram of a system 100 that utilizes LAS to ensure the ability to roll back and rebuild packages while minimizing lock duration and enabling multiple packages to be processed simultaneously in a data pipeline, according to some embodiments. The system 100 may include a source system 102, a LAS 104, a data pipeline 106, and a target system 108. As understood by one of ordinary skill in the art, the system 100 may include multiple target systems 108.

[0017] The source system 102 may be a desktop computer, a server, a virtual machine, a container, a laptop computer, a tablet, a smart phone, or other devices understood by a person of ordinary skill in the art. The source system 102 may also be a software platform for cloud computing.

[0018] The source system 102 may include producer software that reads packages (also referred to as transactions) from the persistent storage 110. A package may represent a set of changes that are fully applied (also referred to as committed) or not applied at all in the target system 108. When a problem occurs in applying a package in the target system 108, the source system 102 may roll back the uncommitted package and attempt to rebuild the package in the target system 108.

[0019] Persistent storage 110 may represent any storage device that retains data after power to the storage device is turned off. For example, persistent storage 110 may be a hard disk drive, a solid-state drive, a database, a file system, an object store, or various other types of storage devices understood by those of ordinary skill in the art. Persistent storage 110 may be communicatively coupled to source system 102.

[0020] The source system 102 may send packets to the LAS 104, and the LAS 104 may send the packets to the target system 108 through the data pipeline 106. In some embodiments, the source system 102 may send packets to the LAS 104, and the LAS 104 may send the packets to multiple target systems 108 through the data pipeline 106. In other words, the LAS 104 may introduce interrupts in the packet processing chain. The LAS 104 may be a software module that is communicatively coupled to the source system 102. The LAS 104 may also be a hardware module, device, or system that is communicatively coupled to the source system 102.

[0021] LAS 104 may write the packet to persistent storage 112. Persistent storage 112 may represent any storage device that retains data after device power is turned off. Persistent storage 112 may be communicatively coupled to LAS 104.

[0022] The target system 108 may be a desktop computer, a server, a virtual machine, a container, a laptop computer, a tablet, a smart phone, or other devices understood by those of ordinary skill in the art. The target system 108 may also be a software platform for cloud computing.

[0023] The target system 108 may include consumer software that receives packages from the data pipeline 106 and writes them to the persistent storage 114. When a package is committed to the target system 108 (e.g., when the package is successfully written to the persistent storage 114), the target system 108 may send a commit notification 120 to the LAS 104. When a problem occurs between the source system 102 and the target system 108 (e.g., when the package fails to be committed in the target system 108), the target system 108 (or the data pipeline 106) may also send a rollback notification 122 to the LAS 104, so that the LAS 104 needs to roll back and rebuild the package.

[0024] Persistent storage 114 may represent any storage device that retains data after the device power is turned off. Persistent storage 114 may be communicatively coupled to source system 102. If source system 102 is the same device or platform as the target system, then persistent storage 114 may be the same as persistent storage 110.

[0025] The data pipeline 106 may include one or more data processing steps 107 that can be performed on packets received from the LAS 104. The one or more data processing steps 107 may be performed on packets before the packets are submitted to the target system 108. The data pipeline 106 may process packets sequentially through each data processing step 107. The data pipeline 106 may also process different packets simultaneously in each data processing step 107. For example, at a given time, the data processing step 107-1 may process a first packet (e.g., a later packet) and the data processing step 107-2 may process a second packet (e.g., an earlier packet).

[0026] The data processing step 107 may perform any set of operations on the packet. The data processing step 107 may also perform filtering steps on the packet. This may potentially result in an empty packet, and therefore no actual data being written to the permanent storage device 114. In this case, the target system 108 may still send a commit notification 120 to the LAS 104 to cause the LAS 104 to update and / or delete the packet in the permanent storage device 112.

[0027] Data processing step 107 may be performed on source system 102, target system 108, or an intermediate system between source system 102 and target system 108. Data processing step 107 may be performed on a desktop computer, a server, a virtual machine, a container, a laptop computer, a tablet, a smart phone, or another device understood by a person of ordinary skill in the art. Data processing step 107 may also be performed on a software platform for cloud computing. Data processing step 107 may be communicatively coupled to a subsequent data processing step 107.

[0028] The system 100 can utilize LAS 104 to solve several technical problems. First, if there is a problem when processing a package through the data pipeline 106, the system 100 can utilize LAS 104 to ensure the ability to roll back and rebuild the package. For example, the system 100 can utilize LAS 104 to roll back and rebuild a package that failed to be submitted to the target system 108 due to a problem between certain locations between the source system 102 and the target system 108 (for example, because there is a network problem or because the target system 108 is temporarily unavailable). Secondly, the system 100 can utilize LAS 104 to minimize the amount of time that other users are blocked from using permanent storage devices 110 and 114. For example, the system 100 can utilize LAS 104 to reduce the amount of time that the producer software on the source system 102 locks the area of ​​the permanent storage device 110. Finally, the system 100 can utilize LAS 104 to increase the throughput of the data pipeline 106.

[0029] LAS 104 may introduce a break in the packet processing chain by providing access to persistent storage 112. Persistent storage 112 may be a small persistent storage device located close to source system 102. Persistent storage 112 may represent any storage device that retains data after power to the device is turned off. Persistent storage 112 may be communicatively coupled to source system 102. If source system 102 also implements LAS 104, persistent storage 112 may be the same as persistent storage 110.

[0030] The LAS 104 can be communicatively coupled to the source system 102. The LAS 104 can also be communicatively coupled to the data pipeline 106. The LAS 104 can receive packets from the source system 102.

[0031] LAS 104 may include an input adapter (also referred to as a LAS IN adapter) to receive packets from source system 102. The input adapter may be an application programming interface (API) of LAS 104. Producer software of source system 102 may utilize the input adapter to send packets to LAS 104 and thus write the packets to persistent storage 112.

[0032] In some embodiments, the "push" producer software on the source system 102 can send the packet to the LAS 104 in a synchronous call. For example, the "push" producer software can call the input adapter of the LAS 104 to send the packet to the LAS 104, and thus write the packet to the permanent storage device 112. The "push" producer software can wait until the call succeeds or fails (e.g., receives a response from the LAS 104). If the call succeeds (e.g., the LAS 104 writes the packet to the permanent storage device 112), the "push" producer software can submit the packet read from the permanent storage device 110. In response to the submission, the source system 102 can release any associated locks on the permanent storage device 110, thereby reducing the lock duration of the permanent storage device 110. If the call fails (e.g., the LAS 104 fails to write the packet to the permanent storage device 112), the "push" producer software can retry calling the LAS 104 to write the packet to the permanent storage device 112. As will be understood by one of ordinary skill in the art, "push" producer software may be software that runs in its own execution environment (eg, outside of the LAS 104).

[0033] In some other embodiments, the "pull" producer software on the source system 102 may be called from an external execution environment (e.g., in the same environment as the LAS 104). For example, the input adapter of the LAS 104 may poll the "pull" producer software on the source system 102 for new data. In response, the "pull" producer software may send a packet representing the new data to the LAS 104, and thus write the packet to the persistent storage 112. If the call is successful (e.g., the LAS 104 writes the packet to the persistent storage 112), the "pull" producer software may commit the packet read from the persistent storage 110. In response to the commit, the source system 102 may release any associated locks on the persistent storage 110, thereby reducing the lock duration of the persistent storage 110. If the call fails (e.g., the LAS 104 fails to write the packet to the persistent storage 112), the "pull" producer may be re-invoked by the input adapter of the LAS 104. As will be understood by one of ordinary skill in the art, "pull" producer software may be software whose invocation occurs in an external execution environment.

[0034] After the LAS 104 receives the packet, the LAS 104 may write the packet to the persistent storage 112. If the LAS 104 successfully writes the packet to the persistent storage 112, the packet may be considered submitted by the source system 102 without risk of data loss. In other words, once the LAS 104 writes the packet to the persistent storage 112, the source system 102 may unlock the associated area of ​​the persistent storage 110 and read the next packet to be sent to the LAS 104. This enables the LAS 104 to begin processing another packet through the data pipeline 106 before the original packet is completely processed through the data pipeline 106. Thus, the LAS 104 may asynchronously process packets in sequence through the data pipeline 106.

[0035] The LAS 104 may append the received packets to an ordered packet list on the permanent storage device 112. The ordered packet list may store packets currently being processed by the data pipeline 106. The ordered packet list may be sorted as each packet is received from the source system 102. As will be appreciated by one of ordinary skill in the art, the ordered packet list may be sorted in a variety of other ways besides the time of receipt. The LAS 104 may maintain the ordered packet list using a variety of data structures such as, but not limited to, queues.

[0036] The LAS 104 may store each packet in an ordered list of packets with various metadata. The LAS 104 may store a subscription identifier for each packet. The subscription identifier may identify a set of logically associated packets. The LAS 104 may store a packet identifier for each packet. The packet identifier may identify the logical position of the packet among other packets in the ordered list of packets for processing by the data pipeline 106. The LAS 104 may store a state for each packet. The state may identify the processing state of the packet in the data pipeline 106. The state of the packet may be "submitted", "in progress", "rolled back", or "new". As understood by those of ordinary skill in the art, a packet may have various other states.

[0037] A state of "Submitted" may indicate that the corresponding packet has been submitted to the permanent storage 114 by the target system 108. A state of "In Progress" may indicate that the corresponding packet is currently being processed in the data pipeline 106 and has not yet been submitted to the permanent storage 114 by the target 108. A state of "Rolled Back" may indicate that the target 108 failed to submit the corresponding packet to the permanent storage 114 and therefore needs to be rebuilt. A state of "New" may indicate that the corresponding packet has not yet started to be processed by the data pipeline 106.

[0038] The LAS 104 may receive a commit notification 120 indicating that the package has been committed to the persistent storage 114 by the target system 108. The LAS 104 may receive the commit notification 120 via an output adapter (also referred to as LAS OUT) of the LAS 104. The output adapter of the LAS 104 may be an API.

[0039] LAS 104 may also receive rollback notifications 122 indicating a problem that occurred between source system 102 and target system 108 (eg, data failed to be committed to persistent storage 114 by target system 108 and therefore needs to be rebuilt). LAS 104 may receive rollback notifications 122 via an output adapter of LAS 104 .

[0040] LAS 104 may append received packets to the ordered packet list until the high water mark of the data is reached. This may prevent the ordered packet list from growing without limit and may allow producer software on source system 102 an opportunity to perform other work.

[0041] In some embodiments, LAS 104 can append received packages to an ordered package list until the number of packages contained in the ordered package list reaches a threshold value. For example, LAS 104 can append received packages to an ordered package list, and the number of packages in the ordered list is less than the threshold number of packages. In some other embodiments, LAS 104 can append received packages to an ordered package list until a fixed number of data is contained in the ordered package list. For example, LAS 104 can append received packages to an ordered package list, and the number of packages in the ordered list is less than the data of the threshold level. In some other embodiments, LAS 104 can append received packages to an ordered package list, until various other threshold values ​​are reached.

[0042] In some embodiments, LAS 104 may append received packets to an ordered list of packets until a global high water mark for the data is reached. The global high water mark for the data may be a threshold level for the data that is independent of whether the next received packet is part of the same logical set of packets (e.g., shares the same subscription identifier).

[0043] In some other embodiments, the LAS 104 may append received packets to an ordered list of packets until a subscription high water mark for the data is reached. The subscription high water mark for the data may be a threshold level of data specific to a subscription identifier.

[0044] The data pipeline 106 may sequentially process packets stored in the ordered packet list maintained by the LAS 104. The LAS 104 may send the next packet marked as "new" in the ordered packet list to the data pipeline 106. Thereafter, the LAS 104 may set the packet status to "in progress" in the ordered packet list.

[0045] In response to receiving the commit notification 120 from the target system 108, the LAS 104 may mark the status of the corresponding package as "committed" in the ordered package list. The LAS 104 may also delete the corresponding package from the permanent storage device 112. In some embodiments, in the case of attempting to submit a package to multiple target systems 108, the LAS 104 may mark the status of the corresponding package as "committed" in the ordered package list in response to receiving the commit notification 120 from each target system 108. The LAS 104 may then also delete the corresponding package from the permanent storage device 112.

[0046] In response to receiving a rollback notification 122 from the target system 108 (or data pipeline 106), the system 100 (e.g., via the LAS 104) can automatically discard all packets in the data pipeline 106 after the rolled back packet. This can prevent duplicates from being created in the persistent storage 114. This can further prevent inconsistencies due to sequential dependencies in the packets.

[0047] In response to receiving the rollback notification 122 from the target system 108 (or the data pipeline 106), the LAS 104 may also roll back and rebuild all subsequent "in progress" packets. In some embodiments, the LAS 104 may identify subsequent "in progress" packets as packets with an "in progress" status and a packet identifier (e.g., a larger packet identifier) ​​that is later than the time when the packet was rolled back. In some other embodiments, the LAS 104 may identify subsequent "in progress" packets as packets with an "in progress" status, a packet identifier (e.g., a larger packet identifier) ​​that is later than the time when the packet was rolled back, and a subscription identifier that is the same as the rolled back packet.

[0048] The LAS 104 may then generate new packets corresponding to all subsequent "in progress" packets. The LAS 104 may change the status of each of the generated packets to "rolled back". The LAS 104 may then sequentially send each of the generated packets to the data pipeline 106. The LAS 104 may change the status of each of the generated packets to "in progress" after sending the corresponding generated packet to the data pipeline 106.

[0049] Figure 2 is an example of an ordered packet list 200 maintained by LAS 104 according to some embodiments. Figure 1 To describe Figure 2 .

[0050] Ordered packet list 200 may include one or more packets received from source system 102. For example, ordered packet list 200 may include packet entries 210, 212, 214, 216, and 218.

[0051] In 200, each packet may include a subscription identifier 202, a packet identifier 204, a state 206, and data

[0052] 208. As will be appreciated by one of ordinary skill in the art, each packet in ordered packet list 200 may include various other types of fields.

[0053] Subscription identifier 202 can uniquely identify a subscription of a package. A subscription of a package can represent a group of logically related packages. For example, a subscription of a package can represent a group of packages associated with a specific user.

[0054] The package identifier 204 can uniquely identify the logical location of a package within a subscription to be processed by the data pipeline 106 and submitted to the target system 108 .

[0055] The status 206 may indicate the processing status of the packet. A status of "Submitted" may indicate that the corresponding packet has been submitted to the permanent storage 114 by the target system 108. A status of "In Progress" may indicate that the corresponding packet is currently being processed in the data pipeline 106 and has not yet been submitted to the permanent storage 114 by the target system 108. A status of "Rolled Back" may indicate that the target 108 failed to submit the corresponding packet to the permanent storage 114 and therefore needs to be rebuilt. A status of "New" may indicate that the corresponding packet has not yet been processed by the data pipeline 106. As will be appreciated by one of ordinary skill in the art, a packet may have a variety of other statuses.

[0056] Data 208 may represent a set of actual data changes that make up a package. For example, data 208 may represent a series of structured query language (SQL) statements. Data 208 may also contain the actual contents of the package. Data 208 may also contain references to the contents of the package. As will be appreciated by those of ordinary skill in the art, data 208 may contain various other types of data.

[0057] Ordered packet list 200 may include packet entries 210, 212, 214, 216, and 218. Each packet entry may represent a packet that has been processed, is being processed, or will be processed in data pipeline 106.

[0058] exist Figure 2 , package entry 210 represents package 0-0 of subscription 0 having data 0-0 that has been successfully submitted to target system 108. After LAS 104 begins processing data entry 210 through data pipeline 106, LAS 104 may have begun processing package entries 212 and 214. However, during the processing of package entries 212 and 214, a failure may have occurred that requires rolling back package entries 212 and 214. In other words, LAS 104 may have received rollback notification 122 from target system 108 (or data pipeline 106). Therefore, LAS 104 may have generated package entries 216 and 218 that contain the same data sets as package entries 212 and 214, respectively. LAS 104 may mark package entries 216 and 218 as "rolled back" to indicate that these package entries represent packages that have been rolled back.

[0059] The system 100 can ensure that no package can be submitted to the target system 108 unless all previous packages have been submitted. In other words, the system 100 can ensure that a rolled back package can only succeed through an "in progress" package that can also be rolled back.

[0060] Because the producer software on the source system 102 sends packets to the LAS 104 to be written to the persistent storage 112, and because the LAS 104 generates new packets for the rollback packets from the persistent storage 112, the system 100 can go back in time and rebuild (also called reprocess) any number of uncommitted packets in the event of a failure while minimizing the lock duration. The producer software on the source system 102 can also go forward in time by submitting packets and processing new packets while the previous packets have not been fully (or at all) processed by the data pipeline 106.

[0061] Figure 3 1 is a flow chart of a method 300 for utilizing LAS to ensure that the lock duration is minimized while reconstructing packets and that multiple packets can be processed simultaneously in a data pipeline according to one embodiment. The method 300 may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps are necessary to perform the disclosure provided herein. Furthermore, some steps may be performed simultaneously or in parallel. Figure 3 The different orders shown are performed as understood by one of ordinary skill in the art.

[0062] Method 300 is a reference Figure 1 and Figure 2 However, the method 300 is not limited to this exemplary embodiment.

[0063] In 302, the LAS 104 receives a package from the source system 102 that may be processed by the data pipeline 106. The package may include a set of changes to be fully applied in the target system 108 or not applied at all if a problem occurs between the source system 102 and the target system 108.

[0064] The LAS 104 can receive packets from the producer software on the source system 102 via an input adapter (also referred to as LAS IN) of the LAS 104. The input adapter can be an API. In some embodiments, the LAS 104 can receive packets from the producer software in a synchronous call. In some other embodiments, the input adapter of the LAS 104 can poll the producer software to obtain new data. In response, the producer software can send packets representing new data to the LAS 104.

[0065] At 304, the LAS 104 stores the packet and its corresponding packet state in the persistent storage 112. The LAS 104 may write the received packet to an ordered list of packets (e.g., the ordered list of packets 200) in the persistent storage 112 being processed by the data pipeline 106. For example, the LAS 104 may append the received packet to the ordered list of packets in the persistent storage 112. The LAS 104 may maintain the ordered list of packets using various data structures such as, but not limited to, queues.

[0066] The LAS 104 may store each packet in an ordered list of packets along with various metadata. The LAS 104 may store a subscription identifier for each packet. The subscription identifier may identify a set of logically associated packets. The LAS 104 may store a packet identifier for each packet. The packet identifier may identify the logical position of the packet among other packets in the ordered list of packets processed by the data pipeline 106. The LAS 104 may store a state for each packet. The state may identify the processing state of the packet in the data pipeline 106. The state of the packet may be "submitted", "in progress", "rolled back", or "new". As understood by those of ordinary skill in the art, a packet may have various other states.

[0067] In some embodiments, the LAS 104 may append received packets to the ordered packet list until a high water mark for data is reached. In some embodiments, the LAS 104 may append received packets to the ordered packet list until the number of packets contained in the ordered packet list reaches a threshold value. In some other embodiments, the LAS 104 may append received packets to the ordered packet list until a fixed amount of data is contained in the ordered packet list. In some other embodiments, the LAS 104 may append received packets to the ordered packet list until various other threshold values ​​are reached.

[0068] In some embodiments, the LAS 104 may append received packets to the ordered packet list until a global high water mark for the data is reached. The global high water mark for the data may be a threshold level for the data that is independent of whether the next received packet is part of the same logical set of packets (e.g., shares the same subscription identifier). In some other embodiments, the LAS 104 may append received packets to the ordered packet list until a subscription high water mark for the data is reached. The subscription high water mark for the data may be a threshold level for the data that is specific to the subscription identifier.

[0069] In 306, the LAS 104 sends the packet to the data pipeline 106. The LAS 104 may send the packet to the data processing step 107 of the data pipeline 106. The data processing step 107 may perform any set of operations on the packet. The data processing step 107 may be executed on the source system 102, the target system 108, or an intermediate system between the source system 102 and the target system 108. The data processing step 107 may be executed on a desktop computer, a laptop computer, a tablet, a smart phone, or other device as understood by one of ordinary skill in the art.

[0070] In response to the transmission, the LAS 104 may update the status of the packet in the ordered list of packets in the persistent storage 112. For example, the LAS 104 may set the status of the packet to "in progress".

[0071] In 308, the LAS 104 receives a commit notification 120 or a rollback notification 122. The LAS 104 may receive the commit notification 120 or the rollback notification 122 via an output adapter (also referred to as LAS OUT) of the LAS 104. The output adapter of the LAS 104 may be an API. If the LAS 104 receives the commit notification 120, the method 300 proceeds to 310. If the LAS 104 receives the commit notification 120, the method 300 proceeds to 312.

[0072] In 310, the LAS 104 deletes the package from the persistent storage 112 in response to receiving the commit notification 120. The LAS 104 may mark the package as "committed" in the ordered package list in response to receiving the commit notification 120. The LAS 104 may also delete the corresponding package from the persistent storage 112.

[0073] In some embodiments, in the case of attempting to submit a package to multiple target systems 108, the LAS 104 may mark the status of the corresponding package as "submitted" in the ordered package list in response to receiving the submission notification 120 from each target system 108. The LAS 104 may then also delete the corresponding package from the permanent storage device 112.

[0074] In 312, LAS 104 identifies all subsequent "in progress" packages in persistent storage 112 in response to receiving rollback notification 122. In some embodiments, LAS 104 may identify subsequent "in progress" packages as packages having an "in progress" status and a package identifier (e.g., a larger package identifier) ​​that is received later than the time when the package was rolled back. In some other embodiments, LAS 104 may identify subsequent "in progress" packages as packages having an "in progress" status, a package identifier (e.g., a larger package identifier) ​​that is received later than the time when the package was rolled back, and a subscription identifier that is the same as the rolled back package.

[0075] In some embodiments, system 100 (eg, via LAS 104) may automatically discard all packets after rolling back a packet in data pipeline 106. This may prevent duplicates from being created in persistent storage 114. This may further prevent inconsistencies from occurring due to sequential dependencies in packets.

[0076] In 314, the LAS 104 generates new packages corresponding to all subsequent "in progress" packages. In some embodiments, the LAS 104 may set the status of each of the generated packages to "rolled back".

[0077] In 316, the LAS 104 sequentially sends the generated packets to the data pipeline 106. The LAS 104 may change the status of each of the generated packets to "in progress" after sending the corresponding generated packet to the data pipeline 106.

[0078] For example, various embodiments may utilize one or more well-known computer systems, such as Figure 4 For example, one or more computer systems 400 may be used to implement any of the embodiments discussed herein and any combinations and sub-combinations thereof. Computer system 400 may be used, for example, to implement Figure 3 Method 300 in .

[0079] Computer system 400 may include one or more processors (also referred to as central processing units or CPUs), such as processor 404. Processor 404 may be connected to a communications infrastructure or bus 406.

[0080] The computer system 400 may also include user input / output device(s) 403 , such as a monitor, keyboard, pointing device, etc., which may communicate with the communication infrastructure 406 through the user input / output interface(s) 402 .

[0081] One or more processors 404 may be a graphics processing unit (GPU). In one embodiment, a GPU may be a processor that is a specialized electronic circuit designed to process math-intensive applications. A GPU may have a parallel architecture that is efficient for parallel processing of large blocks of data, such as math-intensive data commonly used for computer graphics applications, images, videos, and the like.

[0082] Computer system 400 may also include main memory or primary memory 408, such as random access memory (RAM). Main memory 408 may include one or more levels of cache. Main memory 408 may store control logic (ie, computer software) and / or data therein.

[0083] The computer system 400 may also include one or more secondary storage devices or memories 410. The secondary storage 410 may include, for example, a hard disk drive 412 and / or a removable storage device or drive 414. The removable storage drive 414 may be a floppy disk drive, a tape drive, an optical drive, an optical storage device, a tape backup device, and / or any other storage device / drive.

[0084] Removable storage drive 414 may interact with removable storage unit 418. Removable storage unit 418 may include a computer usable or readable storage device having computer software (control logic) and / or data stored thereon. Removable storage unit 418 may be a floppy disk, a magnetic tape, an optical disk, a DVD, an optical storage disk, and / or any other computer data storage device. Removable storage drive 414 may read from and / or write to removable storage unit 418.

[0085] Secondary memory 410 may include other devices, apparatuses, components, instruments, or other methods for enabling computer system 400 to access computer programs and / or other instructions and / or data. Such devices, apparatuses, components, instruments, or other methods may include, for example, a removable storage unit 422 and an interface 420. Examples of removable storage unit 422 and interface 420 may include a program cartridge and cartridge interface (such as found in video game devices), a removable storage chip (e.g., EPROM or PROM) and associated sockets, a memory stick and USB port, a memory card and associated memory card slot, and / or any other removable storage unit and related interface.

[0086] The computer system 400 may also include a communication or network interface 424. The communication interface 424 may enable the computer system 400 to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively represented by reference numeral 428). For example, the communication interface 424 may enable the computer system 400 to communicate with an external or remote device 428 via a communication path 426, which may be wired and / or wireless (or a combination thereof) and may include a LAN, a WAN, the Internet, etc. Control logic and / or data may be sent to and from the computer system 400 via the communication path 426.

[0087] The computer system 400 may also be, to name a few non-limiting examples, a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet, a smart phone, a smart watch or other wearable device, a home appliance, part of the Internet of Things, and / or an embedded system, or any combination thereof.

[0088] Computer system 400 can be a client or server, accessing or hosting any applications and / or data through any delivery paradigm, including but not limited to remote or distributed cloud computing solutions; local or on-site software ("on-site" cloud-based solutions); "as a service" models (e.g., content as a service (CaaS), digital content as a service (DCaaS), software as a service (SaaS), hosted software as a service (MSaaS), platform as a service (PaaS), desktop as a service (DaaS), framework as a service (FaaS), backend as a service (BaaS), mobile backend as a service (MBaaS), infrastructure as a service (IaaS), etc.); and / or a hybrid model including any combination of the above examples or other services or delivery paradigms.

[0089] Any applicable data structures, file formats, and schemes in computer system 400 may be derived from various standards, including but not limited to JavaScript Object Notation (JSON), Extensible Markup Language (XML), another markup language (YAML), Extensible Hypertext Markup Language Standard (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representations alone or in combination. Alternatively, proprietary data structures, formats, or schemes may be used alone or in combination with known or open standards.

[0090] In some embodiments, a tangible, non-transitory device or article of manufacture including a tangible, non-transitory computer-usable or readable medium having control logic (software) stored thereon may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 400, main memory 408, auxiliary memory 410, and removable storage units 418 and 422, and tangible articles of manufacture implementing any combination of the foregoing. When executed by one or more data processing devices (e.g., computer system 400), such control logic may cause such data processing devices to generally operate as described herein.

[0091] Based on the teachings contained in this disclosure, it will be clear to those skilled in the relevant art(s) how to use the Figure 4 The embodiments of the present disclosure may be made and used using data processing devices, computer systems, and / or computer architectures other than those shown in the drawings. Specifically, the embodiments may operate using software, hardware, and / or operating system implementations other than those described herein.

[0092] It should be understood that the detailed description section, but not any other section, is intended to be used to interpret the claims. The other sections may set forth one or more but not all exemplary embodiments contemplated by the inventor(s), and therefore, are not intended to limit the present disclosure or the appended claims in any way.

[0093] Although exemplary embodiments of exemplary fields and applications are described in this disclosure, it should be understood that the disclosure is not limited thereto. Other embodiments and modifications thereof are possible and are within the scope and spirit of the disclosure. For example, but not limiting the generality of this paragraph, the embodiments are not limited to the software, hardware, firmware, and / or entities shown in the drawings and / or described herein. In addition, the embodiments (whether or not explicitly described herein) have important utility for fields and applications beyond the examples described herein.

[0094] The embodiments have been described with the aid of functional building blocks that illustrate implementation schemes for specific functions and relationships thereof. For ease of description, the boundaries of these functional building blocks have been arbitrarily defined herein. Alternative boundaries may be defined as long as the specified functions and relationships (or their equivalents) are properly performed. In addition, alternative embodiments may perform functional blocks, steps, operations, methods, etc., in an order different from that described herein.

[0095] "One embodiment", "embodiment", "exemplary embodiment" or similar phrases mentioned herein indicate that the described embodiments may include specific features, structures or characteristics, but each embodiment does not necessarily include specific features, structures or characteristics. In addition, these phrases do not necessarily refer to the same embodiment. In addition, when describing specific features, structures or characteristics in conjunction with an embodiment, whether or not explicitly mentioned or described herein, such features, structures or characteristics are incorporated into other embodiments within the knowledge of technicians in related (multiple) fields. In addition, the expressions "coupled" and "connected" together with their derivatives can be used to describe some embodiments. These terms are not necessarily synonymous with each other. For example, the terms "connected" and / or "coupled" can be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0096] The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A computer-implemented method for enabling reconstruction of packets processed through a data pipeline, the method comprising: receiving, by at least one processor, a package from a source system for processing through the data pipeline, wherein the package includes a set of changes to be submitted to a target system; storing, by the at least one processor, the packet along with a corresponding packet state in a persistent storage device; sending, by the at least one processor, the packet to the data pipeline in response to the storing; In response to the sending, receiving, by the at least one processor, a submission notification for the package from the target system; as well as in response to receiving the submission notification of the package, deleting, by the at least one processor, the package from the persistent storage; Wherein, the package is the first package, and further comprises: receiving, by the at least one processor, a rollback notification of the second package; identifying, by the at least one processor, a set of subsequent packets in the persistent storage associated with the second packet, wherein each packet in the set of subsequent packets has a corresponding in-progress packet status; generating, by the at least one processor, a set of packets in the persistent storage device based on the identified set of subsequent packets; and sending, by the at least one processor, each packet in the generated set of packets to the data pipeline; Wherein the storage further comprises: storing, by the at least one processor, the packet in the persistent storage device based on the persistent storage device containing data below a data threshold level, wherein the data threshold level is associated with a subscription identifier, and wherein the subscription identifier identifies a group of packages to which the package is logically related; and The subscription identifier is the same as the subscription identifier of the rolled back package.

2. The method of claim 1, wherein the receiving further comprises: The packet is received, by the at least one processor, via a synchronous call from producer software on the source system.

3. The method of claim 1, wherein the receiving further comprises: The packet is received, by the at least one processor, via an application programming interface (API).

4. The method according to claim 1, further comprising: The packet state of the packet in the persistent storage is updated, with the at least one processor, in response to the sending.

5. A system for enabling reconstruction of packets processed through a data pipeline, comprising: Memory; as well as at least one processor coupled to the memory and configured to: receiving a package from a source system for processing through a data pipeline, wherein the package includes a set of changes to be submitted to a target system; storing the package together with the corresponding package state in a permanent storage device; In response to the storing, sending the packet to the data pipeline; In response to the sending, receiving a submission notification for the package from the target system; and in response to receiving the submission notification of the package, deleting the package from the permanent storage; wherein the packet is a first packet, and wherein the at least one processor is further configured to: receiving a rollback notification of the second package; identifying a set of subsequent packets in the persistent storage associated with the second packet, wherein each packet in the set of subsequent packets has a corresponding in-progress packet state; generating a set of packets in the persistent storage device based on the identified set of subsequent packets; and sending each packet in the generated set of packets to the data pipeline; Wherein, for storage, the at least one processor is further configured to: storing the packet in the persistent storage based on the persistent storage containing data below a data threshold level, wherein the data threshold level is associated with a subscription identifier, and wherein the subscription identifier identifies a group of packages to which the package is logically related; and The subscription identifier is the same as the subscription identifier of the rolled back package.

6. The system of claim 5, wherein to receive, the at least one processor is further configured to: The packet is received via a synchronous call from producer software on the source system.

7. The system of claim 5, wherein to receive, the at least one processor is further configured to: The packet is received via an application programming interface API.

8. The system of claim 5, wherein the at least one processor is further configured to: The packet state of the packet in the persistent storage is updated in response to the sending.

9. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: receiving a package from a source system for processing through a data pipeline, wherein the package includes a set of changes to be submitted to a target system; storing the package together with the corresponding package state in a permanent storage device; In response to the storing, sending the packet to the data pipeline; In response to the sending, receiving a submission notification for the package from the target system; as well as In response to receiving the submission notification of the package, deleting the package from the permanent storage device Wherein, the packet is a first packet, and the operations further include: receiving a rollback notification of the second package; identifying a set of subsequent packets in the persistent storage associated with the second packet, wherein each packet in the set of subsequent packets has a corresponding in-progress packet state; generating a set of packets in the persistent storage device based on the identified set of subsequent packets; and sending each packet in the generated set of packets to the data pipeline; Wherein the storage further comprises: storing the packet in the persistent storage based on the persistent storage containing data below a data threshold level, wherein the data threshold level is associated with a subscription identifier, and wherein the subscription identifier identifies a group of packages to which the package is logically related; and The subscription identifier is the same as the subscription identifier of the rolled back package.

10. The non-transitory computer readable medium of claim 9, wherein the receiving further comprises: The packet is received via a synchronous call from producer software on the source system.

11. The non-transitory computer readable medium of claim 9, wherein the receiving further comprises: The packet is received via an application programming interface API.

12. The non-transitory computer readable medium of claim 9, wherein the operations further comprise: The packet state of the packet in the persistent storage is updated in response to the sending.

Images