Method and system for continuous data protection with large key- value store

Abstract

A method of grouping together Continuous Data Protection (CDP) stream entries into a plurality of buffers, where each buffer stores consecutive CDP stream entries such that the entries are sorted in order in each buffer so that an earlier write entry in the CDP stream precedes a write entry which is dependent upon the earlier write entry. Furthermore, indexing each of the CDP Stream entries, in CDP stream order, in the plurality of buffers using a Large Key Value Store (LKVS). Furthermore, as a result of the indexing, identifying CDP Stream entries which have a valid entry in the LKVS as data which has been changed. Thereafter, marking a previous buffer where the data which has been changed has been located, as being stale data.

Description

[0001] METHOD AND SYSTEM FOR CONTINUOUS DATA PROTECTION WITH LARGE KEY- VALUE STORE

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of data management and more specifically, to a method, and a system for continuous data protection with a large key-value store.

[0004] BACKGROUND

[0005] A backup storage deploying continuous data protection (CDP), is used for critical applications where data loss is deemed unacceptable, even in the event of primary storage failure. However, the use of CDP imposes an inherent cost on a host that is required to transfer every write to both a target and to the backup storage, resulting in duplicate updates of the same data. Data recovery from CDP streams includes addressing duplicate updates that require parsing and implementing redundant writes. Furthermore, the CDP includes hotspots, such as updated metadata, adding complexity for storing all the writes that are required during the data recovery. Moreover, the data recovery process often requires replaying the CDP stream until a desired restoration time objective is achieved that further increases overall time.

[0006] Currently, certain attempts have been made to implement the CDP with reduced complexity and overall processing time, such as by combining a host input-output (IO) splitter for capturing an IO to a virtual disk of a virtual machine, integrating database that includes all the writes to a system, utilizing a combination of full and incremental backups, using tiered storages, and the like. However, these attempts fail due to a lack of optimization in underlying storage architecture, resulting in undesirable data overlap between CDP streams and snapshots. Additionally, conventional methods and systems for implementing the CDP lack granularity for performing restoration operations beyond the CDP range. This necessitates complete coverage and has the potential to elongate the restoration process due to duplicate writes, which is undesirable. Thus, there exists a technical problem of how to implement the CDP with reduced processing time and complexity with minimized resource overhead.

[0007] Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with the conventional methods and systems for implementing the CDP.

[0008] SUMMARY

[0009] The present disclosure provides a method, a system and a computer program for continuous data protection (CDP) with a large key-value store. The present disclosure provides a solution to the existing problem of how to implement the CDP with reduced processing time and complexity with minimized resource overhead. An objective of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in the prior art and provides an improved method and an improved system for continuous data protection.

[0010] One or more objectives of the present disclosure are achieved by the solutions provided in the enclosed independent claims. Advantageous implementations of the present disclosure are further defined in the dependent claims.

[0011] In one aspect, the present disclosure provides a method of grouping together Continuous Data Protection (CDP) stream entries into a plurality of buffers, where each buffer stores consecutive CDP stream entries such that the entries are sorted in order in each buffer so that an earlier write entry in the CDP stream precedes a write entry which is dependent upon the earlier write entry. Furthermore, the method includes indexing each of the CDP Stream entries, in CDP stream order, in the plurality of buffers using a Large Key-Value Store (LKVS). As a result of the indexing, identifying CDP Stream entries that have a valid entry in the LKVS as data which has been changed, and marking a previous buffer where the data which has been changed has been located, as being stale data.

[0012] Advantageously, the method is used to provide an efficient, optimized, and effective management and organization of the continuous data protection (CDP) stream entries by grouping the CDP stream entries into the plurality of buffers with sequential sorting. The CDP stream entries are grouped into the plurality of buffers by using the LKVS for maintaining the ease of data accessibility. Furthermore, the indexing of each of the CDP stream entries is beneficial for maintaining order of the CDP stream to provide enhanced and efficient data retrieval and management within the CDP. Thereafter, the identification of the CDP stream entries that possess a valid CDP stream entry in the LKVS, signifying data that has changed ensures precise tracking and recognition of the altered data. Furthermore, the marking of the previous buffer where the data has been changed has been located, as the stale data, ensures efficient data management and data integrity. The marking of the previous buffer as the stale data is achieved by flagging and distinguishing previous buffers containing modified information as stale, thereby facilitating streamlined data processing. As a result, the method is used to provide enhanced, efficient, comprehensive data management and data retrieval in the CDP with optimized CDP backup and reduced recovery time objective, reduced resource consumption, and reduced cost utilization.

[0013] In an implementation form, the LKVS is a log-structured merge (LSM) tree.

[0014] In such an implementation, the LSM tree is utilized for optimized storage utilization, efficient data retrieval, and controlled write amplification that further ensures an optimized CDP backup.

[0015] In another implementation form, the LKVS is a B+ tree.

[0016] Advantageously, the B+ tree is used to provide an efficient and predictable access to the key-value pairs that facilitate the execution of data retrieval operations for large amount of data in an optimized and reliable manner.

[0017] In another implementation form, the indexing pushes updates into the LKVS.

[0018] In such an implementation, the pushing of the updates into the LKVS during indexing ensures an up-to-date representation of the data to provide continuous data protection and an accurate and reliable execution of the data retrieval operation.

[0019] In another implementation form, the indexing pushes only a reference to a respective buffer and an offset inside the respective buffer.

[0020] Advantageously, the pushing of only the reference and the offset inside the respective buffer during indexing leads to an efficient resource utilization, which is further used for managing and retrieving the data, especially during execution of CDP operations.

[0021] In another implementation form, the method further includes the step of performing CDP stream compaction of buffers that contain stale data entries while maintaining the CDP stream order.

[0022] In such an implementation, the CDP stream compaction is used to improve resource consumption while backing up the data, reducing storage requirements, optimizing retention policy for the CDP stream, and reducing the write amplification.

[0023] In another aspect, the present disclosure provides a system comprising means adapted for carrying out all the steps of the method.

[0024] The system achieves all the advantages and technical effects of the method of the present disclosure. In another aspect, the present disclosure provides a computer program comprising instructions for carrying out all the steps of the method when the computer program is executed on a computer system.

[0025] It is to be appreciated that all the aforementioned implementation forms can be combined.

[0026] It has to be noted that all devices, elements, circuitry, units, and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof. It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims.

[0027] Additional aspects, advantages, features, and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative implementations construed in conjunction with the appended claims that follow.

[0028] BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers.

[0030] Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein:

[0031] FIG. 1 is a block diagram that depicts a system configured to optimize continuous data protection with a large key-value store, in accordance with an embodiment of the present disclosure;

[0032] FIG. 2 is a flowchart depicting a method for optimizing continuous data protection with a large key-value store, in accordance with an embodiment of the present disclosure;

[0033] FIG. 3 is an exemplary diagram depicting execution of a CDP stream compaction by LKVS, in accordance with an embodiment of the present disclosure;

[0034] FIG. 4 is an exemplary diagram depicting the execution of the continuous data protection, in accordance with another embodiment of the present disclosure;

[0035] FIG. 5 is an exemplary diagram that depicts the continuous data protection by using a log-structured merge tree-based large key-value store, in accordance with an embodiment of the present disclosure;

[0036] FIG. 6 is an exemplary diagram depicting a continuous data protection layout and indexing of continuous data protection stream entries, in accordance with an embodiment of the present disclosure; FIG. 7 is an exemplary diagram depicting a merging of old write-ahead logs with multiple invalid entries, in accordance with an embodiment of the present disclosure;

[0037] FIG. 8 is an exemplary diagram depicting B+ tree-based large key-value store implementation for indexing of the CDP stream entries, in accordance with an embodiment of the present disclosure; and

[0038] FIG. 9 is an exemplary diagram depicting a continuous data protection-based backup, in accordance with an embodiment of the present disclosure.

[0039] In the accompanying drawings, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the non-underlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing.

[0040] DETAILED DESCRIPTION OF EMBODIMENTS

[0041] The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

[0042] FIG. 1 is a block diagram that depicts a system configured to optimize continuous data protection with a large key-value store, in accordance with an embodiment of the present disclosure. With reference to FIG.1, there is shown a block diagram 100 that includes a system 102. The system 102 includes a processor 104, a plurality of buffers 106, a memory 110, and a network interface 112.

[0043] The system 102 is configured to provide an optimized continuous data protection (CDP) with a large key-value store (LKVS) 114. The system 102 is used to provide data accessibility at any point of time, which is required for data retrieval and data recovery operations with reduced overall data recovery time and reduced complexity. Moreover, the CDP refers to a form of backup that saves every change, which is performed on the data and allows a user to access the data at any point of time through the LKVS 114. The LKVS 114 refers to a type of key-value store, which is neither volatile nor resides in memory and is used to save the data in the form of key-value pairs so that given a key, the store will return the value saved by a user. The LKVS 114 utilizes standard operations that include an ADD operation (i.e., an operation to create a new key-value pair), an UPDATE operation (i.e., an operation to update the value of a previously inserted key), and a DELETE operation (i.e., an operation to delete the key-value pair) to maintain data consistency and data availability for an efficient, effective, and reliable data storage.

[0044] The processor 104 is configured to execute all necessary operations of the system 102. Examples of the processor 104 may include, but are not limited to, a microcontroller, a microprocessor, a central processing unit (CPU), a complex instruction set computing (CISC) processor, an application-specific integrated circuit (ASIC) processor, a reduced instruction set (RISC) processor, a very long instruction word (VLIW) processor, a data processing unit, and other processors or control circuitry.

[0045] The plurality of buffers 106 includes a first buffer 106A, a second buffer 106B, and up to Nlbuffer 106N and each of the buffers includes a group of CDP stream entries. For example, the first buffer 106A includes a first CDP stream entry 108A, a second CDP stream entry 108B and up to NthCDP stream entry 108N. Similarly, the second buffer 106B may also include a group of CDP stream entries that are used for describing CDP operations, which are performed on the keys of the LKVS 114. The memory 110 is configured to store the instructions executable by the processor 104. Examples of the memory 110 may include but are not limited to, an Electrically Erasable Programmable Read-Only Memory (EEPROM), Random Access Memory (RAM), Read-Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, Solid-State Drive (SSD), persistent memory, remote direct memory access (RD MA), or CPU cache memory.

[0046] The network interface 112 may include a hardware or software that is configured to establish communication among the processor 104, the plurality of buffers 106, and the memory 110. Examples of the network interface 112 may include but are not limited to, a computer port, a network socket, a network interface controller (NIC), and any other network interface device.

[0047] There is provided the system 102 that is configured to group together the CDP stream entries into the plurality of buffers 106, where each buffer stores consecutive CDP stream entries such that the entries are sorted in order in each buffer so that an earlier write entry in the CDP stream precedes a write entry which is dependent upon the earlier write entry. In an example, the system 102 may be configured to group the first CDP stream entry 108A, the second CDP stream entry 108B, and up to the NthCDP stream entry 108N together in the first buffer 106A. Similarly, the second buffer 106B, and the Nlbuffer 106N may also include a group of CDP stream entries where each buffer stores consecutive CDP stream entries such that the entries are sorted in order in each buffer so that an earlier write entry in the CDP stream precedes a write entry which is dependent upon the earlier write entry. Moreover, the CDP stream entries describe CDP operations (that are performed on a key or a set of keys, such as the keys that are stored in the LKVS 114. In an example, the CDP operations correspond to writes to an offset and a length of the corresponding writes. The offset and the length are added to the LKVS 114, such as through an ADD operation and if in case, the offset and the length already exist in the LKVS 114, then, in that case, the LKVS 114 is updated accordingly. Moreover, when deleting the CDP entry due to retention, the offset is required to be removed from the tree. However, in the case of CDC operations (e.g., ADD, UPDATE, and DELETE operations), these operations are required to be added to a logical key-value map (LKVM). Firstly, the system 102 is configured to identify dependencies and relationships between the CDP stream entries. Thereafter, the system 102 is configured to group the CDP stream entries based on the identified dependencies and relationships. As the entries are grouped together into the plurality of buffers 106 (e.g., the first buffer 106A, the second buffer 106B, and the like), the system 102 ensures that the order of the CDP stream entries within each buffer reflects the consecutive sequence of the CDP stream entries, such as, the earlier CDP stream entries are placed at the beginning of the buffer and after that, the dependent CDP stream entries are placed accordingly. For example, the first CDP stream entry 108A may be placed before the second CDP stream entry 108B. Moreover, such ordering of the CDP stream entries is used to enhance the data management for handling the CDP stream entries efficiently, thereby ensuring a logical and sequential representation of data changes. The ordering of the CDP stream entries in each of the buffers from the plurality of buffers 106 is used to handle the CDP operations on the CDP stream entries that are required to be performed in order to update the data continuously while ensuring data recovery and data backup. Additionally, the ordering of the CDP stream entries is beneficial to optimize the overall performance of the system 102, especially during the continuous data changes in a reduced time period that is required by the system 102 for data recovery after any malfunction.

[0048] Furthermore, the system 102 is configured to index each of the CDP stream entries, in CDP stream order, in the plurality of buffers 106 using the LKVS 114. The indexing of each CDP stream entry involves creating a structured reference for each CDP stream entry from the CDP stream entries. For example, the first CDP stream entry 108A, the second CDP stream entry 108B and up to the NthCDP stream entry 108N are indexed using the LKVS 114. Moreover, the indexing of the CDP stream entries is performed in the same order in which the CDP stream entries are grouped in each buffer allowing further marking of the stale buffers from the plurality of buffers 106. Additionally, the indexing of each of the CDP stream entries can be used to create a complete CDP-stream-based backup in order to control write amplification. The control on the write amplification reduces the overall data storage cost. Therefore, by utilizing the LKVS 114, the CDP entries are easily accessible due to which the processing time, which is required to retrieve, update, and delete the data, is reduced along with the reduced cost utilization. As a result, the overall performance of the system 102 for the CDP is improved. In accordance with an embodiment, the LKVS 114 is a log-structured merge (LSM) tree. The LSM tree refers to a structure that includes multiple levels, with each level storing a large amount of data. Moreover, the LSM tree structure for the LKVS 114 is used to execute write-intensive operations. The LSM tree minimizes the number of disk input / output operations for WRITE operations, thereby enhancing the overall performance of the system 102. An example of the LSM tree is further described in detail, for example, in FIG. 4. As a result, the LSM tree is utilized for optimized storage utilization, efficient data retrieval, and controlled write amplification that further ensures an optimized CDP backup.

[0049] In accordance with an embodiment, the LKVS 114 is a B+ tree. The B+ tree corresponds to a type of self-balancing tree data structure, widely used to organize, and store key-value pairs, facilitating an efficient data retrieval with reduced overall processing time. Moreover, the B+ tree is utilized to maintain a balanced structure, ensuring consistent performance for various CDP operations, such as ADD operation, DELETE operation, and the like. An example of the LSM tree is further described in detail, for example, in FIG. 4. Advantageously, the B+ tree is used to provide an efficient and predictable access to the keyvalue pairs that facilitate the execution of data retrieval operation for large amount of data in an optimized and reliable manner.

[0050] In accordance with an embodiment, the index pushes updates into the LKVS 114. In other words, the updates are pushed to the LKVS 114, for example, through the UPDATE operation. As a result, the pushing of the updates into the LKVS 114 during indexing ensures an up-to-date representation of the data to provide the continuous data protection and an accurate and reliable execution of the data retrieval operation.

[0051] In accordance with an embodiment, the index pushes only a reference to a respective buffer and an offset inside the respective buffer. In an example, the index pushes the reference to the first buffer 106A and the offset inside the first buffer 106A. Similarly, the index pushes the reference to the second buffer 106B and the offset inside the second buffer 106B. Furthermore, the LKVS 114 implements the index of the respective buffers by using the CDP stream entry key, and the CDP stream entry location (i.e., Write Ahead Log (WAL) buffer ID and offset) as value. By utilizing the reference and the offset of the respective buffer, the index can be used to point the exact location of the data within the plurality of buffers 106. Advantageously, the pushing of only the reference and the offset inside the respective buffer during indexing leads to efficient resource utilization, which is further used for managing and retrieving the data, especially during the execution of the CDP operations, leading to an improved overall performance of the system 102.

[0052] Furthermore, as a result of the indexing, the system 102 is configured to identify the CDP stream entries, which have a valid entry in the LKVS as data, which has been changed. Firstly, the system 102 is configured to group together the CDP stream entries into the plurality of buffers 106. After that, the system 102 is configured to index each of the CDP stream entries in the stream order, creating a data structure that maps the CDP Stream entries to their corresponding entries in the LKVS 114. Moreover, such indexing allows the identification of the CDP stream entries that have a valid match in the LKVS 114, such as by verifying each of the entries in the LKVS 114 whenever the data is changed (e.g., updated, added, or deleted). In other words, the system 102 is configured to identify the CDP stream entries with valid entries in the LKVS 114 that acts as an indicative of the data changes to allow cross-referencing of the CDP stream entries with the corresponding entries in the LKVS 114. Advantageously, the indexing of the CDP stream entries with the valid entry is used to maintain the overall data consistency, data integrity, and data reliability.

[0053] Furthermore, the system 102 is configured to mark a previous buffer where the data which has been changed has been located, as being stale data. The stale data refers to the data that no longer reflects the current (or valid) data of the system 102. By marking the previous buffer, the system 102 is configured to distinguish between current and outdated data, maintaining accuracy and integrity, such as by eliminating the use of the outdated data, contributing to overall data consistency and data reliability. In accordance with an embodiment, the system 102 is further configured to perform the step of CDP stream compaction of buffers that contain stale data entries while maintaining the CDP stream order. The CDP stream compaction refers to a removal of duplication within the CDP stream, such as by considering a last UPDATE operation, which is performed on the corresponding CDP stream entry, or by considering the DELETE operation, which is performed to remove any of the ADD or UPDATE operations on the corresponding CDP stream entry. The LKVS 114 utilizes a standard key-value store semantics to support operations, such as the ADD operation, the UPDATE operation, and the DELETE operation. However, the order of the CDP stream entries does not change after the execution of the CDP stream compaction of the buffers. The CDP stream can be applied to the LKVS 114, which transforms a redundant CDP stream into a compacted form including the unique data. Furthermore, the compaction of the CDP entries allows the creation of an incremental backup even if the incremental backup is not available, for example, there exists no way to expose the differences between a source image and two timestamps or when a target CDP stream format cannot be naively applied to the source image, such as in the case of a change data capture (CDC) stream. As a result, the CDP stream compaction is used to further improve resource consumption while backing up the data, reducing storage requirements, optimizing retention policy for the CDP stream, and reducing the write amplification on the storage of the system 102, thereby improving the overall performance of the system 102.

[0054] Advantageously, the system 102 is configured to provide an efficient, optimized, and effective management of the continuous data protection (CDP) stream entries by grouping the CDP stream entries while using the LKVS. Furthermore, the indexing of each of the CDP stream entries is beneficial for maintaining the order of the CDP stream to provide enhanced and efficient data retrieval and management within the CDP. Thereafter, the identification of the CDP stream entries that possess a valid CDP stream entry in the LKVS, signifying data that has changed ensures precise tracking and recognition of the altered data. Furthermore, the marking of the previous buffer where the data has been changed has been located, as the stale data ensures efficient data management and data integrity by flagging and distinguishing previous buffers containing modified information as stale, thereby facilitating streamlined data processing. As a result, the system 102 is configured to provide enhanced, efficient, comprehensive data management and data retrieval in the CDP with optimized CDP backup and reduced recovery time objective, reduced resource consumption, and reduced cost utilization.

[0055] FIG. 2 is a flowchart depicting a method for optimizing continuous data protection with a large key-value store, in accordance with an embodiment of the present disclosure. With reference to FIG. 2, there is shown a flowchart of a method 200 optimizing the continuous data protection (CDP) with the large key-value store. The method 200 includes steps 202 to 210.

[0056] At step 202, the method 200 includes grouping together CDP stream entries into the plurality of buffers 106, where each buffer stores consecutive CDP stream entries such that the entries are sorted in order in each buffer so that an earlier write entry in the CDP stream precedes a write entry which is dependent upon the earlier write entry. In an example, the first CDP stream entry 108A, the second CDP stream entry 108B, and up to the NlCDP stream entry 108N are grouped together in the first buffer 106A. Similarly, the second buffer 106B, and the Nthbuffer 106N may also include a group of CDP stream entries where each buffer stores consecutive CDP stream entries such that the entries are sorted in order in each buffer so that an earlier write entry in the CDP stream precedes a write entry which is dependent upon the earlier write entry. Moreover, the CDP stream entries describe CDP operations (e.g., ADD operation, UPDATE operation, DELETE operation) that are performed on a key or a set of keys, such as the keys that are stored in the LKVS 114. Firstly, dependencies and relationships between the CDP stream entries are identified. Thereafter, the grouping of the CDP stream entries based on the identified dependencies and relationships is performed. The grouping of the CDP stream entries together into the plurality of buffers 106 (e.g., the first buffer 106A, the second buffer 106B, and the like) ensures that the order of the CDP stream entries within each buffer reflects the consecutive sequence of the CDP stream entries, such as, the earlier CDP stream entries are placed at the beginning of the buffer and after that, the dependent CDP stream entries are placed accordingly. For example, the first CDP stream entry 108A may be placed before the second CDP stream entry 108B. Moreover, such ordering of the CDP stream entries is used to enhance the data management for handling the CDP stream entries efficiently, thereby ensuring a logical and sequential representation of data changes. The ordering of the CDP stream entries in each of the buffers from the plurality of buffers 106 is used to handle the CDP operations on the CDP stream entries that are required to be performed for updating the data continuously, thereby ensuring the continuous data protection, facilitating data recovery and data backup. Additionally, the ordering of the CDP stream entries is beneficial to optimize the overall performance of the system 102, especially during the continuous data changes in a reduced period of time that is required for data recovery after any malfunction.

[0057] At step 204, the method 200 includes indexing each of the CDP Stream entries, in CDP stream order, in the plurality of buffers 106 using the large key-value store (LKVS) 114. The indexing of each CDP stream entry involves creating a structured reference for each CDP stream entry from the CDP stream entries. For example, the first CDP stream entry 108A, the second CDP stream entry 108B, and up to the NthCDP stream entry 108N are indexed using the LKVS 114. Moreover, the indexing of the CDP stream entries is performed in the same order in which the CDP stream entries are grouped in each buffer allowing further marking of the stale buffers from the plurality of buffers 106. Additionally, the indexing of each of the CDP stream entries can be used to create a complete CDP-stream-based backup in order to control write amplification. The control on the write amplification reduces the overall data storage cost. Therefore, by utilizing the LKVS 114, the CDP entries are easily accessible due to which the processing time, which is required to retrieve, update, and delete the data, is reduced along with the reduced cost utilization.

[0058] At step 206, the method 200 includes identifying CDP Stream entries, which have a valid entry in the LKVS as data, which has been changed as a result of the indexing. Firstly, the CDP stream entries are grouped together into the plurality of buffers 106. After that, each of the CDP stream entries is indexed in the stream order, creating a data structure that maps the CDP Stream entries to their corresponding entries in the LKVS 114. Moreover, such indexing allows the identification of the CDP stream entries that have a valid match in the LKVS 114, such as by verifying each of the entries in the LKVS 114 whenever the data is changed (e.g., updated, added, or deleted). In other words, the system 102 is configured to identify the CDP stream entries with valid entries in the LKVS 114 that acts as an indicative of the data changes to allow cross-referencing of the CDP stream entries with the corresponding entries in the LKVS 114. Advantageously, the indexing of the CDP stream entries with the valid entry is used to maintain the overall data consistency, data integrity, and data reliability.

[0059] At step 208, the method 200 includes marking a previous buffer where the data which has been changed has been located, as being stale data. The stale data refers to the data that no longer reflects the current (or valid) data of a system (i.e., the system 102 of FIG. 1). By marking the previous buffer, the current and outdated data can be distinguished, thereby maintaining accuracy and integrity that includes the elimination of the use of outdated data, contributing to overall data consistency and reliability.

[0060] At step 210, the method 200 includes performing CDP stream compaction of buffers that contain stale data entries while maintaining the CDP stream order. The CDP stream compaction refers to a removal of duplication within the CDP stream, such as by considering a last UPDATE operation, which is performed on the corresponding CDP stream entry, or by considering the DELETE operation, which is performed to remove any of the ADD or UPDATE operations on the corresponding CDP stream entry. The LKVS 114 utilizes a standard key -value store semantics to support operations, such as the ADD operation, the UPDATE operation, and the DELETE operation. However, the order of the CDP stream entries does not change after the execution of the CDP stream compaction of the buffers. The CDP stream can be applied to the LKVS 114, which transforms a redundant CDP stream into a compacted form including the unique data. Furthermore, the compaction of the CDP entries allows the creation of an incremental backup even if the incremental backup is not available, for example, there exists no way to expose only the differences between a source image between two timestamps or when a target CDP stream format cannot be naively applied to the source image, such as in the case of a change data capture (CDC) stream. As a result, the CDP stream compaction is used to further improve resource consumption while backing up the data, reducing storage requirements, optimizing retention policy for the CDP stream, and reducing the write amplification on the storage of the system 102 (of FIG.

[0061] 1.

[0062] Advantageously, the method 200 is used to provide an efficient, optimized, and effective management and organization of the continuous data protection (CDP) stream entries by grouping the CDP stream entries into the plurality of buffers with sequential sorting by using the LKVS while maintaining the ease of data accessibility. Furthermore, the indexing of each of the CDP stream entries is beneficial for maintaining the order of the CDP stream to provide enhanced and efficient data retrieval and management within the CDP. Thereafter, the identification of the CDP stream entries that possess a valid CDP stream entry in the LKVS, signifying data that has changed ensures precise tracking and recognition of the altered data. Furthermore, the marking of the previous buffer where the data has been changed has been located, as the stale data ensures efficient data management, and data integrity by flagging and distinguishing previous buffers containing modified information as stale, thereby facilitating streamlined data processing. As a result, the method 200 is used to provide enhanced, efficient, comprehensive data management and data retrieval in the CDP with optimized CDP backup and reduced recovery time objective, reduced resource consumption, and reduced cost utilization.

[0063] The steps 202 to 210 are only illustrative, and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein.

[0064] There is provided a computer program comprising instructions that, when executed by a computer system, cause the computer system to implement the method 200. In an example, the instructions are implemented on the computer-readable media, which include, but are not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Random Access Memory (RAM), Read-Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, a Secure Digital (SD) card, Solid-State Drive (SSD), a computer-readable storage medium, and / or CPU cache memory. In an example, the instructions are generated by a computer program, which is implemented in view of the method 200 for optimizing the CDP with the large key -value store.

[0065] FIG. 3 is an exemplary diagram depicting an execution of a CDP stream compaction by LKVS, in accordance with an embodiment of the present disclosure. FIG. 3 is described in conjunction with elements from FIG. 1. With reference to FIG. 3 , there is shown an exemplary diagram 300 that depicts the use of the LKVS 114 (of FIG. 1) to perform CDP stream compaction in order to reduce the recovery time objective from the CDP stream and optimize retention policy for the CDP stream (i.e., the CDP stream entries). There is shown a time axis 302 that depicts the flow of events with respect to time, CDP stream 304, a first incremental backup 306A, and a second incremental backup 306B.

[0066] In an exemplary scenario, the CDP stream 304 includes the CDP stream entries, for example, the first CDP stream entry 108A, the second CDP stream entry 108B, and up to the NlCDP stream entry 108N that are grouped in each of the plurality of buffers 106, such as by applying each entry to the LKVS 114 (of FIG. 1), thereby creating two incremental backups, such as the first incremental backup 306A, and the second incremental backup 306B. Moreover, each backup is a consolidation of CDP streams within the LKVS 114 and the merging of CDP streams into incremental backups is facilitated by the logical key -value map (LKVM). However, each merging operation is executed in a distinct LKVS table or in a distinct tree. The first incremental backup 306A and the second incremental backup 306B are created by applying sequential time ranges of the CDP stream to the LKVS 114 that includes the ADD operations or the UPDATE operation as a last operation for any updated key and DELETE operation for keys that are not added in such a timeline. Additionally, the DELETE operation for keys is compacted together with all previous ADD and UPDATE operations. Therefore, by using standard key-value store semantics, which support ADD, UPDATE, and DELETE operations, the CDP stream 304 can be applied to the LKVS 114 that transforms the redundant CDP stream into a compacted form containing unique data and allows an efficient data recovery even in cases where a source or a target does not support incremental backup. FIG. 4 is an exemplary diagram depicting the execution of the continuous data protection, in accordance with another embodiment of the present disclosure. FIG. 4 is described in conjunction with elements from FIG. 1 and FIG. 3. With reference to FIG. 4, there is shown an exemplary diagram 400 that depicts the execution of the continuous data protection (CDP) in order to reduce the recovery time objective from the CDP stream and optimize retention policy for the CDP stream (i.e., the CDP stream entries). There is further shown a full backup 402, the time axis 302 that depicts the flow of events with respect to time, the CDP stream 304, the first incremental backup 306A, and the second incremental backup 306B.

[0067] In an exemplary scenario, the full backup 402 is utilized to restore the system 102 to a recovery objective 404 before the first incremental backup 306A and the second incremental backup 306B. In other words, the full backup 402 is used to restore the data, and then the first incremental backup 306A and the second incremental backup 306B are used to restore the data. After that, a tail of the CDP stream is applied, following a timeline to reach the recovery objective 404. The data in each of the CDP stream 304 is compacted into the incremental backup (i.e., the first incremental backup 306, and the second incremental backup 306B), which can be further reclaimed upon reaching a required retention period. Furthermore, the incremental backups (i.e., the first incremental backup 306A and the second incremental backup 306B) can be merged to create coarser-grained increments, allowing more flexibility in terms of recovery time objective (RTO) and service level agreement (SLA) application. Moreover, the SLA application refers to an agreement that defines the service level required by the backup and includes a retention policy for the backups (i.e., the first incremental backup 306A, the second incremental backup 306B, and the full backup 402). The retention policy includes the timelines, for example, the CDP is retained for one week while full snapshots will be retained for one month, and monthly full snapshots for five years. As a result, in such an implementation, multiple disk input / output and computation resources are allocated for compaction (e.g., reading several updates of the key) instead of allocating the multiple disk input / output and computation resources completely for the execution of the data recovery, thereby enhancing the overall performance of the system 102 while ensuring the continuous data protection.

[0068] FIG. 5 is an exemplary diagram that depicts the continuous data protection by using a log-structured merge tree-based large key-value store, in accordance with an embodiment of the present disclosure. FIG. 5 is described in conjunction with elements from FIGs. 1, 3, and 4. With reference to FIG. 5 there is shown an exemplary diagram 500 that depicts the continuous data protection (CDP) by using the log-structured log merge (LSM) tree-based large key-value store (LKVS).

[0069] In the exemplary scenario, the CDP stream 304 is compacted by using the LSM tree-based LKVS with three levels, such as a first level, a second level, and a third level. Moreover, each level includes different CDP stream entries, for example, the first level includes a first CDP entry 502A, a second CDP entry 502B, a third CDP entry 502C, a fourth CDP entry 502D, a fifth CDP entry 502E, and a sixth CDP entry 502F. Similarly, the second level includes a seventh CDP entry 502G, an eighth CDP entry 502H and the third level includes a ninth CDP entry 5021. In an implementation, both a retention mechanism and a merge mechanism in the LSM tree implementation can be used. Moreover, the merge mechanism is used for retaining new CDP stream entries (i.e., at higher levels) and updating older and outdated CDP stream entries that are located at lower levels of the LSM tree. For example, the CDP stream entries, such as the fifth CDP entry 502E, the sixth CDP entry 502F, and the ninth CDP entry 5021 include data that cannot be removed while the CDP entries, such as the first CDP entry 502A, the second CDP entry 502B, the third CDP entry 502C, the fourth CDP entry 502D, the seventh CDP entry 502G and the eighth CDP entry 502H includes the data that is a candidate for reclamation since the data has already been copied to lower levels. Furthermore, a restore path, such as a first restore path 504A, a second restore path 504B, and a third restore path 504C from the full backup 402 is utilized to perform data restoration with reduced overall processing time. Additionally, the utilization of the LSM treebased LKVS allows further control of write amplification as the data is removed only upon merge.

[0070] FIG. 6 is an exemplary diagram depicting a continuous data protection layout and indexing of continuous data protection stream entries, in accordance with an embodiment of the present disclosure. FIG. 6 is described in conjunction with elements from FIGs. 1, 3, 4 and 5. With reference to FIG. 6, there is shown an exemplary diagram 600 that depicts the continuous data protection (CDP) layout and indexing of the continuous data protection (CDP) stream entries.

[0071] In an exemplary scenario, multiple non-overlapping write-ahead logs (WAL) that are ordered at the time of insertion are represented. The WALs include a first WAL 602A, a second WAL 602B, a third WAL 602C, a fourth WAL 602D, a fifth WAL 602E, and a sixth WAL 602F. In an implementation, the WAL refers to a type of file in which only updates are appended to the end of the file. Moreover, the WALs above line 604, for example, the first WAL 602A and the second WAL 602B have passed the CDP service level agreement (SLA) and the WALs below the line 604 include the CDP stream entries, arranged from the oldest to the newest CDP entries. The WAL is required to be stored in the data storage until the retention policy permits the eviction of the WAL and until none of the CDP stream entries are valid CDP entries. Moreover, the CDP is written into the backup by using a pre-defined sized WAL. As a result, the CDP layout and indexing scheme enables the saving of the data only once on the backup storage and provides a way to control write amplification, enhancing the overall performance of the system 102.

[0072] FIG. 7 is an exemplary diagram depicting a merging of old write-ahead logs with multiple invalid entries, in accordance with an embodiment of the present disclosure. FIG. 7 is described in conjunction with elements from FIGs. 1, 3, 4, 5, and 6. With reference to FIG. 7 there is shown an exemplary diagram 700 that depicts the merging of the old write-ahead logs (WAL) with multiple invalid entries being merged into a single entry with no invalid entries (i.e., overwritten entries).

[0073] In an exemplary scenario, an old WAL, such as a first WAL 702A, a second WAL 702B, and a third WAL 702C includes multiple invalid entries (e.g., an invalid entry 704) that are merged into a single entry 706 with no invalid entries. However, in order to meet the WAL condition, the invalid entries (e.g., the invalid entry 704) are not removed in place but are copied into a new WAL, such as the single entry 706. Additionally, the re-ordering of the WAL buffer is possible only if the corresponding WAL contains valid entries. The valid entries are checked through a restore validity check, which is performed for every key as an invalid update may reside in a later buffer caused by the reorder. Moreover, an index of the CDP stream for each key representing the CDP stream entry points to the latest update to allow the system 102 to locate any value that had been part of the CDP stream, even if the corresponding CDP stream had been in older CDP stream WAL.

[0074] FIG. 8 is an exemplary diagram depicting B+ tree-based large key-value store implementation for indexing of the CDP stream entries, in accordance with an embodiment of the present disclosure. FIG. 8 is described in conjunction with elements from FIGs. 1, 3, 4, 5, 6, and 7. With reference to FIG. 8, there is shown an exemplary diagram 800 that depicts the B+ tree-based LKVS implementation for index combined with write-ahead log (WAL) type continuous data protection (CDP) stream. In an implementation scenario, the B+ tree-based LKVS implementation for indexing is combined with a WAL-type CDP stream. For example, some latest CDP stream entries, such as a first latest CDP stream entry 802A, and a second latest CDP stream entry 802B reside in the CDP stream portion of the WAL files, such as the fourth WAL 602D while the latest CDP stream entries reside in the non-CDP (i.e., retention candidates), such as the first WAL 602A and the second WAL 602B. Moreover, when a CDP stream entry is updated through an operation like the UPDATE operation, then, the corresponding LKVS value is moved to a new location and is incremented for the WAL that contains an outdated update, indicating the count of invalid keys in the WAL. The volatility of the WAL concerning invalid CDP stream entries is not as strict as that of the data since the value is based on the CDP stream. Moreover, the latest location of the stream until which the values have been updated needs to be saved, and in case of a crash, there exists a possibility to idempotently replay the CDP stream until the CDP stream order ends and the latest result is achieved. Therefore, such relaxation in the volatility criteria allows the CDP to reduce write amplification.

[0075] Additionally, the CDP stream entries that match the write amplification policy are merged with the already existing CDP stream entries. Furthermore, when restoring from the CDP-stream-based storage, a sequential read and restore from the WAL buffers can be performed due to the reduced invalid data. However, the invalid data is overwritten, such as by applying the UPDATE operation or the DELETE operation in order to provide an efficient data restoration with reduced resource utilization.

[0076] FIG. 9 is an exemplary diagram depicting a continuous data protection-based backup, in accordance with an embodiment of the present disclosure. FIG. 9 is described in conjunction with elements from FIGs. 1, 3, 4, 5, 6, 7, and 8. With reference to FIG. 9, there is shown an exemplary diagram 900 that includes customer server 902, a backup server 904, a database 906, and a change data capture (CDC) agent 908.

[0077] In an exemplary scenario, the database 906 (e.g., a MongoDB database) is created from a storage component (i.e., a block or a file). Moreover, any change performed in the database 906 is performed through an application programming interface (API), such as by using a generic format, which is called a change data capture (CDC). In an implementation, the CDC agent 908 of the customer server 902 is responsible for reading the CDC of changes made to the database 906 and copying the corresponding changes to the backup server 904. The changes made in the database 906 are appended to the latest write-ahead log (WAL) buffer in the CDP stream on the backup server 904 and the latest CDP stream entries allow the system 102 to locate the latest changes that are done to the database 906. However, the database 906 relies on the CDC API, and the only data transferred to the backup server 904 is the CDP stream. As a result, the system 102 is configured to ensure continuous data protection with reduced resource allocation and recovery time objective.

[0078] Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as "including", "comprising", "incorporating", "have", "is" used to describe, and claim the present disclosure are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. The word "exemplary" is used herein to mean "serving as an example, instance or illustration". Any embodiment described as “exemplary” is not necessarily to be constmed as preferred or advantageous over other embodiments or to exclude the incorporation of features from other embodiments. The word "optionally" is used herein to mean "is provided in some embodiments and not provided in other embodiments". It is appreciated that certain features of the present disclosure, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable combination or as suitable in any other described embodiment of the disclosure.

Claims

CLAIMS1. A method (200) comprising steps of: grouping together Continuous Data Protection (CDP) stream entries into a plurality of buffers (106), where each buffer stores consecutive CDP stream entries such that the entries are sorted in order in each buffer so that an earlier write entry in the CDP stream precedes a write entry which is dependent upon the earlier write entry; indexing each of the CDP Stream entries, in CDP stream order, in the plurality of buffers (106) using a Large Key- Value Store (LKVS) (114); as a result of the indexing, identifying CDP Stream entries which have a valid entry in the LKVS (114) as data which has been changed; and marking a previous buffer where the data which has been changed has been located, as being stale data.

2. The method (200) of claim 1 wherein the LKVS (114) is a log-structured merge (LSM) tree.

3. The method (200) of claim 1 wherein the LKVS (114) is a B+ tree.

4. The method (200) of claim 1 wherein the indexing pushes updates into the LKVS (114).

5. The method (200) of claim 4 wherein the indexing pushes only a reference to a respective buffer and an offset inside the respective buffer.

6. The method (200) of claim 1 further including the step of performing CDP stream compaction of buffers that contain stale data entries while maintaining the CDP stream order.

7. A system (102) comprising means adapted for carrying out all the steps of the method (200) according to any preceding method (200) claim.

8. A computer program comprising instructions for carrying out all the steps of the method according to any preceding method (200) claim, when said computer program is executed on a computer system.

Images