System and method for storing data file

Abstract

To provide a computer mounting method, system and program for reducing an amount of duplicate data.SOLUTION: A method 200 stores a data file 202a, divides the data file into chunks 204a1 to 204a3, and constructs a data structure representing the data file. The data structure includes a hierarchical tree 206a representing the chunks. Also, the method generates respective chunk identifiers (IDs) corresponding to the chunks, identifies a non-duplicate chunk from among the chunks on the basis of the respective chunk IDs that are generated, constructs a data pack on the basis of the constructed data structure and the identified non-duplicate chunk, and stores the constructed data pack in a memory.SELECTED DRAWING: Figure 2

Description

[Background technology]

[0001] Conventional systems for storing, organizing, and / or transferring data can be slow and inefficient, resulting in poor performance. Summary of the Invention

[0002] One common technical problem in existing computer systems that store, organize, and / or transfer data, e.g., revision control systems (RCSs, also referred to interchangeably herein as version control systems or VCSs), is how to reduce the amount of data that needs to be transferred and / or stored. For example, when using cloud-based servers, e.g., cloud-based RCSs, the problem arises of minimizing the amount of data transferred and / or stored on the cloud-based servers. Generally, smaller amounts of data mean faster transfer and cheaper storage on the cloud.

[0003] As a non-limiting example, in a traditional RCS, a significant amount of duplicate data may exist. For example, a 1 GB (gigabyte) file may be checked in, i.e., stored, and then a new, slightly modified version of the same file may be checked in. If complete versions of both files are stored, this may consume 2 GB in total, but most of the second gigabyte may be duplicate data. This is just one type of data duplication that may occur. Because data may be stored on a cloud-based system, e.g., a cloud-based RCS, other different types of duplicate data may also need to be reduced to conserve as much storage, e.g., disk-based storage, as possible. For cloud-based systems, it may also be desirable to reduce the amount of data that needs to be transferred to and / or from the cloud-based system. Returning to the non-limiting example of an RCS, a solution that improves checkout execution times and is not dependent on the number of file versions (i.e., deltas) may also be needed.

[0004] Additionally, there may be a need for the ability to expedite the transfer of large files (e.g., transfer files in parallel) without having to retransfer the entire file when an in-progress transfer fails and part of the file has already been successfully transferred.

[0005] The embodiments address these and other problems in existing systems by providing a unique method and system for storing the contents of data files (e.g., binary or text files) within a computer system, such as an RCS, to reduce the amount of duplicate data. Furthermore, the embodiments may be utilized in industries such as semiconductors, as well as any other technical field where computer systems are used for data storage, organization, and / or transfer, for example, the software industry.

[0006] In one embodiment, files checked into an RCS may be partitioned into small chunks via a natural chunking algorithm using a rolling hash or other suitable hash function known to those skilled in the art. These chunks may then be grouped into "data packs" and indexed via their SHA-256 checksums; other suitable known checksum functions may also be used. The "data packs" may be transferred and stored in a cloud-based system or, alternatively, in local storage. A "data pack" may contain one or more chunks, which may comprise part of a larger file or may be a combination of chunks from many smaller files. Grouping chunks into "data packs," instead of storing each chunk as a separate file, may solve the performance issues of transferring many small files to and from cloud-based object stores such as Amazon® S3, among other examples.

[0007] According to another embodiment, chunks may be indexed by their associated SHA-256 checksums, although other suitable known checksum functions may also be used. This may enable embodiments to check for duplicates, which is one exemplary advantage of embodiments, as well as to locate and fetch the chunks needed to reconstruct a file during checkout. Additionally, in one embodiment, checksums may be used to detect corrupted data in a file or other phenomena such as malicious activity, e.g., file tampering.

[0008] In yet another embodiment, the list of chunks that make up a file may be stored as a hierarchical tree, which may be referred to as a "chunk partition hierarchy tree," instead of a flat list. This may allow an embodiment to store only changes to the "chunk partition hierarchy tree" for newer versions of a file, instead of storing a complete list of chunks for each version of the file. This may also reduce the amount of duplicate data, e.g., metadata.

[0009] One embodiment includes, by way of non-limiting example, the following novel components: a) Using natural chunking in computer systems, e.g., RCS; b) indexing the chunks for lookup and duplicate detection; c) storing the list of chunks that make up a file as a "chunk partition hierarchical tree" instead of a flat list; and d) Combining the chunks and structuring them into "data packs" for transfer and storage, for example, in a cloud-based system or alternatively in local storage.

[0010] The above non-limiting example components, individually or in combination, may enable embodiments to (i) significantly reduce the amount of storage required by a computer system, e.g., an RCS; (ii) reduce the time it takes to transfer data to and from a cloud-based system; and (iii) provide fault tolerance against potential data transmission errors.

[0011] An exemplary embodiment is directed to a computer-implemented method for storing a data file. First, the method partitions the data file into multiple chunks. Then, the method constructs a data structure representing the data file. The data structure includes a hierarchical tree representing the multiple chunks. Next, the method generates respective chunk identifiers (IDs) corresponding to the multiple chunks. The method identifies non-duplicate chunks from among the multiple chunks based on the generated respective chunk IDs. Identifying the non-duplicate chunks may include performing a deduplication process on the multiple chunks. Based on the constructed data structure and the identified non-duplicate chunks, the method constructs a data pack. Then, the method stores the constructed data pack in memory, which may be, for example, disk-based storage, primary storage such as RAM, remote storage such as cloud storage, or a database, among other examples.

[0012] In one embodiment, building the data structure includes generating nodes representing the plurality of chunks and generating a hierarchical tree by repeatedly partitioning the generated nodes until a condition is met. According to one such embodiment, the repeatedly partitioning is based on a criterion. In another embodiment, at least one of the condition and the criterion is based on user input. Furthermore, according to yet another embodiment, at least one of the condition and the criterion may be specified in advance.

[0013] In another embodiment, the condition includes that the number of first nodes in the hierarchical tree at a first level is less than the number of second nodes in the hierarchical tree at a second level. According to one such embodiment, the first level is higher than the second level in the hierarchical tree's hierarchical order. Furthermore, in yet another embodiment, the criterion is based on a target average number of child nodes for a given parent node among the plurality of nodes. According to one embodiment, the criterion is based at least in part on a given node ID for a given node of the nodes. In one such embodiment, the given node ID is calculated based on a chunk ID associated with the given node. According to another embodiment, the criterion is based on a node ID of a next level of the hierarchical tree. If the next level is a leaf level (i.e., the nodes at the leaf level are chunks), the criterion is based on the chunk IDs of the chunks at the leaf level. If the next level is an intermediate level, the criterion is based on the node IDs of the nodes at the intermediate level. In one such embodiment, the node ID is a SHA-256 hash value of the respective contents of the node; other suitable known hash functions may also be used. In one embodiment, the content of a node is a list of the IDs (eg, SHA-256 hash values) of its child nodes.

[0014] According to one embodiment, the data file is a first data file and the hierarchical tree is a first hierarchical tree, and in one such embodiment, identifying non-duplicate chunks includes comparing the first hierarchical tree to a second hierarchical tree associated with a second data file.

[0015] In another embodiment, partitioning the data file into chunks includes partitioning the data file into chunks based on at least one of a Rabin-Karp function and a FastCDC function. According to one such embodiment, FastCDC may be an optimized, high-performance approach for content-defined chunking.

[0016] Additionally, in yet another embodiment, partitioning the data file into chunks includes partitioning the data file into chunks based on a target chunk size. According to one embodiment, the target chunk size is based on user input. In another embodiment, the target chunk size may be specified in advance.

[0017] According to one embodiment, generating each chunk ID includes generating each chunk ID using a SHA-256 function.

[0018] In another embodiment, identifying non-duplicate chunks includes querying a data repository index to identify unique chunk IDs corresponding to the non-duplicate chunks. According to one embodiment, chunk ID matching and / or node ID matching may be performed. If node IDs (which may include the ID of the root node) match, then there is a duplicate branch between the two hierarchical trees. If two chunk IDs match, then there is a duplicate leaf between the two hierarchical trees. Such chunk ID and / or node ID matching may be used to identify non-duplicate chunks in embodiments.

[0019] In yet another embodiment, the method further includes generating a metadata segment corresponding to a given data pack of the constructed data packs. According to one such embodiment, the metadata segment includes information about the contents of the given data pack. For example, in one embodiment, the metadata segment may include a table having entries, and the entries may include, for example, an ID of a data item, a type of item (such as a node or chunk), a size of the item, and / or a position of the item within the data pack. According to another embodiment, the method further includes updating a data repository index based on the generated metadata segment.

[0020] According to one embodiment, the memory in which the data pack is stored is associated with a cloud-based server. In one such embodiment, the method further includes transferring the constructed data pack to the cloud-based server before storing. In another embodiment, the data file is a new version of a file in a revision control system (RCS), and the cloud-based server includes a repository for the RCS. According to one such embodiment, the method further includes performing the transfer as part of committing the new version of the file to the repository for the RCS. In yet another embodiment, the method further includes performing the transfer in parallel by instantiating at least two respective transfer threads corresponding to the constructed data pack. According to one embodiment, the method further includes restarting a given transfer thread in response to detecting a failure of a given transfer thread of the at least two instantiated transfer threads.

[0021] Another exemplary embodiment is directed to a computer-based system for storing data files. The system includes a processor and a memory having computer code instructions stored thereon. In one such embodiment, the processor and memory are configured to cause the computer code instructions to cause the system to perform any embodiment or combination of embodiments described herein.

[0022] Yet another exemplary embodiment is directed to a non-transitory computer program product for storing data files, the computer program product including a computer-readable medium having stored thereon computer code instructions that, when executed by a processor, are configured to cause a device associated with the processor to perform any embodiment or combination of embodiments described herein.

[0023] It should be noted that the embodiments of the methods, systems, and computer program products may be configured to implement any embodiment or combination of embodiments described herein. [Brief explanation of the drawings]

[0024] The foregoing will become apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings, in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the embodiments.

[0025] [Figure 1] FIG. 1 is a diagram illustrating the process of partitioning a file, according to one embodiment. [Figure 2] FIG. 2 is a diagram illustrating a process for generating a chunk partition tree according to one embodiment. [Figure 3] FIG. 3 illustrates the process of checking in two files with overlapping content, according to one embodiment. [Figure 4] FIG. 4 is a diagram illustrating a process for grouping and storing data pack files according to one embodiment. [Figure 5] FIG. 5 is a diagram illustrating a commit or update process using a single data pack file, according to one embodiment. [Figure 6] FIG. 6 illustrates a process for storing two different files with overlapping content, according to one embodiment. [Figure 7] FIG. 7 is a flowchart of a method for storing a data file, according to one embodiment. [Figure 8] FIG. 8 is a schematic diagram of a computer network in which embodiments may be implemented. [Figure 9] FIG. 9 is a block diagram illustrating an exemplary embodiment of a computer node in the computer network of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0026] A description of an exemplary embodiment follows.

[0027] Storage systems, such as revision control systems (RCSs), potentially store a lot of duplicate data, for example, by storing multiple versions of each file. Conventional systems may attempt to reduce the amount of data stored by storing deltas for each file version. However, this delta-based approach has drawbacks. As the number of deltas increases (i.e., the number of revisions of a file increases), it becomes more expensive in terms of performance to reconstruct the file. Furthermore, the checkout process may need to traverse the delta history to reconstruct the file. This may also have the drawback of not considering other types of data duplication. For example, two completely unrelated files may also have duplicate data, a problem not addressed by existing systems.

[0028] Embodiments utilize the concept of applying natural chunk partitioning to a file system. A natural chunk partitioning method that can be used in embodiments is described in A. Muthitacharoen, et al., “A Low-bandwidth Network File System,” ACM SIGOPS Operating Systems Review 35(5):174-187 (December 2001), which is incorporated herein by reference in its entirety. Furthermore, embodiments may use these concepts, for example, within cloud-based servers, and their functionality is unique compared to existing approaches. As previously mentioned, RCSs may contain a lot of duplicate data by storing multiple versions of each file. Thus, the data deduplication (removal of duplicate data) functionality implemented by embodiments can reduce the storage requirements and / or increase the efficiency of these types of systems.

[0029] Embodiments may implement a number of novel concepts. For example, as part of applying the concept of natural chunk partitioning to RCS, embodiments may introduce the novel concept of creating a chunk partition hierarchy tree and grouping chunks into "data packs" for transfer and storage, for example, on a cloud-based server. Embodiments go beyond innovations that leverage natural chunk partitioning within RCS to provide these and other new and innovative concepts, which are discussed in detail below.

[0030] In one embodiment, a file checked into an RCS may be partitioned into chunks. According to another embodiment, the chunks may have a target size of, for example, 4 kB (kilobytes) or greater per chunk. Furthermore, in yet another embodiment, the partitioning may be performed using, for example, a rolling hash or other suitable hash function known to those skilled in the art. These chunking and / or partitioning processes may be referred to as "natural chunk partitioning" or "content-defined chunking" because the partitioning criteria may be based on the file content. In one embodiment, the actual chunk size may be arbitrary, but may be relatively close to the target size. According to another embodiment, the natural chunk partitioning may be calculated using, for example, a Rabin-Karp function, a FastCDC function, or any other suitable hash function known in the art.

[0031] According to one embodiment, once a file is partitioned into chunks, an identifier, e.g., a SHA-256 hash value, may be calculated for each chunk for indexing purposes. Note that, according to one embodiment, the identifier, e.g., hash value, may be unique if the corresponding chunk content is unique. Thus, in yet another embodiment, whenever the hash value of a new chunk matches the hash value of another chunk, both chunks may be identical and storage of the new chunk may be skipped. This feature may enable embodiments to check for and prevent the storage of duplicate content.

[0032] FIG. 1 illustrates a process 100 for partitioning files 102a and 102b, e.g., files on a disk, according to one embodiment. Process 100 partitions files 102a and 102b into chunks. Partitioning file 102a results in multiple chunks, including chunks 104a1-104a3, while partitioning file 102b results in a single chunk 104b. For example, a SHA-256 hash value may be calculated for each chunk, e.g., 104a1-104a3 and 104b, for indexing purposes; other suitable known hash functions may also be used. The size of each chunk, e.g., 104a1-104a3 and 104b, may be arbitrary but may, on average, be equal to a target size. A small file, e.g., 102b, may result in only a single chunk, e.g., 104b.

[0033] In one embodiment, when a file, e.g., 102a or 102b, is partitioned, a list of the chunks that make up the file, e.g., 104a1-104a3 or 104b, may be recorded. Such a list may be used to reconstruct the file. However, instead of a traditional flat list, embodiments may generate and record a hierarchical tree. Such a hierarchical tree may be referred to as a "chunk partition hierarchical tree" and may be an embodiment's unique approach to storing the list. Nodes in the hierarchical tree may be referred to as "chunk nodes" and may include a list of their corresponding child nodes. According to one embodiment, only leaf nodes in the hierarchical tree may be actual chunks. In another embodiment, the root node of the hierarchical tree may represent a complete file. According to yet another embodiment, a SHA-256 hash value may be calculated for each node in the tree for indexing purposes, similar to, for example, file chunks. Other suitable known hash functions or other such identifier generation methods may also be used. Embodiments may check for duplicate chunk nodes in a manner similar to file chunks.

[0034] FIG. 2 illustrates a process 200 for generating chunk partition trees 206a and 206b, according to one embodiment. As shown in FIG. 2, hierarchical trees 206a and 206b correspond to files 202a and 202b, respectively. Hierarchical or chunk partition tree 206a includes multiple nodes 208a1-208a6, where nodes 208a1-208a5 are intermediate nodes and node 208a6 is the root node. Similarly, hierarchical or chunk partition tree 206b includes node 208b, which is the root node. File 202b may be a small file, which may result in tree 206b having only a single node 208b.

[0035] Process 200 may generate hierarchical trees 206a and 206b based on chunks, e.g., 204a1-204a3 and 204b, of files 202a and 202b, respectively. For example, file 202a may be partitioned, or chunked, to generate multiple chunks, e.g., 204a1-204a3, among others. Each grouping of chunks is then represented by a respective node 208a1-208a5 in tree 206a. In a particular example, chunks 204a1-204a3 are represented by node 208a1.

[0036] According to one embodiment, non-leaf nodes in a tree may include a list of child nodes, which may be either intermediate or leaf nodes, the latter of which may be file chunks. For example, referring to FIG. 2, root node 208a6 of tree 206a may include a list of intermediate child nodes 208a1-208a5. In turn, intermediate nodes 208a1-208a5 may each include a list referencing a leaf node. For example, intermediate node 208a1 has leaf nodes that are file chunks 204a1-204a3. Similarly, root node 208b of tree 206b may include a list referencing a leaf node that is file chunk 204b.

[0037] Note that in the exemplary embodiment of FIG. 2, intermediate nodes 208a1-208a5 list only leaf-level file chunks, e.g., 204a1-204a3, but in other embodiments, intermediate nodes may also list other intermediate nodes.

[0038] Continuing with FIG. 2, according to one embodiment, process 200 may, for example, calculate a SHA-256 hash value for each node, e.g., 208a1-208a6 and 208b, for indexing purposes, although other suitable known hash functions may also be used.

[0039] Referring again to FIG. 2, in another embodiment, the number of children per non-leaf node, eg, 208a1-208a6 and 208b, may be arbitrary, but on average equal to the target size.

[0040] One reason for embodiments that use a hierarchical tree over a flat list may be to reduce the amount of data stored for each version of a file. According to one embodiment, only the "chunk nodes" that change may need to be stored for a new version of a file. This may result in significant savings in disk space, especially if the file is large and there are several versions of the file. Instead of storing a complete list of the file's chunks for each version of the file, which can be quite expensive, embodiments may store only the changes, e.g., a list of the chunks that are changed.

[0041] According to one embodiment, each level of the hierarchical tree, e.g., 206a and 206b, may be calculated similarly to how natural chunks are calculated for a file. For example, some embodiments may apply a "natural" partitioning of the previous level of hierarchy. In another embodiment, such partitioning may be repeated until the number of nodes in the next level of hierarchy no longer decreases. Furthermore, in yet another embodiment, to determine where to partition, for example, the SHA-256 hash value of each child node may be checked against some criteria; other suitable known hash functions may also be used. For example, a useful criterion may be that the SHA-256 value of the child node ends in "0" (zero). The selected criterion may be determined by a target or desired average number of child nodes per parent. For example, if the selected criterion is that the two lowest digits must be equal to "00" (double zero) rather than the lowest digit being equal to "0" (zero), there may be more children per node on average (in this case, 256 child nodes versus 16 child nodes on average).

[0042] The foregoing procedure may have the property that if two files have matching content, then their corresponding chunk hierarchy trees will also have matching "chunk nodes" up to the point where the files begin to differ. Embodiments may check for duplicate nodes as well as duplicate chunks, allowing for simultaneous determination of duplicate content across multiple chunks instead of checking each chunk one at a time.

[0043] Figure 3 illustrates a process 300 for checking two files 302a and 302b for overlapping content 312, according to one embodiment. In Figure 3, each file 302a and 302b is partitioned to result in a respective plurality of chunks, e.g., 304a1-304a3 and 304b1-304b3, representing each file. The respective plurality of chunks are then used to generate trees 306a and 306b.

[0044] 3, duplicate content 312 in both files 302a and 302b results in trees 306a and 306b (i.e., trees corresponding to files 302a and 302b, respectively) each containing an identical set of leaf nodes, which may be file chunks 304a1-304a3 and 304b1-304b3, respectively. Furthermore, nodes 308a1 and 308b1 having leaf node children of file chunks 304a1-304a3 and 304b1-304b3, respectively, may be identical in the sense that nodes 308a1 and 308b1 may have matching hash values.

[0045] According to one embodiment, if the chunk nodes themselves match, e.g., if two nodes have matching hash values, it may not be necessary to check the contents of the two nodes, e.g., their respective child nodes, for differences. For example, with reference to Figure 3, two nodes 308a1 and 308b1 may have matching hash values, so it may not be necessary to search for differences in the node contents, i.e., the leaf nodes of file chunks 304a1-304a3 and 304b1-304b3, respectively.

[0046] Continuing with FIG. 3, in another embodiment, for example, the SHA-256 hash value of a root node, e.g., 308a2 or 308b2, may also be the hash value of a corresponding file on disk, e.g., 302a or 302b, respectively, or other suitable known hash functions may be used.

[0047] One exemplary advantage of the hierarchy of embodiments, as opposed to a flat listing of chunks of a file, is that it avoids duplicating information for each listing of each version of a file, thereby allowing the listing to be as small as possible. Another exemplary advantage is that embodiments can check for duplicate data at the hierarchical node level, which allows embodiments to skip large sections of a file when determining what content has changed from one version of a file to another, which might otherwise require checking chunks one at a time.

[0048] Yet another innovation of embodiments is how chunks and “chunk nodes” are actually transferred and stored, for example, on a cloud-based server. Storing chunks as one file per chunk or “chunk node” may be too expensive, as this may result in a large number of extremely small files. Also, transferring many very small files separately may be too expensive from a performance perspective. It may be more efficient to transfer data as one large file versus many very small files. This is one reason why methods may be used, such as creating a file of compressed data using tar, gzip, or other known compression tools, and then transferring the compressed file. Embodiments may account for this by grouping chunks and “chunk nodes” together into “data packs.” Instead of transferring and storing individual chunks and “chunk nodes” separately, these “data packs” may be transferred and stored on a cloud-based server, such as a cloud-based RCS. Each data pack may be transferred and stored as a file on disk, and its filename may be its SHA-256 hash value, although other suitable known hash functions may also be used. An exemplary advantage of the data packs of the embodiments over traditional compressed files is that once a data pack is created, it does not need to be unpacked.

[0049] According to one embodiment, a data pack may be a file containing one or more chunks or chunk nodes. There may be no restrictions on how chunks or chunk nodes are grouped into a data pack. The grouping may be sections of a single large file or may span multiple files. In another embodiment, a data pack may also include a table of contents or metadata segments that lists what chunks and chunk nodes the data pack contains and where they are contained. Furthermore, in yet another embodiment, the table of contents may be indexed by an indexer, such as a central indexer. This may be why an embodiment does not need to unpack files. The RCS may be able to determine which data packs contain a particular file chunk and copy only the particular chunk to the destination file during checkout. In one embodiment, the indexer may facilitate identifying duplicate chunks across multiple computer systems, e.g., multiple RCSs.

[0050] Figure 4 illustrates a process 400 for grouping and storing data pack files 414a and 414b, according to one embodiment. In Figure 4, multiple chunks, e.g., 404a-404n, of a file 402 are generated. Next, a hierarchical tree 406 composed of chunks 404a-n and nodes 408a-408f is generated, and based on the tree 406, the process 400 generates and stores data packs 414a and 414b.

[0051] 4, in one embodiment, chunks 404a-404n and chunk nodes 408a-408f of file 402 are grouped into data packs 414a and 414b, which are transferred to and stored in (and optionally retrieved from) a cloud-based server 418, e.g., a cloud-based RCS, or alternatively, stored in (and optionally retrieved from) local storage (not shown) if a cloud-based server is not used. In another embodiment, there may be no restrictions on which data goes into which data pack. For example, as shown in FIG. 4, nodes 408a and 408b are assigned to data pack 414a, and nodes 408c-408f are assigned to data pack 414b. According to yet another embodiment, the data packs 414a and 414b may include a table of content or metadata segments 416a and 416b, respectively, which may list the chunks and chunk nodes within the respective data packs 414a and 414b.

[0052] Continuing with FIG. 4, in one embodiment, in a new commit, for example, when checking in a new version of a file, e.g., 402, new data packs, e.g., 414a and 414b, are created and sent to a cloud-based server, e.g., 418, or alternatively, stored on disk (not shown) with only the new or changed content (not shown). If any new content overlaps with any other content, including previously committed content, this new content may not be included in data pack 414a or 414b. Furthermore, the new content may not be included in a data pack even if the overlap is from an unrelated file (not shown).

[0053] 5 illustrates a process 500 for committing or updating a file 502 using a single data pack file 514, according to one embodiment. In process 500, the file 502 is partitioned into chunks, generating a tree 506, in which only the content 512 is new relative to a previous version of the file 502 (the previous version is not shown). Continuing, the resulting tree 506 is compared with trees of data (including the tree of the previous version of the file 502) to identify new content to include in the data pack 514. In this example, if only the content 512 is changed, the new data includes a root node 508a and its child nodes 508b with leaf nodes of the chunk 504, the latter corresponding to the changed content 512. This new content (root node 508a, child nodes 508b, leaf nodes of chunk 504) is then included in a data pack 514, which is sent to a cloud-based server 518, e.g., a cloud-based RCS, or alternatively, stored on disk (not shown) when checking in a new version of file 502.

[0054] FIG. 6 illustrates a process 600 for storing two different files 602a and 602b with overlapping content 612, according to one embodiment. As shown in FIG. 6, in one embodiment, the same content 612 results in chunks 604a-604c being generated for both files 602a and 602b. Trees 606a and 606b corresponding to the respective files 602a and 602b may then both include the same intermediate node 608 with leaf nodes for chunks 604a-604c. Furthermore, the resulting data pack 614 may contain only unique content and may not include duplicates. This is illustrated in FIG. 6, where only a single instance of node 608 and chunks 604a-604c is included in data pack 614, reflecting the same content for both files 602a and 602b. 6 further illustrates that duplicate content, e.g., 612, from two completely unrelated files, e.g., 602a and 602b, may be stored only once on a cloud-based server, e.g., 618, or alternatively, may be stored on a disk (not shown). Such functionality is inherent in the system of embodiments.

[0055] An exemplary advantage of data packs of embodiments is that very small file partition chunks and chunk nodes can be grouped into larger files and stored, for example, on a cloud-based server, improving transfer speeds and reducing the number of files. As described above in this specification, it may be more efficient to transfer large files relative to many very small files. However, this may be true to a certain extent. For very large files, it may actually be better to split the very large file into smaller pieces and transfer the pieces in parallel. Data packs of embodiments may also solve this problem. Embodiments may group chunks of very large files into separate data packs, for example, based on a certain target size of the data packs. Embodiments may then transfer these data packs in parallel. For example, according to an exemplary embodiment, the target maximum data pack size may be 500 megabytes (MB). Thus, in another exemplary embodiment, if a file exceeds a target maximum size of 500 MB and the resulting data pack is also larger than 500 MB, the file may be split into multiple data packs. Other target maximum data pack and / or file sizes are also suitable. For clarity, it should be noted that a file that exceeds a target maximum size of, for example, 500 MB may still result in data packs smaller than 500 MB due to duplicate content within the file. In other words, according to exemplary embodiments, the criteria for splitting the resulting data pack into sub-data packs may be based on the data pack size, rather than the original file size. In another exemplary embodiment, when a data pack larger than, for example, 500 MB is split into sub-data packs, the sub-data packs may be kept approximately the same size. For example, according to yet another exemplary embodiment, if a data pack of a file is, for example, 501 MB, the data pack may be split into approximately two sub-data packs of, for example, 250.5 MB each, rather than one sub-data pack of 500 MB and another sub-data pack of 1 MB.

[0056] Furthermore, data packs of embodiments may enable optimizing data transfers, for example, by controlling the size and number of files transferred to a cloud-based server. Furthermore, embodiments may enable data packs to be transferred in parallel, while unpacking may not be required. Embodiments may also initiate transfers as the first data pack is being built to minimize the cost of building data packs. For example, embodiments may initiate transfers of data packs once the initial portion of a very large file has been processed.

[0057] According to one embodiment, a computer system such as an RCS allows a user to roll back a file, or revert to an older committed or checked-in version of a file. For a given commit made by a user, there may be an associated hash value created and stored for that commit, which may serve as the commit's ID. In addition, the RCS may store metadata for a given commit (which may include a chunk hierarchy tree) so the system can determine how to reconstruct the files and folders that make up the commit. If a user wants to revert to an older version of a file, the user may provide the hash value (i.e., its ID) of the specific commit to roll back to. The user can obtain the commit hash value by asking the RCS for the file's commit history. This commit history may include the hash values ​​of each commit in which the file was changed, along with a timestamp and a user-provided message for that commit. The commit history may be ordered from most recent to oldest. One embodiment may store metadata that describes how to reconstruct the files of a particular commit as a tree instead of a flat list, minimizing the amount of information stored for each commit. Additionally, an embodiment may only need to capture which chunks have changed since the previous commit of the file, instead of the complete list of chunks every time.

[0058] Exemplary Method Embodiments 7 is a flowchart of a method 700 for storing a data file, according to one embodiment. Method 700 is computer-implemented and may be performed using any computing device, such as a processor, or combination of computing devices known to those skilled in the art.

[0059] Method 700 begins at step 701 by partitioning a data file, e.g., 102a-102b (FIG. 1), 202a-202b (FIG. 2), 302a-302b (FIG. 3), 402 (FIG. 4), 502 (FIG. 5), or 602a-602b (FIG. 6), into chunks, e.g., 104a1-104a3 (FIG. 1), 204a1-204a3 or 204b (FIG. 2), 304a1-304a3 or 304b1-304b3 (FIG. 3), 404a-404n (FIG. 4), 504 (FIG. 5), or 604a-604c (FIG. 6). At step 702, method 700 constructs a data structure representing the data file. The data structure includes a hierarchical tree representing chunks, e.g., 206a-206b (FIG. 2), 306a-306b (FIG. 3), 406 (FIG. 4), 506 (FIG. 5), or 606a-606b (FIG. 6). Next, in step 703, method 700 generates respective chunk identifiers (IDs) corresponding to the chunks. In step 704, method 700 identifies non-duplicate chunks, e.g., 304a1-304a3 or 304b1-304b3 (FIG. 3) or 604a-604c (FIG. 6), from among the chunks based on the generated respective chunk IDs. Next, in step 705, based on the constructed data structure and the identified non-duplicate chunks, the method constructs data packs, e.g., 414a-414b (FIG. 4), 514 (FIG. 5), or 614 (FIG. 6). At step 706, the method 700 then stores the constructed data pack in memory.

[0060] As described above, method 700 is computer-implemented, such that functions and operations such as segmenting (701), constructing (702 and 705), generating (703), identifying (704), and storing (706) are performed automatically by one or more digital processors. Furthermore, method 700 can be implemented using any computer device or combination of computing devices known in the art. Among other examples, method 700 can be implemented using computers / devices 50 and / or 60 described below in connection with FIGS. 8 and 9.

[0061] In one embodiment of method 700, constructing (702) the data structure includes generating nodes representing chunks, e.g., 208a1-208a6 or 208b (FIG. 2), 308a1-308a2 or 308b1-308b2 (FIG. 3), 408a-408f (FIG. 4), 508a-508b (FIG. 5), or 608 (FIG. 6), and generating a hierarchical tree by iteratively partitioning the generated nodes until a condition is met. According to one such embodiment of method 700, the iterative partitioning is based on a criterion. In another embodiment of method 700, the condition includes a first number of nodes in a first level of the hierarchical tree being greater than or equal to a second number of nodes in a second level of the hierarchical tree. According to one such embodiment of method 700, the first level is higher than the second level in the hierarchical order of the hierarchical tree. Additionally, in yet another embodiment of method 700, the criteria is based on a target average number of child nodes for a given parent node of the plurality of nodes. According to one embodiment of method 700, the criteria is based at least in part on a given node ID for the given node of the nodes. In one such embodiment of method 700, the given node ID is calculated based on a chunk ID associated with the given node. According to another embodiment of method 700, at least one of the condition and the criteria is based on user input.

[0062] According to one embodiment of method 700, the data file is a first data file, e.g., 302a (FIG. 3) or 602a (FIG. 6), and the hierarchical tree is a first hierarchical tree, e.g., 306a (FIG. 3) or 606a (FIG. 6). In one such embodiment of method 700, identifying non-duplicate chunks includes comparing the first hierarchical tree to a second hierarchical tree, e.g., 306b (FIG. 3) or 606b (FIG. 6), associated with a second data file, e.g., 302b (FIG. 3) or 602b (FIG. 6).

[0063] In another embodiment of the method 700, partitioning the data file into chunks (701) includes partitioning the data file into chunks based on at least one of a Rabin-Karp function and a FastCDC function.

[0064] Additionally, in yet another embodiment of method 700, dividing the data file into chunks (701) includes dividing the data file into chunks based on a target chunk size. According to one embodiment of method 700, the target chunk size is based on user input.

[0065] According to one embodiment of method 700, generating 703 each chunk ID includes generating each chunk ID using a SHA-256 function.

[0066] In another embodiment of method 700, identifying 704 the non-duplicate chunks includes querying a data repository index to identify unique chunk IDs corresponding to the non-duplicate chunks. Such an embodiment may search the data index using the multiple generated chunk IDs to determine which of the multiple generated chunk IDs are new (unique), i.e., not already in the data index.

[0067] In yet another embodiment, method 700 further includes generating a metadata segment, e.g., 416a-416b (FIG. 4) or 516 (FIG. 5), corresponding to a given data pack of the constructed data packs. According to one such embodiment of method 700, the metadata segment includes information regarding the contents of the given data pack. In one embodiment, method 700 further includes updating a data repository index based on the generated metadata segment.

[0068] According to one embodiment of method 700, the memory (706) in which the at least one data pack is stored is associated with a cloud-based server, e.g., 418 (FIG. 4), 518 (FIG. 5), or 618 (FIG. 6). In one such embodiment, method 700 further includes transferring the constructed data pack to the cloud-based server prior to storing (706). In another embodiment of method 700, the data file is a new version of a file in an RCS, and the cloud-based server includes a repository of the RCS. According to one such embodiment, method 700 further includes performing the transfer as part of committing the new version of the file to the repository. In yet another embodiment, method 700 further includes performing the transfer in parallel by instantiating at least two respective transfer threads corresponding to the constructed data packs. According to one embodiment, method 700 further includes restarting a corresponding transfer thread in response to detecting a failure of a corresponding transfer thread of the at least two instantiated transfer threads.

[0069] advantage Embodiments enable a significant reduction in the amount of storage required by a system, e.g., an RCS. Potentially, an RCS may store a lot of duplicate data, e.g., by storing multiple versions of each file. Conventional systems may attempt to reduce duplication by storing deltas for each file version.

[0070] One novel approach of embodiments is to partition and store files as chunks. Instead of reducing duplicate data from one version of a file to the next, embodiments remove different types of duplicate data wherever it may occur. This may include, for example, duplicate data within the file itself, duplicate data across multiple versions of a file (e.g., no matter how far apart in the revision history), two different, unrelated files, and even files across different revision control repositories, among other examples. The reduction in storage requirements achieved by embodiments can be very significant, for example, when the stored data has many common sections, e.g., containing common IP (intellectual property). This advantage is entirely inherent to revision control using embodiments.

[0071] Advantageously, embodiments make file reconstruction much simpler. Using embodiments, reconstructing a file may consist of simply stitching together the chunks that make up the file, for example, instead of walking the delta tree or history and overlaying the deltas to reconstruct the file. Yet another novel approach of embodiments is also faster as the number of file versions increases, because the approach is independent of the number of file revisions. Again, if file versions are stored as deltas, conventional systems must retraverse the revision history to collect all the deltas.

[0072] Additionally, the natural chunk partitioning of embodiments improves performance when transferring large files to cloud servers. Instead of transferring the entire file sequentially, embodiments can transfer chunks (portions of the file) in parallel. This facilitates the ability to leverage multiple concurrent threads even when transferring only a single file. Also, if a transfer fails, embodiments do not have to start over completely; instead, embodiments can resume transferring only the failed portion.

[0073] The embodiments also provide advantages over existing approaches to data structures, such as Merkle trees, in which each node has only two children. In a chunk partition hierarchical tree according to one embodiment, each node may have more than two children, and the specific number of children may be based on the natural division of the child nodes. Furthermore, in a chunk partition hierarchical tree according to one embodiment, the number of children may vary for each parent node, although it may be a specific target number on average. Advantageously, this may result in more matching nodes between two trees when comparing an original file with a modified version of the same file, or when comparing completely different files with duplicate content. In the chunk partition hierarchical tree according to one embodiment, if the start of a file is changed, only the initial node changes in the tree. However, in a similar situation when using existing approaches with, for example, Merkle trees, all nodes in the tree may change due to a cascading effect. The embodiments avoid such adverse effects, which is particularly advantageous in systems such as RCS.

[0074] Computer Support

[0075] FIG. 8 is a schematic diagram of a computer network in which embodiments may be implemented.

[0076] The client computers / devices 50 and server computers 60 provide processing, storage, and input / output (I / O) devices for executing application programs and the like. The client computers / devices 50 can also be linked to other computing devices, including other client devices / processors 50 and server computers 60, via a communications network 70. The communications network 70 can be part of a remote access network, a global network (e.g., the Internet), a cloud computing server or service, a collection of computers distributed around the world, a local or wide area network, and gateways that currently use respective protocols (e.g., TCP / IP, Bluetooth, etc.) to communicate with each other. Other electronic device / computer network architectures are also suitable.

[0077] FIG. 9 is a block diagram illustrating an exemplary embodiment of a computer node (e.g., client processor / device 50 or server computer 60) in the computer network of FIG. 8. Each computer node 50, 60 includes a system bus 79, which is a series of hardware lines used to transfer data between components of a computer or processing system. The bus 79 is essentially a shared conduit connecting different elements of a computer system (e.g., processor, disk storage, memory, I / O ports, network ports, etc.), allowing information to be transferred between the elements. Attached to the system bus 79 is an I / O device interface 82 for connecting various input / output devices (e.g., keyboard, mouse, display, printer, speakers, etc.) to the computer node 50, 60. A network interface 86 allows the computer node to connect to various other devices attached to a network (e.g., network 70 of FIG. 8). A memory 90 provides volatile storage for computer software instructions 92a and data 94a used to implement one embodiment of the present disclosure (e.g., method 700 described above with reference to FIG. 7). Disk storage 95 provides non-volatile storage for computer software instructions 92b and data 94b used to implement one embodiment of the present disclosure. Central processing unit 84 is also attached to system bus 79 and provides for the execution of computer instructions.

[0078] In one embodiment, the processor routines 92a-92b and data 94a-94b are a computer program product (generally referred to as 92) including a computer-readable medium (e.g., a removable storage medium such as a DVD-ROM, CD-ROM, diskette, tape, etc.) that provides at least a portion of the software instructions to the system of the present disclosure. The computer program product 92 can be installed by any suitable software installation procedure, as known in the art. In another embodiment, at least a portion of the software instructions may be downloaded over a cable, telecommunications, and / or wireless connection. In other embodiments, the program of the present disclosure is a signal product that propagates a computer program, embodied on a signal propagated on a propagation medium (e.g., radio waves, infrared waves, laser waves, sound waves, or electric waves propagated over a global network such as the Internet or other networks). Such a carrier medium or signal provides at least a portion of the software instructions for the routine / program 92 of the present disclosure.

[0079] In alternative embodiments, the propagated signal is an analog carrier wave or a digital signal carried on a propagation medium. For example, the propagated signal may be a digitized signal propagated over a global network (e.g., the Internet), a telecommunications network, or other network (such as network 70 of FIG. 8). In one embodiment, the propagated signal is a signal transmitted over a propagation medium over a period of time, such as instructions for a software application sent as packets over a network over a period of milliseconds, seconds, minutes, or longer. In another embodiment, the computer-readable medium of computer program product 92 is a propagation medium that can be received and read by computer system 50, such as by receiving the propagation medium and identifying the propagated signal embodied in the propagation medium, as described above for a signal product propagating a computer program.

[0080] Generally speaking, the term "carrier medium" or transient carrier wave encompasses the aforementioned transient signals, propagated signals, propagation media, storage media, and the like.

[0081] In other embodiments, program product 92 may be implemented as a so-called software as a service (SaaS) or other installation or communication supporting end users.

[0082] The embodiments or aspects thereof may be implemented in hardware, including but not limited to hardware circuits, firmware, or software. If implemented in software, the software may be stored on any non-transitory computer-readable medium configured to allow a processor to load the software, or a subset of its instructions. The processor is then configured to execute the instructions and operate or cause an apparatus to operate in the manner described herein.

[0083] Furthermore, hardware, firmware, software, routines, or instructions may be described herein as performing certain operations and / or functions of a data processor, but it will be understood that such description contained herein is merely for convenience and that in reality such operations result from a computing device, processor, controller, or other device executing the firmware, software, routines, instructions, etc.

[0084] It should be understood that the flow diagrams, block diagrams, and network diagrams may include more or fewer elements, or may be arranged differently, or may be represented differently, but it should also be understood that a particular implementation may dictate that the block diagrams and network diagrams, and several block diagrams and network diagrams, illustrating the execution of an embodiment are implemented in a particular way.

[0085] Accordingly, further embodiments may also be implemented in a variety of computer architectures, physical, virtual, cloud computers, and / or any combination thereof, and as such, the data processors described herein are intended for illustrative purposes only and not as limitations of the embodiments.

[0086] The teachings of all patents, published applications, and references cited herein are incorporated by reference in their entirety.

[0087] While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments encompassed by the appended claims.

Claims

1. 1. A computer-implemented method for storing a data file, comprising: Partitioning the data file into a plurality of chunks; constructing a data structure representing the data file, the data structure including a hierarchical tree representing the plurality of chunks; generating respective chunk identifiers (IDs) corresponding to the plurality of chunks; identifying at least one non-duplicate chunk from among the plurality of chunks based on the generated respective chunk IDs; (i) constructing at least one data pack based on the constructed data structure and (ii) the identified at least one non-duplicate chunk; and storing the constructed at least one data pack in a memory.

2. constructing the data structure, generating a plurality of nodes representing the plurality of chunks; generating the hierarchical tree by repeatedly partitioning the generated plurality of nodes until a condition is met, wherein the repeatedly partitioning is based on a criterion.

3. 3. The computer-implemented method of claim 2, wherein the condition includes a first number of nodes in the hierarchical tree at a first level being greater than or equal to a second number of nodes in the hierarchical tree at a second level, the first level being higher than the second level in the hierarchical order of the hierarchical tree.

4. The computer-implemented method of claim 2 , wherein the criterion is based on a target average number of child nodes for a given parent node of the plurality of nodes.

5. 3. The computer-implemented method of claim 2, wherein the criteria is based at least in part on a given node ID for a given node of the plurality of nodes, the given node ID being calculated based on one or more chunk IDs associated with the given node.

6. The computer-implemented method of claim 2 , wherein at least one of the condition and the criteria is based on user input.

7. The data file is a first data file, the hierarchical tree is a first hierarchical tree, and identifying the at least one non-duplicate chunk comprises: The computer-implemented method of claim 1 , further comprising comparing the first hierarchy tree to a second hierarchy tree associated with a second data file.

8. partitioning the data file into the plurality of chunks, 2. The computer-implemented method of claim 1, comprising partitioning the data file into the plurality of chunks based on at least one of a Rabin-Karp function and a FastCDC function.

9. partitioning the data file into the plurality of chunks, The computer-implemented method of claim 1 , further comprising partitioning the data file into the plurality of chunks based on a target chunk size.

10. The computer-implemented method of claim 9 , wherein the target chunk size is based on user input.

11. generating each chunk ID, 2. The computer-implemented method of claim 1, comprising generating the respective chunk IDs using a SHA-256 function.

12. identifying the at least one non-duplicate chunk The computer-implemented method of claim 1 , comprising querying a data repository index to identify at least one unique chunk ID corresponding to the at least one non-duplicate chunk.

13. 2. The computer-implemented method of claim 1, further comprising generating a metadata segment corresponding to a given data pack of the constructed at least one data pack, the metadata segment including information regarding content of the given data pack.

14. The computer-implemented method of claim 13 , further comprising updating a data repository index based on the generated metadata segments.

15. The memory is associated with a cloud-based server, and the computer-implemented method comprises: The computer-implemented method of claim 1 , further comprising transferring the constructed at least one data pack to the cloud-based server prior to storing.

16. the data file is a new version of a file in a revision control system (RCS), and the cloud-based server includes a repository for the RCS; 16. The computer-implemented method of claim 15, comprising performing the transfer as part of committing the new version of the file to the repository.

17. 16. The computer-implemented method of claim 15, further comprising performing the transfer in parallel by instantiating at least two respective transfer threads corresponding to the constructed at least one data pack.

18. 20. The computer-implemented method of claim 17, further comprising, in response to detecting a failure of a given transfer thread of the at least two instantiated transfer threads, restarting the given transfer thread.

19. 1. A computer-based system for storing data files, comprising: a processor; a memory having computer code instructions stored thereon, the processor and the memory using the computer code instructions to cause the computer-based system to: Partitioning the data file into a plurality of chunks; constructing a data structure representing the data file, the data structure including a hierarchical tree representing the plurality of chunks; generating respective chunk identifiers (IDs) corresponding to the plurality of chunks; identifying at least one non-duplicate chunk from among the plurality of chunks based on the generated respective chunk IDs; (i) constructing at least one data pack based on the constructed data structure and (ii) the identified at least one non-duplicate chunk; and storing the constructed at least one data pack in a database.

20. 1. A non-transitory computer program product for storing a data file, the non-transitory computer program product comprising a computer readable medium having stored thereon computer code instructions that, when executed by a processor, cause a device associated with the processor to: Partitioning the data file into a plurality of chunks; constructing a data structure representing the data file, the data structure including a hierarchical tree representing the plurality of chunks; generating respective chunk identifiers (IDs) corresponding to the plurality of chunks; identifying at least one non-duplicate chunk from among the plurality of chunks based on the generated respective chunk IDs; (i) constructing at least one data pack based on the constructed data structure and (ii) the identified at least one non-duplicate chunk; storing the constructed at least one data pack in a memory.

Images