Incrementally determining checksum of snapshots

Abstract

Incrementally determining a checksum of snapshots is described. An example method includes identifying, by a data platform implemented by a computing system, a plurality of leaf nodes and a plurality of intermediate nodes in tree data corresponding to a snapshot of a storage system at a particular time, where the intermediate nodes each include one or more pointers identifying one or more of the leaf nodes, and the leaf nodes each include an indication of file system data of the storage system. The method comprises the following steps: determining a checksum of each of the leaf nodes through the data platform; determining, by the data platform, a checksum for each intermediate node based on the checksum for the one or more leaf nodes identified by the pointer for the intermediate node; and storing, by the data platform, the checksum for each of the leaf nodes and each of the intermediate nodes.

Description

Technical Field

[0001] The present disclosure relates to a data platform for a computing system. Background Art

[0002] A data platform that supports computing applications may rely on a primary storage system to support latency-sensitive applications. However, since primary storage is typically more difficult to scale or has a higher cost of scaling, a secondary storage system is typically relied upon to support secondary use cases, such as backup and archiving.

[0003] A file system snapshot may be a copy or representation of the entire file system or a particular subset thereof at a point in time. The snapshot captures the state of files and directories at a particular moment, thus providing a snapshot of how the data of the file system appears at that particular point in time. File system snapshots are typically used for backup and recovery purposes and can provide benefits in terms of data protection and system consistency. File system data may include objects (e.g., files, directories) of the file system, metadata, or both. Summary of the Invention

[0004] Aspects of the present disclosure describe techniques for incrementally generating checksums of snapshots of file system data. File system data, such as objects, may be divided into smaller portions or chunks. For example, multiple individual chunks in one or more chunkfiles may together contain the data of an object of the file system. To assist in snapshot verification, some data platforms may store a checksum for each chunk, which may be used to verify the chunk by comparing the checksum of the chunk with a previously stored checksum of the chunk. However, such data platforms that utilize checksums may not be able to determine in some cases whether a snapshot is consistent (e.g., whether the snapshot completely and accurately represents the complete state of the file system). For example, the checksum does not indicate whether the chunks are in the correct order or out of order. Thus, even if the data platform verifies each chunk in a set of chunks using the checksum, the data stored in a snapshot that includes the set of chunks may still be inconsistent.

[0005] The techniques described herein incrementally generate checksums for snapshots rather than generating checksums for entire snapshots. For example, a data platform can utilize the data structures used to implement a snapshot to generate an aggregated checksum for checksums of multiple blocks directly or indirectly referenced by nodes in the data structure. The aggregated checksum can be stored in association with the node, and the aggregated checksum can be calculated based on the respective checksums of the multiple blocks. When subsequent snapshots are generated based on file system data, resulting in modification of one or more of the multiple blocks, the aggregated checksum is recalculated based on the modified blocks. In this way, the modified aggregated checksum represents an incrementally generated checksum for that portion of the snapshot (e.g., the blocks directly or indirectly referenced by the node) and is an incremental change to the overall checksum data of the snapshot. For portions of the snapshot that have nodes referencing blocks that were not modified relative to a previous snapshot of the file system data, the data platform can avoid recalculating checksums and aggregated checksums based on the unmodified blocks. The data platform can store the incrementally generated checksums (including aggregated checksums) and use the checksums to verify the snapshots.

[0006] The techniques of the present disclosure can provide one or more technical advantages for implementing one or more practical applications. As described above, incrementally generating an aggregated checksum for portions of blocks allows the data platform to avoid recalculating checksums and recalculating checksums for entire snapshots based on unmodified block data. Thus, compared to a data platform that generates checksums for entire snapshots for each snapshot, the techniques can have reduced system requirements (e.g., memory, memory bandwidth, and computing resources) and costs. The techniques can also reduce the time required to complete the snapshot checksum process, thus improving system responsiveness. For example, compared to a data platform that generates checksums for entire snapshots, the techniques can increase the ability of the data platform to perform other tasks (including storage and retrieval of data).

[0007] Although the techniques described in the present disclosure are primarily described in relation to the backup function of a data platform (e.g., verifying a backup in the form of a snapshot), similar techniques can also be applied to an archival function (e.g., verifying an archive) or other similar functions of a data platform. In some examples, in addition to verifying a backup or an archive, the techniques described herein can also be used to verify file system data in a live file system.

[0008] In one example, the present disclosure describes a method that includes identifying, by a data platform implemented by a computing system, a plurality of leaf nodes and a plurality of intermediate nodes in tree-shaped data corresponding to a snapshot of a storage system, where each of the plurality of intermediate nodes includes one or more pointers identifying one or more of the plurality of leaf nodes, and each of the plurality of leaf nodes includes an indication of file system data of the storage system. The method includes: determining, by the data platform, a checksum for each of the plurality of leaf nodes; determining, by the data platform, a checksum for the intermediate nodes based on the respective checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of each of the plurality of intermediate nodes; and storing, by the data platform, the checksum for each of the plurality of leaf nodes and the checksum for each of the plurality of intermediate nodes.

[0009] In another example, the present disclosure describes a computing system that includes a processing device configured to: identify a plurality of leaf nodes and a plurality of intermediate nodes in tree-shaped data corresponding to a snapshot of a storage system, where each of the plurality of intermediate nodes includes one or more pointers identifying one or more of the plurality of leaf nodes, and each of the plurality of leaf nodes includes an indication of file system data of the storage system. The processing device is further configured to: determine a checksum for each of the plurality of leaf nodes; determine a checksum for the intermediate nodes based on the respective checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of each of the plurality of intermediate nodes; and store, by the data platform, the checksum for each of the plurality of leaf nodes and the checksum for each of the plurality of intermediate nodes.

[0010] In another example, the present disclosure describes a computer-readable storage medium that includes instructions that, when executed, cause one or more processors to: identify a plurality of leaf nodes and a plurality of intermediate nodes in tree-shaped data corresponding to a snapshot of a storage system, where each of the plurality of intermediate nodes includes one or more pointers identifying one or more of the plurality of leaf nodes, and each of the plurality of leaf nodes includes an indication of file system data of the storage system. When further executed, the instructions cause the one or more processors to: determine a checksum for each of the plurality of leaf nodes; determine a checksum for the intermediate nodes based on the respective checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of each of the plurality of intermediate nodes; and store, by the data platform, the checksum for each of the plurality of leaf nodes and the checksum for each of the plurality of intermediate nodes.

[0011] Details of one or more embodiments of the present invention will be set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the present invention will be apparent from the specification, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figures 1A to 1B is a block diagram illustrating an example system for incrementally determining a checksum of a snapshot in accordance with one or more aspects of the present disclosure.

[0013] Figure 2 is a block diagram illustrating an example system for incrementally determining a checksum of a snapshot in accordance with the techniques of the present disclosure.

[0014] Figures 3A to 3C is a block diagram illustrating an example tree-shaped data having corresponding multiple sets of nodes in accordance with the techniques of the present disclosure.

[0015] Figures 4A to 4C is a block diagram illustrating an example tree-shaped data having corresponding multiple sets of nodes in accordance with the techniques of the present disclosure. [[ID=S19]]

[0016] Figure 5 is a flowchart illustrating example operations of a data platform during incrementally determining a checksum of a snapshot.

[0017] Figure 6 is a block diagram illustrating an example component for batch verification of a checksum.

[0018] Throughout the specification and the drawings, like reference numerals denote like elements. DETAILED DESCRIPTION

[0019] Figures 1A to 1B is a block diagram illustrating an example system for incrementally determining a checksum of a snapshot in accordance with one or more examples of the present disclosure. In Figure 1A the example, system 100 includes application system 102. Application system 102 represents a collection of hardware devices, software components, and / or data memories that can be used to implement one or more applications or services provided to one or more mobile devices 108 and one or more client devices 109 via network 113. Application system 102 may include one or more physical or virtual computing devices that execute a workload 174 for the application or service. Workload 174 may include one or more virtual machines, containers, Kubernetes pods each including one or more containers, bare-metal processes, and / or other types of workloads.

[0020] In Figure 1AIn the example, the application system 102 includes application servers 170A - 170M (collectively referred to as "application servers 170") connected via a network to a database server 172 that implements a database. Other examples of the application system 102 may include one or more load balancers, web servers, network devices such as switches or gateways, or other devices for implementing one or more applications or services and delivering the one or more applications and services to the mobile device 108 and the client device 109. The application system 102 may include one or more file servers. One or more file servers may implement the primary file system of the application system 102. (In such cases, the file system 153 may be a secondary file system that provides backup, archiving, and / or other services for the primary file system. References to file systems in this document may include the primary file system or the secondary file system, e.g., the primary file system of the application system 102 or the file system 153 operating as the primary file system or the secondary file system.)

[0021] The application system 102 may be located locally and / or in one or more data centers, where each data center is part of a public, private, or hybrid cloud. The application or service may be a distributed application. The application or service may support enterprise software, financial software, office or other productivity software, data analysis software, customer relationship management, web services, educational software, database software, multimedia software, information technology, healthcare software, or other types of applications or services. The application or service may be provided as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), data storage as a service (dSaaS), or other types of services.

[0022] In some examples, the application system 102 may represent an enterprise system that includes one or more workstations in the form of desktop computers, laptop computers, mobile devices, enterprise servers, network devices, and other hardware for supporting enterprise applications. Enterprise applications may include enterprise software, financial software, office or other productivity software, data analysis software, customer relationship management, web services, educational software, database software, multimedia software, information technology, healthcare software, or other types of applications. Enterprise applications may be delivered as services from an external cloud service provider or other provider, executed natively on the application system 102, or both.

[0023] In Figure 1AIn the example, system 100 includes data platform 150, which uses storage system 105 and separate storage system 115 to provide file system 153 and archival functions to application system 102. Data platform 150 implements distributed file system 153 and storage architecture to facilitate access to file system data by application system 102 and to facilitate data transfer between storage system 105 and application system 102 via network 111. In terms of the distributed file system, data platform 150 enables the devices of application system 102 to access file system data via network 111 using a communication protocol as if such file system data were stored locally (e.g., stored on the hard disk of the device of application system 102). Example communication protocols for accessing files and objects include Server Message Block (SMB), Network File System (NFS), or AMAZON Simple Storage Service (S3). File system 153 can be the primary file system or an auxiliary file system of application system 102.

[0024] File system manager 152 represents a collection of hardware devices and software components that implement file system 153 for data platform 150. Examples of file system functions provided by file system manager 152 include storage space management, including deduplication, file naming, directory management, metadata management, partitioning, and access control. File system manager 152 executes communication protocols to facilitate access by application system 102 to files and objects stored in storage system 105 via network 111.

[0025] The data platform 150 includes a storage system 105 having one or more storage devices 180A - 180N (collectively referred to as "storage devices 180"). The storage devices 180 may represent one or more physical or virtual computing and / or storage devices that include or are capable of accessing storage media. Such storage media may include one or more of the following: flash drives, solid state drives (SSDs), hard disk drives (HDDs), various electrically programmable memories (EPROMs) or electrically erasable programmable (EEPROM) memories, and / or other types of storage media for supporting the data platform 150. Different storage devices of the storage devices 180 may have different mixes of various types of storage media. Each of the storage devices 180 may include system memory. Each of the storage devices 180 may be a storage server, a network attached storage (NAS) device, or may represent the disk storage of a computing device. The storage system 105 may be a redundant array of independent disks (RAID) system. In some examples, one or more of the storage devices 180 are both computing devices and storage devices that execute software for the data platform 150, such as the file system manager 152 and the backup manager 154 in the example of system 100. In some examples, a separate computing device (not shown) executes software for the data platform 150, such as the file system manager 152 and the backup manager 154 in the example of system 100. Each of the storage devices 180 may be regarded as and referred to as a "storage node" or simply a "node". The storage devices 180 may represent virtual machines running on a supported hypervisor, cloud virtual machines, physical rack-mounted servers, or computing models installed in a converged platform.

[0026] In various examples, the data platform 150 runs physically, virtually, or cloud-natively. For example, the data platform 150 may be deployed as a physical cluster, a virtual cluster, or a cloud-based cluster that runs in a private cloud, a hybrid private / public cloud, or a public cloud deployed by a cloud service provider. In some examples of system 100, multiple instances of the data platform 150 may be deployed, and the file system 153 may be replicated in each instance. In some cases, the data platform 150 may be a computing cluster representing a single management domain. The number of storage devices 180 may be scaled to meet performance requirements.

[0027] The data platform 150 can implement multiple storage domains and provide the storage domains to one or more tenants or separate workloads 174 that require different data policies. The storage domain can be a data policy domain that determines policies for deduplication, compression, encryption, tiering, and other operations performed on objects stored in the storage domain. In this way, the data platform 150 can provide users with the flexibility to choose global data policies or workload-specific data policies. The data platform 150 can support partitioning.

[0028] A view can be a protocol export residing within a storage domain. A view can inherit the data policy of its storage domain, but additional data policies can be specified for the view. A view can be exported via SMB, NFS, S3, and / or another communication protocol. Policies that determine data processing and storage of the data platform 150 can be assigned at the view level. A protection policy can specify a backup frequency and a retention policy, which can include a data lock period. Snapshots 142 or archives created according to the protection policy inherit the data lock period and retention period specified by the protection policy.

[0029] Each of network 113 and network 111 can be the Internet, or can include or represent any public or private communication network or other network. For example, network 113 can be cellular, near field communication (NFC), satellite, enterprise, service provider, and / or other types of networks capable of transmitting data between computing systems, servers, computing devices, and / or storage devices. One or more of such devices can use any suitable communication technology to transmit and receive data, communications, control signals, and / or other information across network 113 or network 111. Each of network 113 or network 111 can include one or more network hubs, network switches, network routers, satellite dishes, or any other network device. Such network devices or components can be operatively interconnected, thereby for exchanging information between computers, devices, or other components (e.g., between one or more client devices or systems and one or more computer / server / storage devices or systems). Figures 1A to 1B Each of the devices or systems shown can be operatively coupled to network 113 and / or network 111 using one or more network links. The links coupling such devices or systems to network 113 and / or network 111 can be Ethernet, asynchronous transfer mode (ATM), or other types of network connections, and such connections can be wireless and / or wired connections. Figures 1A to 1B One or more of the devices or systems shown or on network 113 and / or network 111 can be in a remote location relative to one or more of the other shown devices or systems.

[0030] The application system 102 can generate objects and other data using the file system 153 provided by the data platform 150. The file system manager 152 can create, manage the objects and data, and store the objects and data into the storage system 105. For this purpose, the application system 102 can alternatively be referred to as the "source system", and the file system 153 used by the application system 102 can alternatively be referred to as the "source file system". The application system 102 can directly communicate with the storage system 105 via the network 111 for some purposes to transfer objects, and communicate with the file system manager 152 via the network 111 for some purposes to indirectly obtain objects or metadata from the storage system 105. The file system manager 152 generates metadata and stores the metadata into the storage system 105. The collection of data stored in the storage system 105 and used to implement the file system 153 is referred to as file system data in this article. The file system data can include the aforementioned metadata and objects. The metadata can include file system objects, tables, trees, or other data structures; metadata generated to support deduplication; or metadata used to support snapshots. For example, as Figure 1A shown in the example of

[0031] the storage system 105 can store the metadata for the file system 153 in a tree data structure, such as the tree data 155. The stored objects can include files, virtual machines, databases, applications, pods, containers, any workload 174, system images, directory information, or other types of objects used by the application system 102. Different types of objects and the same type of objects can be deduplicated relative to each other.

[0032] The storage system 115 includes one or more storage devices 140A - 140X (collectively referred to as "storage devices 140"). The storage devices 140 may represent one or more physical or virtual computing and / or storage devices that include or are capable of accessing storage media. Such storage media may include one or more of the following: flash drives, solid state drives (SSDs), hard disk drives (HDDs), optical discs, various electrically programmable memories (EPROMs) or electrically erasable programmable (EEPROM) memories, and / or other types of storage media. Different storage devices of the storage devices 140 may have different mixes of various types of storage media. Each of the storage devices 140 may include system memory. Each of the storage devices 140 may be a storage server, a network attached storage (NAS) device, or may represent the disk storage of a computing device. The storage system 115 may include a redundant array of independent disks (RAID) system. The storage system 115 is capable of storing a much larger amount of data than the storage system 105. The storage devices 140 may also be configured for long - term storage of information and are more suitable for archival purposes.

[0033] In some examples, the storage system 105 and / or 115 may be storage systems deployed and managed by a cloud storage provider and are referred to as "cloud storage systems". Example cloud storage providers include, for example, AMAZON WEB SERVICES (AWS TM ) of AMAZON company, DROPBOX of DROPBOX company TM , ORACLE CLOUD of ORACLE company TMand GOOGLE CLOUD PLATFORM (GCP) of GOOGLE Inc. In some examples, storage system 115 is co-located with storage system 105 in a data center, on-premises, or in a private, public, or hybrid private / public cloud. Storage system 115 can be regarded as a “backup” or “auxiliary” storage system of the primary storage system 105. Storage system 115 can be referred to as the “external target” of snapshot 142. When deployed and managed by a cloud storage provider, storage system 115 can be referred to as “cloud storage”. Storage system 115 can include one or more interfaces for managing the transfer of data between storage system 105 and storage system 115 and / or between application system 102 and storage system 115. The data platform 150 that supports application system 102 relies on the primary storage system 105 to support latency-sensitive applications. However, since storage system 105 is generally more difficult to scale or has a higher cost of scaling, the data platform 150 can use the auxiliary storage system 115 to support secondary use cases, such as backup and archiving. Generally speaking, a file system backup or snapshot 142 is a copy of the file system 153 used to support protecting the file system 153 for quick recovery (usually due to some data loss in the file system 153), and a file system archive (“archive”) is a copy of the file system 153 used to support long-term retention and viewing. A “copy” of the file system 153 can include such data required to restore or view the state of the file system 153 at the time of backup or archive.

[0034] The backup manager 154 can back up the file system data of the file system 153 at any time according to a backup policy that specifies, for example, backup periodicity and timing (daily, weekly, etc.), which file system data will be backed up, backup retention period, storage location, access control, etc. The initial backup of the file system data corresponds to the state of the file system data at the initial backup time (the backup creation time of the initial backup). According to the backup policy, the initial backup can include a full backup of the file system data or can include a less complete backup of the file system data. For example, the initial backup can include all objects of the file system 153 or one or more selected objects of the file system 153.

[0035] One or more subsequent incremental backups of the file system 153 may correspond to the respective state of the file system 153 at the respective subsequent backup creation times (i.e., after the backup creation time corresponding to the initial backup). A subsequent backup may include an incremental backup of the file system 153. A subsequent backup may correspond to an incremental backup of one or more objects of the file system 153. Some file system data of the file system 153 stored on the storage system 105 at the initial backup creation time may also be stored on the storage system 105 at the subsequent backup creation times. A subsequent incremental backup may include data not previously stored in a backup at the storage system 115. The backup manager 154 may deduplicate the file system data included in a subsequent backup against the file system data included in one or more previous backups (including the initial backup) to reduce the storage amount used. (References to "time" in this disclosure may refer to date and / or time. Time may be associated with a date. For example, multiple backups may occur at different times on the same day.)

[0036] In the system 100, the backup manager 154 stores a backup of the file system data as a snapshot 142 to the storage system 115 using the block file 162. The backup manager 154 may use any of the snapshots 142 to later restore the file system (or a portion thereof) to its state at the snapshot creation time, or for example, may use the snapshot to create or present a new file system (or "view") based on the snapshot. As described above, the backup manager 154 may deduplicate the file system data included in a subsequent snapshot against the file system data included in one or more previous snapshots. For example, the second object of the file system 153 included in a second snapshot may be deduplicated against the first object of the file system 153 included in an earlier first snapshot. The backup manager 154 may remove the data blocks ("blocks") of the second object and generate metadata having a reference (e.g., a pointer) to the blocks stored in a block 164 in one block file 162. The blocks stored in this example are instances of the blocks stored for the first object.

[0037] The backup manager 154 may apply deduplication as part of the write process of writing (i.e., storing) an object of the file system 153 to a snapshot 142 in the storage system 115. Deduplication can be implemented in a variety of ways. For example, the method can be fixed-length or variable-length, the block size of the file system can be fixed or variable, and the deduplication domain can be applied globally or per workload. Fixed-length deduplication involves dividing the data stream at fixed intervals. Variable-length deduplication involves dividing the data stream at variable intervals to improve the ability to match data, regardless of which file system block size method is used. This algorithm is more complex than the fixed-length deduplication algorithm, but may be more effective for most cases and generally produces less metadata. Variable-length deduplication can include variable-length, sliding window deduplication. The length of any deduplication operation (whether fixed-length or variable-length) determines the size of the blocks being deduplicated.

[0038] In some examples, for variable-length deduplication, the block size can be within a fixed range. For example, the backup manager 154 can calculate blocks with a block size in the range of 16 kB to 48 kB. The backup manager 154 can avoid deduplicating objects smaller than 16 kB. In some example implementations, when considering deduplicating the data of an object, the backup manager 154 compares the block identifier (ID) of the data (e.g., the hash value of the entire block) with the existing block IDs of the stored blocks. If a match is found, the backup manager 154 can update the metadata of the object to point to the matching, stored block. If no matching block is found, the backup manager 154 can write the data of the object as a block within a block file 164 in a block file 162 to the memory. Additionally, the backup manager 154 can store the block ID associated with the newly stored block in the block metadata to allow future deduplication against the newly stored block. Generally, for any of the snapshots 142, the block metadata can be used to generate, view, retrieve, or restore an object stored as a block 164 (and references to it) within the block file 162, and this is described in more detail below.

[0039] Each of the chunk files 162 includes a plurality of chunks 164. The chunk files 162 can be of a fixed size (e.g., 8MB) or variable size. The chunk files 162 can be stored in the same location as the snapshot metadata, such as the tree data 120. In some cases, a data structure provided by a cloud storage provider for the storage system 115 can be used to store the chunk files 162. For example, each of the chunk files 162 can be one of the following: an S3 object within an AWS cloud storage bucket, an object within an AZURE Blob storage, an object in an object storage for ORACLE CLOUD, or other similar data structures used within another cloud storage provider's storage system.

[0040] The process of deduplicating multiple objects across multiple snapshots results in the chunk files 162, each of which has a plurality of chunks 164 for a plurality of different objects associated with the plurality of snapshots. In some examples, different snapshots 142 can have objects that are actually copies of the same data, e.g., for objects of a file system that have not been modified. The objects of the snapshots can be represented or "stored" as metadata having references to the chunks, such that access to the objects is enabled. Thus, the description herein of a snapshot 142 "storing", "having", or "including" an object includes instances where the snapshot does not store the data of the object in its native form.

[0041] A user or application associated with the application system 102 can access (e.g., read or write) data stored in the storage system 115. The user or application may delete some data due to malicious attacks (e.g., viruses, ransomware, etc.), rogue or malicious administrators, and / or human errors. The user's credentials may be leaked, and thus, the data stored in the storage system 115 may be subject to a ransomware attack. To reduce the likelihood of accidental or malicious data deletion or corruption, a data lock with a data lock period can be applied to the snapshots.

[0042] As described above, the chunk files 162 can represent objects in a snapshot storage system (shown as "storage system 115", which can also be referred to as "snapshot storage system 115") that conform to the underlying architecture of the snapshot storage system 115. The data platform 150 includes a backup manager 154 that supports archiving data in the form of chunk files 162, which interface with the snapshot storage system 115 to store the chunk files 162 after the chunk files 162 are formed from one or more chunks 164 of the data. The backup manager 154 can apply a process called "deduplication" to the chunks 164 to remove redundant chunks and generate metadata that links the redundant chunks to previously stored chunks 164, thereby reducing the storage consumed (and thus reducing the storage cost in terms of storing the chunks).

[0043] Backup manager 154 may organize and store file system data (e.g., one or more objects or metadata) as tree data 120. In some examples, backup manager 154 may store tree data 120 on storage system 115. Tree data 120 may include one or more tree data structures that include nodes linked by pointers. For example, backup manager 154 may store tree data 120 that includes a tree data structure that stores file system data (e.g., an object or metadata) of a file system at one or more nodes of the tree data structure. Backup manager 154 may traverse tree data 120 to locate file system data (e.g., an object or metadata of a file system). In some examples, backup manager 154 may organize tree data 120 into one or more separate tree data structures. For example, backup manager 154 may store tree data 120 that includes a tree data structure for metadata and a separate tree data structure for objects. In some examples, backup manager 154 may store tree data 120 that includes a tree data structure within another tree data structure (e.g., a nested tree data structure or a subtree).

[0044] A tree data structure includes a plurality of nodes, where each node references one or more other nodes, such as by one or more pointers to the other nodes. A tree structure may include a hierarchy of nodes. For example, a tree data structure may include a root node, one or more intermediate nodes, and one or more leaf nodes, or various subsets thereof. As described above, a tree data structure may include nested tree structures (e.g., subtrees), each of which may include a root node, one or more levels of intermediate nodes, and one or more leaf nodes, or various subsets thereof.

[0045] In some examples, backup manager 154 may utilize a tree structure based on a B+ tree data structure. For example, backup manager 154 may store file system data and retrieve file system data from a tree data structure that includes a root node and intermediate nodes that form an index for locating file system data. In this example, backup manager 154 may store file system data (e.g., an object or metadata) at leaf nodes. In some examples, backup manager 154 may store one or more references (e.g., pointers) to file system data at leaf nodes instead of storing the file system data in the leaf nodes. For example, backup manager 154 may store one or more references (e.g., pointers) to one or more blocks 164 of an object (which may be stored in storage system 115) at leaf nodes instead of storing the object or one or more of its blocks 164 at leaf nodes.

[0046] Snapshot 142 may include tree data 120 (e.g., tree data 120 may be used to store one or more snapshots 142). Each snapshot 142 may record or store the state of the file system at different times. For example, backup manager 154 may store tree data 120, which includes a first snapshot 142 of the state of the entire file system at a first time and a second snapshot 142 that includes incremental changes to the file system since the first snapshot. In some examples, backup manager 154 may store tree data 120, which includes one or more snapshots 142 of the state of the entire file system and one or more snapshots 142 that include incremental changes to the file system since an earlier snapshot.

[0047] Backup manager 154 may traverse the tree data 120 of the snapshots to retrieve a copy (e.g., a backup) of the file system (e.g., the file system data of the file system) at a particular time (such as the time requested by the user). For example, backup manager 154 may locate a snapshot having a timestamp that matches the time requested by the user (or other time) and traverse tree data 120 starting from the root node of tree data 120 to retrieve the data of the snapshot. Backup manager 154 may retrieve one or more incremental snapshots or all of the snapshots of the file system while traversing tree data 120. Backup manager 154 may apply the incremental snapshots to an earlier incremental or full snapshot to generate or output a copy of the file system at a particular time. Additional examples and techniques for storing and retrieving file system data in a tree structure are described in U.S. Patent Application No. 17 / 960,515, “MAINTAINING AND UPDATING A BACKUP VIEW OF AN APPLICATION AND ITS ASSOCIATED OBJECTS,” filed on October 5, 2022, the entire content of which is hereby incorporated by reference.

[0048] According to the techniques of the present disclosure, backup manager 154 includes a checksum module 160 for generating checksums 130 for one or more nodes in tree data 120. Checksum module 160 may implement one or more functions that receive an input and calculate a checksum 130 that represents the input with a relatively small amount of data (e.g., 64 or 128 characters). For example, checksum module 160 may implement various procedures for generating checksums with an associated property such that a change in the order or sequence of the input data results in a different checksum. Some example methods for generating checksums include hash functions (such as Secure Hash Algorithm (e.g., SHA-1 or SHA-2)), bitwise AND, and combining n significant bits of the input data.

[0049] The checksum module 160 can incrementally generate the checksum 130 of a snapshot by generating individual checksums 130 at each node within the snapshot's tree data 120. In operation, the checksum module 160 can utilize the checksums 130 from one or more child nodes (if any) as input to generate the checksum 130 of the parent node. Such checksums are referred to herein as "aggregated checksums". For example, the checksum module 160 can convert each checksum 130 from one or more child nodes into a matrix (e.g., convert a 256-bit checksum into a 16x16 matrix). The checksum module 160 can perform matrix multiplication on each matrix in sequence (e.g., from left to right) to generate the aggregated checksum of the parent node. Some nodes (such as leaf nodes) may not have child nodes. In such cases, the checksum module 160 can use the data within the leaf node or associated with the leaf node to generate the checksum 130 of the leaf node. For example, the checksum module 160 can use the file system data (e.g., objects or metadata) of the leaf node or other data (e.g., pointers) as input to generate the checksum 130.

[0050] In some examples, the backup manager 154 can incrementally generate checksums by generating checksums only for the portions of the file system data that have changed relative to a previous snapshot. For example, a first snapshot can have checksums at each node of the first snapshot's tree data 120, such as can be generated by the checksum module 160. The backup manager 154 can create a second snapshot with tree data 120 that includes pointers to new nodes representing changes since the first snapshot and pointers to nodes in the first snapshot representing file system data that has not changed since the first snapshot. The checksum module 160 can incrementally generate the checksum of the second snapshot by generating checksums for the new nodes in the second snapshot. In this way, the checksum module 160 avoids generating checksums for nodes representing unchanged file system data and avoids generating checksums across the entire snapshot.

[0051] The backup manager 154 can create new nodes by cloning nodes from an earlier snapshot. For example, to include a new leaf node, the backup manager 154 can clone a subtree from the first snapshot that includes a leaf node with now-outdated file system data. The cloned subtree may include the parent and ancestor nodes of the leaf node with the outdated file system data. The backup manager 154 can modify one or more pointers within the cloned subtree to point to the new leaf node instead of the leaf node with the outdated file system data. In this way, the snapshot that includes the cloned subtree (e.g., the second snapshot) includes the changed file system data.

[0052] The checksum module 160 may omit or clear the checksums of each node in the clone subtree to allow the checksum module 160 to identify the nodes that require checksums. The checksum module 160 may generate checksums at each node with an empty or cleared checksum to incrementally generate the checksum of a snapshot including the clone subtree, which is the portion of the nodes in the snapshot representing the changed file system data. In some examples, instead of clearing or omitting the checksums, the checksum manager 160 may include a flag or other data in the clone nodes to identify these nodes as requiring checksums.

[0053] Conversely, some data platforms may generate the checksum of the entire snapshot instead of incrementally generating the checksum. In the case of a large snapshot (e.g., 5 gigabytes (GB) or larger), generating the checksum of the entire snapshot requires a large amount of computation because the entire snapshot may need to be processed to generate the checksum. Snapshots in an enterprise environment can typically reach hundreds of GB, several terabytes (TB) or larger, and a data platform that generates checksums for these snapshots will consume a large amount of computing resources and in some cases take a long time (e.g., several hours). To verify the checksum of the entire snapshot, the data platform generates another checksum for the entire snapshot, thus again requiring similar computing resources and time to generate the checksum for verification purposes.

[0054] To incrementally generate the checksum for the tree data 120, in some examples, the checksum module 160 may use the checksum 130 of each intermediate node that is a child of the root node as input to generate the checksum 130 of the root node. Similarly, the checksum module 160 may use the checksum 130 of each of the children of the intermediate node (e.g., leaf nodes or other intermediate nodes) to generate the checksum 130 of the intermediate node. In some examples, the checksum module 160 may calculate the aggregate checksum of the parent node based on the respective checksums 130 of the child nodes, such as by concatenating the checksums 130 of the child nodes and using the combined checksum 130 as input to generate the aggregate checksum 130 of the parent node. As other examples, the checksum module 160 may calculate the aggregate checksum of the parent node based on the respective checksums 130 of the child nodes of the parent node according to a bitwise AND or XOR (or other bitwise operators), according to a combination of n significant bits, according to the checksum of the checksum 130, or according to another function of the checksums 130 of the child nodes.

[0055] In some examples, when incrementally generating the checksum of a snapshot, the checksum module 160 may use pointers between nodes of the tree data 120 to traverse the tree data 120 of the snapshot and generate a checksum 130 at each node. To traverse the tree data 120, the checksum module 160 may use the pointers between nodes to locate the respective nodes within the tree data 120. In some examples, the checksum module 160 may implement a depth-first traversal process, whereby each child node is traversed before traversing or returning to the parent node, such as a post-order traversal process. In some examples, the checksum module 160 may perform a post-order traversal to traverse the tree data 120 of the snapshot, with the traversal order starting from lower-level nodes (e.g., child nodes) and then proceeding to higher-level nodes (e.g., parent nodes). The checksum module 160 may repeat the post-order traversal process for each subtree within the tree data 120, starting from the leaf nodes in the tree data 120 and ultimately ending at the root node of the tree data 120. By traversing the tree structure from lower level to higher level (e.g., post-order traversal), the checksum 130 from the lower-level nodes (e.g., child nodes) may be used to generate the checksum 130 at the corresponding higher-level nodes (e.g., parent nodes).

[0056] During traversal, the checksum module 160 may determine whether a node includes a pointer, such as to determine whether the node is a leaf node (e.g., a node without child nodes). If the checksum module 160 determines that the node is a leaf node, the checksum module 160 may use the data within or associated with the node as input to generate the checksum 130. For example, the checksum module 160 may generate a checksum for a leaf node having an object (e.g., one or more blocks 164), metadata, or other data stored within or referenced by the leaf node. In some examples, the node type may be determined by an identifier assigned to the node. For example, the checksum module 160 may determine whether a node is a leaf node or another type of node based on the identifier of the node.

[0057] In some examples, the checksum module 160 may initiate the post-order traversal checksum generation process when the backup manager 154 generates a full or incremental snapshot 142 of the file system data. As will be further described below, for an incremental snapshot 142, the checksum module 160 may generate a checksum for the node representing the incremental snapshot 142 without generating a checksum for a previous snapshot 142.

[0058] To verify the snapshot, the backup manager 154 may perform a verification traversal on the tree data 120. The verification traversal may be a traversal as described above (e.g., a post-order traversal). During the verification traversal, the backup manager 154 may generate a verification checksum for each node and compare the verification checksum with the previously generated checksum 130 of the node. The backup manager 154 may generate the verification checksum in the same manner as generating the checksum 130. When determining that the verification checksum of a particular node does not match the checksum, the backup manager 154 may determine that the snapshot is invalid.

[0059] If no mismatched verification checksum is identified, the backup manager 154 may determine that the snapshot is valid. In some examples, the backup manager 154 may determine that the snapshot is valid by examining a portion (e.g., a subtree) of the tree data 120 rather than all of the tree data 120. For example, if the checksum 130 of the root node matches the verification checksum of the root node, the backup manager 154 may determine that the snapshot is valid. To generate the verification checksum of the root node, the backup manager 154 may traverse only the subtree of the tree data 120 that includes the root node, rather than traversing all of the tree data 120.

[0060] In some examples, the backup manager 154 may use batch processing to verify the checksum, such as using a map-reduce process. For example, the backup manager 154 may traverse the tree data 120 to generate one or more maps that store the checksum, the parent node, and one or more child nodes of each node. The backup manager 154 may perform one or more reduce processes on the nodes in the one or more maps to generate the verification checksum for each node, whereby the checksum from the child nodes may be retrieved from the map and combined to generate the verification checksum of the parent node. The backup manager 154 may compare the verification checksum of each node to verify the snapshot.

[0061] Figure 1B The system 190 is Figure 1A a variant of the system 100, where the data platform 150 uses a block file 162 to store the snapshot 142, and the block file is stored to a snapshot storage system 115 that resides locally or, in other words, local to the data platform 150. In some examples of the system 190, the storage system 115 enables a user or an application to create, modify, or delete the block file 162 via the file system manager 152. In the system 190, ​The storage system 105 is a local storage system where the backup manager 154 initially stores and accumulates blocks before they are stored at the storage system 115. The backup manager 154 can store the tree data 120 at the storage system 115, where the tree data includes nodes having references (e.g., pointers) to one or more blocks 164. Although not shown, in some examples, in addition to or instead of the storage system 115, the backup manager 154 can also store the tree data 120 and the checksum 130 at the storage system 105, regardless of whether the storage system 115 is remote or local to the data platform 150.

[0062] ​ is a block diagram illustrating an example system 200 in accordance with the techniques of the present disclosure. ​ The system 200 can be described as ​ the system 100 or ​ an example or alternative implementation of the system 190 (where the block file 162 is written to the local snapshot storage system 115). The backup manager 154 can store the tree data 120 at the local snapshot storage system 115, where the tree data includes one or more nodes having references (e.g., pointers) to the blocks 164. This can be described herein in ​ and ​ the context of ​ one or more aspects.

[0063] In ​ the example, the system 200 includes a network 111, a data platform 150 implemented by a computing system 202, and a storage system 115. In ​ this, the network 111, the data platform 150, and the storage system 115 can correspond to ​ the network 111, the data platform 150, and the storage system 115. Although only one snapshot storage system 115 is depicted, the data platform 150 can use multiple instances of the snapshot storage system 115 to apply the techniques according to the present disclosure. Different instances of the storage system 115 can be deployed by different cloud storage providers, the same cloud storage provider, an enterprise, or other entities.

[0064] Computing system 202 can be implemented as any suitable computing system, such as one or more server computers, workstations, mainframes, appliances, cloud computing systems, and / or other computing systems that can perform operations and / or functions described according to one or more aspects of the present disclosure. In some examples, computing system 202 represents a cloud computing system, server farm, and / or server cluster (or portion thereof) that provides services to other devices or systems. In other examples, computing system 202 can represent or be implemented by one or more virtualized computing instances (e.g., virtual machines, containers) of a cloud computing system, server farm, data center, and / or server cluster.

[0065] exist ​ In the example of FIG, computing system 202 may include one or more communication units 215, one or more input devices 217, one or more output devices 218, and one or more storage devices of local storage system 105. Local storage system 105 may include interface module 226, file system manager 152 and policy 158, as well as backup manager 154 and checksum module 160. One or more of the devices, modules, storage areas, or other components of computing system 202 may be interconnected to enable inter-component communication (physically, communicatively, and / or operationally). In some examples, such connections may be provided through communication channels (e.g., communication channels 212), which may represent one or more of a system bus, a network connection, an inter-process communication data structure, or any other method for transferring data.

[0066] The computing system 202 includes a processing device. ​ In the example of FIG, the processing device includes one or more processors 213, which are configured to implement the operations associated with or in conjunction with the computing system 202. ​ The functionality and / or execution associated with one or more modules shown and described below may be associated with or related to the computing system 202. ​ The one or more processors 213 may be, may be part of, and / or may include processing circuitry that performs operations in accordance with one or more aspects of the present disclosure. Examples of processors 213 include a microprocessor, an application processor, a display controller, an auxiliary processor, one or more sensor hubs, and any other hardware configured to function as a processor, processing unit, or processing device. The computing system 202 may use the one or more processors 213 to perform operations in accordance with one or more aspects of the present disclosure using software, hardware, firmware, or a mixture of hardware, software, and firmware resident in and / or executing at the computing system 202.

[0067] One or more communication units 215 of the computing system 202 may communicate with devices external to the computing system 202 by transmitting and / or receiving data, and in some aspects, may operate as both an input device and an output device. In some examples, the communication unit 215 may communicate with other devices via a network. In other examples, the communication unit 215 may transmit and / or receive radio signals over a radio network (such as a cellular radio network). In other examples, the communication unit 215 of the computing system 202 may transmit and / or receive satellite signals over a satellite network. Examples of the communication unit 215 include network interface cards (e.g., such as Ethernet cards), optical transceivers, radio frequency transceivers, GPS receivers, or any other type of device that can transmit and / or receive information. Other examples of the communication unit 215 may include devices capable of communicating via GPS, NFC, and cellular networks (e.g., 3G, 4G, 5G) and radios found in mobile devices, as well as universal serial bus (USB) controllers, etc. Such communications may comply with, implement, or follow appropriate protocols, including Transmission Control Protocol / Internet Protocol (TCP / IP), Ethernet, NFC, or other technologies or protocols. NFC or other technologies or protocols.

[0068] One or more input devices 217 may represent any input device of the computing system 202 not otherwise separately described herein. The input device 217 may generate, receive, and / or process an input. For example, one or more input devices 217 may generate an input or receive an input from a network, a user input device, or any other type of device for detecting an input from a human or a machine.

[0069] One or more output devices 218 may represent any output device of the computing system 202 not otherwise separately described herein. The output device 218 may generate, present, and / or process an output. For example, one or more output devices 218 may generate, present, and / or process any form of output. The output device 218 may include one or more USB interfaces, video and / or audio output interfaces, or any other type of device capable of generating tactile, audio, visual, video, electrical, or other outputs. Some devices may be used as both an input device and an output device. For example, a communication device may send data to other systems or devices via a network and receive data from other systems or devices.

[0070] One or more storage devices of the local storage system 105 within the computing system 202 can store information for processing during operation of the computing system 202, such as random access memory (RAM), flash memory, solid state drive (SSD), hard disk drive (HDD), etc. The storage devices can store program instructions and / or data associated with one or more modules described in accordance with one or more aspects of the present disclosure. One or more processors 213 and one or more storage devices can provide an operating environment or platform for such modules, which can be implemented as software, but in some examples can include any combination of hardware, firmware, and software. One or more processors 213 can execute instructions, and one or more storage devices of the storage system 105 can store instructions and / or data of one or more modules. The combination of the processor 213 and the local storage system 105 can retrieve, store, and / or execute instructions and / or data of one or more applications, modules, or software. The storage devices of the processor 213 and / or the local storage system 105 can also be operatively coupled to one or more other software and / or hardware components, including but not limited to the computing system 202 and / or one or more components of one or more devices or systems shown as connected to the computing system 202.

[0071] The file system manager 152 can perform functions related to providing the file system 153, as described above with respect to ​ The file system manager 152 can generate and manage file system metadata 232 for constructing the file system data 230 of the file system 153, and store the file system metadata 232 and the file system data 230 into the local storage system 105. The file system metadata 232 can include one or more trees, such as the tree-shaped data 155, which describe the objects within the file system 153 and the file system 153 hierarchy, and can be used to write or retrieve objects within the file system 153. The file system manager 152 can interact and / or cooperate with one or more modules of the computing system 202, including the interface module 226 and the backup manager 154.

[0072] The backup manager 154 can perform functions related to backing up the file system 153, as described above with respect to ​as described, including the operations described above with respect to the checksum module 160. The backup manager 154 may generate one or more snapshots 142 and cause the file system data 230 to be stored as tree data 120 and blocks 164 within block files 162 in the snapshot storage system 115. The backup manager 154 may apply a deduplication process according to one or more policies 158 to selectively deduplicate blocks of objects within the file system data 230. The backup manager 154 may generate and manage the tree data 120 for generating, viewing, retrieving, or restoring any of the snapshots 142. The backup manager 154 may generate and manage the tree data 120 for generating, viewing, retrieving, or restoring an object stored as a block 164 (and references thereto) within a block file 162 for any of the snapshots 142. Logical files may be used to represent and manipulate the stored objects to identify the blocks of the objects.

[0073] The local storage system 105 may store a block table that describes the blocks 164. The block table may include the respective block IDs of the blocks 164 and may contain a pointer to the block file 162 and an offset within the block file 162 for retrieving the block 164 from the storage system 115. The blocks 164 are written to the block file 162 at different offsets. By comparing the new block ID with the block table, the backup manager 154 may determine whether the data already exists on the system. If the block already exists, the data may be discarded and the metadata of the object may be updated to reference the existing block. The backup manager 154 may use the block table to look up the block file identifier of the block file that contains the block.

[0074] The local storage system 105 may include a block file table that describes the respective physical or virtual locations of the block files 162 on the storage system 115, as well as other metadata about the block files, such as checksums, encrypted data, compressed data, etc. In ​ which, the backup manager 154 causes the tree data 120 to be stored to the local storage system 105. In some examples, the backup manager 152 causes some or all of the tree data 120 to be stored to the snapshot storage system 115. Optionally or in combination with the file system manager 152, the backup manager 152 may use the tree data 120 and / or the file system metadata 232 to restore any of the snapshots 142 to the file system implemented by the data platform 150, and the snapshot may be presented by the file system manager 152 to other systems.

[0075] The interface module 226 may perform an interface through which other systems or devices may determine the operations of the file system manager 152 or the backup manager 154. Another system or device may communicate via the interface of the interface module 226 to specify one or more policies 158.

[0076] System 200 can be modified to implement ​ an example of system 190. In the modified system 200, the block file 162 is stored in the local snapshot storage system 115 to support the snapshot 142.

[0077] The interface module 240 of the snapshot storage system 115 can execute an interface through which other systems or devices can create, modify, delete, or extend the expiration time of a write-once-read-many (WORM) lock for any of the block files 162. The interface module 240 can execute and present an API. The interface presented by the interface module 240 can be gRPC, HTTP, RESTful, command line, graphical user interface, network interface, or other interfaces. The interface module 240 can be associated with a usage cost. One or more methods or functions of the interface module 240 may incur a per-use cost (for example, it costs $0.10 to extend the expiration time of the WORM lock for the block file 162).

[0078] ​ is a block diagram showing an example tree data 300 with corresponding multiple sets of nodes according to the technology of the present disclosure. For example, referring to ​ , the tree data 300 can be a tree data structure and can include one or more root nodes 302, one or more intermediate nodes 304, and one or more leaf nodes 306 interconnected by pointers. Although shown as including one layer of intermediate nodes 304 between the root node 302 and the leaf node 306, the tree data 300 can include additional intermediate layers between the root node 302 and the leaf node 306 (for example, intermediate nodes 304 with pointers to other intermediate nodes 304).

[0079] ​ An example of shows the tree data 300 of a snapshot of file system data. The root node 302 includes one or more pointers pointing to one or more other nodes in the snapshot. In some examples, the root node 302 can form an entry point to the snapshot because the snapshot can be retrieved by traversing the tree data 300 starting from the root node 302. The intermediate node 304 can be a node pointed to by another node and includes pointers to other nodes. The leaf node 306 can be a node at the bottom of the tree data 300 and may not have pointers to other nodes. Each node 302, 304, 306 can include a checksum 308.

[0080] In some examples, each node 302, 304, 306 in the tree data 300 can have respectively by ​The node identifier, tree identifier, or both, represented by the node ID and tree ID. The node identifier can be a name that uniquely identifies nodes 302, 304, 306. The tree identifier can be a string or other identifier that identifies the tree-shaped data structure (e.g., snapshot) to which the node belongs.

[0081] As described above, in some examples, the root node 302 and the intermediate node 304 can form an index through which file system data at the leaf node 306 can be located. For example, the root node 302 and the intermediate node 304 can include one or more node keys that indicate which pointer to traverse to locate a particular leaf node 306 (e.g., the leaf node 306 having the desired object data or metadata). In ​ the example, the root node 302 has a node key 2, the intermediate node 304A has a node key 1, and the intermediate node 304B has a node key 3. Each node key can have one or more pointers, and the pointers can be selected for traversal based on a comparison between the node key and the key value. For example, in ​ for each node key, for key values less than or equal to the node key, the first pointer can be traversed, and for values greater than the node key, the second pointer can be traversed. The key value can identify the leaf node 306 that contains or references the desired data. For example, the leaf node 306A has a key value 1, the leaf node 306B has a key value 2, and the leaf node 306C has a key value 3.

[0082] Thus, to locate the leaf node 306A having a key value 1, for example, the backup manager 154 can traverse the first pointer (e.g., the leftmost pointer) of the root node 302 that points to the intermediate node 304A because the key value 1 is less than or equal to the node key 2 at the root node 302. At the intermediate node 304A, the backup manager 154 can traverse the first pointer (e.g., the leftmost pointer) that points to the leaf node 306A because the key value 1 is less than or equal to the node key 1 at the intermediate node 304A. By traversing the leftmost pointer at the intermediate 304A, the backup manager 154 reaches the desired leaf node 306A having a key value 1.

[0083] To locate leaf node 306B with key value 2, for example, backup manager 154 can traverse the first pointer (e.g., the leftmost pointer) of root node 302 that points to intermediate node 304A, because the key value 2 is less than or equal to the node key 2 at root node 302. At intermediate node 304A, backup manager 154 can traverse the second pointer (e.g., the rightmost pointer) that points to leaf node 306B, because the key value 2 is greater than the node key 1 at intermediate node 304A. To locate leaf node 306C with key value 3, for example, backup manager 154 can traverse the second pointer (e.g., the rightmost pointer) of root node 302 that points to intermediate node 304B, because the key value 3 is greater than the node key 2 at root node 302. At intermediate node 304B, backup manager 154 can traverse the first pointer that points to leaf node 306C, because the key value 3 is less than or equal to the node key 3 at intermediate node 304B.

[0084] Each node 302, 304, 30 of the tree - shaped data 300 may include a checksum 308. Backup manager 154 can incrementally generate the checksum of the snapshot by traversing the tree - shaped data 300 and generating checksums 308 at respective nodes 302, 304, 306 of the tree - shaped data 300, where the checksum (e.g., aggregated checksum) of a parent node (e.g., a node having one or more child nodes) is based on the checksums of one or more child nodes. The checksum 308 of leaf node 306 can be based on the data in or associated with leaf node 306 (e.g., the object or metadata indicated by leaf node 306).

[0085] Backup manager 154 can perform a post - order traversal of the tree - shaped data 300, thereby generating a checksum 308 for each child node of a parent node before generating the checksum 308 for the parent node. For example, referring to ​ , backup manager 154 can use the data in or associated with leaf nodes 306A, 306B (such as file system data (e.g., one or more blocks 310 or metadata 314)) to generate checksum 308D ("68B5F") and checksum 308E ("70964") respectively. Backup manager 154 can use checksum 308D ("68B") of leaf node 306A and checksum 308E ("70964") of leaf node 306B as inputs to generate checksum 308B ("25E5D") of intermediate node 304A.

[0086] Continuing with the example, backup manager 154 may use the data in or associated with leaf node 306C to generate a checksum 308F ("1D2F4") for leaf node 306C, and use the checksum 308F ("1D2F4") of leaf node 306C as an input to generate a checksum 308C ("487BF") for intermediate node 304B. Backup manager 154 may use the checksum 308B ("25E5D") of intermediate node 304A and the checksum 308C ("487BF") of intermediate node 304B as inputs to generate a checksum 308A ("E2DCE") for root node 302. Backup manager 154 may store each checksum 308 in the node 302, 304, 306 where it was generated, such as at storage system 115.

[0087] As described above, the tree data 300 may store file system data including objects, metadata, or both. ​ An example of tree data 300 storing a snapshot of objects in a file system is shown. As can be seen, leaf nodes 306 of tree data 300 may be associated with one or more blocks 310 of data of one or more objects. For example, leaf node 306 may include pointers to one or more blocks 310 of a block file 312. Although not shown, in some examples, leaf node 306 may store one or more blocks 310 at leaf node 306 instead of including pointers to blocks 310. ​ The blocks 310 and block file 312 may be ​ examples of blocks 164 and block file 162.

[0088] ​ An example of tree data 300 storing a snapshot of metadata 314 of a file system is shown. As can be seen, whether the tree data 300 stores objects or metadata, the tree data 300 may include the same tree data structure. Thus, backup manager 154 may, for example, traverse ​ the tree data 300 and generate checksums 308 at nodes 302, 304, 306, such as described above with respect to ​ the tree data 300. For example, backup manager 154 may perform a post-order traversal on ​ the tree data 300 and generate checksums 308 during the traversal, as described with respect to ​ the tree data 300. Similarly, at ​ the leaf node 306, the data associated with leaf node 306 may be used as an input to generate a checksum 308. For example, at ​In the example of, at leaf node 306A, the metadata 314 of leaf node 306A can be used to generate checksum 308D. Although shown as storing metadata 314, in some examples, in addition to or instead of metadata 314, the tree data 300 can also store other data at one or more leaf nodes 306.

[0089] The backup manager 154 can perform a post-order traversal on multiple units of the tree data 300. Refer to ​ the example of, the tree data 300A is ​ an example of the tree data 300 of, and the tree data 300B is ​ an example of the tree data 300 of. The leaf node 306B of the tree data 300A includes a reference (e.g., a pointer) to the tree data 300B, such as in the metadata 314. Although not shown, the reference to the tree data 300B can be a reference to the root node of the tree data 300B.

[0090] In some examples, the tree data 300B can be regarded as a subtree of the tree data 300A. Therefore, the backup manager 154 can traverse the combined tree data 300A, 300B and generate and verify the checksum 308 as if the tree data 300A, 300B were a single unit of the tree data. In some examples, the backup manager 154 can independently generate and verify the checksum 308 for each unit of the tree data 300A, 300B. For example, the backup manager 154 can traverse the tree data 300B to generate and verify the checksum of the tree data 300B, and separately traverse the tree data 300A to generate and verify the checksum of the tree data 300A. In this case, the backup manager 154 can separately indicate whether the tree data 300A, the tree data 300B or both are valid or invalid.

[0091] ​ The checksum 308 of is an example. Although shown as a hexadecimal string, the checksum 308 can be a combination of various characters or data, which is different for different input data and even different for input data that is only slightly different (e.g., <1%). As described above, the backup manager 154 can implement various methods for generating the checksum 308, including secure hash algorithms (e.g., SHA-1, SHA-2), bitwise AND, and combining n significant bits of the input data.

[0092] As can be seen, ​ the checksum 308 of is relative to ​ is different because ​ the checksum 308 of is generated based on different data (e.g., metadata 314 instead of block 310). For example,​ The metadata 314 at leaf node 306 of ​ generates checksum 308D ("17AA4"), checksum 308E ("9FB13"), and checksum 308F ("C9B70"). During the post-order traversal checksum generation process, checksums 308D, 308E, 308F propagate upward, thereby generating ​ checksum 308B ("06937") and checksum 308C ("707BD") at intermediate node 304 of ​ the tree-shaped data 300 and checksum 308A ("57487") at root node 302. Therefore,

[0093] the checksum 308 of ​ is different with respect to ​ Similarly, with respect to

[0094] the metadata 314 at leaf node 306B includes a reference to the tree-shaped data 300B. Thus, for this example, assuming that

[0095] the tree-shaped data 300 of

[0096] ​ is otherwise the same, checksum 308E ("3E17B") is different (because leaf node 306B includes a reference to tree-shaped data 300B), which generates checksum 308B ("5F709") at intermediate node 304A and checksum 308A ("8A25E") at root node 302 as checksum 308E propagates upward during the post-order traversal checksum generation process.

[0094] To verify the file system data stored in the tree-shaped data 300, the backup manager 154 can perform a verification traversal (e.g., another post-order traversal) and generate verification checksums as described above with respect to generating checksum 308. For example, as described above with respect to generating checksum 308, the backup manager 154 can use the data associated with leaf node 306 as input to generate the verification checksum of leaf node 306 and use the checksums 308 of the children of intermediate node 304 and root node 302 as input to generate the verification checksums of intermediate node 304 and root node 302 (e.g., collectively referred to as aggregated checksums).

[0095] The backup manager 154 can compare the verification checksum of a node with the previously generated checksum 308 of the node at one or more nodes 302, 304, 306. If each verification checksum matches (e.g., is the same as) the checksum 308 of nodes 302, 304, 306, the backup manager 154 can determine that the tree-shaped data 300 is verified. Alternatively, upon determining that at least one verification checksum of nodes 302, 304, 306 does not match the previously stored checksum 308 of nodes 302, 304, 306, the backup manager 154 can determine that the tree-shaped data 300 is invalid.

[0096] In some examples, the backup manager 154 may traverse only a subtree of the tree data 300 to determine the validity or invalidity of a snapshot. For example, referring to ​ , the backup manager 154 may traverse a subtree that includes the root node 302, the intermediate node 304B, and the leaf node 306C. The backup manager 154 may perform a post-order traversal of the subtree and generate a verification checksum at each node 302, 304B, 306C during the traversal and compare the verification checksum with the checksum 308 at each node 302, 304B, 306C. For example, the verification checksum of the leaf node 306C may be compared with the checksum 308F, the verification checksum of the intermediate node 304B may be compared with the checksum 308C, and the verification checksum of the root node 302 may be compared with the checksum 308A.

[0097] When traversing to the root node 302, if each verification checksum has a matching checksum (based on each node), the backup manager 154 may determine that the snapshot in the tree data 300 is valid. As can be seen, the backup manager 154 may not traverse or generate and compare the verification checksums of the nodes 304A, 306A, 306B outside the subtrees of the nodes 302, 304B, 306C. During the verification traversal, if at least one verification checksum does not match, the backup manager 154 may determine that the snapshot in the tree data 300 is invalid (e.g., contains corrupted data) and output an indication thereof. Once it is determined that the tree data 300 is invalid, the backup manager 154 may stop traversing.

[0098] ​ is a block diagram showing an example tree data 400 having corresponding sets of nodes according to the techniques of the present disclosure. The tree data 400 may be ​ an example of the tree data 300. ​Shows an example of incrementally generating a checksum of a snapshot when file system data changes. In some examples, the backup manager 154 can regenerate the checksum 308 for each node that is a changed, deleted, or newly added node or the parent node of a node with a regenerated checksum. In this way, the checksum 308 can propagate upward from the changed, deleted, or newly added node to the root node 302 in the tree-shaped data 400. In some examples, the backup manager 154 can perform a post-order traversal from the changed nodes 302, 304, 306 to the parent node or ancestor node (such as the root node 302), and generate the checksum 308 at each node 302, 304, 306 during the traversal to propagate the checksum upward in the tree-shaped data 400. At the end of this traversal, the root node 302 can store the checksum 308 resulting from the propagation of the checksum of each intermediate node 304 and leaf node 306 of the root node 302.

[0099] For example, referring to ​ , a new leaf node 306D is added to replace the leaf node 306B. As indicated by its dashed representation, the leaf node 306D and the pointer pointing to it can be removed from the tree-shaped data 400. Since the checksum 308B at the intermediate node 304A is based on the child node checksums 308D and 308E, the removal of the leaf node 306B and the addition of the leaf node 306D result in regenerating the checksum 308B using the checksums 308D, 308G. Since the checksum 308A at the root node 302 is based on the checksum 308B, the checksum 308A is regenerated using the regenerated checksum 308B and the checksum 308C.

[0100] For illustration, the backup manager 154 can generate the checksum 308G of the leaf node 306D, such as using the file system data associated with the leaf node 306D. The backup manager 154 can clear the existing checksums 308 of each parent node and ancestor node (in this case, the intermediate node 304A and the root node 302) of the leaf node 306D. Then, the backup manager 154 can generate the checksum 308 of each such parent node and ancestor node. For example, the backup manager 154 can use the checksums 308D, 308G to generate the checksum 308B of the intermediate node 304A and use the checksums 308B, 308C to generate the checksum 308A of the root node. In some examples, the backup manager 154 can simply overwrite the checksum 308 without first clearing the checksum 308.

[0101] As can be seen, the subtrees including nodes 302, 304A, 306D associated with changes to the tree data 400 (e.g., newly added leaf node 306D and removed leaf node 306B) have updated or regenerated checksums 308A, 308B, 308G. The subtrees including unchanged nodes 304B, 306A, 306C have unchanged checksums 308C, 308D, 308F. Thus, instead of generating a checksum for the entire snapshot represented by the tree data 400 or for all intermediate nodes (regardless of whether the referenced block data or metadata of the subtree has been modified with a new snapshot), the backup manager 154 can generate the checksum for the new snapshot by generating checksums 308 for those nodes associated with the changed part of the snapshot (e.g., the subtree). For example, the backup manager 154 can generate a checksum 308 for each of the nodes associated with the change and the parent and ancestor nodes of the nodes.

[0102] ​ An example where the leaf node 306C is updated is shown. For example, the leaf node 306C may include a pointer to block 310, and the pointer is modified to point to another block 310. In response to the change, the backup manager 154 can clear the checksums 308A, 308C, 308F of the nodes in the subtree of the tree data 400 associated with the change (e.g., the modification of the leaf node 306C). Then, the backup manager 154 can incrementally generate the checksum of the subtree. For example, the backup manager 154 can generate the checksum 308F of the leaf node 306C based on the pointer to block 310 or the data in block 310. The backup manager 154 can use the checksums 308F, 308G to generate the checksum 308C, and use the checksums 308B, 308C to generate the checksum 308A.

[0103] ​ An example where the tree data 400 stores multiple snapshots 402 is shown. In ​ the example, the root node 302A represents the first snapshot 402A, while the root node 302B represents the second snapshot 402B. The snapshot 402A may represent the file system at a first time, while the snapshot 402B may represent the file system at a subsequent second time. The snapshot 402B may include one or more changes made to the file system since the snapshot 402A. For example, in ​In this case, snapshot 402B includes leaf node 306B instead of leaf node 306D. For example, leaf node 306D may be associated with new or updated file system data. Backup manager 154 may traverse the subtree starting from root node 302A to retrieve the file system data of snapshot 402A, and traverse the subtree starting from root node 302B to retrieve the file system data of snapshot 402B. As can be seen, the tree identifier of snapshot 402B (e.g., tree ID: 2) identifies nodes 302B, 304C, 306D added for creating snapshot 402B.

[0104] Backup manager 154 may create snapshot 402B by copying or cloning one or more nodes of an existing snapshot such as snapshot 402A. In some examples, backup manager 154 may clone a subtree of tree-shaped data 400, such as a subtree including one or more nodes 302, 304, 306 related to changes. For example, to replace leaf node 306B, backup manager 154 may clone the subtree including the parent node and ancestor nodes of leaf node 306B (in this case, intermediate node 304A and root node 302A). For example, backup manager 154 may clone (e.g., copy) root node 302A to generate root node 302B, and clone (e.g., copy) intermediate node 304A to generate intermediate node 304C.

[0105] The clone of a node may be the same as the node, including the pointers of the node. For example, although not shown, when cloned, root node 302B may include pointers to intermediate nodes 304A, 304B as seen in root node 302A. Similarly, when cloned, intermediate node 304C may include pointers to leaf nodes 306A, 306B as seen in intermediate node 304A. In some examples, when cloning nodes 302, 304, 306, backup manager 154 may omit or clear the checksum 308 of the node. Thus, the checksum 308 of the cloned node may be empty when cloned and subsequently generated by traversing the cloned node, as will be further described below.

[0106] Backup manager 154 may modify one or clone nodes when creating snapshot 402B. For example, as ​ shown, backup manager 154 may modify intermediate node 304C such that intermediate node 304C has a pointer to leaf node 306D instead of leaf node 306B, because leaf node 306D replaces leaf node 306B in snapshot 402B. Backup manager 154 may modify root node 302B to have a pointer to intermediate node 304C instead of intermediate node 304A.

[0107] Snapshot 402B may include pointers to nodes in other snapshots 402A. In this way, file system data common among the snapshots 402 can be referenced from other snapshots 402 instead of being copied. For example, as ​ shown, both snapshots 402A and 402B include an intermediate node 304B and thus also include a leaf node 306C and the file system data associated therewith. Leaf node 306A is also included in both snapshots 402A and 402B.

[0108] The backup manager 154 can incrementally generate a checksum 308 for each snapshot 402 by performing a post-order traversal as described above. For example, the backup manager 154 can perform a post-order traversal on snapshot 402B starting from the root node 302B. During the traversal, the backup manager 154 can generate a checksum 308 for a node whose checksum is cleared or empty (e.g., a clone node). For example, the backup manager 154 can use the data associated with the leaf node 306D to generate the checksum 308I. Continuing with this example, the backup manager 154 can use the checksums 308D and 308I to generate the checksum 308H and use the checksums 308C and 308H to generate the checksum 308G. At the end of the traversal, the root node 302B of the snapshot 402B contains the checksum 308G of all the data in or referenced by the snapshot 402B. Thus, the checksum 308G at the root node 302B represents the checksum of the entire snapshot 402B.

[0109] The backup manager 154 can verify each snapshot 402 by performing a post-order traversal. For example, to verify snapshot 402A, the backup manager 154 can perform a post-order traversal starting from the root node 302A, and to verify snapshot 402B, the backup manager 154 can perform a post-order traversal starting from the root node 302B. The backup manager 154 can generate and compare verification checksums at each node accessed by the backup manager 154 during the traversal and determine whether the snapshot 402 is valid or invalid based on whether each verification checksum matches the corresponding checksum 308 as described above.

[0110] ​ is a flowchart showing an example operation of the data platform in the process of incrementally determining the checksum of a snapshot. The following is described in the context of ​ and ​ . As ​As shown in the example of , the backup manager 154 can identify multiple leaf nodes 306 and multiple intermediate nodes 304(502) in the tree data 400 corresponding to a backup or snapshot 402A of the storage system 105 at a specific time. For example, the tree data 400 can include a tree data structure storing a snapshot 402A of the file system data of the storage system 105 at a specific time. Each intermediate node 304 can include one or more pointers identifying one or more leaf nodes 306. Each leaf node 306 can include an indication of the file system data of the storage system. As described above, for example, a leaf node 306 can include file system data or include a pointer or other reference to file system data (e.g., block 310) stored outside the leaf node 306.

[0111] The backup manager 154 can determine a checksum 308(504) for each leaf node 306. For example, the backup manager 154 (such as via the checksum module 160) can use the data associated with the leaf node 306A (e.g., file system data or a pointer thereto) as input to generate a checksum 308D for the leaf node 306A. The backup manager 154 can generate a checksum 308 at each leaf node 306 of the snapshot 402A by repeating the checksum generation process described for the leaf node 306A.

[0112] The backup manager 154 can determine a checksum 308 for each individual intermediate node 304A of the intermediate nodes 304 based on the checksums 308 of the leaf nodes 306 identified by the pointers of each individual intermediate node 304A(506). For example, at the intermediate node 304A, the backup manager 154 (such as via the checksum module 160) can use the checksums 308D, 308E of the leaf nodes 306A, 306B to generate a checksum 308B. The backup manager 154 can generate a checksum 308 at each intermediate node 304 of the snapshot 402A by repeating the checksum generation process described for the intermediate node 304A.

[0113] In some examples, determining the checksum 308 for each leaf node 306 and determining the checksum 308 for each intermediate node 304 can include traversing the tree data 400 in a postorder sequence. The checksum 308 for the leaf nodes 306 and the checksum 308 for the intermediate nodes 304 can be determined according to the postorder sequence. For example, the backup manager 154 can generate checksums 308D, 308E and use the checksums 308D, 308E to generate a checksum 308B. Continuing with this example, the backup manager 154 can generate a checksum 308F and use the checksum 308F to generate a checksum 308C. The backup manager 154 can then use the checksums 308B, 308C to generate a checksum 308A.

[0114] A checksum 308A can be generated for the root node 302A in the tree - shaped data 400. For example, the backup manager 154 can identify the root node 302A in the snapshot 402A of the tree - shaped data 400. The root node 302 can include multiple pointers identifying intermediate nodes 304. The backup manager 154 can determine the checksum 308A of the root node 302 based on the checksums 308B, 308C of the intermediate nodes 304A, 304B identified by the pointers of the root node 302A.

[0115] In some examples, a checksum 308 can be generated for a portion of the tree - shaped data 400 (e.g., a subset of nodes) rather than for each node in the tree - shaped data 400. For example, the backup manager 154 can identify a subset of leaf nodes 306 in the tree - shaped data 300 that are associated with changes to the file system data of the storage system 105, and a subset of intermediate nodes 304 in the tree - shaped data 400 that have pointers to the subset of leaf nodes 306. For example, in ​ the example, the backup manager 154 can identify the leaf node 306D associated with a change to the tree - shaped data 400 (e.g., adding the leaf node 306D), and identify the intermediate node 304C that includes a pointer to the leaf node 306D. As can be seen, the subtree of the leaf node 306D, the intermediate node 304C, and the root node 302B corresponds to the change to the tree - shaped data 400 of the file system data of the storage system 105 (and represents the snapshot 402B). The backup manager 154 can regenerate the checksum 308 for each node associated with the change (in this case, the nodes 302, 304A, 306D).

[0116] In some examples, clone nodes can be used to identify a subset of nodes associated with changes to the tree - shaped data 400. For example, the backup manager 154 can identify clone leaf nodes 306 in the tree - shaped data 400 that are associated with changes to the file system data of the storage system 105, and identify a subset of intermediate nodes 304 in the tree - shaped data 400 that have pointers to the clone leaf nodes 306. For example, referring to ​ the example, the backup manager 154 can identify the clone leaf node 306D and the intermediate node 304C that has a pointer to the clone leaf node 306D. The backup manager 154 can also identify the clone root node 302B. As described above, the backup manager 154 can generate the checksum 308 for each clone node 302B, 304C, 306D. In some examples, the clone nodes may have empty or cleared checksums 308. Thus, the backup manager 154 can identify a subset of nodes associated with changes to the tree - shaped data 400 based on whether the nodes have empty or filled checksums 308, where an empty checksum indicates a node associated with a change to the tree - shaped data 400.

[0117] The backup manager 154 can store the checksum 308 of each leaf node 306 and the checksum 308 (508) of each intermediate node 304. For example, with respect to the snapshot 402A, the backup manager 154 can store the checksum 308D of the leaf node 306A, the checksum 308E of the leaf node 306B, and the checksum 308F of the leaf node 306C. Continuing with this example, the backup manager 154 can store the checksum 308B of the intermediate node 304A and the checksum 308C of the checksum 304B. As described above, the backup manager 154 can store the tree data 400 and the checksum 308 on the storage system 115.

[0118] The snapshot 402A of the storage system 105 can be verified by generating verification checksums at one or more nodes 302, 304, 306 of the tree data 400. For example, the backup manager 154 can determine the verification checksum of the intermediate node 304A based on the checksums of the leaf nodes 306A, 306B identified by the pointers from the intermediate node 304A. In some examples, the intermediate node 304A can include multiple pointers identifying at least two leaf nodes 306. In addition to the other nodes 302, 304, 306, the backup manager 154 can also compare the verification checksum of the intermediate node 304A with the checksum 308B of the intermediate node 304A, such as to determine the validity of the snapshot 402A.

[0119] An indication of the integrity (e.g., validity or invalidity) of the snapshot can be generated based on a per-node comparison between one or more verification checksums and the corresponding checksum 308. For example, if the verification checksum of the intermediate node 304A matches the checksum 308B, the user interface of the data platform 150 can output a snapshot integrity indication identifying the snapshot 402A as valid, and if the verification checksum does not match the checksum 308B, output a snapshot integrity indication identifying the snapshot 402A as invalid.

[0120] In some examples, the tree data 400 can include references to one or more subtrees including additional tree data 400. For example, an indication of file system data in the leaf node 306 of the first unit of the tree data can include an indication of the second unit of the tree data, the second unit having leaf nodes 306 including file system data. For example, as described above with respect to ​ the metadata 314 of the leaf node 306B in the tree data 300A can include a reference (e.g., pointer) to the tree data 300B. In this case, the tree data 300A stores a copy of the metadata of the storage system 105, while the tree data 300B stores a copy of the objects of the storage system 105.

[0121] Continuing with this example, the backup manager 154 can determine the checksum 308 for each leaf node in the tree data 300B based on the file system data in one or more of the leaf nodes of the tree data 300B. The backup manager 154 can determine the checksum 308E for the leaf node 306B of the tree data 300A by generating the checksum of the tree data 300B referenced by the leaf node 306B of the tree data 300A. The checksum of the tree data 300B can be the checksum at the root node of the tree data 300B.

[0122] ​ is a block diagram showing an example component for batch verification of checksums. For example, the checksum manager 160 can include one or more mappers 604, 606, one or more reducers 608, 610, or both to perform batch verification of the checksums of the snapshots. For example, in ​ the example, the checksum manager 160 includes a child-to-parent mapper 604, a node checksum manager 606, a child-to-parent reducer 608, and a parent checksum reducer 610. ​ The node 602 of ​ can be an example of a node (e.g., root node, intermediate node, and leaf node) of the tree data 300 as described with respect to ​ The operation of the components of the example of ​ is described in the context of a subtree of the nodes 304A, 306A, 306B of the tree data 300 including

[0123] One or more mappers 604, 606 can receive the node 602 as input and generate one or more maps based on the node 602. For example, the node checksum mapper 606 can scan or traverse the node 602 and output a key-value pair for each node. The key can be the node ID, and the value can be the checksum 308 of the node and a list of the node's child nodes. For example, the node checksum mapper 606 can output the following node checksum map for the subtree of the nodes 304A, 306A, 306B of the tree data 300 including ​ the tree data 300.

[0125] {I1:(25E5D,[68B5F,70964])},

[0126] {L1:(68B5F,[])},

[0127] {L2:(70964,[])}

[0129] ​​The child-to-parent mapper 604 can use the same scan or traversal to generate a mapping that maps child nodes to parent nodes. For example, the child-to-parent mapper 604 can output a mapping with the node ID as the key and the parent node ID as the value, such as the following child-to-parent mapping.

[0131] {L1:I1},

[0132] {L2:I1}

[0134] One or more reducers 608, 610 can process one or more mappings from mappers 604, 606. For example, the child-to-parent reducer 608 can process the output (e.g., mapping) from mappers 604, 606 and group the values by the corresponding keys (e.g., by node ID). The child-to-parent reducer 608 can output a key-value pair for each parent node, where the key is the parent node ID and the value is the child node and the checksum. For example, for node 396A (node ID: L1), the child-to-parent reducer 608 can reduce {L1:(68B5F,[])} and {L1:I1} from the above node checksum and the child-to-parent mapping to output {I1:(L1,68B5F)}.

[0135] The parent node checksum reducer 610 can group the output from the child-to-parent reducer 608 and the node checksum mapper 606 by node ID. At the parent node checksum reducer 610, for each key of all child node IDs, the child node checksum and the parent node checksum can be determined via the mapping or output of at least the node checksum mapper 606, the child-to-parent mapper 604, or both, and the child-to-parent reducer 608 can be used to reduce or group the mapping or output, such as by node ID.

[0136] For example, for node 304A (node ID: I1), the parent checksum reducer 610 can reduce the key-value pairs {I1:(L1,68B5F)}, {I1:(L2,70964)} from the child-to-parent reducer 608 and the key-value pair {I1:(25E5D,[68B5F,70964])} from the node checksum manager 606. The parent checksum reducer 610 can combine the checksums 308 of the child nodes to generate a verification checksum. As described above, the checksum 308 can be generated in various ways. For example, the parent checksum reducer can generate a verification checksum by converting the checksum 308 into a matrix and performing matrix multiplication on the matrix in order (e.g., from left to right), thereby generating a verification checksum.

[0137] ​​The checksum manager 160 can compare the verified checksum with the checksum 308 of the parent node. If the verified checksum does not match the parent node checksum, the parent node, the snapshot, or both can be marked or identified as corrupted. If no mismatched verified checksum is found, the snapshot can be considered valid. For example, the parent checksum reducer 610 can sequentially combine the checksums 308D, 308F (e.g., 68B5F, 70964) of child nodes 308D, 308E (node IDs: L1 and L2) and compare the resulting verified checksum with the checksum 308B of the parent node 304A (node ID: I1). If the verified checksum does not match, the parent checksum reducer 610 can identify the node 304A as corrupted.

[0138] Although the techniques described in this disclosure are primarily described with respect to backup or snapshot functions performed by a backup manager of a data platform, similar techniques can additionally or alternatively be applied to archival, replication, or cloning functions performed by the data platform. In such cases, the snapshots 142 are respectively archives, replicas, or clones.

[0139] For the processes, apparatuses, and other examples or illustrations described herein that are included in any flowcharts, certain operations, actions, steps, or events included in any of the techniques described herein can be performed in a different order, can be added, combined, or entirely omitted (e.g., not all described actions or events are necessary for the practice of the technique). Additionally, in some examples, the operations, actions, steps, or events can be performed simultaneously, such as by multithreading, interrupt handling, or multiple processors, rather than sequentially. Additionally, even if not explicitly identified as being performed automatically, certain operations, actions, steps, or events can be performed automatically. Additionally, certain operations, actions, steps, or events described as being performed automatically can alternatively not be performed automatically, but rather, in some examples, such operations, actions, steps, or events can be performed in response to an input or another event.

[0140] The detailed description set forth herein in connection with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein can be implemented. The detailed description includes specific details for providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0141] According to one or more aspects of the present disclosure, when the context does not otherwise specify, the term "or" may be construed as "and / or". Additionally, although phrases such as "one or more" or "at least one" may be used in some cases; however, in cases where such language is not used and the context does not otherwise specify, those cases may be interpreted as implicitly having such meaning.

[0142] In one or more examples, the described functionality may be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored on and / or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium may include: a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium; or a communication medium, which includes any medium that facilitates the transfer of a computer program from one place to another (e.g., in accordance with a communication protocol). Thus, the computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium, which is non-transitory, or (2) a communication medium such as a signal or a carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0143] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave from a website, server, or other remote source, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather are directed to non-transitory, tangible storage media. As used herein, disk and optical disks include compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), floppy disk, and Blu-ray disk, where disks typically reproduce data magnetically, while optical disks reproduce data optically with a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0144] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the terms "processor" or "processing circuitry" as used herein may each refer to any one of the foregoing structures or any other structure suitable for implementing the described techniques. Additionally, in some examples, the described functionality may be provided in a special purpose hardware and / or software module. Further, the techniques may be fully implemented in one or more circuits or logic elements.

[0145] The processing apparatus used herein may include the processing circuitry as described above. In some examples, the processing apparatus may include at least one processor and at least one memory having computer code that includes a set of instructions that, when executed by the at least one processor, cause the at least one processor to perform any of the functions described herein. In some examples, the processing apparatus may receive the computer code including the set of instructions from at least one memory coupled to the processing apparatus.

[0146] The techniques of the present disclosure may be implemented in a variety of apparatuses or devices, including wireless handsets, mobile or non-mobile computing devices, wearable or non-wearable computing devices, integrated circuits (ICs) or a set of ICs (e.g., a chipset). The present disclosure describes various components, modules, or units to emphasize a functional approach of the apparatus configured to perform the disclosed techniques, but not necessarily requires implementation by different hardware units. Rather, as described above, the various units may be combined in a hardware unit or provided by a collection of interoperating hardware units (including one or more processors as described above) in conjunction with suitable software and / or firmware.

[0147] Aspects of the present invention are provided with reference to the following clauses:

[0148] Clause 1. A method, the method comprising: identifying, by a data platform implemented by a computing system, a plurality of leaf nodes and a plurality of intermediate nodes in tree-shaped data corresponding to a snapshot of a storage system, wherein: each of the plurality of intermediate nodes includes one or more pointers identifying one or more of the plurality of leaf nodes; and each of the plurality of leaf nodes includes an indication of file system data of the storage system; determining, by the data platform, a checksum for each of the plurality of leaf nodes; determining, by the data platform, a checksum for the intermediate nodes based on the respective checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of each of the plurality of intermediate nodes; and storing, by the data platform, the checksum for each of the plurality of leaf nodes and the checksum for each of the plurality of intermediate nodes.

[0149] Clause 2. The method as described in Clause 1, the method further comprising: determining a verification checksum of the first intermediate node by the data platform based on the checksum of one or more of the plurality of leaf nodes identified by one or more pointers of the first intermediate node among the plurality of intermediate nodes; comparing the verification checksum of the first intermediate node with the checksum of the first intermediate node stored by the data platform; and outputting an indication of the integrity of the snapshot based on the comparison through a user interface of the computing system.

[0150] Clause 3. The method as described in Clause 2, wherein the intermediate node includes a plurality of pointers identifying at least two of the plurality of leaf nodes.

[0151] Clause 4. The method as described in Clause 1, the method further comprising: identifying, by the data platform, a root node in the tree-shaped data, wherein the root node includes a plurality of pointers identifying the plurality of intermediate nodes; and determining a checksum of the root node by the data platform based on the checksum of the plurality of intermediate nodes identified by the plurality of pointers of the root node.

[0152] Clause 5. The method as described in Clause 1, wherein determining the checksum of each of the plurality of leaf nodes and determining the checksum of each of the plurality of intermediate nodes includes traversing the tree-shaped data in a postorder sequence, and determining the checksum of each of the plurality of leaf nodes and the checksum of each of the plurality of intermediate nodes according to the postorder sequence.

[0153] Clause 6. The method as described in Clause 1, wherein: identifying the plurality of leaf nodes includes identifying a subset of leaf nodes in the tree-shaped data associated with a change in the file system data of the storage system; and identifying the plurality of intermediate nodes includes identifying a subset of intermediate nodes in the tree-shaped data, the subset of intermediate nodes including one or more pointers pointing to the subset of leaf nodes in the tree-shaped data associated with the change in the file system data of the storage system.

[0154] Clause 7. The method as described in Clause 1, wherein: identifying the plurality of leaf nodes includes identifying a plurality of cloned leaf nodes in the tree-shaped data associated with a change in the file system data of the storage system; and identifying the plurality of intermediate nodes includes identifying a subset of intermediate nodes in the tree-shaped data, the subset of intermediate nodes including one or more pointers pointing to the plurality of cloned leaf nodes in the tree-shaped data associated with the change in the file system data of the storage system.

[0155] Clause 8. The method as described in Clause 1, wherein the tree data is first tree data, and the indication of the file system data of the storage system in the plurality of leaf nodes includes an indication of second tree data having one or more leaf nodes including the file system data.

[0156] Clause 9. The method as described in Clause 8, the method further comprising, for each of the plurality of leaf nodes, determining a checksum of the second tree data based on the file system data in the one or more leaf nodes of the second tree data, wherein determining the checksum of each leaf node in the plurality of leaf nodes in the first tree data includes determining the checksum of the second tree data indicated in the leaf node.

[0157] Clause 10. A computing system, the computing system comprising: a memory storing instructions; and a processing circuit executing the instructions to: identify a plurality of leaf nodes and a plurality of intermediate nodes in tree data corresponding to a snapshot of a storage system, wherein: each of the plurality of intermediate nodes includes one or more pointers identifying one or more of the plurality of leaf nodes; and each of the plurality of leaf nodes includes an indication of the file system data of the storage system; determine a checksum for each of the plurality of leaf nodes; determine a checksum of the intermediate node based on the respective checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of each intermediate node of the plurality of intermediate nodes; and store the checksum of each of the plurality of leaf nodes and the checksum of each of the plurality of intermediate nodes via the data platform.

[0158] Clause 11. The computing system as described in Clause 10, wherein the processing circuit further executes the instructions to: determine a verification checksum of the first intermediate node based on the checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of the first intermediate node of the plurality of intermediate nodes; compare the verification checksum of the first intermediate node with the checksum of the first intermediate node stored by the data platform; and output an indication of the integrity of the snapshot via a user interface of the computing system based on the comparison.

[0159] Clause 12. The method as described in Clause 2, wherein the intermediate node includes a plurality of pointers identifying at least two of the plurality of leaf nodes.

[0160] Clause 13. The computing system as described in Clause 10, wherein the processing circuit further executes the instructions to: identify a root node in the tree data, wherein the root node includes a plurality of pointers identifying the plurality of intermediate nodes; and determine a checksum of the root node based on the checksums of the plurality of intermediate nodes identified by the plurality of pointers of the root node.

[0161] Clause 14. The computing system as described in Clause 10, wherein: to determine the checksum of each of the plurality of leaf nodes and to determine the checksum of each of the plurality of intermediate nodes, the processing circuit further executes the instructions to traverse the tree data in a postorder sequence; and determine the checksum of each of the plurality of leaf nodes and the checksum of each of the plurality of intermediate nodes according to the postorder sequence.

[0162] Clause 15. The computing system as described in Clause 10, wherein: to identify the plurality of leaf nodes, the processing circuit further executes the instructions to identify a subset of leaf nodes in the tree data that are associated with changes to the file system data of the storage system; and to identify the plurality of intermediate nodes, the processing circuit further executes the instructions to identify a subset of intermediate nodes in the tree data, the subset of intermediate nodes including one or more pointers to the subset of leaf nodes in the tree data that are associated with changes to the file system data of the storage system.

[0163] Clause 16. The computing system as described in Clause 10, wherein: to identify the plurality of leaf nodes, the processing circuit further executes the instructions to identify a plurality of cloned leaf nodes in the tree data that are associated with changes to the file system data of the storage system; and to identify the plurality of intermediate nodes, the processing circuit further executes the instructions to identify a subset of intermediate nodes in the tree data, the subset of intermediate nodes including one or more pointers to the plurality of cloned leaf nodes in the tree data that are associated with changes to the file system data of the storage system.

[0164] Clause 17. The computing system as described in Clause 10, wherein the tree data is first tree data, and the indication of the file system data of the storage system in the plurality of leaf nodes includes an indication of second tree data having one or more leaf nodes including the file system data.

[0165] Clause 18. The computing system as described in Clause 17, wherein: the processing circuit further executes the instructions to: for each of the plurality of leaf nodes, determine a checksum of the second tree data based on the file system data in the one or more leaf nodes of the second tree data; and determining the checksum of each of the plurality of leaf nodes in the first tree data includes determining the checksum of the second tree data indicated in the leaf node.

[0166] Clause 19. A computer-readable storage medium, the computer-readable storage medium including instructions that, when executed, cause a processing circuit of a computing system to: identify a plurality of leaf nodes and a plurality of intermediate nodes in tree data corresponding to a snapshot of a storage system, wherein: each of the plurality of intermediate nodes includes one or more pointers identifying one or more of the plurality of leaf nodes; and each of the plurality of leaf nodes includes an indication of file system data of the storage system; determine a checksum for each of the plurality of leaf nodes; determine a checksum of the intermediate node based on the respective checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of each of the plurality of intermediate nodes; and store the checksum of each of the plurality of leaf nodes and the checksum of each of the plurality of intermediate nodes via the data platform.

[0167] Clause 20. The computer-readable storage medium as described in Clause 19, wherein when further executed, the instructions cause the processing circuit of the computing system to: determine a verification checksum of the first intermediate node based on the checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of the first intermediate node of the plurality of intermediate nodes; compare the verification checksum of the first intermediate node with the checksum of the first intermediate node stored by the data platform; and output an indication of the integrity of the snapshot via a user interface of the computing system based on the comparison.

Claims

1. A method, the method comprising: Identifying, by a data platform implemented by a computing system, a plurality of leaf nodes and a plurality of intermediate nodes in tree-shaped data corresponding to a snapshot of a storage system, wherein: Each of the plurality of intermediate nodes includes one or more pointers identifying one or more of the plurality of leaf nodes; and Each of the plurality of leaf nodes includes an indication of file system data of the storage system; Determining, by the data platform, a checksum for each of the plurality of leaf nodes; Determining, by the data platform, a checksum for the intermediate nodes based on the respective checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of each of the plurality of intermediate nodes; and Storing, by the data platform, the checksum for each of the plurality of leaf nodes and the checksum for each of the plurality of intermediate nodes.

2. The method of claim 1, the method further comprising: Determining, by the data platform, a verification checksum for a first intermediate node based on the checksums of the one or more of the plurality of leaf nodes identified by the one or more pointers of the first intermediate node; Comparing the verification checksum of the first intermediate node with the checksum of the first intermediate node stored by the data platform; And Outputting, by a user interface of the computing system, an indication of the integrity of the snapshot based on the comparison.

3. The method according to claim 2, wherein, If the verification checksum of the first intermediate node matches the checksum of the first intermediate node stored by the data platform, the indication of the integrity identifies the snapshot as valid.

4. The method of claim 2, wherein each intermediate node includes a plurality of pointers identifying at least two of the plurality of leaf nodes.

5. The method of claim 1, the method further comprising: Identifying, by the data platform, a root node in the tree-shaped data, wherein the root node includes a plurality of pointers identifying the plurality of intermediate nodes; And Determining, by the data platform, a checksum for the root node based on the checksums of the plurality of intermediate nodes identified by the plurality of pointers of the root node.

6. The method of claim 1, wherein determining the checksum for each of the plurality of leaf nodes and determining the checksum for each of the plurality of intermediate nodes includes traversing the tree-shaped data in a post-order sequence and determining the checksum for each of the plurality of leaf nodes and the checksum for each of the plurality of intermediate nodes according to the post-order sequence.

7. The method of claim 1, wherein: Identifying the plurality of leaf nodes includes identifying a subset of leaf nodes in the tree-shaped data associated with a change in the file system data of the storage system; And Identifying the plurality of intermediate nodes includes identifying a subset of intermediate nodes in the tree data, the subset of intermediate nodes including the one or more pointers that point to the subset of leaf nodes in the tree data that are associated with the change of the file system data of the storage system.

8. The method according to claim 1, wherein: identifying the plurality of leaf nodes includes identifying a plurality of cloned leaf nodes in the tree data that are associated with the change of the file system data of the storage system; and identifying the plurality of intermediate nodes includes identifying a subset of intermediate nodes in the tree data, the subset of intermediate nodes including the one or more pointers that point to the plurality of cloned leaf nodes in the tree data that are associated with the change of the file system data of the storage system.

9. The method according to claim 1, wherein the tree data is first tree data, and the indication of the file system data of the storage system in the plurality of leaf nodes includes an indication of second tree data, the second tree data having one or more leaf nodes including the file system data.

10. The method according to claim 9, the method further comprising, for each of the plurality of leaf nodes, determining a checksum of the second tree data based on the file system data in the one or more leaf nodes of the second tree data, wherein determining the checksum of each leaf node in the plurality of leaf nodes in the first tree data includes determining the checksum of the second tree data indicated in the leaf node.

11. A computing system, the computing system comprising processing means configured to: identify a plurality of leaf nodes and a plurality of intermediate nodes in tree data corresponding to a snapshot of a storage system, wherein: each of the plurality of intermediate nodes includes one or more pointers identifying one or more of the plurality of leaf nodes; and each of the plurality of leaf nodes includes an indication of the file system data of the storage system; determine a checksum for each of the plurality of leaf nodes; determine a checksum of the intermediate node based on the respective checksum of the one or more of the plurality of leaf nodes identified by the one or more pointers of each intermediate node of the plurality of intermediate nodes; and store the checksum of each of the plurality of leaf nodes and the checksum of each of the plurality of intermediate nodes.

12. The computing system according to claim 11, wherein the processing means is further configured to: determine a verification checksum of the first intermediate node based on the checksum of the one or more of the plurality of leaf nodes identified by the one or more pointers of the first intermediate node of the plurality of intermediate nodes; compare the verification checksum of the first intermediate node with the checksum of the first intermediate node; and output an indication of the integrity of the snapshot via a user interface of the computing system based on the comparison.

13. The computing system according to claim 12, wherein, If the verification checksum of the first intermediate node matches the checksum of the first intermediate node, the indication of the integrity of the snapshot identifies the snapshot as valid.

14. The computing system according to any one of claims 11 to 13, wherein the processing device is further configured to: Identify a root node in the tree-shaped data, wherein the root node includes a plurality of pointers identifying the plurality of intermediate nodes; and Determine a checksum of the root node based on the checksums of the plurality of intermediate nodes identified by the plurality of pointers of the root node.

15. A computer-readable storage medium comprising instructions that, when executed, cause one or more processors to perform the method according to any one of claims 1 to 10.

Images