Method implemented by computing system, computing system and computer readable storage medium

Abstract

The invention relates to a method implemented by a computing system, the computing system and a computer readable storage medium. Techniques for performing direct archiving of data block files are described. A computing system including a storage device and processing circuitry having access to the storage device may be configured to perform various aspects of the techniques. The processing circuitry may be configured to predict an incoming data rate of data to be archived to obtain a predicted incoming data rate of the data to be archived, and compare the predicted incoming data rate to a first threshold data rate. The processing circuitry may also be configured to, in response to determining that the predicted incoming data rate exceeds a first threshold data rate, segment the data into chunks, and write the chunks directly to an archive storage system as chunk files.

Description

Technical Field

[0001] The present disclosure relates to a data platform for a computing system. Background Art

[0002] Data platforms that support computing applications may rely on primary storage systems to support latency-sensitive applications. However, because primary storage is typically more difficult or expensive to scale, secondary storage systems are often relied on to support secondary use cases, such as backup and archiving. Summary of the invention

[0003] Aspects of the present disclosure describe techniques for adaptive archiving of data block files. Block files may each represent an object in an archival storage system that conforms to the underlying architecture of the archival storage system. A data platform that supports archiving may interface with the archival storage system to store block files after forming block files from one or more blocks of data. The data platform may apply a process called "deduplication" to the blocks to remove redundant blocks and specify metadata that links the redundant blocks to previously archived blocks, thereby reducing consumed storage (while also reducing storage costs in terms of storage required to store the blocks). The data platform may aggregate the blocks with the metadata to form block files at the archival storage system.

[0004] In certain contexts such as streaming, the data platform may receive a stream of data for which a fixed size (or in other words, a fixed amount) of data is unknown prior to archiving a block. The uncertainty in the size of the data contrasts with receiving all of the data in advance via a write request, where the data platform may know the range of the resulting block and thereby potentially optimize the size of the block file stored to the archival storage system. The uncertainty with respect to streaming of data with respect to the size of the data to be archived may cause the data platform to operate inefficiently with respect to archiving of blocks.

[0005] The data platform may operate inefficiently because, as one example, the data platform may not optimize the size of the block files and instead rely on initially storing the blocks to a storage system local to the data platform (which may also be referred to as a "local storage system"), which consumes additional memory, memory bandwidth, local storage and bandwidth, processing cycles, and / or other computing resources, and also results in latency when archiving the blocks, but provides more efficiently sized block files stored at the archival storage system. As another example, the data platform may avoid consuming computing resources to store blocks to the local storage system and reduce latency when archiving the blocks, but may do so at the expense of inefficiently sized block files (which are objects that include a batch of blocks) stored to the archival storage system, which inefficiently sized block files result in increased consumption of storage resources at the archival storage system and concomitant increased operating costs compared to more efficiently sized block files.

[0006] Rather than statically determining to store a block to a local storage system or to archive the block directly to an archival storage system, the data platform may instead predict the incoming data rate to obtain a predicted incoming data rate. The data platform may predict the incoming data rate for a given source (originating data, which may be referred to as "native format data"), for a given data stream, across streams from the same source that originated the data, and the like. Using this predicted incoming data rate, the data platform may apply one or more threshold data rates to adaptively and / or dynamically select between storing the block to a local storage system or storing (in other words, archiving) the block directly to an archival storage system for purposes of processing the block. The data platform may temporarily store the block in a local repository (e.g., random access memory - RAM and / or flash memory) that is separate from the local storage system used to store at least a portion of a local copy of the file system.

[0007] The technology can provide one or more technical advantages for implementing practical applications. For example, the technology can reduce computing resource consumption at the local storage system and the archival storage system by adaptively determining where the data stream should be stored before archiving the resulting batched blocks as one or more block files at the archival storage system for a given duration. The data platform can implement the technology to archive blocks more efficiently when the predicted incoming data rate indicates that data is to be received in an amount exceeding a threshold rate (for a given duration), thereby archiving blocks faster to potentially achieve an optimally sized (or in other words appropriately sized) cloud block file or block file on the archival storage system. The data platform can store blocks to the local storage system when the predicted incoming data rate indicates that data is to be received in an amount below a threshold rate (for a given duration). Thus, the data platform can reduce consumption of memory, memory bandwidth, and / or other computing resources at the local storage system, reduce disk writes and reads at the local storage system, reduce metadata and garbage collection load, and reduce latency when archiving chunks, while also potentially storing optimally sized chunk files that reduce consumption of computing resources at the archival storage system, such as by reducing metadata bloat caused by a large number of smaller sized chunk files.

[0008] Although the techniques described in this disclosure are primarily described with respect to an archiving function of a data platform, similar techniques may also be applied to a backup function or other similar functions of a data platform (potentially involving deduplication).

[0009] In one example, the present disclosure describes a method comprising: predicting, by a data platform implemented by a computing system, an incoming data rate of data to be archived; comparing the predicted incoming data rate to a first threshold data rate; and in response to determining that the predicted incoming data rate exceeds the first threshold data rate: segmenting, by the data platform, the data into blocks; and writing the blocks directly to an archival storage system as block files.

[0010] In another example, the present disclosure describes a computing system comprising: a storage device; and a processing component having access to the storage device and configured to: predict an incoming data rate of data to be archived to obtain a predicted incoming data rate of the data to be archived; compare the predicted incoming data rate with a first threshold data rate; and in response to determining that the predicted incoming data rate exceeds the first threshold data rate: split the data into blocks; and write the blocks directly to an archival storage system as block files.

[0011] In another example, the present disclosure describes a computer-readable storage medium comprising instructions that, when executed, cause one or more processors to: predict an incoming data rate of data to be archived to obtain a predicted incoming data rate for the data to be archived; compare the predicted incoming data rate to a first threshold data rate; and in response to determining that the predicted incoming data rate exceeds the first threshold data rate: split the data into blocks; and write the blocks directly to an archival storage system as block files. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A to Figure 1B is a block diagram illustrating an example system that performs adaptive direct archiving of data blocks according to one or more aspects of the present disclosure.

[0013] Figure 2 is a block diagram illustrating an example system 200 in accordance with techniques of this disclosure.

[0014] FIG. 3A to FIG. 3C is a diagram illustrating an example mode of operation of a data platform performing adaptive direct archiving of block files created from one or more streams.

[0015] Figure 4 is a flow chart illustrating example operation of a data platform in performing adaptive direct archiving of block files in accordance with various aspects of the techniques described in this disclosure.

[0016] Like reference numerals refer to like elements throughout the text and drawings. DETAILED DESCRIPTION

[0017] Figure 1A to Figure 1Bis a block diagram illustrating an example system for performing adaptive direct archiving of data blocks according to one or more examples of the present disclosure. Figure 1A In the example of , system 100 includes application system 102. Application system 102 represents a collection of hardware devices, software components, and / or data stores that can be used to implement one or more applications or services provided to one or more mobile devices 108 and one or more client devices 109 via network 113. Application system 102 may include one or more physical or virtual computing devices that execute workloads 174 for the applications or services. Workload 174 may include one or more virtual machines, containers, Kubernetes pods each including one or more containers, bare metal processes, and / or other types of workloads.

[0018] exist Figure 1A In the example of , the application system 102 includes application servers 170A-170M (collectively referred to as "application servers 170") connected to a database server 172 that implements a database via a network. Other instances of the application system 102 may include one or more load balancers, web servers, network devices (such as switches or gateways), or other devices for implementing one or more applications or services and delivering one or more applications or services to the mobile device 108 and the client device 109. The application system 102 may include one or more file servers. The one or more file servers may implement the primary file system of the application system 102. (In such an example, the file system 153 may be a secondary file system that provides backup, archiving, and / or other services for the primary file system. References to the file system herein may include a primary file system or a secondary file system, for example, the primary file system of the application system 102 or a file system 153 that operates as a primary file system or a secondary file system).

[0019] The application system 102 may be deployed internally and / or located in one or more data centers, each of which is part of a public cloud, a private cloud, or a hybrid cloud. The application or service may be a distributed application. The application or service may support enterprise software, financial software, office or other productivity software, data analysis software, end-user relationship management, network services, educational software, database software, multimedia software, information technology, healthcare software, or other types of applications or services. The application or service may be provided in the form of software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), data storage as a service (dSaaS), or other types of services.

[0020] In some examples, the application system 102 may represent an enterprise system that includes one or more workstations in the form of desktop computers, laptop computers, mobile devices, enterprise servers, network devices, and other hardware to support enterprise applications. Enterprise applications may include enterprise software, financial software, office or other productivity software, data analysis software, end-user relationship management, web services, educational software, database software, multimedia software, information technology, healthcare software, or other types of applications. Enterprise applications may be delivered as a service from an external cloud service provider or other provider, executed natively on the application system 102, or both.

[0021] exist Figure 1A In the example of , the system 100 includes a data platform 150, which uses the storage system 105 and the separate storage system 115 to provide a file system 153 and an archiving function to the application system 102. The data platform 150 implements a distributed file system 153 and a storage architecture to facilitate the application system 102 to access the file system data and to facilitate the transmission of data between the storage system 105 and the application system 102 through the network 111. In the case of a distributed file system, the data platform 150 enables the device of the application system 102 to access the file system data through the network 111 using a communication protocol, as if such file system data is stored locally (for example, stored on the hard disk of the device of the application system 102). Example communication protocols for accessing files and objects include server message blocks (SMB), network file systems (NFS), or AMAZON simple storage services (S3). The file system 153 can be the primary file system or a secondary file system of the application system 102.

[0022] The file system manager 152 represents a collection of hardware devices and software components that implement the file system 153 for the data platform 150. Examples of file system functions provided by the file system manager 152 include storage space management, including deduplication, file naming, directory management, metadata management, partitioning, and access control. The file system manager 152 executes communication protocols to facilitate the application system 102 to access files and objects stored in the storage system 105 through the network 111.

[0023] The data platform 150 includes a storage system 105 having one or more storage devices 180A-180N (collectively referred to as "storage devices 180"). Storage devices 180 may represent one or more physical or virtual computing and / or storage devices that include storage media or otherwise have access to storage media. Such storage media may include one or more of a flash drive, a solid state drive (SSD), a hard disk drive (HDD), an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory form and / or other types of storage media used to support the data platform 150. Different storage devices in the storage devices 180 may have different types of storage media combinations. Each of the storage devices 180 may include system memory. Each of the storage devices 180 may be a storage server, a network attached storage (NAS) device, or may represent disk storage of a computer device. The storage system 105 may be a redundant array of independent disks (RAID) system. In some examples, one or more of the storage devices 180 is both a computing device and a storage device that executes software for the data platform 150, such as the file system manager 152 and the archive manager 154 in the example of the system 100, and stores objects and metadata for the data platform 150 to a storage medium. In some examples, a separate computing device (not shown) executes software for the data platform 150, such as the file system manager 152 and the archive manager 154 in the example of the system 100. Each of the storage devices 180 may be considered and referred to as a "storage node" or simply a "node". The storage 180 may represent a virtual machine running on a supported hypervisor, a cloud virtual machine, a physical rack server, or a computing model installed in a converged platform.

[0024] In various examples, the data platform 150 runs natively on a physical system, virtually, or in the cloud. For example, the data platform 150 can be deployed as a physical cluster, a virtual cluster, or a cloud-based cluster that runs in a private cloud, a hybrid private / public cloud, or a public cloud deployed by a cloud service provider. In some examples of the system 100, multiple instances of the data platform 150 can be deployed, and the file system 153 can be replicated between the various instances. In some cases, the data platform 150 can be a computing cluster representing a single management domain. The number of storage devices 180 can be expanded to meet performance requirements.

[0025] The data platform 150 may implement and provide multiple storage domains to one or more tenants, or to isolate workloads 174 that require different data policies. A storage domain may be a data policy domain that determines policies for deduplication, compression, encryption, tiering, and other operations performed on objects stored using the storage domain. In this way, the data platform 150 may provide users with the flexibility to choose global data policies or workload-specific data policies. The data platform 150 may support partitioning.

[0026] A view may be a protocol export that resides within a storage domain. A view may inherit the data policy of its storage domain, but additional data policies may be specified for the view. Views may be exported via SMB, NFS, S3, and / or another communication protocol. Policies that determine data processing and storage by the data platform 150 may be assigned at the view level. A protection policy may specify backup frequency and retention policy, which may include a data lock period. Archives 142 or snapshots created in accordance with a protection policy inherit the data lock period and retention period specified by the protection policy.

[0027] Each of network 113 and network 111 may be the Internet, or may include or represent any public or private communication network or other network. For example, network 113 may be cellular, Wi-Fi®, ZigBee®, Bluetooth®, near field communication (NFC), satellite, enterprise, service provider and / or other types of networks that support data transfer between computing systems, servers, computing devices and / or storage devices. One or more of such devices may use any suitable communication technology to transmit and receive data, commands, control signals and / or other information across network 113 or network 111. Each of network 113 or network 111 may include one or more network hubs, network switches, network routers, satellite antennas or any other network equipment. Such network devices or components may be operably coupled to each other, thereby providing information exchange between computers, devices or other components (e.g., between one or more client devices or systems and one or more computers / servers / storage devices or systems). Figure 1A to Figure 1B Each of the devices or systems shown in the figure can be operatively coupled to the network 113 and / or the network 111 using one or more network links. The links coupling such devices or systems to the network 113 and / or the network 111 can be Ethernet, asynchronous transfer mode (ATM), or other types of network connections, and such connections can be wireless and / or wired connections. Figure 1A to Figure 1B One or more of the devices or systems shown or otherwise on network 113 and / or network 111 may be remotely located relative to one or more of the other shown devices or systems.

[0028] The application system 102 may generate objects and other data using the file system 153 provided by the data platform 150, and the file system manager 152 may create, manage and store the objects and data in the storage system 105. For this purpose, the application system 102 may be alternatively referred to as a "source system", and the file system 153 used for the application system 102 may be alternatively referred to as a "source file system". The application system 102 may communicate directly with the storage system 105 through the network 111 to transfer objects for some purposes, and may communicate with the file system manager 152 through the network 111 to indirectly obtain objects or metadata from the storage system 105 for some purposes. The file system manager 152 may generate metadata and store the metadata in the storage system 105. The collection of data stored in the storage system 105 and used to implement the file system 153 is referred to as file system data in this article. The file system data may include the aforementioned metadata and objects. The metadata may include file system objects, tables, trees or other data structures; metadata generated to support deduplication; or metadata used to support snapshots. The stored objects may include any of files, virtual machines, databases, applications, pods, containers, workloads 174, system images, directory information, or other types of objects used by application system 102. Objects of different types and objects of the same type may be deduplicated relative to each other.

[0029] Data platform 150 includes archive manager 154 that provides archiving for file system data of file system 153. In the example of system 100, archive manager 154 archives file system data stored by storage system 105 to storage system 115 via network 111.

[0030] The storage system 115 includes one or more storage devices 140A-140X (collectively referred to as "storage devices 140"). Storage device 140 may represent one or more physical or virtual computing and / or storage devices that include storage media or otherwise have access to storage media. Such storage media may include one or more of a flash drive, a solid state drive (SSD), a hard disk drive (HDD), an optical disk, an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory form and / or other types of storage media. Different storage devices of storage device 140 may have different types of storage media combinations. Each of storage devices 140 may include system memory. Each of storage devices 140 may be a storage server, a network attached storage (NAS) device, or may represent disk storage of a computer device. Storage system 115 may include a redundant array of independent disks (RAID) system. Storage system 115 may be capable of storing much larger amounts of data than storage system 105. Storage device 140 may also be configured for long-term storage of information, more suitable for archival purposes.

[0031] In some examples, storage systems 105 and / or 115 may be storage systems deployed and managed by a cloud storage provider and are referred to as “cloud storage systems”. Example cloud storage providers include, for example, AMAZON WEBSERVICES (AWS™) of AMAZON, AZURE® of MICROSOFT, DROPBOX™ of DROPBOX, ORACLE CLOUD™ of ORACLE, and GOOGLE CLOUD PLATFORM (GCP) of GOOGLE. In some examples, storage system 115 is co-located with storage system 105 in a data center, locally, or in a private, public, or hybrid private / public cloud. Storage system 115 may be considered a “backup” or “secondary” storage system for primary storage system 105. Storage system 115 may be referred to as an “external target” of archive 142. Storage system 115 may be referred to as “cloud storage” when deployed and managed by a cloud storage provider. The storage system 115 may include one or more interfaces for managing the transfer of data between the storage system 105 and the storage system 115 and / or between the application system 102 and the storage system 115. The data platform 150 supporting the application system 102 relies on the primary storage system 105 to support latency-sensitive applications. However, because the storage system 105 is generally more difficult or more expensive to scale, the data platform 150 may use the secondary storage system 115 to support secondary use cases, such as backup and archive. Generally speaking, a file system backup is a copy of the file system 153 used to support the protection of the file system 153 for rapid recovery (usually due to the loss of some data in the file system 153), while a file system archive ("archive") is a copy of the file system 153 used to support long-term retention and review. A "copy" of the file system 153 may include such data as is needed to restore or view the state of the file system 153 at the time of the backup or archive.

[0032] Archive manager 154 can archive the file system data of file system 153 at any time according to an archive strategy, which specifies, for example, archive periodicity and timing (daily, weekly, etc.), which file system data will be archived, archive retention period, storage location, access control, etc. The initial archive of file system data corresponds to the state of the file system data at the initial archive time (archive creation time of the initial archive). Depending on the archive strategy, the initial archive may include a complete archive of the file system data, or may include a less complete archive of the file system data. For example, the initial archive may include all objects of the file system 153 or one or more selected objects of the file system 153.

[0033] One or more subsequent incremental archives of file system 153 may correspond to corresponding states of file system 153 at corresponding subsequent archive creation times (i.e., after the archive creation time corresponding to the initial archive). Subsequent archives may include incremental archives of file system 153. Subsequent archives may correspond to incremental archives of one or more objects of file system 153. Some file system data of file system 153 stored on storage system 105 at the initial archive creation time may also be stored on storage system 105 at subsequent archive creation times. Subsequent incremental archives may include data that was not previously archived to storage system 115. Archive manager 154 may deduplicate file system data included in subsequent archives against file system data included in one or more previous archives (including the initial archive) to reduce the amount of storage used. (References to "time" in this disclosure may refer to dates and / or times. Times may be associated with dates. For example, multiple archives may occur at different times on the same day).

[0034] In system 100, archive manager 154 archives file system data to storage system 115 as archives 142 using block files 162. Archive manager 154 may use any of archives 142 to later restore the file system (or portion thereof) to its state at the time of archive creation, or, for example, may use the archive to create or present a new file system (or "view") based on the archive. As described above, archive manager 154 may deduplicate file system data included in subsequent archives against file system data included in one or more previous archives. For example, a second object of file system 153 and included in a second archive may be deduplicated against a first object of file system 153 and included in an earlier first archive. Archive manager 154 may remove data blocks ("blocks") of the second object and generate metadata having references (e.g., pointers) to the stored blocks in blocks 164 in one of block files 162. The stored blocks in this example are instances of blocks stored for the first object.

[0035] Archive manager 154 may apply deduplication as part of the write process of writing (i.e., storing) objects of file system 153 to an archive 142 in storage system 115. Deduplication may be implemented in a variety of ways. For example, the method may be fixed length or variable length, the block size used for the file system may be fixed or variable, and the deduplication domain may be applied globally or per workload. Fixed length deduplication involves delimiting a data stream at fixed intervals. Variable length deduplication involves delimiting a data stream at variable intervals to improve the ability to match data regardless of the file system block size method being used. This algorithm is more complex than the fixed length deduplication algorithm, but is more efficient for most situations and generally generates less metadata. Variable length deduplication may include variable length, sliding window deduplication. The length of any deduplication operation (whether fixed length or variable length) determines the size of the blocks being deduplicated.

[0036] In some examples, for variable length deduplication, the block size may be within a fixed range. For example, archive manager 154 may calculate blocks with block sizes in the range of 16 kB to 48 kB. Archive manager 154 may avoid deduplication of objects smaller than 16 kB. In some example implementations, when considering deduplication of data for an object, archive manager 154 may compare a block identifier (ID) of the data (e.g., a hash value of the entire block) with an existing block ID of a stored block. If a match is found, archive manager 154 may update metadata for the object to point to the matching, stored block. If no matching block is found, archive manager 154 may write the data of the object to memory as one of blocks 164 of one of block files 162. In addition, archive manager 154 may store the block ID in association with the newly stored block in the block metadata to allow future deduplication against the newly stored block. In general, for any of the archives 142, the chunk metadata may be used to generate, view, retrieve, or restore objects stored as chunks 164 (and references thereto) within the chunk files 162, and this is described in more detail below.

[0037] Each of the block files 162 includes a plurality of blocks 164. The block files 162 may be of a fixed size (e.g., 8 MB) or of a variable size. The block files 162 may be stored using a data structure provided by a cloud storage provider for the storage system 115. For example, each of the block files 162 may be one of the following: an S3 object within an AWS cloud storage bucket, an object within an AZURE Blob storage device, an object in an object storage device for ORACLE CLOUD, or other similar data structures used within another cloud storage provider storage system. Any of the block files 162 may be subject to a write-once, read-many (WORM) lock expiration time. The WORM lock for an S3 object is referred to as an "object lock," and the WORM lock for an object within an AZURE Blob storage is referred to as "blob immutability."

[0038] The process of deduplicating multiple objects across multiple archives results in block files 162, each of which has multiple blocks 164 for multiple different objects associated with the multiple archives. In some examples, different archives 142 may have objects that are actually copies of the same data, for example, objects that have not been modified with respect to the file system. Archived objects may be represented or "stored" as metadata with references to the blocks, which enables access to the archived objects. Therefore, descriptions herein of an archive "storing," "having," or "including" an object include instances in which the archive does not store the object's data in its native form.

[0039] The initial archive and one or more subsequent incremental archives of archive 142 may each be associated with a corresponding retention period and, in some cases, with a data lock period for the archive. As described above, a data management policy (not shown) may specify a retention period for an archive and a data lock period for the archive. The retention period for an archive may be the amount of time that an archive and blocks referenced by archived objects are to be stored before the archive and blocks are eligible to be removed from a storage device. The retention period for an archive may begin when the archive is stored (archive creation time). Block files containing blocks referenced by archived objects and subject to the archive's retention period but not to the archive's data lock period may be modified at any time before the expiration of the retention period. The nature of such modifications must be such as to be able to preserve the data referenced by the archived objects.

[0040] A user or application associated with the application system 102 may have access to (e.g., read or write) the archived data stored in the storage system 115. The user or application may delete some data due to malicious attacks (e.g., viruses, ransomware, etc.), malicious administrators, and / or human errors. The user's credentials may be leaked, and therefore, the archived data stored in the storage system 115 may be attacked by ransomware. In order to reduce the possibility of accidental or malicious data deletion or corruption, a data lock with a data lock period may be applied to the archive.

[0041] As described above, the block file 162 may represent an object in an archival storage system (illustrated as “storage system 115”, and may also be referred to as “archival storage system 115”) that conforms to the underlying architecture of the archival storage system 115. The data platform 150 includes an archive manager 154 that supports archiving data in the form of block files 162, which interface with the archival storage system 115 to store the block files 162 after forming the block files 162 from one or more blocks 164 of data. The archive manager 154 may apply a process known as “de-duplication” to the blocks 164 to remove redundant blocks and generate metadata that links the redundant blocks to previously archived blocks 164, thereby reducing consumed storage (thereby reducing storage costs in terms of storage required to store the blocks). The archive manager 154 may aggregate the blocks 164 using the metadata to form the block files 162 at the archival storage system 115.

[0042] In certain contexts, such as streaming, the data platform 150 may receive a stream of data from the application system 102 for which a fixed size (or in other words, a fixed amount) of data is unknown prior to archiving the data as a block 164. The uncertainty in the size of the data contrasts with receiving all of the data in advance via a write request, in which case the archive manager 154 may know the extent of the resulting block 164 and thereby potentially optimize the size of the block file 162 stored to the archival storage system 115. The uncertainty in the size of the data to be archived with respect to streaming of data may cause the archive manager 154 to operate inefficiently with respect to archiving of the blocks 164.

[0043] The data may be in a format corresponding to the format of the source from which the data originated, or may be in a format in which the data is generated and is generally referred to as native format data. Although the following discussion provides examples with respect to native format data, it should be understood that the described techniques relate to data in any format.

[0044] The archive manager 154 may operate inefficiently in such a context because, as one example, the archive manager 154 may fail to optimize the size of the chunk files 162 and instead rely on storing the chunks 164 to a storage system 105 local to the data platform 115 (which may also be referred to as a “local storage system 105”), which consumes additional memory, memory bandwidth, processing cycles, and / or other computing resources, and also results in latency when archiving the chunks 164, but provides a more efficiently sized chunk file 162. As another example, the archive manager 154 may avoid consuming computing resources to store the chunks 164 to the local storage system 105 and reduce latency when archiving the chunks, but may do so at the expense of an inefficiently sized chunk file 162 that results in increased consumption of storage resources at the archive storage system 115 and an attendant increase in storage costs compared to a more efficiently sized chunk file 162.

[0045] These inefficiencies may result, for example, in the context of streaming when the application system 102 does not allow the data platform agent to execute or otherwise operate within the application system 102. In the event that the application system 102 does operate with a data platform agent, the data platform 150 may communicate with the data platform agent, which may segment the native format data into chunks local to the application system 102 in which the approximate size of the native format data may be more readily obtained (given that the data platform agent may have access to the primary file system of the application system 102, it may be easier to obtain the approximate size of the underlying native format data, such as the file size of a file). However, given that the data platform agent may not be aware of other chunks stored to or by the data platform 150 and / or the archival storage system 115, the data platform agent may not perform deduplication with respect to the chunks. Furthermore, the data platform agent may issue a write request defining the size of the native format data for chunking and deduplication even when the native format data is not segmented into chunks, rather than opening a stream with an undefined native format data size, wherein such size metadata may allow the data platform to operate more efficiently in chunking and deduplication of the native format data. Thus, the data platform agent may have more access to native format data (which may refer to data in a format that is native to or otherwise supported by the application system 102) and potentially optimize the size of chunks, resulting in a more optimized chunk file 162.

[0046] In situations where the application system 102 does not permit execution of a data platform agent, the archive manager 154 may receive one or more native format data streams without making much, if any, access to the primary file system of the application system 102. This lack of access may lead to the inefficiencies discussed above and the resulting issues with latency, consumption of computing resources, and power consumption.

[0047] According to techniques of this disclosure, archive manager 154 may include an archive director module 184 (illustrated as and may be referred to as “archive director 184”) that, rather than statically determining to initially store and accumulate chunks 164 to local storage system 105 or to continue to archive chunks 164 directly to archive storage system 115, may instead predict incoming data rates to obtain predicted incoming data rates (PIDRs) 185. Archive director 184 may predict incoming data rates for a given source (originating native format data, such as application system 102), for a given native format data stream, across different streams from the same source and / or tenants originating native format data, and the like. Using PIDR 185, archive director 184 may apply one or more threshold data rates (TDR) 187 to dynamically and / or adaptively determine whether to initially store and accumulate blocks 164 (obtained from one or more streams) to local storage system 105 or to archive blocks 164 directly to archive storage system 115 over a certain period of time (however, archive director 184 may temporarily store blocks 164 in local repository 107—e.g., random access memory—RAM).

[0048] In operation, the archive director 184 may predict the incoming data rate of data in a native format (e.g., received directly from the application system 102 - meaning there is no intermediate data platform agent to forward the block 164 itself) to obtain the PIDR 185. The archive director 184 may utilize machine learning (ML) and / or artificial intelligence (AI, which is a subset of machine learning) to perform predictions on the incoming data rate to obtain the PIDR 185. The archive director 184 may apply the ML and / or AI (ML / AI) model to a stream, i.e., an aggregate of streams from one or more sources of a given tenant (but typically not across tenants due to privacy considerations), etc. based on past incoming data rates of the stream, i.e., an aggregate of two or more streams from one or more sources of a given tenant, etc. to obtain the PIDR 185.

[0049] Examples of ML / AI models may include neural networks, deep neural networks, dense neural networks, etc. Although described with respect to ML / AI models, the techniques described in the present disclosure are also applicable to other types of ML / AI models, including rule-based models, finite state machines, etc. In addition, while PIDR 185 is discussed as being obtained by applying an ML / AI model, other forms of prediction may occur to approximate PIDR 185, for example using time series analysis (TSA) and / or time series forecasting (TSF).

[0050] Regardless, machine learning generally enables archive director 184 to analyze input data and identify actions to be performed in response to the input data. Each ML / AI model can be trained using training data that reflects possible input data (e.g., a time series of incoming data rates). The training data can be labeled or unlabeled (meaning that the correct action to take based on a sample of the training data is explicitly stated or not, respectively).

[0051] The training of the ML / AI model can be supervised (i.e., a designer (such as a computer programmer) can direct the training to guide the ML / AI model to identify the correct action based on the input data) or unsupervised (i.e., the designer does not guide the ML / AI model to identify the correct action based on the input data). In some cases, the ML / AI model is trained by a combination of labeled and unlabeled training data, a combination of supervised and unsupervised training, or a possible combination thereof. Examples of machine learning include nearest neighbors, naive Bayes, decision trees, linear regression, support vector machines, neural networks, k-means clustering, Q-learning, temporal differences, deep adversarial networks, evolutionary algorithms or other supervised learning algorithms, unsupervised learning algorithms, semi-supervised learning algorithms, or reinforcement learning algorithms to train one or more models.

[0052] The archive director 184 may segment the native format data into chunks 164 in response to determining that the PIDR 185 exceeds (or may be equal to) a first TDR 187. This first TDR 187 may define a threshold data rate above which direct archiving of the chunks 164 to the archive storage system 115 may be performed preferentially. The threshold data rate may be expressed as a bandwidth, such as one megabyte per second (1 MB / s), 2 MB / s, 0.5 MB / s. That is, an efficient (or in other words, a target) archive chunk file size may be 2MB, and assuming that data will accumulate in the local repository 107 for up to 100 milliseconds (ms), the TDR 187 may be 20 MB / s. In some cases, this threshold data rate may be adaptive, in which case the archive director 184 may adapt the TDR 187 based on the current load (e.g., the amount of native format data being received, the computing resources consumed by the archive director 184 or more generally by the data platform 150, etc.). Furthermore, the threshold data rate may be different for different formats of native format data, where, for example, there may be a first threshold data rate for images, a second threshold data rate for documents, etc. Furthermore, an AI / ML model (similar to that described above) may be applied to determine the TDR 187 .

[0053] The archive director 184 may write one or more chunks 164 directly to the archive storage system 105 as a chunk file 162, again in response to determining that the PIDR 185 exceeds the first TDR 187. Again, writing chunks 164 directly may include temporarily storing chunks 164 in a local repository 107 (e.g., a cache memory, RAM, or other very high-speed (meaning low-latency) memory (perhaps flash memory) separate from the local storage system 105) for the purpose of processing the native format data into chunks 164. In this sense, writing chunks 164 directly may occur without accumulating chunks 164 (over time) in the local storage system 105 to form an optimally sized chunk file 162, but may occur solely for the purpose of segmenting the incoming native format data into chunks 164. The local repository 107 may represent a memory or other low-latency storage device used by the processes executed by the data platform 150 that temporarily stores data and is separate from the local storage system 105 used to accumulate blocks 164 to form block files of optimal size for low data rate streams and / or store at least a portion of a local copy of the file system. In other words, the local repository 107 may represent a low-latency memory (e.g., RAM and / or flash memory) that provides lower latency than the storage system 105, although the total storage capacity is limited compared to the storage system 105.

[0054] The archive director 184 may segment the native format data into chunks 164 and write the chunks 164 to the local storage system 105 of the data platform 150 in response to determining that the PIDR 185 is lower than (or possibly equal to) the first TDR 187. The PIDR 185 being lower than (or equal to) the first TDR 187 may indicate that the expected incoming data rate (which is another way of referring to the PIDR 185) is insufficient to formulate an optimally sized chunk file 162 during direct archiving of the chunks 164, and the archive director 184 may effectively locally buffer the chunks 164 of the native format data to the solid-state drive and / or hard disk drive of the local storage system 105. In some examples, the archive director 184 may select between a solid state drive (SSD) and / or a hard disk drive (HDD) of the local storage system 105 to buffer the chunks 164 based on a percentage below the first TDR 187, where a lower percentage (e.g., within 10-20% of TDR 187) causes the archive director 184 to store to the SSD, while a higher percentage (e.g., 20% or more of TDR 187) causes the archive director 184 to store to the HDD. Regardless, in response to determining that the chunks 164 written to the local storage system 105 exceed a threshold data amount (TDA) 189 ("TDA 189") for forming a chunk file 162 (which may be approximately 8 MB or 16 MB), the archive director 184 may write the chunks 164 to the archive storage system 115 as chunk files 162. In this manner, the TDA 189 defines a preferred size for the chunk files 162 for the data platform 150.

[0055] In this way, the techniques may provide one or more technical advantages that enable practical applications. For example, the techniques may reduce computing resource consumption at the local storage system 105 and the archive storage system 115 by adaptively determining where the native format data stream should be stored before archiving the resulting chunk 164 for a given duration. The archive director 184 may implement the techniques to archive the chunk 164 more efficiently when the PIDR 185 indicates that a large amount of native format data is to be received in an amount above a threshold data rate (for a given duration, or in other words, an amount of time), thereby archiving the chunk 164 more quickly into a potentially optimally sized chunk file 162. The archive director 184 may store the chunk 164 to the local storage system 105 when the PIDR 185 indicates that the native format data is to be received in an amount below a threshold data rate (for a given duration). Thus, the data platform 150 can reduce consumption of memory, memory bandwidth, and / or other computing resources at the local storage system 105, reduce disk writes and reads at the local storage system 105, reduce metadata and garbage collection loads, and reduce latency when archiving chunks 164, while also potentially storing optimally sized chunk files 162 that reduce consumption of computing resources at the archival storage system 115, such as by reducing metadata bloat caused by a large number of smaller sized chunk files 162.

[0056] In some cases, the PIDR 185 may not reflect the actual incoming data rate because the streaming of native format data may change suddenly due to a variety of different reasons, such as network bandwidth constraints, network latency, network load, changes in the local load of the data platform 150 (in terms of computing resources, number of streams received), unexpected errors, etc. Although the archive director 184 may still attempt to directly archive the chunk 164 when the PIDR 185 exceeds the TDR 187 in this case, the archive director 184 may create a suboptimal sized chunk file 162, but still reduce the delay in archiving the chunk 164 as a chunk file 162 or fall back to storing the data in the local repository 107. This delay is reduced because the archive director 184 may only delay the direct archiving of the chunk 164 for a limited amount of time (e.g., 100 milliseconds-ms), which may be significantly faster than when the archive director 184 initially stores and accumulates the chunk 164 to the local storage system 105 (which may occur within seconds or minutes, or even longer). Likewise, when PIDR 185 is below TDR 187 but when the actual incoming data rate exceeds TDR 187, archive director 184 may still buffer chunks 164 to local storage system 105 in an anticipatory manner, but more quickly formulate optimally sized chunk files 162. In other words, archive director 185 may realize the aforementioned benefits as a percentage of how accurate PIDR 185 is to the actual incoming data rate.

[0057] Figure 1B The system 190 is Figure 1A A variation of system 100 in which data platform 150 stores archive 142 using block files 162 stored to an archive storage system 115 that resides locally, or in other words, local to data platform 150. In some examples of system 190, storage system 115 may enable a user or application to create, modify, or delete block files 162 through file system manager 152. In system 190, Figure 1B Storage system 105 is the local storage system that archive manager 154 initially uses to store and accumulate chunks before they are archived to storage system 115 .

[0058] Figure 2 is a block diagram illustrating an example system 200 in accordance with techniques of this disclosure. Figure 2 The system 200 can be described as Figure 1A System 100 or Figure 1B An example or alternative implementation of a system 190 in which block files 162 are written to a local archive storage system 115. Figure 1A and Figure 1B Description in context Figure 2 one or more aspects of .

[0059] exist Figure 2 In the example of , system 200 includes network 111, data platform 150 implemented by computing system 202, and storage system 115. Figure 2 In the example, the network 111, the data platform 150 and the storage system 115 may correspond to Figure 1A 1 , a data platform 150, and a storage system 115. Although only one archival storage system 115 is depicted, the data platform 150 may apply techniques according to the present disclosure using multiple instances of the archival storage system 115. Different instances of the storage system 115 may be deployed by different cloud storage providers, the same cloud storage provider, an enterprise, or other entities.

[0060] The computing system 202 may be implemented as any suitable computing system, such as one or more server computers, workstations, mainframes, electrical appliances, cloud computing systems, and / or other computing systems that may be capable of performing operations and / or functions described according to one or more aspects of the present disclosure. In some examples, the computing system 202 represents a cloud computing system, a server farm, and / or a server cluster (or a portion thereof) that provides services to other devices or systems. In other examples, the computing system 202 may represent or be implemented by one or more virtualized computing instances (e.g., virtual machines, containers) of a cloud computing system, a server farm, a data center, and / or a server cluster.

[0061] exist Figure 2 In the example of , computing system 202 may include one or more communication units 215, one or more input devices 217, one or more output devices 218, and one or more storage devices of local storage system 105. Local storage system 105 includes interface module 226, file system manager 152, archive manager 154, policy 158, archive metadata 222, and block metadata 220. One or more of the devices, modules, storage areas, or other components of computing system 202 may be interconnected to enable inter-component communication (physically, communicatively, and / or operationally). In some examples, such connection may be provided by a communication channel (e.g., communication channel 212), which may represent one or more of a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

[0062] Computing system 202 includes processing components. Figure 2 In the example of FIG. 1 , the processing component includes one or more processors 213, wherein the one or more processors are configured to implement the computing system 202 associated with or with Figure 2 2 and described below and / or execute instructions associated with the above. One or more processors 213 can be a processing circuit, can be a part of a processing circuit, and / or can include a processing circuit system that performs operations according to one or more aspects of the present disclosure. Examples of processors 213 include microprocessors, application processors, display controllers, auxiliary processors, one or more sensor hubs, and any other hardware configured to function as a processor, processing unit, or processing device. Computing system 202 can use one or more processors 213 to perform operations according to one or more aspects of the present disclosure using software, hardware, firmware, or a mixture of hardware, software, and firmware resident in and / or executed at computing system 202.

[0063] One or more communication units 215 of computing system 202 can communicate with devices outside computing system 202 by transmitting and / or receiving data, and in some aspects, can operate as both input devices and output devices. In some examples, communication unit 215 can communicate with other devices through a network. In other examples, communication unit 215 can send and / or receive radio signals on a radio network (such as a cellular radio network). In other examples, communication unit 215 of computing system 202 can transmit and / or receive satellite signals on a satellite network. Examples of communication unit 215 include network interface cards (e.g., Ethernet cards), optical transceivers, radio frequency transceivers, GPS receivers, or any other type of device that can send and / or receive information. Other examples of communication unit 215 may include devices that can communicate through Bluetooth®, GPS, NFC, ZigBee®, and cellular networks (e.g., 3G, 4G, 5G) and Wi-Fi® radios in mobile devices and universal serial bus (USB) controllers, etc. Such communications may adhere to, implement or comply with appropriate protocols, including Transmission Control Protocol / Internet Protocol (TCP / IP), Ethernet, Bluetooth®, NFC or other technologies or protocols.

[0064] The one or more input devices 217 may represent any input device of the computing system 202 that is not otherwise separately described herein. The input device 217 may generate, receive, and / or process input. For example, the one or more input devices 217 may generate or receive input from a network, a user input device, or any other type of device for detecting input from a person or a machine.

[0065] One or more output devices 218 may represent any output device of computing system 202 that is not otherwise described separately herein. Output device 218 may generate, present, and / or process output. For example, one or more output devices 218 may generate, present, and / or process output in any form. Output device 218 may include one or more USB interfaces, video and / or audio output interfaces, or any other type of device capable of generating tactile, audio, visual, video, electrical, or other output. Some devices may be used as both input devices and output devices. For example, a communication device may send data to other systems or devices and receive data from other systems or devices over a network.

[0066] One or more storage devices of the local storage system 105 within the computing system 202 may store information for processing during operation of the computing system 202, such as random access memory (RAM), flash memory, solid state drive (SSD), hard disk drive (HDD), etc. The storage device may store program instructions and / or data associated with one or more modules described according to one or more aspects of the present disclosure. One or more processors 213 and one or more storage devices may provide an operating environment or platform for such modules, which may be implemented as software, but may include any combination of hardware, firmware, and software in some examples. One or more processors 213 may execute instructions, and one or more storage devices of the storage system 105 may store instructions and / or data of one or more modules. The combination of the processor 213 and the local storage system 105 may retrieve, store, and / or execute instructions and / or data of one or more applications, modules, or software. The processor 213 and / or the storage device of the local storage system 105 may also be operably coupled to one or more other software and / or hardware components, including but not limited to the computing system 202 and / or one or more components of one or more devices or systems shown as being connected to the computing system 202.

[0067] The file system manager 152 may perform functions related to providing the file system 153, as described above with respect to Figure 1A The file system manager 152 may generate and manage file system metadata 232 for constructing file system data 230 of the file system 153, and store the file system metadata 232 and the file system data 230 to the local storage system 105. The file system metadata 232 may include one or more trees that describe objects within the file system 153 and the file system 153 hierarchy and may be used to write or retrieve objects within the file system 153. The file system metadata 232 may reference either the block metadata 220 or the archive metadata 222, and vice versa. The file system manager 152 may interact and / or operate in coordination with one or more modules of the computing system 202, including the interface module 226 and the backup manager 154.

[0068] Archive manager 154 may perform functions associated with archive file system 153, as described above with respect to Figure 1AThe archive manager 154 may generate and manage the archive metadata 222 for generating, viewing, retrieving, or restoring any of the archives 142. The archive metadata 222 may include the corresponding original data lock period of the archive 142. The archive manager 154 may generate and manage the block metadata 220 for objects stored as blocks 164 (and references thereto) within the block files 162 for any of the archives 142. The stored objects may be represented and manipulated using logical files to identify the blocks of the objects. The block metadata 220 may include a block table describing the blocks 164. The block table may include a corresponding block ID for block 164 and may contain a pointer to block file 162 and an offset within block file 162 for retrieving block 164 from storage system 115. Block 164 is written to block file 162 at a different offset. By comparing the new block ID to the block table, archive manager 154 may determine whether the data already exists on the system. If the block already exists, the data may be discarded and the metadata of the object may be updated to reference the existing block. Archive manager 154 may use the block table to look up the block file identifier of the block file containing the block.

[0069] The block metadata 220 may include a block file table that describes the corresponding physical or virtual location of the block file 162 on the storage system 115, as well as other metadata about the block file, such as checksums, encryption data, compression data, etc. Figure 2 142 to a file system implemented by data platform 150, which can be presented by file system manager 152 to other systems.

[0070] The interface module 226 may implement an interface through which other systems or devices may determine the operation of the file system manager 152 or the archive manager 154. Another system or device may communicate through the interface of the interface module 226 to specify one or more policies 158.

[0071] System 200 can be modified to implement Figure 1B In the modified system 200, the block files 162 are stored to the local archive storage system 115 to support the archive 142.

[0072] The interface module 240 of the archive storage system 115 may execute an interface through which other systems or devices may create, modify, delete, or extend a WORM lock expiration time for any of the block files 162. The interface module 240 may execute and present an API. The interface presented by the interface module 240 may be a gRPC, HTTP, RESTful, command line, graphical user, Web, or other interface. The interface module 240 may be associated with a usage cost. One or more methods or functions of the interface module 240 may impose a single usage cost (e.g., $0.10 to extend the WORM lock expiration time of the block file 162).

[0073] FIG. 3A to FIG. 3C is a diagram illustrating an example mode of operation of a data platform performing adaptive direct archiving of block files created from one or more streams. Figure 3A , the archive director 184 may predict the incoming data rate of data in a native format (e.g., received directly from the application system 102 - meaning there is no intermediate data platform agent to forward the block 164 itself) to obtain the PIDR 185. As described above, the archive director 184 may utilize machine learning (ML) and / or artificial intelligence (AI, which is a subset of machine learning) to perform predictions on the incoming data rate to obtain the PIDR 185. The archive director 184 may apply the ML and / or AI (ML / AI) model to the streams, i.e., the aggregation of streams from one or more sources of a given tenant (but typically not across tenants due to privacy considerations), etc. based on the past incoming data rates of the streams, i.e., the aggregation of two or more streams from a given source, etc. to obtain the PIDR 185.

[0074] Archive director 184 may receive a flow 300 and perform chunking and deduplication (302) on flow 300, where flow 300 may also be referred to as data flow 300. Archive director 184 may maintain a table or other data structure for each flow 300, which may store a flow identifier (ID) and a corresponding PIDR 185. Archive director 184 may identify flow 300 (e.g., by an Internet Protocol—IP—address; a packet flow five-tuple—which may include a source IP address, a destination IP address, a source port, a destination port, and a transport protocol; an application ID; or any other unique identifying information associated with flow 300) to obtain the flow ID. Archive director 184 may use the flow ID to access the table to identify an entry in the table that specifies the PIDR 185 of flow 300.

[0075] Archive director 184 may compare PIDR 185 to TDR 187 (304), where archive director 184 may select an appropriate TDR 187 based on the transport protocol, native file format, or other metadata associated with stream 300. In response to determining that PIDR 185 exceeds (or may be equal to) first TDR 187 ("yes" 304), archive director 184 may store block 164 directly to archive storage system 115 as block file 162 (306). Likewise, storing block 164 directly to archive storage system 115 may include temporarily storing block 164 in local repository 107 (e.g., cache memory, RAM, or other very high-speed (meaning low-latency) memory (e.g., possibly flash memory)) separate from local storage system 105 for the purpose of processing native format data into block 164. These chunks 164 are not accumulated, but rather formed and stored directly to the archive storage system 115, thereby forming the chunk file 162 before a batch of optimally sized chunks 164 is determined.

[0076] The archive director 184 may write the block 164 to the local storage system 105 of the data platform 150 in response to determining that the PIDR 185 is less than (or possibly equal to) the first TDR 187 ("No" 304) (308). The PIDR 185 being less than (or equal to) the first TDR 187 may indicate that the expected incoming data rate (which is another way to refer to the PIDR 185) is insufficient to formulate an optimally sized block file 162, and the archive director 184 may effectively buffer the block 164 of native format data locally to a solid-state drive and / or hard disk drive of the local storage system 105 (the solid-state drive and / or hard disk drive are examples of a second storage tier compared to the first storage tier, which may include RAM, flash memory, or other low-latency forms of memory compared to the second storage tier). In some examples, archive director 184 may select between a solid state drive (SSD) and / or a hard disk drive (HDD) of local storage system 105 to buffer blocks 164 based on a percentage below first TDR 187, with a lower percentage (e.g., within 10-20% of TDR 187) being stored to the SSD and a higher percentage (e.g., 20% or more of TDR 187) being stored to the HDD. In any event, in response to determining that a batch of blocks written to local storage system 105 exceeds a threshold data amount (TDA) 189 for forming a block file (which may be approximately 8 MB or 16 MB), archive director 184 may write the batch of blocks to archive storage system 115 as one of block files 162, or in other words, copy the batch of blocks to one of block files 162 within archive storage system 115 (310). Writing the batch of blocks from local storage system 105 to archive storage system 115 involves reading the batch of blocks from local storage system 105.

[0077] Next reference Figure 3B For example, the archive director 184 can be used with the above Figure 3A However, instead of processing a single stream 300, archive director 184 may process multiple different streams from one or more sources of the same tenant. Archive director 184 may be similar to the operations discussed above with respect to Figure 3A The aggregate stream 320 is processed in the same manner as described in the examples (302-310).

[0078] exist Figure 3CIn the example of , archive director 184 can layer multiple different storage processes together in a more granular (or finer-grained) archive storage architecture. Archive director 184 can receive stream 300, which can also be referred to as file stream 300 because the size of the native format data provided by stream 300 is unknown (330). It should be understood that archive director 184 can apply a finer-grained archive storage scheme to multiple streams from one or more sources of the same tenant, similar to Figure 3B .

[0079] Not like Figure 3A and Figure 3B 180, the archive director 184 may first obtain the PIDR 185 and compare the PIDR 185 with the first TDR 187 (332). The first TDR 187 may be greater than the second TDR 187. In any case, the archive director 184 may perform chunking and deduplication to obtain the chunk 164 in response to determining that the PIDR 185 is greater than (or equal to) the first TDR 187 ("yes" 332), and then store the chunk 164 directly to the archive storage system 115 as the chunk file 162 (334, 336).

[0080] In response to determining that PIDR 185 is less than (or equal to) first TDR 187 (“No” 332), archive director 184 may obtain second TDR 187 and compare PIDR 185 to second TDR 187 (338). Archive director 184 may determine whether local repository 107 has sufficient main memory space (or in other words, storage capacity) to store file stream 300 (in its native format, meaning without chunking and deduplication) (340) in response to determining that PIDR 185 is greater than second TDR 187 (“Yes” 338).

[0081] In response to determining that there is sufficient main memory space in the local repository 107 ("yes" 340), the archive director 184 may store the file stream 300 to the RAM of the local storage system 105 so that the file stream 300 is buffered in its normal format in the low-latency memory (342). Given that no chunking and deduplication are performed, this path of the archive storage architecture may be referred to as hydration (which may also be referred to as hydration), and thereby maintain hydration. The archive director 184 may rehydrate the file stream range (e.g., when the file stream range meets the block size, or when the main memory space (which is another way to refer to the local repository 107) becomes limited, in which case the block size may not meet the optimal size, which may be 8 MB as an example), thereby flushing the file stream range back to the active process in the manner described above (344). Given that available main memory space can change rapidly, archive director 184 may determine that main memory space is insufficient ("NO" 340), and return file stream 300 to active processing in the manner described above in response to determining that main memory space is insufficient.

[0082] In this regard, in response to determining that the PIDR 185 is below the first TDR 187 and exceeds the second TDR 187, the archive director 184 may write the native format data to the main memory (RAM or flash memory) of the local repository 107 of the data platform 150. In response to determining that a time limit (e.g., 100 ms) for writing the native format data to the main memory of the local repository 107 has been exceeded or the amount of the native format data exceeds the TDA 189, the archive director 184 may replay the receipt of the native format data, segment the native format data into blocks 164, and write the blocks 164 to the archive storage system 115.

[0083] Regardless, the archive director may perform chunking and deduplication on the file stream 300 to obtain chunks 184 (346) in response to determining that the PIDR 185 is greater than (or equal to) the TDR 187. The archive director 184 may then store the chunks 164 to the SSD and / or HDD of the local storage system 105 in the manner described above (348). The archive director 184 may then copy the batch of chunks 164 to the archive storage system 115 as chunk files 162 (350), thereby effectively storing the chunks 164 to the archive storage system (336).

[0084] Figure 4 1 is a flow chart illustrating an example operation of a data platform according to various techniques described in this disclosure when performing adaptive direct archiving on a block file. Figure 4As shown in the example of FIG. 1 , the archive director 184 may predict the incoming data rate of the native format data to obtain a PIDR 185 (400). The archive director 184 may segment the native format data into chunks 164 (402) in response to determining that the PIDR 185 exceeds (or may equal) the first TDR 187. The archive director 184 may write one or more chunks 164 directly to the archive storage system 105 as chunk files 162, again in response to determining that the PIDR 185 exceeds the first TDR 187 (404).

[0085] Although the techniques described in this disclosure are primarily described with respect to the archive function performed by the archive manager of the data platform, similar techniques may be additionally or alternatively applied to the backup, replication, cloning, or snapshot functions performed by the data platform. In such cases, the archive 142 will be a backup, replica, clone, or snapshot, respectively.

[0086] For the processes, devices, and other examples or illustrations described herein, including in any flowchart or flow diagram, certain operations, actions, steps, or events included in any of the techniques described herein may be performed in a different order, may be added, merged, or omitted entirely (e.g., not all described actions or events are necessary for the practice of the technique). In addition, in some examples, operations, actions, steps, or events may be performed simultaneously, such as by multithreading, interrupt processing, or multiple processors, rather than sequentially. In addition, certain operations, actions, steps, or events may also be automatically performed even if not explicitly identified as being automatically performed. Moreover, certain operations, actions, steps, or events described as being automatically performed may alternatively not be automatically performed, but, in some examples, such operations, actions, steps, or events may be performed in response to an input or another event.

[0087] The detailed description set forth herein in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent only configurations in which the concepts described herein may be practiced. The specific embodiments include specific details for the purpose of providing a comprehensive understanding of the various concepts. However, it is apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, in order to avoid confusion of such concepts, well-known structures and components are shown in block diagram form.

[0088] According to one or more aspects of the present disclosure, the term "or" may be interpreted as "and / or" if the context does not otherwise dictate. In addition, while phrases such as "one or more" or "at least one" may have been used in some instances, they are not used in other instances; if the context does not dictate otherwise, these instances may be interpreted as having an implicit meaning in the absence of such language.

[0089] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on and / or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or a communication medium, including any medium that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this manner, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0090] For example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, flash memory or any other medium that can be used to store the desired instruction or data structure form of program code and can be accessed by the computer. In addition, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit instructions from a website, server or other remote source, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave) is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other transient media, but instead include tangible storage media for non-transient states. Disks and optical disks such as those used include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, wherein disks usually reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media.

[0091] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the terms "processor" or "processing circuit system" as used herein may each refer to any of the aforementioned structures or any other structure suitable for implementing the described techniques. In addition, in some examples, the described functionality may be provided within dedicated hardware and / or software modules. Moreover, the described techniques may be implemented entirely in one or more circuits or logic elements.

[0092] As used herein, a processing component may include a processing circuit system as described above. In some examples, the processing component may include at least one processor and at least one memory having a computer code, the computer code including an instruction set, which when executed by the at least one processor causes the at least one processor to perform any of the functions described herein. In some examples, the processing component may receive the computer code including the instruction set from at least one memory coupled to the processing component.

[0093] The techniques of the present disclosure may be implemented in a variety of devices or equipment, including wireless handsets, mobile or non-mobile computing devices, wearable or non-wearable computing devices, integrated circuits (ICs) or IC sets (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize the functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in hardware units or provided by a collection of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

[0094] Therefore, various aspects of the technology can achieve the following embodiments: Embodiment 1. A method comprising: predicting, by a data platform implemented by a computing system, an incoming data rate of native format data to be archived; comparing the predicted incoming data rate with a first threshold data rate; and in response to determining that the predicted incoming data rate exceeds the first threshold data rate: dividing, by the data platform, the native format data into blocks; and writing the blocks directly to an archive storage system as block files.

[0095] Embodiment 2. The method as described in Embodiment 1 further includes: in response to determining that the predicted incoming data rate is lower than the first threshold data rate and exceeds a second threshold data rate, writing the native format data to the local repository of the data platform; in response to determining that the time limit for writing the native format data to the local repository has been exceeded or the amount of the native format data exceeds a threshold data amount: replaying the reception of the native format data; dividing the native format data into the blocks; and writing the blocks to the archive storage system.

[0096] Embodiment 3. The method of embodiment 2, wherein writing the native format data to the local storage repository comprises: writing the native format data to one or more of a random access memory or a flash memory of the local storage repository.

[0097] Embodiment 4. The method as described in any combination of embodiments 1 to 3, further comprising: in response to determining that the predicted incoming data rate is lower than the first threshold data rate: splitting the native format data into the blocks; and writing the blocks to a local storage system of the data platform.

[0098] Embodiment 5. The method of embodiment 4, wherein writing the block to the local storage system of the data platform comprises: writing the block to one or more of a solid state drive and a hard disk drive of the local storage system.

[0099] Embodiment 6 The method of embodiment 4 further comprising: in response to determining that the block written to the local storage system exceeds a threshold amount of data for forming a block file, writing the block as a block file to the archival storage system.

[0100] Embodiment 7. A method as described in any combination of Embodiments 1 to 6, wherein predicting the incoming data rate of the native format data includes: applying one or more of a machine learning model and an artificial intelligence model to the native format data to obtain the predicted incoming data rate.

[0101] Embodiment 8. A method as described in any combination of Embodiments 1 to 7, wherein writing the blocks directly to the archival storage system as the block files comprises: writing the blocks to the archival storage system as the block files without accumulating the blocks, and writing the blocks to a local storage system of the data platform rather than a local storage repository separate from the local storage system within a threshold time limit.

[0102] Embodiment 9. The method of any combination of Embodiments 1 to 8, wherein segmenting the native format data into the blocks comprises segmenting the native format data into the blocks regardless of whether a size of the native format data exceeds a minimum block file size.

[0103] Embodiment 10 The method of any combination of Embodiments 1 to 9, wherein the first threshold data rate comprises one or more threshold data rates that vary depending on a source or format of the native format data.

[0104] Embodiment 11. The method of any combination of Embodiments 1 to 10, wherein the source of the native format data does not allow installation of an agent to manage transmission of the native format data to the data platform.

[0105] Embodiment 12. The method as recited in any combination of Embodiments 1 to 11, further comprising: receiving native format data from a source as a data stream.

[0106] Embodiment 13. The method as recited in any combination of Embodiments 1 to 12, wherein the archival storage system resides in and is supported by a cloud computing infrastructure, or resides locally with the data platform.

[0107] Embodiment 14. A computing system comprising: a storage device; and a processing circuit system having access to the storage device and configured to: predict an incoming data rate of native format data to be archived to obtain a predicted incoming data rate of the native format data to be archived; compare the predicted incoming data rate with a first threshold data rate; and in response to determining that the predicted incoming data rate exceeds the first threshold data rate: split the native format data into blocks; and write the blocks directly to an archival storage system as block files.

[0108] Embodiment 15. A computing system as described in Embodiment 14, wherein the processing circuit system is further configured to: in response to determining that the predicted incoming data rate is lower than the first threshold data rate and exceeds a second threshold data rate, write the native format data to the local repository of the data platform; in response to determining that a time limit for writing the native format data to the local repository has been exceeded or the amount of the native format data exceeds a threshold data amount: replay the receipt of the native format data; split the native format data into the blocks; and write the blocks to the archival storage system.

[0109] Embodiment 16. The computing system of Embodiment 15, wherein the processing circuitry is configured to write the native format data to one or more of a random access memory or a flash memory of the local repository.

[0110] Embodiment 17. A computing system as described in any combination of Embodiments 14 to 16, wherein the processing circuit system is further configured to: in response to determining that the predicted incoming data rate is lower than the first threshold data rate: split the native format data into the blocks; and write the blocks to a local storage system of the data platform.

[0111] Embodiment 18. The computing system of Embodiment 17, wherein the processing circuitry is configured to write the block to one or more of a solid state drive and a hard disk drive of the local storage system.

[0112] Embodiment 19. The method of Embodiment 17, wherein the processing circuitry is further configured to: in response to determining that the block written to the local storage system exceeds a threshold amount of data for forming a block file, write the block to the archival storage system as a block file.

[0113] Embodiment 20. A computer-readable storage medium comprising: instructions that, when executed, configure a processing circuit system of a computing system to: predict an incoming data rate of native format data to be archived to obtain a predicted incoming data rate of the native format data to be archived; compare the predicted incoming data rate with a first threshold data rate; and in response to determining that the predicted incoming data rate exceeds the first threshold data rate: split the native format data into blocks; and write the blocks directly to an archival storage system as block files.

Claims

1. A method implemented by a computing system, comprising: predicting, by a data platform implemented by the computing system, an incoming data rate of data to be archived; comparing the predicted incoming data rate to a first threshold data rate; as well as In response to determining that the predicted incoming data rate exceeds the first threshold data rate: dividing the data into blocks by the data platform; and The chunks are written directly to the archive storage system as chunk files.

2. The method of claim 1, further comprising: In response to determining that the predicted incoming data rate is below the first threshold data rate and exceeds a second threshold data rate, writing the data to a local repository of the data platform; In response to determining that a time limit for writing the data to the local repository has been exceeded or that an amount of the data exceeds a threshold amount of data: replaying the receipt of said data; dividing the data into the blocks; and writing the blocks to the archival storage system, and optionally Wherein, writing the data into the local storage repository comprises: writing the data into one or more of a random access memory or a flash memory of the local storage repository.

3. The method of claim 1, further comprising: In response to determining that the predicted incoming data rate is below the first threshold data rate: dividing the data into the blocks; and The blocks are written to the local storage system of the data platform.

4. The method of claim 3, wherein: Writing the block to the local storage system of the data platform includes writing the block to one or more of a solid state drive and a hard disk drive of the local storage system.

5. The method of claim 3, further comprising: In response to determining that the block written to the local storage system exceeds a threshold amount of data for forming a block file, the block is written to the archival storage system as a block file.

6. The method of claim 1, wherein: Predicting the incoming data rate of the data includes applying one or more of a machine learning model and an artificial intelligence model to the data to obtain the predicted incoming data rate.

7. The method of claim 1, wherein: Writing the block directly to the archival storage system as the block file includes: writing the block to the archival storage system as the block file without accumulating the block, and writing the block to a local storage system of the data platform rather than a local repository separate from the local storage system within a threshold time limit.

8. The method of claim 1, wherein: Splitting the data into the blocks includes splitting the data into the blocks regardless of whether a size of the data exceeds a minimum block file size.

9. The method of claim 1, wherein: The first threshold data rate includes one or more threshold data rates that vary depending on a source or format of the data.

10. The method of claim 1, wherein: The source of the data does not allow installation of an agent to manage the transfer of the data to the data platform.

11. A computing system comprising: Storage device; as well as a processing member having access to the storage device and configured to: predicting an incoming data rate of data to be archived to obtain a predicted incoming data rate of the data to be archived; comparing the predicted incoming data rate to a first threshold data rate; and In response to determining that the predicted incoming data rate exceeds a first threshold data rate: dividing the data into blocks; and The chunks are written directly to the archive storage system as chunk files.

12. The computing system of claim 11, wherein: The processing circuit system is further configured to: In response to determining that the predicted incoming data rate is below the first threshold data rate and exceeds a second threshold data rate, writing the data to a local repository of the data platform; In response to determining that a time limit for writing the data to the local repository has been exceeded or that an amount of the data exceeds a threshold amount of data: replaying the receipt of said data; dividing the data into the blocks; and writing the blocks to the archival storage system, and optionally The processing circuit system is configured to write the data to one or more of a random access memory or a flash memory of the local storage repository.

13. The computing system of claim 11, wherein: The processing circuit system is further configured to: In response to determining that the predicted incoming data rate is below the first threshold data rate: dividing the data into the blocks; and The blocks are written to the local storage system of the data platform.

14. The computing system of claim 13, wherein: The processing circuitry is further configured to, in response to determining that the block written to the local storage system exceeds a threshold amount of data for forming a block file, write the block to the archival storage system as a block file.

15. A computer-readable storage medium comprising instructions which, when executed, cause one or more processors to perform the method of any one of claims 1 to 10.

Images