Intelligent job scheduling for pipelines using backlog indicators

Abstract

The invention relates to intelligent job scheduling for pipelines using backlog indicators. Techniques are described for configuring a data platform to schedule workloads using backlog indicators. For example, a processing device of a data platform may obtain a generic backlog indicator of a workload to be executed by the data platform. Each of the workloads may specify one or more storage system maintenance operations. A processing device may obtain a customized backlog indicator for at least a subset of the workloads. A processing device may calculate a single weighted backlog indicator value for each of the workloads by applying a configurable weight to the generic backlog indicator and the customized backlog indicator. The data platform may schedule the workloads for execution on the data platform based on the single weighted backlog indicator value calculated for each workload. In some examples, the data platform processes the workload according to the schedule.

Description

[0001] Claiming priority

[0002] This application claims the benefit of Indian Provisional Patent Application No. 202341056996 filed on August 25, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a data platform for a computing system. Background Art

[0004] Data platforms that support computational applications are required to perform a variety of tasks, including periodically repeating end-user tasks, background tasks, and overhead tasks, all of which support the direct or indirect goals of the end-users and the overall efficiency of the data platform. Many of these tasks are scheduled based on a fixed run frequency associated with a given task. Due to the limited compute resources allocated to run such tasks, end users and platform administrators may encounter situations where critical jobs cannot be scheduled in a timely manner due to an overall lack of available system resources, resulting in an increase in the data platform's work backlog and ultimately subsequent impacts, including degraded operational performance, missing service level agreements (SLAs), and possible running out of space on clusters that rely on primary storage systems to support latency-sensitive applications. Summary of the invention

[0005] Various aspects of the present disclosure describe techniques for performing intelligent job scheduling of pipelines based on the use of both custom backlog indicators and generic backlog indicators associated with enqueuing pipelines. A pipeline (also referred to as a workload) may be associated with a generic backlog indicator, which may represent a measure of time that a given workload remains in an unscheduled state after the workload has completed a previous execution. Certain pipelines or workloads may additionally be associated with custom backlog indicators, which may be configured according to specific implementation needs, including, for example, a measure of utilization of a specified one or more resources within a data platform that are affected by the data platform executing the corresponding workload.

[0006] The techniques described in this disclosure can optimize the scheduling of workloads and pipelines by using general backlog indicators and workload-specific customized backlog indicators, rather than scheduling workloads primarily based on the associated fixed run frequency of any given task, which can result in excessive utilization and excessive execution delays.

[0007] The use of generic backlog indicators associated with enqueue pipelines and custom backlog indicators (when available and already associated with a portion or subset of the enqueue pipelines) improves the operational efficiency of a data platform or custom configured computing system by minimizing delayed execution of jobs, excessive computational burden on certain computing clusters, and can result in greater responsiveness of the data platform. For example, intelligent scheduling can eliminate situations where end users are unable to execute critical jobs by specified deadlines due to lack of computing resources, which in turn leads to an increase in work backlogs, with the side effects of degraded performance, missed SLAs, and running out of data storage space on the cluster. In addition, it is not feasible to manually address each end user's concerns due to the perceived significant degradation in the operational performance of the data platform. Therefore, an automated and improved technique for intelligent scheduling of workloads is provided, such as intelligent scheduling of workloads, algorithms, and routines related to garbage collection or maintenance operations.

[0008] Not all workloads, pipelines, and algorithms are dedicated to garbage removal operations. While the specific operation depends on the specific pipeline implementation, alternatives to garbage collection operations include workloads that perform collection of system-wide statistics, operational statistics on a per-cluster, per-end-user, per-workload, or per-datacenter basis, workloads that fix data replication factors, erasure coding (EC) operations, deduplication operations, cloud archive operations, stale session identification and removal operations, etc.

[0009] For example, a workload may specify one or more storage system maintenance operations for a storage system under the control of the data platform. Examples of such storage system maintenance operations include garbage removal operations, data replication redundancy operations, storage node recovery operations, and server message block (SMB) session removal operations. For example, a garbage removal operation may remove data from one or more storage systems in response to determining that the data platform has identified the data as available for removal from one or more storage systems managed by the data platform. In one example, a workload / pipeline that implements a garbage collection and / or data removal process identifies which data is available for removal by initially identifying metadata that references the deleted data, and then, if the deleted data is not referenced by any metadata, the garbage collection and / or data removal process removes the deleted data.

[0010] A data replication redundancy operation may replicate a data instance to create a new operational replica of the data instance in response to determining that a count of an operational replica of the data instance is below a configured redundancy minimum. In some examples, a storage node recovery operation may clone, replicate, and / or copy a data instance to a new storage node in response to determining that a storage node failure has caused a count of an operational data instance to be below a configured redundancy minimum. Another example of a storage system maintenance operation includes a server message block (SMB) session removal operation that removes one or more SMB sessions from one or more storage systems in response to determining that the data platform has identified the one or more SMB sessions as available for removal from one or more storage systems managed by the data platform.

[0011] In one example, various aspects of these techniques relate to a method. An example method may include obtaining, by a processing device of a data platform, a general backlog indicator for a plurality of workloads to be executed by the data platform. In such an example, each of the plurality of workloads may specify one or more storage system maintenance operations for one or more storage systems managed by the data platform. In some examples, the processing device of the data platform obtains a customized backlog indicator for at least a subset of the plurality of workloads to be executed by the data platform. The processing device may calculate a single weighted backlog indicator value for each of the plurality of workloads to be executed by the data platform. For example, a single weighted backlog indicator value may be calculated by applying a configurable weight to each of the general backlog indicator and the customized backlog indicator of the corresponding workload from the plurality of workloads. In some examples, the processing device schedules the plurality of workloads for execution on the data platform based on the single weighted backlog indicator value calculated for each of the plurality of workloads. In such an example, the data platform may process the plurality of workloads according to the schedule.

[0012] In another example, various aspects of these technologies relate to a data platform having a storage system and a processing device that can access the storage system. In this example, the processing device is configured to perform various operations. In this example, the processing device can obtain a customized backlog indicator for at least a subset of multiple workloads to be executed by the data platform. The processing device can calculate a single weighted backlog indicator value for each of the multiple workloads to be executed by the data platform by applying a configurable weight to each of the general backlog indicator and the customized backlog indicator of the corresponding workload from the multiple workloads. In some examples, the processing device schedules multiple workloads for execution on the data platform based on the single weighted backlog indicator value calculated for each of the multiple workloads. In this example, the data platform can process the multiple workloads according to the schedule.

[0013] In another example, various aspects of these technologies relate to a computer-readable storage medium having instructions that, when executed, configure one or more processors to perform various operations. In this example, the instructions, when executed, may configure one or more processors to obtain a general backlog indicator for multiple workloads to be executed by a data platform. In this example, each of the multiple workloads may specify one or more storage system maintenance operations for one or more storage systems managed by the data platform. In some examples, the instructions configure one or more processors to obtain a customized backlog indicator for at least a subset of the multiple workloads to be executed by the data platform. The instructions may configure one or more processors to calculate a single weighted backlog indicator value for each of the multiple workloads to be executed by the data platform. For example, a single weighted backlog indicator value may be calculated by applying a configurable weight to each of the general backlog indicator and the customized backlog indicator of the corresponding workload from the multiple workloads. In some examples, the instructions may configure one or more processors to schedule multiple workloads for execution on the data platform based on a single weighted backlog indicator value calculated for each of the multiple workloads. In this example, the data platform may process multiple workloads according to the schedule. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A and Figure 1B is a block diagram illustrating an example system for intelligent job scheduling of an execution pipeline according to one or more aspects of the present disclosure.

[0015] Figure 2 is a graph illustrating the relationship between multiplier values ​​and run interval seconds according to one or more aspects of the present disclosure.

[0016] Figure 3 is a block diagram illustrating an example system in accordance with techniques of this disclosure.

[0017] Figure 4 is a flow chart illustrating an example mode of operation of a data platform 150 to perform intelligent job scheduling of workloads based on the use of custom and generic backlog indicators associated with enqueued workloads in accordance with techniques of this disclosure.

[0018] Like reference characters refer to like elements throughout the text and drawings. DETAILED DESCRIPTION

[0019] Figure 1A and Figure 1B is a block diagram illustrating an example system for intelligent job scheduling of an execution pipeline according to one or more aspects of the present disclosure. Figure 1AIn the example of , system 100 includes application system 102. Application system 102 represents a collection of hardware devices, software components, and / or data storage that can be used to implement one or more applications or services provided to one or more mobile devices 108 and one or more client devices 109 via network 113. Application system 102 may include one or more physical or virtual computing devices that execute workloads 174 of the application or service. Workload 174 may include one or more virtual machines, containers, Kubernetes pods, each of which includes one or more containers, bare metal processes, and / or other types of workloads.

[0020] exist Figure 1A In the example of , application system 102 includes application servers 170A to 170M (collectively referred to as "application servers 170") connected to a database server 172 that implements a database via a network. Other examples of application system 102 may include one or more load balancers, web servers, network devices (such as switches or gateways), or other devices for implementing one or more applications or services and delivering one or more applications or services to mobile devices 108 and client devices 109. Application system 102 may include one or more file servers. One or more file servers may implement a primary file system for application system 102. (In such instances, file system 153 may be a secondary file system that provides backup, archiving, and / or other services for the primary file system. References to file systems herein may include a primary file system or a secondary file system, for example, a primary file system of application system 102 or file system 153 that is used as a primary file system or a secondary file system.)

[0021] The application system 102 may be located in a venue and / or in one or more data centers, each of which is part of a public cloud, a private cloud, or a hybrid cloud. The application or service may be a distributed application. The application or service may support enterprise software, financial software, office or other productivity software, data analysis software, end-user relationship management, web services, educational software, database software, multimedia software, information technology, healthcare software, or other types of applications or services. The application or service may be provided as a service (-aaS) as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), data storage as a service (dSaaS), or other types of services.

[0022] In some examples, application system 102 may represent an enterprise system that includes one or more workstations in the form of desktop computers, laptop computers, mobile devices, enterprise servers, network devices, and other hardware to support enterprise applications. Enterprise applications may include enterprise software, financial software, office or other productivity software, data analysis software, end-user relationship management, web services, educational software, database software, multimedia software, information technology, healthcare software, or other types of applications. Enterprise applications may be delivered as a service from an external cloud service provider or other provider, executed locally on application system 102, or both.

[0023] exist Figure 1A In the example of , the system 100 includes a data platform 150, which uses the storage system 105 and the separate storage system 115 to provide the application system 102 with a file system 153 and archiving functions. The data platform 150 implements a distributed file system 153 and a storage architecture to facilitate the application system 102 to access the file system data and to facilitate the transmission of data between the storage system 105 and the application system 102 through the network 111. Using the distributed file system, the data platform 150 enables the device of the application system 102 to use a communication protocol to access the file system data through the network 111, as if such file system data is stored locally (for example, stored to the hard disk of the device of the application system 102). Example communication protocols for accessing files and objects include server message blocks (SMB), network file systems (NFS), or AMAZON simple storage services (S3). The file system 153 can be the primary file system or a secondary file system of the application system 102.

[0024] The file system manager 152 represents a collection of hardware devices and software components that implement the file system 153 for the data platform 150. Examples of file system functions provided by the file system manager 152 include storage space management, which includes deduplication, file naming, directory management, metadata management, partitioning, and access control. The file system manager 152 executes communication protocols to facilitate the application system 102 to access files and objects stored in the storage system 105 through the network 111.

[0025] exist Figure 1B In the example of , system 190 includes data platform 150, which uses storage system 105 and separate storage system 115 to provide file system 153 and archiving functions to application system 102. Data platform 150 implements distributed file system 153 and storage architecture to facilitate access to file system data by application system 102 and to facilitate data transmission between storage system 105 and application system 102 over network 111. As depicted here, storage system 115 is represented as being collocated with data platform 150.

[0026] The data platform 150 includes a storage system 105 having one or more storage devices 180A to 180N (collectively, "storage devices 180"). Storage devices 180 may represent one or more physical or virtual computers and / or storage devices that include or otherwise have access to storage media. Such storage media may include one or more of the following: a flash drive, a solid-state drive (SSD), a hard disk drive (HDD), an electrically programmable memory (EPROM) or electrically erasable programmable (EEPROM) memory format, and / or other types of storage media used to support the data platform 150. Different storage devices of storage devices 180 may have a mix of different types of storage media.

[0027] Each storage device in the storage device 180 may include system memory. Each storage device in the storage system 180 may be a storage server, a network attached storage (NAS) device, or may represent disk storage of a computer device. The storage system 105 may be a redundant array of independent disks (RAID) system. In some examples, one or more of the storage devices 180 is both a computing device and a storage device, which executes the software of the data platform 150, such as the file system manager 152 and the scheduler 154 in the example of the system 100, and stores the objects and metadata of the data platform 150 to the storage medium. In some examples, a separate computer device (not shown) may execute the software of the data platform 150, such as the file system manager 152 and the scheduler 154 in the example of the system 100. Each storage device in the storage device 180 may be considered and referred to as a "storage node" or simply a "node". The storage device 180 may represent a virtual machine running on a supported hypervisor, a cloud virtual machine, a physical rack server, or a computing model installed in a converged platform.

[0028] In various examples, data platform 150 may run locally on a physical system, virtually, or in a cloud. For example, data platform 150 may be deployed as a physical cluster, a virtual cluster, or a cloud-based cluster running in a private cloud, a hybrid private / public cloud, or a public cloud deployed by a cloud service provider. In some examples of system 100, multiple instances of data platform 150 may be deployed, and file system 153 may be replicated between instances. In some cases, data platform 150 may be a computing cluster representing a single management domain. The number of storage devices 180 may be scaled to meet performance requirements.

[0029] The data platform 150 may implement multiple storage domains and provide multiple storage domains to one or more tenants, or isolate workloads 174 that require different data policies. A storage domain may be a data policy domain that determines the policies for deduplication, compression, encryption, tiering, and other operations performed with respect to objects stored using the storage domain. In this way, the data platform 150 may provide users with the flexibility to select global data policies or workload-specific data policies. The data platform 150 may support partitioning.

[0030] A view is a protocol export that resides within a storage domain. A view inherits the data policy from its storage domain, but additional data policies can be specified for the view. Views can be exported via SMB, NFS, S3, and / or another communication protocol. Policies that determine data processing and storage for the data platform 150 can be assigned at the view level. A protection policy can specify backup frequency and retention policy, which can include a data lock period. Archives 142 or snapshots created according to a protection policy inherit the data lock period and retention period specified by the protection policy.

[0031] Each of network 113 and network 111 may be the Internet or may include or represent any public or private communication network or other network. For example, network 113 may be a cellular, Near field communication (NFC), satellite, enterprise, service provider, and / or other types of networks capable of transferring data between computing systems, servers, computing devices, and / or storage devices. One or more of such devices may use any suitable communication technology to transmit and receive data, commands, control signals, and / or other information over network 113 or network 111. Each of network 113 or network 111 may include one or more network hubs, network switches, network routers, satellite antennas, or any other network equipment.

[0032] Such network devices or components may be operatively coupled to each other to provide for information exchange between computers, devices, or other components (eg, between one or more client devices or systems and one or more computer / server / storage devices or systems). Figure 1A to Figure 1B Each of the devices or systems shown may be operatively coupled to network 113 and / or network 111 using one or more network links. The links coupling such devices or systems to network 113 and / or network 111 may be Ethernet, asynchronous transfer mode (ATM), or other types of network connections, and such connections may be wireless and / or wired connections. Figure 1A and Figure 1B One or more of the devices or systems shown or otherwise on network 113 and / or network 111 may be remotely located relative to one or more of the other shown devices or systems.

[0033] The application system 102 uses the file system 153 provided by the data platform 150 to generate objects and other data, and the file system manager 152 creates, manages and stores the other data in the storage system 105. Therefore, the application system 102 may be alternatively referred to as a "source system", and the file system 153 of the application system 102 may be alternatively referred to as a "source file system". The application system 102 may communicate directly with the storage system 105 through the network 111 to transfer objects for some purposes, and communicate with the file system manager 152 through the network 111 to indirectly obtain objects or metadata from the storage system 105 for some purposes.

[0034] The file system manager 152 generates metadata and stores it to the storage system 105. The data set stored to the storage system 105 and used to implement the file system 153 is referred to herein as file system data. The file system data may include the aforementioned metadata and objects. The metadata may include file system objects, tables, trees, or other data structures; metadata generated to support deduplication; or metadata used to support snapshots. The stored objects may include files, virtual machines, databases, applications, pods, containers, any workload 174, system images, directory information, or other types of objects used by the application system 102. Objects of different types and objects of the same type can be deduplicated relative to each other.

[0035] The storage system 115 includes one or more storage devices 140A to 140X (collectively referred to as "storage devices 140"). The storage device 140 may represent one or more physical or virtual computers and / or storage devices that include or otherwise have access to storage media. Such storage media may include one or more of the following: a flash drive, a solid-state drive (SSD), a hard disk drive (HDD), an optical disk, an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory form, and / or other types of storage media. Different storage devices of the storage device 140 may have a mixture of different types of storage media. Each storage device in the storage device 140 may include system memory. Each storage device in the storage system 140 may be a storage server, a network attached storage (NAS) device, or may represent disk storage of a computer device. The storage system 115 may include a redundant array of independent disks (RAID) system. The storage system 115 may be capable of storing much larger amounts of data than the storage system 105. The storage device 140 may be further configured for long-term storage of information that is more suitable for archival purposes.

[0036] In some examples, storage systems 105 and / or 115 may be storage systems deployed and managed by a cloud storage provider and are referred to as "cloud storage systems". Example cloud storage providers include, for example, Amazon Web Services (AWS) of Amazon Web Services, Inc. TM ), Microsoft DROPBOX from DROPBOX TM Oracle Cloud TM and Google Cloud Platform TM (GCP). In some examples, storage system 115 is co-located with storage system 105 in a data center, on premises, or in a private cloud, public cloud, or hybrid private / public cloud. Storage system 115 can be considered a "backup" or "secondary" storage system for primary storage system 105. Storage system 115 can be referred to as an "external target" for archive 142. Storage system 115 can be referred to as "cloud storage" when deployed and managed by a cloud storage provider.

[0037] The storage system 115 may include one or more interfaces for managing data transfer between the storage system 105 and the storage system 115 and / or between the application system 102 and the storage system 115. The data platform 150 supporting the application system 102 relies on the primary storage system 105 to support latency-sensitive applications. However, because the storage system 105 is generally more difficult or expensive to scale, the data platform 150 may use the secondary storage system 115 to support secondary use cases such as backup and archive. Generally speaking, a file system backup is a copy of the file system 153 to support protection of the file system 153 for rapid recovery, which is usually due to the loss of some data in the file system 153, and a file system archive ("archive") is a copy of the file system 153 to support longer-term retention and review. A "copy" of the file system 153 may include such data as is needed to restore or view the state of the file system 153 at the time of the backup or archive. Each "copy" of the file system 153 may be referred to as a "replica," "copy instance," "operational copy," and / or "data instance."

[0038] Archive manager 154 can archive the file system data of file system 153 at any time according to an archive policy, which specifies, for example, archive periodicity and timing (daily, weekly, etc.), which file system data will be archived, archive retention period, storage location, access control, etc. The initial archive of file system data can correspond to the state of the file system data at the initial archive time (archive creation time of the initial archive). Depending on the archive policy, the initial archive may include a complete archive of the file system data, or may include an incomplete archive of the file system data. For example, the initial archive may include all objects of the file system 153 or one or more selected objects of the file system 153.

[0039] One or more subsequent incremental archives of file system 153 may correspond to respective states of file system 153 at respective subsequent archive creation times (i.e., after the archive creation time corresponding to the initial archive). Subsequent archives may include incremental archives of file system 153. Subsequent archives may correspond to incremental archives of one or more objects of file system 153. Some file system data of file system 153 stored on storage system 105 at the time of initial archive creation may also be stored on storage system 105 at the time of subsequent archive creation. Subsequent incremental archives may include data that was not previously archived to storage system 115. Archive manager 154 may deduplicate file system data included in subsequent archives based on file system data included in one or more previous archives (including the initial archive) to reduce the amount of storage used. (References to "time" in this disclosure may refer to dates and / or times. Times may be associated with dates. For example, multiple archives may appear at different times on the same date.)

[0040] In this example, the data platform 150 includes a scheduler 154 that can provide scheduling services for overhead tasks, background tasks, and end-user tasks (collectively, workloads 174) that interact with the file system 153. In the example of the system 100, the scheduler 154 periodically schedules repetitive workloads 174 that interact with file system data stored by the storage system 105 based on a scaled backlog indicator (SBI) 185. The priority manager 184 can calculate the scaled backlog indicator 185 as a composite value or score on behalf of the scheduler 154 based on various metrics (including a generic backlog indicator (GBI) 189, a customized backlog indicator (CBI) 187, and the urgency 107 of the specific workload 174) according to the policy 158 established and configured for the scheduler 154.

[0041] Continuing with this example, the scheduler 154 provides scheduling services for various workloads 174. For example, the scheduler 154 may utilize defined backlog indicators to intelligently schedule various tasks, jobs, or workloads 174 that are performed on behalf of platform end users or platform administrators. The backlog indicator may be a value between 0 and 100 having two parts, specifically, a generic indicator or generic backlog indicator 189, which is used to provide fairness between different pipelines based on the desired scheduling frequency of the workload 174 in question, and a custom indicator or custom backlog indicator 187, which may be determined by an algorithm that defines the backlog for a given workload 174. The framework of the scheduler 154 may allow for the addition of more and more custom backlog indicators for different pipelines. The compute workloads 174 may be scheduled in descending order of their backlog indicator values, which may be dynamically recalculated based on the configuration policy 158 of the scheduler.

[0042] The data platform 150 may perform various repetitive periodic computing workloads 174 associated with maintenance and / or optimization of the storage systems 105, 115 within and / or under the control of the data platform 150. Figure 1A In an example of , each of the storage systems 105, 115 includes a maintenance engine 145. The maintenance engine 145 may include one or more storage system maintenance operations 146, logic and / or execution modules. For example, the maintenance engine 145 may perform a variety of storage system 105, 115 optimization operations, cleanup operations, and / or SLA compliance operations. In some examples, one or more storage system maintenance operations 146 may include a garbage removal operation. For example, the garbage removal operation may remove data from one or more storage systems in response to determining that the data platform has identified the data as available for removal from the one or more storage systems.

[0043] In other examples, one or more storage system maintenance operations 146 may include data copy redundancy operations. Such data copy redundancy operations may copy the data instance to create a new operational copy of the data instance in response to determining that the count of the operational copy of the data instance is below the configured redundancy minimum. In other examples, one or more storage system maintenance operations 146 may include data copy redundancy operations. In the storage systems 105, 115, there may be a large number of storage nodes operating simultaneously, and when one of the storage nodes encounters a failure (e.g., a software problem or a hardware failure), the failed storage node may cause the previous operating node and the data instance to be offline. Therefore, the data copy redundancy operation may clone the data instance to a new storage node in response to determining that a storage node failure within one or more storage systems 105, 115 reduces the count of the operational data instance to below the configured redundancy minimum. In yet another example, one or more storage system maintenance operations 146 may include a server message block (SMB) session removal operation. Such an SMB session removal operation may remove one or more SMB sessions from one or more storage systems in response to determining that the data platform 150 has identified one or more SMB sessions as available for removal.

[0044] exist Figure 1BIn an example of FIG. 1 , the maintenance engine 145 is depicted in greater detail. In this example, the maintenance engine 145 also includes a server message block (SMB) cleaner 160, a garbage collector 162, an instance replicator 166, and a node cloning module 168. In some examples, the garbage removal operations performed by the garbage collector 162 remove garbage 164 that has accumulated on the cluster. In some cases, the data platform 150 may be unable to keep up with the garbage collection workload 174 using fixed interval scheduling because the requested garbage collection ("GC") workload 174 may be excessively delayed or initiated but then canceled before it can complete its processing objectives. The garbage collector 162 may perform garbage collection configured by a processing device (e.g., processing circuitry) of the data platform 150 to remove garbage 164 identified within the storage system 115 as a maintenance and / or processing optimization operation of the data platform 150 itself or on a per-end-user basis. The scheduler 154 may optionally be configured to make garbage collection on the data platform 150 more aggressive by setting certain configurable settings, including using configurable flags, such as "Google Command Line Flags" or "gflags". The gflags defined at "gflags.github.io / gflags" are not specific to garbage collection, but can be used to prioritize the garbage collection and / or data removal process. Setting these gflags will typically require manual intervention, and they are typically configured in a reactive rather than proactive manner to address higher cluster utilization and growing estimated garbage 164 for end users. Restoring these gflags may be performed manually, automatically, after a configurable pause period and / or when cluster utilization drops to a level that meets a configured threshold, indicating that accumulated garbage is under control.

[0045] Optionally, the data platform 150 may expose an "aggressive GC UI button" to an administrator-accessible UI and, optionally, to an end-user UI that is configured to apply aggressive GC gflags with a single button click. However, this approach may not always be suitable for exposure to end users, and therefore may require management support involvement to evaluate the operational performance of the end-user cluster state before applying aggressive GC settings, resulting in delays and additional costs. For example, consider the problem of cluster sizing where there is not much garbage 164 to collect. Even if the option is exposed directly to the end user, the end user setting aggressive GC will not solve the root problem, and therefore, due diligence by management support will still be required because exposing the configuration functionality to the end user may simply result in reduced performance of the data platform 150, which is counter-intuitive due to the increased computational burden associated with activating the end-user configured aggressive GC settings.

[0046] While establishing an "aggressive GC button" through an end user or management UI can improve the process by providing a simplified single button click and configuration for automatically restoring gflags when deadlines are reached, greater optimization can be achieved through the use of the scheduler 154 and specifically through intelligent job scheduling of the execution pipeline.

[0047] Processes other than the garbage collector 162 may utilize the general backlog indicator 189. For example, the server message block (SMB) cleaner 160 may utilize the general backlog indicator 189 for a server message block (SMB) session removal operation. The instance replicator 166 may utilize the general backlog indicator 189 for a data replication redundancy operation. The node cloning module 168 may utilize the general backlog indicator 189 for a storage node recovery operation. Similarly, the custom backlog indicator 187 may optionally be configured to be used with various maintenance operations 146, although this is not required. When the custom backlog indicator 187 is configured for any of the garbage collection operations, SMB session removal operations, data replication redundancy operations, and / or storage node recovery operations, the priority manager 184 may use the values ​​corresponding to each of the general backlog indicator 189 and the custom backlog indicator 187 for each respective workload 174 to calculate a single weighted backlog indicator value.

[0048] The instance replicator 166 can operate as a replication monitor and repairer. For example, when it is determined that the number of operational copies of the data instance of the storage system 105, 115 is less than a configurable minimum or less than the number required by the SLA, the instance replicator 166 can automatically create a new operational instance. For example, a new operational instance can be created by replicating the master data instance or by copying and / or replicating one of the remaining operational copies. In this way, the maintenance engine 145 can maintain and / or save a configurable minimum number of simultaneous copies of the data instance. The node cloning module 168 can operate to replicate, duplicate and / or clone storage nodes within the storage system 105, 115. Such storage nodes can have data replicas, and therefore cloning the storage nodes will produce new data instance replicas.

[0049] In other cases, the cloned storage node supplies an operational storage node, and the new data instance copy can be overwritten, repeated and / or copied to the storage node. When a failed storage clone is found, the computing load on the data platform 150 changes and / or the SLA that specifies the total amount of available and operational storage nodes increases changes, the node cloning module 168 can automatically clone the storage node. In any case, the workload 174 can be executed and checked periodically for the total number of operating storage nodes. When the total number of operating storage nodes meets the threshold, SLA or other standards, the workload 174 can be completed very quickly, given that any storage node recovery operation will not be executed or is quickly executed and completed when the operational storage node conditions are met. On the contrary, when the storage node recovery operation must clone one or more storage nodes, the computing load may be very large. Therefore, the storage node recovery operation may not be completed in time, resulting in a backlog of the workload 174. Therefore, the priority manager 184 can process the resulting backlog based on the general backlog indicator 189 associated with the workload 174 of the storage node recovery operation and the optional custom backlog indicator 187 when configured.

[0050] In a similar example, the instance replicator 166 can use the scheduled workload 174 to initiate data replication redundancy operations at a periodic frequency. As in the previous example, many such workloads 174 can be executed to completion very quickly because the number of operating data instances meets the minimum value. However, consider an example of discovering insufficient replication. For example, the end-user SLA specifies at least four simultaneous data instance replicas, and the instance replicator 166 only identifies two current operating instances. In this case, due to the low redundancy, not only does the SLA not meet the requirements, but the data within the data instance may also be at risk. Although this situation is extremely rare in the event of occurrence, if the workload 174 responsible for performing data replication redundancy operations causes any backlog, the priority manager 184 will require a very high level of urgency. As before, the priority manager 184 can use the general backlog indicator 189 and the optional customized backlog indicator 187 associated with the corresponding workload 174 to resolve the resulting backlog problem, so as to prioritize and potentially scale the backlog indicators, thereby speeding up the recovery of insufficient data instance replicas on the storage system 105, 115.

[0051] Another example is a cleanup session for a Server Message Block (SMB) protocol session. The Server Message Block protocol or "SMB protocol" is a network file sharing protocol that allows applications of the data platform 150 to read and write files and request services from server programs in a computer network. The SMB protocol can be used on top of its TCP / IP protocol or other network protocols. Similar to garbage collection of unwanted garbage 164, which consumes storage resources on the storage systems 105, 115, discarded, erroneous and / or malicious SMB protocol sessions can exhaust the available computing resources of the data platform 150. This is because the SMB protocol session is open and remains active, but is no longer needed. The SMB scrubber 160 can check the open sessions and initiate a workload 174 to perform an SMB session removal operation. When the SMB scrubber 160 finds few or no erroneous SMB sessions, the SMB session removal operation will be performed very quickly or not at all. However, when the data platform 150 is unable to complete the periodically requested SMB session removal operation due to, for example, an extended period of high computing load, the SMB session removal operation will cause a repeated backlog. As a result of this backlog over time, SMB session removal operations will require longer and longer periods of time to complete as the total number of SMB sessions to be cleaned increases. Therefore, the priority manager 184 can again use the generic backlog indicator 189 and optional custom backlog indicators 187 associated with the corresponding workload 174 to resolve the resulting backlog to prioritize SMB session removal operations on the storage systems 105, 115.

[0052] Maintenance operations 146 may correspond to algorithms that lack any knowledge about the amount of work to be performed each time they are initiated. Such algorithms may configure maintenance operations 146 workloads 174 by scanning meta tables and databases to identify the work to be performed. For example, the algorithm may scan various resources to identify how much garbage 164 is marked as available for removal from storage systems 105, 115, how many SMB sessions need to be cleaned, how many storage nodes need to be cloned, and / or how many data instance replicas need to be created. However, when the configured workload 174 cannot be completed within a single run, additional runs may be required to complete the unfinished tasks of the previously configured workload 174. In other words, when the algorithm is subsequently repeated, the remaining unfinished tasks may flow indirectly into the new workload 174, and again scan the meta tables and databases to identify the work to be performed to configure the workload 174 to perform the required maintenance operations 146. In some examples, the workload 174 is executed in two operational phases. The first operational phase may use an algorithm for identifying work to identify and discover the work that needs to be completed by scanning the meta tables and databases to configure the maintenance operations 146 into the workload 174. The second operating phase may then execute the configured workload 174 to perform the work configured into the workload 174 .

[0053] Deadlines or deadline expiration can be configured to terminate and / or cancel workloads that are executing but not completed or workloads that failed to start (and therefore never completed) within a configurable time period. Deadlines ensure that workloads do not use outdated, stale, and / or wrong information to perform work. Deadline configuration may be advantageous because data becomes stale over time. The more dynamic the data within the storage system 105, 115, the greater the likelihood that such data will become outdated within any given unit of time. In some examples, a configured workload may consume hours and sometimes days to complete. Therefore, the freshness of the workload 174 will decrease over time and eventually become irrelevant. The workload 174 that is relevant at the time of creation may be useless after a few hours or days. When the data platform 150 lacks sufficient computing resources to determine the work to be performed (first phase operation) and perform the work (second phase operation) due to excessive backlogs, the workload 174 may operate on wrong information and thus waste computing resources of the data platform 150. Thus, the priority manager 154 may resolve such backlogs that result in stale workloads 174 by using a single weighted backlog indicator value calculated for the workload to be executed by the data platform, by applying configurable weights to each of the generic backlog indicator and the custom backlog indicator for the respective workload.

[0054] In some examples, specialized flags may be set, such as "gflags" for prioritizing garbage collection tasks or other maintenance operations 146. For example, for pipelines and workloads 174 that exhibit high metadata utilization, gflags are used to prioritize metadata reduction algorithms, such as executing the SnapTreeDedup pipeline and increasing the QoS weight of Snap Tree actions. However, in some cases, these aggressive settings may cause greater latency impact on backup and replication workloads 174, and in this example, applying these settings may require frequent monitoring of the cluster and detailed re-tuning to appropriately adjust the settings to minimize latency and avoid negatively impacting the backup and replication workloads executed by the data platform.

[0055] According to various aspects of these technologies, such as Figure 1AThe illustrated data platform 150 can perform intelligent job scheduling of pipelines through a scheduler 154 based on the use of both custom backlog indicators 187 and generic backlog indicators 189 associated with enqueued pipelines and workloads. The scheduler 154 can perform intelligent job scheduling of pipelines to address the balance between high metadata utilization and system responsiveness to support low-latency applications by programmatically and automatically adjusting the data platform 150 algorithms and determining scheduling priorities based on the backlog amount of a given workload 174. In addition, the scheduler 154 can be configured to make smarter scheduling decisions not only for garbage collection but for any workload 174 and algorithm executed by the data platform 150, which is achieved by implementing generic scheduling logic that prevents other workloads 174 from competing for the limited computing resources of the data platform 150 and causing resource starvation.

[0056] In this case, scheduler 154 may use policies 158 and values ​​determined by priority manager 184 to help automate these processes and eliminate any need for human intervention, while potentially achieving desired operational metrics for data platform 150 .

[0057] Optimizations by the scheduler 154 may result in a reduction in garbage estimates that is faster than the process caused by activating the aggressive GCUI button. Prioritization of algorithms by the scheduler 154 may be based on stack sorting. The scheduler 154 may use an urgency value associated with each pipeline for prioritization. The scheduler 154 may prioritize scheduling algorithms based on the backlog of their algorithms. If the workload 174 backlog associated with an algorithm continues to increase, the scheduler 154 may trigger an alarm. The fairness of scheduling algorithms and workloads may be configured by detecting resource scarcity and re-prioritizing. For example, other workload algorithms should still receive run allocations even if there is a large deletion backlog.

[0058] If a pipeline does not need to run, the scheduler 154 can avoid running the pipeline, for example, when there is no associated cloud layer, there is no need to run the ChunkCloudSpiller operation that allocates too much computing demand to the associated cloud layer. Automatic adjustment of workload priority scheduling is based on the backlog of workloads 174. Certain workloads are only scheduled for SystemHigh priority actions or actions with priorities that meet thresholds. The scheduler 154 prevents the starvation of critical workload algorithms in shard / time slot scheduling. The scheduler 154 can provide reserved time slots for critical pipelines based on the urgency of the pipeline (high, critical). The scheduler may divide and reserve a separate pool of time slots for this dynamic consumption. The scheduler 154 can be configured to allow critical pipelines to exceed the reserved time slots by 10%. The scheduler 154 can limit the number of actions that issue shards by separating the limit from the number of nodes. The scheduler 154 can recalibrate the number of shards of the line scanner based on a configurable fixed data size limit.

[0059] The scheduler 154 can speed up MR stage execution. For example, the scheduler can increase parallel line range scanning based on cluster utilization and / or reduce the admission of new pipelines as cluster utilization increases. The scheduler does not need to globally enforce optimization across multiple services, and the scope can be limited to specific services, pods, clusters, clients and end users, clouds, etc. The scheduler 154 does not need to automatically adjust all acceleration flags, such as gflags or other configurable flags. The scheduler 154 can improve pipeline deadline extension by extending / shortening time based on the quality or type of action, or can adopt an interleaved mode without changing the backlog deadline. The scheduler 154 can change block IO weights or ignore them.

[0060] Architecture Overview: Intelligent job scheduling for pipelines performs systematic and dynamic automatic adjustments in the following areas:

[0061] 1. Prioritize job scheduling.

[0062] 2. Prioritize shard scheduling.

[0063] 3. Prioritize workload deployment actions.

[0064] 4. Automatically adjust workload deployment throttling settings.

[0065] 5. Prioritize the line range scans.

[0066] 6. Prioritize disk IO.

[0067] 7. Extend the assembly line deadline.

[0068] Prioritize job scheduling: The decision to re-prioritize jobs or change gflag settings can be based on demand. The demand can be defined based on the backlog indicator.

[0069] Backlog indicator: A backlog indicator for a workload deployment algorithm may provide the extent of pending work, with a value in the range of zero to one hundred or in the range [0,100], and may be published periodically for each runnable workload executed by the data platform 150. According to various embodiments, the backlog indicator may meet the following requirements: 1. The backlog indicator value may represent the pending work of the algorithm for the workload 174, such as the estimated garbage (e.g., stale data) of the workload that needs to be cleared from the system. 2. In order to be fair to other algorithms, the backlog indicator value may be comparable to other backlog indicators published by other algorithms. 3. The value may be updated periodically to reflect the current state of the backlog and prevent resource starvation when pipelines are not scheduled.

[0070] Optionally, the scheduler may run on a data platform 150, for which the backlog indicator is not implemented for all pipelines with schedulable workloads, and therefore, the backlog indicator may be calculated as a composite value from two parts: specifically, a generic backlog indicator (GBI) 189 and a custom backlog indicator (CBI) 187, where the composite score is defined as a final backlog indicator, which is a weighted sum of the generic backlog indicator 189 and the custom backlog indicator 187, respectively.

[0071] The weighted sum final backlog indicator can be calculated as follows:

[0072] β=w c ·B c +(1-w)·B g ,

[0073] where term B is the backlog indicator, B c is a custom backlog indicator, B g is the general backlog indicator, and w c is the weight of the custom backlog indicator, where the configurable default value is "0.4".

[0074] If a particular pipeline does not implement a custom backlog indicator, the scheduler 154 may still operate to schedule the associated workload, and the custom backlog indicator will set the weight to 0, so that B = B g .

[0075] For the comparability requirement of backlog indicators, the weight of custom backlog indicators is limited so that the generic backlog indicator still provides a baseline for workload algorithms that implement custom backlog indicators.

[0076] Generic backlog indicator: To implement the generic backlog indicator, a generic parameter applicable to all algorithms may be utilized. For example, the generic backlog indicator may be a weighted sum of at least two backlog indicators, including: (1) a run interval backlog indicator, which is a backlog indicator for algorithms that have scheduling delays beyond their expected run interval; and (2) a deadline backlog indicator, which is a backlog indicator for algorithms that have reached deadlines in the last few runs.

[0077] The general backlog indicator can be calculated as follows:

[0078] B g =w d ·B d +(1-w d ).B r ,

[0079] The term B g is the general backlog indicator, B d is the deadline backlog indicator, B r is the run interval backlog indicator, and w d is the weight of the deadline backlog indicator, where the configurable default value is 0.25.

[0080] Run interval backlog indicator: The run interval backlog indicator may represent a measure of the length of time or period of time that a workload 174 algorithm has not been scheduled after the last run of a particular workload algorithm has ended. A workload algorithm is considered backlogged if its execution has not been successfully scheduled and deployed for execution by the scheduler within its configured run interval. Therefore, the longer the workload algorithm has not been scheduled, the scheduler 154 may increase the backlog indicator value, which is calculated as follows:

[0081]

[0082] The term B r is the run interval backlog indicator, T cur is the current timestamp, T last is the uptime for systems where the last run end timestamp or run history is not available, term "R" is the run interval of the pipeline with workload, and term "M" is the run interval multiplier at which the backlog is considered to be 100%.

[0083] In the example formula above, the term "M" represents a parameter that affects how quickly or slowly the algorithm for a workload accumulates a backlog when the workload is not scheduled by the scheduler (relative to the run interval of the pipeline). Generally, for short run interval algorithms, it is preferred to accumulate at a slower rate, while for algorithms scheduled with longer run intervals, it is preferred to accumulate at a faster rate.

[0084] Consider the following diagram:

[0085] Table 1. Algorithm run interval chart

[0086]

[0087] Therefore, an exponential decay function is provided for the multiplier. One such decay function can be expressed as follows:

[0088] f(t)=a.(1-r) t +b; where a>0, b≥0, 0≤r<1.

[0089] By trying different parameter values, it is shown that the values ​​shown in Table 2 can provide a reasonable and ideal fitting effect for the exponential decay function.

[0090] Table 2. Parameters:

[0091]

[0092] Figure 2 is a graph 200 showing the relationship between the multiplier value and the number of seconds between runs, which corresponds to the graph shown in Table 3 below. Figure 2 In the example of , the vertical axis represents the multiplier value (205), and the horizontal axis represents the run interval seconds (210).

[0093] Table 3. Run interval seconds and multipliers:

[0094]

[0095] Deadline Backlog Indicator: The Deadline Backlog Indicator measures the number of consecutive deadlines reached in the last few runs of the pipeline, with a configurable maximum backtracking of five (5) runs. For example, if the most recent run completed without reaching a deadline, then there is no deadline backlog, even if the previous run reached the deadline. In general, the algorithm for a workload that reached a deadline may not have completed its work, and therefore, the Deadline Backlog Indicator is also an indirect measure of the backlog of pending work.

[0096] The deadline backlog indicator can be calculated as follows:

[0097] B d =D*100 / 5,

[0098] The term B d is the deadline backlog indicator, and the term "D" represents the number of consecutive deadlines.

[0099] Custom Backlog Indicators: It is expected that the generic backlog indicator by itself will generally be sufficient to meet the requirements of most workload algorithms, as the generic backlog indicator provides some notion of a backlog of pending work and can adequately prevent resource starvation. However, in some cases, such as for garbage collection operations, custom backlog indicators are implemented that provide configurability to provide additional priority boosts controlled by the actual pending work. Alternatively, the scheduler may consume, reference, or utilize other custom backlog indicators to implement different workload algorithms, such as SnapTreeDedupe, ReplicationFixer, TierRebalancer, ChunkCloudSpiller, and so on. For maintenance operations 146 workload algorithms that perform other tasks such as garbage collection, the custom backlog indicator is configured to publish statistics for its backlog indicator and issue an alarm if its backlog is observed to increase repeatedly over multiple samples. Therefore, the scheduler 154 can reuse the same published custom backlog indicators for the purpose of intelligent scheduling. At a high level, the custom backlog indicator can be configured as a combination of the following backlog indicators: 1. Block GC backlog and 2. Erasure Coding (EC) conversion backlog. Erasure coding (EC for short) uses a mathematical technique to store a piece of data on N disks in such a way that even if a few disks are potentially lost or actually lost (e.g., accidentally erased), it still allows the data to be rebuilt from the remaining disks.

[0100] According to such embodiments, the backlog indicator is calculated by the weighted sum of the garbage collection (GC) backlog indicator and the EC backlog indicator, and then scaled down based on the current utilization %, resulting in a scaled backlog indicator (SBI) 185. For example, if the cluster utilization is 40%, the custom backlog indicator will be scaled down to 140 / 2 = 70%, when the utilization is 80%, then the value is 180 / 2 = 90%, and so on.

[0101] In this example, the custom backlog indicator can be calculated as follows:

[0102]

[0103] The term B c is a custom backlog indicator, B gc is the GC or garbage collection backlog indicator, B ec is the EC backlog indicator, w gc is the weight of the GC backlog indicator, with a configurable default value of 0.75, and "U" is the cluster utilization percentage (%).

[0104] Urgency Scaling of Backlog Indicator: The scheduler 154 may utilize an urgency value, as depicted at element 107. Figure 1A and Figure 1B Urgency 107, shown as being connected to priority manager 184, can be used to enforce stack ordering of algorithms, where the urgency value is statically determined for all pipelines. The urgency of all critical pipelines should be higher than any non-critical pipeline. Some algorithms can implement garbage collection, and the associated urgency value can be automatically adjusted based on cluster utilization. However, the use of dynamic urgency is not unique to garbage collection algorithms. Therefore, scheduler 154 can use urgency values ​​to implement dynamic urgency based on the work backlog of the specific algorithm, pipeline, or workload, regardless of the type of work. For example, scheduler 154 can implement dynamic urgency for other management tasks, overhead work performed on behalf of data platform 150, or end-user specific work that has been associated with sufficient urgency 107 values ​​by data platform 150 or administrators of data platform 150. Similarly, scheduler 154 can use the urgency 107 value of the algorithm to schedule the priority of any running shard under the control or influence of the scheduler. In this example, the urgency value may be additionally utilized to prioritize the scheduling of jobs by including the urgency value as part of the backlog indicator.

[0105] Alternative approaches for the scheduler 154 to incorporate urgency 107 into job scheduling include: (1) configuring the priority manager 184 to add urgency to the generic backlog indicator using a weighted value, where the weight of the urgency is configurable. Caution is required, as too small a weight may result in insufficient variation, while too large a weight may weaken the impact of other backlog indicators; and (2) configuring the scheduler 154 to classify the job run set using urgency to enforce strict stack ordering. This approach may affect algorithms that schedule in descending order of backlog indicator values. In this example, using strict stack ordering may provide an unfair advantage to algorithms that return high urgency values, but may not have sufficient backlog, so careful selection is required.

[0106] According to some example embodiments, the urgency value of the algorithms scheduled for execution by the scheduler 154 may be selected to scale the overall backlog indicator, with the following advantages: (1) the scheduler 154 may preferentially schedule critical algorithms as long as there is a backlog for the critical algorithms; and (2) the scheduler 154 may schedule non-critical algorithms, and if the backlog of the non-critical algorithms increases, these algorithms may still be prioritized, but the threshold for these algorithms to obtain this priority will be higher than that of critical (higher urgency) algorithms.

[0107] A similar scaling scheme can be employed for the backlog indicator for urgency as is used for the garbage collection custom backlog indicator in the context of cluster utilization.

[0108] The scaled backlog indicator (SBI) 185 may be calculated as follows:

[0109] B s =B.(1+U c / U max ) / 2)

[0110] The term B s is the scaling backlog indicator, where U c is the urgency of the workload algorithm, and where U max is the maximum urgency value.

[0111] To limit the impact of scaling, bounded values ​​are defined for urgency, acting as a type of suppression. For example, the following levels are configurable and can be defined as defaults, including: non-urgent (0), low (1), medium (2), high (3), and critical (4).

[0112] Publishing Backlog Indicators: According to certain embodiments, the priority manager 184 may periodically publish backlog indicators for all pipelines that the alarm processor has made accessible to the scheduler 154. The frequency of publishing is different for each pipeline and is a factor of its run interval, with a configurable default value of 4. For example, for a maintenance operation 146 (e.g., garbage collection, etc.), if the run interval is 4 hours, it will be published once every hour. Backlog indicators may be published additionally as work is completed, as the backlog from the last run may have changed (e.g., the backlog of work has decreased), and deadlines may have been met.

[0113] Job Scheduling Using Backlog Indicators: The last published backlog indicator value output by the priority manager 184 can be used by the scheduler 154 to prioritize scheduling in the job run queue. The job run queue can be set up with a custom comparator that can be modified to compare first to the backlog indicator of the pipeline, then to the last start time, etc. In this example, if the algorithm has a backlog, the scheduler 154 will not skip it, even if it has not reached its run interval. In this case, the generic backlog may not have a run interval backlog published by the priority manager 184, but it can still have other types of backlogs. In addition, the scheduler 154 can use custom backlogs to prioritize scheduling of the algorithm in question.

[0114] Prioritizing Slice Scheduling Operations: In various embodiments of intelligent scheduling implemented by the scheduler 154, the backlog may be expanded based on a prioritization metric provided by the priority manager 184 to schedule slices of the workload algorithms being run. For example, the operation of the scheduler 154 will schedule slices based on the backlog indicator of the algorithm issued by the priority manager 184. The operation may combine the backlog indicator with the current run time of the algorithm to penalize algorithms that run for a long time. For slices that are runnable but waiting to be scheduled by the scheduler 154, the priority manager 184 may add another waiting time indicator to increase (actually increase sharply) the priority of the slice while it is waiting in the queue to be picked up by the scheduler 154. Through such operations, the waiting time indicator may be faster or slower according to different algorithms. Start time fair queuing (SFQ) may be used to keep the runnable slices classified according to the above priority scheme. Some algorithms may additionally use reserved time slots, as described above, and the scheduling logic of the scheduler 154 may therefore be configured to honor those reservations.

[0115] Prioritize Scheduler Actions: The action priority of GC, metadata GC, and block cloud overflow actions can be automatically adjusted based on the cluster storage utilization threshold (metadata utilization for snapshot tree actions). However, the scheduler 154 can be configured with the following optional parameters to achieve further improvements in the automatic adjustment of action priorities implemented by the scheduler: 1.) The threshold for high priority actions can be configured to 75% by default, and the critical priority actions are configured to 85% by default. However, configuring the lower threshold for high priority to 70% provides additional space between high action priority and critical action priority. 2.) For GC actions, the scheduler can be configured to issue a high priority action when the backlog indicator value meets or exceeds a configurable threshold.

[0116] Similar to the use of backlog indicators, the quantification of the quality or type of actions can be a normalized value between [0,100]. Certain embodiments refer to the quantification of the quality or type of actions as a "utility score" and can be processed similarly to the maintenance operations 146 (e.g., garbage collection, etc.) workload to classify actions. Using this configuration: 1.) The host can periodically query the utility score from the running shards, perhaps through QuerySlavesOp. 2.) The shard can calculate the utility score as a moving average of the internal utility scores of the set of actions currently issued. For example, the maintenance operations 146 workload classifies actions based on how much data will be garbage collected, how many sessions will be cleaned, how many data instances will be replicated, and / or how many storage nodes will be cloned. 3.) The utility score can also be adjusted based on the % of kCASErrors encountered. 4.) If the utility score drops below certain thresholds, the host can expire the job based on a deadline being reached. Utility scores may be implemented based on deadlines for specific maintenance operations 146 pipelines / workloads only (such as garbage collection only), while other pipelines / workloads configured to use a generic or default implementation based on action_exec_time_min_pct will expire jobs after 50% of the actions have been executed when configured using the default values. Other jobs may be configured with custom parameters that differ from the default values ​​at any time.

[0117] Implementation Alternatives: 1.) In various examples, the configuration uses a single number called a backlog indicator to represent the priority of a workload, pipeline, or algorithm, which can be calculated using multiple sub-indicators; however, alternative implementations may output or expose the underlying sub-indicators for further fine-grained customization as needed. 2.) In various examples, custom and generic backlog indicators are provided, where the generic indicator provides a common base comparison and fairness between algorithms, while the custom indicator is customizable to fit the specific needs of a workload or pipeline and represents actual pending work within a workload or pipeline algorithm. 3.) In various examples, the configuration disables pipelines based on cluster utilization to simulate settings similar to aggressive GC gflags. However, caution is required when using this approach, as it may keep pipelines disabled indefinitely if cluster utilization never sufficiently decreases below a threshold. Without management support from the data platform, the scheduler may not be able to address end-user concerns about the adjustments needed for their environment if the underlying issue is a 29izeng issue, stale snapshot footprint, or incorrect retention policy, etc. 4.) Using only custom backlogs to determine algorithm priorities and automatically adjusting gflags is another implementation option. However, this approach may not provide sufficient fairness in job prioritization and may cause resource starvation for other pipelines. 5.) Extending / shortening pipeline deadlines based on the quality or type of actions is another implementation option. However, doing so requires collecting additional information from the emission slices of the running pipeline actions, which increases the complexity and computational overhead of the data platform 150. In some instances, the priority of this feature decreases with staggered running of pipelines, because staggered running allows the MR stage of the next pipeline run to be scheduled in parallel with the emission of the previous run of actions.

[0118] Scheduler Indicator Details: There are two different backlog indicator implementations within the scheduler: 1.) Backlog Indicator for throttling; and 2.) Backlog Indicator for alerting.

[0119] Backlog indicator for throttling: This backlog indicator is implemented to avoid limiting the speed of the scheduler when any pipeline has a backlog. The indicator can be a Boolean value indicating whether a pipeline has a backlog.

[0120] Backlog indicators for alerting: The master publishes backlog indicators for different pipelines. The master also generates alerts based on the backlog indicators published for the pipelines. The process looks for positive slopes in the backlog indicator graph. If the number of positive slopes exceeds 70% in the total samples, the process raises an alert identifying the backlog algorithm. The backlog indicator value can be scaled based on the current cluster usage, for example, the same backlog indicator value can be increased or decreased based on the current cluster utilization.

[0121] Merging Backlog Indicators: Optionally, rather than maintaining two different implementations of the backlog indicators as discussed above, the process may allow them to be moved to one place and possibly merged.

[0122]

[00136] General Backlog Indicator: The scheduler 154 or priority manager 184 may implement a general backlog indicator based on the average run interval of a pipeline exceeding a configured run interval. The process measures the run interval from the start time of the last run to the current timestamp, however, for large clusters where pipeline run times are longer than the run interval, an alternative configuration may be selected. The average run time may be calculated as the difference between the start time of the current run and the end time of the last run, which represents quiet time when the pipeline is not running. This configuration should avoid marking a pipeline as backlogged when the pipeline run time exceeds the configured run interval value. For runs that reach their deadlines, the average run time provides an indication that the pipeline has reached a backlog.

[0123] There are two ways that reaching a deadline can affect the backlog indicator, and an implementation can select one or both of these ways to indicate a backlog. For runs that fail to be scheduled and result in runs reaching a deadline, measuring the average run interval from the last successful run will cause the average run interval to be displayed as an increased value, thereby increasing the backlog indicator value. Alternatively or in addition, the process can add an explicit weighted value for the expiration of the deadline to the backlog indicator value. Further enhance the general indicator, taking the duration of the pipeline into account to avoid situations where pipeline delays with short run intervals do not cause the scheduler to accelerate itself when it indicates a backlog. This can be configured by scaling the backlog indicator according to the pipeline run interval. For the weighted combination of the backlog indicator values, another weighted boost can be applied to mission-critical pipelines so that the backlog indicator values ​​of these mission-critical pipelines increase faster than non-mission-critical pipelines. Another configurable indicator is the number of actions issued by the corresponding pipeline in a continuous run or the time spent in the action execution phase. When the numbers for this metric are increasing, it indicates that the backlog is increasing because more work is being done on the pipeline in question.

[0124] Garbage Collection Backlog Indicator: For garbage collection specific workload implementations, the scheduler 154 may measure the backlog as an increase in the storage reduction rate, and may optionally perform it on the garbage collection pipeline. The priority manager 184 may calculate the difference between the last storage reduction rate and the average reduction rate as a percentage value, and may additionally map this backlog indicator value in the range [0,100]. The garbage collection implementation of the backlog indicator calculates an average weight of different types of indicators, such as block GC backlog and EC backlog. By adding new types of indicators to this weighted average, the weights can be adjusted using the gflag for each type of indicator. The encoded sensor provides an estimate of the amount of garbage emitted by the previous garbage collection run, which is calculated by performing a reachability analysis on all blocks present in the cluster.

[0125] Combining different types of indicators into a weighted average can be challenging, and sometimes data for one or more indicators may not be readily available, for example, the GPSR pipeline may not have run between runs of the garbage collection pipeline and workload. Those unavailable indicators may be excluded, and then the processing will distribute the weighted average among the available indicators. However, if the value of the combination varies depending on the presence or absence of an indicator, this may create undesirable fluctuations in the results, which may further lead to false alarms on positive slopes of the backlog indicator.

[0126] To avoid this, the following enhanced configuration can be used: When all component indicator data are available, the process will publish the first backlog indicator value. In further calculations, when one of the component indicators is missing, the historical value of the missing indicator can be used instead, while publishing the component indicator value in the statistics along with the overall indicator. This process can help administrators and developers debug which component is causing the backlog when performance anomalies occur.

[0127] Smart Pipeline Scheduling: In addition to the current uses of the backlog indicator described above, the backlog indicator value can also be used as an input to the pipeline scheduler to prioritize pipelines based on their backlog indicator values. By default, the master job scheduler selects pipelines from a categorized set of job runs that holds jobs in categorized order of the least recently run pipeline. This logic provides fairness to the algorithm and avoids resource starvation. The job run comparator can be further enhanced to retrieve the backlog indicator value of the pipeline. Based on the retrieved value, the scheduler can then prioritize pipelines with higher backlog indicator values. Scheduling based solely on backlog may starve other pipelines that have not run for extended periods of time. However, with the generic backlog indicator, the scheduler already takes into account how long the workload or pipeline in question has been overdue for the next periodic run, which will eventually provide a catch-up for the resource-starved pipelines. Regardless, this logic requires fine-tuning through multiple iterations to optimize the parameters.

[0128] In this manner, the processing performed by the data platform 150 implements the following operations, including: obtaining, by the data platform, a generic backlog indicator for a plurality of workloads to be executed by the data platform, and obtaining, by the data platform, a customized backlog indicator for at least a subset of the plurality of workloads to be executed by the data platform. According to this example, for each of the workloads enqueued for execution by the data platform, the data platform further applies a weight to each of the generic backlog indicator and the customized backlog indicator for the corresponding workload to obtain a single weighted backlog indicator value. The scheduler of the data platform can then proceed to schedule the workload for execution on the data platform based on the single weighted backlog indicator value corresponding to each respective workload.

[0129] In another example, the data platform can further determine an urgency value for each respective workload and then scale the single weighted backlog indicator value based on the determined urgency value to obtain a scaled backlog indicator value for each respective workload. In this example, scheduling the workload for execution on the data platform can be based on the scaled backlog indicator value for each respective workload.

[0130] In another example, the computing system may further apply a configurable multiplier to the single weighted backlog indicator value to determine a run interval for the general backlog indicator within a configurable range. In such an example, the run interval represents how fast or slow the corresponding workload accumulates a backlog while remaining in an unscheduled state.

[0131] Still further, in another example, the general backlog indicator may include a measure of time that an algorithm associated with a workload has not been scheduled after a previous execution of the same workload ended.

[0132] In yet another example, a custom backlog indicator may include a measure of utilization of a specified one or more resources within the data platform that are affected by execution of an algorithm associated with the workload.

[0133] According to the above example, the data platform 150 may perform operations including obtaining, at the data platform 150, a generic backlog indicator or “GBI” 189 for the workloads 174 to be executed by the data platform 150. In addition, the data platform 150 may obtain a customized backlog indicator or “CBI” 187 for at least a subset of the workloads 174 to be executed by the data platform 150. The priority manager 184 of the data platform 150 may perform operations for calculating a single weighted backlog indicator value for each of the workloads 174 to be executed by the data platform based on various policies 158 of the scheduler 154. For example, the scheduler 154 may define a policy 158 that causes the priority manager 184 to apply a configurable weight to each of the generic backlog indicator 189 and the customized backlog indicator 187 for a corresponding one of the workloads 174. The scheduler 154 may then continue to schedule workloads 174 for execution on the data platform 150 based on a single weighted backlog indicator value corresponding to each respective workload, either responsively or based on the calculated generic backlog indicator 189 and custom backlog indicator 187 determined by the priority manager 184 .

[0134] Figure 3 is a block diagram illustrating an example system 300 in accordance with techniques of this disclosure. Figure 3 The system 300 can be described as Figure 1A System 100 or Figure 1B An example or alternative implementation of a system 190 of (wherein the scheduler 154 performs intelligent job scheduling that affects the pipeline of the local archive storage system 115). Figure 1A and Figure 1B Described in the context of Figure 3 one or more aspects of .

[0135] exist Figure 3 In the example of , system 300 includes network 111, data platform 150 implemented by computing system 302, and storage system 115. Figure 3 In the example, the network 111, the data platform 150 and the storage system 115 may correspond to Figure 1A 1 , a data platform 150, and a storage system 115. Although only one archival storage system 115 is depicted, the data platform 150 may apply techniques according to the present disclosure using multiple instances of the archival storage system 115. Different instances of the storage system 115 may be deployed by different cloud storage providers, the same cloud storage provider, an enterprise, or other entities.

[0136] The computing system 302 may be implemented as any suitable computing system, such as one or more server computers, workstations, mainframes, devices, cloud computing systems, and / or other computing systems capable of performing operations and / or functions described according to one or more aspects of the present disclosure. In some examples, the computing system 302 represents a cloud computing system, a server farm, and / or a server cluster (or a portion thereof) that provides services to other devices or systems. In other examples, the computing system 302 may represent or be implemented by one or more virtualized computing instances (e.g., virtual machines, containers) of a cloud computing system, a server farm, a data center, and / or a server cluster.

[0137] exist Figure 3 In the example of , computing system 302 may include one or more communication units 315, one or more input devices 317, one or more output devices 318, and one or more storage devices of local storage system 105. Local storage system 105 includes interface module 326, file system manager 152, archive manager 154, policy 158, archive metadata 322, and block metadata 320. One or more devices, modules, storage areas, or other components of computing system 302 may be interconnected to enable inter-component communication (physically, communicatively, and / or operationally). In some examples, such connection may be provided by a communication channel (e.g., communication channel 312), which may represent one or more of a system bus, a network connection, an inter-process communication data structure, or any other method for transferring data.

[0138] The computing system 302 includes a processing device. Figure 3 In the example of FIG. 1 , the processing device includes one or more processors 313, wherein the one or more processors are configured to implement functions associated with the computing system 302 and / or perform operations associated with the computing system 302. Figure 3 , and described below. The one or more processors 313 may be, may be a part of, and / or may include processing circuitry that performs operations in accordance with one or more aspects of the present disclosure. Examples of processor 313 include a microprocessor, an application processor, a display controller, an auxiliary processor, one or more sensor hubs, and any other hardware configured to function as a processor, processing unit, or processing device. The computing system 302 may use one or more processors 313 to perform operations in accordance with one or more aspects of the present disclosure using software, hardware, firmware, or a mixture of hardware, software, and firmware resident in and / or executed at the computing system 302.

[0139] One or more communication units 315 of computing system 302 may communicate with devices external to computing system 302 by transmitting and / or receiving data, and in some aspects may operate as both an input device and an output device. In some examples, communication unit 315 may communicate with other devices over a network. In other examples, communication unit 315 may send and / or receive radio signals over a radio network (such as a cellular radio network). In other examples, communication unit 315 of computing system 302 may transmit and / or receive satellite signals over a satellite network. Examples of communication unit 315 include a network interface card (e.g., such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and / or receive information. Other examples of communication unit 315 may include a device that is capable of transmitting information over a network. GPS, NFC, and cellular networks (e.g., 3G, 4G, 5G) and in mobile devices devices that communicate with other devices such as radios and Universal Serial Bus (USB) controllers. Such communications may comply with, implement, or follow appropriate protocols, including Transmission Control Protocol / Internet Protocol (TCP / IP), Ethernet, NFC or other technologies or protocols.

[0140] One or more input devices 317 may represent any input device of computing system 302 not separately described herein. Input device 317 may obtain, generate, receive, and / or process input. For example, one or more input devices 317 may generate or receive input from a network, a user input device, or any other type of device for detecting input from a human or machine.

[0141] One or more output devices 318 may represent any output device of computing system 302 not described separately herein. Output device 318 may generate, present and / or process output. For example, one or more output devices 318 may generate, present and / or process output in any form. Output device 318 may include one or more USB interfaces, video and / or audio output interfaces, or any other type of device capable of generating tactile, audio, visual, video, electrical or other output. Some devices may be used as both input devices and output devices. For example, a communication device may send data to other systems or devices via a network and receive data therefrom.

[0142] One or more storage devices of the local storage system 105 within the computing system 302 may store information for processing during operation of the computing system 302, such as random access memory (RAM), flash memory, solid state disk (SSD), hard disk drive (HDD), etc. The storage device may store program instructions and / or data associated with one or more modules described according to one or more aspects of the present disclosure. One or more processors 313 and one or more storage devices may provide an operating environment or platform for such modules, which may be implemented as software, but may include any combination of hardware, firmware, and software in some examples. One or more processors 313 may execute instructions, and one or more storage devices of the storage system 105 may store instructions and / or data of one or more modules. The combination of the processor 313 and the local storage system 105 may retrieve, store, and / or execute instructions and / or data of one or more applications, modules, or software. The processor 313 and / or the storage device of the local storage system 105 may also be operably coupled to one or more other software and / or hardware components, including but not limited to one or more components of the computing system 302 and / or one or more devices or systems shown as being connected to the computing system 302.

[0143] The file system manager 152 may perform functions related to providing the file system 153, as described above with respect to Figure 1A The file system manager 152 may generate and manage file system metadata 332 to construct file system data 330 of the file system 153, and store the file system metadata 332 and the file system data 330 to the local storage system 105. The file system metadata 332 may include one or more trees describing objects within the file system 153 and the hierarchical structure of the file system 153, and may be used to write or retrieve objects within the file system 153. The file system metadata 332 may be referenced by the scheduler 154 to support intelligent job scheduling of the execution pipeline, and vice versa. The file system manager 152 may interact and / or operate in coordination with one or more modules of the computing system 302, including the interface module 326 and the scheduler 154.

[0144] Scheduler 154 may perform archival functions related to scheduling workloads 174, tasks, algorithms, and pipelines, as described above with respect to Figure 1A The above includes operations described above with respect to coordination with the priority manager 184. The scheduler 154 may reference, update, and manage the archive metadata 322 for coordinating garbage collection algorithms and other operations affecting the storage system 115.

[0145] The interface module 326 may implement an interface through which other systems or devices may determine the operation of the file system manager 152 or the archive manager 154. Another system or device may communicate through the interface of the interface module 326 to specify one or more policies 158.

[0146] System 300 can be modified to implement Figure 1B In the modified system 300 , the workload 174 may perform a garbage collection routine on the local archive storage system 115 or the archive 142 .

[0147] Figure 4 is a flow chart illustrating an example mode of operation of a data platform 150 to perform intelligent job scheduling of workloads based on the use of custom and generic backlog indicators associated with enqueued workloads in accordance with the techniques of this disclosure. Figure 1A System 100, Figure 1B System 190 and Figure 3 The operating mode is described with reference to the computing system 302 and the storage systems 105 , 115 .

[0148] The data platform 150 may obtain a generic backlog indicator for each workload to be executed (405). For example, a processing device (e.g., processing circuitry) of the data platform 150 may obtain a generic backlog indicator for a plurality of workloads to be executed by the data platform. In such an example, each of the plurality of workloads may specify one or more storage system maintenance operations for one or more storage systems managed by the data platform. The data platform 150 may obtain a customized backlog indicator for at least a portion of the workloads (410). For example, the processing device may obtain a customized backlog indicator for at least a subset of the plurality of workloads to be executed by the data platform.

[0149] The data platform 150 may calculate a single weighted backlog indicator value for each workload to be executed (415). For example, a processing device of the data platform 150 may calculate a single weighted backlog indicator value for each of a plurality of workloads to be executed by the data platform. In such an example, the processing device may calculate each single weighted backlog indicator value by applying a configurable weight to each of a generic backlog indicator and a custom backlog indicator for a corresponding workload from the plurality of workloads. The data platform 150 may schedule the workloads using the single weighted backlog indicator value for each workload to be executed (420). For example, the processing device may schedule the plurality of workloads for execution on the data platform based on the single weighted backlog indicator value calculated for each of the plurality of workloads. The data platform 150 may process the workloads according to the schedule (425). For example, the data platform 150 may process the workloads according to the schedule in descending order of the single weighted backlog indicator values.

[0150] In some examples, one or more storage system maintenance operations include a garbage removal operation that removes data from one or more storage systems in response to determining that the data platform has identified the data as available for removal from one or more storage systems managed by the data platform. In some examples, one or more storage system maintenance operations include a data replication redundancy operation that replicates a data instance to create a new operational replica of the data instance in response to determining that the count of an operational replica of the data instance is below a configured redundancy minimum. In some examples, one or more storage system maintenance operations include a storage node recovery operation that clones a data instance to a new storage node in response to determining that a storage node failure within one or more storage systems causes the count of operational data instances to be below a configured redundancy minimum. In some examples, one or more storage system maintenance operations include a server message block (SMB) session removal operation that removes one or more SMB sessions from one or more storage systems in response to determining that the data platform has identified the one or more SMB sessions as available for removal from one or more storage systems managed by the data platform.

[0151] The processing device may configure the data platform 150 to perform further operations, including obtaining an urgency value for each of the plurality of workloads. The processing device may calculate a scaled backlog indicator value for each respective workload in the plurality of workloads by scaling a single weighted backlog indicator value corresponding to each respective workload in the plurality of workloads based on the urgency value obtained for the respective workload from the plurality of workloads. In such an example, the processing device may schedule the plurality of workloads for execution using the scaled backlog indicator value for each respective workload in the plurality of workloads, or the processing device may schedule the plurality of workloads for execution using the scaled backlog indicator value for each respective workload in the plurality of workloads.

[0152] In some examples, obtaining an urgency value for each respective workload in a plurality of workloads includes: configuring a first portion of the plurality of workloads as critical workloads; configuring a second portion of the plurality of workloads as non-critical workloads; and enforcing stack ordering of the plurality of workloads by requiring each critical workload to be configured with an urgency value that is higher than any urgency value associated with any non-critical workload.

[0153] In some instances, data platform 150 can apply a configurable multiplier to a single weighted backlog indicator value to determine a run interval for a general backlog indicator within a configurable range. In some examples, the run interval represents how quickly or slowly a corresponding workload accumulates a backlog while remaining in an unscheduled state.

[0154] In some examples, the general backlog indicator includes a measure of a time that each respective workload from the plurality of workloads remains in an unscheduled state after the respective workload from the plurality of workloads finishes a previous execution. In some examples, the custom backlog indicator includes a measure of utilization of specified one or more resources within the data platform that are affected by the data platform executing the respective workload from the plurality of workloads.

[0155] In some examples, each respective workload from a plurality of workloads embodies at least one of the following: an algorithm to be scheduled for execution through a data platform; a periodic algorithm to be scheduled for execution through a data platform using a repetitive schedule; an iterative algorithm to be scheduled for execution through a data platform using a configurable frequency; a process that specifies one or more executable tasks to be executed through a data platform; a job that specifies one or more executable tasks to be executed through a data platform; an overhead task that specifies one or more executable tasks to be executed through a data platform, the overhead task not being uniquely associated with any end user of the data platform; a temporary end-user task that specifies one or more executable tasks to be executed through a data platform on behalf of or at the request of a uniquely specified end user of the data platform; an optimization routine that specifies one or more executable tasks to be executed through a data platform, the optimization routine being configured to improve the operational efficiency of the data platform; and a garbage collection routine that is configured to increase the available storage space of the data platform at least by removing data that has been identified by the data platform as available for removal from one or more storage systems managed by the data platform.

[0156] In at least one example, obtaining a common backlog indicator for a plurality of workloads to be executed by a data platform includes calculating a common backlog indicator for each of the plurality of workloads based on one or more of: a configurable urgency value for each respective workload in the plurality of workloads; a backlog indicator value for each respective workload in the plurality of workloads, the backlog indicator value representing a time period during which the respective workload from the plurality of workloads failed to initiate execution at the data platform after exceeding an initially scheduled execution time of the respective workload from the plurality of workloads; a quiet time value for each respective workload in the plurality of workloads, the quiet time value representing a time period during which the respective workload from the plurality of workloads was not executed between repeated executions of the respective workload from the plurality of workloads; a workload deadline value for each respective workload in the plurality of workloads, the workload deadline value indicating that the respective workload from the plurality of workloads encountered a configurable urgency value. a workload deadline count for each respective workload from the plurality of workloads, the workload deadline count indicating the number of times the respective workload from the plurality of workloads encountered a configurable deadline within a configurable historical time period without completing execution at the data platform, or a historical number of execution attempts of the respective workload from the plurality of workloads; an issue action count for each respective workload from the plurality of workloads, the issue action count indicating the number of outputs of executable tasks executed to completion by the data platform as part of each respective workload from the plurality of workloads; a total issue time based on each workload, the total issue time based on each workload being calculated for each respective workload from the plurality of workloads to indicate the total amount of processing time required to fully complete one or more executable tasks executed by the data platform as part of the respective workload from the plurality of workloads. In some examples, the data platform 150 may store the general backlog indicator calculated for each of the plurality of workloads as being uniquely associated with the respective workload from the plurality of workloads for which the general backlog indicator was calculated.

[0157] In some examples, obtaining a customized backlog indicator for a plurality of workloads to be executed by the data platform includes calculating a customized backlog indicator for at least a subset of the plurality of workloads based on one or more of: a workload-specific total utilization value for a corresponding workload from the plurality of workloads to be scheduled; a workload-specific total estimated garbage value for a corresponding workload from the plurality of workloads to be scheduled; an end-user activated user interface action or button indicating that the data platform has received input associated with a request to apply an aggressive garbage collection operation at the data platform; and a request for a corresponding workload from the plurality of workloads to be scheduled. an end-user-specific upgrade request by an end-user; a service tier corresponding to the end-user associated with the corresponding workload from the plurality of workloads to be scheduled; a total amount of pending workloads corresponding to the end-user associated with the corresponding workload from the plurality of workloads to be scheduled; a cluster utilization value corresponding to the cluster utilized by the end-user associated with the corresponding workload from the plurality of workloads to be scheduled; an SLA issued for the corresponding workload from the plurality of workloads to be scheduled; and storing a custom backlog indicator calculated for each of the plurality of workloads as uniquely associated with the corresponding workload from the plurality of workloads for which the custom backlog indicator is calculated.

[0158] Although the techniques described in this disclosure are primarily described with respect to the archive functions performed by the archive manager of the data platform, similar techniques may additionally or alternatively be applied to the backup, replication, cloning, and / or snapshot functions performed by the data platform. In such cases, the archive 142 would be a backup, replica, clone, or snapshot, respectively.

[0159] For the processes, devices, and other examples or illustrations described herein, including in any flow charts or flowcharts, certain operations, actions, steps, or events included in any technology described herein may be performed in a different order, may be added, merged, or omitted entirely (e.g., not all described actions or events are necessary for the practice of the technology). In addition, in some examples, operations, actions, steps, or events may be performed simultaneously, such as by multithreading, interrupt processing, or multiple processors, rather than in sequence. In addition, even if not explicitly identified as automatically executed, certain operations, actions, steps, or events may also be automatically executed. In addition, certain operations, actions, steps, or events described as automatically executed may alternatively not be automatically executed, but in some examples, such operations, actions, steps, or events may be executed in response to an input or another event.

[0160] The detailed description set forth herein in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configuration in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a comprehensive understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some cases, well-known structures and components are shown in block diagram form in order to avoid obscuring such concepts.

[0161] According to one or more aspects of the present disclosure, when the context does not otherwise dictate, the term "or" may be interpreted as "and / or". In addition, although phrases such as "one or more" or "at least one" may be used in some cases, they are not used in other cases; if the context does not dictate otherwise, they may be interpreted as having such an implicit meaning in the absence of such language.

[0162] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium, including any medium that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transient, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0163] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, optical fiber cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, then the coaxial cable, optical fiber cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave) is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transient media, but are directed to non-transient tangible storage media. Disks and optical disks such as those used include compact discs (CDs), laser optical disks, optical disks, digital versatile discs (DVDs), floppy disks and blue-ray discs, wherein disks usually reproduce data magnetically, and optical discs reproduce data optically with lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0164] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" or "processing circuitry" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementing the described techniques, respectively. In addition, in some examples, the described functionality may be provided within dedicated hardware and / or software modules. In addition, the described techniques may be fully implemented in one or more circuits or logic elements.

[0165] As used herein, a processing device may include processing circuitry as described above. In some examples, a processing device may include at least one processor and at least one memory having computer code, the computer code including a set of instructions that, when executed by the at least one processor, causes the at least one processor to perform any of the functions described herein. In some examples, the processing device may receive computer code including a set of instructions from at least one memory coupled to the processing device.

[0166] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, mobile or non-mobile computing devices, wearable or non-wearable computing devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in hardware units or provided by a set of interoperable hardware units (including one or more processors as described above) in combination with appropriate software and / or firmware.

[0167] Aspects of the present invention are provided with reference to the following clauses:

[0168] Item 1. A method comprising: obtaining, by processing circuitry of a data platform, a general backlog indicator for a plurality of workloads to be executed by the data platform, wherein each of the plurality of workloads specifies one or more storage system maintenance operations for one or more storage systems managed by the data platform; obtaining, by the processing circuitry of the data platform, a customized backlog indicator for at least a subset of the plurality of workloads to be executed by the data platform; calculating, by the processing circuitry using a priority manager, a single weighted backlog indicator value for each of the plurality of workloads to be executed by the data platform by applying configurable weights to the general backlog indicator and each of the customized backlog indicators for a corresponding workload from the plurality of workloads; scheduling, by the processing circuitry using a scheduler, the plurality of workloads for execution on the data platform based on the single weighted backlog indicator value calculated for each of the plurality of workloads; and processing, by the processing circuitry of the data platform, the plurality of workloads according to the schedule.

[0169] Clause 2. A method as described in Clause 1, wherein the one or more storage system maintenance operations include at least one of the following: a garbage removal operation, which removes the data from the one or more storage systems managed by the data platform in response to determining that the data platform has identified the data as available for removal from the one or more storage systems managed by the data platform; a data replication redundancy operation, which replicates the data instance to create a new operational replica of the data instance in response to determining that the count of the operational replica of the data instance is lower than the configured redundancy minimum; a storage node recovery operation, which clones the data instance to a new storage node in response to determining that a storage node failure within the one or more storage systems causes the count of the operational data instance to be lower than the configured redundancy minimum; and a server message block (SMB) session removal operation, which removes the one or more SMB sessions from the one or more storage systems in response to determining that the data platform has identified the one or more SMB sessions as available for removal from the one or more storage systems managed by the data platform.

[0170] Clause 3. A method as described in Clause 1, further comprising: obtaining an urgency value for each of the multiple workloads; calculating a scaled backlog indicator value for each corresponding workload in the multiple workloads by scaling the single weighted backlog indicator value corresponding to each corresponding workload in the multiple workloads based on the urgency value obtained for the corresponding workload from the multiple workloads; and wherein scheduling the multiple workloads for execution on the data platform based on the single weighted backlog indicator value corresponding to each corresponding workload in the multiple workloads includes one of the following: scheduling the multiple workloads for execution in descending order according to the single weighted backlog indicator of each corresponding workload in the multiple workloads, or scheduling the multiple workloads for execution using the scaled backlog indicator value of each corresponding workload in the multiple workloads.

[0171] Clause 4. A method as described in Clause 3, wherein obtaining the urgency value for each corresponding workload in the multiple workloads includes: configuring a first portion of the multiple workloads as critical workloads; configuring a second portion of the multiple workloads as non-critical workloads; and forcing stack sorting of the multiple workloads by requiring each of the critical workloads to be configured with an urgency value that is higher than any urgency value associated with any of the non-critical workloads.

[0172] Clause 5. A method as described in Clause 1, further comprising: applying a configurable multiplier to the single weighted backlog indicator value to determine an operating interval of the general backlog indicator within a configurable range; and wherein the operating interval represents how quickly or slowly the corresponding workload accumulates a backlog while remaining in an unscheduled state.

[0173] Clause 6. The method of clause 1, wherein the general backlog indicator comprises a measure of a time that each respective workload from the plurality of workloads remains in an unscheduled state after the respective workload from the plurality of workloads finishes a previous execution.

[0174] Clause 7. The method of clause 1, wherein the customized backlog indicator comprises a measure of utilization of specified one or more resources within the data platform affected by execution of the corresponding workload from the plurality of workloads by the data platform.

[0175] Clause 8. A method as described in clause 1, wherein each corresponding workload from the multiple workloads embodies at least one of the following: an algorithm to be scheduled for execution through the data platform; a periodic algorithm to be scheduled using repeated scheduling for execution through the data platform; an iterative algorithm to be scheduled using a configurable frequency for execution through the data platform; a process that specifies one or more executable tasks to be executed through the data platform; a job that specifies the one or more executable tasks to be executed through the data platform; an overhead task that specifies the one or more executable tasks to be executed through the data platform, the overhead task not being uniquely associated with any customer of the data platform; a temporary customer task that specifies the one or more executable tasks to be executed through the data platform on behalf of or at the request of a uniquely specified customer of the data platform; an optimization routine that specifies the one or more executable tasks to be executed through the data platform, the optimization routine being configured to improve the operating efficiency of the data platform; and a garbage collection routine that is configured to increase the available storage space of the data platform at least by removing data that has been identified by the data platform as available for removal from one or more storage systems managed by the data platform.

[0176] Clause 9. A method as described in clause 1, wherein obtaining the common backlog indicator of the multiple workloads to be executed by the data platform includes: calculating the common backlog indicator of each of the multiple workloads based on one or more of the following: a configurable urgency value of each corresponding workload in the multiple workloads; a backlog indicator value of each corresponding workload in the multiple workloads, the backlog indicator value indicating a time period during which the corresponding workload from the multiple workloads failed to initiate execution at the data platform after exceeding an initially scheduled execution time of the corresponding workload from the multiple workloads; a quiet time value of each corresponding workload in the multiple workloads, the quiet time value indicating a time period during which the corresponding workload from the multiple workloads was not executed between repeated executions of the corresponding workload from the multiple workloads; a workload deadline value of each corresponding workload in the multiple workloads, the workload deadline value indicating that the corresponding workload from the multiple workloads encountered a configurable deadline and did not complete execution at the data platform; a workload deadline count for each respective workload in the workloads, the workload deadline count indicating the number of times the respective workload from the plurality of workloads encountered a configurable deadline within a configurable historical time period without completing execution at the data platform, or a historical number of execution attempts of the respective workload from the plurality of workloads; an issue action count for each respective workload in the plurality of workloads, the issue action count indicating the number of outputs of executable tasks executed to completion by the data platform as part of each respective workload from the plurality of workloads; a total issue time based on each workload, the total issue time based on each workload being calculated for each respective workload in the plurality of workloads to indicate the total amount of processing time required to fully complete one or more executable tasks executed by the data platform as part of the respective workload from the plurality of workloads; and storing the general backlog indicator calculated for each of the plurality of workloads as being uniquely associated with the respective workload from the plurality of workloads for which the general backlog indicator was calculated.

[0177] Clause 10. A method as described in clause 1, wherein obtaining the customized backlog indicator for the multiple workloads to be executed by the data platform includes: calculating the customized backlog indicator for at least a subset of the multiple workloads based on one or more of the following: a workload-specific total utilization value for the corresponding workload from the multiple workloads to be scheduled; a workload-specific total estimated garbage value for the corresponding workload from the multiple workloads to be scheduled; a customer-activated user interface action or button, the customer-activated user interface action or button indicating that the data platform has received input associated with a request to apply an aggressive garbage collection operation at the data platform; and a request for a corresponding workload from the multiple workloads to be scheduled. a customer-specific upgrade request of the customer associated with the respective workload from the plurality of workloads to be scheduled; a service tier corresponding to the customer associated with the respective workload from the plurality of workloads to be scheduled; a total amount of pending workloads corresponding to the customer associated with the respective workload from the plurality of workloads to be scheduled; a cluster utilization value corresponding to a cluster utilized by the customer associated with the respective workload from the plurality of workloads to be scheduled; an SLA issued for the respective workload from the plurality of workloads to be scheduled; and storing the custom backlog indicator calculated for each of the plurality of workloads as uniquely associated with the respective workload from the plurality of workloads for which the custom backlog indicator is calculated.

[0178] Item 11. A computing system comprising: a priority manager; a scheduler; a storage device; and a processing circuit capable of accessing the storage device and configured to: obtain, by the processing circuit, a general backlog indicator for a plurality of workloads to be executed by the data platform, wherein each of the plurality of workloads specifies one or more storage system maintenance operations for one or more storage systems managed by the data platform; obtain, by the processing circuit, a customized backlog indicator for at least a subset of the plurality of workloads to be executed by the data platform; calculate, by the processing circuit, using the priority manager a single weighted backlog indicator value for each of the plurality of workloads to be executed by the data platform by applying a configurable weight to the general backlog indicator and each of the customized backlog indicators for a corresponding workload from the plurality of workloads; schedule, by the processing circuit, using the scheduler to schedule the plurality of workloads for execution on the data platform based on the single weighted backlog indicator value calculated for each of the plurality of workloads; and process, by the processing circuit, the plurality of workloads according to the schedule.

[0179] Clause 12. A method as described in Clause 11, wherein the one or more storage system maintenance operations include at least one of the following: a garbage removal operation, which removes the data from the one or more storage systems managed by the data platform in response to determining that the data platform has identified the data as available for removal from the one or more storage systems managed by the data platform; a data replication redundancy operation, which replicates the data instance to create a new operational replica of the data instance in response to determining that the count of the operational replica of the data instance is lower than the configured redundancy minimum; a storage node recovery operation, which clones the data instance to a new storage node in response to determining that a storage node failure within the one or more storage systems causes the count of the operational data instance to be lower than the configured redundancy minimum; and a server message block (SMB) session removal operation, which removes the one or more SMB sessions from the one or more storage systems in response to determining that the data platform has identified the one or more SMB sessions as available for removal from the one or more storage systems managed by the data platform.

[0180] Clause 13. A computing system as described in Clause 11, wherein the processing circuit is further configured to: obtain an urgency value for each of the multiple workloads; calculate a scaled backlog indicator value for each corresponding workload in the multiple workloads by scaling the single weighted backlog indicator value corresponding to each corresponding workload in the multiple workloads based on the urgency value obtained for the corresponding workload from the multiple workloads; and wherein the computing system is used to: schedule the multiple workloads for execution in descending order according to the single weighted backlog indicator of each corresponding workload in the multiple workloads, or schedule the multiple workloads for execution using the scaled backlog indicator value of each corresponding workload in the multiple workloads.

[0181] Clause 14. A computing system as described in Clause 13, wherein the computing system is used to obtain the urgency value for each corresponding workload in the multiple workloads, including the computing system being used to: configure a first portion of the multiple workloads as critical workloads; configure a second portion of the multiple workloads as non-critical workloads; and force stack sorting of the multiple workloads by requiring that each of the critical workloads be configured with an urgency value that is higher than any urgency value associated with any of the non-critical workloads.

[0182] Clause 15. A computing system as described in Clause 11, wherein the computing system is used to further apply a configurable multiplier to the single weighted backlog indicator value to determine a run interval of the general backlog indicator within a configurable range; and wherein the run interval represents how quickly or slowly the corresponding workload accumulates a backlog while remaining in an unscheduled state.

[0183] Clause 16. The computing system of Clause 11, wherein the general backlog indicator comprises a measure of a time that each respective workload from the plurality of workloads remains in an unscheduled state after the respective workload from the plurality of workloads finishes a previous execution.

[0184] Clause 17. The computing system of Clause 11, wherein the customized backlog indicator comprises a measure of utilization of specified one or more resources within the computing system that are affected by execution of the indicated corresponding workload from the plurality of workloads by the computing system.

[0185] Item 18. A computer-readable storage medium comprising instructions that, when executed, configure processing circuitry of a computing system to: obtain a general backlog indicator for a plurality of workloads to be executed by the data platform, wherein each of the plurality of workloads specifies one or more storage system maintenance operations for one or more storage systems managed by the data platform; obtain a customized backlog indicator for at least a subset of the plurality of workloads to be executed by the data platform; calculate a single weighted backlog indicator value for each of the plurality of workloads to be executed by the data platform by applying a configurable weight to the general backlog indicator and each of the customized backlog indicators for a corresponding workload from the plurality of workloads; schedule the plurality of workloads for execution on the data platform based on the single weighted backlog indicator value calculated for each of the plurality of workloads; and process the plurality of workloads according to the schedule.

[0186] Clause 19. A computer-readable storage medium as described in Clause 18, wherein the one or more storage system maintenance operations include at least one of the following: a garbage removal operation, which removes the data from the one or more storage systems managed by the data platform in response to determining that the data platform has identified the data as available for removal from the one or more storage systems managed by the data platform; a data replication redundancy operation, which replicates the data instance to create a new operational replica of the data instance in response to determining that the count of the operational replica of the data instance is lower than the configured redundancy minimum; a storage node recovery operation, which clones the data instance to a new storage node in response to determining that a storage node failure within the one or more storage systems causes the count of the operational data instance to be lower than the configured redundancy minimum; and a server message block (SMB) session removal operation, which removes the one or more SMB sessions from the one or more storage systems in response to determining that the data platform has identified the one or more SMB sessions as available for removal from the one or more storage systems managed by the data platform.

[0187] Clause 20. A computer-readable storage medium as described in Clause 18, wherein the instructions, when executed, further configure the processing circuit of the computing system to: obtain an urgency value for each of the multiple workloads; calculate a scaled backlog indicator value for each corresponding workload in the multiple workloads by scaling the single weighted backlog indicator value corresponding to each corresponding workload in the multiple workloads based on the urgency value obtained for the corresponding workload from the multiple workloads; and wherein the instructions configure the processing circuit of the computing system to: schedule the multiple workloads for execution in descending order according to the single weighted backlog indicator of each corresponding workload in the multiple workloads, or schedule the multiple workloads for execution using the scaled backlog indicator value of each corresponding workload in the multiple workloads.

Claims

1. A method comprising: obtaining, by a processing device of a data platform, a common backlog indicator for a plurality of workloads to be executed by the data platform, wherein each of the plurality of workloads specifies one or more storage system maintenance operations for one or more storage systems managed by the data platform; obtaining, by the processing device, a customized backlog indicator for at least a subset of the plurality of workloads to be executed by the data platform; calculating, by the processing device, a single weighted backlog indicator value for each of the plurality of workloads to be executed by the data platform by applying a configurable weight to each of the generic backlog indicator and the custom backlog indicator for a corresponding workload from the plurality of workloads; scheduling, by the processing device, the plurality of workloads for execution on the data platform based on the single weighted backlog indicator value calculated for each of the plurality of workloads; as well as The plurality of workloads are processed by the processing device according to the schedule.

2. The method of claim 1 , wherein the one or more storage system maintenance operations include at least one of: a garbage removal operation that removes the data from the one or more storage systems in response to determining that the data platform has identified the data as available for removal from the one or more storage systems managed by the data platform; a data copy redundancy operation that, in response to determining that a count of an operational copy of a data instance is below a configured redundancy minimum, copies the data instance to create a new operational copy of the data instance; a storage node recovery operation, the storage node recovery operation cloning the data instance to a new storage node in response to determining that a storage node failure within the one or more storage systems causes the count of the operational data instance to fall below the configured redundancy minimum; as well as A server message block (SMB) session removal operation that removes one or more SMB sessions from the one or more storage systems managed by the data platform in response to determining that the data platform has identified the one or more SMB sessions as available for removal from the one or more storage systems managed by the data platform.

3. The method of claim 1, further comprising: obtaining an urgency value for each of the plurality of workloads; calculating a scaled backlog indicator value for each respective workload of the plurality of workloads by scaling the single weighted backlog indicator value corresponding to each respective workload of the plurality of workloads based on the urgency value obtained for the respective workload from the plurality of workloads; and wherein scheduling the plurality of workloads for execution on the data platform based on the single weighted backlog indicator value corresponding to each respective workload of the plurality of workloads comprises one of: scheduling the plurality of workloads for execution in descending order according to the single weighted backlog indicator of each respective workload in the plurality of workloads, or The plurality of workloads are scheduled for execution using the scaled backlog indicator value for each respective workload in the plurality of workloads.

4. The method of claim 3, wherein obtaining the urgency value of each corresponding workload in the plurality of workloads comprises: configuring a first portion of the plurality of workloads as critical workloads; configuring a second portion of the plurality of workloads as non-critical workloads; as well as Stack ordering of the plurality of workloads is enforced by requiring each of the critical workloads to be configured with an urgency value that is higher than any urgency value associated with any of the non-critical workloads.

5. The method of claim 1, further comprising: applying a configurable multiplier to the single weighted backlog indicator value to determine a running interval for the universal backlog indicator within a configurable range; and The run interval represents how quickly or slowly the corresponding workload accumulates a backlog while remaining in an unscheduled state.

6. The method of claim 1, wherein the general backlog indicator comprises a measure of a time that each respective workload from the plurality of workloads remains in an unscheduled state after the respective workload from the plurality of workloads finishes a previous execution.

7. The method of claim 1, wherein the customized backlog indicator comprises a measure of utilization of specified one or more resources within the data platform that are affected by execution of the corresponding workload from the plurality of workloads by the data platform.

8. The method of claim 1, wherein each respective workload from the plurality of workloads embodies at least one of: an algorithm to be scheduled for execution by the data platform; a periodic algorithm to be scheduled for execution by the data platform using a recurring schedule; an iterative algorithm to be scheduled using a configurable frequency for execution by the data platform; a process of specifying one or more executable tasks to be performed by the data platform; a job specifying the one or more executable tasks to be performed by the data platform; specifying an overhead task of the one or more executable tasks to be performed by the data platform, the overhead task not being uniquely associated with any end user of the data platform; a temporary end-user task that specifies the one or more executable tasks to be performed via the data platform on behalf of or at the request of a uniquely specified end-user of the data platform; specifying an optimization routine for the one or more executable tasks to be performed by the data platform, the optimization routine being configured to improve an operational efficiency of the data platform; as well as A garbage collection routine configured to increase available storage space of the data platform at least by removing data that has been identified by the data platform as available for removal from one or more storage systems managed by the data platform.

9. The method of claim 1 , wherein obtaining the common backlog indicator for the plurality of workloads to be executed by the data platform comprises: The common backlog indicator for each of the plurality of workloads is calculated based on one or more of: a configurable urgency value for each respective workload in the plurality of workloads; a backlog indicator value for each respective workload from the plurality of workloads, the backlog indicator value representing a period of time during which the respective workload from the plurality of workloads failed to initiate execution at the data platform after exceeding an initially scheduled execution time of the respective workload from the plurality of workloads; a quiet time value for each respective workload from the plurality of workloads, the quiet time value representing a time period during which the respective workload from the plurality of workloads is not executed between repeated executions of the respective workload from the plurality of workloads; a workload deadline value for each respective workload of the plurality of workloads, the workload deadline value indicating that the respective workload from the plurality of workloads encountered a configurable deadline without completing execution at the data platform; a workload deadline count for each respective workload from the plurality of workloads, the workload deadline count indicating a number of times the respective workload from the plurality of workloads encountered a configurable deadline within a configurable historical time period without completing execution at the data platform, or a historical number of execution attempts for the respective workload from the plurality of workloads; an issue action count for each respective workload in the plurality of workloads, the issue action count indicating a number of outputs of executable tasks executed to completion by the data platform as part of each respective workload in the plurality of workloads; a per-workload-based aggregate issue time, the per-workload-based aggregate issue time being calculated for each respective workload from the plurality of workloads to indicate an amount of total processing time required to fully complete one or more executable tasks executed by the data platform as part of the respective workload from the plurality of workloads; as well as The general backlog indicator calculated for each of the plurality of workloads is stored uniquely associated with the respective workload from the plurality of workloads for which the general backlog indicator was calculated.

10. The method of claim 1, wherein obtaining the customized backlog indicator for the plurality of workloads to be executed by the data platform comprises: The customized backlog indicator for at least a subset of the plurality of workloads is calculated based on one or more of: a workload-specific total utilization value of the corresponding workload from the plurality of workloads to be scheduled; a workload-specific total estimated garbage value of the corresponding workload from the plurality of workloads to be scheduled; a customer-activated user interface action or button indicating that the data platform has received input associated with a request to apply an aggressive garbage collection operation at the data platform; a customer-specific upgrade request for the end user associated with the respective workload from the plurality of workloads to be scheduled; a service tier corresponding to the end user associated with the respective workload from the plurality of workloads to be scheduled; a total amount of pending workloads corresponding to the end users associated with the respective workloads from the plurality of workloads to be scheduled; a cluster utilization value corresponding to a cluster utilized by the end user associated with the respective workload from the plurality of workloads to be scheduled; an SLA issued for the corresponding workload from the plurality of workloads to be scheduled; as well as The customized backlog indicator calculated for each of the plurality of workloads is stored uniquely associated with the respective workload from the plurality of workloads for which the customized backlog indicator was calculated.

11. A computing system comprising: Storage devices; as well as a processing device, the processing device being capable of accessing the storage device and being configured to: obtaining a common backlog indicator for a plurality of workloads to be executed by the data platform, wherein each of the plurality of workloads specifies one or more storage system maintenance operations for one or more storage systems managed by the data platform; obtaining a customized backlog indicator for at least a subset of the plurality of workloads to be executed by the data platform; calculating a single weighted backlog indicator value for each of the plurality of workloads to be executed by the data platform by applying a configurable weight to each of the generic backlog indicator and the custom backlog indicator for a corresponding workload from the plurality of workloads; scheduling the plurality of workloads for execution on the data platform based on the single weighted backlog indicator value calculated for each of the plurality of workloads; as well as The plurality of workloads are processed according to the schedule.

12. The computing system of claim 11, wherein the one or more storage system maintenance operations include at least one of: a garbage removal operation that removes the data from the one or more storage systems in response to determining that the data platform has identified the data as available for removal from the one or more storage systems managed by the data platform; a data copy redundancy operation that, in response to determining that a count of an operational copy of a data instance is below a configured redundancy minimum, copies the data instance to create a new operational copy of the data instance; a storage node recovery operation, the storage node recovery operation cloning the data instance to a new storage node in response to determining that a storage node failure within the one or more storage systems causes the count of the operational data instance to fall below the configured redundancy minimum; as well as A server message block (SMB) session removal operation that removes one or more SMB sessions from the one or more storage systems managed by the data platform in response to determining that the data platform has identified the one or more SMB sessions as available for removal from the one or more storage systems managed by the data platform.

13. The computing system of claim 11, wherein the processing device is further configured to: obtaining an urgency value for each of the plurality of workloads; calculating a scaled backlog indicator value for each respective workload from the plurality of workloads by scaling the single weighted backlog indicator value corresponding to each respective workload from the plurality of workloads based on the urgency value obtained for the respective workload from the plurality of workloads; and The computing system is used to: scheduling the plurality of workloads for execution in descending order according to the single weighted backlog indicator of each respective workload in the plurality of workloads, or The plurality of workloads are scheduled for execution using the scaled backlog indicator value for each respective workload in the plurality of workloads.

14. A computing system as claimed in any one of claims 11 to 13: wherein the processing means is further configured to apply a configurable multiplier to the single weighted backlog indicator value to determine a running interval of the general backlog indicator within a configurable range; and The run interval represents how quickly or slowly the corresponding workload accumulates a backlog while remaining in an unscheduled state.

15. A computer-readable storage medium comprising instructions which, when executed, cause one or more processors to perform the method of any one of claims 1 to 10.

Images