Synchronous object placement for information lifecycle management
By synchronously applying ILM policies during data ingestion, the problems of low data ingestion efficiency and untimely application of ILM policies in the prior art are solved, and more efficient and secure data storage and protection are achieved.
Patent Information
- Application Number
- CN202011232377.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-11-07
- Filing Date
- 2020-11-06
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-11-06
AI Technical Summary
Existing distributed storage systems are inefficient in data ingestion and cannot guarantee the timely application of information lifecycle management (ILM) policies, which may lead to data protection and availability issues.
By synchronously applying ILM policies during data ingestion, creating copies of objects and storing them in multiple storage locations, ensuring that data is copied in multiple places and meeting ILM rules, providing higher data security and efficiency.
It improves the resource utilization rate of data during ingestion, ensures the timely application of ILM policies, provides better data security and storage system reliability, and reduces resource consumption and processing load.
Smart Images

Figure CN112783436B_ABST
Abstract
Description
Technical Field
[0001] The present specification relates generally to the field of data processing and, more particularly, to database and file management or data structures. Background Art
[0002] Information lifecycle management (LIM) generally includes policies, processes, practices, services and tools used to keep the business / regulatory requirements of information consistent with the appropriate and cost-effective infrastructure from the time the information is created to its final configuration. ILM can keep information consistent with business requirements through management policies and service levels associated with applications, metadata and data. The organization specifies the ILM rule set to be used for data. The collection of ILM rules can be specified in the ILM policy. Some factors that affect the ILM strategy include the cost of managing enterprise data, compliance with various laws and regulations across various jurisdictions and data fields (e.g., health-related data), litigation preparation, and enterprise-scale content management. In general, accessibility requirements and the value of data decrease over time. Therefore, the ILM strategy will usually store data of lower value in a manner that reflects the decline in data value (e.g., fewer copies, less resource-intensive data protection, higher latency, etc.). Summary of the invention
[0003] The present specification provides a method, including: receiving, by a node of a distributed storage system, a request to establish a connection to save an object to the distributed storage system; capturing, by the node, metadata associated with the object; determining a storage location for a copy of the object based on evaluating information lifecycle management (ILM) policy rules and metadata associated with the object; and creating a copy of the object at the storage location based on the evaluated ILM policy rules.
[0004] In one embodiment, creating a copy of the object at the storage location is based on a determination that a copy of the object can be made at the time of ingestion.
[0005] In one embodiment, the method further includes returning an ingest success message to the client application that sent the request to establish the connection.
[0006] In one embodiment, a copy of the object is created at the storage location without storing a temporary copy of the object in the distributed storage system.
[0007] In one embodiment, creating a copy of the object at the storage location is made synchronously with ingesting the object.
[0008] In one embodiment, the storage location includes a plurality of storage locations, and creating a copy of the object includes storing data segments of the copy of the object at different storage locations in the plurality of storage locations, and storing parity segments of the copy of the object at other storage locations in the plurality of storage locations.
[0009] In one embodiment, the metadata includes at least one of: ingest time, last access time, object name, part of object name, object keyword, creation location, object size, user metadata, and object tag data.
[0010] This specification provides a computing device, including:
[0011] a memory including a machine-readable medium including machine-executable code, the machine-readable medium having stored thereon instructions for executing a method of ingesting an object into a distributed storage system;
[0012] A processor, the processor being coupled to the memory, the processor being configured to execute the machine executable code so that the processor:
[0013] receiving a request to establish a connection to save an object to a distributed storage system;
[0014] Capture metadata associated with objects;
[0015] Determining a storage location for a copy of the object based on evaluating information lifecycle management (ILM) policy rules and metadata associated with the object; and
[0016] A determination is made that a copy cannot be made to the determined storage location.
[0017] In one embodiment, the machine executable code is further configured to cause the processor to: send a message indicating that ingestion of the object based on the determined ingestion options failed, the determined ingestion options requiring compliance with ILM policy rules in ingestion of the object.
[0018] In one embodiment, the machine executable code is further configured to cause the processor to: store the object based on a fallback ILM policy rule.
[0019] In one embodiment, the machine executable code is further configured to cause the processor to: store the object based on the fallback ILM policy rule, and return an ingest success message to the client application that sent the request to establish the connection.
[0020] In one embodiment, the machine executable code is further configured to cause the processor to: store two temporary copies of the object at two different nodes of the distributed storage system.
[0021] In one embodiment, the machine executable code is further configured to cause the processor to: store a replica of the object at each of a plurality of different nodes of the distributed storage system, and perform asynchronous evaluation of the ILM policy rules.
[0022] In one embodiment, the machine executable code is further configured to cause the processor to: store a temporary copy of the object at each of a plurality of different nodes of the distributed storage system, and perform asynchronous evaluation of the ILM policy rules on the temporary copy of the object.
[0023] In one embodiment, determining that a copy cannot be made to the determined storage location includes determining that the storage location is temporarily unavailable.
[0024] The present specification provides a non-transitory machine-readable medium having stored thereon instructions for performing a method, the instructions including machine-executable code, which, when executed by at least one machine, causes the machine to:
[0025] receiving a request to establish a connection to save an object to a distributed storage system;
[0026] Capture metadata associated with objects;
[0027] determining a storage location for a copy of the object based on evaluating information lifecycle management (ILM) policy rules and metadata associated with the object; and
[0028] A copy of the object is created at a storage location based on the evaluated ILM policy rules.
[0029] In one embodiment, the machine executable code, when executed by at least one machine, further causes the machine to determine that a copy of the object can be stored at a storage location at the time of ingestion, wherein creating a copy of the object at the storage location is based on determining that a copy of the object can be made at the time of ingestion.
[0030] In one embodiment, the machine executable code, when executed by at least one machine, further causes the machine to return an ingest success message to the client application that sent the request to establish the connection.
[0031] In one embodiment, creating a copy of the object at the storage location is made synchronously with ingesting the object.
[0032] In one embodiment, the storage location includes a plurality of storage locations, and creating a copy of the object includes storing data segments of the copy of the object at different storage locations in the plurality of storage locations, and storing parity segments of the copy of the object at other storage locations in the plurality of storage locations. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Embodiments of the present disclosure will be better understood by referring to the accompanying drawings.
[0034] Figure 1 A conceptual diagram of a distributed storage system for applying ILM policies at ingest time and throughout the lifecycle of an object.
[0035] Figure 2 is a diagram illustrating ILM management of an object during the various stages of its life cycle.
[0036] Figure 3 is an exemplary interface illustrating ILM rules for operating a distributed storage system.
[0037] Figure 4 is a flow chart illustrating an exemplary ILM strategy for operating a distributed storage system.
[0038] Figure 5 is a flow diagram of example operations for ingesting objects in a distributed storage system.
[0039] Fig. 6A Exemplary ILM rules are illustrated.
[0040] FIG. 6B to FIG. 6D The diagram illustrates the use of an erasure coding algorithm on object data.
[0041] Figure 7 is a flow chart of example operations for synchronized object placement for information lifecycle management.
[0042] Figure 8 is a flow chart of example operations for synchronized object placement for information lifecycle management.
[0043] Fig. 9 An example computer system with an information lifecycle management ingestion scanner is depicted. DETAILED DESCRIPTION
[0044] The following description includes example systems, methods, techniques, and program flows for implementing embodiments of the present disclosure. However, it is understood that the present disclosure can be practiced without these specific details. In other examples, well-known instruction instances, protocols, structures, and techniques are not shown in detail to avoid confusing the description.
[0045] Overview
[0046] Users of a geographically distributed large-scale storage system ("storage system", "distributed storage system" or "grid") may configure information lifecycle management (ILM) rules for their objects. However, at ingestion time, two replicated copies of the user's data may be in the same site for redundancy, and ILM is then applied afterwards to create the final placement. For example, a user may wish to erasure encode their data so that the storage system can immediately create two replicated copies in the same site and then asynchronously erasure encode the data after ingestion.
[0047] Two major issues with this workflow can be addressed in this article using synchronous ILM. First, there is inefficiency because for ILM policies other than those that require two replicated copies for the same site (e.g., the ILM policy is the same as the default), the storage system may be doing wasted work: the storage system may have to create two replicated copies that are ultimately discarded. Second, the user has little visibility into when the storage system has finished applying the ILM policy to the ingested object.
[0048] In accordance with aspects of the present disclosure, a storage system can be configured to apply data lifecycle policies synchronously at ingestion time. To illustrate the difference between the two approaches, an example storage system is presented in which a user has an ILM rule to create a replicated copy in each of two sites A and B.
[0049] In the first method:
[0050] 1. The storage system creates two replicated copies in site A.
[0051] 2. Return a successful response (eg, a Hypertext Transfer Protocol (HTTP) "200 OK" success status response code for an S3 PUT request) and queue the object for lifecycle management.
[0052] 3. Asynchronous lifecycle rules are activated.
[0053] 4. The storage system creates another replica in Site B to satisfy the ILM policy.
[0054] 5. The storage system deletes one of the replicas replicated in site A.
[0055] Note that in the first approach, because the storage system returns a success response (200 OK) to the client before applying ILM, the storage system may not be able to guarantee anything about the application of lifecycle management. The storage system may attempt to apply the ILM policy immediately, but the storage system cannot guarantee it. In the first approach, the ILM policy can be applied asynchronously as follows:
[0056] 1. The client performs a request to store an object (eg, a PUT request), the storage system may queue the object into storage for ILM processing, and then return a successful response (eg, 200 OK).
[0057] 2. If the storage system is under heavy load, the storage system may choose to cancel the initial storage enqueue of the object and postpone scanning the ILM.
[0058] 2a. The scanning process can be used to handle rescanning objects in the storage system and applying ILM when needed. When the scanning system defers actions from asynchronous ILM, the user must now wait for the scan to execute it again. An example of this scanning process is described in U.S. Patent Publication No. 2018 / 0314706A1, entitled "Iterative Object Scanning For Information Lifecycle Management", the entire contents of which are incorporated herein by reference.
[0059] 3. If the storage system has not cancelled the initial storage enqueue of the object, the storage system may attempt to perform ILM immediately. However, if application of the ILM policy is not feasible at this time (e.g., the entire site of the storage system is down or unreachable), the operating system may defer this until a later time and again rely on scanning ILM.
[0060] Due to the nature of the asynchronous process: a success response is returned after the initial ingestion but before the ILM rules are applied, the storage system cannot guarantee the certainty that ILM has been applied to the objects just ingested. Synchronous ILM makes this possible.
[0061] In a second method, according to an embodiment of the present disclosure:
[0062] 1. The client executes a request to store an object (e.g., a PUT request), and the storage system creates a replicated copy in site A and a replicated copy in site B.
[0063] 2. The storage system can return a successful response ("200OK" to an S3 PUT request)
[0064] In this second method, the storage system resource consumption for the PUT request is to create one less copy of the replicated object and use one less delete / unlink. A copy of the replicated (or erasure coded) object is created without storing a temporary copy of the object in the distributed storage system. Embodiments of the present disclosure ensure that the storage system reduces the resource consumption of the PUT request (e.g., by the storage system creating one less copy of the replicated object and requiring one less delete / unlink).
[0065] In the synchronous ILM embodiment, there are three available options for the user: balanced, strict, and dual commit. Two-phase commit refers to the behavior of the storage node of the distributed storage system creating two replicated copies in the same site and then applying ILM asynchronously. Selecting two-phase commit can be advantageous in some scenarios, such as for storage system instances with high inter-site latency and sufficient hardware resource backup (e.g., non-CPU or I / O limited, so there is no need to improve the efficiency of ingesting objects at the same time). In other embodiments, other default behaviors can be provided before applying ILM policies asynchronously, such as creating a single copy at the site, three or more copies at the site, or more complex defaults. Strict refers to the behavior of the storage system trying to apply the ILM policy synchronously at ingestion, and if it cannot, the S3 / Swift PUT will fail. This ensures that the storage system can guarantee that ILM has been applied to the most recently ingested objects. Balanced refers to the behavior of the storage system trying to apply ILM synchronously, but if the storage system cannot (e.g., failure to connect to the remote site), the storage system can fall back to two-phase commit or another fallback rule. Some storage systems may set balanced as the default setting for new ILM policies. The balanced setting may be usable just as well as two-phase commit, but most of the time, has the efficiency benefit of being synchronous. Using a balanced approach may be advantageous when the user does not require the guarantees of an ILM application at ingestion time, but does want to take advantage of the efficiency benefits.
[0066] Applying ILM policies synchronously at ingestion provides many advantages. Storage systems according to embodiments of the present disclosure have the ability to give guaranteed application of lifecycle policies. This may be desirable for customers who have a service level agreement (SLA) for data protection and want to be able to guarantee that their data meets their SLA. For example, an SLA may have a statement such as "we will have replicas of all data in at least two geographic locations."
[0067] The storage system according to the embodiments of the present disclosure may also improve efficiency by synchronously applying ILM; this may reduce consumption of I / O, CPU, and / or other resources when ingesting data (for example, if the ILM policy has a rule to create one replica at each site and there are two sites). In a deterministic system according to the previous method, the flow would be to create two replicated copies in site A, then the storage system may create another replicated copy in site B, and then delete one of the replicated copies in site A. In contrast, according to the embodiments of the present disclosure, ILM rules are synchronously applied at ingestion time to remove one of the replicates and remove the item (removal).
[0068] Synchronous ILM additionally provides better data security. For example, object data is immediately protected as specified in the placement instructions of the ILM rule, which can be configured to protect against a variety of different failure conditions, including the failure of more than one storage location. In addition, synchronous ILM provides more efficient grid operations. Each object is processed only once when it is ingested. Because the distributed storage system does not need to track or delete temporary copies, less processing load and less database space are consumed.
[0069] Example diagram
[0070] Figure 1 1 is a conceptual diagram of a distributed storage system for applying ILM policies at ingestion and throughout the life cycle of an object. A distributed storage system is geographically distributed across multiple sites including sites 105, 133, 135, and 137 (as examples of any number of sites), which communicate via a network including a wide area network (WAN). Each of these sites can accommodate multiple storage nodes and storage devices. A storage node includes a collection of processes (application processes, services, etc.) that store object data and metadata in a storage device and access the object data and metadata in the storage device. The collection of processes can be encapsulated by a virtual machine or a physical host. A storage node at any site of sites 105, 133, 135, and 137 can ingest an object into a grid. Ingestion refers to the operation of storing an object in a grid by one or more storage nodes based on a client request and (multiple) any governing ILM policies. The ingestion process includes assigning an object identifier to an object based on an object namespace defined for the grid. Ingestion can include objects being divided into their content data and metadata, caching, replication, erasure coding, and the like. The interface to the distributed storage system may include an object storage application programming interface (API) for storing objects in the grid, such as an S3-type API (based on the Amazon Simple Storage Service (S3) API) or a Swift-type (based on the OpenStack Object Storage Project Swift API) (to mention only two examples to which embodiments of the present disclosure may be applied). The API reference manual for S3 may be found in "Amazon Simple Storage Service API Reference API Version 2006-03-01", the entire contents of which are incorporated herein by reference.
[0071] Ingestion begins when a client application (e.g., an S3 or Swift client-based application) establishes a connection to save an object to the storage system, and completes when the storage system returns an "ingest success" message to the client. Depending on how ILM is specified, object data can be protected during ingestion by applying ILM instructions immediately (synchronous put), or by creating a temporary copy and applying ILM later (two-phase commit).
[0072] The storage nodes of the grid apply ILM policies 120 to objects at ingestion time and throughout the life of the objects in the grid. Each storage node of the grid is responsible for a different area of the object namespace 123. A subset of storage nodes ("management nodes") at each site in the distributed storage system maintains a copy of the ILM policy 120. Modifications or replacements of the ILM policy can be made in one of the management nodes and transmitted to other management nodes at different sites throughout the storage system. The constituent ILM rules are extracted from the ILM policy 120 and accessed by the ILM rule applicator 125. To address the situation of changes in the ILM policy, the grid maintains proposed and current ILM instructions (ILM policy identifiers 122) in a distributed data repository (or distributed database) 131 accessible to the grid storage nodes. To enhance practicality, the proposed and current ILM policies are identified by identifiers derived from the constituent rules to capture differences in the rules. Examples of ILM rules include replication rules, storage tiering or layering rules, data protection rules, and the like. ILM rule sets are generally represented as ILM policies for consistent organization of rules including precedence. To apply an ILM policy or rule set, a storage node evaluates the metadata of an object according to each rule in the rule set in order of priority, and determines whether an ILM task will be performed based on the rule evaluation. As an illustration, placement rules and storage classification rules can be triggered based on the size and age of an object, causing the object content data to be moved to a storage node at a different site that is assigned to a lower level storage pool. Distributed data repository 131 hosts object metadata, although different distributed data repositories can be used for object metadata and ILM policy identifier 122. Because ILM policies can be changed and distributed data repository 131 can be an eventually consistent distributed data repository, storage nodes across the grid can apply different versions of an ILM policy or different ILM policies.
[0073] For this example illustration, the storage node 107 at the site 105 includes a content delivery service 109, a storage subsystem 115, a distributed data repository service 117, an ILM scanner 121, and an ILM rule applicator 125. The content delivery service 109 manages the initial operations for the ingestion of an object. The initial ingestion operations handled by the content delivery service 109 may include request processing, data storage, storage management, data transfer to another storage node, and operations of a storage protocol interface. Data storage operations may include local caching of object content data and routing or storage of object metadata. The initial ingestion operations may include applying ingestion-based ILM rules. For example, the distributed storage system at the storage node 107 at the site 105 may protect objects during ingestion by performing synchronous placement, which evaluates ILM and makes a copy that satisfies the requirements as the object is ingested; or by performing a default ILM such as a two-phase commit, which creates a temporary copy and then evaluates ILM later. An administrator user can specify the method to be used for each object when creating an ILM rule.
[0074] The storage subsystem 115 interfaces with storage devices and / or external storage services for storing data to storage devices (physical or virtual) in response to commands or requests from the content delivery service 109. The distributed data repository service 117 performs operations corresponding to the distributed data repository 131, including managing a local instance 119 of the distributed data repository 131 including metadata for objects in the grid. The distributed data repository service 117 processes requests for the distributed data repository 131 from the content delivery service 109 and the ILM scanner 121. The ILM scanner 121 may continuously scan for object metadata of objects within the region(s) 123 of the object namespace autonomously assigned to the ILM scanner 121. The ILM scanner 121 requests data metadata from the distributed data repository service 117 and queues the object metadata into a queue set 127 ("ILM metadata queue") based on an evaluation priority. The ILM applier 125 selects object metadata from the ILM metadata queue 127, evaluates the object metadata against the ILM rule set of the ILM policy 120, and executes the resulting ILM task depending on whether the task is risky.
[0075] In some examples, ILM changes can occur while a multi-part upload is in progress. Each part of the upload is placed according to the rules that were active when the part was ingested; when the multi-part upload is complete, some parts of the object may not meet the current ILM requirements. In these cases, the ingestion of the object will not fail. Instead, any parts that were not placed correctly can be queued for ILM re-evaluation and moved to the correct location later.
[0076] against Figure 1 , client 101 requests to store object 103 to the grid. Storage node 107 at site 105 receives the request and object 103. Content delivery service 109 determines an object identifier for object 103 within object namespace 123. Content delivery service 109 divides object 103 into object metadata 111 and content data 113, and associates both with the object identifier of object 103. Object metadata 111 may indicate data size, data type, creation date, date / time of ingestion, data owner, etc. In addition to metadata already indicated in object 103, metadata 111 may include metadata created by storage node 107 (e.g., a site identifier corresponding to the ingest storage node). In some examples, a user can determine whether to have content delivery service 109 perform an inline ILM evaluation upon ingestion of object 103, or to have content delivery service 109 store data content 113 locally in a storage device at site 105 via storage subsystem 115 prior to an initial ILM evaluation (e.g., prior to sharding and fragmenting distribution across multiple storage nodes of a distributed storage system according to an erasure coding scheme).
[0077] Thus, when the default or two-phase commit option is selected, the metadata 111 may initially indicate the storage node 107 as the location of the content data 113. The content delivery service 109 requests the distributed data repository service 117 to store the metadata 111 in the local instance 119 of the distributed data repository 131. The content delivery service 109 also inserts the metadata into the ILM metadata queue 127. The ILM metadata queue 127 may include a first priority queue and a second priority queue. The content delivery service 109 inserts the metadata of the object at the time of ingestion into the queue 127 of a higher priority (i.e., the first priority queue). The ILM rule applicator 125 may be dequeued in a manner that is biased toward the first priority queue without starving the second priority queue (e.g., weighted round-robin).
[0078] However, when other options (such as strict or balanced options) are selected, local caching does not have to be implemented as part of ingestion. The content delivery service 109 can be programmed to evaluate the object metadata 111 according to the ILM policy 120 at ingest time and perform the ILM tasks determined from the rule evaluation, rather than asynchronously delegating to the ILM rule applicator 125.
[0079] Figure 2 is a diagram illustrating ILM management of an object during various stages of its life cycle.
[0080] Ingestion: Ingestion begins when a client application (e.g., an S3 or Swift client application) establishes a connection to save an object to the distributed storage system at block 202, and completes when the distributed storage system returns an "ingest success" message to the client. Object data is protected during ingestion by applying ILM instructions immediately at block 204 (synchronous put), or by creating a temporary copy and applying ILM later (two-phase commit), depending on how the ILM requirements are specified.
[0081] Replica Management: After creating the number and type of object replicas specified in the ILM placement instructions, the distributed storage system manages the object location and protects the object from loss.
[0082] ILM Scanning and Assessment: At block 208, the distributed storage system may continuously scan the list of objects stored in the grid and check whether the current replicas meet the ILM requirements. In other embodiments, the distributed storage system periodically scans the objects at regular intervals. In other embodiments, the distributed storage system may not periodically scan the objects at regular intervals, but may scan based on triggers. Triggers may include movement of an object, changes in the metadata of an object, and an increase in the frequency of access to an object.
[0083] When a different type, number, or location of object replicas is required, the distributed storage system creates, deletes, or moves replicas as needed. Background Verification: At block 210, the distributed storage system continuously performs background verification to check the integrity of object data. At block 212, if a problem is discovered, the distributed storage system automatically creates a new object replica or replaces the erasure coded object fragment at a location that meets the current ILM requirements.
[0084] Object Deletion: Management of an object ends when a copy is removed from the distributed storage system. At box 214, the object may be deleted as a result of a client delete request; or at box 216, the object may be deleted as a result of an ILM delete, an S3 bucket lifecycle expiration, or an automatic delete triggered by the end of a compliant S3 bucket retention period. After the client requests object removal, at box 218, the distributed storage system determines whether synchronous (e.g., immediate) removal is possible (e.g., if all copies are stored within the distributed storage system and not on the cloud, or access to tape backups is slow, or all sites are accessible). If so, at box 220, the object copy is removed and the space is reclaimed. If not, (or when the deletion is triggered by ILM), at box 222, the object is marked for deletion and the copy is queued for removal. Next, at box 224, the object copy is removed.
[0085] Figure 3 is an example interface for operating ILM rules for a distributed storage system. ILM rules may include three elements: (1) Filtering criteria: The basic and advanced filtering criteria of a rule define which objects the rule applies to. If an object matches the filtering criteria, the distributed storage system applies the rule and creates a copy of the object specified in the placement instructions of the rule. (2) Placement instructions: The placement instructions of a rule define the number, type, and location of object copies. Each rule may include a sequence of placement instructions to change the number, type, and location of object copies over time. When the time period of a placement expires, the instructions in the next placement are automatically applied by the next ILM evaluation. (3) Ingestion behavior: The ingestion behavior of a rule defines what happens when a client (such as S3 or Swift) saves an object to the grid. The ingestion behavior controls whether object copies are placed immediately according to the instructions in the rule, or whether temporary copies are made and the placement instructions are applied later.
[0086] Placement instructions determine where, when, and how object data is stored. An ILM rule may include one or more placement instructions. Each placement instruction applies to a single time period. When a user creates a placement instruction, the user can specify when the placement applies (time period), which type of copy to create (replicated or erasure coded), and where to store the copy (one or more storage locations). Within a single rule, a user can specify multiple placements for a time period, and specify placement instructions for more than one time period.
[0087] When a user defines a set of placement instructions for a rule, the user may be asked to ensure that at least one placement instruction starts on day 0 (e.g., at ingestion), that there are no gaps between defined time periods, and that the final placement instruction continues forever / indefinitely or continues until any object copies are no longer needed. As each time period in the rule expires, the content placement instructions for the next time period may be applied. New object copies are created and any unneeded copies are deleted.
[0088] like Figure 3 As illustrated in the example in , an exemplary ILM rule is applied to data belonging to Tenant A in a distributed storage system. The rule requires the distributed storage system to make three replicated copies of the ingested object and store each copy for two years in different data centers. After two years, one copy is moved to permanent (or indefinite) archival storage. The ILM rule uses a balanced option for ingestion behavior: as long as Tenant A saves the object to the distributed storage system, the three-copy placement instruction is applied, unless it is impossible to make the three requested copies immediately. In that case, the distributed storage system immediately makes two temporary copies on two different storage nodes and makes the required copies later.
[0089] When users create ILM rules, they can specify filtering criteria to identify which objects the rule applies to. Filtering rules can be simple or complex. In the simplest case, a rule may not specify any filtering criteria. A rule without filtering criteria is applied to an object, and the rule will be applied in the case where each item in the data has the same storage requirements. An example of a rule without filtering criteria is the inventory rule "Make 2 copies", which permanently stores two replicated copies of the object on any two storage nodes. If the user does not have more specific storage needs, the "Make 2 copies" rule can be used for objects. Users can also include "Make 2 copies" as a default rule in the ILM policy to provide storage instructions for objects that do not meet any of the criteria in the filtering criteria in other rules. Basic filtering criteria allow users to apply different rules to large, clear groups of objects. Available filters can be created using the "Create ILM Rules" wizard for tenant accounts, for S3 buckets, or for Swift containers (as two examples). These basic filters allow users to apply different rules to a large number of objects in a simple way. For example, a company's financial records may need to be stored to meet regulatory requirements, while data from the marketing department may need to be stored to facilitate daily operations. After creating separate tenant accounts for each department or separating data from different departments into separate S3 buckets, users can easily create one rule that applies to all financial records and a second rule that applies to all market data. Advanced filtering options within the Create ILM Rule wizard can give users granular control. Users can create filter rules to select objects based on one or more of the following object attributes: ingestion time, last access time, all or part of the object name (keyword), S3 bucket region (location constraint), object size, user metadata, and S3 object tags. Table 1 describes the different metadata types that can be used as filtering criteria.
[0090]
[0091]
[0092]
[0093] Table 1
[0094] Users can use advanced filtering to create very specific filtering criteria. For example, objects stored by the hospital's imaging department may be frequently used when they are less than 30 days old, and infrequently used thereafter; whereas objects containing patient visit information may need to be replicated to the billing department at the health network's headquarters. Users can create filters that identify each type of object based on object name, size, S3 object tags, or any other relevant criteria, and then create separate rules to store each group of objects appropriately.
[0095] Users can also combine basic and advanced filtering rules in a single rule as needed. For example, the marketing department may want to store large image files differently from their vendor records, while the HR department may need to centrally store personnel records in specific geographic locations and policy information. In this case, users can create rules that filter by tenant account to separate records from each department, while using advanced filters in each rule that identify the specific type of object to which the rule applies.
[0096] Figure 4 is a flow chart illustrating an exemplary ILM policy for operating a distributed storage system. An information lifecycle management (ILM) policy includes an ordered set of ILM rules that determine how the distributed storage system manages objects over time. In the exemplary ILM policy, the active ILM policy for the distributed storage system controls the placement, duration, and data protection of all objects. When a client stores an object to the distributed storage system, the object can be evaluated according to the ordered set of ILM rules in the active policy as follows: If the filter criteria for the first rule in the policy matches the metadata of the object, the object is ingested according to the ingestion behavior of that rule and stored according to the placement instructions of that rule. If the filter criteria for the first rule does not match the metadata of the object, the object is evaluated according to each subsequent rule in the policy until a match is made. If no rule matches the metadata of the object, the ingestion behavior and placement instructions for the default rule in the policy are applied.
[0097] To manage objects, users can create information lifecycle management (ILM) rule sets and organize them into ILM policies. Each object ingested into the system can be evaluated against an active policy. When a rule in a policy matches an object's metadata, the instructions in the rule determine what actions the distributed storage system takes to replicate and store that object. ILM rules can define: (1) Which objects should be stored. Rules can apply to all objects, objects belonging to a specific tenant account or bucket / container, or objects that contain specific metadata values. (2) Storage device type and location. Objects can be stored on storage nodes, in cloud storage pools, or on archive nodes. (3) The type of object copies made. Copies can be replicated or erasure coded. (4) For replicated copies, the number of copies made. (5) For erasure coded copies, the erasure coding algorithm used. (6) Changes in the storage location and type of copies of objects over time. (7) How object data is protected (synchronous placement or two-phase commit) as objects are ingested into the grid. In some examples, object metadata may not be managed by ILM rules. Alternatively, object metadata can be stored in a database called a metadata repository (e.g., an Apache Cassandra database). Multiple (e.g., three) copies of object metadata can be automatically maintained at each site to protect data from loss. The copies can be load balanced across all storage nodes of the distributed storage system.
[0098] At box 402, when an object is ingested, the active policy is applied. In the illustrated example, at box 404, all objects belonging to tenant A are matched by "Rule 1" and stored as three replicated copies at three data centers. Objects belonging to other tenants are not matched by the first rule, so they are evaluated by the next rule in the policy. At box 406, "Rule 2" is applied to objects stored by any tenant that are larger than a defined size (e.g., 200KB in the illustrated example). These larger objects are stored across two data center sites using erasure coding. Objects of 200KB (in the example) or smaller are not matched by the second rule, so they are evaluated by the third rule. At box 408, "Rule 3" is the default rule of the policy. The default rule is applied to any object that does not match any other rule in the policy. In this example, the default rule makes two replicated copies of all objects that are 200KB or smaller that do not belong to tenant A.
[0099] Figure 5is a flow chart of example operations for ingesting an object in a distributed storage system. At box 502, method 500 begins by a client application saving an object to a distributed storage system. The distributed storage system can evaluate which ingestion behavior the user selected when creating an ILM rule / ILM policy for ingestion. When the user creates an ILM rule, the user can specify whether the distributed storage system should use a two-phase commit, balanced, or strict option to protect the object at ingestion. Depending on the user selection, the distributed storage system makes a temporary copy and queues the object for later ILM evaluation, or uses synchronous placement and immediately makes a copy that meets the ILM requirements. Each of the three options for protecting objects at ingestion (balanced, strict, or two-phase commit) is applicable in some situations.
[0100] Two-phase commit: When the user selects two-phase commit, the distributed storage system immediately makes temporary copies on two different storage nodes and returns an "ingestion success" message to the client. The object is queued for ILM evaluation, and a copy that satisfies the regular placement instructions is made later. Two-phase commit provides a fast response to the client so that the distributed storage system can process subsequent requests more quickly, and the redundant temporary copies protect the data from the loss of storage nodes before ILM can be evaluated. However, if the temporary copy does not satisfy the regular placement instructions, the two-phase commit may be less efficient because the distributed storage system must create, track, and delete object copies that are ultimately not needed.
[0101] Strict: When the user selects the strict option, the distributed storage system can use synchronous placement at ingestion time and immediately make copies of the objects specified in the placement instructions of the rule. If creating these copies is not possible, for example because the required storage location is temporarily unavailable, then ingestion fails. The client can retry the operation at a later time. The strict option ensures that objects are always provided with the protections against loss specified in the placement instructions in the rule. For example, an object can be protected against the failure of more than one storage node or against the complete failure of an entire site if those protections are present in the rule. However, there can be a higher level of ingestion failures when using the strict option because transient problems can make creating the requested copies temporarily impossible.
[0102] Balanced: When the user selects the balanced option, the distributed storage system also uses synchronous placement at ingest time and immediately makes all replicas specified in the regular placement instructions. In contrast to the strict option, if it is not possible to make these replicas immediately, the distributed storage system falls back to using two-phase commit. Alternatively, different default rules are used. The balanced option provides a high level of data security, grid performance, and ingestion success. Ingestion may take longer because the distributed storage system may need to create a replica of the erasure coded object or a remote before it can return an "ingest success" message to the client. In some embodiments of the distributed storage system, the balanced option may be provided as the default unless changed by the user.
[0103] Table 2 provides the advantages of each of the ingestion behavior options for protecting data.
[0104]
[0105] Table 2
[0106] If the distributed storage system evaluating the ILM rules or policies determines that the ingestion behavior is strict, then at 504, the distributed storage system can determine whether a day 0 copy can be made immediately to the storage location determined by the ILM rules. The determined storage location can be based on evaluating the ILM policy according to the metadata associated with the object. The day 0 copy can include an immediate copy of the object to the expected location. The day 0 copy includes evaluating whether a copy of the object can be made according to the ILM policy. Possible reasons why the day 0 copy cannot be made are if: the storage site is unavailable (e.g., connection error), or the storage site is a storage site that cannot accept the day 0 placement. In the case of a connection error, the distributed storage system (or the storage node of the distributed storage system) can attempt to reconnect with the unavailable resource. In an example embodiment, the storage site that cannot accept the day 0 placement can include cloud storage pools (through third-party cloud hosting services, such as Amazon Web Services (AWS) and Microsoft AZURE), and archive nodes (e.g., tape-based archive storage devices) that may not be able to store objects at the time of ingestion based on speed / bandwidth requirements. Alternatively, third-party cloud services and / or archive storage devices can accept day 0 placement.
[0107] If the day-0 copy cannot be made immediately (block 504, no branch), the distributed storage system may send an ingest failure message to the client application that made the request at block 506. In this example, the object is not copied to the distributed storage system, and the ingest failure message may alert the client application that the object sent is not stored in the distributed storage system. The client application may resubmit the request to store the object.
[0108] If a day-0 copy can be made immediately (block 504, yes branch), then a copy is created by the distributed storage system to satisfy the ILM at block 508. The distributed storage system may send an "ingest success" message to the client application indicating that one or more copies of the object are stored in the distributed storage system from ingestion according to the ILM policy.
[0109] If the distributed storage system evaluating the ILM rules or policies determines that the ingestion behavior is balanced, then at 512, the distributed storage system can determine whether a day-0 copy can be made immediately to the storage location determined by the ILM rules. If a day-0 copy can be made immediately (block 512, yes branch), then at block 508, a copy is created by the distributed storage system to satisfy the ILM. The distributed storage system can send an "ingest success" message to the client application indicating that one or more copies of the object were stored from ingestion into the distributed storage system in accordance with the ILM policy. Alternatively, the ingest success message can indicate that the object was ingested without indicating whether the ILM policy has been followed.
[0110] If the day-0 copy cannot be made immediately (block 512, no branch), the distributed storage system may attempt to use a two-phase commit or another alternate process.
[0111] If the distributed storage system evaluating the ILM rules or policies determines that the ingestion behavior is a two-phase commit, or if the storage of the object fails the balancing rules, then at box 514, the distributed storage system may store a temporary copy of the object. The temporary copy may be made without regard to the ILM rules or policies. At box 516, the distributed storage system may send an "ingest success" message to the client application. The message may indicate that one or more copies of the object are stored in the distributed storage system. The ingest success message may indicate that a temporary copy is made. Alternatively, the ingest success message does not indicate whether the object copy is temporarily stored or permanently stored in the distributed storage system.
[0112] At block 518, the distributed storage system may asynchronously queue the object for ILM evaluation. At block 520, a copy of the object is created to satisfy ILM. Additional copies of the object may be made if they do not already exist due to temporarily stored copies. At block 522, any temporary copies that are not needed are deleted.
[0113] Objects can be stored in a distributed storage system using a variety of techniques including replication and erasure coding. When a distributed storage system matches an object to an ILM rule that is configured to create replicated copies, the system creates an exact copy of the object data and stores the copy on a storage node or archive node. When a user configures an ILM rule to create replicated copies, the user can specify how many copies should be created, where those copies should be placed, and how long the copies should be kept in each location.
[0114] For example, Fig. 6A An exemplary ILM rule is illustrated that specifies that two replicated copies of each object are placed in a storage pool containing three storage nodes. When a distributed storage system matches an object to this rule, the distributed storage system creates two copies of the object, placing each copy on a different storage node in the storage pool. The two copies may be placed on any two of the three available storage nodes. In this case, the rule places the object copy on storage nodes 2 and 3. Because there are two copies, the object can be retrieved if any of the nodes in the storage pool fails. In some examples, the distributed storage system stores only one replicated copy of an object on any given storage node. In this example, if the distributed storage system includes three storage nodes and the ILM is created to create 4 copies of the object, only three copies will be made - one copy on each storage node.
[0115] Erasure coding is the second method used by distributed storage systems to store object data. When a distributed storage system matches an object to an ILM rule configured to create an erasure-coded replica, it slices the object data into data fragments, computes additional parity fragments, and stores each fragment on a different storage node. When the object is accessed, it is reassembled using the stored fragments. If a data or parity fragment is damaged or lost, the erasure coding algorithm can recreate the fragment using a subset of the remaining data and parity fragments.
[0116] Distributed storage systems can use erasure coding algorithms such as the Reed-Solomon erasure coding algorithm, which divides an object into k data fragments and calculates m parity fragments (expressed using k+m notation). The k+m=n fragments are spread across n storage nodes to provide data protection. An object can withstand up to m lost or damaged fragments. k fragments may be needed to retrieve or repair an object.
[0117] Figure 6BThe use of erasure coding on the data of an object is illustrated. In the illustrated example, the ILM rule uses a 4+2 erasure coding scheme, each object is split into four equal data fragments, and two parity fragments are calculated from the object data. In a 6+3 erasure coding scheme, each of the six fragments is stored on a different node across three data centers to provide data protection against node failure or site loss. Other erasure coding schemes such as 8+2, 6+2, 9+3, 2+1, 4+1, and 6+1 are available.
[0118] The 4+2 erasure coding scheme requires at least nine storage nodes, three storage nodes at each of the three different sites. Figure 6C As shown, as long as any four of the six fragments (data or parity) remain available, the object can be retrieved. Up to two fragments can be lost without loss of object data. If an entire data center site is lost, the object can still be retrieved or repaired as long as all of the other fragments remain accessible. Fig.6D As shown, if more than two storage nodes are lost, the object is unavailable. Erasure coding can provide improved reliability, availability, and storage efficiency when compared to replication. However, an increased number of nodes and sites may be required, and there may be increased retrieval latency when erasure coding is used across multiple geographically distributed sites.
[0119] ILM rules can be created to perform erasure coding on data (or data above a certain threshold size, such as 1MB or 200KB). At ingestion time, the distributed storage system can evaluate the ILM rules to determine whether the two-phase commit, strict, or balanced option is selected. Each of the above options can be used with erasure coding. However, creating erasure-coded objects on multiple storage nodes may take longer and may have a greater chance of failure than using replication. In other embodiments, ILM rules that use erasure coding will not apply erasure coding in a two-phase commit until a temporary copy of the object is evaluated. In some embodiments, the strict option may not be available for erasure coding, or have a greater chance of failure.
[0120] Figure 7706 is a flow chart of example operations for synchronous object placement for information lifecycle management. At box 702, a distributed storage system (or a storage node of a distributed storage system) may receive a request to establish a connection to save an object to the distributed storage system. At box 704, a node of the distributed storage system may capture metadata associated with the object. The metadata may include an ingest time, a last access time, an object name, a portion of an object name, an object keyword, a creation location, an object size, user metadata, and / or object tag data. At box 706, the distributed storage system may determine a storage location for a copy of the object based on evaluating an ILM policy rule and metadata associated with the object.
[0121] At box 708, the distributed storage system may create copies of the object at storage locations based on the evaluated ILM policy rules. Creating copies of the object at one or more storage locations is based on determining that a copy of the object can be made at the time of ingestion. Creating the copy of the object can be made synchronously with ingesting the object. The distributed storage system can return an "ingest success" message to the client application. In some examples, creating the copy of the object includes storing data fragments of the copy of the object at different storage locations in the plurality of storage locations, and storing parity fragments of the copy of the object at other locations in the plurality of storage locations.
[0122] Figure 8 8 is a flow chart of example operations for synchronized object placement for information lifecycle management. At block 802, a distributed storage system may receive a request to establish a connection to save an object to the distributed storage system. At block 804, a node of the distributed storage system may capture metadata associated with the object. At block 806, the distributed storage system may determine a storage location for a copy of the object based on evaluating policy rules and metadata associated with the object. At block 808, the distributed storage system may determine that a copy cannot be made to the determined storage location.
[0123] In an exemplary embodiment, a distributed storage system may send a message indicating a failure to ingest an object based on a determined ingestion option that requires compliance with ILM policy rules when ingesting the object. In another exemplary embodiment, a distributed storage system may store the object based on a fallback ILM policy rule and return an ingestion success message to a client application. In another embodiment, a distributed storage system may store two temporary copies of an object at two different nodes of a distributed storage system. In yet another embodiment, a distributed storage system may store a copy of the object at each of a plurality of different nodes of a distributed storage system and perform an asynchronous evaluation of the ILM policy rules. The evaluation may be performed on a temporary object copy. The distributed storage system may determine that a copy cannot be made to a determined storage location because the storage location is temporarily unavailable.
[0124] The flowchart is provided to help understand the description and is not used to limit the scope of the claims. The flowchart describes example operations that can be varied within the scope of the claims. Additional operations can be performed; fewer operations can be performed; operations can be performed in parallel, and operations can be performed in different orders. It should be understood that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented by program code. The program code can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable machine or device.
[0125] It should be understood that various aspects of the present disclosure may be implemented as systems, methods, or program codes / instructions stored in one or more machine-readable media. Accordingly, various aspects may take the form of hardware, software (including firmware, resident software, microcode, etc.), or a combination of software and hardware aspects that may be generally referred to herein as "circuits," "modules," or "systems." The functionality presented as individual modules / units in the example illustrations may be organized differently according to any of the following: platform (operating system and / or hardware), application ecosystem, interface, programmer preference, programming language, administrator preference, etc.
[0126] Any combination of one or more machine-readable media may be utilized. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable storage medium may be, for example, but not limited to, a system, apparatus, or device that uses any or a combination of electronic, magnetic, optical, electromagnetic, infrared, or semiconductor technologies to store program code or machine executable code. More specific examples (a non-exhaustive list) of machine-readable storage media would include the following: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a machine-readable storage medium may be any tangible medium that may contain or store a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. A machine-readable storage medium is not a machine-readable signal medium.
[0127] A machine-readable signal medium may include a propagated data signal having machine-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof, and a machine-readable signal medium may be a machine-readable medium that is not a machine-readable storage medium and that may communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0128] Program code embodied on a machine-readable medium may be transmitted using any appropriate medium, including but not limited to radio, telephone line, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
[0129] Program code for performing operations of various aspects of the present disclosure may be implemented as instructions in various forms depending on the implementation. For example, the instructions may be written in any combination of one or more programming languages, including: an object-oriented programming language (such as Programming languages (such as C++, etc.), dynamic programming languages (such as Python), scripting languages (such as the Perl programming language or the PowerShell scripting language), and traditional procedural programming languages (such as the "C" programming language or similar programming languages). The program code may execute entirely on a stand-alone machine, may execute across multiple machines in a distributed fashion, and may execute on one machine and or accept input on another machine while providing results.
[0130] Program code / instructions may also be stored in a machine-readable medium that can direct a machine to work in a specific manner, so that the instructions stored in the machine-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or block diagram or block.
[0131] Fig. 9 An example computer system with an information lifecycle management iteration scanner is depicted. The computer system includes a processor 901 (which may include multiple processors, multiple cores, multiple nodes, and / or implement multithreading, etc.). The computer system includes a memory 907. The memory 907 can be any of the possible implementations of system memory (e.g., one or more of the following: cache, SRAM, DRAM, zero capacitor RAM, two-transistor RAM, eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM, etc.) or machine-readable media described above. The computer system also includes a bus 903 (e.g., PCI, ISA, PCI-Express, bus, Bus, NuBus, etc.) and a network interface 605 (e.g., a Fibre Channel interface, an Ethernet interface, an Internet Small Computer System Interface, a SONET interface, a wireless interface, etc.). The system also includes a set of storage devices 915, which may be magnetic storage devices, optical storage devices, solid-state storage devices, and / or a mixture of storage devices. The computer system may submit read and write requests via the network interface 905 or a separate interface (e.g., a small computer system interface). The system also includes an ILM ingestion scanner 911. The ILM scanner 911 may apply the ILM rule set of the distributed storage system to the object based on the object metadata synchronously at the time of ingestion. Any of the described functions may be implemented partially (or completely) in hardware and / or on the processor 901. For example, the functionality may be implemented in logic implemented in the processor 901 using an application-specific integrated circuit, in a coprocessor on a peripheral device or card, etc. In addition, the implementation may include fewer or additional functions not shown in the figure. Fig. 9 Components in the processor 901 (e.g., video card, audio card, additional network interface, peripheral devices, etc.). The processor 901 and the network interface 905 are coupled to the bus 903. Although illustrated as being coupled to the bus 903, the memory 707 may be coupled to the processor 901.
[0132] Although aspects of the present disclosure are described with reference to various implementations and developments, it should be understood that these aspects are exemplary and the scope of the claims is not limited to these aspects. In general, the synchronization object placement techniques using ILM as described herein can be implemented with any hardware system or hardware system consistent facilities. Many variations, modifications, additions or improvements are possible.
[0133] Multiple instances of components, operations or structures described herein may be provided as single instances. Ultimately, the boundaries between various components, operations and data repositories are more or less arbitrary, and specific operations are described in the context of specific illustrative configurations. Other assemblies of functions are contemplated and fall within the scope of this disclosure. Typically, the structures and functions presented as separate components in the example configurations may be implemented as combined structures or components. Similarly, the structures and functions presented as single components may be implemented as separate components. These and other variations, modifications, additions or improvements may fall within the scope of this disclosure.
[0134] Unless specifically stated otherwise, the phrase "at least one of..." preceding a list with the conjunction "and" should not be taken as an exclusive list and should not be interpreted as a category list with one item from each category. A statement stating "at least one of A, B, and C" may be violated by only one of the listed items, multiple items of the listed items, and one or more of the items in the list and another item not listed.
Claims
1. A calculation method, comprising: Receiving, by a node of a distributed storage system, a request from a client application to save an object to the distributed storage system at a first storage site of the distributed storage system; capturing, by the node, metadata associated with the object synchronously with ingestion of the object; determining, synchronously with ingesting the object, a second storage site of the distributed storage system for a copy of the object based on evaluating information lifecycle management (ILM) policy rules and the metadata associated with the object, the second storage site being different from the first storage site; storing the copy of the object at the second storage site synchronously with ingestion of the object based on the evaluated ILM policy rules; as well as After storing the copy of the object at the second storage site, a response is returned to the client application that sent the request, the response indicating that the object was ingested.
2. The method of claim 1, wherein storing the copy of the object at the second storage site is based on determining that the copy of the object can be made synchronously with ingestion.
3. The method of claim 1, wherein storing the copy of the object at the second storage site is made without storing a temporary copy of the object in the distributed storage system.
4. A method according to claim 1, wherein the second storage site includes multiple storage locations, and storing the copy of the object includes storing data fragments of the copy of the object at different storage locations among the multiple storage locations, and storing parity fragments of the copy of the object at other storage locations among the multiple storage locations.
5. The method of claim 1, wherein the metadata comprises at least one of: ingestion time, last access time, object name, a portion of the object name, object keywords, creation location, object size, user metadata, and object tag data.
6. A computing device comprising: a memory including a machine-readable medium including machine-executable code having stored thereon instructions for executing a method of ingesting an object into a distributed storage system; a processor coupled to the memory, the processor being configured to execute the machine executable code so that the processor: receiving a request from a client application to save an object to the distributed storage system at a first storage site of the distributed storage system; capturing metadata associated with the object synchronously with ingesting the object; determining, synchronously with ingesting the object, a second storage site of the distributed storage system for a copy of the object based on evaluating information lifecycle management (ILM) policy rules and the metadata associated with the object, the second storage site being different from the first storage site; as well as Synchronously with ingesting the object, it is determined that the copy cannot be made to the determined second storage site.
7. The computing device according to claim 6, wherein the machine executable code is further configured to cause the processor to: Synchronously with ingesting the object, a message is sent indicating a failure to ingest the object based on determined ingest options requiring compliance with the ILM policy rules.
8. The computing device of claim 6, wherein the machine executable code is further configured to cause the processor to: The object is stored based on a fallback ILM policy rule.
9. The computing device of claim 8, wherein the machine executable code is further configured to cause the processor to: In response to storing the object based on the fallback ILM policy rule, returning a response to the client application that sent the request to establish the connection, the response indicating that the object was ingested.
10. The computing device of claim 6, wherein the machine executable code is further configured to cause the processor to: Two temporary copies of the object are stored at two different nodes of the distributed storage system.
11. The computing device of claim 6, wherein the machine executable code is further configured to cause the processor to: In response to determining that the copy cannot be made to the determined second storage site, a copy of the object is stored at each of a plurality of different nodes of the distributed storage system, and an asynchronous evaluation of the ILM policy rule is performed.
12. The computing device of claim 6, wherein the machine executable code is further configured to cause the processor to: A temporary object copy is stored at each of a plurality of different nodes of the distributed storage system, and an asynchronous evaluation of the ILM policy rules is performed on the temporary object copy.
13. The computing device of claim 6, wherein determining that the copy cannot be made to the determined second storage site comprises: It is determined that the storage location for the second storage site is temporarily unavailable.
14. A non-transitory machine-readable medium having stored thereon instructions for performing a method, the instructions comprising machine-executable code which, when executed by at least one machine, causes the machine to: receiving a request from a client application to store an object to a first storage site of a distributed storage system; capturing metadata associated with the object synchronously with ingesting the object; determining, synchronously with ingesting the object, a second storage site of the distributed storage system for a copy of the object based on evaluating information lifecycle management (ILM) policy rules and the metadata associated with the object, the second storage site being different from the first storage site; as well as The copy of the object is stored synchronously with ingestion of the object at the storage location based on the evaluated ILM policy rules.
15. A non-transitory machine-readable medium according to claim 14, wherein the machine executable code, when executed by the at least one machine, also causes the machine to determine synchronously with ingestion that the copy of the object can be stored at the second storage site, wherein storing the copy of the object at the second storage site is based on determining that the copy of the object can be made synchronously with ingestion.
16. The non-transitory machine-readable medium of claim 14, wherein the machine-executable code, when executed by the at least one machine, further causes the machine to return a response to the client application that sent the request, the response indicating that the object was ingested.
17. A non-transitory machine-readable medium according to claim 14, wherein the second storage site includes a plurality of storage locations, and storing the copy of the object includes storing data fragments of the copy of the object at different storage locations among the plurality of storage locations, and storing parity fragments of the copy of the object at other storage locations among the plurality of storage locations.
