Non-transitory machine-readable medium, method, and system for forwarding storage operation requests to storage systems using the underlying volume identifiers

Abstract

Example implementations refer to virtual persistent volumes. In one example, a storage operation request includes a volume identifier. A volume mapping corresponding to the volume identifier is identified. Underlying volume identifiers are identified based on the volume mapping. The underlying volume identifiers refer to underlying storage volumes that form at least part of a virtual persistent volume associated with the volume identifier. The storage operation request is forwarded to the storage systems on which the underlying volumes are located, using the underlying volume identifiers.

Description

BACKGROUNDContainers are one type of virtualization. A container may contain an application packaged together with dependencies and libraries. A containerized application may use or generate persistent data.The present invention relates to a non-transitory machine readable medium, method and system for forwarding memory operation requests to memory systems using underlying data bearer detections.US 2019 / 0 310 873 A1 discloses connecting storage resources to virtual machines, wherein different host bus adapters are each assigned to different storage area networks.It is an object of the present invention to simplify and more effectively automate the management and management of storage of complex applications using multiple types of memory. This object is achieved by the subject matters of the independent claims.BRIEF DESCRIPTION OF THE DRAWINGSVarious examples will be described below with reference to the following figures. FIG. 1 illustrates an example system that creates a virtual persistent volume that summarizes a plurality of different underlying memories and is presented to a containerized application. FIG. 2 shows an example system that forwards a store operation request to a storage system corresponding to an underlying storage volume of a virtual persistent volume. FIG. 3 shows an example method that includes forwarding a store operation request to a storage system corresponding to an underlying storage volume of a virtual persistent volume. FIG. 4 shows an example method that includes creating a new virtual persistent volume representing recovered data. FIG. 5 shows an example method that includes rollback a state of a virtual persistent volume based on a transaction log. FIG. 6 shows an example method that includes identifying additional volume maps. FIG. 7 shows an example system including a machine readable medium including instructions to pass a memory operation request to a memory system corresponding to an underlying storage volume of a virtual persistent volume. FIG. 8 shows an example system including a machine readable medium including instructions to respond to a restore request and instructions to reset a state of a virtual persistent volume based on a transaction log.DETAILED DESCRIPTIONContainer technology is a paradigm of computer virtualization in which an application is packaged in a container along with dependencies and libraries to provide an isolated environment for execution of the application. Such an application may be referred to as a containerized application. Many containers may run on a single operating system, but each container is isolated from other containers by nature. In this way, the container paradigm may be understood as virtualization of the operating system. Containers may be lighter in weight than other forms of virtualization, such as virtual machines that virtualize hardware. For example, each virtual machine may have its own copy of an operating kernel, while in contrast multiple containers may share an operating kernel.Containerized applications may require memory to store persistent data. Container orchestrators (e.g., Kubernetes) may provide the ability to provide some memory for a container. However, there are many types of memory, including, but not limited to, memory that is local to the container, remote to the container, hardware (e.g., locally attached drive), software defined (e.g., a file system, virtualized or containerized memory, storage provided via an API, etc.), or memory having a combination of the foregoing aspects. Previous attempts to provide container storage may not provide the flexibility of the storage configuration and the data services that users and administrators desire container environments. For example, a container orchestrator may be limited to providing a storage type for an application. Other types of systems may attempt to concatenate multiple volumes into a single volume, but such concatenate may not have the flexibility to provide particular data services without disrupting user access to the data.To address the challenges and provide flexible and robust persistent storage accessible to containers, the examples described herein relate to a storage virtualization system and a policy engine thereof that can create a virtual persistent volume that summarizes a plurality of different underlying storage volumes that can be provided by different types of storage systems. The virtual persistent volume may take the form of a hierarchical structure, such as a Merchle tree, that relates data objects of the containerized application through their content-based signatures to a root object.Moreover, users or administrators may wish to perform various storage operations, such as snapshot, save, or restore operations, on the aforementioned virtual persistent volume. Accordingly, further examples described herein relate to the memory virtualization system and its policy engine that maps the memory operation on the virtual persistent volume to memory operations on the constituting underlying memory volume of the virtual persistent volume. In particular, an example memory virtualization system may receive a memory operation request that includes a volume identifier associated with a virtual persistent volume. The memory virtualization system may then identify a memory map corresponding to the volume identifier. The memory map contains an underlying volume identifier for each underlying memory volume that makes up the virtual persistent volume. The memory virtualization system may then forward the memory operation request to any memory system that provides an underlying memory volume by using the underlying volume identifier for that underlying memory volume. Thus, the storage virtualization system may function as a fan-out control point to manage the underlying storage volumes. Moreover, management and management of storage of complex applications using multiple types of memory can be simplified and automated.FIG. 1 shows an example of a computer system 100 supporting and participating in a container environment 120. The computing system 100 includes a processing resource 102 that may include a microcontroller, a microprocessor, one or more central processing unit cores, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The computing system 100 includes a machine readable medium 104, which may be non-transitory and may include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory, a hard disk drive, etc.The processing resource 102 may execute instructions 105 (i.e., programming or software code) stored on the machine readable medium 104 to perform functions of the computer system 100, such as providing a container orchestrator 122 and a containerized storage system 130, as described further below. In particular, each of the components of the containerized storage virtualization system 130 may be implemented as executable instructions 105, including the data service provider 134, the storage virtualization unit 136, the volume manager 138, the adapters 139, the policy engine 142, and the container storage interface API 144. Containerized applications 124, 126, and container memory interface plug-ins 128 may also be implemented as instructions included in executable instructions 105. Additionally or alternatively, the processing resource 102 may include electronic circuitry for executing the functionality described herein.Computer system 100 may also include other hardware components, such as physical memory 106. The physical storage 106 may include any physical storage device, such as a hard disk drive, a solid state drive, or the like, or a plurality of such storage devices (e.g., an array of hard disks), and may be locally attached (i.e., installed) to the computer system 100. In some implementations, physical memory 106 may be accessed as a block storage device.In some cases, the computer system 100 may also include a local file system 108, which may be implemented as a layer above the physical storage 106. For example, an operating system 107 may be executed on the computer system 100 (due to the processing resource 102 executing certain instructions 105 related to the operating system), and the operating system 107 may provide a file system 108 for storing data on the physical storage 106.The computer system 100 may be in communication with other computer systems, such as the computer system 110, e.g., via a wired and / or wireless network. The other computing systems may be similar to computing system 100 and each include at least one processing resource and a machine readable medium. The computer systems 100 and 110 may each execute software (i.e., the processing resource 102 executes certain instructions 105) to employ nodes of a container orchestrator 122. In other words, container orchestrator 122 may have a cluster architecture that includes container orchestrator nodes of computer systems connected in a cluster. Container orchestrator 122 acts as a platform for providing and managing containerized applications over the cluster of computer systems. Container orchestrator 122, the containerized applications provided by it, and other container resources (such as container storage) are considered part of container environment 120. In contrast, other elements may operate outside of container environment 120, such as local file system 108 and operating system 107 of computer system 100.In FIG. 1, containerized applications 124 and 126 are illustratively provided locally on computing system 100 via container orchestrator 122. Thus, it can be said that the applications 124, 126 are remote from other nodes, such as the computer system 110. The containerized applications 124, 126 may represent microservices, user applications, or the like, and are sometimes also referred to simply as containers. Container environment 120 may also host containerized applications (not shown) that are local to other computer systems in the cluster.Container orchestrator 122 also employs a containerized storage virtualization system 130, which is described in more detail below. In the example of FIG. 1, the containerized memory virtualization system 130 is also executed locally on the computer system 100 and its processing resource 102. In some implementations, other containerized storage virtualization systems of other computer systems are also included in the cluster in the container environment 120, although they are not shown in FIG. 1. For example, containerized storage virtualization system 130 may serve as a node in a storage virtualization platform having a cluster architecture in which multiple containerized storage virtualization systems (at least some of which may be executed on different and separate physical computer systems) cooperate to store and manage data in a storage virtualization layer.Container orchestrator 122 may also include a standardized container storage interface 123. Container storage interface 123 has a plug-in architecture and may provide storage in the form of a persistent volume from a storage source by using a corresponding one of a plurality of available container storage interface plug-ins 128. "Providing" memory may refer to the process of allocating a certain amount of memory capacity and providing that allocation to a consumer. Plug-ins 128 may be provided by various providers and may enable (i.e., make available for use and / or consumption) an associated storage system for container storage interface 123. Non-limiting examples of plug-ins include a block protocol plug-in (e.g., based on the Internet small computer systems interface or iSC protocol), a file protocol plug-in (e.g., based on the network file system or NFS protocol, the common Internet file system or CIFS protocol, the server message block or SMB protocol), a public cloud persistent volume plug-in, and other plug-ins based on any other type of memory (e.g., customized drivers). For convenience, individual ones of the plug-ins 128 may be referred to herein as plug-in 128 (e.g., a block protocol plug-in 128 or a file protocol plug-in 128). A plug-in 128 may undergo an installation and setup process in the container environment 120 as required for that plug-in (e.g., login or other configuration details entry). In some cases, one or more of the plug-ins 128 may be executed as a containerized application. However, a persistent volume provided via the plug-in 128 may be limited to a single type of memory corresponding to that plug-in. In contrast, the containerized storage virtualization system 130 disclosed herein may be advantageously used for the creation of persistent volumes that blend multiple underlying storage types.The containerized storage virtualization system 130 (also referred to as storage virtualization system 130) runs in one or more containers and may be implemented in executable instructions 105. As will be seen, the memory virtualization system 130 provides an additional memory virtualization layer between a requesting application and one or more memory assignments provided via the container memory interface 123. The memory virtualization system 130 includes a data path 132 and a control plane 140. Data path 132 includes data service provider 134, memory verifier 136, and volume manager 138. Data path 132 may also include storage adapters 139 for accessing local storage (i.e., storage on the same computer system 100 on which storage virtualization system 130 is hosted), such as a block adapter for mounting local physical storage 106, stacked file and block adapters for accessing local file system 108 as a block device, or other storage adapters.The control plane 140 includes a policy engine 142 and a container storage interface (API) 144. During an initialization phase of the storage virtualization system 130, the control plane 140 may receive from an administrator or container orchestrator 122 a list of the available container storage interface plug-ins 128 in the container environment 120. The control plane 140 may also acknowledge the storage adapters 139 available for the data path 132 for mounting local storage. The control plane 140 may also maintain a list of features of the memory associated with each of these available plug-ins 128, such as performance features (e.g., latency, IOPS, or input / output operations per second, etc.), security features (e.g., encryption, isolation, etc.), privacy features (e.g., available RAID or redundant array of independent disks planes), cost features (e.g., dollar per GB), or other features.The functionality of the datapath functions 132 and the control plane functions 140 will now be described in connection with providing memory for containerized applications 124, 126. A containerized application, such as application 124, may request memory from container orchestrator 122. For example, the application 124 may require a persistent volume to store data. In some implementations, the application 124 may pass one or more requests with the request, such as a capacity request, a power request (e.g., latency, IOPS, etc.), a privacy request (e.g., RAID level), a cost request (e.g., dollar per GB), a security request, an tiering request (e.g., certain amounts of hot and cold storage), or another request. In some implementations, container orchestrator 122 may manage a storage abstraction called "orchestrator persistent volume" in response to the request.The container orchestrator 122 may use the container storage interface 123 to send the request to the storage virtualization system 130 via the container storage interface API 144 of the control plane 140 (where the interface 144 acts as a server). In this manner, the containerized storage virtualization system 130 may be understood to function as a storage provider for the container orchestrator 122. In some implementations, the control plane 140 may assign an identifier to the request such that each request may be individually identified, particularly with respect to the memory provided for each request in the manner described herein.The policy engine 142 of the container orchestrator 122 analyzes the request to determine which memory types meet the requests of the request. For example, the control plane 140 may have one or more of the following types available for storage provisioning: physical storage 106, local file system 108, remote storage system 160, virtualized storage 162, or public cloud storage 164. Additionally, more than one of the illustrated types of memory may be present, but is not shown for clarity. For example, multiple remote memories 160 may be available from which memory assignments may be provided.The remote storage system 160 shown in FIG. 1 may represent either a file system (e.g., an NAS file server), a block storage system (e.g., a SAN storage device), or any other type of storage system remote from the computer system 100, and thus also from the containerized storage virtualization system 130. Being remote may mean, for example, that the computer system 100 communicates with the remote storage system 160 via a network connection or the like.Virtualized storage 162 may represent any persistent volume in a virtual environment existing, such as a persistent volume in container environment 120, including container storage provided via container storage interface 123 independent of container storage virtualization system 130. In some examples, virtualized storage 162 may represent container storage provided by a container storage virtualization system other than system 130 (e.g., hosted on a node other than computer system 100). In some examples, virtualized memory 162 may represent memory provided by a virtual machine or a hypervisor-based software-defined storage platform.Policy engine 142 may determine a mixture of the aforementioned memory types. For example, policy engine 142 may compare the request to the available memory types to determine a match that is as great as possible. Illustratively, the request may require a certain amount of high speed memory and a certain amount of low cost archive class memory. The policy engine may determine that the physical memory 106 meets the requirements for the high-speed memory (e.g., due in part to locality and because this example is a high-speed medium) and a block storage device 160 meets the requirements for the low-cost archive memory. Additional example implementations of policy engine 142 are described further below, e.g., with reference to FIG. 2.Next, the control plane 140 uses the adapters 139 and / or container memory interface plug-ins 128 to provide each type of memory in the specified mix and obtain a hang point for each memory provided. A hang point allows access to the provided memory by a load, such as data path 132, as described further below.As an example of local provisioning, the control plane 140 may use a block adapter from the adapters 139 to provide an association from the physical memory 106 and acquire a local block suspension point 170 (e.g., a local host device suspension point) to access that association. As another example, the control plane 140 may use stacked file and block adapters to provide an assignment from the local file system 108 and acquire a local file system hang-up point 172 to access that assignment as a block device (i.e., "file as a block device").To provide storage via the plug-ins 128, the control plane 140 communicates via the container storage interface API 144 with the storage interface 123 to request that a plug-in 128 provide an association from its associated storage and provide a hang-in point back to the control plane 140. Assuming that the remote storage system 160 represents a remote block device (e.g., a SAN memory array external to the computer system 100), the control plane 140 may request (via 144 and 123) that a block protocol plug-in 128 (e.g., based on the iSC protocol) provide an assignment from the remote block storage system 160 and provide a remote volume mount point 174 (e.g., iSC target and LUN or logical unit number) to access this assignment. As another example, remote storage system 160 may represent a remote file device (e.g., an NAS file server), and control plane 140 may request (via 144 and 123) that a file log plug-in 128 (e.g., based on the NFS log) provide an assignment from remote storage system 160 of the file type and provide a remote volume mount point 174 (e.g., an IP address and an export name under NFS) to access this assignment. In some implementations, the control plane 140 may use a block protocol plug-in 128 to provide from the physical storage 106 or a file protocol plug-in 128 to provide from the local file system 108 rather than using an adapter 139.As another example of providing via a plug-in, the control plane 140 may request (via 144 and 123) that a plug-in 128 that matches the virtualized storage 162 provide an assignment from the virtualized storage 162 and provide a hang-in point for the virtualized storage 176 to access that assignment. As another example, the control plane 140 may request (via 144 and 123) that a public cloud plug-in 128 provide an association from the public cloud storage 164. In turn, the public cloud plug-in 128 may provide a public cloud persistent volume hang point 178 for access to this mapping.Although a local block mount point 170, a local file system mount point 172, a remote volume mount point 174, a virtualized storage mount point 176, and a public cloud persistent volume mount point 178 are illustrated in FIG. 1, more or fewer mount points and mount points may be requested by the control plane 140 in any combination and captured via the adapters 139 or plug-ins 128. In various cases, multiple local block hang-in points 170, multiple local file system hang-in points 172, multiple remote volume hang-in points 174, multiple virtualized storage hang-in points 176, and / or multiple persistent public cloud volume hang-in points 178 may be requested and captured by the control plane 140. Moreover, storage system 160 and remote hang point 174 may each represent one or more of the same or different types of remote storage and a hang point thereof, including block, file, or other types of remote accessible storage. The particular combination of storage hang-up points requested and acquired by the control plane 140 may depend on the storage request of the containerized application 124, and more particularly, on the handling of that storage request by the policy engine 142.Once the one or more storage hang points (e.g., 170, 172, 174, 176, or 178) are detected by the control plane 140 in accordance with the policy engine 142, the control plane 140 passes the detected hang points to the data path 132. The control plane 140 may identify the hang points as pertaining to a particular request by associating the hang points with the request identifier, for example. As described, the data path 132 consumes and merges (i.e., aggregates) the hang-in points to create a virtual persistent volume 156 presented by a hang-in point 180 of the requesting containerized application 124. In this way, the allocated memory corresponding to the acquired hang points (e.g., 170, 172, 174, 176, or 178) may be referred to as the underlying memory (or similar, backend memory) of the virtual persistent volume 156. Thus, the containerized application 124 reads and writes data to the virtual persistent volume 156. Before describing the creation of the virtual persistent volume 156, operational aspects of the data path 132 will first be described.Data path 132 includes memory virtualization layer 136, which maintains object-based memory virtualization layer 150. A purpose of the memory virtualization layer 150 is to decouple the location at which data is stored (i.e., memory assignments accessed via the suspension points 170, 172, 174, 176, and / or 178) from how the data is presented to a consumer of the data (e.g., a containerized application 124). In this way, data services such as migration, backup, snapshotting, replication, deduplication, compression, and others may be performed on any mixture of the underlying storage and with reduced, minimal, or even no disruption to a consumer of the data.An aspect of the memory virtualization layer 150 is that the memory virtualization unit 136 stores data as "objects" in an object memory 152. In particular, the object storage 152 may store various types of objects, including data objects and metadata objects. Data relating to the containerized application 124, including files and / or directories, is formed from one or more data objects. Metadata objects may serve, among other things, to organize the data objects in a meaningful and ordered manner, as described further below. In some implementations, each data object in object storage may be a fixed amount of data, such as 4 or 8 kibibyte data, and metadata objects may also be a fixed amount of data, such as 1 kibibyte.An object-based storage virtualization layer 150 may be different from block level storage (e.g., implemented in a SAN and represented via a storage protocol such as iSC or Fibre Channel) and file level storage (e.g., a file system that manages data in a file hierarchy and represented via a file level protocol such as NFS or SMB / CIFS), although the object-based storage virtualization layer 150 may submit block or file level storage protocols (e.g., by abstraction via suspension points 180, 182, as will be described).The storage verifier 136 maintains an object index 154 that tracks a signature, a physical address, and a reference counter for each object (data object and metadata object) in the object storage 152. The signature of an object may be a cryptographic digest of the content of that object using a hash function such as SHA-1, SHA-256, MD5, etc. Therefore, the signature may also be referred to as a content-based signature. The reference counter in object index 154 refers to the number of times the associated object is referenced across all virtual persistent volumes (including 156, 158) in storage virtualization layer 150.The physical address in the object index 154 refers to the actual physical location of the object. Although in some examples, the object storage 152 may be understood as a storage construct to describe the storage of objects within the storage virtualization layer 150, it may also be understood that the objects are physically stored on the underlying storage at the physical address. Because data path 132 may utilize multiple memory hang-in points, the respective hang-in point may be a portion of the physical address. In addition, the physical address may include a location within the memory allocation of a particular hang point. For example, if the hang-in point refers to the physical memory 106, the physical address may include a logical block number. If the hang point refers to a public cloud storage 164, the physical address may include a reference to a cloud object in the syntax of the corresponding hosting provider. Volume manager 138 is configured to perform data reads and writes for particular physical addresses, e.g., the physical addresses stored in object index 154.In some implementations, the containerized storage virtualization system 130 may use an additional indirect layer between the object index 154 and the actual physical storage location of the objects. In such implementations, the volume manager 138 may assign and / or partition the underlying memory assignments into extents (or also referred to as mini-volumes) at each suspension point provided by the control plane 140. The additional layer of indirection may be implemented by storing a virtual address in place of a physical address in object index 154 in conjunction with an object signature and managing an extent table mapping a given virtual address to an extent and thus the corresponding underlying memory. Thus, to access an object based on a virtual address, the volume manager 138 may first identify the area targeted by the virtual address by using the area table and a first portion of the virtual address, and then locate the object within the area using a second portion of the virtual address. In this manner, certain data services, such as migration and tiering between extents, may be efficiently performed by updating only the extent identifiers in the extent table, rather than updating a large amount of intra-memory or persistent references to the data objects (i.e., each address affected in the object index 154) and re-generating various logical addresses, indices, and other data structures used in managing the storage system.Within the storage virtualization layer 150, the storage virtualization unit 136 manages one or more virtual persistent volumes backed up by the object storage 152. In some implementations, a containerized application will be connected to a virtual PV in a one-to-one relationship. For example, in the example illustration of FIG. 1, the containerized application 124 is connected to the virtual PV 156 and the containerized application 126 is connected to the virtual PV 158. In some implementations, each virtual PV is mapped by the container orchestrator 122 to a corresponding persistent orchestrator volume managed by the container orchestrator 122, and the requesting containerized application 124 accesses the storage of the virtual PV 156 via the persistent orchestrator volume.In other cases, containerized applications and virtual PVs may be interconnected in one-to-multiple, many-to-one, or many-to-many relationships. For purposes of illustration only, virtual PVs will now be described with reference to virtual PV 156, although the same description may apply to other virtual PVs, such as virtual PV 158 and other virtual PVs not shown.In one implementation, the virtual persistent volume 156 may be an organization of metadata objects and data objects stored in the object storage 152, where the organization hierarchically relates the data objects to a root object by associated content-based signatures. In an example, the virtual PV 156 may be a Merchle tree (also referred to as a hash tree) or other hierarchical arrangement (e.g., directed acyclic graphs, etc.). In the case of a hierarchical Merkle tree, data objects may be located at the lowest tree level of each branch (also referred to as leaf level furthest from the root object) and such data objects may be referred to as leaf data objects. As described above, the data objects form the containerized application 124 data, such as files and directories.Within the hierarchical arrangement, a parent object refers to an object that contains as its content the signatures of children. For example, a parent object of leaf-level data objects is a metadata object that stores as content the signatures of its parent leaf-level data objects. In this manner, the signatures of objects at each level are collected into parent objects at a next higher level in the hierarchical arrangement until the root object is reached. The root object is thus also a metadata object which stores the signatures of the respective child objects as content. Viewed from another perspective, the hierarchical arrangement expands in a direction from the root object to the leaf level - a metadata object at any level can extend to a number of child nodes specified by a predefined branching factor. A metadata object may be capable of storing a set of signatures that corresponds at least to the branching factor of the hierarchical arrangement, such that it may contain the signatures of all children.Any change in the virtual PV data (i.e., new data, changed data, deleted data) results in a change in the content of one or more leaf-level data objects, resulting in a change in the content-based signatures of those changed data objects, thereby propagating the changes in content and signatures up through the parent nodes to the parent object. Thus, a virtual PV 156 may be uniquely identified by its parent object at a particular time (also referred to as snapshot), particularly by its parent object signature.Another aspect of the virtual PV 156 is that, in some implementations, a particular file or directory from the containerized application 124 may be stored in a corresponding subtree arrangement within the virtual PV 156. In other words, the virtual PV 156 may be divided into sub-trees, each of which corresponds to a corresponding file or directory of the containerized application 124.Because files and directories consist of one or more data objects and these data objects are arranged in the virtual PV 156 and their sub-trees by referencing associated data object signatures, in some implementations, each of the data objects may be stored physically only once in the object storage 152 and referenced by their respective signatures in multiple metadata objects in the virtual PV 156 or in another virtual PV (e.g., 158) in the storage virtualization layer 150. In this way, the data may be deduplicated by the memory virtualization unit 136. Similarly, metadata objects may be stored once and referenced multiple times by corresponding signatures. The number of times a data object or metadata object is referenced in the storage virtualization layer 150, may be recorded in the corresponding reference counters of the object index 154. In some implementations, the data deduplication may be performed inline during a write operation, as opposed to post-processed or near-line deduplication, and in this way, the storage of data may be described as being natively deduplication in the memory virtualization layer 150 and between the virtual PVs 156, 158.In use cases where security plays a role, including multi-latency scenarios, separate object memories may be used for each sensitive security domain. Thus, sensitive data can be isolated in a secured object memory without participating in the deduplication of other virtual PVs that are not in the security domain.In order for the containerized application 124 to access the virtual PV 156, the data path 132 may provide a hang-in point 180. The data path 132 may provide any type of suspension point from a plurality of types of suspension points, including, but not limited to, a block type suspension point (e.g., iSC compatible), a file type suspension point (e.g., a file system in user space or a FUSE interface or NFS, SMB, or CIFS compatible), a key / value release suspension point (e.g., a noSQ volume or an amazone S3 compatible API), and other types of suspension points. In this manner, the hang-in point may be understood to contribute to a complete abstraction of the memory access, as the containerized application 124 is provided with any type of memory access needed by the containerized application 124 (e.g., file log or block log, etc.), regardless of the underlying type of memory that the virtual PV 156 is made up of (e.g., regardless of software or hardware-based, block or file, local or remote, etc.).The type of hang point, i.e., the type of abstraction, may be selected by the user or predefined according to the containerized application 124 requesting the storage (i.e., based on a class of the containerized application specified by the container orchestrator 122, etc.). The type of abstraction may be indicated to the containerized storage virtualization system 130 via the storage request received at the control plane 140.In the example of FIG. 1, containerized storage virtualization system 130 may also provide a hang point 182 for containerized application 126 to access a virtual persistent volume 158 created in response to a storage request from containerized application 126 to container orchestrator 122, in a similar manner as described above for hang point 180 of virtual PV 156 for containerized application 124. The virtual PVs 156 and 158 may both include respective marker trees for organizing respective records, while the object memory 152 stores the data of both virtual PVs 156 and 158 in a dedicated manner.In operation (i.e., after the virtual PV 156 is created and the suspension point 180 is provided to the application 124), the storage virtualization system 130 may service input / output (I / O) requests from the containerized application 124 that are directed to a virtual PV 156 via the suspension point 180. For example, to service a read request received via the hang-in point 180, the memory verifier 136 may identify the signatures of data objects in the virtual PV that are addressed by the read request (i.e., which may include traversing the virtual PV's Merkle tree structure based on an address of the read request) and determine the physical addresses of these data object signatures from the object index 154. In some implementations, the physical address of a data object may indicate the hang point of the underlying storage assignment at which the data object is stored (e.g., one or more of hang points 170, 172, 174, 176, or 178). The storage virtualization system may then read, via the volume manager 138, the data objects using the physical addresses (or using a virtual address and an extent table as described above) and return the read data to the containerized application 124 via the suspension point 180.To service a write request, in example implementations, the memory verifier 136 may receive the data to be written to the virtual PV 156 from the containerized application 124, check whether the data includes any data objects that are new to the object memory 152 based on content-based signatures, and write the new data objects to the object memory (i.e., data objects that are not already present in the data memory). In some implementations, the memory verifier 136 may compress the new data objects prior to writing to the object memory 152. The process of writing to the object memory 152 may specifically include the control over which underlying memory map (e.g., 106, 108, 160, 162, or 164) the new data objects are written. In some implementations, the containerized application in the write request may indicate into which underlying memory map the data is to be written. In some implementations, new data may be written to a local storage area of the virtual PV 156 by default, e.g., to the locally attached physical memory 106, which may provide a "hot" memory optimized for frequent access. In some implementations, the containerized application may specify a particular policy or service level agreement for writing the data, and the memory virtualization system may determine which underlying memory allocation satisfies that policy or SLA. The memory virtualization system then uses the hang point (e.g., 170, 172, 174, 176, or 178) corresponding to this underlying memory allocation to write the data objects. The storage virtualization system also adds the signature of the data objects to the metadata objects of the virtual PV 156.Presenting data in a virtual PV 156 that is natively deduplicated and uniquely identified by a root object signature may enable efficient data services, including those provided by the data service provider 134. For example, the data service provider 134 may perform snapshot-based backups, replications, migrations, tiering, redundancy-based data backup (e.g., redundant array of independent nodes, also referred to as RAIN or RAID), or other functions without limitation. The data service provider 134 may execute the data services with the virtual PV 156 in a manner that is transparent or non-disruptive to the containerized application 124. For example, the data services may be executed without changing the suspension point 180, and in the case of some data services without input or instruction (e.g., configuration details, setup commands, etc.) by a user, containerized application 124, or container orchestrator 122. Moreover, in some examples, the data service provider 134 may manipulate data primarily at the storage virtualization layer 150 such that the data services are executed in a manner independent of the various underlying storage mounts and the particular composition of the virtual PV 156. In other words, the data service provider 134 may consistently and smoothly execute a common set of data services regardless of what type of underlying storage the virtual PV 156 forms. The aforementioned technical advantages may be enabled, for example, by the virtual PV 156 decoupling the underlying memory from the containerized application 124.For example, the data service provider 134 may perform an efficient snapshot-based backup service. In some implementations, the difference between temporal snapshotes of a hierarchically-arranged virtual PV 156 can be efficiently performed by comparing object signatures in an iterative top-down manner starting at the root object to find metadata and data objects that are different. For example, in a process for saving a current state of the virtual PV 156 (i.e., a current snapshot), the current snapshot may be on a primary system (e.g., computer system 100) and an older, previously saved snapshot may already be on a backup system (e.g., computer system 110). In this example, the difference between the current snapshot and the legacy snapshot may be determined by comparing signatures of the snapshot in the manner described above, and the backup system may be searched to determine whether the metadata or data objects that are different are already present on the backup system (i.e., in an object store of the backup system). Only the metadata or data objects that do not exist are copied from the primary system to the backup system, thereby reducing data traffic and improving backup times. In other implementations, snapshot-based fuses may be performed on the same primary system instead of or in addition to the backup system in a similar manner.For example, snapshot-based backup may be performed on a scheduled basis without interrupting the containerized application 124. Moreover, snapshot-based backup may be performed primarily on the software virtualization layer 150, thereby directly avoiding the complexity of managing each individual underlying memory. Similar to the backup process, a restore process can also be performed with a comparison of the metadata or data objects to be restored with the objects already present on the restore destination and a transfer of only those data that are not present on the restore destination.The data service provider 134 may also perform a migration process. The migration may move data objects between different underlying memories within a virtual PV 156, including between different local memories, between local and remote memories, between different remote memories, between different public cloud memories, between public cloud memories and non-public cloud memories (either local or remote), or between other combinations of underlying memories. The migration is handled at the storage virtualization layer 150 by associating the new physical address of a moved data object with the unchanged content-based signature in the object index 154, making the migration transparent to the containerized application 124.As another example, the data service provider 134 may migrate the virtual PV 156 to another computing system. For example, in some cases, it may be useful to migrate a virtual PV 156 to be near a workload that the virtual PV 156 uses. In an example scenario, the container orchestrator 122 may move the containerized application 124 to another computer system (e.g., from the source computer system 100 to the target computer system 110) for load balancing or other reason, and the virtual PV 156 may need to be migrated to be proximate to the containerized application 124. In some implementations, the storage virtualization system 130 may migrate the management of the virtual PV 156 to another storage virtualization system on the target computing system 110 to which the containerized application 124 has been migrated. The data service provider 134 may also migrate some of the data in the virtual PV 156, such as migrating data objects that were local to the computer system 100 (e.g., on the underlying physical memory 106) to the physical memory of the other computer system 110, which may be useful for maintaining the storage locality and other performance characteristics of the virtual PV 156. Such migration may include identifying whether the target computer system 110 already has a copy of the metadata or data objects to be migrated, and transmitting only data objects that do not exist on the target computer system 110. At the same time, the same suspension point 180 for the containerized application 124 may be maintained and uninterrupted.The data service provider 134 may also perform data tiering within a virtual PV 156, i.e., move data between different types of underlying storage that have different characteristics and may correspond to different storage policies. For example, the tiering may be implemented by allocating and / or sharing the constituent memory allocations of the virtual PV 156 among different extents, as described above. Under certain triggering conditions, the data service provider 134 (in conjunction with the volume manager 138, in some implementations) may move data objects from one extent to another extent and update the extent table accordingly. Example triggering conditions may be an elevated security status of data that may cause the data service provider 134 to move that data from the public cloud storage 164 to a non-cloud storage 106, 108, 160, or 162; aging data that may cause the data service provider 134 to move that data into an archiving class of storage (e.g., remote storage 160); recent frequent accesses to data that may cause the data service provider 134 to move that data into a high-performance storage (e.g., local physical storage 106); or other types of conditions. By moving and managing data at the storage virtualization level 150, data tiering may be performed over any type of underlying storage and without interrupting the hang point 180 or the containerized application 124.The data service provider 134 may also support redundancy-based data backup. For example, the data service provider 134 may provide RAID backup. For example, the data service provider 134 (in conjunction with the volume manager 138, in some implementations) may create a RAID set of underlying storage assignments or within an underlying storage assignment (e.g., in cases where the local physical storage 106 includes a set of drives). The data service provider 134 may include or cooperate with a software RAID controller to write objects of the virtual PV 156 to the RAID set according to a RAID scheme such as RAID 1, RAID 5, or RAID 6.The data service provider 134 may also provide RAIN privacy by replicating or mirroring virtual PV 156 (also referred to as primary virtual PV for simplicity) data for purposes of privacy and high availability in accordance with the principles of the RAIN architecture. In some implementations, the data service provider 134 may initially replicate or mirror data when the data is received as a write request from the containerized application 124 at the storage virtualization layer 150. In some implementations, the replicated data may form a virtual PV replicate, which may be similar in shape to a virtual PV (e.g., including a Merchle tree) and may be managed by and local to another storage virtualization system on another computer system 110, relative to the primary virtual PV local to the computer system 100. Additionally or alternatively, the virtual PV replicate may consist of an underlying memory that is different from and / or separate from the underlying memory that consists of the primary virtual PV 156. Thus, if data cannot be recovered on the primary virtual PV, the data may be recovered from the virtual PV replica using a failover method.In summary, by a containerized memory virtualization system 130 that summarizes different memory types into a virtual persistent volume that can be represented as any number of available memory abstractions, the container memory data path can be highly virtualized end-to-end. In this way, users of containerized applications may receive a high degree of flexibility in requesting any composition of the underlying memory to meet the performance requirements (or other requirements) of an application while at the same time being able to utilize the memory using any type of abstraction suitable for the application. Moreover, a consistent set of memory services can be provided, regardless of the composition of the underlying memory and regardless of the type of memory access abstraction used to represent the virtual PV.Turning now to FIG. 2, an example system including a policy engine 242 is shown. FIG. 2 also shows other elements including a container orchestrator 222, a container storage interface 223, a containerized application 224, a data path 232, a container storage interface API 244, and one or more storage plug-ins (e.g., a storage plug-in A 228- 1, a storage plug-in B 228- 2). In some implementations, it may be assumed that the aforementioned elements of FIG. 2 exist within the same container environment. Policy engine 242 and the elements of FIG. 2 (i.e., 222, 223, 224, 232, 244, 228- 1, 228- 2) may each be implemented as hardware or as any combination of hardware and programming to implement their respective functionalities as described herein. For example, the programming may consist of processor-executable instructions stored on a non-transitory machine-readable medium, and the hardware may include a processing resource for fetching and / or executing these instructions. For example, such a machine-readable medium and processing resource may be analogous to the machine-readable medium 104 and processing resource 102 of FIG. 1, respectively, in many respects. The hardware may also include electronic circuitry or logic.Container orchestrator 222 may be analogous to container orchestrator 122 described above in many respects. Container orchestrator 222 may serve as a platform for the provisioning and management of containerized applications, such as containerized application 224, which may be similar to containerized applications 124, 126 described above. Container orchestrator 222 may include container storage interface 223, which may be analogous to container storage interface 123 described above. Container storage interface 223 may enable a plug-in to free up (i.e., make available for use and / or consumption) a storage system for container orchestrator 222 and an associated container environment. For example, the plug-in 228- 1 for memory A may provide the container orchestrator 222 with the storage system 229- 1, and the plug-in 228- 2 for memory B may provide the container orchestrator 222 with the storage system 229- 2. The plug-ins 228- 1 and 228- 2 (also commonly referred to as plug-in(s) 228) may be similar to the plug-ins 128 described above, and each of the storage systems 229- 1 and 229- 2 (also commonly referred to as storage, storage type(s) or storage system(s) 229) may be local storage devices (e.g., local storage devices (e.g., locally attached hard disk drives), local or remote storage arrays, software defined storage, cloud storage, or the like, and may be similar to any of the storage systems or storage types described above, such as physical storage 106, local file system 108, remote storage system 160, virtualized storage 162, or public cloud storage 164.For a storage plug-in 228 deployed in the container orchestrator 222, the container orchestrator 222 may manage an object called a storage class that includes parameters (e.g., in the form of key / value pairs) that describe an associated storage 229. A storage class may describe features of the offered memory (e.g., providable as a persistent storage medium via container memory interface 223), and the described features may include quality of service levels, backup policies, capacity, cost, performance (e.g., IOPS, latency), redundancy, recovery time target (RTO), recovery point target (RPO), or other actions. Different storage classes may describe different storage offers. In some cases, a memory 229 associated with a provider may have multiple classes of memory. Different memories 229 from different providers may have different memory classes. Storage classes may not be standardized between providers or even within a provider, as different storage classes may use different parameters. In FIG. 2, the memory A plug-in 228- 1 may have at least one associated memory class 226- 1 for memory A, and the memory B plug-in 228- 2 may have at least one associated memory class 226- 2 for memory B. Storage classes, such as memory A storage class 226- 1 and memory B storage class 226- 2, may be referred to herein generally as storage class(s) 226. Although two storage plug-ins 228 and two storage systems 229 are shown in FIG. 2 as an example, in various implementations more or fewer storage plug-ins and / or storage classes may be present and available.In some implementations, policy engine 242 may form at least a portion of a memory virtualization system. For example, the policy engine 242 may serve as or form part of the containerized storage virtualization system 130 of FIG. 1, and more particularly, may serve as or form part of the control plane 140 and / or the policy engine 142 of FIG. 1. As will be described, policy engine 242 may be useful, among other things, for virtualizing control plane operations to provide containerized applications (e.g., application 224) with a virtual persistent volume backed up by one or more types of available memory to meet a policy goal, although the available memory types are described by non-standardized memory classes. As described further below, policy engine 242 may also intelligently manage storage operation requests directed to the virtual persistent volume by, among other things, forwarding the storage operation request to the various storage types that support the virtual persistent volume in a multicast method.Policy engine 242 may include and / or interact with one or more elements, each of which may be implemented as hardware or as any combination of hardware and programming to implement its respective functions as described herein. In some implementations, policy engine 242 may include scheduler 246, discovery agent 248, virtual PV manager 250 including provisioning 252 and operation router 253, volume extent and replica manager 254, container storage interface mapping 256, and / or data path manager 258. In some implementations, the policy engine 242 may communicate (e.g., bidirectionally communicate) with the container storage interface 223 via a container storage interface API 244, which may be analogous to the container storage interface API 144 described above. In some implementations, policy engine 242 may communicate with data path 232, which may be analogous to data path 132 described above, via data path manager 258, for example.Discovery agent 248 may determine which plug-ins 228 and associated storage 229 are available in the container environment. Additionally, discovery agent 248 may retrieve storage classes 226 corresponding to discovered plug-ins 228 and storage 229 and register retrieved storage classes 226 with policy engine 242. For example, discovery agent 248 may collect the aforementioned information about plugins 228, memory 229, and memory classes 226 by querying container orchestrator 222 or container memory interface 223.In some implementations, the discovery agent 248 may operate at the start or during an initialization phase of a storage virtualization system to which the policy engine 242 belongs and / or may operate periodically while the storage virtualization system is running. To illustrate, a non-limiting example of a container environment may include three to six types of storage plug-ins 228. In various implementations, there may be one or more storage classes for each plug-in 228.The storage classes 226 registered by the policy engine 242 may not be standardized as described above. In other words, different storage classes may use different parameters, or more specifically, different keys and / or different values for parameters defined as key / value pairs. Policy engine 242 may use a common schema 260 to translate each memory class 226 into a corresponding memory profile. For example, memory class 226- 1 of memory A may be translated into profile 262- 1 of memory A, and memory class 226- 2 of memory B may be translated into profile 262- 2 of memory B. Memory profiles, such as memory A memory profile 262- 1 and memory B memory profile 262- 2, may be referred to generally herein as memory profile(s) 262. In some implementations, the common schema 260 may be manually encoded, e.g., based on the administrator's knowledge of the storage classes 226. The common schema 260 may be understood as a mapping between proprietary storage class parameters and a common language. In some implementations, as described below, the common language of the common schema 260 may be designed in correlation with the creation of the virtual PV storage class 227. In some implementations, the memory profiles 262 may be static, i.e., are not expected to change after translation.To illustrate, the joint schema 260 may be useful when a same level of performance in the memory class 226- 1 of memory A and in the memory class 226- 1 of memory B has the values A and B, respectively, and is thus translated into a same level of performance value X in the profile 262- 1 of memory A and 262- 2 of memory B. Thus, the storage profiles 262- 1 and 262- 2 (translated from storage classes 226- 1 and 226- 2, respectively) use a common language to describe storage offers for memories 229- 1 and 229- 2, respectively.The policy engine 242 may also present or publish one or more virtual PV storage classes 227 to the container orchestrator 222. A virtual PV storage class 227 may be accessible to users in addition to storage classes 226. Similar to the storage classes 226, a virtual PV storage class 227 may describe the characteristics of a providable persistent volume, particularly a persistent volume that may be provided by the policy engine 242 or by a storage virtualization system including the policy engine 242. In some implementations, a virtual PV storage class 227 and a virtual PV provided according to the virtual PV storage class 227 may be considered to have or meet a default or initial storage policy.The container orchestrator 222 may be presented or published with various virtual PV storage classes 227, such as storage classes for various service levels, such as gold, silver, and bronze, each with different parameters. For example, each of the various virtual PV storage classes 227 may be designed by an administrator for various purposes of the virtual storage policy or predefined by a provider providing the policy engine 242. For illustrative purposes only, the virtual PV memory class 227 shown in FIG. 2 has the example characteristic of a "golden" memory class with the parameters type: graded, power: high, and cost: high.Unlike the storage classes 226, the virtual PV storage class 227 is natively compliant with the shared scheme 260. In other words, the key / value pairs of the virtual PV storage class 227 are in the same language as the common schema 260 and need not be translated further. In some implementations, the virtual PV storage classes 227 may be defined first, and the shared schema 260 may be created based on the parameters of the virtual PV storage classes 227.Application 224 may request a persistent volume from container orchestrator 222, either independently or via user input. The request may also include provisioning requests, e.g., indicating that the virtual PV storage class 227 is the desired storage class for the requested persistent volume. Other requests submitted with the request may be, for example, a desired capacity. Container orchestrator 222 may, in turn, forward a request to create volume 230 to policy engine 242, for example, via container storage interface 223 and container storage interface API 244. In some implementations, container orchestrator 222 may extract the parameters of virtual PV storage class 227 and pass the parameters to policy engine 242 with request 230. The request 230 may be received within the policy engine 242 from the virtual PV manager 250.Virtual PV manager 250 may include various operators for virtual PV manipulation managed by policy engine 242, including operators for providing new virtual PVs, for deleting virtual PVs, for handling read / write I / O requests directed to virtual PVs, or other operations. These operators may each be hardware or a combination of hardware and programming to implement their respective functionalities. Virtual PV manager 250 may include, for example, operator 252 to provide new virtual PVs. The virtual PV manager 250 may also include an operation router 253 to route storage operations for existing virtual PVs (e.g., backup or restore operations) to the underlying storage systems (also referred to as backend storage systems). The functionality of virtual PV manager 250 and its provisioning 252 and operation router 253 will now be discussed in more detail.In response to request 230, provisioning system 252 begins creating a virtual PV that is ultimately returned as virtual PV 280 (i.e., including at least one volume identifier) to container orchestrator 222 for consumption or use as persistent storage by containerized application 224. The provisioning unit 252 determines that the virtual PV storage class 227 has been identified in the request 230 as the desired type of virtual PV to be created. The provisioning unit 252 then evaluates each parameter in the requested virtual PV storage class 227 and determines which storage profile 262 has a corresponding parameter that can satisfy and satisfy the parameter of the virtual PV storage class 227. In some implementations, a parameter in the virtual PV storage class 227 is a minimum, such that a closest match of the corresponding parameter among the storage profiles 262 exceeds the parameter value in the storage class 227. This matching of a parameter of the virtual PV storage class 227 with a parameter of a storage profile 262 is enabled by the virtual PV storage class 227 and the storage profiles 262 all conforming to and in language with the common schema 260. For example, if a requested parameter indicates "performance:high", the provisioning unit 252 would identify a storage profile 262 that also contains "performance:high".In some implementations, the provider 252 may match the parameters of the virtual PV storage class 227 with the parameters of the storage profiles 262 based on the best match. For example, if no parameters of storage profiles 262 were found that match or exceed the parameters of virtual PV storage class 227, the closest parameter may be selected. For example, if a requested parameter indicates Power:high, provider 252 may select a Power:Medium storage profile over other Power:Low storage profiles.In some implementations, the provisioning unit 252 may also determine whether the mapping of the virtual PV storage class 227 to a particular storage profile 262 would violate a resource limit or quota of a corresponding plug-in 228, and if an violation were occurring, avoid mapping to that plug-in 228. In some implementations, the provisioning unit 252 may check the remaining capacity of a memory 229 with an out-of-band query command (e.g., not via plug-ins 228, container memory interface API 244, or container memory interface 223).In some implementations, the provisioning unit 252 may determine that multiple storage profiles 262 (e.g., more than one) may be combined to satisfy a virtual PV storage class 227 parameter. For example, a request 230 may request the power:high and 2 terabyte (TB) capacity. In this mapping, the memory profiles 262- 1 and 262- 2 with the performance:high parameters are, however, constrained to 1 TB volumes, either due to remaining capacity or other resource constraints. In this case, the provider 252 may decide to combine 1 TB capacity from each of the storage profiles 262- 1 and 262- 2 to meet the requirement for 2 TB performance:high.In some cases, the provisioning unit 252 may determine that a single storage profile 262 satisfies all of the parameters of the virtual PV storage class 227. In some cases, the provider 252 may determine that multiple different storage profiles 262 are required to satisfy the parameters of the virtual PV storage class 227. For example, a virtual PV memory class 227 may be of a graded type, and may indicate a high performance primary memory (e.g., low latency and / or high IOPS) in a first parameter, and an archive class memory (e.g., low cost and / or high write speed) for backup in a second parameter. In this example, for purposes of illustration, it is assumed that profile 262- 1 for memory A is associated with a high-performance memory array 229- 1 and profile 262- 2 for memory B is associated with an archive / backup memory system, and supplier 252 may determine that the first parameter is satisfied by profile 262- 1 for memory A and the second parameter is satisfied by profile 262- 2 for memory B.Finally, the provisioning unit 252 may map the parameters of the virtual PV storage class 227 to parameters of one or more storage profiles 262 that serve as proxies for the storage classes 226 so that the parameters of the virtual PV storage class 227 are satisfied. Thus, the provisioning unit 252 also identifies the plug-ins 228 (corresponding to the mapped storage profiles 262) to be used to compile a virtual PV with the characteristics of the virtual PV storage class 227.The provider 252 coordinates with the container memory interface Figure 256 to provide volumes of each of the memory plug-ins 228 associated with the depicted profiles 262 using their associated memory classes 226. The provider 252 informs the container storage interface mapping 256 of the profiles 262 and the plug-ins 228 that the provider 252 has identified and mapped to the virtual PV storage class 227, and the container storage interface mapping 256 assumes the low-level replacement of each parameter in the virtual PV storage class 227 with one or more of the provided volumes using the plug-ins 228 identified by the provider 252. In this process, the mapping of container memory interface 256 may translate parameters of virtual PV memory class 227 into parameters in memory classes 226 and then provide volumes using plug-ins 228 and these translated parameters (since plug-ins 228 are compatible with memory classes 226). The provision of volumes using plug-ins 228 can be effected similarly to the provision described above in FIG. 1 using plug-ins 128.To illustrate, if the provider 252 determines that the profile 262- 1 of the memory A satisfies a first parameter of the virtual PV memory class 227, the container memory interface mapping 256 may translate this first parameter into a corresponding parameter of the memory class 226- 1 of the memory A using the common schema 260 and then issue a volume build request 270 to the memory A plug-in 228- 1 specifying the translated parameter of the memory class 226- 1 of the memory A (and in some examples additional parameters such as capacity) via the container memory interface API 244 and the container memory interface 223. In turn, the memory A plug-in 228-1 provides memory from the memory system 229-1 according to request 270 and returns a volume 272-1 that can be identified by a volume identifier or persistent volume object or other control plane means for handling a provided volume. Container storage interface mapping 256 may repeat the process sequentially or in parallel for each parameter of virtual PV storage class 227. Another provisioning request may return a volume identifier or other handle for volume 272- 2 from storage system 229- 2 via storage B plug-in 228- 2. In other examples, more or fewer storage volumes may be returned. Volumes 272- 1 and 272- 2 may collectively be referred to as volumes 272 or generally as volume 272.In some implementations, volume extension and replicate manager 254 and / or data path manager 258 (in conjunction with data path 232) may provide additional functionality to create and / or manage the virtual PV. The functionality of volume extension and replicate manager 254 may cooperate with data path manager 258 / data path 232 or be independent of data path manager 258 / data path 232.In some implementations, data path manager 258 may send provisioning information 266 to data path 232. The provisioning information 266 may be volume information provided via plug-ins 228 to satisfy the PV virtual storage class 227, such as storage type(s), volume identifiers, etc. (e.g., identifying volume information 272). Data path 232 may then consume these provided volumes to create the virtual PV in a similar manner as described above with respect to FIG. 1. In particular, data path 232 may create the virtual PV as an organization (e.g., Merchle tree) of metadata objects and data objects that are stored in object storage and hierarchically connected by associated content-based signatures to a root object. In some implementations, the hierarchically organized object storage supported virtual PV is returned to container orchestrator 222 as virtual PV 280, and data path 232 may be involved in handling I / O requests as described above.An extent, also referred to as a mini-volume, refers to an organization unit within the virtual PV. The virtual PV may include a single extent (e.g., corresponding to a single provided volume) or multiple extents. An extent may be directed to a particular volume provided via a plug-in 228, or a subdivision thereof. The subdivision of the virtual PV into extents may allow I / Os to be efficiently driven to the intended destination, e.g., in the case of animal storage with high-performance hot data animal extents and cold data archive or backup animal extents. In this way, a single virtual PV may have automated built-in tiering functions, making data tiering simple and invisible to users and administrators. In another example, extents may be useful to manage separate volumes to be combined to satisfy a particular virtual PV storage class parameter. The volume extent and replicate manager 254 may manage extents using an extent table in a similar manner as described above with respect to FIG. 1, or offload that management, in whole or in part, to the data path 232.In some implementations, the volume extension and replica manager 254 may coordinate a backup policy specified by the virtual PV storage class 227 throughout the lifecycle of the virtual PV. During the provisioning phase (i.e., after provider 252 identifies primary storage and backup storage from profiles 262 to satisfy virtual PV storage class 227), volume extension and replica manager 254 may coordinate with container storage interface map 256 to first provide a backup volume (e.g., from backup storage system 229- 2 via plug-in 228- 2) and then a primary storage volume (e.g., from primary storage system 229- 1 via plug-in 228- 1) using a provisioning request that identifies the backup volume as a replication target. The storage system hosting the primary storage volume (e.g., storage system 229- 1) may then use its own replication mechanism to backup data to the backup volume identified in the provisioning request (e.g., in storage system 229- 2). In some implementations, a scheduler 246 may operate to schedule backups at a particular frequency or according to a particular schedule (e.g., as defined in virtual PV storage class 227). At each scheduled instance, scheduler 246 may instruct primary storage volume plug-in (e.g., 228-1) to backup data to the target backup volume. In this way, a single virtual PV 280 may contain an automatic backup, thereby facilitating backup configuration and maintenance for users and administrators.Once the one or more volumes have been provided via plug-ins 228 in accordance with the virtual PV storage class 227 mapping to virtual PV storage profile(s) 262, a volume mapping is recorded in the virtualization map 264 data structure that maps the individual volumes provided to the virtual PV to be created. In FIG. 2, virtualization maps 264 show various example mappings for various representations described herein, but it should be understood that virtualization maps 264 may include more or fewer mappings and / or mappings of various types. As an example of an assignment, if a VOL( 2) was provided from the storage A plug-in 228- 1 (e.g., VOL( 2) is a volume identifier for volume 272- 1) and a VOL( 3) was provided from the storage B plug-in 228- 2 (e.g., VOL( 3) is a volume identifier for volume 272- 2) to satisfy the storage class 227 of the virtual PV, then a mapping of VIRTUAL_PV( 1) (volume identifier for virtual PV 280) comprised of VOL( 2) and VOL( 3) is included in the virtualization maps 264 as illustrated in FIG. 2. It is to be understood that the foregoing is an illustration and that a virtual PV may consist of more or less volumes provided. When multiple virtual PVs are requested (e.g., for the same virtual PV storage class or for different virtual PV storage classes), additional assignments are stored in the virtualization card 264 data structure. In this sense, the data structure of the virtualization maps 264 may be considered dynamic because new maps may be added when new virtual PVs are created and existing maps may be deleted when virtual PVs are deleted.Policy engine 242 may return virtual PV 280 by at least sending the virtual persistent volume 281 identifier (e.g., volume ID VIRTUAL_PV(1)) to container orchestrator 222 via container storage interface API 244 and container storage interface 223, and container orchestrator 222 may in turn enable containerized application 224 to access virtual PV 280. The container orchestrator 222 may associate or associate the virtual PV identifier 281 with the virtual PV 280 of the containerized application 224 in the container orchestrator 222 metadata. Subsequently, the containerized application 224 may perform reads and / or writes to the virtual PV 280. The memory virtualization system to which policy engine 242 belongs may service I / O requests directed to virtual PV 280 by communicating those I / O requests with the volumes of memory 229 associated with virtual PV 280.In various implementations and scenarios, the type of virtual PV 280 provided by policy engine 242 may be different. For example, if a single underlying volume is mapped and provided for the virtual PV storage class 227, the policy engine 242 may provide the provided volume (e.g., a volume 272) directly as a virtual PV 280 without including the data path 232. In another example, a single underlying provisioned volume (e.g., volume 272) may be passed to data path 232 and virtualized as described above (e.g., as a hierarchical structure of objects identified by content-based signatures), and this virtualized data path memory is provisioned as virtual PV 280. In another example, multiple underlying volumes (e.g., volumes 272- 1, 272- 2) are mapped and provided for the virtual PV memory class 227, these volumes are virtualized over the data path 232 as described above, and this virtualized memory over the data path is provided as the virtual PV 280. In FIG. 2, a virtual PV 280, shown in phantom, includes, for example, a volume of storage system 229- 1 and a volume of storage system 229- 2 provided with their respective plug-ins 228- 1 and 228- 2.Based on the above, policy engine 242 may automatically identify and construct an optimal or near optimal composition of memory to achieve a policy goal represented in a virtual PV memory class 227 exposed in a container orchestrator 222. Policy engine 242 may utilize an ecosystem of storage plug-ins 228 and storage classes 226 without the intensive manual work of writing code to support each plug-in 228. Moreover, policy engine 242 may improve the mobility of containerized applications, as when a containerized application enters a new container environment, policy engine 242 may automatically construct the correct persistent storage in the new container environment from the storage available and recognizable in that new container environment.Policy engine 242 may also include operation router 253. The operation router 253 may be useful for handling storage operation requests 290 received via the API 244. For example, a user or administrator operating in the container environment in which the container orchestrator 222 is located may attempt to perform a particular operation on existing virtual PVs in the container environment (e.g., virtual PV 280 associated with the application 224). In some examples, the memory operation request 290 may be a standardized request that conforms to a standard specification associated with the container memory interface 223. Example requests for storage operations may include a snapshot operation, a backup operation, a restore operation, a request to create a volume group, or other storage operations.A user or administrator (e.g., via a command line interface, graphical user interface, or the like), an application (e.g., 224), a script in the container environment, or other means for connecting to the container orchestrator 222 may send the storage operation request 290, e.g., via API 244, to a storage virtualization system (e.g., 130) or its policy engine 242. Within the memory virtualization system and the policy engine 242, the operation router 253 receives the memory operation request 290.The memory operation request 290 includes at least one volume identifier. The volume identifier may be a PV virtual volume identifier associated with a virtual PV or a volume group identifier, as described below. In some cases, the memory operation request 290 includes a plurality of virtual PV volume identifiers associated with corresponding virtual PVs. For example, referring to FIG. 2, the storage operation request 290 may include the volume identifier 281 (i.e., VIRTUAL_PV( 1)) of the virtual PV 280. A virtual PV identified by the memory operation request 290 is an existing virtual PV and may consist of one or more underlying memory volumes (e.g., 272- 1, 272- 2) provided by memory systems (e.g., 229- 1, 229- 2) and merged into a hierarchical data structure (e.g., Merchle tree) that relates data objects to a root object by content-based signatures, as described above.The operation router 253 references the virtualization cards 264 to identify a volume map corresponding to the volume identifier included in the storage operation request 290. Assuming the storage operation request 290 is to act on the virtual PV 280 by referencing the virtual PV identifier 281 of the virtual PV 280, the operation router 253 may match this virtual PV identifier 281 to an entry in the virtualization maps 264 (e.g., VIRTUAL_PV( 1)). The operation router 253 then searches the volume mapping for VIRTUAL_PV( 1) in the virtualization maps 264 and identifies the underlying volume identifiers (e.g., VOL( 2), VOL( 3)) corresponding to the underlying storage volumes (e.g., 272- 1, 272- 2) of the virtual PV 280. In some examples, the operation router 253 may recursively check the underlying volumes of an identified volume map to determine whether these underlying volumes have additional volume maps (i.e., interleaved volume maps), such as in an implementation that includes volume groups, as described further below.The operation router 253 then forwards, via the container storage interface API 244 and the container storage interface 223, the storage operation request 290 to each storage system (e.g., 229-1, 229-2) hosting the underlying storage volumes (e.g., 272-1, 272-2), using the underlying volume identifiers (e.g., VOL(2), VOL(3)) as arguments of the respective forwarded storage operation requests. That is, for example, a memory operation request 292 with VOL( 2) as an argument is output to the memory system 229- 1 via the memory A plug-in 228- 1, and a memory operation request 294 with VOL( 3) as an argument is output to the memory system 229- 2 via the memory B plug-in 228- 2.The operation router 253 may forward the plurality of storage operation requests to backend storage systems 229 in parallel, sequentially, or in a combination thereof when there are a plurality of underlying volumes. Apart from different volume identifier arguments, the storage operation requests 292, 294 may be identical (i.e., a copy of) to the received storage operation request 290 in some cases and implementations due to standardization by the container storage interface 223. In such cases, the forwarding of the memory operation request 290 may be understood as memory operation request 292, 294 to issue the same memory operation request with arguments specific to an underlying memory volume.If a store request contains multiple volume identifiers, the store request may also be forwarded to the underlying volumes for each of these volume identifiers in a similar manner. Since the operation router 253 may forward the storage operation request 290 to a plurality of storage systems 229 and is not limited to forwarding the storage operation request 290 to a single storage system 229, the operation router 253 may be understood as a multicast router.After processing requests 292, 294, storage systems 229- 1, 229- 2 may return success or failure messages to operation router 253. In some implementations, the policy engine 242 may return an error message to the container orchestrator 222 when an error message is received. If no error messages are received, the policy engine 242 may return a success message to the container orchestrator 222. In some implementations, the operation router 253 may have capabilities for error handling. The operation router 253 may maintain a transaction log for a storage operation request 290 to be executed on a virtual PV 280. The operation router 253 may track the type of memory operation requests (e.g., 292, 294) forwarded to the backend memory systems (e.g., 229-1, 229-2) for execution against the underlying memory volumes, such as the type of operation forwarded and their arguments. In some implementations, when a backend storage system reports an error in the execution of the forwarded storage operation request, the operation router 253 may reset the state of the virtual PV 280 completely. For example, if the storage operation request 290 was a snapshot request (described below), the operation router 253 may reset the state of the virtual PV 280 by issuing delete requests to each backend storage system that has reported success in creating an underlying storage volume snapshot, thereby deleting the incomplete snapshot.Due to the above, the operation router 253 may process storage operation requests directed to virtual PVs that consist of multiple underlying storage volumes in an efficient and automated manner that is transparent to and without intervention by the users, containerized application 224, and container orchestrator 222.Some non-limiting examples of requests for memory operations are described below. Although the examples are described with reference to elements of FIG. 2, it should be understood that the principles of the examples may be similarly extended to other computing environments and to other virtual PVs that consist of more or less underlying storage volumes.In an example, a snapshot storage operation request 290 (also referred to herein as a snapshot request) may be received from the policy engine 242, e.g., via the API 244, with the virtual PV identifier of the virtual PV 280 (e.g., VIRTUAL_PV( 1)) as an argument to create a snapshot of the virtual PV 280. The operation router 253 matches the request 290 with the volume mapping of VIRTUAL_PV(1):VOL (2), VOL (3) from the virtualization cards 264. The operation router 253 then forwards the snapshot request to the backend storage systems: as snapshot request 292 to storage system 229-1 with VOL(2) as an argument and as snapshot request 294 to storage system 229-2 with VOL(3) as an argument. The storage systems 229- 1 and 229- 2 may perform snapshot (e.g., copy-on-write or reject-on-write snapshot) on volumes identified by VOL( 2) and VOL( 3), and return VOL( 2).SNAP( 1) and VOL( 3).SNAP( 1), respectively, to the policy engine 242. Virtual PV manager 250 may then update the data structure of virtualization maps 264 to modify the volume mapping of the resulting snapshot of virtual PV 280 as VIRTUAL_PV(1).SNAP( 1):VOL( 2). SNAP(1),VOL(3).SNAP (1).In another example, the policy engine 242 may receive a backup type storage operation request 290 (also referred to herein as a backup request), e.g., via the API 244, with the virtual PV identifier of the virtual PV 280 (e.g., virtual_pv( 1)) as an argument to create a full backup of the virtual PV 280 (as opposed to an incremental backup or snapshot). The store operation request 290 may also include a destination (also referred to as a destination) for the backup. The target may be, for example, another storage system in the container environment, in another container environment, a public cloud storage system, or another storage system. In some implementations, a backup by the storage operation request 290 may be an additional and supplemental backup mechanism to the scheduled backups being handled by the volume extension and replica manager 254. After identifying the volume allocation as described above, the operation router 253 forwards the backup request to the backend storage systems: as backup request 292 to the storage system 229- 1 with VOL( 2) and destination as arguments, and as backup request 294 to the storage system 229- 2 with VOL( 3) and destination as arguments. Storage systems 229-1 and 229-2 may then perform a full backup (i.e., a full copy) for the volumes identified by VOL(2) and VOL(3), respectively, to the destination indicated in requests 292, 294. In some implementations, storage systems 229- 1 and 229- 2 may return volume identifiers for the backups, such as VOL(2).BCKP( 1) and VOL(3).BCKP( 1), respectively, and virtual PV manager 250 may then update the data structure of virtualization assignments 264 to include the volume assignment of the resulting backup of virtual PV 280 as VIRTUAL_PV(1).BCKP( 1): VOL( 2). BCKP(1),VOL (3). BCKP(1).In another example, a restore request 290 (also referred to herein as a restore request) may be received from the policy engine 242, e.g., via the API 244, with the virtual PV identifier of the virtual PV 280 (e.g., VIRTUAL_PV(1).SNAP( 1)) as an argument to restore a backup or snapshot of the virtual PV 280. As described above, the operation router 253 may identify the volume identifiers for underlying virtual PV storage volumes indicated in the restore request 290. For example, VIRTUAL_PV(1).SNAP (1) would be identified as mapped to VOL(2).SNAP(1) and VOL(3).SNAP(1).In some implementations, the container environment (i.e., particularly the container storage interface) may support restore-in-place, i.e., snapshot or backup data is restored to the existing virtual PV 280. In restore-in-place implementations, the operation router 253 may forward a restore request 292 to the storage system 229- 1 with VOL( 2).SNAP( 1) as an argument and a restore request 294 to the storage system 229- 2 with VOL( 3).SNAP( 1) as an argument. The storage systems 229- 1 and 229- 2 may then restore a backup or snapshot to VOL( 2) and VOL( 3), respectively, thereby effectively restoring a backup or snapshot of VIRTUAL_PV( 1).In some implementations, the container environment may not support a restore-in-place function. In such implementations, the operation router 253 may issue requests to create volumes to the corresponding backend storage systems using the underlying volume identifiers for each of the underlying storage volumes. For example, the operation router 253 may issue a volume create request 292 to the storage system 229- 1 with VOL( 2).SNAP( 1) as an argument and a volume create request 294 to the storage system 229- 2 with VOL( 3).SNAP( 1) as an argument. Storage system 229- 1 may then create a new volume VOL( 4) from VOL( 2).SNAP( 1), and storage system 229- 2 may create a new volume VOL( 5) from VOL( 3).SNAP( 1). The virtual PV manager 250 may then aggregate (e.g., using the data path 232 as described above) the new volumes VOL( 4), VOL( 5) resulting from the requests to create volumes 292, 294 into a new virtual PV identified as VIRTUAL_PV( 2) and store a new volume map VIRTUAL_PV(2):VOL( 4), VOL( 5) in the virtualization maps 264. VIRTUAL_PV( 1) may still exist next to VIRTUAL_PV( 1). The policy engine 242 may return VIRTUAL_PV( 2) to the container orchestrator 222, for example, in response to the restore request 290.As another example of a store operation, the operation router 253 may receive a store operation request of the "delete" type 290 (also referred to herein as a "delete request"), e.g., via the API 244, with a virtual PV identifier of a virtual PV to be deleted (e.g., virtual_PV( 1) for the virtual PV 280). In response, the operation router 253 may identify the volume mapping from the virtualization cards 264 as described above and issue memory requests for deletion (e.g., 292, 294) to the backend storage systems (e.g., 229- 1, 229- 2) using the underlying volume identifiers from the identified volume mapping (e.g., VOL( 2), VOL( 3)). If these memory delete requests are successful, the virtual PV manager 250 may delete the virtual PV identifier (e.g., VIRTUAL_PV(1)) from the data structure of the virtualization cards 264.In some implementations, the virtual PV manager 250 may include a volume group creation function. For example, policy engine 242 may receive, via API 244, a volume group create storage operation request 290 (also referred to herein as a volume group create request) that includes the volume identifiers of the virtual PVs to be grouped, such as, for purposes of illustration, VIRTUAL_PV( 1) and VIRTUAL_PV( 2) (which may be different from the VIRTUAL_PV( 1) and VIRTUAL_PV( 2) described above in other examples). In response to the volume group create request 290, the virtual PV manager 250 may create a new volume mapping in the virtualization mapping data structure 264 that maps a new volume group identifier to the plurality of virtual PV identifiers: e.g., VOL_GROUP(A):VIRTUAL_PV( 2). The new volume group identifier VOL_GROUP(A) may be returned to the container orchestrator 222.A volume group identifier may function in many ways similar to a virtual persistent volume identifier, particularly with respect to memory operation requests, and both may therefore be commonly referred to as volume identifier in some implementations. Thus, various aspects of handling a virtual persistent volume identifier described herein may be similarly applied to a volume group identifier.For example, a store operation request 290 with the group identifier VOL_GROUP(A) as an argument may be received from the policy engine 242, for example, via the API 244. In response, the operation router 253 may identify the VOL_GROUP(A) volume mapping from the virtualization maps 264 and recursively check the volumes mapped to VOL_GROUP(A) until the underlying volumes of the backend storage systems have been identified. For example, the operation router 253 may determine that the VOL_GROUP(A) includes the virtual volumes VIRTUAL_PV( 1) and VIRTUAL_PV( 2). By further checking virtualization maps 264, operation router 253 may determine that VIRTUAL_PV( 1) and VIRTUAL_PV( 2) each include underlying volumes (e.g., VIRTUAL_PV( 1)_VOL( 2) and VOL( 3)). Thus, it can be assumed that one or more levels of volume assignments are nested within the group identifier type of the identifier submitted with request 290. Once the operations router 253 has identified the underlying volume identifiers, the operations router 253 may forward the storage operation request 290 to the backend storage systems for those underlying volumes. In this way, volume groups may be used for efficient and consistent memory management across multiple virtual PVs and may also be used for policy-based memory management.FIGS. 3-6 are flowcharts illustrating various example methods. In some implementations, one or more blocks of the methods may be performed substantially simultaneously or in a different order than shown. In some implementations, a method may include more or fewer blocks than illustrated. In some implementations, one or more of the blocks of a method may be contiguous and / or repeating at certain times. In some implementations, the blocks of the methods may be combined.The methods illustrated in FIGS. 3-6 may be implemented in the form of executable instructions stored on a machine readable medium and executed by a processing resource and / or in the form of electronic circuitry. For example, aspects of the methods may be described below as being executed by a memory virtualization system, an example of which may be the containerized memory virtualization system 130 running on a hardware processor resource 102 of the computer system 100 described above. Aspects of the methods may further be attributed to a policy engine of such a memory virtualization system, such as policy engine 242 described above. In addition, other aspects of the methods described below may be described with reference to other elements shown in FIG. 2 for non-limiting illustrative purposes.FIG. 3 is a flow diagram illustrating an example method 300. The method 300 begins at block 302 and continues at block 304 where a memory virtualization system receives a memory operation request 290 that includes a volume identifier. The volume identifier may be of various types, such as a virtual persistent volume identifier (e.g., virtual PV identifier 281, VIRTUAL_PV(1) associated with virtual PV 280), or a volume group identifier representing a group of virtual PVs. The storage operation request 290 may be received from a container orchestrator 222 via a container storage interface 223 and its API 244.At block 306, the memory virtualization system identifies a volume map corresponding to the volume identifier received with the memory operation request at block 304. For example, the storage virtualization system may reference a data structure virtualization cards 264 for volume assignments.At block 308, based on the volume assignment identified at block 306, the memory virtualization system identifies underlying volume identifiers for underlying storage volumes that form at least a portion of a virtual PV associated with the volume identifier included in the received memory operation request. For example, in the illustration of FIG. 2, the memory virtualization system identifies the underlying volume identifiers VOL( 2) and VOL( 3) mapped to VIRTUAL_PV( 1), the volume identifier of the memory operation request 290. The underlying volumes identified by VOL(2) and VOL(3) form at least a portion of the virtual PV 280 that is associated with VIRTUAL_PV(1).At block 310, the memory virtualization system forwards the memory operation request 290 to the backend memory systems 229- 1, 229- 2 on which the underlying memory volumes are located, respectively, using the identifiers of the underlying volumes identified at block 308. The method 300 ends in block 312. In this way, requests for storage operations on a virtual PV that is comprised of multiple underlying volumes can be processed seamless and transparent to the requester. In this manner, various types of memory operation requests may be handled, including snapshot requests, backup requests (e.g., full backup), restore requests, erase requests, or other requests.FIG. 4 is a flow diagram illustrating an example method 400 that may be useful for handling memory operation requests of the "Restore" type in environments where "Restore-in-Place" is not supported. The method 400 begins at block 402 and proceeds to block 404. In block 404, the memory virtualization system receives a memory operation request that is a restore request. The restore request includes a volume identifier of the volume to be restored (e.g., VIRTUAL_PV(1).SNAP (1)).At block 406, the memory virtualization system identifies a volume map corresponding to the volume identifier received at block 404 and identifies the underlying volume identifiers based on this volume map (e.g., VOL( 2).SNAP( 1) and VOL(3).SNAP( 1)). Block 406 may be analogous to blocks 306, 308 described above in many respects.At block 408, the storage virtualization system issues requests to the corresponding storage systems 229 to create volumes using each of the underlying volume identifiers identified at block 406. Issuing the requests to create volumes may be understood as forwarding the store operation request to the backend store, as described in block 310 above. The backend storage systems 229 then create new volumes from the underlying volumes identified by the underlying volume identifiers passed as arguments with the volume creation requests (e.g., by cloning). The backend storage systems 229 may return new volume identifiers for the newly created volumes to the storage virtualization system (e.g., VOL( 4) created from VOL( 2).SNAP( 1) and VOL( 5) created from VOL(3).SNAP( 1)).At block 410, the memory virtualization system aggregates, e.g., via a data path, new volumes resulting from the issue of the requests to create volumes at block 408 into a new virtual persistent volume representing recovered data. The storage virtualization system may add a new volume map for the newly aggregated virtual PV to the virtualization maps 264 (e.g., VIRTUAL_PV(2):VOL( 4), VOL( 5)). The method 400 ends in block 412.FIG. 5 is a flow diagram illustrating an example method 500. In some implementations, the method 500 may be performed after block 304 of the method 300 and may be useful for fault handling during execution of the method 300. Method 500 begins at block 502 and continues until block 504. In block 504, the memory virtualization system maintains a transaction log for a memory operation request 290. As with the methods described above, the storage operation request 290 includes a volume identifier associated with a virtual PV or a group of virtual PVs on which to perform the storage operation. Underlying volume identifiers can be identified from the volume identifier via a volume assignment as described above.Block 506 may be analogous to block 310 described above in many respects. At block 506, the memory virtualization system may forward the memory operation request to backend storage systems 229 using the underlying volume identifiers (e.g., identified in a similar manner as at blocks 306, 308). Each of the backend storage systems independently processes the forwarded storage operation requests 292, 294. The backend storage systems 229 return either a success or an error message to the storage virtualization system. The transaction log includes one or more of the following information: the forwarded storage operation requests 292, 294; success or failure messages returned by the backend storage systems 229; volume identifiers returned by the backend storage systems (e.g., if the forwarded storage operation was successful, e.g., in a restore or snapshot operation); or other information resulting from the storage operation requests forwarded to the backend storage.At block 508, the memory virtualization system determines whether an error has occurred during execution of a forwarded memory operation request. For example, the storage virtualization system checks whether a backend storage system 229 has returned an error message. If no error messages have been returned ("NO" in block 508), the method 500 may end in block 512.If an error has occurred ("YES" in block 508), the memory virtualization system in response returns a state of the virtual PV based on the transaction log. The rolling back may include, for example, resetting to a state of the underlying storage volumes of the virtual PV prior to executing the forwarded storage operation requests. The rolling back may include issuing memory operation requests to the backend memory that undo all successful memory operation requests forwarded from block 506. Illustratively, if a forwarded store operation request has created a snapshot, the snapshot volume identifier is stored in the transaction log, and rolling back the virtual PV state involves issuing a delete request for the snapshot volume identifier. After rolling back the virtual PV, the method 500 ends in block 512.In some implementations, the memory virtualization system may return a success (if "NO" in block 508) or an error (if "YES" in block 508) to the requesting instance that issues the memory operation request 290 in the first instance. In this manner, the memory operation request 290 may be considered completed only when all forwarded memory operation requests 292, 294 have been successfully completed to prevent partially completed operations leaving a virtual PV in an inconsistent state. Such fault handling by method 300 may be useful to make storage operations on composite virtual PVs transparent to users and applications.FIG. 6 is a flow diagram illustrating an example method 600. In some implementations, the method 600 may be useful to execute a store operation request consistently across a group of virtual PVs. The method 600 begins at block 602 and proceeds to block 604, where the memory virtualization system receives a request to create a volume group that includes a plurality of virtual PV identifiers.In response to this volume group creation request, the storage virtualization system creates a new volume map that includes a volume group identifier associated with the plurality of virtual PV identifiers (e.g., VOL_GROUP(A):VIRTUAL_PV( 1), VIRTUAL_PV( 2)) at block 606. This new volume map may be stored in a data structure with virtualization maps 264.At a later time, the memory virtualization system may receive a memory operation request 290 that includes a volume identifier, also referred to as a volume identifier for lookup for the purposes of FIG. 6, at block 608. The volume identifier received with the memory operation request 290 may be a volume group identifier (e.g., VOL_GROUP(A)).At block 610, the memory virtualization system identifies a volume mapping corresponding to the looked up volume identifier. At block 612, the memory virtualization system identifies constituent volume identifiers (e.g., VIRTUAL_PV( 1), VIRTUAL_PV( 2)) associated with the look-up volume identifier (e.g., VOL_GROUP(A)).At block 614, the memory virtualization system determines whether an identifier of a constituent of the volume identified at block 612 is associated with another volume mapping (i.e., an interleaved volume mapping). In particular, the storage virtualization system may determine whether a constituent volume identifier is present in the virtualization cards 264. If not, the method 600 proceeds to block 618, where the memory virtualization system forwards the memory operation request to the identified constituent volume identifiers in a manner corresponding to the block 310 described above.If a constituent volume identifier is associated with another volume assignment ("YES" at block 614, such as would be the case with VIRTUAL_PV(1) and VIRTUAL_PV(2), the constituent volume identifiers are from VOL_GROUP(A)), the method 600 proceeds to block 616, where the constituent volume identifier (e.g., VIRTUAL_PV(1) and VIRTUAL_PV(2)) becomes the next trailing volume identifier. The method 600 then continues to identify constituent volume identifiers for the next persistent volume identifier (e.g., VOL( 2), VOL( 3) for VIRTUAL_PV( 1) and VOL( 4), VOL( 5) for VIRTUAL_PV( 2)).Blocks 610, 612, 614, 616 as a whole recursively check whether a volume map contains another volume map to another volume identifier and identify another volume map for the other volume identifier until the underlying volume identifiers for each of the underlying storage volumes are determined. Blocks 610, 612, 614, 616 may be implemented as part of blocks 306 and / or 308 described above.Illustrated in FIGS. 7 and 8 are example systems 700 and 800, respectively, that include non-transitory machine readable media 704 and 804, respectively, encoded with example instructions executable by the processing resources 702 and 804, respectively. In some implementations, the systems 700, 800 may be useful to implement aspects of the memory virtualization system 130 of FIG. 1 or the policy engine 242 of FIG. 2, or to perform aspects of the methods 300, 400, 500, and / or 600 of FIGS. 3-6. For example, the instructions encoded on machine-readable media 704 and / or 804 may be included in instructions 105 of FIG. 1. In some implementations, the functionality described with respect to FIG. 2 may be included in the instructions encoded on machine-readable media 704 and / or 804.The processing resources 702, 802 may include a microcontroller, a microprocessor, central processor core(s), an ASIC, an FPGA, and / or other hardware device suitable for retrieving and / or executing instructions from the machine readable media 704, 804 to perform functions with respect to various examples. Additionally or alternatively, the processing resources 702, 802 may include or be coupled to electronic circuitry or dedicated logic for executing some or all of the functions of the instructions described herein.The machine readable media 704, 804 may be any medium suitable for storing executable instructions, such as RAM, ROM, EEPROM, flash memory, a hard disk drive, an optical disk, or the like. In some example implementations, the machine readable media 704, 804 may be a tangible, non-transitory medium. Machine-readable media 704, 804 may be disposed in systems 700, 800, respectively, where executable instructions may be considered installed or embedded on the system. Alternatively, the machine readable media 704, 804 may be a portable (e.g., external) storage medium that may be part of an installation package.As described further below, the machine readable media 704, 804 may be encoded with a set of executable instructions. It should be appreciated that some or all of the executable instructions and / or electronic circuits included in a package may be included in another package shown in the figures or in another package not shown in alternative implementations. Some implementations may include more or fewer commands than are shown in FIGS. 7 and 8.Referring to FIG. 7, the machine readable medium 704 includes instructions 706, 708, 710. Instructions 706, when executed, cause processing resource 702 to receive from a container orchestrator a store operation request that includes a volume identifier. The store operation request may be a save request, a snapshot request, a restore request, an erase request, or another type of request. The volume identifier may be a virtual PV identifier associated with a virtual PV comprised of at least one underlying storage volume provided by storage systems. The volume identifier may be a volume group identifier consisting of multiple virtual PVs.Instructions 708, when executed, cause the processing resource 702 to identify a volume mapping corresponding to the volume identifier. Instructions 708 may also cause the processing resource 702 to identify underlying volume identifiers for the one or more underlying storage volumes that form at least a portion of a virtual PV associated with the volume identifier included in the received storage operation request. In some implementations, instructions 708 may act recursively on the volume identifier (e.g., in the manner described in FIG. 6 ), particularly when a volume map is associated with other volume maps, which may be the case when the volume identifier received with the memory operation request was a volume group identifier.Instructions 710, when executed, cause processing resource 702 to forward the memory operation request to each memory system corresponding to an underlying memory volume using an identifier of the underlying volume identified by instructions 708.Referring to FIG. 8, the machine readable medium 804 includes instructions 806, 808, 810, 812, 814. Instructions 806, when executed, cause the processing resource 802 to respond to a store operation request, which is a restore request, by issuing requests to corresponding backend storage systems for creating volumes using underlying volume identifiers for each underlying volume. Instructions 806 then cause the processing resource 802 to aggregate new volumes resulting from the requests to create volumes into a new virtual persistent volume representing recovered data. The instructions 806 may be executed as part of the instructions 710 to pass a store operation request of the "restore" type, particularly in an environment that does not support, e.g., "restore-in-place.".Instructions 808, when executed, cause the processing resource 802 to respond to a request to create a volume group by creating a new volume mapping that includes a volume group identifier associated with a plurality of virtual PV identifiers.The instructions 810, 812, 814 may be useful for fault handling. The instructions 810, when executed, cause the processing resource 802 to maintain a transaction log for a store operation request. Instructions 812, when executed, cause processing resource 802 to determine that an error has occurred in the execution of a forwarded storage operation request in a particular storage type system hosting an underlying storage volume. Instructions 814, when executed, cause the processing resource 802 to respond to such an error by resetting a state of the virtual PV based on the transaction protocol.

Claims

A non-transitory machine-readable medium storing instructions for a policy engine (142, 242) of a memory virtualization system (130) that, when executed, cause a processing resource (102) to: receive, from a container orchestrator (122, 222), a memory operation request (290, 292, 294) including a volume identifier, the volume identifier being a virtual persistent volume identifier associated with a virtual persistent volume (156, 158, 230, 272) comprised of at least one underlying memory volume (156, 158, 230, 272) provided by memory systems (160, 229), the virtual persistent volume (156, 158, 230, 272), 272) consisting of the at least one underlying storage volume (156, 158, 230, 272) merged into a hierarchical data structure that relates data objects to a root object by content-based signatures; identifying a volume mapping corresponding to the volume identifier, the volume mapping indicating an underlying volume identifier for each of the at least one underlying storage volumes (156, 158, 230, 272); and forwarding the storage operation request (290, 292, 294) to each storage system (160, 229) corresponding to the at least one underlying storage volume (156, 158, 230, 272) using the underlying volume identifier.The non-transitory machine readable medium of claim 1, wherein the memory operation request (290, 292, 294) is a snapshot request.The non-transitory machine-readable medium of claim 1, wherein the storage operation request (290, 292, 294) is a restore request (290, 292, 294), and the instructions, when executed, cause the processing resource (102) to, in response to the storage operation request (290, 292, 294) being a restore request (290, 292, 294),: issue a request (230, 270) to generate a volume using the underlying volume identifier for each of the at least one underlying storage volumes (156, 158, 230, 272) to each corresponding storage type system as part of forwarding the storage operation request (290, 292, 294) and issue new volumes resulting from the create volume request, aggregating to a new virtual persistent volume (156, 158, 230, 272) representing recovered data.The non-transitory machine-readable medium of claim 1, wherein the instructions, when executed, cause the processing resource (102) to respond to a volume group generation request (290) by generating a new volume mapping including a volume group identifier associated with a plurality of virtual persistent volume identifiers, wherein to identify the volume mapping, the instructions cause the processing resource (102) to recursively check whether the volume mapping includes another volume mapping to virtual persistent volume identifiers until the underlying volume identifier is determined for each of the at least one underlying storage volumes (156, 158, 230, 272).The non-transitory machine readable medium of claim 1, wherein the memory operation request (290, 292, 294) is a full backup request (294).The non-transitory machine-readable medium of claim 1, wherein the instructions, when executed, cause the processing resource (102) to maintain a transaction log for the storage operation request (290, 292, 294); determine that an error has occurred in execution of the storage operation request (290, 292, 294) on a given storage system (160, 229) of the at least one underlying storage volume (156, 158, 230, 272); and in response to the determination of the error, reset a state of the virtual persistent volume (156, 158, 230, 272) based on the transaction log.A method comprising: receiving, by a hardware processor-based memory virtualization system (130) in a container environment, a memory operation request (290, 292, 294) from a container orchestrator (122, 222) that includes a volume identifier, wherein a type of the volume identifier is a virtual persistent volume identifier; identifying, by the memory virtualization system (130), a volume mapping corresponding to the volume identifier; identifying, by the storage virtualization system and based on the identified volume mapping, underlying volume identifiers for underlying storage volumes (156, 158, 230, 272) that form at least a portion of a virtual persistent volume (156, 158, 230, 272) associated with the volume identifier included in the received storage operation request (290, 292, 294), the virtual persistent volume being composed of the underlying storage volumes (156, 158, 230, 272) merged into a hierarchical data structure that relates data objects to a root object by content-based signatures; Forwarding the memory operation request (290, 292, 294) by the memory virtualization system (130) using the underlying volume identifiers to memory systems (160, 229) on which the underlying memory volumes are respectively located, wherein forwarding the memory operation request (290, 292, 294) is performed via a container storage interface application programming interface.The method of claim 7, wherein the store operation request (290, 292, 294) is a snapshot request.The method of claim 7, wherein the store operation request (290, 292, 294) is a full backup request (294).The method of claim 7, further comprising responding to the store operation request (290, 292, 294) being a restore request (290, 292, 294) by: issuing a volume generation request (230, 270) using each of the underlying volume identifiers to a corresponding one of the storage systems (160, 229) as part of forwarding the store operation request (290, 292, 294) and aggregating new volumes resulting from issuing the volume generation request (230, 270) for each of the underlying volume identifiers into a new virtual persistent volume (156, 158, 230, 272) representing restored data.The method of claim 7, further comprising: responding to a create volume group request (290) by generating a new volume mapping that includes a volume group identifier associated with a plurality of virtual persistent volume identifiers, wherein identifying the underlying volume identifiers includes a recursive check of whether the identified volume mapping includes a further volume mapping to a different volume identifier, and identifying a further volume mapping for the different volume identifier until the underlying volume identifiers are determined for each of the underlying storage volumes (156, 158, 230, 272).The method of claim 7, further comprising: maintaining a transaction log for the storage operation request (290, 292, 294); determining that an error has occurred during execution of the storage operation request (290, 292, 294), forwarded to a storage system (160, 229), of an underlying storage volume (156, 158, 230, 272) of the underlying storage volumes; and responsive to the determination of the error, rolling back a state of the virtual persistent volume (156, 158, 230, 272) based on the transaction log.A system, comprising: a hardware processing resource (102); and a non-transitory machine readable medium storing instructions that, when executed, cause the hardware processing resource (102) to: receive a storage operation request (290, 292, 294) from a container orchestrator (122, 222) that includes a volume identifier, wherein a type of the volume identifier is a virtual persistent volume identifier to identify a volume mapping corresponding to the volume identifier, identify, based on the identified volume mapping, underlying volume identifiers for underlying storage volumes (156, 158, 230, 272) that form at least a portion of a virtual persistent volume (156, 158, 230, 272), which is associated with the volume identifier contained in the received store operation request (290, 292, 294), wherein the virtual persistent volume (156, 158, 230, 272) consists of the underlying store volumes (156, 158, 230, 272) merged into a hierarchical data structure that relates data objects to a root object by content-based signatures; To forward the memory operation request (290, 292, 294) to the memory systems (160, 229) on which the underlying memory volumes (156, 158, 230, 272) are located using the underlying volume identifiers, the memory operation request (290, 292, 294) being forwarded to the memory systems (160, 229) via a container storage interface application programming interface.The system of claim 13, wherein the store operation request (290, 292, 294) is a snapshot request or a full backup request (294).The system of claim 13, wherein the storage operation request (290, 292, 294) is a restore request (290, 292, 294), and the instructions, when executed, cause the hardware processing resource (102) to respond to the restore request (290, 292, 294) by: issuing a volume generation request (230, 270) using each of the underlying volume identifiers to a corresponding one of the storage systems (160, 229) as part of forwarding the storage operation request (290, 292, 294) and aggregating new volumes resulting from issuing the volume generation request "(230, 270) for each of the underlying volume identifiers into a new virtual persistent volume (156, 158, 230, 272), represents the recovered data.The system of claim 13, wherein the instructions, when executed, cause the hardware processing resource (102) to respond to a volume group generation request (290) by generating a new volume map including a volume group identifier associated with a plurality of virtual persistent volume identifiers, wherein the instructions that cause the hardware processing resource (102) to identify the underlying volume identifiers include instructions to recursively check whether the identified volume map includes a further volume map to a different volume identifier and identify a further volume map to the different volume identifier until the underlying volume identifiers for each of the underlying storage volumes (156, 158, 230, respectively, 272).The system of claim 13, wherein the instructions, when executed, cause the hardware processing resource (102) to: maintain a transaction log for the storage operation request (290, 292, 294); determine that an error has occurred in execution of the storage operation request (290, 292, 294), forwarded to a storage system (160, 229), of an underlying storage volume (156, 158, 230, 272) of the underlying storage volumes (156, 158, 230, 272); and reset a state of the virtual persistent volume (156, 158, 230, 272) based on the transaction log in response to the determination of the error.

Images