Non-transitory, machine-readable medium, method, and system for changing virtual persistent volumes based on performance metrics analysis

Abstract

A non-transitory, machine-readable medium (104) comprising instructions (105) for a storage virtualization system (130) in a container environment (120), the instructions (105), when executed, causing a processing resource (102) to: to identify a first application data storage profile (284-1) from a plurality of preconfigured application data storage profiles (284), each containing preconfigured storage parameters for a corresponding application type, based on a comparison of information in the plurality of application data storage profiles (284) with information in an application manifest (282) of a containerized application (124, 126, 224) assigned a first virtual persistent volume (156, 158, 280), wherein the information in the application manifest (282) compared to the information in the plurality of application data storage profiles (284) includes a name of a container image on which the containerized application (124, 126, 224) is based, and wherein each identified application data storage profile of the plurality of application data storage profiles (284) identifies a respective collection of performance metrics (292, 294, 296),which are relevant for the optimization of virtual persistent volumes for the respective application type; , receive values ​​of first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) for use by an optimizer (253) in optimizing the first virtual persistent volume (156, 158, 280), wherein the first performance metrics (292, 294, 296) relate to the use of the first virtual persistent volume (156, 158, 280) by the containerized application (124, 126, 224) and are a subset of the available metrics to be collected; and modifying the first virtual persistent volume (156, 158, 280) by the optimizer (253) based on an analysis of the first performance metrics (292, 294, 296) identified by the first application, wherein modifying the first virtual persistent volume (156, 158, 280) comprises: enabling caches from a disabled state by changing a cache from the disabled state to an enabled state for the first virtual persistent volume (156, 158, 280) based on the optimizer's (253) analysis of the first performance metrics (292, 294, 296).

Description

BACKGROUND

[0001] Containers are a type of virtualization. A container can contain an application packaged together with dependencies and libraries. A containerized application can use or create persistent data. US 2013 / 0 297 871 A1 describes a computer program product with a computer-readable medium on which computer program logic for providing data access in a storage system is materially recorded.US 2012 / 0 317 355 A1 describes a storage device connected via a network to a host computer requesting a data write operation, comprising multiple types of storage devices with different performance, and a controller that manages the storage areas provided by each of the multiple types of storage devices using storage tiers of multiple different types and allocates the storage areas in page units to virtual disks from any of the multiple types of storage tiers in accordance with a data write request from the host computer. US 2004 / 0 123 062 A1 describes a method, a system, and a computer program product that enables a user to provide a general specification of storage requirements without having detailed knowledge of underlying vendor-specific APIs and command-line programs for each possible storage device used to fulfill the storage requirements.It is an object of the invention to propose a non-transitory, machine-readable medium, a method, and a system for modifying virtual persistent volumes based on the analysis of performance metrics. This object is achieved by a non-transitory, machine-readable medium according to claim 1, a method according to claim 8, and a system according to claim 15. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various examples are described below with reference to the following figures. Fig. Figure 1 shows an example system that creates a virtual persistent volume that aggregates a variety of underlying storage and presents it to a containerized application. Fig. Figure 2 shows an example system that modifies a virtual persistent volume based on performance metrics related to the usage of the virtual persistent volume. Fig. 3 shows an example method that includes modifying a virtual persistent volume based on performance metrics related to the usage of the virtual persistent volume. Fig. Figure 4 shows an example method that includes analyzing performance metrics to identify a storage class and provisioning a new underlying storage volume using the identified storage class. Fig. 5 shows an example system with machine-readable media containing instructions for modifying a virtual persistent volume based on performance metrics related to the usage of the virtual persistent volume. DETAILED DESCRIPTION

[0003] Container technology is a paradigm of computer virtualization in which an application, along with dependencies and libraries, is packaged in a container to create an isolated environment for running the application. Such an application can be referred to as a containerized application. Many containers can run on top of a single operating system, but each container is inherently isolated from other containers. In this way, the container paradigm can be thought of as the virtualization of the operating system. Containers can be lighter than other forms of virtualization, such as virtual machines that virtualize hardware. For example, each virtual machine can have its own copy of an operating system kernel, whereas, in contrast, multiple containers can share an operating system kernel.

[0004] Containerized applications may require storage to store persistent data. Container orchestrators (e.g., Kubernetes) can provide the ability to provision some storage to a container. However, there are many types of storage, including but not limited to storage that is local to the container, remote to the container, hardware-based (e.g., a locally attached drive), software-defined (e.g., a file system, virtualized or containerized storage, storage provided via an API, etc.), or storage that has a combination of the above. Previous efforts to provide container storage may not provide the flexibility of storage configuration and data services that users and administrators of container environments desire. For example, a container orchestrator may be limited to providing one type of storage to an application.Other systems may attempt to concatenate multiple disks into a single disk, but such concatenation may not have the flexibility to provide certain data services without disrupting user access to the data.

[0005] To address these challenges and provide flexible and robust persistent storage accessible to containers, the examples described herein refer to a storage virtualization system and policy engine that can create a virtual persistent volume that aggregates a variety of underlying storage volumes provided by various types of storage systems available in a container environment. The virtual persistent volume can take the form of a hierarchical structure, such as a Merkle tree, that relates data objects of the containerized application through their content-based signatures back to a root object.

[0006] After the initial creation of the virtual persistent disk, the storage-related performance requirements of a containerized application may change due to increased application load (e.g., increased read or write IOPS (input / output operations per second), increasing latency, etc.), application autoscaling, or a change in the nature of the application workload (e.g., switching from write-oriented to read-oriented operations or vice versa, switching to archive-oriented operations, etc.). In such situations, it would be useful if the persistent storage of a containerized application could be automatically optimized in response to the application's changing storage-related performance requirements.

[0007] Accordingly, further examples described herein relate to the storage virtualization system monitoring and analyzing performance metrics of the containerized application and / or the storage virtualization system itself (e.g., particularly with respect to a datapath component thereof that handles operations of the virtual persistent volume). The storage virtualization system may modify the virtual persistent volume based on the performance metrics. For example, in one implementation, the storage virtualization system may identify a storage class or classes available in the container environment that may more optimally meet the performance requirements of the containerized application determined through an analysis of the performance metrics.The storage virtualization system can then provision one or more new underlying storage volumes from these identified storage classes and modify the virtual persistent volume to include these new underlying storage volumes. Based on the above, the storage virtualization system can automatically (e.g., without human intervention) and periodically (e.g., in real-time, near-real-time, on a scheduled basis, etc.) optimize the performance of a virtual persistent volume to address the storage requirements of an associated containerized application.

[0008] Fig. 1 shows an example of a computer system 100 that supports and participates in a container environment 120. The computing system 100 includes a processing resource 102, which may include a microcontroller, a microprocessor, one or more central processing cores, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc. The computer system 100 includes a machine-readable medium 104, which may be non-volatile and may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, a hard disk drive, etc.

[0009] The processing resource 102 can execute instructions 105 (i.e., programming or software code) stored on the machine-readable medium 104 to perform functions of the computer system 100, such as providing a container orchestrator 122 and a containerized storage system 130, as described further below. In particular, each of the components of the containerized storage virtualization system 130 can be implemented as executable instructions 105, including the data service provider 134, the storage virtualizer 136, the volume manager 138, the adapters 139, the policy engine 142, and the container storage interface API 144. Containerized applications 124, 126 and container storage interface plug-ins 128 can also be implemented as instructions in the executable instructions 105.Additionally or alternatively, the processing resource 102 may include electronic circuitry for performing the functions described herein.

[0010] Computer system 100 may also include other hardware components, such as physical memory 106. Physical memory 106 may comprise any physical storage device, such as a hard disk drive, a solid-state drive, or the like, or a plurality of such storage devices (e.g., an array of hard disks), and may be locally attached (i.e., installed) to computer system 100. In some implementations, physical memory 106 may be accessed as a block storage device.

[0011] In some cases, computer system 100 may also include a local file system 108, which may be implemented as a layer above physical storage 106. For example, an operating system 107 may be running on computer system 100 (due to processing resource 102 executing certain instructions 105 related to the operating system), and operating system 107 may provide a file system 108 to store data on physical storage 106.

[0012] Computing system 100 may communicate with other computing systems, such as computing system 110, for example, via a wired and / or wireless network. The other computing systems may be similar to computing system 100 and may each include at least one processing resource and a machine-readable medium. Computing systems 100 and 110 may each execute software (i.e., processing resource 102 executes specific instructions 105) to deploy nodes of a container orchestrator 122. In other words, container orchestrator 122 may have a cluster architecture that includes the container orchestrator nodes of computing systems connected in a cluster. Container orchestrator 122 acts as a platform for deploying and managing containerized applications within the cluster of computing systems.The container orchestrator 122, the containerized applications it provides, and other container resources (such as container storage) are considered part of a container environment 120. In contrast, other elements may operate outside of the container environment 120, such as the local file system 108 and an operating system 107 of the computing system 100.

[0013] In Fig. 1, for illustrative purposes, containerized applications 124 and 126 are deployed locally on computing system 100 via container orchestrator 122. Applications 124, 126 can therefore be said to be remote from other nodes, such as computer system 110. Containerized applications 124, 126 may represent microservices, user applications, or the like and are sometimes simply referred to as containers. Container environment 120 may also host containerized applications (not shown) that are local to other computer systems in the cluster.

[0014] The container orchestrator 122 also employs a containerized storage virtualization system 130, which is described in more detail below. In the example of Fig. 1, the containerized storage virtualization system 130 is also executed locally on the computer system 100 and its processing resource 102. In some implementations, other containerized storage virtualization systems of other computer systems in the cluster are also included in the container environment 120, although they are Fig. 1 are not shown. For example, the containerized storage virtualization system 130 may serve as a node in a storage virtualization platform having a clustered architecture in which multiple containerized storage virtualization systems (at least some of which may be running on different and separate physical computing systems) cooperate to store and manage data in a storage virtualization layer.

[0015] The container orchestrator 122 may also include a standardized container storage interface 123. The container storage interface 123 has a plug-in architecture and may provision storage in the form of a persistent volume from a storage source using a corresponding plug-in from a variety of available container storage interface plug-ins 128. "Provisioning" storage may refer to the process of allocating a specific amount of storage capacity and making that allocation available to a consumer. Plug-ins 128 may be provided by various vendors and may expose an associated storage system to the container storage interface 123 (i.e., make it available for use and / or consumption). Non-limiting examples of plug-ins include a block protocol plug-in (e.g.,based on the Internet Small Computer Systems Interface or iSCSI protocol), a file protocol plug-in (e.g., based on the Network File System or NFS protocol, the Common Internet File System or CIFS protocol, the Server Message Block or SMB protocol), a public cloud persistent volume plug-in, and other plug-ins based on any other storage type (e.g., custom drivers). For convenience, individual ones of the plug-ins 128 may be referred to herein as a plug-in 128 (e.g., a block protocol plug-in 128 or a file protocol plug-in 128). A plug-in 128 may undergo an installation and setup process in the container environment 120 as required by that plug-in (e.g., entering credentials or other configuration details). In some cases, one or more of the plug-ins 128 may be executed as a containerized application.However, a persistent volume provided by plug-in 128 may be limited to a single storage type corresponding to that plug-in. In contrast, the containerized storage virtualization system 130 disclosed herein may be advantageously used to create persistent volumes that combine multiple underlying storage types.

[0016] The containerized storage virtualization system 130 (also referred to as storage virtualization system 130) runs in one or more containers and can be implemented in executable instructions 105. As will be seen, the storage virtualization system 130 provides an additional storage virtualization layer between a requesting application and one or more storage allocations provided via the container storage interface 123. The storage virtualization system 130 includes a data path 132 and a control plane 140. The data path 132 includes a data service provider 134, a storage virtualizer 136, and a volume manager 138. The data path 132 may also include storage adapters 139 for accessing local storage (i.e., storage on the same computer system 100 on which the storage virtualization system 130 is hosted), such asa block adapter for mounting the local physical storage 106, stacked file and block adapters for accessing the local file system 108 as a block device, or other storage adapters.

[0017] The control plane 140 includes a policy engine 142 and a container storage interface API (Application Programming Interface) 144. During an initialization phase of the storage virtualization system 130, the control plane 140 may receive a list of available container storage interface plug-ins 128 in the container environment 120 from an administrator or the container orchestrator 122. The control plane 140 may also confirm the storage adapters 139 available to the datapath 132 for mounting the local storage. The control plane 140 may also maintain a list of characteristics of the storage associated with each of these available plug-ins 128, such as: B. Performance characteristics (e.g. latency, IOPS or input / output operations per second, etc.), security features (e.g. encryption, isolation, etc.), data protection features (e.g. available RAID or redundant Array of Independent Disks levels), cost characteristics (e.g.dollars per GB) or other features.

[0018] The functionality of the data path functions 132 and the control plane functions 140 will now be described in the context of providing storage to containerized applications 124, 126. A containerized application, such as application 124, may request storage space from the container orchestrator 122. For example, the application 124 may require a persistent volume to store data. In some implementations, the application 124 may pass one or more requirements with the request, such as a capacity requirement, a performance requirement (e.g., latency, IOPS, etc.), a data protection requirement (e.g., RAID level), a cost requirement (e.g., dollars per GB), a security requirement, a tiering requirement (e.g., specific amounts of hot and cold storage), or another requirement.In some implementations, the container orchestrator 122 may manage a storage abstraction called an Orchestrator Persistent Volume in response to the request.

[0019] The container orchestrator 122 may use the container storage interface 123 to send the request to the storage virtualization system 130 via the container storage interface API 144 of the control plane 140 (where the interface 144 acts as a server). In this way, the containerized storage virtualization system 130 may be understood as acting as a storage provider for the container orchestrator 122. In some implementations, the control plane 140 may assign an identifier to the request so that each request can be individually identified, particularly with regard to the storage provided for each request in the manner described herein.

[0020] The policy engine 142 of the container orchestrator 122 analyzes the request to determine which storage types meet the requirements of the request. For example, the control plane 140 may have one or more of the following types available for storage provisioning: physical storage 106, the local file system 108, a remote storage system 160, virtualized storage 162, or public cloud storage 164. Furthermore, more than one of the illustrated storage types may be present, but they are not shown for clarity. For example, multiple remote storage systems 160 may be available from which storage allocations can be provisioned.

[0021] The Fig. The remote storage system 160 illustrated in Figure 1 may be either a file system (e.g., a NAS file server), a block storage system (e.g., a SAN storage device), or another type of storage system that is remote from the computer system 100 and, thus, also from the containerized storage virtualization system 130. Remote may mean, for example, that the computer system 100 communicates with the remote storage system 160 via a network connection or the like.

[0022] Virtualized storage 162 may represent any existing persistent volume in a virtual environment, such as a persistent volume in container environment 120, including container storage provisioned via container storage interface 123 independently of container storage virtualization system 130. In some examples, virtualized storage 162 may represent container storage provided by a container storage virtualization system other than system 130 (e.g., hosted on a different node than computer system 100). In some examples, virtualized storage 162 may represent storage provided by a virtual machine or a hypervisor-based software-defined storage platform.

[0023] The policy engine 142 may determine a mix of the aforementioned storage types. For example, the policy engine 142 may compare the request with the available storage types to determine the best possible match. To illustrate, the request may require a certain amount of high-speed storage and a certain amount of low-cost archival-class storage. The policy engine may determine that physical storage 106 meets the high-speed storage requirements (e.g., due in part to locality and because it is a high-speed medium in this example) and a block storage device 160 meets the low-cost archival storage requirements. Additional example implementations of the policy engine 142 are described further below, e.g., with reference to Fig. 2.

[0024] The control plane 140 then uses the adapters 139 and / or container storage interface plug-ins 128 to provision each storage type in the specified mix and obtain a mount point for each provisioned storage. A mount point enables access to the provisioned storage by a consumer, e.g., the data path 132, as described further below.

[0025] As an example of a local provisioning, the control plane 140 may use a block adapter from the adapters 139 to provide a mapping from the physical storage 106 and acquire a local block mount point 170 (e.g., a local host device mount point) to access this mapping. As another example, the control plane 140 may use stacked file and block adapters to provide a mapping from the local file system 108 and acquire a local file system mount point 172 to access this mapping as a block device (i.e., "file as block device").

[0026] To provision storage via plug-ins 128, control plane 140 communicates with storage interface 123 via container storage interface API 144 to request that a plug-in 128 provide a mapping from its associated storage and provide a mount point back to control plane 140. Assuming remote storage system 160 represents a remote block device (e.g., a SAN storage array external to computer system 100), control plane 140 may request (via 144 and 123) that a block protocol plug-in 128 (e.g., based on the iSCSI protocol) provide a mapping from remote block storage system 160 and provide a remote volume mount point 174 (e.g., iSCSI target and LUN or Logical Unit Number) for accessing that mapping. As another example, the remote storage system 160 may be a remote file device (e.g.,a NAS file server), and the control plane 140 may request (via 144 and 123) that a file protocol plug-in 128 (e.g., based on the NFS protocol) provide a mapping from the remote file storage system 160 and provide a remote volume mount point 174 (e.g., an IP address and export name under NFS) for accessing that mapping. In some implementations, the control plane 140 may use a block protocol plug-in 128 to serve from the physical storage 106 or a file protocol plug-in 128 to serve from the local file system 108, instead of using an adapter 139.

[0027] As another example of provisioning via a plug-in, the control plane 140 (via 144 and 123) may request that a plug-in 128 that matches the virtualized storage 162 provide a mapping from the virtualized storage 162 and provide a mount point to the virtualized storage 176 for accessing that mapping. As another example, the control plane 140 (via 144 and 123) may request that a public cloud plug-in 128 provide a mapping from the public cloud storage 164. In turn, the public cloud plug-in 128 may provide a public cloud persistent volume mount point 178 for accessing that mapping.

[0028] Although in Fig. 1 illustrates a local block mount point 170, a local file system mount point 172, a remote volume mount point 174, a virtualized storage mount point 176, and a public cloud persistent volume mount point 178, more or fewer mount points and mount points in any combination may be requested by the control plane 140 and acquired via the adapters 139 or plug-ins 128. In various cases, multiple local block mount points 170, multiple local file system mount points 172, multiple remote volume mount points 174, multiple virtualized storage mount points 176, and / or multiple public cloud persistent volume mount points 178 may be requested and acquired by the control plane 140.Furthermore, storage system 160 and remote mount point 174 may each represent one or more of the same or different types of remote storage and a mount point thereof, including block, file, or other types of remotely accessible storage. The particular combination of storage mount points requested and acquired by control plane 140 may depend on the storage requirement of containerized application 124, and in particular, on the handling of that storage requirement by policy engine 142.

[0029] Once the one or more storage mount points (e.g., 170, 172, 174, 176, or 178) have been acquired by the control plane 140 in accordance with the policy engine 142, the control plane 140 passes the acquired mount points to the data path 132. The control plane 140 may identify the mount points as belonging to a particular request, for example, by associating the mount points with the request identifier. As will be described, the data path 132 consumes the mount points and merges them (i.e., aggregates them) to create a virtual persistent volume 156, which is presented by a mount point 180 to the requesting containerized application 124. In this way, the allocated storage corresponding to the acquired mount points (e.g., 170, 172, 174, 176, or 178) can be referred to as the underlying storage (or similarly, backend storage) of the virtual persistent volume 156.The containerized application 124 therefore reads and writes data to the virtual persistent disk 156. Before describing the creation of the virtual persistent disk 156, operational aspects of the data path 132 are first described.

[0030] Data path 132 includes storage virtualizer 136, which maintains an object-based storage virtualization layer 150. One purpose of storage virtualization layer 150 is to decouple the location of data storage (i.e., the storage allocations accessed via mount points 170, 172, 174, 176, and / or 178) from how the data is presented to a data consumer (e.g., a containerized application 124). In this way, data services such as migration, backup, snapshotting, replication, deduplication, compression, and others can be performed on any mix of underlying storage and with reduced, minimal, or even no disruption to a consumer of the data.

[0031] One aspect of the storage virtualization layer 150 is that the storage virtualizer 136 stores data as "objects" in an object store 152. In particular, the object store 152 may store various types of objects, including data objects and metadata objects. Data associated with the containerized application 124, including files and / or directories, consists of one or more data objects. Metadata objects may be useful, among other things, for organizing the data objects in a meaningful and orderly manner, as described further below. In some implementations, each data object in the object store may be a fixed amount of data, e.g., 4 or 8 kibibytes, and metadata objects may also be a fixed amount of data, e.g., 1 kibibyte.

[0032] An object-based storage virtualization layer 150 may be distinct from block storage (e.g., implemented in a SAN and represented over a storage protocol such as iSCSI or Fibre Channel) and file storage (e.g., a file system that manages data in a file hierarchy and represented over a file protocol such as NFS or SMB / CIFS), although the object-based storage virtualization layer 150 may underlie block or file storage protocols (e.g., by abstraction over mount points 180, 182, as described below).

[0033] The storage virtualizer 136 maintains an object index 154 that records a signature, a physical address, and a reference count for each object (data object and metadata object) in the object store 152. An object's signature may be a cryptographic digest of that object's contents using a hash function such as SHA-1, SHA-256, MD5, etc. Therefore, the signature may also be referred to as a content-based signature. The reference count in the object index 154 refers to the number of references to the associated object across all virtual persistent volumes (including 156, 158) in the storage virtualization layer 150.

[0034] The physical address in object index 154 refers to the actual physical location of the object. While in some examples, object storage 152 may be understood as a storage construct to describe the storage of objects within storage virtualization layer 150, the objects may also be physically stored on the underlying storage at the physical address. Because data path 132 may utilize multiple storage mount points, the specific mount point may be part of the physical address. Furthermore, the physical address may include a location within the memory map of a particular mount point. For example, if the mount point refers to physical storage 106, the physical address may include a logical block number.If the mount point refers to a public cloud storage 164, the physical address may contain a reference to a cloud object in the syntax of the corresponding hosting provider. The volume manager 138 is configured to perform data read and write operations at specific physical addresses, for example, the physical addresses stored in the object index 154.

[0035] In some implementations, the containerized storage virtualization system 130 may employ an additional indirect layer between the object index 154 and the actual physical storage location of the objects. In such implementations, the volume manager 138 may allocate and / or partition the underlying storage allocations into extents (also referred to as mini-volumes) at each mount point provided by the control plane 140. The additional indirect layer may be implemented by storing a virtual address instead of a physical address in the object index 154 in conjunction with an object signature and by maintaining an extent table that maps a given virtual address to an extent and, thus, to the corresponding underlying storage.Thus, to access an object based on a virtual address, the volume manager 138 may first identify the extent targeted by the virtual address using the extent table and a first portion of the virtual address, and then locate the object within the extent using a second portion of the virtual address. In this way, certain data services, such as migration and tiering between extents, may be performed efficiently by updating only the extent identifiers in the extent table, rather than updating a large number of in-memory or persisted references to the data objects (i.e., each affected address in the object index 154) and rebuilding various logical addresses, indexes, and other data structures used in managing the storage system.

[0036] Within the storage virtualization layer 150, the storage virtualizer 136 manages one or more virtual persistent volumes secured by the object store 152. In some implementations, a containerized application is connected to a virtual PV in a one-to-one relationship. For example, in the example illustration of Fig. 1, the containerized application 124 is connected to the virtual PV 156, and the containerized application 126 is connected to the virtual PV 158. In some implementations, each virtual PV is mapped by the container orchestrator 122 to a corresponding persistent orchestrator volume managed by the container orchestrator 122, and the requesting containerized application 124 accesses the storage of the virtual PV 156 via the persistent orchestrator volume.

[0037] In other cases, containerized applications and virtual PVs may be connected in one-to-many, many-to-one, or many-to-many relationships. For illustrative purposes, virtual PVs will now be described with reference to virtual PV 156, although the same description may apply to other virtual PVs, such as virtual PV 158 and other virtual PVs not shown.

[0038] In one implementation, the virtual persistent disk 156 may be an organization of metadata objects and data objects stored in the object store 152, where the organization hierarchically relates the data objects to a root object through associated content-based signatures. In one example, the virtual PV 156 may be a Merkle tree (also referred to as a hash tree) or another hierarchical arrangement (e.g., directed acyclic graphs, etc.). In the case of a hierarchical Merkle tree, the data objects may be located at the lowest tree level of each branch (also referred to as the leaf level, which is farthest from the root object), and such data objects may be referred to as leaf data objects. As described above, the data objects consist of the data of the containerized application 124, such as files and directories.

[0039] Within the hierarchical arrangement, a parent object refers to an object that contains the signatures of its child objects as content. For example, a parent object of leaf-level data objects is a metadata object that stores the signatures of its child leaf-level data objects as content. In this way, the signatures of objects at each level are collected in parent objects at the next higher level in the hierarchical arrangement until the root object is reached. The root object is therefore also a metadata object that stores the signatures of its respective child objects as content. Viewed from another perspective, the hierarchical arrangement expands in a direction from the root object to the leaf level—a metadata object at a given level can extend to a number of child nodes determined by a predefined branching factor.A metadata object can store a set of signatures that is at least equal to the branching factor of the hierarchical arrangement, so that it can contain the signatures of all child objects.

[0040] Any change to the virtual PV's data (i.e., new data, modified data, deleted data) results in a change in the content of one or more leaf-level data objects, resulting in a change in the content-based signatures of these modified data objects. The changes in content and signatures propagate upward through the parent nodes to the root object. Thus, a virtual PV 156 can be uniquely identified at a specific point in time (also called a snapshot) by its root object, and in particular, by its root object signature.

[0041] Another aspect of the virtual PV 156 is that, in some implementations, a particular file or directory from the containerized application 124 may be stored in a corresponding subtree arrangement within the virtual PV 156. In other words, the virtual PV 156 may be divided into subtrees, each of which corresponds to a corresponding file or directory of the containerized application 124.

[0042] Because files and directories consist of one or more data objects, and these data objects are arranged in the virtual PV 156 and its subtrees by reference to associated data object signatures, in some implementations, each of the data objects may be physically stored only once in the object store 152 and referenced by their respective signatures in multiple metadata objects in the virtual PV 156 or in another virtual PV (e.g., 158) in the storage virtualization layer 150. In this way, the data may be deduplicated by the storage virtualizer 136. Similarly, metadata objects may be stored once and referenced multiple times by corresponding signatures. The number of references to a data object or metadata object in the storage virtualization layer 150 may be recorded in the corresponding reference counts of the object index 154.In some implementations, data deduplication may be performed inline during a write operation, as opposed to post-processed or near-line deduplication, and in this way, data storage may be described as natively deduplicated in the storage virtualization layer 150 and between the virtual PVs 156, 158.

[0043] For use cases where security is a concern, including multi-tenant scenarios, separate object stores can be used for each sensitive security domain. This allows sensitive data to be isolated in a secured object store without participating in deduplication of other virtual PVs that are not part of the security domain.

[0044] To enable the containerized application 124 to access the virtual PV 156, the data path 132 may provide a mount point 180. The data path 132 may provide any of a variety of mount point types, including, but not limited to, a block-type mount point (e.g., iSCSI-compatible), a file-type mount point (e.g., a user-space file system or a FUSE interface, or NFS, SMB, or CIFS-compatible), a key / value share mount point (e.g., a NoSQL volume or an Amazon S3-compatible API), and other mount point types. In this way, the mount point can be understood as contributing to a complete abstraction of storage access, as any type of storage access required by the containerized application 124 (e.g., file protocol or block protocol, etc.) is provided to the containerized application 124.), regardless of the underlying storage type that makes up the virtual PV 156 (e.g., regardless of software or hardware based, block or file, local or remote, etc.).

[0045] The mount point type, i.e., the type of abstraction, can be selected by the user or predefined according to the containerized application 124 requesting the storage (i.e., based on a class of the containerized application, specified by the container orchestrator 122, etc.). The type of abstraction can be communicated to the containerized storage virtualization system 130 via the storage request received at the control plane 140.

[0046] In the example of Fig. 1, the containerized storage virtualization system 130 may also provide a mount point 182 for the containerized application 126 to access a virtual persistent volume 158 created in response to a storage request from the containerized application 126 to the container orchestrator 122, in a manner similar to that described above for the mount point 180 of the virtual PV 156 for the containerized application 124. The virtual PVs 156 and 158 may each contain Merkle trees to organize their respective data sets, while the object store 152 stores the data of both virtual PVs 156 and 158 in a deduplicated manner.

[0047] In operation (i.e., after the virtual PV 156 has been created and the mount point 180 has been made available to the application 124), the storage virtualization system 130 may service input / output (I / O) requests from the containerized application 124 directed to a virtual PV 156 via the mount point 180. For example, to service a read request received via the mount point 180, the storage virtualizer 136 may identify the signatures of data objects in the virtual PV addressed by the read request (i.e., which may include traversing the Merkle tree structure of the virtual PV based on an address of the read request) and determine the physical addresses of those data object signatures from the object index 154. In some implementations, the physical address of a data object may indicate the mount point of the underlying memory allocation where the data object is stored (e.g.,one or more of mount points 170, 172, 174, 176, or 178). The storage virtualization system can then, via volume manager 138, read the data objects using the physical addresses (or using a virtual address and extent table as described above) and return the read data to containerized application 124 via mount point 180.

[0048] To service a write request, in example implementations, storage virtualizer 136 may receive data to be written to virtual PV 156 from containerized application 124, check whether the data includes data objects that are new to object store 152 based on content-based signatures, and write the new data objects (i.e., data objects that are not already present in the data store) to the object store. In some implementations, storage virtualizer 136 may compress the new data objects before writing them to object store 152. In particular, the process of writing to object store 152 may include controlling which underlying memory map (e.g., 106, 108, 160, 162, or 164) the new data objects are written to.In some implementations, the containerized application can specify in the write request which underlying storage map the data should be written to. In some implementations, new data can be written by default to a local storage area of ​​the virtual PV 156, such as the locally attached physical storage 106, which provides a "hot" storage tier optimized for frequent access. In some implementations, the containerized application can specify a specific policy or service level agreement for writing the data, and the storage virtualization system can determine which underlying storage map satisfies that policy or SLA. The storage virtualization system then uses the mount point (e.g., 170, 172, 174, 176, or 178) corresponding to that underlying storage map to write the data objects.The storage virtualization system also adds the signature of the data objects to the metadata objects of the virtual PV 156.

[0049] Representing data in a virtual PV 156 that is natively deduplicated and uniquely identified by a root object signature can enable efficient data services, including those provided by the data service provider 134. For example, the data service provider 134 can perform, without limitation, snapshot-based backups, replications, migrations, tiering, redundancy-based data protection (e.g., redundant array of independent nodes, also referred to as RAIN or RAID), or other functions. The data service provider 134 can perform the data services with the virtual PV 156 in a manner that is transparent or non-intrusive to the containerized application 124. For example, the data services can be deployed without changing the mount point 180 and, in the case of some data services, without requiring any input or instruction (e.g., configuration details, setup commands, etc.).) by a user, the containerized application 124, or the container orchestrator 122. Furthermore, in some examples, the data service provider 134 may manipulate data primarily at the storage virtualization layer 150 such that the data services execute independently of the various underlying storage mounts and the particular composition of the virtual PV 156. In other words, the data service provider 134 may perform a common set of data services in a non-disruptive manner regardless of the type of underlying storage composing the virtual PV 156. The aforementioned technical advantages may be achieved, for example, by the virtual PV 156 decoupling the underlying storage from the containerized application 124.

[0050] For example, the data service provider 134 may perform an efficient snapshot-based data backup service. In some implementations, the difference between temporal snapshots of a hierarchically arranged virtual PV 156 may be efficiently achieved by comparing object signatures in a top-down iteration, starting with the root object, to find metadata and data objects that differ. For example, in an operation to back up a current state of the virtual PV 156 (i.e., a current snapshot), the current snapshot may be located on a primary system (e.g., computer system 100) and an older, previously backed up snapshot may already exist on a backup system (e.g., computer system 110).In this example, the difference between the current snapshot and the older snapshot can be determined by comparing the signatures of the snapshots in the manner described previously, and the backup system can be searched to determine if the metadata or data objects that differ already exist on the backup system (i.e., in an object store of the backup system). Only the metadata or data objects that do not exist are copied from the primary system to the backup system, thereby reducing data traffic and improving backup times. In other implementations, snapshot-based backups can be performed on the same primary system instead of the backup system, or in addition to it, in a similar manner.

[0051] For example, the snapshot-based backup can be performed on a schedule without interrupting the containerized application 124. Furthermore, the snapshot-based backup can be performed primarily on the software virtualization layer 150, directly avoiding the complexity of managing each underlying storage. Similar to the backup process, a restore process can also proceed by comparing the metadata or data objects to be restored with the objects already present on the restore target and transferring only the data not present on the restore target.

[0052] The data service provider 134 may also perform a migration operation. The migration may move data objects between different underlying stores within a virtual PV 156, including between different local stores, between local and remote stores, between different remote stores, between different public cloud stores, between public cloud stores and non-public cloud stores (either local or remote), or between other combinations of underlying stores. The migration is performed at the storage virtualization layer 150 by mapping the new physical address of a moved data object to the unchanged content-based signature in the object index 154, making the migration transparent to the containerized application 124.

[0053] As another example, the data service provider 134 may migrate the virtual PV 156 to another computing system. For example, in some cases, it may be beneficial to migrate a virtual PV 156 to be proximate to a workload that uses the virtual PV 156. In an example scenario, the container orchestrator 122 may move the containerized application 124 to another computing system (e.g., from the source computing system 100 to the destination computing system 110) for load balancing or other reasons, and the virtual PV 156 may need to be migrated to be proximate the containerized application 124. In some implementations, the storage virtualization system 130 may migrate management of the virtual PV 156 to another storage virtualization system on the destination computing system 110 to which the containerized application 124 has been migrated.The data service provider 134 may also migrate some of the data in the virtual PV 156, such as migrating data objects that were local to the computing system 100 (e.g., on the underlying physical storage 106) to the physical storage of the other computing system 110, which may be useful for maintaining storage locality and other performance characteristics of the virtual PV 156. Such a migration may involve determining whether the target computing system 110 already has a copy of the metadata or data objects to be migrated and transferring only data objects that do not exist on the target computing system 110. At the same time, the same mount point 180 for the containerized application 124 may be maintained and uninterrupted.

[0054] The data service provider 134 may also perform data tiering within a virtual PV 156, i.e., move data between different types of underlying storage, which may have different characteristics and conform to different storage policies. Tiering may be performed, for example, by allocating and / or partitioning the constituent storage allocations of the virtual PV 156 across different extents, as previously described. Under certain triggering conditions, the data service provider 134 (in conjunction with the volume manager 138 in some implementations) may move data objects from one extent to another and update the extent table accordingly.Triggering conditions may include, for example, an increased security status of data, which may cause the data service provider 134 to move that data from the public cloud storage 164 to non-cloud storage 106, 108, 160, or 162; the aging of data, which may cause the data service provider 134 to move that data to an archival class of storage (e.g., remote storage 160); recent frequent access to data, which may cause the data service provider 134 to move that data to high-performance storage (e.g., local physical storage 106); or other types of conditions. By moving and managing data at the storage virtualization layer 150, data tiering can be performed across any type of underlying storage and without disruption to the mount point 180 or the containerized application 124.

[0055] The data service provider 134 may also support redundancy-based data protection. For example, the data service provider 134 may provide RAID data protection. For example, the data service provider 134 (in conjunction with the volume manager 138 in some implementations) may create a RAID set via underlying storage allocations or within an underlying storage allocation (e.g., in cases where the local physical storage 106 comprises a set of drives). The data service provider 134 may have a software RAID controller or may work with a hardware RAID controller to write virtual PV 156 objects to the RAID set according to a RAID scheme such as RAID 1, RAID 5, or RAID 6.

[0056] The data service provider 134 may also provide RAIN data protection by replicating or mirroring data of a virtual PV 156 (also referred to as the primary virtual PV for convenience) for data protection and high availability purposes, consistent with the principles of the RAIN architecture. In some implementations, the data service provider 134 may initially replicate or mirror data when the data enters the storage virtualization layer 150 as a write request from the containerized application 124. In some implementations, the replicated data may form a virtual PV replica, which may have a similar shape to a virtual PV (e.g., including a Merkle tree) and is managed by another storage virtualization system on a different computing system 110 and is local there, compared to the primary virtual PV, which is local to the computing system 100.Additionally or alternatively, the virtual PV replica may consist of underlying storage that is different and / or separate from the underlying storage that comprises the primary virtual PV 156. Thus, if data on the primary virtual PV is not recoverable, the data may be recovered from the virtual PV replica using a failover procedure.

[0057] In summary, a containerized storage virtualization system 130, which aggregates various storage types into a virtual, persistent volume that can be represented as any number of available storage abstractions, enables the containerized storage datapath to be highly virtualized from end to end. This allows users of containerized applications to obtain a high degree of flexibility in requiring any composition of underlying storage to meet the performance (or other) requirements of an application, while still being able to utilize the storage using any type of abstraction appropriate for the application.Furthermore, a consistent set of storage services can be provided regardless of the composition of the underlying storage and regardless of the type of storage access abstraction used to represent the virtual PV.

[0058] Fig. 2 shows an example system that includes a policy engine 242. Fig. 2 also shows other elements, including a container orchestrator 222, a container storage interface 223, a containerized application 224, a data path 232, a container storage interface API 244, and one or more storage plugins (e.g., a storage plugin A 228-1, a storage plugin B 228-2, a storage plugin C 228-3). In some implementations, the aforementioned elements may be considered Fig. 2 are present in the same container environment. The Policy Engine 242 and the elements of Fig. 2 (i.e., 222, 223, 224, 232, 244, 228-1, 228-2, 228-3) may each be implemented as hardware or as any combination of hardware and programming to implement their respective functions as described herein. The programming may, for example, be processor-executable instructions stored on a non-transitory machine-readable medium, and the hardware may include a processing resource for retrieving and / or executing those instructions. For example, such a machine-readable medium and the processing resource may be analogous in many respects to the machine-readable medium 104 and the processing resource 102, respectively, of Fig. 1. The hardware may also include electronic circuits or logic.

[0059] The container orchestrator 222 may be comparable in many respects to the container orchestrator 122 described above. The container orchestrator 222 may serve as a platform for deploying and managing containerized applications, such as the containerized application 224, which may be similar to the containerized applications 124, 126 described above. The containerized application 224 may contain or be connected to one or more containers, each of which may contain one or more microservices, and the one or more containers may be organized into one or more pods.

[0060] The container orchestrator 222 may include a container storage interface 223, which may be analogous to the container storage interface 123 described above. The container storage interface 223 may enable a plug-in to make a storage system available (i.e., for use and / or consumption) to the container orchestrator 222 and an associated container environment, which may be analogous to the container environment 120 described above. For example, the plug-in 228-1 for storage A may make the storage system 229-1 available to the container orchestrator 222, the plug-in 228-2 for storage B may make the storage system 229-2 available to the container orchestrator 222, and the plug-in 228-3 for storage C may make the storage system 229-3 available to the container orchestrator 222.The plug-ins 228-1, 228-2, and 228-3 (also generally referred to as plug-in(s) 228) may be similar to the plug-ins 128 described above, and each of the storage systems 229-1, 229-2, and 229-3 (also generally referred to as storage, storage type(s), or storage system(s) 229) may be a local storage device (e.g., local storage devices (e.g., locally attached hard disk drives), local or remote storage arrays, software-defined storage, cloud storage, or the like), and may be comparable to any of the storage systems or storage types described above, such as the physical storage 106, the local file system 108, the remote storage system 160, the virtualized storage 162, or the public cloud storage 164.

[0061] For a storage plug-in 228 deployed with the container orchestrator 222, the container orchestrator 222 may manage an object called a storage class containing parameters (e.g., in the form of key / value pairs) describing an associated storage 229. A storage class may describe characteristics of the offered storage (e.g., provisionable as a persistent disk via the container storage interface 223), and the described characteristics may include quality of service levels, backup policies, capacity, cost, performance (e.g., IOPS, latency), redundancy, recovery time objective (RTO), recovery point objective (RPO), or other measures. Different storage classes may describe different storage offerings. In some cases, a storage 229 associated with a provider may have multiple storage classes. Different storage 229 from different providers may have different storage classes.Storage classes may not be standardized between vendors or even within a vendor, as different storage classes may use different parameters.

[0062] In Fig. 2, memory A plug-in 228-1 may have at least one associated storage class 226-1, memory B plug-in 228-2 may have at least one associated storage class 226-2, and memory C plug-in 228-3 may have at least one associated storage class 226-3. Storage classes, such as storage class A 226-1, storage class B 226-2, and storage class C 226-3, may be referred to herein generally as storage class(es) 226. Although in Fig. 2 three memory slots 228, three memory systems 229 and three memory classes 226 are shown as an example, more or fewer memory slots, memory systems and / or memory classes may be present and available in different implementations.

[0063] In some implementations, the policy engine 242 may form at least a part of a storage virtualization system, which may itself be a containerized application with one or more microservices. For example, the policy engine 242 may be implemented as part of the containerized storage virtualization system 130 of Fig. 1, and in particular, it may serve as part of the control plane 140 and / or the policy engine 142 of Fig. 1 or form part of it. As described below, the policy engine 242 may be useful, among other things, for virtualizing control plane operations to provide containerized applications (e.g., application 224) with a virtual persistent volume supported by one or more types of available storage to achieve a policy objective, although the available storage types are described by non-standardized storage classes. Furthermore, as described below, the policy engine 242 may intelligently and automatically optimize the virtual persistent volume, e.g., based on the storage-related performance requirements of an associated containerized application.

[0064] The policy engine 242 may include and / or interact with one or more elements, each of which may be implemented as hardware or as a combination of hardware and programming, to implement their respective functions as described herein. In some implementations, the policy engine 242 may include a scheduler 246, a discovery agent 248, a virtual PV manager 250 including a provisioner 252 and an optimizer 253, a volume extent and replica manager 254, a container storage interface mapping 256, a data path manager 258, and / or an application storage monitor 259. In some implementations, the policy engine 242 may communicate with the container storage interface 223 (e.g., bidirectionally) via a container storage interface API 244, which may be analogous to the container storage interface API 144 described above.In some implementations, the policy engine 242 may communicate with a data path 232, which may be analogous to the data path 132 described above, for example, via the data path manager 258.

[0065] The discovery agent 248 can determine which plug-ins 228 and associated storages 229 are available in the container environment. Furthermore, the discovery agent 248 can retrieve storage classes 226 corresponding to the discovered plug-ins 228 and storages 229 and register the retrieved storage classes 226 with the policy engine 242. For example, the discovery agent 248 can collect the aforementioned information about plug-ins 228, storages 229, and storage classes 226 by querying the container orchestrator 222 or the container storage interface 223.

[0066] In some implementations, the discovery agent 248 may be deployed at startup or during the initialization phase of a storage virtualization system that includes the policy engine 242, and / or periodically during operation of the storage virtualization system. For illustrative purposes, a non-limiting example of a container environment may include three to six types of storage plug-ins 228. In various implementations, there may be one or more storage classes for each plug-in 228.

[0067] The storage classes 226 registered by the policy engine 242 may not be standardized, as described above. In other words, different storage classes may use different parameters, or more precisely, different keys and / or different values ​​for parameters defined as key / value pairs. The policy engine 242 may use a common schema 260 to translate each storage class 226 into a corresponding storage profile. For example, storage class 226-1 of storage A may be translated into profile 262-1 of storage A, storage class 226-2 of storage B into profile 262-2 of storage B, and so on. Storage profiles, such as storage profile 262-1 for storage A and storage profile 262-2 for storage B, may be referred to generally herein as storage profile(s) 262. In some implementations, the common schema 260 may be manually coded, e.g.,based on the administrator's knowledge of the storage classes 226. The common schema 260 can be understood as a FIG. between vendor-specific storage class parameters and a common language. In some implementations, as described below, the common language of the common schema 260 can be designed in correlation with the creation of the virtual PV storage class 227. In some implementations, the storage profiles 262 can be static, meaning they are not expected to change after translation.

[0068] To illustrate, the common schema 260 may be useful when one and the same performance level has the values ​​A and B in the storage class 226-1 of storage A and in the storage class 226-1 of storage B, respectively, and is thus translated into the same performance level value X in the profile 262-1 of storage A and in the profile 262-2 of storage B. Thus, the storage profiles 262-1 and 262-2 (translated from the storage classes 226-1 and 226-2, respectively) use a common language to describe the storage offerings for storage 229-1 and 229-2, respectively.

[0069] The policy engine 242 may also present or publish one or more virtual PV storage classes 227 to the container orchestrator 222. A virtual PV storage class 227 may be made accessible to users alongside the storage classes 226. Similar to the storage classes 226, a virtual PV storage class 227 may describe the properties of a provisionable persistent volume, particularly a persistent volume provisionable by the policy engine 242 or by a storage virtualization system that includes the policy engine 242. In some implementations, a virtual PV storage class 227 and a virtual PV provisioned according to the virtual PV storage class 227 may be assumed to have or satisfy a default or initial storage policy.

[0070] Different virtual PV storage classes 227 can be presented or published to the container orchestrator 222, e.g., storage classes for different service levels, such as gold, silver, bronze, or dynamic, each with different parameters. For example, each of the different virtual PV storage classes 227 can be designed by an administrator for different virtual storage policy purposes or predefined by a provider providing the policy engine 242. For illustration, the Fig. 2 illustrates a "Dynamic-H" storage class with the parameters type: tiered, performance: high, cost: high, and auto-optimize: yes. Such parameters are useful for determining an initial configuration for a virtual PV (e.g., an initial configuration with "high" performance and cost, although other dynamic classes with medium or low performance may also be available), and the virtual PV may be modified by the optimizer 253 (due to the "Auto-Optimize: Yes" parameter), as described further below. Another example of a virtual PV storage class (not illustrated) may be a "silver" storage class with the parameters type: tiered, performance: medium, cost: medium, and auto-optimize: no, which is not automatically optimized by the optimizer 253.

[0071] Unlike storage classes 226, virtual PV storage class 227 conforms to shared schema 260 by default. In other words, the key / value pairs of virtual PV storage class 227 are in the same language as shared schema 260 and do not require further translation. In some implementations, virtual PV storage classes 227 can be defined first, and shared schema 260 can be created based on the parameters of virtual PV storage classes 227.

[0072] The application 224 may request a persistent volume from the container orchestrator 222 either independently or via user input. The request may also include provisioning requirements, such as specifying that the virtual PV storage class 227 is the desired storage class for the requested persistent volume. Other requirements conveyed with the request may include, for example, a desired capacity. The container orchestrator 222, in turn, may forward a request to create a volume 230 to the policy engine 242, for example, via the container storage interface 223 and the container storage interface API 244. In some implementations, the container orchestrator 222 may extract the parameters of the virtual PV storage class 227 and pass the parameters to the policy engine 242 with the request 230.The request 230 can be received within the policy engine 242 from the Virtual PV Manager 250.

[0073] The virtual PV manager 250 may include various operators for manipulating the virtual PVs managed by the policy engine 242, including operators for provisioning new virtual PVs, deleting virtual PVs, handling read / write I / O requests directed to virtual PVs, or other operations. These operators may each be hardware or a combination of hardware and programming to implement their respective functions. For example, the virtual PV manager 250 may include an operator 252 that provisions new virtual PVs. The virtual PV manager 250 may also include an optimizer 253 that automatically optimizes virtual PVs based on the respective performance metrics associated with those virtual PVs. The functionality of the virtual PV manager 250, the provisioner 252, and the optimizer 253 will now be explained in more detail.

[0074] In response to request 230, provisioner 252 begins creating a virtual PV, which is ultimately returned as virtual PV 280 (i.e., including at least one volume identifier) ​​to container orchestrator 222 for consumption or use as persistent storage by containerized application 224. Provisioner 252 determines that storage class 227 of the virtual PV was identified in request 230 as the desired type of virtual PV to be created. Provisioner 252 then evaluates each parameter in the requested virtual PV storage class 227 and determines which storage profile 262 has a corresponding parameter that can satisfy the parameter of virtual PV storage class 227.In some implementations, a parameter in the virtual PV storage class 227 is a minimum such that a closest match of the corresponding parameter among the storage profiles 262 exceeds the parameter value in the storage class 227. This matching of a parameter of the virtual PV storage class 227 to a parameter of a storage profile 262 is enabled by the fact that the virtual PV storage class 227 and the storage profiles 262 all conform to and are in the language of the common schema 260. For example, if a requested parameter specifies performance:high, the provisioner 252 would identify a storage profile 262 that also contains performance:high.

[0075] In some implementations, the provisioner 252 may match the parameters of the virtual PV storage class 227 with the parameters of the storage profiles 262 based on the best match. For example, if no parameters of storage profiles 262 are found that match or exceed the parameters of the virtual PV storage class 227, the closest parameter may be selected. For example, if a requested parameter specifies performance: high, the provisioner 252 may select a storage profile with performance: medium over other storage profiles with performance: low.

[0076] In some implementations, the provisioner 252 may also determine whether assigning the virtual PV storage class 227 to a particular storage profile 262 would violate a resource limit or quota of an associated plug-in 228, and if so, avoid assigning it to that plug-in 228. In some implementations, the provisioner 252 may check the remaining capacity of a storage 229 using an out-of-band query command (e.g., not via plug-ins 228, the container storage interface API 244, or the container storage interface 223).

[0077] In some implementations, the provisioner 252 may determine that multiple storage profiles 262 (e.g., more than one) can be combined to satisfy a parameter of the virtual PV storage class 227. For example, a request 230 may require performance:high and 2 terabytes (TB) of capacity. In this FIG., however, storage profiles 262-1 and 262-2 with the performance:high parameters are limited to 1 TB volumes, either due to remaining capacity or other resource constraints. In this case, the provisioner 252 may decide to combine 1 TB of capacity from each of the storage profiles 262-1 and 262-2 to satisfy the 2 TB performance:high request.

[0078] In some cases, the provisioner 252 may determine that a single storage profile 262 satisfies all of the parameters of the virtual PV storage class 227. In some cases, the provider 252 may determine that multiple different storage profiles 262 are required to satisfy the parameters of the virtual PV storage class 227. For example, a virtual PV storage class 227 may be of a tiered type and specify high-performance primary storage (e.g., with low latency and / or high IOPS) in a first parameter and archival storage (e.g., with low cost and / or high write speed) for backup in a second parameter.This example assumes that storage profile 262-1 is associated with a high-performance storage array 229-1 and storage profile 262-2 is associated with an archive / backup storage system, so that provisioner 252 can determine that the first parameter is satisfied by storage profile 262-1 and the second parameter is satisfied by storage profile 262-2.

[0079] Finally, the provisioner 252 may create a mapping of the parameters of the virtual PV storage class 227 to parameters of one or more storage profiles 262 that serve as proxies for the storage classes 226, such that the parameters of the virtual PV storage class 227 are satisfied. In this manner, the provisioner 252 also identifies the plug-ins 228 (corresponding to the associated storage profiles 262) to be used to assemble a virtual PV with the characteristics of the virtual PV storage class 227.

[0080] The provisioner 252 coordinates with the container storage interface map 256 to provision volumes from each of the storage plug-ins 228 associated with the mapped profiles 262 using their associated storage classes 226. The provisioner 252 informs the container storage interface map 256 of the profiles 262 and the plug-ins 228 that the provisioner 252 has identified and mapped to the virtual PV storage class 227, and the container storage interface map 256 handles the low-level replacement of each parameter in the virtual PV storage class 227 with one or more provisioned volumes using the plug-ins 228 identified by the provisioner 252.In this process, the container storage interface mapper 256 can translate parameters of the virtual PV storage class 227 into parameters in the storage classes 226 and then provision volumes using the plugins 228 and these translated parameters (since the plugins 228 are compatible with the storage classes 226). Provisioning volumes using plugins 228 can be similar to the process described above in . Fig. 1 described deployment can be carried out using plug-ins 128.

[0081] To illustrate, if the provisioner 252 determines that the profile 262-1 of Storage A satisfies a first parameter of the virtual PV storage class 227, the container storage interface mapper 256 may translate this first parameter into a corresponding parameter of the storage class 226-1 of Storage A using the common schema 260 and then issue a create volume request 270 specifying the translated parameter of the storage class 226-1 of Storage A (and, in some examples, additional parameters such as capacity) via the container storage interface API 244 and the container storage interface 223. In return, the Storage A plug-in 228-1 provides storage, e.g.,a volume 272 from the storage system 229-1 according to the request 270 and returns the volume 272, which may be identified by a volume identifier or a persistent volume object or other control plane means for handling a provisioned volume. The container storage interface mapping 256 may repeat the process for each parameter of the virtual PV storage class 227 sequentially or in parallel. Although in . Fig. 2, for the sake of clarity, a provisioning request with plug-ins 228 is shown, it should be understood that in some cases, data carriers may be provided by more than one plug-in 228 according to the allocation determined by the provider 252.

[0082] In some implementations, the volume extension and replica manager 254 and / or the data path manager 258 (in conjunction with the data path 232) may provide additional functionality for creating and / or managing the virtual PV. The functionality of the volume extension and replication manager 254 may work in conjunction with the data path manager 258 / data path 232 or may be independent of the data path manager 258 / data path 232.

[0083] In some implementations, the data path manager 258 may send provisioning information 266 to the data path 232. The provisioning information 266 may be information about the volumes provisioned via plug-ins 228 to satisfy the virtual PV storage class 227, e.g., storage type(s), volume identifiers, etc. (e.g., identifying information about the volume 272). The data path 232 may then use these provisioned volumes to provision the virtual PV in a manner similar to that described above with respect to Fig. 1. In particular, the data path 232 may create the virtual PV as an organization (e.g., a Merkle tree) of metadata objects and data objects stored in the object store, which are hierarchically related by associated content-based signatures up to a root object. In some implementations, the hierarchically organized virtual PV backed by the object store is returned to the container orchestrator 222 as virtual PV 280, and the data path 232 may participate in servicing I / O requests as described above.

[0084] An extent, also called a mini-volume, is an organizational unit within the virtual PV. The virtual PV can span a single extent (corresponding, for example, to a single provisioned volume) or multiple extents. An extent can target a specific volume provisioned via a plug-in 228 or a subdivision thereof. Dividing the virtual PV into extents can enable I / O to be efficiently directed to the intended destination, for example, in the case of tiered storage with high-performance tier extents for hot data and archival or backup tier extents for cold data. In this way, a single virtual PV can have automated, built-in tiering capabilities, making data tiering simple and invisible to users and administrators.In another example, extents may be useful for managing separate volumes that are to be combined to satisfy a specific parameter of a virtual PV storage class. The volume extension and replica manager 254 may manage extents using an extent table in a similar manner as described above with respect to [ ]. Fig. 1, or shift this management in whole or in part to the data path 232.

[0085] In some implementations, the Volume Extension and Replica Manager 254 may coordinate a backup policy specified by the storage class 227 of the virtual PV throughout the lifecycle of the virtual PV. During the provisioning phase (i.e., after the provisioner 252 has identified primary storage and backup storage from profiles 262 to satisfy the storage class 227 of the virtual PV), the Volume Extension and Replica Manager 254 may coordinate with the container storage interface map 256 to first provision a backup volume (e.g., from a backup storage system 229-2 via plug-in 228-2) and then a primary storage volume (e.g., from a primary storage system 229-1 via plug-in 228-1) using a provisioning request that identifies the backup volume as the replication target. The storage system hosting the primary storage volume (e.g.,Storage system 229-1) can then utilize its own replication mechanism to back up data to the backup volume identified in the provisioning request (e.g., storage system 229-2). In some implementations, a scheduler 246 can operate to schedule backups at a specific frequency or according to a specific schedule (e.g., as defined in virtual PV storage class 227). For each scheduled instance, scheduler 246 can instruct the plug-in for the primary storage volume (e.g., 228-1) to back up data to the target backup volume. In this way, a single virtual PV 280 can include automated backup, simplifying backup configuration and maintenance for users and administrators.

[0086] Once the one or more volumes have been provisioned via plug-ins 228 in accordance with the mapping of storage class 227 to storage profile(s) 262 of the virtual PV, a volume mapping is recorded in the virtualization mapping data structure 264 that associates each provisioned volume with the virtual PV to be created. An example of a mapping is that if a VOL(2) was provisioned by the storage-A plug-in 228-1 (e.g., VOL(2) is a volume identifier for volume 272) and a VOL(3) was provisioned by the storage-B plug-in 228-2 to satisfy the storage class 227 of the virtual PV, then a mapping of VIRTUAL_PV(1) (volume ID for virtual PV 280), consisting of VOL(2) and VOL(3), is recorded in the virtualization maps 264, as shown in Fig. 2. It is understood that the above is an illustration and that a virtual PV may consist of more or fewer provisioned volumes. If multiple virtual PVs are requested (e.g., for the same virtual PV storage class or different virtual PV storage classes), additional mappings are stored in the virtualization card data structure 264. In this sense, the virtualization card data structure 264 can be considered dynamic, since new cards may be added when new virtual PVs are created and existing cards may be deleted when virtual PVs are deleted.

[0087] The policy engine 242 may return the virtual PV 280 by sending at least the identifier of the virtual persistent volume 281 (e.g., the volume ID VIRTUAL_PV(1)) to the container orchestrator 222 via the container storage interface API 244 and the container storage interface 223, and the container orchestrator 222 may, in turn, provide access to the virtual PV 280 to the containerized application 224. The container orchestrator 222 may map or associate the virtual PV identifier 281 of the virtual PV 280 with the containerized application 224 in the metadata of the container orchestrator 222. Subsequently, the containerized application 224 may perform read and / or write operations on the virtual PV 280.The storage virtualization system, including the policy engine 242, may service I / O requests directed to the virtual PV 280 by mediating those I / O requests with the storage volumes 229 associated with the virtual PV 280.

[0088] In different implementations and scenarios, the type of virtual PV 280 provisioned by the policy engine 242 may vary. For example, if a single underlying volume is mapped and provisioned for the virtual PV storage class 227, the policy engine 242 may provision the provisioned volume (e.g., a volume 272) directly as a virtual PV 280 without involving the data path 232. In another example, a single underlying provisioned volume (e.g., a volume 272) may be passed to the data path 232 and virtualized as described above (e.g., as a hierarchical structure of objects identified by content-based signatures), and this virtualized data path storage is provisioned as a virtual PV 280.In another example, multiple underlying volumes are mapped and provisioned for the virtual PV storage class 227, these volumes are virtualized via the data path 232 as described above, and this data path virtualized storage is provisioned as virtual PV 280. In . Fig. 2, a dashed virtual PV 280 includes, for example, a volume from storage system 229-1 and a volume from storage system 229-2, which are provided via their respective plug-ins 228-1 and 228-2.

[0089] Based on the foregoing, the policy engine 242 can automatically identify and construct a tailored collection of stores to achieve a policy goal represented in a virtual PV storage class 227 exposed in a container orchestrator 222. The policy engine 242 can utilize an ecosystem of storage plug-ins 228 and storage classes 226 without the intensive manual work of writing code to support each plug-in 228. Furthermore, the policy engine 242 can improve the mobility of containerized applications because when a containerized application moves to a new container environment, the policy engine 242 can automatically construct the correct persistent storage in the new container environment from the storage available and discoverable in that new container environment.

[0090] As previously mentioned, the virtual PV 280, as initially created and provisioned by the policy engine 242 in response to a request to create a volume 230 and in accordance with the virtual PV's storage class 227, may be considered to have a default storage policy. That is, the virtual PV 280 may initially have characteristics based on a predefined virtual PV storage class, created, for example, by an administrator. In some implementations, the policy engine 242 may further tune the virtual PV 280 based on information about the requesting containerized application 224 to further optimize the virtual PV 280 for that application 224. This tuning may occur before or while the containerized application 224 uses the virtual PV 280 as persistent storage.

[0091] At the beginning of reconciliation, the policy engine 242 can use the volume identifier 281 of the virtual PV 280 to request and read from the container orchestrator 222 an application manifest 282 of the containerized application 224 associated with this volume identifier. For example, a deployed containerized application 224 can be represented by one or more objects (e.g., Kubernetes objects), and these objects can be expressed as one or more JSON (JavaScript Object Notation) or YAML (YAML Ain't Markup Language) files. The manifest 282 is the collection of these objects (e.g., YAML files), and the manifest 282 can be understood as a definition of the containerized application 224.

[0092] The objects of the manifest 282 may each contain a label associated with the containerized application 224. To locate the manifest 282, the policy engine 242 may use the volume identifier of the virtual PV 280 to locate a persistent volume object with that volume identifier, read the label of that persistent volume object, and locate other objects with the same label.

[0093] The manifest 282 may specify properties such as a name, the image on which the containerized application 224 is based, a number of replicas (e.g., a number of compute replicas, which may relate to scaling), labels, a pod definition of which the containerized application 224 is a member, a reference to another application that uses or is related to the containerized application 224, or other properties. Examples of images may be Postgres, Nginx, MongoDB, MySQL, or other images. Such details may not be communicated with the create volume request 230, so the policy engine 242 has no idea what the storage requested via the create volume request 230 is intended for or what the nature of the containerized application 224 is.

[0094] The policy engine 242 may analyze the application manifest 282 by comparing information of the manifest 282 with one or more preconfigured application data store profiles, such as the one in Fig. 2 and the application data store profile B 284-2 (also referred to generally as application data store profile(s) 284), although more or fewer application data store profiles may be available in various examples. The policy engine 242 may then identify a particular application data store profile that corresponds to the manifest. For example, each of the application data store profiles 284 is predetermined, as explained further below, to provide a tailored store for a corresponding application type. In some implementations, an application data store profile 284 may include customization criteria metadata indicating which application type the profile is tailored for, including metadata such as a name, an image name, labels, number of replicas (e.g., compute replicas), and / or other properties.The policy engine 242 may identify a best match between the properties of the application manifest 282 and the metadata information of the application data store profiles 284 (i.e., a best match may be an application data store profile that matches more of the information in the manifest 282 than other application data store profiles). The best match may be based on various logic, rules, filters, weightings, or the like, or any combination thereof (e.g., in some implementations, a match with the name of a container image may be given the highest weight).

[0095] An application data store profile 284 may be a profile that describes the characteristics of a data store used by a particular application, and in particular, the characteristics may be optimized for the particular application. The application data store profiles 284 may be preconfigured based on an understanding of storage usage by corresponding application types. The application data store profiles 284 may be preconfigured manually by developers or in a semi-automated or automated manner using machine learning techniques. In some implementations, the application data store profiles 284 may be in the form of, for example, human-readable text files.

[0096] The understanding of memory usage on which Profiles 284 are based can be gained in several ways. For example, in some cases, an application type associated with a particular image may be assumed to use a particular memory type based on domain knowledge or analysis of the image's source code (e.g., in the case of open source). In other examples, instances of the application type may be deployed, and while the instances are running, memory-related calls (e.g., at a file system level) may be intercepted or other memory-related behaviors may be observed (e.g., I / O patterns, etc.). These test observations and intercepted calls can be analyzed to infer memory usage.Memory usage determined by any of the aforementioned methods may include caching patterns, I / O patterns, replication behavior, or other characteristics that can be used as a prime indicator of how the corresponding application type utilizes memory. Different application types may be analyzed using any of the aforementioned methods, and the application types may include different deployment configurations of the same application image (e.g., a fast Postgres instance, a small Postgres instance, a large Postgres instance). In some implementations, the analyzed application types may be selected based on the most common and / or popular types of containerized applications deployed.

[0097] Each application data store profile 284 may be structured to include storage parameters for configuring a virtual PV to adapt it to the storage usage determined for the corresponding application type. The storage parameters may specify aspects of a virtual PV, such as the type of underlying storage (e.g., with desired performance or cost characteristics), storage capacity, tiering, a replication policy, a backup policy, a data prioritization policy, or other storage characteristics. The storage usage of different application types and configurations can vary greatly, so the storage parameters specified in the application data store profiles 284 may also vary. In addition to the storage parameters, an application data store profile 284 may also specify relevant performance metrics that can be used by the optimizer 253 to automatically optimize the storage (e.g.,a virtual persistent volume) for the corresponding application type, as described further below. Once created, the application datastore profiles 284 can be made available to the policy engine 242 to tune virtual PVs based on application manifests.

[0098] In some implementations, the profile engine 242 may receive and store one or more custom application data store profiles. For example, a user may create a custom application data store profile that is added to the preconfigured application data store profiles 284. For example, custom application data store profiles may be useful for users with specific application and storage deployments not anticipated by the preconfigured application data store profiles 284. The author of a custom application data store profile may also provide the associated metadata for the fitting criteria and best-fit algorithm.

[0099] After the policy engine 242 has identified an application datastore profile 284 that matches the application manifest 282, the policy engine 242 (via the above-described functionality of the virtual PV manager 250, the container storage interface mapper 256, the volume extension and replica manager 254, and / or the datapath manager 258) may modify the virtual PV 280 according to the storage parameters of the identified application datastore profile 284, such that the virtual PV 280 has an updated application-centric storage policy. In many cases, the changes to the virtual PV 280 may be made in a manner that is transparent and non-disruptive to the containerized application 224, particularly when the virtual PV 280 is virtualized and abstracted via the datapath 232 and changes to the datapath 232 (e.g.,Changes to extents or to the object-based hierarchical arrangement of data) or to the underlying storage 229. Any changes to the volumes that comprise the virtual PV 280 can be updated in the virtualization cards 264 without changing the volume identifier for the virtual PV 280. In some implementations, the policy engine 242 does not need to send updates to the container orchestrator 222 about the changed virtual PV 280 because the volume identifier remains the same as before the change.

[0100] The policy engine 242 may also include an optimizer 253, which, in conjunction with an application storage monitor 259 and a monitoring system 290, may automatically optimize virtual PVs in the following manner. Virtual PVs created from a dynamic storage class (e.g., Auto-Optimize: Yes) may be flagged for optimization by the optimizer 253. The optimization may be performed by the optimizer 253 in real-time or near-real-time, regularly, periodically, or on a schedule, depending, for example, in part on the frequency of metric collection. The optimization may include changing a composition of a virtual PV based on an analysis of performance metrics retrieved or collected by containerized applications and their microservices related to the use and operation of the virtual PV.Optimization can be performed with or without tuning based on parameters of an application data store profile, as described above.

[0101] First, example sources of performance metrics are described. In some implementations, metrics instrumentation may be integrated into the datapath 232 and / or the containerized application 224 (or one or more microservices thereof) to collect metrics 292 and metrics 294 about the performance of the datapath 232 and the containerized application 224, respectively. The metrics 292, 294 may be time-series performance data in the form of a counter (e.g., a monotonically increasing value), a gauge (e.g., a value that can go up or down), a histogram, or other forms. The metrics 292, 294 may include general performance data related to processor (CPU) or memory utilization, memory-related performance information such as read IOPS, write IOPS, latency per read, latency per write, etc., or other types of performance information. Each of the metrics can be identified by a metric name.In some implementations, the metrics 292, 294 may be sent to a gateway or made available for retrieval (e.g., as an HTTP endpoint) by the monitoring system 290. The instrumentation and presentation of the metrics 292, 294 may be in accordance with a standard, e.g., a standard defined for Prometheus.

[0102] The metrics 292 may relate to the performance of the datapath 232 itself. In some implementations, the datapath 232 may manage multiple virtual PVs, and the datapath 232 may be instrumented to provide performance metrics separately for each virtual PV. In some implementations, the datapath 232 may also be instrumented to provide performance metrics associated with the backend storage systems 229 based on the interactions between the datapath 232 and these backends during service I / O to the underlying storage volumes.

[0103] In order to optimize a virtual PV such as virtual PV 280, policy engine 242 (or more specifically, optimizer 253) may use the virtual PV identifier to locate an application manifest 282 associated with the containerized application 224 that uses virtual PV 280. Based on application manifest 282, policy engine 242 may identify microservices associated with containerized application 224 that may serve as targets for scrapping performance metrics.

[0104] The policy engine 242 may also identify an application datastore profile 284 that matches the application manifest 282, as described above (e.g., by matching metadata such as an image name). The identified application datastore profile 284 may indicate which particular performance metrics are relevant as inputs to the optimizer 253 for optimizing a datastore (i.e., the virtual PV 280) used by the application type corresponding to the identified application datastore profile 284. For example, the application datastore profile 284 may indicate that a metric to be collected includes one or more of the following metrics: read IOPS, write IOPS, combined read and write IOPS, latency per read, latency per write, CPU utilization, memory utilization, or other performance metrics.The application data store profile 284 may also indicate that a metric related to a particular virtual PV should be collected (e.g., if the data path 232 can export the data path performance metrics individually per virtual PV). Such an indication of the relevant metrics to be collected may be useful in cases where the available metrics among metrics 292, 294 are numerous and of varying relevance or usefulness for optimizing the virtual PV 280, which in some cases is due to metrics being collected for many other purposes. The application data store profile 284 may also indicate from which targets performance metrics should be collected, such as the containerized application (e.g., 224), the data path 232, and / or other related applications or microservices associated with the use of the virtual PV 280.

[0105] In some implementations, the application data store profile 284 may also include a metric name mapping between a metric to be retrieved and corresponding different metric names specific to containerized applications. The different metric names may arise due to non-standardization among containerized application developers, but it can be assumed that such different metric names effectively refer to the same or similar metric.

[0106] A monitoring system (which may also be referred to as a monitoring server) may be deployed in the container environment to collect metrics from instrumented applications or microservices, either directly (e.g., by scraping HTTP endpoints of instrumented microservices) or, for example, via a pushed metrics gateway. The monitoring system 290 may itself be a containerized application running on a processing resource in the container environment. The monitoring system 290 may be a single monitoring system or a collection of monitoring systems.

[0107] In some implementations, a monitoring system may be launched in the container environment in conjunction with the launch of an instrumented application. For example, a monitoring system 290 may be launched for the datapath 232 as a microservice, the storage virtualization system to which the datapath 232 belongs, or the containerized application 224. In some implementations, the monitoring system 290 may be configured to collect metrics from each instrumented containerized application, even if the monitoring system 290 was not launched for that specific application. In some implementations, the monitoring system 290 may include a monitoring system that collects metrics from another monitoring system, and such a monitoring system 290 may be referred to as a monitoring federation.In some implementations, monitoring system 290 may be a centralized monitoring service launched to collect metrics from multiple containerized applications. Monitoring system 290 may also process the collected metrics, for example, by averaging the metrics. An example implementation of monitoring system 290 may include a Prometheus server.

[0108] The policy engine 242 may configure the monitoring system 290 according to all metrics and targets defined in the application data store profile 284 and using target information determined from the application manifest 282. In some implementations, a sample monitoring system 290 may be configured using a YAML file that specifies a target (e.g., an HTTP endpoint), a metric name, a sampling rate, and / or other parameters. The metric name may be specific to the target based on the mapping contained in the application data store profile 284. The monitoring system 290 thus configured may collect the desired metrics from metrics 292 and / or metrics 294 and store the collected metrics as metrics 296.

[0109] The policy engine 242 may include an application memory monitor 259 that receives some or all of the metrics 296 from the monitoring system 290 and forwards these received metrics to the optimizer 253. In some implementations, the application memory monitor 259 may retrieve metrics 296 from the monitoring system 290 via an API of the monitoring system 290. The application memory monitor 259 may specify a time range of the metrics 296 to be collected.

[0110] The optimizer 253 may analyze the metrics received from the application storage monitor 259 to determine an optimization strategy for the virtual PV 280. For example, the optimizer 253 may include preconfigured logic or rules that prescribe certain actions to be performed on the virtual PV 280 in response to certain values ​​or trends observed by the metrics. Example actions may include adding capacity to the virtual PV 280, changing the underlying volume composition of the virtual PV 280, enabling or disabling storage services (e.g., tiering, snapshotting, replication, migration, redundancy-based backup, caching, or other services), or other actions.Adding capacity or changing the underlying volume composition may be accomplished by adding underlying volumes as extents managed by data path 232 in the manner described above.

[0111] For example, the optimizer 253 may change the composition of the virtual PV 280 if the optimizer 253 determines that the characteristics of a particular storage profile 262 (and thus the storage class 226) are better suited to the way the application 224 uses the virtual PV 280, based on values ​​or trends observed from the metrics. Characteristics of the storage profiles 262 that may assist in such a determination include the characteristics described above, such as high / medium / low performance, as well as information about the media type (e.g., hard disk drive, solid state disk, etc.), the performance of the media (e.g., 7200 RPM or 5400 RPM for hard disk drive), the maximum supported IOPS, or other storage characteristics.After identifying one or more storage profiles 262 suitable for actual storage according to the metrics, the optimizer 253, in conjunction with the provisioner 252, may provision one or more new underlying volumes from these storage profiles 262 and migrate data of the virtual PV 280 to these new underlying volumes, thereby changing the virtual PV 280 to the virtual PV 280'. The composition of the underlying volume for the virtual PV 280' may be updated in the virtualization maps 264, but the virtual PV 280' may be identified by the same virtual PV identifier 281 as the virtual PV 280.

[0112] The following describes some non-limiting examples of optimizer 253 logic that changes the virtual PV composition or services in response to the metrics. In some implementations, optimizer 253 may determine that the metrics indicate a shift in I / O toward a more read-oriented state (e.g., indicated by increasing read IOPS and / or decreasing write IOPS metrics), and optimizer 253 may attempt to identify a storage profile 262 that represents favorable storage for read operations, such as SSD (Solid State Disk) or NVMe (Non-Volatile Memory Express) class storage.On the other hand, if the optimizer determines that the metrics indicate a shift in I / O toward a more write-oriented nature, the optimizer 253 may avoid storage profiles 262 that represent storage that may have limited write bandwidth, such as SSD.

[0113] In some implementations, optimizer 253 may determine that metrics indicate that virtual PV 280 is being used in a read-only capacity (e.g., sustained zero write IOPS or write IOPS below a threshold). In response, optimizer 253 may attempt to identify a storage profile 262 representing archival-class storage that can replace (through migration) some or all of virtual PV 280.

[0114] In some implementations, the optimizer 253 may determine that the metrics indicate increasing latency. In response, the optimizer function 253 may attempt to identify a storage profile 262 that represents a more powerful and / or scalable storage system than the one currently used in the virtual PV 280, which may be added to the virtual PV 280 or may replace all or part of the virtual PV 280 (through migration). For example, the optimizer 253 may maintain a performance ranking of vendor systems and compare the ranking to the system names in the storage classes 262.

[0115] In some implementations, the optimizer 253 may determine that the metrics indicate that the read IOPS, write IOPS, or total IOPS are at or approaching the maximum supported IOPS of the underlying storage of the virtual PV 280. In response, the optimizer 253 may attempt to identify a storage profile 262 with a higher supported IOPS that can be added to the virtual PV 280 or replace part or all of the virtual PV 280 (through migration).

[0116] In some implementations, optimizer 253 may detect that the number of I / Os for virtual PV 280 (e.g., indicated by read IOPS, write IOPS, or total IOPS) has increased or is increasing. In response, optimizer 253 may cause datapath manager 258 to issue provisioning information 266 to enable a cache service on datapath 232 for virtual PV 280 or to allocate additional cache on datapath 232 for virtual PV 280.

[0117] In some implementations, the optimizer 253 may analyze metrics related to the performance of backend storage systems used by the virtual PV 280 and detect a load imbalance between the backend storage systems. For example, an overloaded backend storage system may represent a bottleneck. In response, the optimizer 253 may identify a storage profile 262 that performs comparable to or better than the overloaded backend and provision a new underlying volume to be added to the virtual PV 280 or to partially or completely replace the virtual PV 280 to improve load balancing.

[0118] The Fig. 3 and Fig. 4 are flowcharts illustrating various example methods. In some embodiments, one or more blocks of the methods may be performed substantially concurrently or in a different order than illustrated. In some embodiments, a method may include more or fewer blocks than illustrated. In some embodiments, one or more blocks of a method may be continuous and / or repeating at certain times. In some embodiments, blocks of the methods may be combined.

[0119] The Fig. 3 and Fig. 4 may be implemented in the form of executable instructions stored on a machine-readable medium and executed by a processing resource, and / or in the form of electronic circuits. For example, aspects of the methods may be described below as being executed by a storage virtualization system, which may be, for example, the containerized storage virtualization system 130 running on a hardware processor resource 102 of the computer system 100 described above, and may in particular be executed by a policy engine of such a storage virtualization system, such as the policy engine 242 described above. Furthermore, other aspects of the methods described below may be described with reference to other Fig. 2 elements shown are described for non-limiting illustration purposes.

[0120] Fig. 3 is a flowchart illustrating an example method 300. The method 300 begins with block 302 and proceeds to block 304, where the storage virtualization system may identify an application data storage profile (e.g., 284-1) from a plurality of application data storage profiles 284 based on an application manifest 282 of a containerized application 224 assigned a virtual persistent volume 280. For example, as described above, the storage virtualization system may use a virtual PV identifier of the virtual PV 280 to locate the application manifest 282 and may match metadata (e.g., image name, etc.) of the application manifest 282 with that of the application data storage profiles 284.An application data storage profile 284 may contain storage parameters preconfigured for a particular application type and may also specifically specify performance metrics relevant for optimizing the storage used by that particular application type (e.g., a virtual PV). As described above, the virtual PV 280 is formed from at least one underlying storage volume 272 by a data path 232 of the storage virtualization system.

[0121] In some implementations, the storage virtualization system may configure a monitoring system 290 to collect performance metrics, particularly the metrics identified in the application data storage profile 284-1 identified in block 304, from instrumented applications and their microservices. The performance metrics may be storage performance metrics such as IOPS (read IOPS and / or write IOPS), latency (read and / or write latency), or other metrics. An example target for collecting metrics includes the datapath 232, which may provide storage performance metrics about the datapath related to input / output operations for the virtual PV 280. An additional or alternative example target for collecting metrics is the containerized application 224. The monitoring system 290 stores the collected metrics 296.

[0122] In block 306, the storage virtualization system receives the performance metrics specified in the application data storage profile 284. In some implementations, the storage virtualization system retrieves the metrics from the monitoring system 290 (e.g., via an API call).

[0123] In block 308, the storage virtualization system modifies the virtual PV 280 based on an analysis of the performance metrics received in block 306. The method 300 ends in block 312. In this way, the storage virtualization system may utilize a microservice monitoring framework to automatically optimize a virtual persistent volume.

[0124] Fig. 4 is a flowchart illustrating an example method 400. The method 400 begins at block 402 and proceeds to block 404, where a storage virtualization system determines whether a storage class associated with a virtual PV 280 (e.g., storage class 227, after which the virtual PV 280 was created) is of a dynamic type. If the storage class is not a dynamic type ("NO" in block 404), the method 400 performs no optimization on the virtual PV 280 and ends at block 416. If the storage class is a dynamic type ("YES" in block 404), the method 400 proceeds to block 406.

[0125] Block 406 may be analogous to block 304 in many respects. In block 406, the storage virtualization system identifies an application data storage profile 284 based on an application manifest 282 of a containerized application 224 that uses the virtual PV 280. Block 408 may be analogous to block 306. In block 408, the storage virtualization system receives performance metrics indicating how the containerized application 224 utilizes the virtual PV 280. As described above, the metrics may be drawn from the metrics 296 of a monitoring system 290 configured to collect metrics according to the application data storage profile 284 from instrumented applications or microservices.

[0126] In block 410, the storage virtualization system analyzes the performance metrics received in block 408. The storage virtualization system determines trends in the performance metrics, as described above with reference to optimizer 253, and attempts to identify one or more storage classes 226 available in the container environment, as represented by storage profiles 262, that are optimized for or correspond to the determined trends.

[0127] In block 412, the storage virtualization system provisions new underlying storage volumes using the one or more storage classes 226 identified in block 410 (via corresponding storage profiles 262).

[0128] In block 414, the storage virtualization system modifies the virtual PV 280 (e.g., into a virtual PV 280') based on the analysis performed in block 410 by incorporating the new underlying storage volumes provided in block 412. In some implementations, the virtual PV 280 may be modified by adding the new underlying storage volumes to the virtual PV 280, and in some scenarios, by migrating data from existing underlying storage volumes to the new underlying storage volumes. The mapping of the virtual PV 280 to the underlying storage media may be updated in the virtualization cards 264.

[0129] The method 400 ends in block 416. The storage virtualization system may periodically repeat the method 400 for the virtual PV 280.

[0130] Fig. 5 shows an example system 500 that includes a non-transitory, machine-readable medium 504 encoded with example instructions that can be executed by a processing resource 502. In some implementations, the system 500 can be used to implement aspects of the storage virtualization system 130 of Fig. 1 or the Policy Engine 242 from Fig. 2 or for the implementation of aspects of methods 300 and / or 400 of Fig. 3 or 4 may be useful. For example, the instructions encoded on the machine-readable medium 504 may be included in the instructions 105 of Fig. 1. In some implementations, the Fig. 2 may be included in the instructions encoded on the machine-readable medium 504.

[0131] Processing resource 502 may include a microcontroller, a microprocessor, central processing unit core(s), an ASIC, an FPGA, and / or other hardware device suitable for retrieving and / or executing instructions from machine-readable medium 504 to perform functions related to various examples. Additionally or alternatively, processing resource 502 may include or be coupled to electronic circuitry or dedicated logic for performing some or all of the functions of the instructions described herein.

[0132] Machine-readable medium 504 may be any medium suitable for storing executable instructions, such as RAM, ROM, EEPROM, flash memory, a hard disk drive, an optical disk, or the like. In some embodiments, machine-readable medium 504 may be a tangible, non-transferable medium. Machine-readable medium 504 may be located within system 500, where the executable instructions may be considered installed or embedded in the system. Alternatively, machine-readable medium 504 may be a portable (e.g., external) storage medium that is part of an installation package.

[0133] As described further below, the machine-readable medium 504 may be encoded with a set of executable instructions. It is understood that some or all of the executable instructions and / or electronic circuitry contained in one package may, in alternative implementations, be contained in a different package shown in the figures or in a different package not shown. Some embodiments may contain more or fewer instructions than in Fig. 5 are shown.

[0134] With reference to Fig.5, the machine-readable medium 504 contains instructions 506, 508, 510. When executed, the instructions 506 cause the processing resource 502 to identify an application data storage profile from a plurality of application data storage profiles, each containing preconfigured storage parameters for a corresponding application type, based on an application manifest of a containerized application to which a virtual PV is assigned. The virtual PV is assembled from at least one underlying storage volume by a data path of a storage virtualization system.

[0135] The identified application datastore profile displays performance metrics relevant for optimizing the virtual PV. The performance metrics are related to the containerized application's usage of the virtual persistent volume and can include, for example, IOPS (e.g., read IOPS, write IOPS, total IOPS), latency (e.g., read latency, write latency), or other storage-related performance metrics. In some examples, the performance metrics can be storage performance metrics across the datapath, and in other examples, across the datapath that specifically relate to I / O operations for the virtual PV and not to I / O operations for other virtual PVs. In some examples, the performance metrics are storage performance metrics of the containerized application.

[0136] In some implementations, machine-readable medium 504 contains instructions for configuring a monitoring system in the container environment to retrieve the performance metrics specified in the identified application data storage profile from instrumented applications or microservices related to the data path or containerized application and to store the retrieved metrics in the monitoring system. The monitoring system may have been launched in the container environment for or with the storage virtualization system.

[0137] When executed, instructions 508 cause processing resource 502 to receive the performance metrics. In some implementations, instructions 508 may cause processing resource 502 to retrieve the performance metrics from the monitoring system, e.g., via an API call or other interface means. As described above, the retrieved performance metrics may originate from the datapath, the containerized application, or both.

[0138] Instructions 510, when executed, cause the processing resource 502 to modify the virtual PV based on an analysis of the performance metrics. In some implementations, the machine-readable medium 504 includes instructions for performing the analysis, including identifying a storage class among the storage classes available in the container environment according to the performance metrics (i.e., certain performance metrics or trends of metrics may trigger logic to search for a storage class with specific parameters, as described in the examples above with respect to the optimizer 253) and determining that the virtual PV should be modified to include a new underlying storage volume provisioned using the identified storage class.

[0139] In some implementations, machine-readable medium 504 may contain other instructions. For example, additional instructions may cause processing resource 502 to make the execution of instructions 506, 508, 510 dependent on whether a storage class of the virtual PV (e.g., the storage class of the virtual PV) is of a dynamic type.

[0140] In the foregoing description, numerous details are set forth to facilitate understanding of the subject matter disclosed herein. However, the invention may be practiced without some or all of these details. Other implementations may include modifications, combinations, and variations of the details described above. The following claims are intended to cover such modifications and variations.

Claims

[1] A non-transitory, machine-readable medium (104) comprising instructions (105) for a storage virtualization system (130) in a container environment (120), the instructions (105), when executed, causing a processing resource (102) to: to identify a first application data storage profile (284-1) from a plurality of preconfigured application data storage profiles (284), each containing preconfigured storage parameters for a corresponding application type, based on a comparison of information in the plurality of application data storage profiles (284) with information in an application manifest (282) of a containerized application (124, 126, 224) assigned a first virtual persistent volume (156, 158, 280), wherein the information in the application manifest (282) compared to the information in the plurality of application data storage profiles (284) includes a name of a container image on which the containerized application (124, 126, 224) is based, and wherein each identified application data storage profile of the plurality of application data storage profiles (284) identifies a respective collection of performance metrics (292, 294, 296),relevant for optimizing virtual persistent volumes for the respective application type; receive values ​​of first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) for use by an optimizer (253) in optimizing the first virtual persistent volume (156, 158, 280), wherein the first performance metrics (292, 294, 296) relate to the use of the first virtual persistent volume (156, 158, 280) by the containerized application (124, 126, 224) and are a subset of the available metrics to be collected; and modifying the first virtual persistent volume (156, 158, 280) by the optimizer (253) based on an analysis of the first performance metrics (292, 294, 296) identified by the first application, wherein modifying the first virtual persistent volume (156, 158, 280) comprises: enabling caches from a disabled state by changing a cache from the disabled state to an enabled state for the first virtual persistent volume (156, 158, 280) based on the optimizer's (253) analysis of the first performance metrics (292, 294, 296). [2] The non-transitory, machine-readable medium (104) of claim 1, wherein the first virtual persistent volume (156, 158, 280) is composed of at least one first underlying storage volume (106, 108, 160, 162, 164, 272) by a data path (132) of the storage virtualization system (130), and wherein modifying the first virtual persistent volume (156, 158, 280) by the optimizer (253) comprises: Selecting a given storage profile from different storage profiles (262) for different storage types, respectively, based on the analysis of the first performance metrics (292, 294, 296) identified by the first application data storage profile, and Providing a second underlying storage volume (106, 108, 160, 162, 164, 272) using the given storage profile to include it in the first virtual persistent volume (156, 158, 280), wherein, prior to the modification of the first virtual persistent volume (156, 158, 280), the first virtual persistent volume (156, 158, 280) is composed of the first underlying storage volume (106, 108, 160, 162, 164, 272) and a third underlying storage volume (106, 108, 160, 162, 164, 272), wherein a combination of a parameter of a first storage profile (262-1) for the first underlying storage volume (106, 108, 160, 162, 164, 272) and a parameter of a second storage profile (262-2) for the third underlying storage volume (106, 108, 160, 162, 164, 272) defines a parameter for a storage class (226) of the first virtual persistent volume (156, 158, 280), and wherein the different storage profiles (262) comprise the first and second storage profiles (262-1, 262-2). [3] The non-transitory machine-readable medium (104) of claim 1, wherein the first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) comprise storage performance metrics of the containerized application (124, 126, 224). [4] The non-transitory machine-readable medium (104) of claim 1, wherein the first application data storage profile (284-1) comprises an assignment between names of the first performance metrics (292, 294, 296) specific to the containerized application (124, 126, 224) and corresponding names of metrics to be stored by a monitoring system (290). [5] The non-transitory machine-readable medium (104) of claim 1, wherein the instructions (105), when executed, cause the processing resource (102) to: Determining whether a storage class (226) of the first virtual persistent volume (156, 158, 280) is of a dynamic type; and in response to determining that the storage class (226) of the first virtual persistent volume (156, 158, 280) is of the dynamic type, perform receiving the values ​​of the first performance metrics (292, 294, 296) and modifying the first virtual persistent volume (156, 158, 280). [6] The non-transitory, machine-readable medium (104) of claim 2, wherein selecting the given storage profile based on the analysis of the first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) comprises determining that the given storage profile is better suited than another storage profile of the various storage profiles (262) for how the containerized application (124, 126, 224) uses the first virtual persistent volume (156, 158, 280). [7] The non-transitory machine-readable medium (104) of claim 3, wherein the storage performance metrics include input / output per second and latency. [8] A method (300) comprising: Identifying (304), by a storage virtualization system (130) having a hardware processor, a first application data storage profile (284-1) from a plurality of preconfigured application data storage profiles (284), each containing preconfigured storage parameters for a respective application type, based on a comparison of information in the plurality of application data storage profiles (284) with information in an application manifest (282) of a containerized application (124, 126, 224) assigned a first virtual persistent volume (156, 158, 280), wherein the information in the application manifest (282) compared to the information in the plurality of application data storage profiles (284) includes a name of a container image on which the containerized application (124, 126, 224) is based,and wherein each identified application data storage profile of the plurality of application data storage profiles (284) identifies a respective collection of performance metrics (292, 294, 296) relevant to the optimization of virtual persistent volumes for the respective application type;, Receiving (306), by the storage virtualization system (130), values ​​of first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) for use by an optimizer (253) in optimizing the first virtual persistent volume (156, 158, 280), wherein the first performance metrics (292, 294, 296) relate to the use of the first virtual persistent volume (156, 158, 280) by the containerized application (124, 126, 224) and are a subset of the available metrics to be collected; and Changing (308) the first virtual persistent volume (156, 158, 280) by the optimizer (253) in the storage virtualization system (130) based on an analysis of the first performance metrics (292, 294, 296) identified by the first application data storage profile, wherein changing the first virtual persistent volume (156, 158, 280) comprises: Enabling caches from a disabled state by changing a cache from the disabled state to an enabled state for the first virtual persistent volume (156, 158, 280) based on the optimizer's (253) analysis of the first performance metrics (292, 294, 296). [9] The method (300) of claim 8, wherein the first virtual persistent volume (156, 158, 280) is composed of at least one underlying first storage volume (106, 108, 160, 162, 164, 272) by a data path (132) of the storage virtualization system (130), and wherein modifying the first virtual persistent volume (156, 158, 280) by the optimizer (253) comprises: Selecting a given storage profile from the various storage profiles (262) for different storage types, respectively, based on the analysis of the first performance metrics (292, 294, 296) identified by the first application data storage profile, and Providing a second underlying storage volume (106, 108, 160, 162, 164, 272) using the given storage profile to include it in the first virtual persistent volume (156, 158, 280), wherein, prior to changing the first virtual persistent volume (156, 158, 280), the first virtual persistent volume (156, 158, 280) is composed of the first underlying storage volume (106, 108, 160, 162, 164, 272) and a third underlying storage volume (106, 108, 160, 162, 164, 272), wherein a combination of a parameter of a first storage profile (262-1) for the first underlying storage volume (106, 108, 160, 162, 164, 272) and a parameter of a second storage profile (262-1) for the third underlying storage volume (106, 108, 160, 162, 164, 272) is a parameter for a storage class (226) of the first virtual persistent volume (156, 158, 280), and wherein the different storage profiles (262) comprise the first and second storage profiles (262-1, 262-2). [10] The method (300) of claim 9, wherein the values ​​of the first performance metrics (292, 294, 296) identified by the first application data storage profile indicate that input / output operations directed to the first virtual persistent volume (156, 158, 280) include more read operations than write operations, and wherein selecting the given storage profile includes selecting a storage profile that is more favorable for read operations than for write operations. [11] Method (300) according to claim 8, comprising: Determining (404), by a monitoring system (290) based on the first application data storage profile (284-1), which performance metrics (292, 294, 296) from the available metrics are for optimizing the first virtual persistent volume (156, 158, 280), and Collecting the values ​​of the first performance metrics (292, 294, 296) identified by the first application based on the determination by the monitoring system (290). [12] The method (300) of claim 8, further comprising: Controlling whether the reception and modification should be performed based on whether a storage class (226) of the first virtual persistent volume (156, 158, 280) is of a dynamic type. [13] The method (300) of claim 9, wherein selecting the given storage profile based on the analysis of the first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) comprises determining that the given storage profile is better suited than another storage profile of the various storage profiles (262) for how the containerized application (124, 126, 224) uses the first virtual persistent volume (156, 158, 280). [14] The method (300) of claim 8, wherein the first performance metrics (292, 294, 296) determined by the first application data storage profile (284-1) include input / output per second and latency. [15] A system (500) comprising: a processing resource (102); and a non-transitory machine-readable medium (104) storing instructions (105) for a storage virtualization system (130) that, when executed, cause the processing resource (102) to: to identify a first application data storage profile (284-1) from a plurality of preconfigured application data storage profiles (284), each containing preconfigured storage parameters for a corresponding application type, based on a comparison of information in the plurality of application data storage profiles (284) with information in an application manifest (282) of a containerized application (124, 126, 224) assigned a first virtual persistent volume (156, 158, 280), wherein the information in the application manifest (282) compared to the information in the plurality of application data storage profiles (284) includes a name of a container image on which the containerized application (124, 126, 224) is based, and wherein each identified application data storage profile of the plurality of application data storage profiles (284) identifies a respective collection of performance metrics (292, 294, 296),relevant for optimizing virtual persistent volumes for the respective application type; receive values ​​of first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) for use by an optimizer (253) in optimizing the first virtual persistent volume (156, 158, 280), wherein the first performance metrics (292, 294, 296) determined by the first application data storage profile (284-1) relate to the use of the first virtual persistent volume (156, 158, 280) by the containerized application (124, 126, 224) and are a subset of the available metrics to be collected; and modifying the first virtual persistent volume (156, 158, 280) by the optimizer (253) based on an analysis of the first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1), wherein modifying the first virtual persistent volume (156, 158, 280) comprises: enabling caches from a disabled state by changing a cache from the disabled state to an enabled state for the first virtual persistent volume (156, 158, 280) based on the optimizer's (253) analysis of the first performance metrics (292, 294, 296). [16] The system (500) of claim 15, wherein the first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) comprise storage performance metrics of a data path (132) for the first virtual persistent volume (156, 158, 280) and not other virtual persistent volumes, and wherein the storage performance metrics comprise input / output per second and latency. [17] The system (500) of claim 15, wherein the first virtual persistent volume (156, 158, 280) is composed of at least one first underlying storage volume (106, 108, 160, 162, 164, 272) by a data path (132) of the storage virtualization system (130), and wherein modifying the first virtual persistent volume (156, 158, 280) by the optimizer (253) comprises: Selecting (410) a given storage profile from various storage profiles (262) for different storage types, respectively, based on the analysis of the first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1), and Providing (412) a second underlying storage volume (106, 108, 160, 162, 164, 272) using the given storage profile to include it in the first virtual persistent volume (156, 158, 280), and where the different storage types include local storage, remote storage, and virtualized storage. [18] The system (500) of claim 15, wherein the first virtual persistent volume (156, 158, 280) is composed of at least one first underlying storage volume (106, 108, 160, 162, 164, 272) by a data path (132) of the storage virtualization system (130), and wherein modifying the first virtual persistent volume (156, 158, 280) by the optimizer (253) comprises: Selecting (410) a given storage profile of the various storage profiles (262) from various storage profiles (262) for respective different storage types based on the analysis of the first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1), and Providing (412) a second underlying storage volume (106, 108, 160, 162, 164, 272) using the given storage profile to include it in the first virtual persistent volume (156, 158, 280), wherein, prior to changing the first virtual persistent volume (156, 158, 280), the first virtual persistent volume (156, 158, 280) is composed of the first underlying storage volume (106, 108, 160, 162, 164, 272) and a third underlying storage volume (106, 108, 160, 162, 164, 272), wherein a combination of a parameter of a first storage profile (262-1) for the first underlying storage volume (106, 108, 160, 162, 164, 272) and a parameter of a second storage profile (262-2) for the third underlying storage volume (106, 108, 160, 162, 164, 272) defines a parameter for a storage class (226) of the first virtual persistent volume (156, 158, 280), and wherein the different storage profiles (262) comprise the first and second storage profiles (262-1, 262-2). [19] The system (500) of claim 18, wherein selecting the given storage profile based on the analysis of the first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) comprises determining that the given storage profile is better suited than another storage profile of the various storage profiles (262) for how the containerized application (124, 126, 224) uses the first virtual persistent volume (156, 158, 280). [20] The system (500) of claim 18, wherein the values ​​of the first performance metrics (292, 294, 296) identified by the first application data storage profile (284-1) indicate that input / output operations directed to the first virtual persistent volume (156, 158, 280) include more read operations than write operations, and wherein selecting the given storage profile includes selecting a storage profile that is better for read operations than for write operations.

Images