IMPROVED MANAGEMENT OF THE AVAILABILITY OF STORAGE REPOSITORIES IN A VIRTUAL ENVIRONMENT

Abstract

Procedure, comprehensive: Provisioning multiple storage repositories to a host, the provisioning comprising a single provisioning operation for each of the storage repositories of the multiple storage repositories to make a file system for the storage repository accessible in a host directory as a provisioning section; Identifying a requirement to initiate a plurality of virtual clusters on the host, wherein a first virtual cluster comprises multiple containers and a second virtual cluster comprises one or more containers, wherein the multiple containers of the first virtual cluster and the one or more containers of the second virtual cluster share resources of an operating system of the host; Identify, by the host, first permissions assigned to the first virtual cluster and second permissions assigned to the second virtual cluster; Determine, by the host, at least one first storage repository from the multiple storage repositories for the first virtual cluster based on the first permissions; and determine, by the host, at least one second storage repository from the multiple storage repositories for the second virtual cluster based on the second permissions; and based on the determination, initiation, by the host, of the execution of the multiple containers of the first virtual cluster in the host, wherein applications running in the multiple containers have access to at least one file system that is at least equivalent to the first storage repository, and initiation, by the host, of the execution of one or more containers of the second virtual cluster in the host, wherein one or more applications running in the one or more containers have access to at least one file system that is at least equivalent to the second storage repository.

Description

TECHNICAL BACKGROUND

[0001] A growing number of data-intensive distributed applications are being developed to meet various needs, such as processing very large amounts of data that are difficult for a single computer to handle. Instead, computer clusters are used to distribute various tasks, such as organizing and accessing the data and performing related operations on the data. Various large-scale processing applications and frameworks have been developed to interact with such large amounts of data, including Hive, HBase, Hadoop, and Spark.

[0002] At the same time, virtualization techniques have gained popularity and are now common in data centers and other computing environments where it is useful to increase the efficiency with which computing resources are used. In a virtualized environment, one or more virtual nodes are instantiated on an underlying physical computer and share the underlying computer's resources. Accordingly, instead of implementing a single node per host computer system, multiple nodes can be deployed on a host to more efficiently utilize the computer system's processing resources. These virtual nodes can include full operating system virtual machines, containers such as Linux containers or Docker containers, jails, or other similar types of virtual containment nodes.Although virtualization techniques offer greater efficiency in computing environments, difficulties often arise in allocating resources to individual virtual nodes. Particularly as the number of virtual nodes in an environment increases, managing the provisioning and administration of data resources for each of the virtual nodes can be difficult and cumbersome. US 2016 / 0 359 955 A1 describes systems for high-performance computing with a memory control architecture implemented by a plurality of nodes, where a node comprises combinations of executable containers running in cooperation with virtual machines running on top of a hypervisor. US 7 801 894 B1 describes a harvester for collecting metadata of managed objects across file systems, which are generally not interoperable in an enterprise environment.WO 2012 / 100092 A2 describes improvements for a virtualization environment adapted for the development and deployment of at least one software workload, wherein the virtualization environment has a metamodel framework that enables the assignment of a policy to the software workload during development of the workload, which policy is applied during deployment of the software workload. US 2009 / 0043881 A1 describes techniques for efficiently notifying servers of cache expiration indications that their local cache data is stale and should not be used. SUMMARY

[0003] The technology described herein improves the management of the availability of storage repositories in a virtual computing environment. In one implementation, a host provisions one or more storage repositories and identifies a request to initiate a virtual cluster on the host. In response to the request, the host identifies the permissions associated with the virtual cluster, determines at least one storage repository from one or more storage repositories associated with the virtual cluster, and initiates execution of the virtual cluster with access to a file system associated with the at least one storage repository. It is an object of the invention to propose a method, a computing device, and a non-transitory computer-readable storage medium for improved management of the availability of storage repositories in a virtual environment.This object is achieved by a method according to the invention according to claim 1, a computing device according to the invention according to claim 8 and a non-volatile computer-readable storage medium according to the invention according to claim 14. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows a computing environment to improve the allocation of storage resources to virtual clusters according to one implementation. Fig. Figure 2 shows an operation of a host to improve the allocation of storage resources to virtual clusters according to one implementation. Fig. 3 shows an operational scenario for managing storage resources for virtual clusters according to one implementation. Fig. Figure 4A shows a directory view for a host according to one implementation. Fig. Figure 4B shows a directory view for a virtual node according to one implementation. Fig. 5A - 5B illustrate an operational scenario for mounting a new storage repository on a host according to one implementation. Fig. 6 shows a management computer system according to one implementation. Fig. 7 shows a host computer system according to one implementation. DETAILED DESCRIPTION

[0004] Fig. 1 shows a computing environment 100 for improving the allocation of storage resources to virtual clusters according to one implementation. Computing environment 100 includes hosts 110-111 and available storage 115. Host 110 includes virtual nodes 120-122 and host directory 130, and host 111 includes virtual nodes 123-124 and host directory 131. Available storage 115 includes storage repositories 140-142, each of which may include its own file system, a distributed file system, or other similar data storage repository. Available storage 115 may be stored on one or more computing systems, a storage area network (SAN), a networked attached storage unit, or other separate storage device accessible by hosts 110-111.

[0005] In operation, a management system (not shown) may identify virtual clusters for deployment on hosts 110-111 of computing environment 100. These virtual clusters may include virtual container nodes such as Docker containers, Linux containers, or other similar namespace-based containers. Instead of requiring a separate operating system required for virtual machines, containers may share resources from the host computing system, where the resources may include kernel resources from the host operating system and may further include repositories and other approved resources that may be shared with other containers or processes running on the host.Although resources can be shared between containers on a host, containers are deployed with their own identifier space and have private access to the operating system, file system structure, and network interfaces. The operating system may also be responsible for allocating processing resources, storage resources, network resources, and other similar resources to the containerized endpoint.

[0006] In the present implementation, virtual nodes 120-124 may be running as part of one or more virtual clusters that require access to storage repositories 140-142. To provide access to the various repositories, hosts 110-111 may mount at least a portion of storage repositories 140-142 and may provide access to storage repositories 140-142 based on the permissions assigned to the individual cluster. In some implementations, when mounting the storage repositories, the file system for the mounted directory in the host directory may be accessible. For example, when storage repositories 140-141 are mounted on host 110, the file systems for the corresponding repository become available in host directory 130 as mount sections 151-152.Similarly, when storage repository 142 is deployed on host 111, the file system for storage repository 142 becomes available as mount sections 154. When a cluster is deployed on a host, the host may identify mount sections for the cluster to access and allow the file system of each virtual node to access the required mount sections. For example, virtual nodes 120-121 may comprise a first cluster with permissions to access data in storage repositories 140-141. Therefore, when containers are initiated on host 110, the host operating system or other platform for virtual nodes 120-121 may include mount sections 151-152 in the file system that are available to the virtual nodes.Additionally, virtual nodes 123-124 may represent a secondary virtual cluster that has been assigned permissions to access storage repository 142. Consequently, when virtual nodes 123-124 are deployed on host 111, the directory for the virtual nodes may include the mount portion 154 corresponding to storage repository 142. Once access to the corresponding storage repository has been provided, processes in the container may access data in the repository as needed. Advantageously, by deploying the storage repositories on the host and forwarding access to the storage repositories to the associated virtual nodes, a single deployment operation may be performed instead of deploying for each individual node of a processing cluster.This can improve the speed and efficiency of repository deployment because the mount can occur before the virtual nodes are initiated on a host and may only require a single mount for each repository on the host.

[0007] In some implementations, the clusters deployed on hosts 110-111 may comprise large data processing clusters capable of processing data in parallel with other virtual nodes in the same cluster. Consequently, the storage repositories required for access may include distributed file systems or other large data repositories accessible by multiple virtual nodes to efficiently process the data.

[0008] Fig. 2 shows an operation 200 of a host to improve the allocation of storage resources to virtual clusters according to one implementation. The operational processes 200 are described in the following sections with reference to systems and elements of the computing environment 100 of Fig. 1 described in brackets.

[0009] As illustrated, operation 200, which may be performed by either host 110 or host 111, provisions one or more storage repositories on the host (201). Upon mounting the storage repositories to the host, addressing may be provided to the host to communicatively couple the host to the storage repository, where the addressing may include an Internet Protocol (IP) address, a network path, or some other similar addressing information. In addition, the host may further provide permissions for the host to access the desired storage repository. For example, if storage repository 140 is to be provisioned on host 110, host 110 may provide address information for accessing the storage repository and may further provide credentials for accessing the storage repository.The credentials may include a username, password, token, key, or other similar credential for accessing storage repository 140. Once the information has been provisioned for the storage repository, the file system for the repository may be made available in the file system of the corresponding host. Referring to the example in computing environment 100, when storage repository 140 is provisioned on host 110, a provision section 151 is added to host directory 130 corresponding to the file system for the newly available storage repository. Similarly, when storage repository 142 is provisioned on host 111, an associated provision section 154 is made available in host directory 131 corresponding to host 111.Although the example computing environment 100 shows storage repositories deployed on separate hosts, in some examples, the same storage repository may be deployed on multiple hosts. For example, storage repository 142 may be deployed on both host 110 and host 111.

[0010] Once the storage repositories are deployed on a respective host, the host may identify (202) a request to initiate a virtual cluster at least partially on the host. For example, a management system or service associated with computing environment 100 may identify requests to initiate a virtual cluster and determine one or more hosts capable of supporting the request for the cluster. After the hosts capable of supporting the virtual cluster are identified, a notification may be generated for the host indicating the request for the virtual cluster.In response to the request for the virtual cluster, operation 200 further identifies (203) permissions associated with the virtual cluster and determines (204) at least one storage repository from the one or more storage repositories for the virtual cluster based on the permissions.

[0011] In some implementations, hosts 110-111 may manage permissions for different tenants (organizations or departments of an organization) or users of computing environment 100, where each of the tenants may have access to different ones of the storage repositories. Therefore, when a first tenant generates a request for a new cluster, the first tenant may be granted access to a first set of one or more repositories. However, when a second tenant generates a request for a new cluster, the second tenant may be provided with a second set of one or more repositories. In some examples, different tenants may share one or more of the repositories. Thus, a first tenant may have access to storage repository 141, while a second tenant may have access to the same storage repository.In some implementations, each of the hosts may maintain one or more data structures, such as tables, linked lists, or other similar data structures, that can match authorization information with one or more repositories. When a request for a new cluster is received, the host may identify authorization information associated with the request (tenant ID, user ID, password, token, etc.) and compare the authorization information with the data structure to identify one or more repositories to make available to the cluster.

[0012] In some implementations, the management system or service for the computing environment 100 may be used to generate the various permissions for the deployed cluster. When provisioning the permissions, the management service may specify one or more repositories to make available to each of the tenants. This permission information may be maintained in one or more tables, trees, or other data structures. For example, when the tenant issues a request for a new cluster, the management system may consult one or more data structures to determine which repositories to make available based on the tenant identifier.In some implementations, when generating the request for a new cluster, the user or administrator associated with the new cluster may define the specific repositories to be made available to the cluster. Thus, the permissions can provide one or more repository IDs with additional credentials associated with the one or more repositories for the new cluster.

[0013] Once at least one storage repository is designated for the new cluster, operation 200 further initiates (205) execution of the virtual cluster with access to a file system corresponding to the at least one storage repository. As described herein, the platform or operating system associated with the containers may define resources made available to the virtual nodes when virtual nodes are deployed as containers. Thus, a new container may be provided with access to the operating system kernel, and operation 200 may further ensure that processes in each of the virtual nodes can access data from required storage repositories associated with the containers.Referring to the example of virtual nodes 120-121, virtual nodes 120-121 may be granted access to mount parts 151-152 corresponding to the file systems of storage repositories 140-141.

[0014] In at least one implementation, when provisioning a cluster, the administrator or user associated with the request can select an image for the virtual cluster, where the image can contain the necessary configuration and deployment files for the application(s) running on the virtual nodes of the cluster. For example, a user can select an image corresponding to a large-scale processing framework such as Hadoop or Spark, where the framework can be deployed as one or more virtual nodes capable of processing data in parallel. Once the image has been selected, along with other resource requirements (processing, memory, storage repositories, and the like), the virtual nodes can be initialized with additional resources allocated or provisioned by the associated host.These additional resources may include providing access to storage repositories associated with the cluster request or providing access to other files available on the host.

[0015] Fig. 3 shows an operational scenario 300 for managing storage resources for virtual clusters according to one implementation. The operational scenario 300 includes the host 310 and the available storage 315. The host 310 includes the virtual nodes 320-322 and the host directory 330, where the host directory 330 contains the host section 350 and the mount sections 351-353. The available storage 315 further includes storage repositories 340-342, where the storage repositories 340-342 correspond to the mount sections 351-353.

[0016] In operation, host 310 may mount storage repositories 340-342 in step 1 so that the corresponding mount sections 351-353 become available in host directory 330. This mount operation may include retrieving the necessary addressing and / or credential information to access storage repositories 340-342 and retrieving the addressing and / or credential information once to mount the file systems associated with the repositories as mount sections 351-353. Once the repositories are mounted on host 310, virtual nodes of clusters 360-361 may be initiated, where each of the clusters may be provided with authorization information to access one or more of storage repositories 340-342 in step 2.

[0017] Referring to the example of the first cluster 360, when virtual nodes 320-321 are initiated, they may be assigned credentials to access storage repositories 340-341. These credentials may include a tenant ID associated with the first cluster 360, definitions of the requesting administrator specifying the required repositories, or other similar credentials specifying the required storage repositories. As a result, the directories for virtual nodes 320-321 may include access to mount portions 351-352 shared on or by host 310.Although mount portions 351-352 are made available to virtual nodes 320-321, virtual nodes 320-321 may not be able to identify that storage repository 342 is mounted on the host or made available to other virtual nodes running on the host. In contrast, the second cluster 361 with virtual node 322 is associated with storage repository 342 and can access the data using mount portion 353 made accessible in the file system directory for virtual node 322. Because storage repositories 340-341 are not associated with virtual node 322, host 310 does not grant permissions to access or view mount portions 351-352.

[0018] Although the previous example described deploying the repositories on host 310 prior to initiating the virtual nodes, in some examples, storage repositories may be deployed during the execution of the virtual nodes. For example, the first cluster 360 may be initiated with access to storage repository 340 using mount portion 351. However, after initiation, storage repository 341 may be deployed on host 310, and permissions may be assigned to the first cluster 360 to provide access to the cluster. As a result, the operating system or other platform for virtual nodes 320-321 may modify the file system of virtual nodes 320-321 so that the virtual nodes can access the new mount portion 352 corresponding to storage repository 341.

[0019] Fig. 4A shows a directory view for a host according to one implementation. Directory view 401 includes root directory 405, directories 410-413, and files 420-428. Directory 410 contains files 420-424 representing the contents of host section 350 from operational scenario 300 of FIG. 3. Directories 411-413 each correspond to one of mount sections 351-353, where mount sections 351-353 represent the file systems for storage repositories 340-342 of operational scenario 300 of FIG. 3.

[0020] In operation, a virtual node host may provide storage repositories that may be made available to individual virtual nodes. Here, three different storage repositories are mounted on the host so that the host can access files and directories located in the repositories. These portions of directory view 401 contain directories 411-413, each corresponding to one of the storage repositories 340-342. After the various repositories are mounted or attached to the host, virtual nodes may be initiated, where the virtual nodes share kernel resources provided by the host and represented as host section 350, and may further be assigned access to one or more resources, such as the file systems for mounted storage repositories.

[0021] Fig. 4B shows a directory view 402 for a virtual node according to one implementation. The directory view 402 is representative of a directory view for the virtual node 320 of the operational scenario 300 of FIG. 3. The directory view 402 includes the root directory 405, the directories 410-412, and the files 420-428. The directory 410 includes files 420-424 representing the contents of the host section 350 from the operational scenario 300 of FIG. 3. The directories 411-412 each correspond to one of the mount sections 351-352, where the mount sections 351-352 represent the file systems for the storage repositories 340-341 of the operational scenario 300 of FIG. 3. Although not shown in the example of Figure 4B, it should be understood that each virtual node can have its own unique files and directories in addition to those of the shared host files and storage repository files.These files can be used to configure and run one or more applications or processes on a single virtual node. Furthermore, it should be understood that the virtual node may only have access to a portion of the host's data, including at least the kernel, even though it is represented by all the files from the host.

[0022] As described herein, for virtual node 320, the host may determine the permissions associated with the nodes and provide access to storage repositories based on the permissions. Because virtual nodes 320 are assigned permissions to access storage repositories 340-341, directory view 402 may include mount parts 351-352 corresponding to repositories 340-341. However, while making the file systems for storage repositories 340-341 accessible to the corresponding nodes, the host may ensure that storage repository 342 cannot be accessed or identified as mounted on the host. In particular, the host may ensure that while the repository is mounted and available in the host directory, the repository is not available in virtual nodes without permission to access the repository.

[0023] In some implementations, the repositories and resources assigned to virtual nodes may be dynamic during the execution of the virtual nodes. For example, one or more storage repositories assigned to a cluster may be added, removed, or otherwise changed during the execution of the cluster. For example, virtual node 320 is initially provided access to provisioning section 351, but an administrator or user associated with the cluster may change the cluster's permissions. Consequently, the host may revoke permissions to access the repository, making the data from the repository inaccessible to the virtual node. Similarly, when a new repository is mounted to the host, the permissions for the host may be changed to allow one or more virtual nodes running on it to access the new repository.These permissions can add the deployed repository to the file system directory of the one or more virtual nodes, allowing applications on the one or more virtual nodes to access the required files and directories.

[0024] In some implementations, each of the storage repositories may be associated with one or more tenants, where the tenants may include different organizations or different segments of an organization. For example, tenants for an organization may include a development tenant, a financial tenant, a legal tenant, and the like. Consequently, when generating a request for a new cluster, the host may identify a tenant associated with the request and designate one or more repositories associated with the tenant. Once the repositories are identified, the repositories may be made available in the file system directory of each of the nodes associated with the cluster.In some examples, when a cluster request is identified by the host, the host may identify permissions associated with the request, where the permissions may include a tenant identifier, tenant password, key, token, or other similar permissions. The permissions may then be used to identify one or more repositories to assign to the cluster.

[0025] Figures 5A-5B illustrate an operational scenario for mounting a new storage repository on a host according to one implementation. Fig. 5A includes host 510 and available storage 515. Host 510 includes virtual nodes 520-523 and host directory 530 with host portion 550 and mount portions 551-553. Virtual nodes 520-521 correspond to first cluster 560, while virtual nodes 522-523 correspond to second cluster 561. Available storage 515 includes storage repositories 540-543, which may represent various file systems and distributed file systems that may be provided by one or more devices. In FIG. 5A, storage repositories 540-542 are attached or mounted to host 510, with the file systems represented as mount portions 551-553. In Figure 5B, storage repositories 540-543 are attached or mounted, and an additional file system is shown in host directory 530 as mount section 554.

[0026] Referring to Figure 5A, storage repositories 540-542 are initially deployed on host 510 to provide data resources to virtual nodes 520-523, where virtual nodes 520-523 are representative of nodes operating in clusters 560-561 to provide various data operations to the data in storage repositories 540-542. To make the data available, host 510 may initiate a mount operation for each of the storage repositories, where the mount operation attaches the file system of the storage repositories and makes the file systems available from a mount point on the host system. Here, the file systems are represented in host directory 530 as mount parts 551-553.

[0027] Once the repositories are deployed on host 510, host 510 can make the repositories available to virtual nodes running on host 510. When determining which of the repositories to assign to which of the virtual nodes, each parent cluster associated with the virtual node can be assigned permissions that determine the repositories for each node. These permissions can be set based on the tenant associated with the cluster (e.g., accounting department, development department, etc.), can be based on a token provided during cluster generation, can be based on a username and password provided during cluster generation, or can be provided in another similar manner.As shown in Figure 5A, the first cluster 560 is provided with access to the storage repositories 540-541, while the second cluster 561 is provided with access to the storage repository 543. As a result, when the virtual nodes are provisioned, each of the virtual nodes may be provided with file system access to the corresponding mount sections of the mount sections 551-553. In particular, the virtual nodes 520-521 may each have access permissions to the mount sections 551-552, and the virtual nodes 522 may access the mount section 553.

[0028] In Fig. 5B shows Fig. 5B, when a new storage repository 543 is mounted on the host 510. Similar to the operations described above for the storage repositories 540-542, when the host 510 initiates a mount operation of the storage repository 543, the repository is mounted to the host and the contents of the file system are made available in the host directory 530, with the contents associated with the storage repository 543 shown as a mount portion 554 in the host directory 543. Once mounted, the host can determine whether one or more of the virtual nodes are associated with the newly mounted storage repository. This determination may be based on the tenant associated with the cluster, username or password information associated with the cluster, or other similar authorization information.For example, when the storage repository is deployed, the repository may be associated with permission information that specifies tenants, clusters, or another classifier that can identify virtual nodes that can access and process data from the repository.

[0029] In the example of Figure 5B, host 510 determines that second cluster 561 includes the necessary permissions to access storage repository 543. As a result, host 510 may add a mount portion 554 to the directories for each of virtual nodes 522-523, allowing the virtual nodes to access the necessary files and directories from storage repository 543. When accessing the repository, the permissions may further specify which of the clusters or tenants may read from the repository, which of the clusters or tenants may write to the repository, or which of the tenants or clusters may perform another operation with respect to the data in the repository.

[0030] Fig. 6 shows a management computer system 600 according to one implementation. Computer system 600 is representative of any computer system or systems with which the various operating architectures, processes, scenarios, and sequences disclosed herein for a management system may be implemented. Computer system 600 is an exemplary management system that may be used to initiate and configure clusters on host systems, as described herein. Computer system 600 includes communications interface 601, user interface 602, and processing system 603. Processing system 603 is connected to communications interface 601 and user interface 602. Processing system 603 includes processing circuitry 605 and a storage device 606 that stores operating software 607.The computer system 600 may include other known components such as a battery and a housing, which are not shown for clarity.

[0031] The communication interface 601 includes components that communicate over communication links, such as network cards, ports, radio frequency (RF), processing circuits, and software or some other communication devices. The communication interface 601 may be configured to communicate over metallic, wireless, or optical connections. The communication interface 601 may be configured to use time-division multiplexing (TDM), Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or another communication format—including combinations thereof. In at least one implementation, the communication interface 601 may be used to communicate with one or more hosts of a computing environment, where the hosts execute virtual nodes to provide various processing operations.

[0032] User interface 602 includes components that interact with a user to receive user input and present media and / or information. User interface 602 may include a speaker, a microphone, buttons, lights, a display screen, a touchscreen, a touchpad, a scroll wheel, a communications port, or other user input / output device—including combinations thereof. User interface 602 may be omitted in some examples.

[0033] Processing circuitry 605 includes a microprocessor and other circuitry that retrieves and executes operating software 607 from storage device 606. Storage device 606 may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Storage device 606 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems. Storage device 606 may include additional elements, such as a controller for reading operating software 607.Examples of storage media include random-access memory, read-only memory, magnetic disks, optical disks, and flash memory, as well as any combination or variation thereof, or any other type of storage media. In some implementations, the storage medium may be a non-transitory storage medium. In some cases, at least a portion of the storage medium may be transient. In no case is the storage medium a propagated signal.

[0034] Processing circuitry 605 is typically mounted on a printed circuit board, which may also house storage device 606 and portions of communications interface 601 and user interface 602. Operating software 607 includes computer programs, firmware, or another form of machine-readable program instructions. Operating software 607 includes request module 608 and allocation module 609, although any number of software modules may provide similar operation. Operating software 607 may further include an operating system, utilities, drivers, network interfaces, applications, or another type of software. When executed by processing circuitry 605, operating software 607 instructs processing system 603 to operate computer system 600 as described herein.

[0035] In one implementation, management computer system 600 may be used to assign virtual clusters and the corresponding virtual nodes to one or more host systems in a computing environment. These host systems may include physical computing elements such as server computer systems, or may include virtual machines running on a physical computer system. In operation, requests module 608 may identify a request for a cluster, where the request may identify processing resources to be allocated to the cluster, storage resources to be allocated to the cluster, storage repositories to be made available to the cluster, the type of software to be made available to the cluster, the tenant associated with the cluster, or similar information about the cluster.In response to the request, the allocation module 609 may generate a request that is provided to one or more host computer systems to support the new virtual cluster. In some implementations, in addition to the request, the allocation module 609 may provide information about the resources (processing, memory, and the like) required by the cluster and may further include permission information for the storage repositories associated with the cluster. These permissions may identify the specific repositories available to the cluster, the tenant associated with the cluster, a key or token for accessing repositories, or other similar permissions that allow the one or more host computer systems to identify the storage repositories available to the cluster.

[0036] In some examples, management computer system 600 may further be used to dynamically change the configuration of a cluster, where the changes may include changing the repositories made available to the cluster. These changes may include requests to add a storage repository to an executing cluster, remove a storage repository from an executing cluster, or a similar operation.

[0037] Although shown as separate from a host computer system, it should be understood that the request module 608 and the allocation module 609 may be colocated on a host for the virtual nodes. For example, when a request is generated, the allocation module 609 may allocate the necessary processing and storage repository resources before the virtual nodes are initiated to support a cluster request.

[0038] Fig. 7 shows a host computer system 700 according to one implementation. Computer system 700 is representative of any computer system or systems with which the various operating architectures, processes, scenarios, and sequences disclosed herein for a host may be implemented. Computer system 700 is an example of hosts 110-111, 310, and 510, although other examples may exist. Computer system 700 includes communications interface 701, user interface 702, and processing system 703. Processing system 703 is coupled to communications interface 701 and user interface 702. Processing system 703 includes processing circuitry 705 and a storage device 706 that stores operating software 707. Computer system 700 may include other known components, such as a battery and a housing, which are not shown for clarity.

[0039] The communication interface 701 includes components that communicate over communication links, such as network cards, ports, radio frequency (RF), processing circuits, and software or some other communication devices. The communication interface 701 may be configured to communicate over metallic, wireless, or optical connections. The communication interface 701 may be configured to use time-division multiplexing (TDM), Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or another communication format—including combinations thereof. In at least one implementation, the communication interface 701 may be used to communicate with one or more other hosts of a computing environment and a management system, such as the management computer system 600, for configuration information related to executable virtual nodes.Communication interface 701 may further communicate with one or more storage repositories that may be located on other computer systems or storage systems. These storage repositories may be coupled to computer system 700 using Ethernet, Fibre Channel, Peripheral Component Interconnect Express (PCIe), or another similar connection, where the storage repository may be mountable and accessible by multiple hosts in a computing environment.

[0040] User interface 702 includes components that interact with a user to receive user input and present media and / or information. User interface 702 may include a speaker, a microphone, buttons, lights, a display screen, a touchscreen, a touchpad, a scroll wheel, a communications port, or other user input / output device—including combinations thereof. User interface 702 may be omitted in some examples.

[0041] Processing circuitry 705 includes a microprocessor and other circuitry that retrieves and executes operating software 707 from storage device 706. Storage device 706 may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Storage device 706 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems. Storage device 706 may include additional elements, such as a controller for reading operating software 707.Examples of storage media include random-access memory, read-only memory, magnetic disks, optical disks, and flash memory, as well as any combination or variation thereof, or any other type of storage media. In some implementations, the storage medium may be a non-transitory storage medium. In some cases, at least a portion of the storage medium may be transient. In no case is the storage medium a propagated signal.

[0042] The processing circuitry 705 is typically mounted on a printed circuit board, which may also house the storage device 706 and portions of the communications interface 701 and the user interface 702. The operating software 707 includes computer programs, firmware, or another form of machine-readable program instructions. The operating software 707 includes the data request module 708 and the virtual nodes 709, although any number of software modules may provide similar operation. The operating software 707 may further include an operating system, utilities, drivers, network interfaces, applications, or another type of software. When executed by the processing circuitry 705, the operating software 707 instructs the processing system 703 to operate the computer system 700 as described herein.

[0043] In one implementation, virtual nodes 709, which in some examples may include containers, are executed by processing system 703 to provide various data processing operations. In some examples, virtual nodes 709 may represent nodes operating in one or more clusters that provide large-scale data processing operations on data sets, where the data sets may be provided to the clusters as one or more storage repositories, each storing data in file systems or distributed file systems. To provide access to each of the storage repositories, management module 708 may provision one or more storage repositories on host 700, where provisioning the repositories enables the file systems associated with the repositories to be made available on the host.

[0044] Once the repositories are provisioned and the file systems are available to the host directory, clusters may be provisioned on the host 700, where the clusters may include one or more of the virtual nodes 709. When a cluster is to be provisioned, the management module 708 may determine one or more repositories to be made available to the cluster based on permissions associated with the request for provisioning. These permissions may explicitly define the repositories to be made available to the cluster, identify a tenant associated with the cluster, define a token or key associated with the cluster, or provide other information to define the repositories to be made available to the cluster.Once the repositories are identified, the management module 708 can initiate the cluster with each virtual node of the cluster granting access to the required storage repositories. When providing access to the required storage repositories, the host can traverse the file systems associated with the repositories, allowing the virtual nodes to access the data in the repositories. When traversing the file systems, a single mount can be performed for the host, while the file system for the repositories can be forwarded to the individual virtual nodes.

[0045] Although it was described above that access to data repositories is provided at the time of initiation, resources can be dynamically added to or removed from the cluster. In some implementations, resources may include the allocation of data repositories, where an administrator can request that a new repository be provisioned on host 700, which is then propagated to one or more clusters operating on the host. Similarly, an administrator may wish to remove access to a storage repository for one or more clusters. Therefore, instead of deregistering the repository from each of the virtual nodes, the repository can be deregistered from the host, preventing access from nodes that may have had pass-through access to the repository.This way, instead of maintaining a repository mount for each individual node of a cluster, the host can deploy the repository once and provide permissions and accessibility to the repository for each associated node on the host.

[0046] Back to the elements of Fig.1, hosts 110-111 may each include communication interfaces, network interfaces, processing systems, computer systems, microprocessors, storage systems, storage media, or some other processing devices or software systems, and may be distributed across multiple devices. Examples of hosts 110-111 may include software such as an operating system, protocols, databases, utilities, drivers, network software, and other software stored on a computer-readable medium. Hosts 110-111, in some examples, may include one or more server computer systems, desktop computer systems, laptop computer systems, or other computer systems, including combinations thereof. In some implementations, hosts 110-111 may include virtual machines that include abstracted physical computing elements and an operating system that may provide a platform for the virtual nodes.In these examples, hosts 110-111 may be running on the same physical computer system or may be running on multiple computer systems.

[0047] The available memory 115 may be located on any computer system or all computer systems, which may include communication interfaces, network interfaces, processing systems, computer systems, microprocessors, storage systems, storage media, or some other processing devices or software systems, and may be distributed across multiple devices. The available memory 115 may be located on one or more server computer systems, desktop computer systems, laptop computer systems, or other computer systems, including combinations thereof.

[0048] Communication between hosts 110-111 and available storage 115 may use metal, glass, optics, air, space, or another material as the transport medium. Communication between hosts 110-111 and available storage 115 may use various communication protocols, such as time-division multiplexing (TDM), asynchronous transfer mode (ATM), Internet Protocol (IP), Ethernet, synchronous optical network (SONET), hybrid fiber coax (HFC), circuit-switched, communication signaling, wireless communication, or another communication format, including combinations, enhancements, or variations thereof. Communication between hosts 110-111 and available storage 115 may be a direct connection or may involve intermediate networks, systems, or devices, and may include a logical network connection transported over multiple physical links.In some implementations, hosts 110-111 may communicate with available storage 115 using Ethernet. However, it is understood that communication may include PCIe, Fibre Channel, or another similar communication protocol.

[0049] The included descriptions and figures depict specific implementations to teach those skilled in the art how to make and use the best mode. To convey inventive principles, some conventional aspects have been simplified or omitted. Those skilled in the art will appreciate variations of these implementations that fall within the scope of the invention. Those skilled in the art will also recognize that the features described above can be combined in various ways to form multiple implementations. Consequently, the invention is not limited to the specific implementations described above, but only to the claims and their equivalents.

Claims

[1] Method comprising: Providing multiple storage repositories to a host, the provisioning comprising a single mount operation for each of the storage repositories of the multiple storage repositories to make a file system for the storage repository accessible in a host directory as a mount section; Identifying a request to initiate a plurality of virtual clusters on the host, wherein a first virtual cluster comprises a plurality of containers and a second virtual cluster comprises one or more containers, wherein the plurality of containers of the first virtual cluster and the one or more containers of the second virtual cluster share resources of an operating system of the host; identifying, by the host, first permissions associated with the first virtual cluster and second permissions associated with the second virtual cluster; Determining, by the host, the at least one first storage repository from the plurality of storage repositories for the first virtual cluster based on the first permissions and determining, by the host, at least one second storage repository from the plurality of storage repositories for the second virtual cluster based on the second permissions; and based on determining, initiating, by the host, execution of the plurality of containers of the first virtual cluster in the host, wherein applications executing in the plurality of containers have access to at least one file system corresponding to at least the first storage repository, and initiating, by the host, execution of the one or more containers of the second virtual cluster in the host, wherein one or more applications executing in the one or more containers have access to at least one file system corresponding to at least the second storage repository. [2] The method of claim 1, wherein the host comprises a virtual machine in which the first virtual cluster with the plurality of containers and the second virtual cluster with the one or more containers are deployed. [3] The method of any one of claims 1 to 2, wherein the one or more storage repositories comprise one or more distributed file systems. [4] The method of any one of claims 1 to 3, further comprising the host identifying at least a first mount section in the host accessible to the first virtual cluster to access at least the first storage repository, and the host identifying at least a second mount section in the host accessible to the second virtual cluster to access at least the second storage repository. [5] The method of any one of claims 1 to 4, wherein the first credentials associated with the first virtual cluster comprise a first tenant ID, and wherein determining the at least one first storage repository from the plurality of storage repositories for the first virtual cluster is based on the first tenant ID, and wherein the second credentials associated with the second virtual cluster comprise a second tenant ID, and wherein determining the at least one second storage repository from the plurality of storage repositories for the second virtual cluster is based on the second tenant ID. [6] The method of claim 5, further comprising: Providing at least one additional storage repository on the host; Determining whether the at least one additional storage repository is associated with the first virtual cluster or the second virtual cluster; and if the additional storage repository is associated with the first virtual cluster or the second virtual cluster, providing access, by the first virtual cluster or the second virtual cluster, to at least one file system corresponding to the at least one additional storage repository. [7] The method of any one of claims 1 to 6, wherein the one or more storage repositories are accessible from the host via Ethernet, Fibre Channel or Peripheral Component Interconnect Express (PCIe). [8] A computing device comprising: one or more non-transitory computer-readable storage media; a processing system operatively coupled to the one or more non-transitory computer-readable storage media; and Program instructions stored on one or more non-transitory computer-readable storage media which, when executed by the processing system, instruct the processing system to: Providing multiple storage repositories to a host, the provisioning comprising a single mount operation for each of the storage repositories of the one or more storage repositories to make a file system for the storage repository accessible in a host directory as a mount section; Identifying a request to initiate a plurality of virtual clusters on the host, wherein a first virtual cluster comprises a plurality of containers and a second virtual cluster comprises one or more containers, wherein the plurality of containers of the first virtual cluster and the one or more containers of the second virtual cluster share resources of an operating system of the host; Identifying first permissions associated with the first virtual cluster and second permissions associated with the second virtual cluster; Determining at least one first storage repository from the plurality of storage repositories for the first virtual cluster based on the first permissions, and determining at least one second storage repository from the plurality of storage repositories for the second virtual cluster based on the second permissions; and based on the determining, initiating execution of the plurality of containers of the first virtual cluster in the host, wherein applications executing in the plurality of containers have access to at least one file system corresponding to at least the first storage repository, and initiating, by the host, execution of the one or more containers of the second virtual cluster in the host, wherein one or more applications executing in the one or more containers have access to at least one file system corresponding to at least the second storage repository. [9] The computing device of claim 8, wherein the host comprises a virtual machine in which the first virtual cluster with the plurality of containers and the second virtual cluster with the one or more containers are deployed. [10] The computing device of any of claims 8 or 9, wherein the one or more storage repositories comprise one or more distributed file systems. [11] The computing device of any one of claims 8 to 10, wherein the first credentials associated with the first virtual cluster comprise a first tenant ID, and wherein determining the at least one first storage repository for the first virtual cluster from the plurality of storage repositories is based on the first tenant ID, and wherein the second credentials associated with the second virtual cluster comprise a second tenant ID, and wherein determining the at least one second storage repository from the plurality of storage repositories for the second virtual cluster is based on the second tenant ID. [12] The computing device of any one of claims 8 to 11, wherein the program instructions further instruct the processing system to: Provide at least one additional storage repository on the host; Determining which of the first virtual cluster or the second virtual cluster is associated with the additional storage repository; in response to determining that the first virtual cluster is associated with the additional storage repository, providing access of the first virtual cluster to a file system corresponding to the additional storage repository. [13] The computing device of any one of claims 8 to 12, wherein the one or more storage repositories are accessible from the host via Ethernet, Fibre Channel, or Peripheral Component Interconnect Express (PCIe). [14] A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to: Providing multiple storage repositories to a host, the provisioning comprising a single mount operation for each of the storage repositories of the multiple storage repositories to make a file system for the storage repository accessible in a host directory as a mount section; Identifying a request to initiate a plurality of virtual clusters on the host, wherein a first virtual cluster comprises a plurality of containers and a second virtual cluster comprises one or more containers, wherein the plurality of containers of the first virtual cluster and the one or more containers of the second virtual cluster share resources of an operating system of the host; in response to the request, identifying first permissions associated with the first virtual cluster and second permissions associated with the second virtual cluster; Determining at least one first storage repository from the plurality of storage repositories for the first virtual cluster based on the first permissions, and determining at least one second storage repository from the plurality of storage repositories for the second virtual cluster based on the second permissions; and based on the determining, initiating execution of the plurality of containers of the first virtual cluster in the host, wherein applications executing in the plurality of containers have access to at least one file system corresponding to at least the first storage repository, and initiating, by the host, execution of the one or more containers of the second virtual cluster in the host, wherein one or more applications executing in the one or more containers have access to at least one file system corresponding to at least the second storage repository. [15] The non-transitory computer-readable storage medium of claim 14, wherein the one or more storage repositories comprise one or more distributed file systems. [16] The non-transitory computer-readable storage medium of claim 14 or 15, wherein the first credentials associated with the first virtual cluster comprise a first tenant ID, and wherein determining the at least one first storage repository from the plurality of storage repositories for the first virtual cluster is based on the first tenant ID, and wherein the second credentials associated with the second virtual cluster comprise a second tenant ID, and wherein determining the at least one second storage repository from the plurality of storage repositories for the second virtual cluster is based on the second tenant ID. [17] The non-transitory computer-readable storage medium of any of claims 14 to 16, wherein the program instructions further include instructions instructing the processor to: Providing at least one additional storage repository on the host; Determining which of the first virtual cluster or the second virtual cluster is assigned to the additional storage repository; and in response to determining that the first virtual cluster is associated with the additional storage repository, providing access of the first virtual cluster to a file system corresponding to the additional storage repository.

Images